Files
parsedmarc/docker-compose.dashboard-dev.yml
T
62514bd72a Add per-domain DMARC compliance percentage to all aggregate dashboards (#834)
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)

The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:

- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
  a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
  sum of message_count), pivoted on header_from.keyword. The time field
  is date_begin rather than the multi-valued date_range, which TSVB's
  per-value date histogram would double-count. Editing (not rendering)
  the panel on Kibana 8.x requires the metrics:allowStringIndices
  advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
  field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
  COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
  stats, per the SPL eval-in-stats syntax.

All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).

Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.

Closes #112

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix over-time charts double-counting reports via multi-valued date_range

date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.

Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.

All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.

Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review comments on PR #834

- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed  DMARC" -> "failed DMARC" and the
  backticked `ruf ` trailing space in the RUF explainer panel, and
  rename the "SMPT TLS failure details" visualization to "SMTP TLS
  failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
  plugin-install restart so a hang fails with a clear timeout message
  instead of an opaque downstream curl error.

The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* AGENTS.md: reviews must cover prose and hunk context, not just function

Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address second round of Copilot review comments

- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
  matching the repo convention where the release commit assigns the
  version number (see 855d267 for 10.2.4).
- docker-compose.yml: make the dev-stack Elasticsearch heap overridable
  via ES_JAVA_OPTS in .env (default unchanged at 2g), using the compose
  file's existing ${VAR:-default} idiom, for smaller machines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 19:23:55 -04:00

75 lines
2.2 KiB
YAML

name: parsedmarc-dashboards
include:
- docker-compose.yml
services:
kibana:
image: docker.elastic.co/kibana/kibana:8.19.7
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- "127.0.0.1:5601:5601"
depends_on:
elasticsearch:
condition: service_healthy
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3
environment:
- OPENSEARCH_HOSTS=["https://opensearch:9200"]
ports:
- "127.0.0.1:5602:5601"
depends_on:
opensearch:
condition: service_healthy
grafana:
image: grafana/grafana:latest
environment:
# Grafana reads GF_SECURITY_ADMIN_PASSWORD, not GRAFANA_PASSWORD. Default
# to "admin" so the login matches the bootstrap script's GRAFANA_PASSWORD
# default; set GRAFANA_PASSWORD in .env to change both in lockstep.
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
# Grafana >= 13 no longer bundles the Elasticsearch datasource, but
# installing it via GF_INSTALL_PLUGINS crash-loops: the image ships a
# root-owned plugins-bundled/elasticsearch remnant the installer cannot
# replace. The bootstrap script installs it via `grafana cli` instead.
- GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel
ports:
- "127.0.0.1:3000:3000"
depends_on:
elasticsearch:
condition: service_healthy
postgresql:
condition: service_healthy
postgresql:
image: postgres:17
environment:
- POSTGRES_USER=${POSTGRESQL_USER:-parsedmarc}
- POSTGRES_PASSWORD=${POSTGRESQL_PASSWORD:-parsedmarc}
- POSTGRES_DB=${POSTGRESQL_DB:-parsedmarc}
ports:
- "127.0.0.1:5432:5432"
healthcheck:
test:
[
"CMD-SHELL",
"pg_isready -U ${POSTGRESQL_USER:-parsedmarc} -d ${POSTGRESQL_DB:-parsedmarc}"
]
interval: 5s
timeout: 5s
retries: 20
splunk:
image: splunk/splunk:latest
environment:
- SPLUNK_START_ARGS=--accept-license
- "SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com"
- SPLUNK_PASSWORD=${SPLUNK_PASSWORD}
- SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
ports:
- "127.0.0.1:8000:8000"
- "127.0.0.1:8088:8088"