Commit Graph
4 Commits
Author SHA1 Message Date
Sean WhalenandClaude Fable 5 62514bd72a Add per-domain DMARC compliance percentage to all aggregate dashboards (#834)
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)

The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:

- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
  a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
  sum of message_count), pivoted on header_from.keyword. The time field
  is date_begin rather than the multi-valued date_range, which TSVB's
  per-value date histogram would double-count. Editing (not rendering)
  the panel on Kibana 8.x requires the metrics:allowStringIndices
  advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
  field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
  COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
  stats, per the SPL eval-in-stats syntax.

All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).

Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.

Closes #112

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix over-time charts double-counting reports via multi-valued date_range

date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.

Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.

All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.

Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review comments on PR #834

- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed  DMARC" -> "failed DMARC" and the
  backticked `ruf ` trailing space in the RUF explainer panel, and
  rename the "SMPT TLS failure details" visualization to "SMTP TLS
  failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
  plugin-install restart so a hang fails with a clear timeout message
  instead of an opaque downstream curl error.

The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* AGENTS.md: reviews must cover prose and hunk context, not just function

Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address second round of Copilot review comments

- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
  matching the repo convention where the release commit assigns the
  version number (see 855d267 for 10.2.4).
- docker-compose.yml: make the dev-stack Elasticsearch heap overridable
  via ES_JAVA_OPTS in .env (default unchanged at 2g), using the compose
  file's existing ${VAR:-default} idiom, for smaller machines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 19:23:55 -04:00
Sean WhalenandClaude Sonnet 5 1986835f51 Fix daily-cadence charts defaulting to sub-day time buckets (#828) (#830)
DMARC aggregate reports post one data point per reporting period
(typically daily), but several dashboard time-series panels used a
dynamically-computed bucket interval that scales with the viewed time
range/panel width instead of the data's actual cadence, producing a
spiked/gapped chart whenever that computed interval landed below a day:

- OpenSearch Dashboards: both date_histogram aggregations used
  "interval": "auto". Fixed to the "d" unit code (verified against
  OpenSearch-Dashboards' and Kibana's own _interval_options.ts /
  parse_interval.ts source — "day" is not a valid value and would throw
  at render time).
- Grafana + PostgreSQL: 7 query targets used
  $__timeGroup(col, $__interval); hardcoded to $__timeGroup(col, '1d'),
  matching the equivalent already-correct panels in the Grafana +
  Elasticsearch dashboard.
- Splunk: both `timechart` panels had no explicit span, so Splunk's
  100-bin auto-algorithm only coincidentally produced daily buckets for
  the dashboard's default 7-day view. Added span=1d per Splunk's own
  timechart documentation.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 20:17:57 -04:00
Sean WhalenandClaude Opus 4.7 e104f1118c Land 10.0.3 changes on master (#785)
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.

This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 13:54:40 -04:00
Sean WhalenandClaude Opus 4.7 2b3cd32b9c chore: move PostgreSQL Grafana dashboard into dashboards/grafana/ (#780)
The PostgreSQL dashboard shipped at the repo-root grafana/ by oversight;
every other dashboard source lives under dashboards/ (opensearch/,
grafana/, splunk/). Move it next to the existing Grafana dashboard, list
it in dashboards/README.md, and fix the CHANGELOG path reference.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:52:33 -04:00