Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)

The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:

- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
  a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
  sum of message_count), pivoted on header_from.keyword. The time field
  is date_begin rather than the multi-valued date_range, which TSVB's
  per-value date histogram would double-count. Editing (not rendering)
  the panel on Kibana 8.x requires the metrics:allowStringIndices
  advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
  field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
  COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
  stats, per the SPL eval-in-stats syntax.

All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).

Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.

Closes #112

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-07-20 21:25:00 -04:00
co-authored by Claude Fable 5
parent 855d267650
commit 47fb50e76a
8 changed files with 163 additions and 46 deletions
+17 -6
View File
@@ -44,18 +44,29 @@ disposition over time.
Under the graphs you will find the most useful data tables on the dashboard. On
the left, there is a list of organizations that are sending you DMARC reports.
In the center, there is a list of sending servers grouped by the base domain
in their reverse DNS. On the right, there is a list of email from domains,
sorted by message volume.
in their reverse DNS. On the right, there is the "Message volume and DMARC
compliance by from domain" table, which lists email from domains with their
message volume and a percentage of those messages that passed DMARC.
By hovering your mouse over a data table value and using the magnifying glass
icons, you can filter on our filter out different values. Start by looking at
the Message Sources by Reverse DNS table. Find a sender that you recognize,
such as an email marketing service, hover over it, and click on the plus (+)
magnifying glass icon, to add a filter that only shows results for that sender.
Now, look at the Message From Header table to the right. That shows you the
domains that a sender is sending as, which might tell you which brand/business
is using a particular service. With that information, you can contact them and
have them set up DKIM.
Now, look at the Message volume and DMARC compliance by from domain table to
the right. That shows you the domains that a sender is sending as, and what
share of that traffic is passing DMARC, which might tell you which
brand/business is using a particular service. With that information, you can
contact them and have them set up DKIM.
:::{note}
The "Message volume and DMARC compliance by from domain" table is a TSVB
visualization, used because per-domain compliance percentages require a
Filter Ratio metric that agg-based data tables can't compute. It renders
correctly on Kibana 8.x as imported, but *editing* it requires first enabling
the `metrics:allowStringIndices` advanced setting, since it references the
`dmarc_aggregate*` index as a string pattern, which Elastic has deprecated.
:::
:::{note}
If you have a lot of B2C customers, you may see a high volume of emails as