amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-07-26 18:34:56 +00:00
Code Issues Projects Releases Wiki Activity

Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)

Browse Source 
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:

- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
  a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
  sum of message_count), pivoted on header_from.keyword. The time field
  is date_begin rather than the multi-valued date_range, which TSVB's
  per-value date histogram would double-count. Editing (not rendering)
  the panel on Kibana 8.x requires the metrics:allowStringIndices
  advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
  field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
  COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
  stats, per the SPL eval-in-stats syntax.

All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).

Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.

Closes #112

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-07-20 21:25:00 -04:00
co-authored by Claude Fable 5
parent 855d267650
commit 47fb50e76a
8 changed files with 163 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+7
View File
@@ -1,5 +1,12 @@
# Changelog
## 10.2.5
### New features
- **Added a per-domain DMARC compliance percentage to the aggregate dashboards of every provider** ([#112](https://github.com/domainaware/parsedmarc/issues/112)): OpenSearch Dashboards/Kibana, Grafana Elasticsearch, Grafana PostgreSQL, and Splunk. The from-domain volume table on each dashboard is now "Message volume and DMARC compliance by from domain", with columns for From Domain, Messages, and % DMARC Compliant.
- On OpenSearch Dashboards/Kibana, the table is now a TSVB visualization using a Filter Ratio metric (passed messages over total messages per `header_from`), since the previous agg-based data table can't compute a per-domain ratio. Editing the imported visualization on Kibana 8.x requires first enabling the `metrics:allowStringIndices` advanced setting.
## 10.2.4
### Bug fixes
dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json
+9 -2
View File
@@ -840,6 +840,13 @@
"value": { "mode": "gradient", "type": "color-background" }
}
]
},
{
"matcher": { "id": "byName", "options": "% DMARC Compliant" },
"properties": [
{ "id": "custom.width", "value": 160 },
{ "id": "unit", "value": "percent" }
]
}
]
},
@@ -851,7 +858,7 @@
"cellHeight": "sm",
"footer": { "show": false }
},
"title": "Reports by From Domain",
"title": "Message volume and DMARC compliance by from domain",
"type": "table",
"targets": [
{
@@ -861,7 +868,7 @@
},
"editorMode": "code",
"format": "table",
"rawSql": "SELECT\n COALESCE(r.header_from, 'unknown') AS \"From Domain\",\n SUM(r.message_count) AS \"Messages\"\nFROM dmarc_aggregate_record r\nJOIN dmarc_aggregate_report rpt ON rpt.id = r.report_id\nWHERE $__timeFilter(rpt.begin_date)\n AND r.header_from IN ($fromdomain)\nGROUP BY r.header_from\nORDER BY \"Messages\" DESC",
"rawSql": "SELECT\n COALESCE(r.header_from, 'unknown') AS \"From Domain\",\n SUM(r.message_count) AS \"Messages\",\n ROUND(100.0 * COALESCE(SUM(r.message_count) FILTER (WHERE r.dmarc_passed), 0) / NULLIF(SUM(r.message_count), 0), 1) AS \"% DMARC Compliant\"\nFROM dmarc_aggregate_record r\nJOIN dmarc_aggregate_report rpt ON rpt.id = r.report_id\nWHERE $__timeFilter(rpt.begin_date)\n AND r.header_from IN ($fromdomain)\nGROUP BY r.header_from\nORDER BY \"Messages\" DESC",
"refId": "A"
}
]
dashboards/grafana/Grafana-DMARC_Reports.json
+91 -7
View File
@@ -1837,7 +1837,7 @@
{
"matcher": {
"id": "byName",
"options": "Header From"
"options": "From Domain"
},
"properties": [
{
@@ -1845,8 +1845,8 @@
"value": [
{
"targetBlank": true,
"title": "Open ${__data.fields[\"header_from.keyword\"]} in new window",
"url": "https://${__data.fields[\"header_from.keyword\"]}"
"title": "Open ${__data.fields[\"From Domain\"]} in new window",
"url": "https://${__data.fields[\"From Domain\"]}"
}
]
}
@@ -1879,6 +1879,26 @@
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "% DMARC Compliant"
},
"properties": [
{
"id": "unit",
"value": "percentunit"
},
{
"id": "decimals",
"value": 1
},
{
"id": "noValue",
"value": "0"
}
]
}
]
},
@@ -1932,20 +1952,84 @@
"query": "header_from.keyword:$fromdomain",
"refId": "A",
"timeField": "date_range"
},
{
"bucketAggs": [
{
"$$hashKey": "object:389",
"fake": true,
"field": "header_from.keyword",
"id": "6",
"settings": {
"min_doc_count": 1,
"missing": "none",
"order": "desc",
"orderBy": "4",
"size": "0"
},
"type": "terms"
}
],
"hide": false,
"metrics": [
{
"$$hashKey": "object:387",
"field": "message_count",
"id": "4",
"meta": {},
"settings": {},
"type": "sum"
}
],
"query": "header_from.keyword:$fromdomain AND passed_dmarc:true",
"refId": "B",
"timeField": "date_range"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Message Volume by Header From",
"title": "Message volume and DMARC compliance by from domain",
"transformations": [
{
"id": "joinByField",
"options": {
"byField": "header_from.keyword",
"mode": "outer"
}
},
{
"id": "calculateField",
"options": {
"alias": "% DMARC Compliant",
"mode": "binary",
"binary": {
"left": {
"matcher": {
"id": "byName",
"options": "Sum 2"
}
},
"operator": "/",
"right": {
"matcher": {
"id": "byName",
"options": "Sum 1"
}
}
},
"replaceFields": false
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"excludeByName": {
"Sum 2": true
},
"indexByName": {},
"renameByName": {
"Sum": "Messages",
"header_from.keyword": "Header From"
"Sum 1": "Messages",
"header_from.keyword": "From Domain"
}
}
}
dashboards/opensearch/opensearch_dashboards.ndjson
+27 -27
View File
File diff suppressed because one or more lines are too long
dashboards/splunk/dmarc_aggregate_dashboard.xml
+6 -3
View File
@@ -188,16 +188,19 @@
</table>
</panel>
<panel>
<title>Message volume by header from</title>
<title>Message volume and DMARC compliance by from domain</title>
<table>
<search base="base_search">
<query>| stats sum(message_count) as message_count by header_from | sort -message_count</query>
<query>| stats sum(message_count) as Messages, sum(eval(if(passed_dmarc="true", message_count, 0))) as passed by header_from | eval "% DMARC Compliant"=round(passed/Messages*100, 1) | fields - passed | rename header_from as "From Domain" | sort -Messages</query>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
<format type="number" field="sum(message_count)">
<format type="number" field="Messages">
<option name="precision">0</option>
</format>
<format type="number" field="% DMARC Compliant">
<option name="precision">1</option>
</format>
</table>
</panel>
</row>
docker-compose.dashboard-dev.yml
+3 -1
View File
@@ -31,7 +31,9 @@ services:
# to "admin" so the login matches the bootstrap script's GRAFANA_PASSWORD
# default; set GRAFANA_PASSWORD in .env to change both in lockstep.
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
- GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel
# "elasticsearch" is required on Grafana >= 13, which no longer bundles
# the Elasticsearch datasource as a core plugin.
- GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel,elasticsearch
ports:
- "127.0.0.1:3000:3000"
depends_on:
docker-compose.yml
+3
View File
@@ -11,6 +11,9 @@ services:
- bootstrap.memory_lock=true
- xpack.security.enabled=false
- xpack.license.self_generated.type=basic
# Without an explicit heap, ES sizes it to 50% of host RAM and mlocks it
# (bootstrap.memory_lock), which OOM-kills the container on large hosts.
- ES_JAVA_OPTS=-Xms2g -Xmx2g
ports:
- "127.0.0.1:9200:9200"
ulimits:
docs/source/kibana.md
+17 -6
View File
@@ -44,18 +44,29 @@ disposition over time.
Under the graphs you will find the most useful data tables on the dashboard. On
the left, there is a list of organizations that are sending you DMARC reports.
In the center, there is a list of sending servers grouped by the base domain
in their reverse DNS. On the right, there is a list of email from domains,
sorted by message volume.
in their reverse DNS. On the right, there is the "Message volume and DMARC
compliance by from domain" table, which lists email from domains with their
message volume and a percentage of those messages that passed DMARC.
By hovering your mouse over a data table value and using the magnifying glass
icons, you can filter on our filter out different values. Start by looking at
the Message Sources by Reverse DNS table. Find a sender that you recognize,
such as an email marketing service, hover over it, and click on the plus (+)
magnifying glass icon, to add a filter that only shows results for that sender.
Now, look at the Message From Header table to the right. That shows you the
domains that a sender is sending as, which might tell you which brand/business
is using a particular service. With that information, you can contact them and
have them set up DKIM.
Now, look at the Message volume and DMARC compliance by from domain table to
the right. That shows you the domains that a sender is sending as, and what
share of that traffic is passing DMARC, which might tell you which
brand/business is using a particular service. With that information, you can
contact them and have them set up DKIM.
:::{note}
The "Message volume and DMARC compliance by from domain" table is a TSVB
visualization, used because per-domain compliance percentages require a
Filter Ratio metric that agg-based data tables can't compute. It renders
correctly on Kibana 8.x as imported, but *editing* it requires first enabling
the `metrics:allowStringIndices` advanced setting, since it references the
`dmarc_aggregate*` index as a string pattern, which Elastic has deprecated.
:::
:::{note}
If you have a lot of B2C customers, you may see a high volume of emails as