mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-03 14:12:20 +00:00
Add per-domain DMARC compliance percentage to all aggregate dashboards (#834)
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:
- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
sum of message_count), pivoted on header_from.keyword. The time field
is date_begin rather than the multi-valued date_range, which TSVB's
per-value date histogram would double-count. Editing (not rendering)
the panel on Kibana 8.x requires the metrics:allowStringIndices
advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
stats, per the SPL eval-in-stats syntax.
All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).
Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.
Closes #112
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix over-time charts double-counting reports via multi-valued date_range
date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.
Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.
All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.
Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review comments on PR #834
- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed DMARC" -> "failed DMARC" and the
backticked `ruf ` trailing space in the RUF explainer panel, and
rename the "SMPT TLS failure details" visualization to "SMTP TLS
failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
plugin-install restart so a hang fails with a clear timeout message
instead of an opaque downstream curl error.
The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* AGENTS.md: reviews must cover prose and hunk context, not just function
Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second round of Copilot review comments
- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
matching the repo convention where the release commit assigns the
version number (see 855d267 for 10.2.4).
- docker-compose.yml: make the dev-stack Elasticsearch heap overridable
via ES_JAVA_OPTS in .env (default unchanged at 2g), using the compose
file's existing ${VAR:-default} idiom, for smaller machines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
co-authored by
Claude Fable 5
Claude Fable 5
parent
855d267650
commit
62514bd72a
@@ -174,6 +174,14 @@ If a config file is listed in `.gitignore`, treat its contents as secret. Do not
|
||||
|
||||
Before rewriting a tracked list/data file from freshly-generated content (anything under `parsedmarc/resources/maps/`, CSVs, `.txt` lists), check the existing file first — `git show HEAD:<path> | wc -l`, `git log -1 -- <path>`, `git diff --stat`. Files like `known_unknown_base_reverse_dns.txt` and `base_reverse_dns_map.csv` accumulate manually-curated entries across many sessions, and a "fresh" regeneration that drops the row count is almost certainly destroying prior work. If the new content is meant to *add* rather than *replace*, use a merge/append pattern. Treat any unexpected row-count drop in the pending diff as a red flag.
|
||||
|
||||
## Review passes cover prose, not just function
|
||||
|
||||
A review that only verifies functional/numeric correctness (queries return the right values, files import cleanly, types check) will sail past exactly the defects a text-first reviewer catches. On PR #834, four such misses survived a thorough functional review: two long-standing typos inside the OSD ndjson ("SMPT TLS", "filed DMARC"), a typo on an *unchanged* line adjacent to a docs edit, and hand-written bootstrap glue that duplicated the script's existing `wait_for()` helper. Rules drawn from that:
|
||||
|
||||
- **Whole-file canonical exports put every line in the diff — review them as text, too.** Re-exporting `dashboards/opensearch/opensearch_dashboards.ndjson` or a Grafana JSON from a running instance rewrites the entire file, so pre-existing user-facing strings (saved-object titles, markdown panels, column labels) are formally part of the change. A semantic before/after comparison ("attributes identical") proves no unintended changes but deliberately looks through pre-existing content problems; add one text-level pass over titles and markdown before committing.
|
||||
- **Proofread the whole hunk around prose edits, not just the `+`/`-` lines.** Typos one line away from an edit are in the reviewer's context window and fair game; they should be in yours.
|
||||
- **Code written mid-incident gets the same review bar as planned code.** Before writing new shell/infra glue while firefighting, check the file for an existing helper that already does it (e.g. `wait_for()` in `dashboard-dev-bootstrap.sh`), and give your own inline code the same scrutiny you'd give a subagent's.
|
||||
|
||||
## Releases
|
||||
|
||||
A release isn't done until built artifacts are attached to the GitHub release page. Full sequence:
|
||||
|
||||
@@ -1,5 +1,16 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
### New features
|
||||
|
||||
- **Added a per-domain DMARC compliance percentage to the aggregate dashboards of every provider** ([#112](https://github.com/domainaware/parsedmarc/issues/112)): OpenSearch Dashboards/Kibana, Grafana Elasticsearch, Grafana PostgreSQL, and Splunk. The from-domain volume table on each dashboard is now "Message volume and DMARC compliance by from domain", with columns for From Domain, Messages, and % DMARC Compliant.
|
||||
- On OpenSearch Dashboards/Kibana, the table is now a TSVB visualization using a Filter Ratio metric (passed messages over total messages per `header_from`), since the previous agg-based data table can't compute a per-domain ratio. Editing the imported visualization on Kibana 8.x requires first enabling the `metrics:allowStringIndices` advanced setting.
|
||||
|
||||
### Bug fixes
|
||||
|
||||
- **The Elasticsearch/OpenSearch aggregate dashboards' over-time charts (and the Grafana ES dashboard's summary pies and time series) bucketed on the multi-valued `date_range` field**; a date histogram counts a report once per value, double-counting any report whose begin and end dates fall in different buckets. All date histograms and time-range filters now use the single-valued `date_begin`, matching the report-begin semantics of the PostgreSQL (`begin_date`) and Splunk (`_time` = interval begin) dashboards.
|
||||
|
||||
## 10.2.4
|
||||
|
||||
### Bug fixes
|
||||
|
||||
@@ -340,6 +340,33 @@ curl -sS -X POST 'http://localhost:5602/api/saved_objects/_import?overwrite=true
|
||||
--form file=@dashboards/opensearch/opensearch_dashboards.ndjson | sed 's/^/ /'
|
||||
echo " (imported into OSD tenant: ${OSD_TENANT})"
|
||||
|
||||
log "Ensuring Grafana Elasticsearch datasource plugin is installed"
|
||||
# Grafana >= 13 no longer bundles the Elasticsearch datasource plugin, and
|
||||
# GF_INSTALL_PLUGINS cannot install it (the image ships a root-owned
|
||||
# plugins-bundled/elasticsearch remnant its background installer fails to
|
||||
# replace). `grafana cli` installs into /var/lib/grafana/plugins, which works;
|
||||
# a restart is needed for Grafana to load it.
|
||||
code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
||||
-o /dev/null -w "%{http_code}" \
|
||||
"http://localhost:3000/api/plugins/elasticsearch/settings")
|
||||
if [ "$code" != "200" ]; then
|
||||
"${COMPOSE[@]}" exec -T grafana grafana cli plugins install elasticsearch \
|
||||
| sed 's/^/ /'
|
||||
"${COMPOSE[@]}" restart grafana >/dev/null
|
||||
wait_for "Grafana (after plugin install)" \
|
||||
curl -sf http://localhost:3000/api/health
|
||||
code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
||||
-o /dev/null -w "%{http_code}" \
|
||||
"http://localhost:3000/api/plugins/elasticsearch/settings")
|
||||
if [ "$code" != "200" ]; then
|
||||
echo "ERROR: elasticsearch datasource plugin failed to install" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo " installed elasticsearch datasource plugin"
|
||||
else
|
||||
echo " elasticsearch datasource plugin already installed"
|
||||
fi
|
||||
|
||||
log "Configuring Grafana datasources"
|
||||
# Two Elasticsearch datasources, one per index family, matching the dashboard's
|
||||
# template variables (dmarc-ag and dmarc-fo). Skipped when already present.
|
||||
@@ -347,7 +374,7 @@ declare -a GF_DS_NAMES=("dmarc-ag" "dmarc-fo")
|
||||
# dmarc_f* matches both pre-rename dmarc_forensic* and post-rename
|
||||
# dmarc_failure* indices, mirroring the OpenSearch/Kibana dashboards.
|
||||
declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_f*")
|
||||
declare -a GF_DS_TIME=("date_range" "arrival_date")
|
||||
declare -a GF_DS_TIME=("date_begin" "arrival_date")
|
||||
for i in 0 1; do
|
||||
name="${GF_DS_NAMES[$i]}"
|
||||
code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
||||
|
||||
@@ -840,6 +840,13 @@
|
||||
"value": { "mode": "gradient", "type": "color-background" }
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "% DMARC Compliant" },
|
||||
"properties": [
|
||||
{ "id": "custom.width", "value": 160 },
|
||||
{ "id": "unit", "value": "percent" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -851,7 +858,7 @@
|
||||
"cellHeight": "sm",
|
||||
"footer": { "show": false }
|
||||
},
|
||||
"title": "Reports by From Domain",
|
||||
"title": "Message volume and DMARC compliance by from domain",
|
||||
"type": "table",
|
||||
"targets": [
|
||||
{
|
||||
@@ -861,7 +868,7 @@
|
||||
},
|
||||
"editorMode": "code",
|
||||
"format": "table",
|
||||
"rawSql": "SELECT\n COALESCE(r.header_from, 'unknown') AS \"From Domain\",\n SUM(r.message_count) AS \"Messages\"\nFROM dmarc_aggregate_record r\nJOIN dmarc_aggregate_report rpt ON rpt.id = r.report_id\nWHERE $__timeFilter(rpt.begin_date)\n AND r.header_from IN ($fromdomain)\nGROUP BY r.header_from\nORDER BY \"Messages\" DESC",
|
||||
"rawSql": "SELECT\n COALESCE(r.header_from, 'unknown') AS \"From Domain\",\n SUM(r.message_count) AS \"Messages\",\n ROUND(100.0 * COALESCE(SUM(r.message_count) FILTER (WHERE r.dmarc_passed), 0) / NULLIF(SUM(r.message_count), 0), 1) AS \"% DMARC Compliant\"\nFROM dmarc_aggregate_record r\nJOIN dmarc_aggregate_report rpt ON rpt.id = r.report_id\nWHERE $__timeFilter(rpt.begin_date)\n AND r.header_from IN ($fromdomain)\nGROUP BY r.header_from\nORDER BY \"Messages\" DESC",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
|
||||
@@ -188,7 +188,7 @@
|
||||
},
|
||||
{
|
||||
"$$hashKey": "object:244",
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "auto",
|
||||
@@ -211,7 +211,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -277,7 +277,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "auto",
|
||||
@@ -299,7 +299,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -369,7 +369,7 @@
|
||||
},
|
||||
{
|
||||
"$$hashKey": "object:386",
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "auto",
|
||||
@@ -392,7 +392,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -622,7 +622,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "1d",
|
||||
@@ -644,7 +644,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -852,7 +852,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "1d",
|
||||
@@ -874,7 +874,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -977,7 +977,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "1d",
|
||||
@@ -1001,7 +1001,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1104,7 +1104,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "1d",
|
||||
@@ -1128,7 +1128,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1231,7 +1231,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "1d",
|
||||
@@ -1255,7 +1255,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1373,7 +1373,7 @@
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "2",
|
||||
"settings": {
|
||||
"fixed_interval": "1d",
|
||||
@@ -1395,7 +1395,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1463,7 +1463,7 @@
|
||||
{
|
||||
"$$hashKey": "object:430",
|
||||
"fake": true,
|
||||
"field": "date_range",
|
||||
"field": "date_begin",
|
||||
"id": "6",
|
||||
"settings": {
|
||||
"fixed_interval": "auto",
|
||||
@@ -1486,7 +1486,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1606,7 +1606,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1756,7 +1756,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
},
|
||||
{
|
||||
"bucketAggs": [
|
||||
@@ -1788,7 +1788,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "B",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -1837,7 +1837,7 @@
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Header From"
|
||||
"options": "From Domain"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
@@ -1845,8 +1845,8 @@
|
||||
"value": [
|
||||
{
|
||||
"targetBlank": true,
|
||||
"title": "Open ${__data.fields[\"header_from.keyword\"]} in new window",
|
||||
"url": "https://${__data.fields[\"header_from.keyword\"]}"
|
||||
"title": "Open ${__data.fields[\"From Domain\"]} in new window",
|
||||
"url": "https://${__data.fields[\"From Domain\"]}"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1879,6 +1879,26 @@
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "% DMARC Compliant"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "unit",
|
||||
"value": "percentunit"
|
||||
},
|
||||
{
|
||||
"id": "decimals",
|
||||
"value": 1
|
||||
},
|
||||
{
|
||||
"id": "noValue",
|
||||
"value": "0"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -1931,21 +1951,85 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
},
|
||||
{
|
||||
"bucketAggs": [
|
||||
{
|
||||
"$$hashKey": "object:389",
|
||||
"fake": true,
|
||||
"field": "header_from.keyword",
|
||||
"id": "6",
|
||||
"settings": {
|
||||
"min_doc_count": 1,
|
||||
"missing": "none",
|
||||
"order": "desc",
|
||||
"orderBy": "4",
|
||||
"size": "0"
|
||||
},
|
||||
"type": "terms"
|
||||
}
|
||||
],
|
||||
"hide": false,
|
||||
"metrics": [
|
||||
{
|
||||
"$$hashKey": "object:387",
|
||||
"field": "message_count",
|
||||
"id": "4",
|
||||
"meta": {},
|
||||
"settings": {},
|
||||
"type": "sum"
|
||||
}
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain AND passed_dmarc:true",
|
||||
"refId": "B",
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "Message Volume by Header From",
|
||||
"title": "Message volume and DMARC compliance by from domain",
|
||||
"transformations": [
|
||||
{
|
||||
"id": "joinByField",
|
||||
"options": {
|
||||
"byField": "header_from.keyword",
|
||||
"mode": "outer"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "calculateField",
|
||||
"options": {
|
||||
"alias": "% DMARC Compliant",
|
||||
"mode": "binary",
|
||||
"binary": {
|
||||
"left": {
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Sum 2"
|
||||
}
|
||||
},
|
||||
"operator": "/",
|
||||
"right": {
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Sum 1"
|
||||
}
|
||||
}
|
||||
},
|
||||
"replaceFields": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"excludeByName": {
|
||||
"Sum 2": true
|
||||
},
|
||||
"indexByName": {},
|
||||
"renameByName": {
|
||||
"Sum": "Messages",
|
||||
"header_from.keyword": "Header From"
|
||||
"Sum 1": "Messages",
|
||||
"header_from.keyword": "From Domain"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2020,7 +2104,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -2198,7 +2282,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -2567,7 +2651,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -2920,7 +3004,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -3173,7 +3257,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -3400,7 +3484,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -3645,7 +3729,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -4558,7 +4642,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
@@ -4742,7 +4826,7 @@
|
||||
],
|
||||
"query": "header_from.keyword:$fromdomain",
|
||||
"refId": "A",
|
||||
"timeField": "date_range"
|
||||
"timeField": "date_begin"
|
||||
}
|
||||
],
|
||||
"timeFrom": null,
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -188,16 +188,19 @@
|
||||
</table>
|
||||
</panel>
|
||||
<panel>
|
||||
<title>Message volume by header from</title>
|
||||
<title>Message volume and DMARC compliance by from domain</title>
|
||||
<table>
|
||||
<search base="base_search">
|
||||
<query>| stats sum(message_count) as message_count by header_from | sort -message_count</query>
|
||||
<query>| stats sum(message_count) as Messages, sum(eval(if(passed_dmarc="true", message_count, 0))) as passed by header_from | eval "% DMARC Compliant"=round(passed/Messages*100, 1) | fields - passed | rename header_from as "From Domain" | sort -Messages</query>
|
||||
</search>
|
||||
<option name="drilldown">none</option>
|
||||
<option name="refresh.display">progressbar</option>
|
||||
<format type="number" field="sum(message_count)">
|
||||
<format type="number" field="Messages">
|
||||
<option name="precision">0</option>
|
||||
</format>
|
||||
<format type="number" field="% DMARC Compliant">
|
||||
<option name="precision">1</option>
|
||||
</format>
|
||||
</table>
|
||||
</panel>
|
||||
</row>
|
||||
|
||||
@@ -31,6 +31,10 @@ services:
|
||||
# to "admin" so the login matches the bootstrap script's GRAFANA_PASSWORD
|
||||
# default; set GRAFANA_PASSWORD in .env to change both in lockstep.
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
|
||||
# Grafana >= 13 no longer bundles the Elasticsearch datasource, but
|
||||
# installing it via GF_INSTALL_PLUGINS crash-loops: the image ships a
|
||||
# root-owned plugins-bundled/elasticsearch remnant the installer cannot
|
||||
# replace. The bootstrap script installs it via `grafana cli` instead.
|
||||
- GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
|
||||
@@ -11,6 +11,10 @@ services:
|
||||
- bootstrap.memory_lock=true
|
||||
- xpack.security.enabled=false
|
||||
- xpack.license.self_generated.type=basic
|
||||
# Without an explicit heap, ES sizes it to 50% of host RAM and mlocks it
|
||||
# (bootstrap.memory_lock), which OOM-kills the container on large hosts.
|
||||
# Override via ES_JAVA_OPTS in .env for smaller machines.
|
||||
- ES_JAVA_OPTS=${ES_JAVA_OPTS:--Xms2g -Xmx2g}
|
||||
ports:
|
||||
- "127.0.0.1:9200:9200"
|
||||
ulimits:
|
||||
|
||||
+18
-7
@@ -44,18 +44,29 @@ disposition over time.
|
||||
Under the graphs you will find the most useful data tables on the dashboard. On
|
||||
the left, there is a list of organizations that are sending you DMARC reports.
|
||||
In the center, there is a list of sending servers grouped by the base domain
|
||||
in their reverse DNS. On the right, there is a list of email from domains,
|
||||
sorted by message volume.
|
||||
in their reverse DNS. On the right, there is the "Message volume and DMARC
|
||||
compliance by from domain" table, which lists email from domains with their
|
||||
message volume and a percentage of those messages that passed DMARC.
|
||||
|
||||
By hovering your mouse over a data table value and using the magnifying glass
|
||||
icons, you can filter on our filter out different values. Start by looking at
|
||||
icons, you can filter on or filter out different values. Start by looking at
|
||||
the Message Sources by Reverse DNS table. Find a sender that you recognize,
|
||||
such as an email marketing service, hover over it, and click on the plus (+)
|
||||
magnifying glass icon, to add a filter that only shows results for that sender.
|
||||
Now, look at the Message From Header table to the right. That shows you the
|
||||
domains that a sender is sending as, which might tell you which brand/business
|
||||
is using a particular service. With that information, you can contact them and
|
||||
have them set up DKIM.
|
||||
Now, look at the Message volume and DMARC compliance by from domain table to
|
||||
the right. That shows you the domains that a sender is sending as, and what
|
||||
share of that traffic is passing DMARC, which might tell you which
|
||||
brand/business is using a particular service. With that information, you can
|
||||
contact them and have them set up DKIM.
|
||||
|
||||
:::{note}
|
||||
The "Message volume and DMARC compliance by from domain" table is a TSVB
|
||||
visualization, used because per-domain compliance percentages require a
|
||||
Filter Ratio metric that agg-based data tables can't compute. It renders
|
||||
correctly on Kibana 8.x as imported, but *editing* it requires first enabling
|
||||
the `metrics:allowStringIndices` advanced setting, since it references the
|
||||
`dmarc_aggregate*` index as a string pattern, which Elastic has deprecated.
|
||||
:::
|
||||
|
||||
:::{note}
|
||||
If you have a lot of B2C customers, you may see a high volume of emails as
|
||||
|
||||
Reference in New Issue
Block a user