The PostgreSQL dashboard shipped at the repo-root grafana/ by oversight;
every other dashboard source lives under dashboards/ (opensearch/,
grafana/, splunk/). Move it next to the existing Grafana dashboard, list
it in dashboards/README.md, and fix the CHANGELOG path reference.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The forensic→failure rename (#659) left a few loose ends and one deliberate
hold-back. This closes them.
Leftover rename misses (broken paths / stale canonical names):
- CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/*
- dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml
→ dmarc_failure_dashboard.xml (the file was already renamed; the import path
and view name were not)
- docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example
- samples/parsedmarc.ini: save_forensic → save_failure
- pyproject.toml, README.md: canonical "failure" naming
(ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.)
Archive subfolder rename + on-startup migration:
- New failure reports now archive to <archive>/Failure (was <archive>/Forensic).
- _migrate_forensic_archive_folder() runs once on startup (best-effort):
renames Forensic→Failure when no Failure folder exists yet, merges the two
when both exist, no-ops when there's no legacy folder, and logs-and-skips a
mailbox it can't reorganize (warn, don't crash). This consolidates pre- and
post-rename failure reports into one folder, replacing the previously
documented decision to keep the folder named Forensic to avoid a split
archive. Uses the folder-management API (folder_exists / rename_folder /
merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0.
Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana):
- Forensic panel titles + the datasource label → Failure; the fo-column display
label and its linked byName field-override matcher both → "Failure Policy"
(changed together so the column-width override keeps matching).
- dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both
pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana
dashboards); RESEED wipe loop now also clears dmarc_failure* indices.
- Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an
orphan export accidentally committed in #736 and referenced by nothing.
Tests (tests/test_init.py):
- TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via
mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full
get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds).
- TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir
can't reproduce — a backend that raises mid-operation must warn, not crash.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @ 2cda5bf (9.9.0), which added the ASN
source attribution work (#712, #713, #714, #715). Individual Copilot
iteration commits squashed into this single commit — the per-commit
history on the feature branch was iterative (add tests, fix lint,
move field, revert, etc.) and not worth preserving; GitHub squash-
merges PRs anyway.
New fields from the DMARCbis XSD, plumbed through types, parsing, CSV
output, and the Elasticsearch / OpenSearch mappings:
- ``np`` — non-existent subdomain policy (``none`` / ``quarantine`` /
``reject``)
- ``testing`` — testing mode flag (``n`` / ``y``), replaces RFC 7489
``pct``
- ``discovery_method`` — policy discovery method (``psl`` /
``treewalk``)
- ``generator`` — report generator software identifier (metadata)
- ``human_result`` — optional descriptive text on DKIM / SPF results
RFC 7489 reports parse with ``None`` for DMARCbis-only fields.
Forensic reports have been renamed to failure reports throughout the
project to reflect the proper naming since RFC 7489.
- Core: ``types.py``, ``__init__.py`` — ``ForensicReport`` →
``FailureReport``, ``parse_forensic_report`` →
``parse_failure_report``, report type ``"failure"``.
- Output modules: ``elastic.py``, ``opensearch.py``, ``splunk.py``,
``kafkaclient.py``, ``syslog.py``, ``gelf.py``, ``webhook.py``,
``loganalytics.py``, ``s3.py``.
- CLI: ``cli.py`` — args, config keys, index names
(``dmarc_failure``).
- Docs + dashboards: all markdown, Grafana JSON, Kibana NDJSON,
Splunk XML.
Backward compatibility preserved: old function / type names remain as
aliases (``parse_forensic_report = parse_failure_report``,
``ForensicReport = FailureReport``, etc.), CLI accepts both the old
(``save_forensic``, ``forensic_topic``) and new (``save_failure``,
``failure_topic``) config keys, and updated dashboards query both
old and new index / sourcetype names so data from before and after
the rename appears together.
Merge conflicts resolved in ``parsedmarc/constants.py`` (took bis's
10.0.0 bump), ``parsedmarc/__init__.py`` (combined bis's "failure"
wording with master's IPinfo MMDB mention), ``parsedmarc/elastic.py``
and ``parsedmarc/opensearch.py`` (kept master's ``source_asn`` /
``source_asn_name`` / ``source_asn_domain`` on the failure doc path
while renaming ``forensic_report`` → ``failure_report``), and
``CHANGELOG.md`` (10.0.0 entry now sits above the 9.9.0 entry).
All 324 tests pass; ``ruff check`` / ``ruff format --check`` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Apply post-RFC review fixes: RFC 9990 detection, langAttrString, CFWS-aware RUF parsing
Aligns the implementation with the final RFCs (9989/9990/9991) instead of
inferring DMARCbis support from the version element or the namespace alone.
Aggregate parsing (RFC 9990):
- _text() helper unwraps langAttrString values (extra_contact_info, error,
comment, human_result, generator) — when reporters include the lang
attribute, xmltodict yields {"#text": ..., "@lang": ...} dicts instead
of strings; the parser now stores the text payload in both shapes.
- New xml_namespace field on AggregateReport records the declared XML
namespace (urn:ietf:params:xml:ns:dmarc-2.0 for RFC 9990 reports).
- RFC 9990 detection accepts namespaceless reports that follow the
RFC 9990 shape (presence of np / testing / discovery_method / generator),
so reporters that don't declare the namespace still receive RFC 9990-
aware validation.
- Warnings: missing DKIM <selector> (REQUIRED in RFC 9990); legacy
forwarded / sampled_out policy-override types (removed by RFC 9990);
unknown policy-override types per the RFC 9990 enumeration.
- xml_namespace added to Elasticsearch and OpenSearch document mappings.
Failure parsing (RFC 9991):
- Identity-Alignment and Auth-Failure are split on commas with CFWS
whitespace stripped per the RFC 9991 ABNF; previously "dkim, spf"
yielded ["dkim", " spf"] with a leading space on the second token.
- Warnings logged when either REQUIRED field is missing.
Terminology: every reference to "DMARCbis" in code, tests, sample
filenames, AGENTS.md, and CHANGELOG.md is replaced with the appropriate
RFC number (9989 for the policy spec, 9990 for aggregate reports, 9991
for failure reports). Sample contents are unchanged.
Docs: corrects the prior claim that fo was dropped from RFC 9990 (only
pct was), reframes testing as a new field (not a pct replacement, since
RFC 9989 Appendix A.6 removed pct with no per-message substitute), and
documents the policy_override_reason enum changes (added policy_test_mode;
removed forwarded / sampled_out).
Tests: 8 new tests covering xml_namespace capture, RFC 9990 detection
from field shape, missing-DKIM-selector warning, legacy-override-type
warning, langAttrString unwrapping across all four affected elements,
and CFWS-aware Identity-Alignment / Auth-Failure parsing plus their
missing-field warnings. 276 tests total, all passing; ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refactored the SMTP TLS Splunk dashboard to use a base search for improved query efficiency and maintainability. Updated input token names and adjusted search queries for better organization and clarity.
* Align Grafana dashboard with OpenSearch Dashboards source-of-truth
Adds the two aggregate-DMARC panels that exist on the OSD dashboard but
were missing from the bundled Grafana dashboard:
- "Message sources by name and type" — buckets by source_name + source_type,
sums message_count per (name, type) tuple. Mirrors the OSD viz from 9.4.x.
- "Message sources by Autonomous System" — buckets by source_asn +
source_as_name + source_as_domain, sums message_count per ASN. Mirrors
the OSD viz added in 9.9.0 with the IPinfo Lite ASN integration.
Both panels are patterned on the existing "Reporting Organisations" panel
(same datasource $datasourceag, same sum(message_count) metric, same
gradient-gauge "Messages" column with rename transforms). They sit at
the bottom of the existing layout (gridPos y=129 and y=140) so the
existing panel positions are unchanged.
Verified against the bundled grafana/grafana:12.3.0: dashboard import
returns status=success, both panels render with real data from the
sample-corpus indexes, and the ES aggregations (terms on source_name
+ source_type, numeric terms on source_asn) return the expected results.
Out of scope:
- Extras in the Grafana dashboard that aren't on OSD (SPF/DKIM Results
Over Time, Alignment Over Time, Stat overview, Published Policies,
Forensic IP / country tables) are left in place. They were
community-contributed and likely valued by some users.
- Migrating the deprecated `graph` and `grafana-worldmap-panel` panel
types to modern timeseries / geomap is a separate, larger task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Grafana: migrate deprecated graph and worldmap panels
Replaces the 6 legacy `graph` panels with `timeseries` panels and the
2 legacy `grafana-worldmap-panel` panels with `geomap` panels. Both
deprecated plugins still rendered in Grafana 12 via auto-migration but
were flagged for removal; this ships the modern saved shape.
graph -> timeseries (6 panels):
SPF Results Over Time, DKIM Results Over Time, SPF Alignment Over Time,
DKIM Alignment Over Time, DMARC Passage Over Time, Message Disposition
Over Time. Panel `aliasColors` (e.g. {true: dark-green, false: dark-red})
are translated into per-series `fieldConfig.overrides` so the green/red
by-pass-fail colorings carry forward; legacy graph fields (lines, fill,
yaxes, tooltip etc.) are dropped in favor of the new
`fieldConfig.defaults.custom` block and `options.legend` / `options.tooltip`.
worldmap -> geomap (2 panels):
Map of Message Source Countries (aggregate), Forensic Sample Sources
by Country (forensic). The legacy `locationData=countries` lookup-by-ISO
becomes a geomap markers layer with `location.mode=lookup`,
`gazetteer=public/gazetteer/countries.json`, and `lookup=source_country.keyword`
— same input data, modern renderer. Drops the date_histogram bucket
from the geomap targets since the map is a snapshot over the panel
time range, not a time series.
Verified against the bundled grafana/grafana:12.3.0: dashboard imports
with status=success and `version=19`, live panel types now report
`{timeseries: 6, geomap: 2, table: 14, grafana-piechart-panel: 3,
stat: 1, row: 3}` — no more `graph` or `grafana-worldmap-panel` entries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* OSD: fix aggregate dashboard metrics to sum(message_count)
13 panels on the DMARC aggregate dashboard were aggregating with `count`
(number of OSD docs) when they should have been summing `message_count`.
Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from
the XML and carries an integer message_count, so doc-counting reports
"distinct sources" rather than "messages". Panels with titles like "Message
volume by header from", "DMARC passage over time", etc. were producing
misleading numbers.
Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by
reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and
DKIM details. (DMARC failure email samples kept count — one OSD doc per
RUF sample, so it's correct. SMTP TLS panels untouched — they sum the
right session-count fields.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Splunk: align dashboards with OSD and fix query bugs
Aggregate dashboard:
- Add "Message sources by Autonomous System" panel (source_asn / as_name /
as_domain), formatted "AS<n>" at render with eval, matching the OSD addition.
- DKIM details: add the missing dkim_aligned column.
- SPF details: reorder columns to OSD order (spf_aligned at end).
- Map / country titles renamed to match OSD ("Map of message sources by
country", "Message sources by country").
- Map widget: stats count by Country -> stats sum(message_count) by
Country, so the choropleth shades by message volume not record count.
- fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain,
source_country to mirror OSD's missing-bucket labels.
- charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC
pies and the DMARC-passage timechart.
Forensic dashboard:
- Restructure to match OSD's two-panel layout (markdown + samples table).
- Drop the country map / IP table / country-ISO table panels (not in OSD).
- Samples table columns aligned to OSD: arrival_date_utc, source.ip_address,
from, subject, reply_to, authentication_results.
- Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=*
required field to exist; LinkedIn RUF sample with null From was filtered out).
SMTP TLS dashboard:
- Reorder metrics to OSD order (successful before failed).
- Domains panel: add policy_type bucket.
- Failure details: replace search-time `failed_session_count>0` (which
doesn't evaluate against multivalued JSON paths in Splunk) with
`result_type=*` for presence + post-stats `where failed_sessions>0`.
Drop _time/successful_sessions columns; reorder to match OSD.
- Wire the existing policy_type input into all three searches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add dashboard-dev bootstrap script and VSCode task
dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml,
seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini,
and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana,
and Splunk. Idempotent: existence checks skip provisioning that's already
done; only the dashboard imports re-run unconditionally on every invocation
(that's the point of running it after a dashboard edit).
Notable provisioning quirks the script handles:
- Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships
with indexes=[] and index=default; rewrites it to allow the email index.
- ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily
parsedmarc indexes via _cat/indices and deletes one at a time.
- Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1
deletes and recreates the email index (then re-applies the HEC token).
- OSD security plugin tenants: imports target global_tenant explicitly
via the securitytenant header so they're visible to the shared workspace
rather than landing in the API user's private tenant. Override with
OSD_TENANT=<name>.
- Splunk ships an in-product announcement view (scheduled_export_dashboard)
with sharing=global; the script narrows it to sharing=app so it stops
showing up in every app's dashboards list.
Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs
the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump version to 9.10.3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: warn against the "Create new objects with unique IDs" import mode
OSD's import dialog has two modes: the default "Check for existing objects"
(which honors saved-object IDs and overwrites in place when "Automatically
overwrite conflicts" is on) and "Create new objects with unique IDs" (which
imports under fresh UUIDs and leaves the buggy originals untouched). Picking
the second one means the dashboards keep rendering the wrong numbers because
the originals are never replaced. Spell that out so users don't fall into
the trap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* OSD: label the metric column "messages" instead of "Sum of message_count"
OSD's table column header defaults to "Sum of message_count" when the
metric agg has no customLabel. "messages" reads better and matches what
the panels are actually counting.
Applies to all 15 aggregate-DMARC visualizations that use sum(message_count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable
Trim the verbose technical exposition; lead each fix with the user-visible
symptom. Move the action-required call out to its own header in upgrade
notes so the re-import instructions don't get lost in a wall of text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Move per-tool dashboard exports under a single dashboards/ directory
Consolidates the four sibling top-level folders (kibana/, opensearch/,
grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/.
Updates the only path references in tracked files: bootstrap script (5
lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in
docs/source/elasticsearch.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* OSD: restore the "DKIM alignment" panel title on the aggregate dashboard
The DKIM alignment panel had no title override in panelsJSON, so OSD fell
back to the visualization's own name ("Aggregate DMARC DKIM alignment").
Every other pie/table on the same dashboard sets a clean title (SPF
alignment, Passed DMARC, etc.) — this was a stray regression. Set the
panel title to "DKIM alignment" to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Splunk: color the message-disposition timechart by severity
Reject is red, quarantine is yellow, none is green — same semantic
mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage
timechart, applied via charting.fieldColors. Matches OSD's existing
color overrides on the equivalent viz.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: clarify that "Create new objects with unique IDs" is the default
The OSD import dialog defaults to that mode — users have to actively
switch away from it, not just avoid picking it. Reword the upgrade note
to lead with the switch and explain why the default would silently
preserve the bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sean Whalen
Copilot
Sean Whalen