mirror of https://github.com/domainaware/parsedmarc.git synced 2026-05-21 11:25:23 +00:00

Files

T

Copilot ae1e5adb66 Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659 )

* Add DMARCbis report support; rename forensic→failure project-wide

Rebased on top of master @ 2cda5bf (9.9.0), which added the ASN
source attribution work (#712, #713, #714, #715). Individual Copilot
iteration commits squashed into this single commit — the per-commit
history on the feature branch was iterative (add tests, fix lint,
move field, revert, etc.) and not worth preserving; GitHub squash-
merges PRs anyway.

New fields from the DMARCbis XSD, plumbed through types, parsing, CSV
output, and the Elasticsearch / OpenSearch mappings:

- ``np`` — non-existent subdomain policy (``none`` / ``quarantine`` /
  ``reject``)
- ``testing`` — testing mode flag (``n`` / ``y``), replaces RFC 7489
  ``pct``
- ``discovery_method`` — policy discovery method (``psl`` /
  ``treewalk``)
- ``generator`` — report generator software identifier (metadata)
- ``human_result`` — optional descriptive text on DKIM / SPF results

RFC 7489 reports parse with ``None`` for DMARCbis-only fields.

Forensic reports have been renamed to failure reports throughout the
project to reflect the proper naming since RFC 7489.

- Core: ``types.py``, ``__init__.py`` — ``ForensicReport`` →
  ``FailureReport``, ``parse_forensic_report`` →
  ``parse_failure_report``, report type ``"failure"``.
- Output modules: ``elastic.py``, ``opensearch.py``, ``splunk.py``,
  ``kafkaclient.py``, ``syslog.py``, ``gelf.py``, ``webhook.py``,
  ``loganalytics.py``, ``s3.py``.
- CLI: ``cli.py`` — args, config keys, index names
  (``dmarc_failure``).
- Docs + dashboards: all markdown, Grafana JSON, Kibana NDJSON,
  Splunk XML.

Backward compatibility preserved: old function / type names remain as
aliases (``parse_forensic_report = parse_failure_report``,
``ForensicReport = FailureReport``, etc.), CLI accepts both the old
(``save_forensic``, ``forensic_topic``) and new (``save_failure``,
``failure_topic``) config keys, and updated dashboards query both
old and new index / sourcetype names so data from before and after
the rename appears together.

Merge conflicts resolved in ``parsedmarc/constants.py`` (took bis's
10.0.0 bump), ``parsedmarc/__init__.py`` (combined bis's "failure"
wording with master's IPinfo MMDB mention), ``parsedmarc/elastic.py``
and ``parsedmarc/opensearch.py`` (kept master's ``source_asn`` /
``source_asn_name`` / ``source_asn_domain`` on the failure doc path
while renaming ``forensic_report`` → ``failure_report``), and
``CHANGELOG.md`` (10.0.0 entry now sits above the 9.9.0 entry).

All 324 tests pass; ``ruff check`` / ``ruff format --check`` clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Apply post-RFC review fixes: RFC 9990 detection, langAttrString, CFWS-aware RUF parsing

Aligns the implementation with the final RFCs (9989/9990/9991) instead of
inferring DMARCbis support from the version element or the namespace alone.

Aggregate parsing (RFC 9990):
- _text() helper unwraps langAttrString values (extra_contact_info, error,
  comment, human_result, generator) — when reporters include the lang
  attribute, xmltodict yields {"#text": ..., "@lang": ...} dicts instead
  of strings; the parser now stores the text payload in both shapes.
- New xml_namespace field on AggregateReport records the declared XML
  namespace (urn:ietf:params:xml:ns:dmarc-2.0 for RFC 9990 reports).
- RFC 9990 detection accepts namespaceless reports that follow the
  RFC 9990 shape (presence of np / testing / discovery_method / generator),
  so reporters that don't declare the namespace still receive RFC 9990-
  aware validation.
- Warnings: missing DKIM <selector> (REQUIRED in RFC 9990); legacy
  forwarded / sampled_out policy-override types (removed by RFC 9990);
  unknown policy-override types per the RFC 9990 enumeration.
- xml_namespace added to Elasticsearch and OpenSearch document mappings.

Failure parsing (RFC 9991):
- Identity-Alignment and Auth-Failure are split on commas with CFWS
  whitespace stripped per the RFC 9991 ABNF; previously "dkim, spf"
  yielded ["dkim", " spf"] with a leading space on the second token.
- Warnings logged when either REQUIRED field is missing.

Terminology: every reference to "DMARCbis" in code, tests, sample
filenames, AGENTS.md, and CHANGELOG.md is replaced with the appropriate
RFC number (9989 for the policy spec, 9990 for aggregate reports, 9991
for failure reports). Sample contents are unchanged.

Docs: corrects the prior claim that fo was dropped from RFC 9990 (only
pct was), reframes testing as a new field (not a pct replacement, since
RFC 9989 Appendix A.6 removed pct with no per-message substitute), and
documents the policy_override_reason enum changes (added policy_test_mode;
removed forwarded / sampled_out).

Tests: 8 new tests covering xml_namespace capture, RFC 9990 detection
from field shape, missing-DKIM-selector warning, legacy-override-type
warning, langAttrString unwrapping across all four affected elements,
and CFWS-aware Identity-Alignment / Auth-Failure parsing plus their
missing-field warnings. 276 tests total, all passing; ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-20 18:51:08 -04:00

grafana

Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659 )

2026-05-20 18:51:08 -04:00

opensearch

Fix typo in source ip viz

2026-04-27 15:20:45 -04:00

splunk

Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659 )

2026-05-20 18:51:08 -04:00

README.md

Update dashboard documentation

2026-05-03 12:36:06 -04:00

README.md

Dashboard development

This directory holds the dashboard sources that ship with parsedmarc:

opensearch/opensearch_dashboards.ndjson — the source-of-truth saved-objects export. It is imported into both OpenSearch Dashboards and Kibana (the file format is compatible with both).
grafana/Grafana-DMARC_Reports.json — the Grafana dashboard, with two Elasticsearch datasources (dmarc-ag, dmarc-fo).
splunk/ — three Splunk dashboard XML views (dmarc_aggregate, dmarc_forensic, smtp_tls).

Edits to any of these files should be exported from a running instance after authoring the change in the UI, not hand-edited (with the occasional exception of small XML tweaks for Splunk).

The dev stack

docker-compose.dashboard-dev.yml brings up every viz target at once so a single dashboard change can be authored and re-exported across all four UIs in one session. It include:s docker-compose.yml for the Elasticsearch and OpenSearch backends, then layers on Kibana, OpenSearch Dashboards, Grafana, and Splunk.

Service	URL	Credentials
Elasticsearch	http://localhost:9200	(security disabled)
OpenSearch	https://localhost:9201	`admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD`
Kibana	http://localhost:5601	(security disabled)
OpenSearch Dashboards	http://localhost:5602	`admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD`
Grafana	http://localhost:3000	`admin` / `$GRAFANA_PASSWORD`
Splunk Web / HEC	http://localhost:8000 / https://localhost:8088	`admin` / `$SPLUNK_PASSWORD`, HEC token `$SPLUNK_HEC_TOKEN`

All ports bind to 127.0.0.1 only.

Prerequisites

Docker with the Compose v2 plugin.
A repo-root .env defining the secrets the compose file references:
```
OPENSEARCH_INITIAL_ADMIN_PASSWORD=...
SPLUNK_PASSWORD=...
SPLUNK_HEC_TOKEN=...
GRAFANA_PASSWORD=...
```
Pick any values you like — these are local-only dev secrets. Both .env and parsedmarc*.ini are gitignored. The matching values must also appear in parsedmarc-dev.ini, which the bootstrap script feeds to the parsedmarc CLI for sample-data ingestion.
The parsedmarc CLI on PATH (or in ./venv/bin/) — pip install -e .[build] from the repo root works. Override the lookup with PARSEDMARC_BIN=/path/to/parsedmarc if needed.

One-shot bootstrap

dashboard-dev-bootstrap.sh is the normal entry point. It is idempotent — re-run it any time:

./dashboard-dev-bootstrap.sh

It does, in order:

docker compose -f docker-compose.dashboard-dev.yml up -d and waits for every service's health endpoint.
Provisions Splunk: creates the email index, creates the DMARC app, configures the auto-created HEC token to allow the email index, and scopes the search-app's "scheduled export" announcement view away from global so it stops appearing in the DMARC app's dashboard list.
Seeds Elasticsearch, OpenSearch, and Splunk with parsedmarc-parsed sample reports (from samples/) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass RESEED=1 to wipe and re-seed all three backends.
Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is edit in the UI → export → save into this directory → re-run the bootstrap script to verify the file imports cleanly into a fresh service.

VS Code users can run this via the Dev Dashboard: Bootstrap task in .vscode/tasks.json. Dev Dashboard: Up brings the stack up without importing or seeding.

Editing a dashboard

After running the bootstrap script once, the round trip for each platform is:

OpenSearch Dashboards (and Kibana)

Edit the dashboard at http://localhost:5602/ (OpenSearch Dashboards) — this is the canonical authoring surface.
Stack Management → Saved Objects → Export, select the DMARC dashboard, include related objects, and save the resulting .ndjson over opensearch/opensearch_dashboards.ndjson.
Re-run ./dashboard-dev-bootstrap.sh to confirm it re-imports cleanly into both OSD and Kibana. The Kibana CI workflow (.github/workflows/dashboards.yml) also imports the same file on every PR that touches it.

OSD imports default to the global_tenant so other admins on the instance can see the result. Set OSD_TENANT=... to import elsewhere.

Grafana

Edit the dashboard at http://localhost:3000/.
Dashboard settings → JSON Model, copy the JSON, save it to grafana/Grafana-DMARC_Reports.json.
Re-run the bootstrap script.

The bootstrap script provisions two elasticsearch datasources (dmarc-ag for dmarc_aggregate*, dmarc-fo for dmarc_forensic*) on first run; existing datasources are left alone.

Splunk

Edit the dashboard at http://localhost:8000/ inside the DMARC app.
Open the dashboard's Source view, copy the XML, and paste it over the matching file in splunk/ (dmarc_aggregate_dashboard.xml, dmarc_forensic_dashboard.xml, or smtp_tls_dashboard.xml).
Re-run the bootstrap script. It re-imports each view via DELETE + POST to the splunkd management API.

Reseeding sample data

RESEED=1 ./dashboard-dev-bootstrap.sh

Wipes every dmarc_aggregate* / dmarc_forensic* / smtp_tls* index from ES and OS, drops and recreates the Splunk email index, then re-runs the parsedmarc CLI against the curated sample list. Use this after changing parsedmarc's enrichment or output schemas.

Tearing the stack down

docker compose -f docker-compose.dashboard-dev.yml down          # stop containers, keep volumes
docker compose -f docker-compose.dashboard-dev.yml down -v       # also drop volumes (full reset)