- Raise the compliance table's TSVB pivot_rows from 1000 to 10000,
matching the 10k terms size of the agg-based table it replaced, so
large installations don't lose per-domain rows.
- Guard the Splunk compliance column's division explicitly:
if(Messages>0, round(...), null()). SPL already returns NULL on
division by zero, and null (not 0.0%) matches the PostgreSQL panel's
NULLIF behavior for zero-message domains, so this makes the intended
semantics explicit without changing results.
Both verified against the dashboard dev stack: the guarded SPL returns
identical values on seeded data, and the ndjson was round-tripped
through OSD (global tenant) and re-imports 26/26 into both OSD and
Kibana 8.19.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:
- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
sum of message_count), pivoted on header_from.keyword. The time field
is date_begin rather than the multi-valued date_range, which TSVB's
per-value date histogram would double-count. Editing (not rendering)
the panel on Kibana 8.x requires the metrics:allowStringIndices
advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
stats, per the SPL eval-in-stats syntax.
All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).
Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.
Closes#112
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix over-time charts double-counting reports via multi-valued date_range
date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.
Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.
All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.
Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review comments on PR #834
- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed DMARC" -> "failed DMARC" and the
backticked `ruf ` trailing space in the RUF explainer panel, and
rename the "SMPT TLS failure details" visualization to "SMTP TLS
failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
plugin-install restart so a hang fails with a clear timeout message
instead of an opaque downstream curl error.
The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* AGENTS.md: reviews must cover prose and hunk context, not just function
Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second round of Copilot review comments
- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
matching the repo convention where the release commit assigns the
version number (see 855d267 for 10.2.4).
- docker-compose.yml: make the dev-stack Elasticsearch heap overridable
via ES_JAVA_OPTS in .env (default unchanged at 2g), using the compose
file's existing ${VAR:-default} idiom, for smaller machines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
DMARC aggregate reports post one data point per reporting period
(typically daily), but several dashboard time-series panels used a
dynamically-computed bucket interval that scales with the viewed time
range/panel width instead of the data's actual cadence, producing a
spiked/gapped chart whenever that computed interval landed below a day:
- OpenSearch Dashboards: both date_histogram aggregations used
"interval": "auto". Fixed to the "d" unit code (verified against
OpenSearch-Dashboards' and Kibana's own _interval_options.ts /
parse_interval.ts source — "day" is not a valid value and would throw
at render time).
- Grafana + PostgreSQL: 7 query targets used
$__timeGroup(col, $__interval); hardcoded to $__timeGroup(col, '1d'),
matching the equivalent already-correct panels in the Grafana +
Elasticsearch dashboard.
- Splunk: both `timechart` panels had no explicit span, so Splunk's
100-bin auto-algorithm only coincidentally produced daily buckets for
the dashboard's default 7-day view. Added span=1d per Splunk's own
timechart documentation.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.
This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @ 2cda5bf (9.9.0), which added the ASN
source attribution work (#712, #713, #714, #715). Individual Copilot
iteration commits squashed into this single commit — the per-commit
history on the feature branch was iterative (add tests, fix lint,
move field, revert, etc.) and not worth preserving; GitHub squash-
merges PRs anyway.
New fields from the DMARCbis XSD, plumbed through types, parsing, CSV
output, and the Elasticsearch / OpenSearch mappings:
- ``np`` — non-existent subdomain policy (``none`` / ``quarantine`` /
``reject``)
- ``testing`` — testing mode flag (``n`` / ``y``), replaces RFC 7489
``pct``
- ``discovery_method`` — policy discovery method (``psl`` /
``treewalk``)
- ``generator`` — report generator software identifier (metadata)
- ``human_result`` — optional descriptive text on DKIM / SPF results
RFC 7489 reports parse with ``None`` for DMARCbis-only fields.
Forensic reports have been renamed to failure reports throughout the
project to reflect the proper naming since RFC 7489.
- Core: ``types.py``, ``__init__.py`` — ``ForensicReport`` →
``FailureReport``, ``parse_forensic_report`` →
``parse_failure_report``, report type ``"failure"``.
- Output modules: ``elastic.py``, ``opensearch.py``, ``splunk.py``,
``kafkaclient.py``, ``syslog.py``, ``gelf.py``, ``webhook.py``,
``loganalytics.py``, ``s3.py``.
- CLI: ``cli.py`` — args, config keys, index names
(``dmarc_failure``).
- Docs + dashboards: all markdown, Grafana JSON, Kibana NDJSON,
Splunk XML.
Backward compatibility preserved: old function / type names remain as
aliases (``parse_forensic_report = parse_failure_report``,
``ForensicReport = FailureReport``, etc.), CLI accepts both the old
(``save_forensic``, ``forensic_topic``) and new (``save_failure``,
``failure_topic``) config keys, and updated dashboards query both
old and new index / sourcetype names so data from before and after
the rename appears together.
Merge conflicts resolved in ``parsedmarc/constants.py`` (took bis's
10.0.0 bump), ``parsedmarc/__init__.py`` (combined bis's "failure"
wording with master's IPinfo MMDB mention), ``parsedmarc/elastic.py``
and ``parsedmarc/opensearch.py`` (kept master's ``source_asn`` /
``source_asn_name`` / ``source_asn_domain`` on the failure doc path
while renaming ``forensic_report`` → ``failure_report``), and
``CHANGELOG.md`` (10.0.0 entry now sits above the 9.9.0 entry).
All 324 tests pass; ``ruff check`` / ``ruff format --check`` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Apply post-RFC review fixes: RFC 9990 detection, langAttrString, CFWS-aware RUF parsing
Aligns the implementation with the final RFCs (9989/9990/9991) instead of
inferring DMARCbis support from the version element or the namespace alone.
Aggregate parsing (RFC 9990):
- _text() helper unwraps langAttrString values (extra_contact_info, error,
comment, human_result, generator) — when reporters include the lang
attribute, xmltodict yields {"#text": ..., "@lang": ...} dicts instead
of strings; the parser now stores the text payload in both shapes.
- New xml_namespace field on AggregateReport records the declared XML
namespace (urn:ietf:params:xml:ns:dmarc-2.0 for RFC 9990 reports).
- RFC 9990 detection accepts namespaceless reports that follow the
RFC 9990 shape (presence of np / testing / discovery_method / generator),
so reporters that don't declare the namespace still receive RFC 9990-
aware validation.
- Warnings: missing DKIM <selector> (REQUIRED in RFC 9990); legacy
forwarded / sampled_out policy-override types (removed by RFC 9990);
unknown policy-override types per the RFC 9990 enumeration.
- xml_namespace added to Elasticsearch and OpenSearch document mappings.
Failure parsing (RFC 9991):
- Identity-Alignment and Auth-Failure are split on commas with CFWS
whitespace stripped per the RFC 9991 ABNF; previously "dkim, spf"
yielded ["dkim", " spf"] with a leading space on the second token.
- Warnings logged when either REQUIRED field is missing.
Terminology: every reference to "DMARCbis" in code, tests, sample
filenames, AGENTS.md, and CHANGELOG.md is replaced with the appropriate
RFC number (9989 for the policy spec, 9990 for aggregate reports, 9991
for failure reports). Sample contents are unchanged.
Docs: corrects the prior claim that fo was dropped from RFC 9990 (only
pct was), reframes testing as a new field (not a pct replacement, since
RFC 9989 Appendix A.6 removed pct with no per-message substitute), and
documents the policy_override_reason enum changes (added policy_test_mode;
removed forwarded / sampled_out).
Tests: 8 new tests covering xml_namespace capture, RFC 9990 detection
from field shape, missing-DKIM-selector warning, legacy-override-type
warning, langAttrString unwrapping across all four affected elements,
and CFWS-aware Identity-Alignment / Auth-Failure parsing plus their
missing-field warnings. 276 tests total, all passing; ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refactored the SMTP TLS Splunk dashboard to use a base search for improved query efficiency and maintainability. Updated input token names and adjusted search queries for better organization and clarity.
* OSD: fix aggregate dashboard metrics to sum(message_count)
13 panels on the DMARC aggregate dashboard were aggregating with `count`
(number of OSD docs) when they should have been summing `message_count`.
Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from
the XML and carries an integer message_count, so doc-counting reports
"distinct sources" rather than "messages". Panels with titles like "Message
volume by header from", "DMARC passage over time", etc. were producing
misleading numbers.
Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by
reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and
DKIM details. (DMARC failure email samples kept count — one OSD doc per
RUF sample, so it's correct. SMTP TLS panels untouched — they sum the
right session-count fields.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Splunk: align dashboards with OSD and fix query bugs
Aggregate dashboard:
- Add "Message sources by Autonomous System" panel (source_asn / as_name /
as_domain), formatted "AS<n>" at render with eval, matching the OSD addition.
- DKIM details: add the missing dkim_aligned column.
- SPF details: reorder columns to OSD order (spf_aligned at end).
- Map / country titles renamed to match OSD ("Map of message sources by
country", "Message sources by country").
- Map widget: stats count by Country -> stats sum(message_count) by
Country, so the choropleth shades by message volume not record count.
- fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain,
source_country to mirror OSD's missing-bucket labels.
- charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC
pies and the DMARC-passage timechart.
Forensic dashboard:
- Restructure to match OSD's two-panel layout (markdown + samples table).
- Drop the country map / IP table / country-ISO table panels (not in OSD).
- Samples table columns aligned to OSD: arrival_date_utc, source.ip_address,
from, subject, reply_to, authentication_results.
- Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=*
required field to exist; LinkedIn RUF sample with null From was filtered out).
SMTP TLS dashboard:
- Reorder metrics to OSD order (successful before failed).
- Domains panel: add policy_type bucket.
- Failure details: replace search-time `failed_session_count>0` (which
doesn't evaluate against multivalued JSON paths in Splunk) with
`result_type=*` for presence + post-stats `where failed_sessions>0`.
Drop _time/successful_sessions columns; reorder to match OSD.
- Wire the existing policy_type input into all three searches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add dashboard-dev bootstrap script and VSCode task
dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml,
seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini,
and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana,
and Splunk. Idempotent: existence checks skip provisioning that's already
done; only the dashboard imports re-run unconditionally on every invocation
(that's the point of running it after a dashboard edit).
Notable provisioning quirks the script handles:
- Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships
with indexes=[] and index=default; rewrites it to allow the email index.
- ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily
parsedmarc indexes via _cat/indices and deletes one at a time.
- Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1
deletes and recreates the email index (then re-applies the HEC token).
- OSD security plugin tenants: imports target global_tenant explicitly
via the securitytenant header so they're visible to the shared workspace
rather than landing in the API user's private tenant. Override with
OSD_TENANT=<name>.
- Splunk ships an in-product announcement view (scheduled_export_dashboard)
with sharing=global; the script narrows it to sharing=app so it stops
showing up in every app's dashboards list.
Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs
the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump version to 9.10.3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: warn against the "Create new objects with unique IDs" import mode
OSD's import dialog has two modes: the default "Check for existing objects"
(which honors saved-object IDs and overwrites in place when "Automatically
overwrite conflicts" is on) and "Create new objects with unique IDs" (which
imports under fresh UUIDs and leaves the buggy originals untouched). Picking
the second one means the dashboards keep rendering the wrong numbers because
the originals are never replaced. Spell that out so users don't fall into
the trap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* OSD: label the metric column "messages" instead of "Sum of message_count"
OSD's table column header defaults to "Sum of message_count" when the
metric agg has no customLabel. "messages" reads better and matches what
the panels are actually counting.
Applies to all 15 aggregate-DMARC visualizations that use sum(message_count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable
Trim the verbose technical exposition; lead each fix with the user-visible
symptom. Move the action-required call out to its own header in upgrade
notes so the re-import instructions don't get lost in a wall of text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Move per-tool dashboard exports under a single dashboards/ directory
Consolidates the four sibling top-level folders (kibana/, opensearch/,
grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/.
Updates the only path references in tracked files: bootstrap script (5
lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in
docs/source/elasticsearch.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* OSD: restore the "DKIM alignment" panel title on the aggregate dashboard
The DKIM alignment panel had no title override in panelsJSON, so OSD fell
back to the visualization's own name ("Aggregate DMARC DKIM alignment").
Every other pie/table on the same dashboard sets a clean title (SPF
alignment, Passed DMARC, etc.) — this was a stray regression. Set the
panel title to "DKIM alignment" to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Splunk: color the message-disposition timechart by severity
Reject is red, quarantine is yellow, none is green — same semantic
mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage
timechart, applied via charting.fieldColors. Matches OSD's existing
color overrides on the equivalent viz.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: clarify that "Create new objects with unique IDs" is the default
The OSD import dialog defaults to that mode — users have to actively
switch away from it, not just avoid picking it. Reword the upgrade note
to lead with the switch and explain why the default would silently
preserve the bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sean Whalen