mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-01 13:12:18 +00:00
a03b5074eeba084e9973ca4d7063aa2a772285d8
23
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
a421747a79 |
Add a Domain policy dropdown filter to the Splunk aggregate dashboard (#854) (#855)
Adds a dropdown that filters every panel on the published DMARC policy (published_policy.p) or subdomain policy (published_policy.sp), mirroring the Message disposition dropdown's choices and default and sitting immediately before it, per the issue spec. Both fields are always present on parsedmarc-written events (sp defaults to p in the parser), so the wildcard default matches everything, consistent with the other raw-search filters. Also fixes the Splunk docs page's XML-files link, which still pointed at the pre-#736 splunk/ path and 404ed. Closes #854 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
23c5ea9ad6 |
Fix DKIM/SPF and SMTP TLS detail-table cross-products in dashboards (#169) (#839)
* Fix DKIM/SPF alignment detail cross-product in dashboards (#169) Elasticsearch and OpenSearch dynamic-map the dkim_results/spf_results object arrays as `object` (create_indexes never registers the DSL document mappings), so Lucene flattens each array into independent multi-valued fields and stacked terms aggregations on dkim_results.selector/.domain/.result return every combination of values across a report's signatures — each phantom row repeating the full message count. Aggregate documents now also carry dkim_results_combined and spf_results_combined: one "selector / domain / result" ("scope / domain / result") string per auth result, composed in add_dkim_result/add_spf_result. The Kibana/OpenSearch Dashboards and Grafana (Elasticsearch) alignment-detail tables aggregate those instead, and the Splunk detail panels pair the values with mvzip/mvexpand. A documented idempotent _update_by_query backfills documents saved by older versions; the query matches only documents that have auth results and lack the combined fields, because an `exists` query cannot see an empty array. Also corrects the dead _SPFResult.results (plural) declaration to `result` (the save path always wrote the singular key), fixes the result parameter annotations on add_dkim_result/add_spf_result, and removes the Grafana dmarcian.com DKIM-checker data link, which required the separate domain/selector columns. The SMTP TLS visualizations have the same class of defect and are tracked separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review findings on #839 Reword the combined-field regression test docstrings: the DKIM/SPF auth results are dynamic-mapped as plain `object`, not the `nested` mapping type the previous wording implied — the distinction is the crux of the fix. Also drop the inert renameByName entries Copilot flagged on the Grafana Overview and DKIM Alignment Details panels, which referenced fields those panels' queries no longer (or never) produced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add PR #839 review lessons to AGENTS.md Extend the "Review passes cover prose" section with two rules from the #839 Copilot findings: docstrings/comments get the same text-level review pass as docs and dashboard labels (with suspicion for dual-use terms like "nested" near Elasticsearch code), and inert config entries inside hunks a PR already rewrites should be cleaned rather than preserved to minimize the diff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reflow create_indexes comments flagged by Copilot The line wrap placed "#169" directly after the comment marker, so the raw source read "# #169;". Reword so the issue reference stays on one line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Extend the hunk-proofreading rule with rendered-text wraps Fold the PR #839 second-round Copilot lesson into the existing rule: proofread how wrapped lines render (comment markers, punctuation at wrap points), not just the wording itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Rename Overview combined-result column labels (Copilot round 3) The Overview table's "DKIM Auth Result" / "SPF Auth Result" labels were kept when the columns switched to the combined "selector / domain / result" values, leaving the headers misleading. Rename them to match the detail panels' convention and retarget the byName width overrides that matched the old labels, widening them for the longer values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document per-signature row semantics in the alignment tables A message carrying multiple DKIM signatures appears once per signature in the details tables, so summing the messages column across rows can exceed the total message count. State that explicitly rather than leaving readers to infer it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot round-4 findings on dashboards Fix three pre-existing saved-object title typos in the OpenSearch ndjson (leading space on "Aggregate DMARC passed DMARC", trailing space on "Aggregate DMARC reporting organizations", double space in "map of message sources by country"), in both the top-level title and the embedded visState title. Normalize the Splunk DKIM details placeholders: the base search's fillnull renders wholly-missing DKIM fields as the literal string "null", so unsigned mail showed "null / null / null" while the SPF panel shows "none". Rewrite the values to "none" after the signature split, where the fields are single-valued and the mvzip pairing cannot be disturbed. Verified against the dev Splunk that no truncation or mis-pairing occurs either way, since fillnull guarantees the fields are never actually null. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Backfill combined DKIM/SPF fields automatically at startup migrate_indexes() now backfills dkim_results_combined and spf_results_combined on aggregate documents saved by older versions, so ES/OS users get historical data in the reworked alignment tables without running the documented _update_by_query by hand. The backfill is submitted as a non-blocking background task (wait_for_completion=false, conflicts=proceed) guarded by a cheap count query, making repeated startups a fast no-op once an index is backfilled; any cluster error is logged as a warning and retried at the next startup rather than raised. The manual command remains documented for users who upgrade dashboards without pointing the new parsedmarc at the cluster or who want to control write-load timing. The legacy published_policy.fo long-to-text reindex migration in the OpenSearch module is kept ahead of the new backfill, for clusters upgraded from very old data. Verified end-to-end against the live dev environment: a real CLI startup backfilled 9 stripped OpenSearch documents (logged with task ID) while the already-backfilled Elasticsearch side stayed silent, and a second startup was silent on both engines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Match backfill guard on either domain or result subfield End-to-end upgrade testing (real parsedmarc 10.2.4 ingest, then a branch startup) surfaced that an exists query cannot see an empty string: a text field with no tokens is invisible to exists. The parsers we audited never store an auth result with an empty or missing domain — they drop such entries entirely, so the previous domain-only guard was sufficient for their data — but the storage shape of every historical parsedmarc version can't be audited, so the guard (and the documented manual command) now matches either the domain or the result subfield per protocol. Matching either costs nothing and cannot skip a document that has something to backfill. Verified by recomputing expected combined values from _source for all 2,299 documents on both engines: every document with stored auth results has exactly the recomputed pairs, and the legacy fo migration correctly did not fire on typeless indexes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add auth-result filter controls to the Kibana/OSD aggregate dashboard The combined per-signature columns fixed the #169 cross-product but left no way to click-filter by an individual selector, domain, or result. Add an "Aggregate DMARC auth result filters" input_control_vis panel above the SPF/DKIM details tables with six option-list dropdowns (DKIM selector/domain/result, SPF scope/domain/result) that emit ordinary dashboard-wide filter pills. Works on both Kibana 8.19 and OpenSearch Dashboards 3, verified by driving the controls in both UIs against the issue's two-signature repro report. Documented in kibana.md, including the flat-mapping caveat: combining two component filters matches documents where any signature satisfies each condition individually. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scale Grafana source-country map markers with message volume The "Map of Message Source Countries" panel drew fixed 5 px dark-green markers at 50% opacity — nearly invisible on the dark basemap, so the panel read as empty even when data was flowing (verified via the query API). Markers now scale with Sum(message_count) (min 4, max 30 px) at 0.8 opacity in a higher-contrast green. Pre-existing issue; the identically-styled failure-dashboard map panel is intentionally left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Correct the nested-mapping rationale in the create_indexes comments The comments claimed Kibana/OSD/Grafana "cannot terms-aggregate fields inside a nested mapping" — too absolute. Fact-checked empirically and against primary docs: Kibana/OSD visual editors (Lens and classic Visualize) do not support nested fields, but Vega panels can run nested aggregations (they just cannot render tables, per Elastic's docs), and Grafana >= 9.4 has a nested bucket aggregation (grafana/grafana#62301) but no reverse_nested, so parent-level metrics like Sum(message_count) return 0 inside per-signature buckets (reproduced live). Conclusion unchanged: the dynamic object mapping stays load-bearing for the shipped dashboards. PR #839's body was updated with the same correction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Pin ruff exactly, matching the existing pyright pin rationale CI installs the [build] extra fresh on every run, and ruff was the one lint tool left unpinned. ruff 0.16.0 (released this week) began flagging this codebase's str.format() house style, so every PR started failing lint on lines it never touched. Pin to 0.15.21 — the version the codebase is clean under — with the same bump-deliberately comment pyright carries. Upgrading to 0.16 and converting to f-strings can be its own PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot findings: harden migrate_indexes, normalize panel titles Three unresolved review threads, all verified against cli.py's re-raising init handler before fixing: - elastic.py/opensearch.py: connections.get_connection() sat outside migrate_indexes()'s try/except, so a connection-registration failure would abort startup despite the docstring's promise that migration errors are caught and logged. Now caught, logged, and skipped until the next startup. - opensearch.py: the legacy published_policy.fo migration loop did unguarded network I/O (exists/get_field_mapping/reindex/delete), so a transient cluster error aborted startup on the OpenSearch path while the identical situation on the Elasticsearch path was logged and survived. Each index's migration attempt is now wrapped, warns, and moves on. - opensearch_dashboards.ndjson: normalized two pre-existing panel titles in the aggregate dashboard's panelsJSON ("Reporting organizations " trailing space, "Map of message sources by country" double space). Regression tests assert migrate_indexes never propagates connection or per-index cluster errors on either backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Clarify that Nested() on auth-result fields is in-memory shape only Copilot flagged that _AggregateReportDoc declares dkim_results and spf_results with Nested(...) while the create_indexes comment insists the stored mapping must stay dynamic `object`. Both are true: the Nested declaration only shapes the DSL's in-memory document building and is never installed as a mapping, because create_indexes skips Index.document() registration. Say so at both sites, in both backends, so nobody "fixes" the mismatch by registering the mapping — which would install real nested mappings and blank the dashboards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refer to the filter panel by its displayed title in docs and CHANGELOG The dashboard convention is a short panel display title backed by a long-form saved-object name ("SPF details" / "Aggregate DMARC SPF details"), and the new controls panel follows it. The docs and CHANGELOG named the panel by its saved-object title, which is not what a user sees on the dashboard; use the displayed "Auth result filters" instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix misspelled column label in the failure email samples table The "DMARC failure email samples" visualization labeled its authentication_results column "autentication_results". The underlying field reference was already correct; only the user-facing customLabel was misspelled. A sweep of every title and customLabel in the ndjson found no other misspellings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Extend the combined-field fix to SMTP TLS documents SMTP TLS reports have the same cross-product defect as the DKIM/SPF alignment tables (issue #169), one level deeper: policies is an object array and each policy's failure_details is an object array inside it, so stacked terms aggregations on their subfields fabricate rows. Documents now also carry policies_combined ("domain / type" per policy) and failure_details_combined ("domain / type / result / sending mta / receiving ip / mx" per failure detail), composed at save time with the same "none" fallbacks as the aggregate fields. migrate_indexes() gains smtp_tls_indexes and backfills old documents with the same guarded, non-blocking update_by_query pattern; cli.py wires the index name in on both backends, and the manual _update_by_query command is documented. Also fixes two adjacent dead fields: add_failure_details stored additional_information_uri under the wrong constructor kwarg (additional_information), and receiving_mx_hostname had no declaration despite always being stored. Verified live on ES 8.19 and OpenSearch 3: a two-policy repro report yields exactly 2 policy rows and 2 failure-detail rows via the combined fields where the old stacked aggregations return 4 of each; the startup backfill converted the 4 pre-existing sample documents on both engines with zero recompute mismatches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Rework the SMTP TLS dashboards onto the combined fields Kibana/OSD: "SMTP TLS domains" replaces its stacked policy_domain × policy_type terms with one terms agg on policies_combined.keyword; "SMTP TLS failure details" replaces six stacked terms spanning both array levels with one on failure_details_combined.keyword; the smtp_tls* index-pattern field cache gains the new fields. The "reporting organizations" table only buckets on doc-level org_name and needed no change. Splunk: the base search now expands policies at the JSON level (spath + mvexpand) so policy fields are scalars per event, and the failure details panel expands the second level the same way — sums are the detail's own failed_session_count, correctly paired. Verified via the search REST API: a two-policy repro returns exactly one row per real failure detail with per-detail counts. kibana.md documents the per-policy/per-detail row semantics and the honest caveat that session-count sums remain per report document. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Persist additional_info_uri from parsed SMTP TLS failure details Copilot caught that the savers read additional_information_uri from the parsed failure-detail dict, but the parser's key is additional_info_uri (SMTPTLSFailureDetailsOptional in types.py, set in parse_smtp_tls_report_json), so the URI was never persisted — the read-side half of the dead-field bug whose write-side half (wrong constructor kwarg) was fixed earlier. Read the parser's key first, keeping the long-form key as a fallback for dicts built by other callers. Regression test proven to fail on the unfixed savers. Also restructured the expected combined-string test values into named locals so no implicit string concatenation sits inside a list literal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Say "inner doc", not "nested doc", in the singular-key test docstrings Final review sweep: in this codebase "nested" is reserved for the Elasticsearch mapping type, and these InnerDoc-serialization docstrings used it colloquially — the same dual-use-term trap documented in AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
72bece4686 |
Add dashboard screenshot harness to the dev tooling (#840)
* Add dashboard screenshot harness to the dev tooling dashboard-dev-screenshots.py drives headless Chromium (Playwright) against the dashboard dev stack and captures how Kibana, OpenSearch Dashboards, Grafana, and Splunk actually render the current sample data — for end-to-end verification of dashboard changes and PR evidence. It encodes the platform quirks that otherwise cost time to rediscover: fixed render waits instead of networkidle (Kibana/OSD dashboards poll forever), pinning OSD to the global tenant (a stale private-tenant copy silently screenshots old dashboards), and driving the Grafana and Splunk login forms rather than HTTP basic auth. The output directory is gitignored; usage is documented in dashboards/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review findings on the screenshot harness - Only require OPENSEARCH_INITIAL_ADMIN_PASSWORD / SPLUNK_PASSWORD when the osd / splunk targets are selected, and fail fast before launching Playwright with a clear message naming the missing variable(s). - Honor GRAFANA_USER (defaulting to admin) to match the dev bootstrap script instead of hard-coding the Grafana username. - Print a full traceback to stderr when a target fails, instead of just the exception message, while still continuing to the next target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Always close the browser when a screenshot target fails Copilot round 2: each target only closed its Chromium instance on the happy path, so a mid-run Playwright failure leaked a headless browser while the script continued to the next target. Wrap each target body in try/finally; closing the browser also closes its contexts and pages, so the single b.close() covers the OSD context too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Exit with an install hint when playwright is missing Copilot round 3: playwright is deliberately not a project dependency, so a bare import failure produced a raw ModuleNotFoundError traceback. Catch ImportError and exit with the same one-time install command the module docstring documents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
6acacd01d1 |
Address post-merge Copilot findings from #834 (#836)
- Raise the compliance table's TSVB pivot_rows from 1000 to 10000, matching the 10k terms size of the agg-based table it replaced, so large installations don't lose per-domain rows. - Guard the Splunk compliance column's division explicitly: if(Messages>0, round(...), null()). SPL already returns NULL on division by zero, and null (not 0.0%) matches the PostgreSQL panel's NULLIF behavior for zero-message domains, so this makes the intended semantics explicit without changing results. Both verified against the dashboard dev stack: the guarded SPL returns identical values on seeded data, and the ndjson was round-tripped through OSD (global tenant) and re-imports 26/26 into both OSD and Kibana 8.19. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
62514bd72a |
Add per-domain DMARC compliance percentage to all aggregate dashboards (#834)
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:
- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
sum of message_count), pivoted on header_from.keyword. The time field
is date_begin rather than the multi-valued date_range, which TSVB's
per-value date histogram would double-count. Editing (not rendering)
the panel on Kibana 8.x requires the metrics:allowStringIndices
advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
stats, per the SPL eval-in-stats syntax.
All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).
Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.
Closes #112
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix over-time charts double-counting reports via multi-valued date_range
date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.
Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.
All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.
Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review comments on PR #834
- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed DMARC" -> "failed DMARC" and the
backticked `ruf ` trailing space in the RUF explainer panel, and
rename the "SMPT TLS failure details" visualization to "SMTP TLS
failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
plugin-install restart so a hang fails with a clear timeout message
instead of an opaque downstream curl error.
The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* AGENTS.md: reviews must cover prose and hunk context, not just function
Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second round of Copilot review comments
- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
matching the repo convention where the release commit assigns the
version number (see
|
||
|
|
1986835f51 |
Fix daily-cadence charts defaulting to sub-day time buckets (#828) (#830)
DMARC aggregate reports post one data point per reporting period (typically daily), but several dashboard time-series panels used a dynamically-computed bucket interval that scales with the viewed time range/panel width instead of the data's actual cadence, producing a spiked/gapped chart whenever that computed interval landed below a day: - OpenSearch Dashboards: both date_histogram aggregations used "interval": "auto". Fixed to the "d" unit code (verified against OpenSearch-Dashboards' and Kibana's own _interval_options.ts / parse_interval.ts source — "day" is not a valid value and would throw at render time). - Grafana + PostgreSQL: 7 query targets used $__timeGroup(col, $__interval); hardcoded to $__timeGroup(col, '1d'), matching the equivalent already-correct panels in the Grafana + Elasticsearch dashboard. - Splunk: both `timechart` panels had no explicit span, so Splunk's 100-bin auto-algorithm only coincidentally produced daily buckets for the dashboard's default 7-day view. Added span=1d per Splunk's own timechart documentation. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
e104f1118c |
Land 10.0.3 changes on master (#785)
PR #784 was stacked on the #783 branch and its base was never retargeted to master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master. master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes. This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the >=2.2.1 floor is already on master from 10.0.2). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
d92593f2da | Add RFC 9990 fields to opensearch_dashboards.json | ||
|
|
180fc581fe |
fix: OSD Global-tenant import + dropped report files with glob metacharacters; validate dev stack on OpenSearch 3.x with PostgreSQL (#781)
* fix: import OpenSearch dashboards into the real Global tenant dashboard-dev-bootstrap.sh sent `securitytenant: global_tenant`. The OpenSearch security plugin reads that header as a tenant *name*, and `global_tenant` is a sample custom tenant from the security demo config -- not the shared Global tenant, whose token is the literal `global`. The import therefore landed in a separate `global_tenant` tenant (its own `.kibana_<hash>_globaltenant_1` index) and the dashboards were invisible to anyone viewing the Global tenant in OpenSearch Dashboards. Verified against the live dev cluster: `_find` under `securitytenant: global` returned 26 objects and `.kibana_1` (the Global tenant index the UI reads) went from 2 to 67 docs after re-importing with the fix. An empty/omitted header read 0 from Global -- it falls back to the user's configured default tenant -- so `global` is the only reliable token. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't drop report files whose names contain glob metacharacters The CLI expanded every file argument with glob(), which treats [, ], *, and ? as pattern syntax. A literal path like "[Netease DMARC Failure Report] Rent Reminder.eml" -- the bracketed shape many providers use for emailed failure reports -- was read as a character class, matched nothing, and was dropped before reaching the parser, with no error. File arguments that exist on disk are now taken literally; only non-existent paths are globbed, so shell-style wildcards still expand. Also adds "postgresql" to _KNOWN_SECTIONS so PARSEDMARC_POSTGRESQL_* env vars (and their _FILE Docker-secret variants) resolve like every other backend -- the PostgreSQL backend is new in 10.0.0, so this completes the unreleased feature rather than fixing a released regression, and is documented under the PostgreSQL enhancement, not Bug fixes. Regression tests added for both. Verified end-to-end: all four samples/failure/*.eml now index (the bracketed Netease report included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: validate dashboards on OpenSearch 3.x and add PostgreSQL to the dev stack The dev stack ran OpenSearch Dashboards 3.x against OpenSearch 2.x, an unsupported cross-major pairing. Bump opensearch to :3 (validated on 3.6.0: OSD import into the Global tenant and all dashboards work). Add a postgresql service plus bootstrap wiring so the new PostgreSQL backend is exercised alongside the others: wait for PG, seed it via PARSEDMARC_POSTGRESQL_* env vars on the same parsedmarc run, wipe it on RESEED, create a Grafana grafana-postgresql-datasource (uid dmarc-pg), and import dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json. PG seeding is gated on psycopg being importable: parsedmarc aborts the whole run (exit 1, nothing written to any backend) when a configured output backend can't initialize, so wiring in PG without the optional extra would silently zero ES/OS/Splunk too. When psycopg is absent the script warns and skips PG, leaving the other backends seeded. Also fix the Grafana admin password env: the container was given GRAFANA_PASSWORD, which Grafana ignores -- it reads GF_SECURITY_ADMIN_PASSWORD. Defaults to admin to match the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: list PostgreSQL on the premade-dashboards features bullet PostgreSQL ships a premade Grafana dashboard (dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json), so it belongs on the "for use with premade dashboards" bullet alongside Elasticsearch, OpenSearch, and Splunk rather than on the plain-output-destinations line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: clear stale org_email mapping conflict in the OpenSearch dashboards The aggregate index pattern in dashboards/opensearch/opensearch_dashboards.ndjson shipped a cached field-list snapshot where org_email was a text/object conflict, plus leftover org_email.#text and org_email.#text.keyword subfields. Those came from a cluster that had indexed a langAttrString email dict ({"#text": ..., "@lang": ...}) before the parser unwrapped it. org_email is mapped as Text() and parse_aggregate_report_xml now unwraps a dict email to a plain string, so current data is consistently text -- a clean cluster's _field_caps reports no conflict. Cleared the frozen conflict and the two artifact subfields, leaving org_email (text) and org_email.keyword, matching the live mapping. Verified: re-importing the corrected ndjson yields an index pattern with org_email as a plain text field and zero conflicts; only the aggregate index-pattern line changed, all other saved objects byte-identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: seed the RFC 9990 (dmarc-2.0) aggregate samples samples/aggregate/rfc9990-sample.xml and rfc9990-example.net!...xml were not in the bootstrap's SAMPLE_FILES, so the dev stack only ever indexed RFC 7489 reports and the new DMARCbis fields (np, testing, discovery_method, generator, xml_namespace) never appeared in the OpenSearch/Kibana indices or were available to the dashboards. Added both samples (one declares the urn:ietf:params:xml:ns:dmarc-2.0 namespace, the other is namespaceless RFC 9990-shaped, covering both detection paths). Verified the seeded data now carries np/testing/ discovery_method/generator and xml_namespace=urn:ietf:params:xml:ns:dmarc-2.0; OpenSearch Dashboards surfaces them on an index-pattern field-list refresh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: auto-resolve (or create) a venv for the seed and ensure psycopg The seed previously required parsedmarc to be pre-installed and only warned-and-skipped PostgreSQL when psycopg was missing. Resolve the seed environment by precedence instead: 1. explicit PARSEDMARC_BIN -> used as-is, nothing installed 2. active $VIRTUAL_ENV 3. existing repo venv/ or .venv/ 4. otherwise create $REPO_ROOT/venv For cases 2-4, run `pip install -e .[postgresql]` only when the CLI or psycopg is missing, so the dev stack can populate Postgres out of the box without a manual install step. The explicit-PARSEDMARC_BIN path is left untouched (and the psycopg seed guard still warns/skips if that env lacks the extra). Verified: a RESEED run resolves the active venv, seeds ES/OS/Splunk/PG including the RFC 9990 fields, with no output-client errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
2b3cd32b9c |
chore: move PostgreSQL Grafana dashboard into dashboards/grafana/ (#780)
The PostgreSQL dashboard shipped at the repo-root grafana/ by oversight; every other dashboard source lives under dashboards/ (opensearch/, grafana/, splunk/). Move it next to the existing Grafana dashboard, list it in dashboards/README.md, and fix the CHANGELOG path reference. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
a6778707d7 |
Finish forensic→failure rename: archive-folder migration + dashboard/doc cleanup (#776)
The forensic→failure rename (#659) left a few loose ends and one deliberate hold-back. This closes them. Leftover rename misses (broken paths / stale canonical names): - CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/* - dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml → dmarc_failure_dashboard.xml (the file was already renamed; the import path and view name were not) - docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example - samples/parsedmarc.ini: save_forensic → save_failure - pyproject.toml, README.md: canonical "failure" naming (ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.) Archive subfolder rename + on-startup migration: - New failure reports now archive to <archive>/Failure (was <archive>/Forensic). - _migrate_forensic_archive_folder() runs once on startup (best-effort): renames Forensic→Failure when no Failure folder exists yet, merges the two when both exist, no-ops when there's no legacy folder, and logs-and-skips a mailbox it can't reorganize (warn, don't crash). This consolidates pre- and post-rename failure reports into one folder, replacing the previously documented decision to keep the folder named Forensic to avoid a split archive. Uses the folder-management API (folder_exists / rename_folder / merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0. Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana): - Forensic panel titles + the datasource label → Failure; the fo-column display label and its linked byName field-override matcher both → "Failure Policy" (changed together so the column-width override keeps matching). - dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana dashboards); RESEED wipe loop now also clears dmarc_failure* indices. - Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an orphan export accidentally committed in #736 and referenced by nothing. Tests (tests/test_init.py): - TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds). - TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir can't reproduce — a backend that raises mid-operation must warn, not crash. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
ae1e5adb66 |
Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659)
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @
|
||
 Sean Whalen
|
02a8014893 | Fix splunk SMTP TLS dashboard: add additional renames for failure details and adjust stats query | ||
|
Sean Whalen
|
8317ffcde8 | Fix rename syntax for parsed_sample headers in Splunk DMARC forensic dashboard | ||
|
Sean WhalenandGitHub
|
3b9e678533 |
Refactor SMTP TLS dashboard with base search
Refactored the SMTP TLS Splunk dashboard to use a base search for improved query efficiency and maintainability. Updated input token names and adjusted search queries for better organization and clarity. |
||
|
Sean Whalen
|
5ba72d2783 | Add source AS name to fillnull and search queries in DMARC aggregate dashboard | ||
|
Sean Whalen
|
e40b53da64 | Enhance Splunk DMARC aggregate dashboard: add source AS name dropdown and update search queries | ||
|
Sean Whalen
|
fe296ca869 |
Update dashboard documentation
- Introduced a new README.md for dashboard development with detailed instructions. - Removed outdated README files for Grafana and Splunk dashboards. |
||
|
Sean Whalen
|
900ee22525 | Make map and country list side by side in the Splunk DMARC aggregate dashboard XML | ||
|
Sean Whalen
|
e709839f79 | Fix typo in source ip viz | ||
|
|
265bf64240 |
Align Grafana dashboard with OpenSearch Dashboards source-of-truth (#738)
* Align Grafana dashboard with OpenSearch Dashboards source-of-truth Adds the two aggregate-DMARC panels that exist on the OSD dashboard but were missing from the bundled Grafana dashboard: - "Message sources by name and type" — buckets by source_name + source_type, sums message_count per (name, type) tuple. Mirrors the OSD viz from 9.4.x. - "Message sources by Autonomous System" — buckets by source_asn + source_as_name + source_as_domain, sums message_count per ASN. Mirrors the OSD viz added in 9.9.0 with the IPinfo Lite ASN integration. Both panels are patterned on the existing "Reporting Organisations" panel (same datasource $datasourceag, same sum(message_count) metric, same gradient-gauge "Messages" column with rename transforms). They sit at the bottom of the existing layout (gridPos y=129 and y=140) so the existing panel positions are unchanged. Verified against the bundled grafana/grafana:12.3.0: dashboard import returns status=success, both panels render with real data from the sample-corpus indexes, and the ES aggregations (terms on source_name + source_type, numeric terms on source_asn) return the expected results. Out of scope: - Extras in the Grafana dashboard that aren't on OSD (SPF/DKIM Results Over Time, Alignment Over Time, Stat overview, Published Policies, Forensic IP / country tables) are left in place. They were community-contributed and likely valued by some users. - Migrating the deprecated `graph` and `grafana-worldmap-panel` panel types to modern timeseries / geomap is a separate, larger task. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Grafana: migrate deprecated graph and worldmap panels Replaces the 6 legacy `graph` panels with `timeseries` panels and the 2 legacy `grafana-worldmap-panel` panels with `geomap` panels. Both deprecated plugins still rendered in Grafana 12 via auto-migration but were flagged for removal; this ships the modern saved shape. graph -> timeseries (6 panels): SPF Results Over Time, DKIM Results Over Time, SPF Alignment Over Time, DKIM Alignment Over Time, DMARC Passage Over Time, Message Disposition Over Time. Panel `aliasColors` (e.g. {true: dark-green, false: dark-red}) are translated into per-series `fieldConfig.overrides` so the green/red by-pass-fail colorings carry forward; legacy graph fields (lines, fill, yaxes, tooltip etc.) are dropped in favor of the new `fieldConfig.defaults.custom` block and `options.legend` / `options.tooltip`. worldmap -> geomap (2 panels): Map of Message Source Countries (aggregate), Forensic Sample Sources by Country (forensic). The legacy `locationData=countries` lookup-by-ISO becomes a geomap markers layer with `location.mode=lookup`, `gazetteer=public/gazetteer/countries.json`, and `lookup=source_country.keyword` — same input data, modern renderer. Drops the date_histogram bucket from the geomap targets since the map is a snapshot over the panel time range, not a time series. Verified against the bundled grafana/grafana:12.3.0: dashboard imports with status=success and `version=19`, live panel types now report `{timeseries: 6, geomap: 2, table: 14, grafana-piechart-panel: 3, stat: 1, row: 3}` — no more `graph` or `grafana-worldmap-panel` entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
4e8c28bbc0 |
Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737)
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
826e78c390 |
Fix DMARC dashboard metrics (OSD + Splunk) and add dashboard-dev bootstrap (#736)
* OSD: fix aggregate dashboard metrics to sum(message_count) 13 panels on the DMARC aggregate dashboard were aggregating with `count` (number of OSD docs) when they should have been summing `message_count`. Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from the XML and carries an integer message_count, so doc-counting reports "distinct sources" rather than "messages". Panels with titles like "Message volume by header from", "DMARC passage over time", etc. were producing misleading numbers. Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and DKIM details. (DMARC failure email samples kept count — one OSD doc per RUF sample, so it's correct. SMTP TLS panels untouched — they sum the right session-count fields.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: align dashboards with OSD and fix query bugs Aggregate dashboard: - Add "Message sources by Autonomous System" panel (source_asn / as_name / as_domain), formatted "AS<n>" at render with eval, matching the OSD addition. - DKIM details: add the missing dkim_aligned column. - SPF details: reorder columns to OSD order (spf_aligned at end). - Map / country titles renamed to match OSD ("Map of message sources by country", "Message sources by country"). - Map widget: stats count by Country -> stats sum(message_count) by Country, so the choropleth shades by message volume not record count. - fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain, source_country to mirror OSD's missing-bucket labels. - charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart. Forensic dashboard: - Restructure to match OSD's two-panel layout (markdown + samples table). - Drop the country map / IP table / country-ISO table panels (not in OSD). - Samples table columns aligned to OSD: arrival_date_utc, source.ip_address, from, subject, reply_to, authentication_results. - Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=* required field to exist; LinkedIn RUF sample with null From was filtered out). SMTP TLS dashboard: - Reorder metrics to OSD order (successful before failed). - Domains panel: add policy_type bucket. - Failure details: replace search-time `failed_session_count>0` (which doesn't evaluate against multivalued JSON paths in Splunk) with `result_type=*` for presence + post-stats `where failed_sessions>0`. Drop _time/successful_sessions columns; reorder to match OSD. - Wire the existing policy_type input into all three searches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add dashboard-dev bootstrap script and VSCode task dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml, seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini, and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana, and Splunk. Idempotent: existence checks skip provisioning that's already done; only the dashboard imports re-run unconditionally on every invocation (that's the point of running it after a dashboard edit). Notable provisioning quirks the script handles: - Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships with indexes=[] and index=default; rewrites it to allow the email index. - ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily parsedmarc indexes via _cat/indices and deletes one at a time. - Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1 deletes and recreates the email index (then re-applies the HEC token). - OSD security plugin tenants: imports target global_tenant explicitly via the securitytenant header so they're visible to the shared workspace rather than landing in the API user's private tenant. Override with OSD_TENANT=<name>. - Splunk ships an in-product announcement view (scheduled_export_dashboard) with sharing=global; the script narrows it to sharing=app so it stops showing up in every app's dashboards list. Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump version to 9.10.3 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: warn against the "Create new objects with unique IDs" import mode OSD's import dialog has two modes: the default "Check for existing objects" (which honors saved-object IDs and overwrites in place when "Automatically overwrite conflicts" is on) and "Create new objects with unique IDs" (which imports under fresh UUIDs and leaves the buggy originals untouched). Picking the second one means the dashboards keep rendering the wrong numbers because the originals are never replaced. Spell that out so users don't fall into the trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: label the metric column "messages" instead of "Sum of message_count" OSD's table column header defaults to "Sum of message_count" when the metric agg has no customLabel. "messages" reads better and matches what the panels are actually counting. Applies to all 15 aggregate-DMARC visualizations that use sum(message_count). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable Trim the verbose technical exposition; lead each fix with the user-visible symptom. Move the action-required call out to its own header in upgrade notes so the re-import instructions don't get lost in a wall of text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move per-tool dashboard exports under a single dashboards/ directory Consolidates the four sibling top-level folders (kibana/, opensearch/, grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/. Updates the only path references in tracked files: bootstrap script (5 lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in docs/source/elasticsearch.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: restore the "DKIM alignment" panel title on the aggregate dashboard The DKIM alignment panel had no title override in panelsJSON, so OSD fell back to the visualization's own name ("Aggregate DMARC DKIM alignment"). Every other pie/table on the same dashboard sets a clean title (SPF alignment, Passed DMARC, etc.) — this was a stray regression. Set the panel title to "DKIM alignment" to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: color the message-disposition timechart by severity Reject is red, quarantine is yellow, none is green — same semantic mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart, applied via charting.fieldColors. Matches OSD's existing color overrides on the equivalent viz. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: clarify that "Create new objects with unique IDs" is the default The OSD import dialog defaults to that mode — users have to actively switch away from it, not just avoid picking it. Reword the upgrade note to lead with the switch and explain why the default would silently preserve the bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |