mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-01 21:22:18 +00:00
e892de794b34c042cde7c7df50e6a4afe72a5331
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Sean WhalenandClaude Fable 5
|
2ebe26681e |
Document per-signature row semantics in the alignment tables
A message carrying multiple DKIM signatures appears once per signature in the details tables, so summing the messages column across rows can exceed the total message count. State that explicitly rather than leaving readers to infer it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Fable 5
|
6148002a43 |
Fix DKIM/SPF alignment detail cross-product in dashboards (#169)
Elasticsearch and OpenSearch dynamic-map the dkim_results/spf_results
object arrays as `object` (create_indexes never registers the DSL
document mappings), so Lucene flattens each array into independent
multi-valued fields and stacked terms aggregations on
dkim_results.selector/.domain/.result return every combination of
values across a report's signatures — each phantom row repeating the
full message count.
Aggregate documents now also carry dkim_results_combined and
spf_results_combined: one "selector / domain / result"
("scope / domain / result") string per auth result, composed in
add_dkim_result/add_spf_result. The Kibana/OpenSearch Dashboards and
Grafana (Elasticsearch) alignment-detail tables aggregate those
instead, and the Splunk detail panels pair the values with
mvzip/mvexpand. A documented idempotent _update_by_query backfills
documents saved by older versions; the query matches only documents
that have auth results and lack the combined fields, because an
`exists` query cannot see an empty array.
Also corrects the dead _SPFResult.results (plural) declaration to
`result` (the save path always wrote the singular key), fixes the
result parameter annotations on add_dkim_result/add_spf_result, and
removes the Grafana dmarcian.com DKIM-checker data link, which
required the separate domain/selector columns.
The SMTP TLS visualizations have the same class of defect and are
tracked separately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
62514bd72a |
Add per-domain DMARC compliance percentage to all aggregate dashboards (#834)
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:
- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
sum of message_count), pivoted on header_from.keyword. The time field
is date_begin rather than the multi-valued date_range, which TSVB's
per-value date histogram would double-count. Editing (not rendering)
the panel on Kibana 8.x requires the metrics:allowStringIndices
advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
stats, per the SPL eval-in-stats syntax.
All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).
Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.
Closes #112
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix over-time charts double-counting reports via multi-valued date_range
date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.
Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.
All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.
Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review comments on PR #834
- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed DMARC" -> "failed DMARC" and the
backticked `ruf ` trailing space in the RUF explainer panel, and
rename the "SMPT TLS failure details" visualization to "SMTP TLS
failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
plugin-install restart so a hang fails with a clear timeout message
instead of an opaque downstream curl error.
The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* AGENTS.md: reviews must cover prose and hunk context, not just function
Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second round of Copilot review comments
- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
matching the repo convention where the release commit assigns the
version number (see
|
||
|
ae1e5adb66 |
Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659)
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @
|
||
|
4e8c28bbc0 |
Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737)
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
 
|
bf69ea8ccc |
Fix typos (#413)
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com> |
||
 Sean Whalen
|
cd475255c5 | Documentation cleanup | ||
|
2b35b785c6 |
Split and Organise documentation files (#404)
* Set global TOC collapse to false * Split documentation I tried to split the index.md file into logical parts, not changing the contents. I did add a space and change one HTTP URL to HTTPS. --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |