* feat: add grafana loki output
Push aggregate, failure and SMTP TLS reports as JSON log lines to the
Loki push API. Static labels and labels taken from report fields are
configurable, host defaults to the FQDN. The timestamp is either the
ingest time or the report begin time. Supports basic auth, bearer
tokens, X-Scope-OrgID and batching. Push errors are reported as output
errors so messages are not archived.
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
* feat(dashboards): add grafana dashboard for the loki output
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
* Address Copilot review on the Loki output
Responds to Copilot's review of #926:
- Treat any push status other than 200/204 as a failure. Loki returns
block_ingestion_status_code (260 by default) when block_ingestion_until
blocks ingestion, and the logs are discarded (Loki configuration
reference); the old >= 300 check counted that as a successful save, so
the CLI could archive mailbox messages whose reports were never stored.
- Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate
CSV rows carry the record's interval_begin, which the parser builds as
naive UTC wall-clock time (to_utc=True); reading it as local time shifted
entries by the host's UTC offset (+4h on America/New_York).
- Group the Published Policies dashboard dedup by org_name as well as
report_id: report IDs are only unique per reporter (the PostgreSQL key
is UNIQUE (org_name, report_id, ...)).
- Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback
with the automatic step yields overlapping rolling totals, so the sum
counted the same sessions repeatedly.
Adds regression tests for each, including a non-UTC timezone test against
a real sample report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address second Copilot review on the Loki output
Responds to Copilot's second review of #926 (previously-missed findings):
- Strip each [loki] label_fields entry. _str_to_list() only lstrips, so
"report_type , domain" produced "report_type " and LokiClient refused to
start with "Invalid Loki label name 'report_type '".
- Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat
inside the ES-has-docs guard, so after the ES container was recreated a
reseed appended a second copy of the samples to Loki, which does not
deduplicate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address third Copilot review on the Loki output
Responds to Copilot's third review of #926 (previously-missed findings):
- Keep a base URL's query string when appending the push path. The path
was string-concatenated, so https://gw.example.org?route=logs became
...?route=logs/loki/api/v1/push, hitting the root path with a corrupted
query. The path is now set on the parsed httpx.URL.
- Push each report's entries in timestamp order. With timestamp =
report_begin, a normalized multi-day aggregate report yields rows
source-major (every day of one source, then day 1 of the next), all in
one stream. Loki rejects entries older than the newest line minus half
of max_chunk_age (1h by default) with too_far_behind (Loki configuration
docs, "Accept out-of-order writes"), so later sources were partly
rejected. The whole entry list is now sorted before batching.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(loki): do not follow redirects and drop forensic alias
httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError.
Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias.
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
* fix(dashboards): use the existing Loki datasource uid in the dev bootstrap
When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead.
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
---------
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Make the output and mailbox integrations optional extras (#883)
Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).
cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.
The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.
Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Patch psycopg presence in the PostgreSQL CLI wiring tests
CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs
- The optional-integration and Graph error-type import guards now catch
ModuleNotFoundError instead of ImportError, so only a genuinely absent
package reads as a missing extra; a broken-but-present SDK fails
loudly with its real error instead of masquerading as one. The test
blocker raises ModuleNotFoundError accordingly — the exact exception a
missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
an output module (it also serves the msgraph/gmail_api mailbox
sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
INI key the code reads is password (cli.py _parse_config).
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Quote extras specs in copy-paste install commands
From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix upgrade command in the changelog
* Documentation review: accuracy, spelling, grammar, and clarity pass
A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:
- usage.md: documented six missing [general] options (the CSV/JSON
filename options, prettify_json, normalize_timespan_threshold_hours),
the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
the postgresql env-var prefix; corrected the maildir_path default
(None, not INBOX — cli.py Namespace defaults), the mailbox
check_timeout option name, the systemd restart interval (RestartSec
is 5m), and merged the duplicate silent entry; quoted every
copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
the dashboards filename (opensearch_dashboards.ndjson, matching the
file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
(was parsedmarc.service — a copy-paste error that left DavMail
unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
(7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
matching the feature list. CONTRIBUTING: pre-PR checks now include
ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
the PostgreSQL backend the compose stack runs.
Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Docstring review: accuracy, spelling, grammar, and clarity pass
Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:
- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
splunk/loganalytics save functions described aggregate or failure
reports they do not handle; LogAnalyticsException claimed to be an
Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
enumeration omitted smtp_tls; parse_failure_report typed msg_date as
str (it is datetime); strip_attachment_payloads claimed payloads are
replaced with None (the key is deleted); kafkaclient's failure and
SMTP TLS savers claimed per-record slicing while sending the whole
list in one message (docstrings now describe reality — whether
slicing was intended is flagged for follow-up); the postgres savers
claimed to take parse_report_file's return value but receive the
inner report dict; elastic/opensearch save functions' Raises listed
only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
get_ip_address_country), enumeration completeness fixed
(get_ip_address_info's 9 result keys, maps script outputs, TSV
columns), and the stale 44-industry-types count corrected to the
46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
including two that overstated coverage of the elastic/opensearch
address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
files alongside aggregate and failure, mirrored into usage.md's
CLI-help block; --offline's doubled spaces removed (rendered help
unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
security is enabled and auto-configured on first startup since 8.0
(not "8.7 secure mode"), so the settings are verified, not
hand-written.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: import OpenSearch dashboards into the real Global tenant
dashboard-dev-bootstrap.sh sent `securitytenant: global_tenant`. The
OpenSearch security plugin reads that header as a tenant *name*, and
`global_tenant` is a sample custom tenant from the security demo config
-- not the shared Global tenant, whose token is the literal `global`.
The import therefore landed in a separate `global_tenant` tenant (its
own `.kibana_<hash>_globaltenant_1` index) and the dashboards were
invisible to anyone viewing the Global tenant in OpenSearch Dashboards.
Verified against the live dev cluster: `_find` under `securitytenant:
global` returned 26 objects and `.kibana_1` (the Global tenant index the
UI reads) went from 2 to 67 docs after re-importing with the fix. An
empty/omitted header read 0 from Global -- it falls back to the user's
configured default tenant -- so `global` is the only reliable token.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: don't drop report files whose names contain glob metacharacters
The CLI expanded every file argument with glob(), which treats [, ], *,
and ? as pattern syntax. A literal path like
"[Netease DMARC Failure Report] Rent Reminder.eml" -- the bracketed shape
many providers use for emailed failure reports -- was read as a character
class, matched nothing, and was dropped before reaching the parser, with
no error. File arguments that exist on disk are now taken literally; only
non-existent paths are globbed, so shell-style wildcards still expand.
Also adds "postgresql" to _KNOWN_SECTIONS so PARSEDMARC_POSTGRESQL_* env
vars (and their _FILE Docker-secret variants) resolve like every other
backend -- the PostgreSQL backend is new in 10.0.0, so this completes the
unreleased feature rather than fixing a released regression, and is
documented under the PostgreSQL enhancement, not Bug fixes.
Regression tests added for both. Verified end-to-end: all four
samples/failure/*.eml now index (the bracketed Netease report included).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dev: validate dashboards on OpenSearch 3.x and add PostgreSQL to the dev stack
The dev stack ran OpenSearch Dashboards 3.x against OpenSearch 2.x, an
unsupported cross-major pairing. Bump opensearch to :3 (validated on
3.6.0: OSD import into the Global tenant and all dashboards work).
Add a postgresql service plus bootstrap wiring so the new PostgreSQL
backend is exercised alongside the others: wait for PG, seed it via
PARSEDMARC_POSTGRESQL_* env vars on the same parsedmarc run, wipe it on
RESEED, create a Grafana grafana-postgresql-datasource (uid dmarc-pg),
and import dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json.
PG seeding is gated on psycopg being importable: parsedmarc aborts the
whole run (exit 1, nothing written to any backend) when a configured
output backend can't initialize, so wiring in PG without the optional
extra would silently zero ES/OS/Splunk too. When psycopg is absent the
script warns and skips PG, leaving the other backends seeded.
Also fix the Grafana admin password env: the container was given
GRAFANA_PASSWORD, which Grafana ignores -- it reads
GF_SECURITY_ADMIN_PASSWORD. Defaults to admin to match the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: list PostgreSQL on the premade-dashboards features bullet
PostgreSQL ships a premade Grafana dashboard
(dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json), so it belongs
on the "for use with premade dashboards" bullet alongside Elasticsearch,
OpenSearch, and Splunk rather than on the plain-output-destinations line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: clear stale org_email mapping conflict in the OpenSearch dashboards
The aggregate index pattern in dashboards/opensearch/opensearch_dashboards.ndjson
shipped a cached field-list snapshot where org_email was a text/object
conflict, plus leftover org_email.#text and org_email.#text.keyword
subfields. Those came from a cluster that had indexed a langAttrString
email dict ({"#text": ..., "@lang": ...}) before the parser unwrapped it.
org_email is mapped as Text() and parse_aggregate_report_xml now unwraps a
dict email to a plain string, so current data is consistently text -- a
clean cluster's _field_caps reports no conflict. Cleared the frozen
conflict and the two artifact subfields, leaving org_email (text) and
org_email.keyword, matching the live mapping.
Verified: re-importing the corrected ndjson yields an index pattern with
org_email as a plain text field and zero conflicts; only the aggregate
index-pattern line changed, all other saved objects byte-identical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dev: seed the RFC 9990 (dmarc-2.0) aggregate samples
samples/aggregate/rfc9990-sample.xml and rfc9990-example.net!...xml were
not in the bootstrap's SAMPLE_FILES, so the dev stack only ever indexed
RFC 7489 reports and the new DMARCbis fields (np, testing,
discovery_method, generator, xml_namespace) never appeared in the
OpenSearch/Kibana indices or were available to the dashboards.
Added both samples (one declares the urn:ietf:params:xml:ns:dmarc-2.0
namespace, the other is namespaceless RFC 9990-shaped, covering both
detection paths). Verified the seeded data now carries np/testing/
discovery_method/generator and xml_namespace=urn:ietf:params:xml:ns:dmarc-2.0;
OpenSearch Dashboards surfaces them on an index-pattern field-list refresh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dev: auto-resolve (or create) a venv for the seed and ensure psycopg
The seed previously required parsedmarc to be pre-installed and only
warned-and-skipped PostgreSQL when psycopg was missing. Resolve the seed
environment by precedence instead:
1. explicit PARSEDMARC_BIN -> used as-is, nothing installed
2. active $VIRTUAL_ENV
3. existing repo venv/ or .venv/
4. otherwise create $REPO_ROOT/venv
For cases 2-4, run `pip install -e .[postgresql]` only when the CLI or
psycopg is missing, so the dev stack can populate Postgres out of the box
without a manual install step. The explicit-PARSEDMARC_BIN path is left
untouched (and the psycopg seed guard still warns/skips if that env lacks
the extra).
Verified: a RESEED run resolves the active venv, seeds ES/OS/Splunk/PG
including the RFC 9990 fields, with no output-client errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: note DMARC RFC support in the features list
The features list only mentioned "draft and 1.0" aggregate reports. Spell
out the standards parsedmarc parses: RFC 7489 (legacy DMARC) and the final
DMARC standard RFC 9989 with RFC 9990 aggregate reports, RFC 6591 and
RFC 9991 failure reports, and RFC 8460 SMTP TLS reports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: align Python compatibility table pipes (MD060)
The emoji cells were padded for display width, leaving the source pipes
misaligned by character count and tripping markdownlint MD060. Re-pad so
every row's pipes line up by codepoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: list all optional output destinations; fix table emoji alignment
Expand the features list to cover every output sink: Elasticsearch,
OpenSearch, Splunk, and PostgreSQL (premade dashboards), plus Kafka,
Amazon S3, Azure Log Analytics (Microsoft Sentinel), Graylog (GELF),
syslog, and HTTP webhooks.
Also re-pad the Python compatibility table using display width (the
status emoji render two columns wide), which is what markdownlint MD060
measures — the previous codepoint-based padding still tripped the rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: separate PostgreSQL from the premade-dashboards clause
PostgreSQL is a storage target without bundled premade dashboards, so it
shouldn't sit inside the "for use with premade dashboards" phrase next to
Elasticsearch/OpenSearch/Splunk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: move PostgreSQL to the non-dashboard outputs line
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: use compact markdown tables
Switch the markdown tables (Python compatibility, env-var section mapping)
to compact single-space format. It reads cleanly in a text editor and
sidesteps the column-alignment churn that emoji/variable-width content
caused with padded tables (markdownlint MD060). The reStructuredText grid
table in dmarc.md is left as-is — it relies on multi-line cells markdown
can't express.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The forensic→failure rename (#659) left a few loose ends and one deliberate
hold-back. This closes them.
Leftover rename misses (broken paths / stale canonical names):
- CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/*
- dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml
→ dmarc_failure_dashboard.xml (the file was already renamed; the import path
and view name were not)
- docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example
- samples/parsedmarc.ini: save_forensic → save_failure
- pyproject.toml, README.md: canonical "failure" naming
(ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.)
Archive subfolder rename + on-startup migration:
- New failure reports now archive to <archive>/Failure (was <archive>/Forensic).
- _migrate_forensic_archive_folder() runs once on startup (best-effort):
renames Forensic→Failure when no Failure folder exists yet, merges the two
when both exist, no-ops when there's no legacy folder, and logs-and-skips a
mailbox it can't reorganize (warn, don't crash). This consolidates pre- and
post-rename failure reports into one folder, replacing the previously
documented decision to keep the folder named Forensic to avoid a split
archive. Uses the folder-management API (folder_exists / rename_folder /
merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0.
Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana):
- Forensic panel titles + the datasource label → Failure; the fo-column display
label and its linked byName field-override matcher both → "Failure Policy"
(changed together so the column-width override keeps matching).
- dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both
pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana
dashboards); RESEED wipe loop now also clears dmarc_failure* indices.
- Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an
orphan export accidentally committed in #736 and referenced by nothing.
Tests (tests/test_init.py):
- TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via
mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full
get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds).
- TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir
can't reproduce — a backend that raises mid-operation must warn, not crash.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ivan The Geek
Sean Whalen
Kili
Copilotandseanthegeek
Anael Mobilia
daminoux
Christian Clauss
Andras
Pierce
Sean Whalen