Google's content-hub parsers tag dropped logs (drop { tag => "TAG_..." })
so they surface correctly in the unparsed-log views, rather than bare drop{}.
Use TAG_MALFORMED_ENCODING for the two JSON-extraction/parse failures (matches
content-hub) and TAG_UNSUPPORTED for valid JSON that matches no parsedmarc
report shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two corrections confirmed against Google's official content-hub parsers
(content/parsers/third_party/community/*/cbn):
1. Numbers as numbers. count, source_asn, successful_session_count and
failed_session_count were being stored in additional.fields as string_value.
Store them as number_value instead (build string -> convert to uinteger ->
rename to number_value, the content-hub idiom), so SecOps can range-query and
sort them, per parsedmarc's "store numbers as numbers" rule. Booleans stay
string_value (content-hub never uses bool_value) and are still converted in
step 1b for the == "true"/"false" comparisons.
2. Conditional guards. Replaced bare `if [field] {` with `if [field] != "" {`
(76 guards + the detection cascade + policy_override). After 1a initializes
every tested field to "", a bare `if` is true for an empty field (Logstash/CBN
semantics), which would misfire detection and emit empty labels. content-hub
uses `!= ""` ~111x vs 2 bare (both flags); parser flags (no_json_payload,
not_json, *_nan) correctly stay bare.
Verified: braces balance, no stray bare field-guards, all if-tested fields
initialized, all four numeric fields emit number_value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add github.com/chronicle/content-hub (Google's official third-party SecOps
parser repo) to the README references and re-anchor the in-code citations to
it. Its current CBN parsers (e.g. CLOUDFLARE_PAGESHIELD, Copyright 2025 Google
SecOps) confirm both fixes this parser makes: initialize every field before the
json{} filter, and convert JSON booleans/numbers to strings before comparison.
Replaces the dated "How to parse JSON data" citation with the authoritative,
actively-maintained source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a short, skippable callout explaining what a parser / configuration-based
normalizer (CBN) is and how it fits the SecOps ingest flow (log type → parser →
UDM event), so the README serves newcomers without slowing experienced users.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the remaining official Google docs the parser is built on (parser tips
& troubleshooting, manage parsers, UDM search, Bindplane install) and a
clearly-separated "Additional sources and tooling" section for the community
resources that drove the JSON type-handling and field-init fixes
(thatsiemguy's Parsing 101, the Corelight production parser, chronicle/cbn-tool).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xml_schema is aggregate-only (failure/SMTP TLS rows don't carry it) and a
distinctive, non-generic field name, addressing the concern that "domain"
could be confused with other logs. parsedmarc defaults xml_schema to "draft"
when the report omits <version> (parsedmarc/__init__.py:832), so it survives a
missing version element -- unlike a field with no default.
It is also a native JSON string straight out of the json{} filter, so unlike
dmarc_aligned it needs no convert step to be testable, keeping detection
independent of the type-conversion in step 1b. xml_schema is added to the
pre-json init block (required for any if-tested field); domain stays
initialized since it is still mapped to target.hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two CBN behaviors, confirmed against Google's own "How to parse JSON data"
guide (statedump shows JSON true/199 retaining boolean/integer type) and the
published Corelight production parser:
1. The json{} filter preserves the original JSON type, so parsedmarc's boolean
*_aligned / testing / normalized_timespan and numeric count / *_session_count
/ source_asn would never match string comparisons. Add a mutate{convert} step
turning them into strings before any == "true"/"false" test or %{...} use.
2. CBN raises _failed_parsing_ when an `if [field]` references a field absent
from the log, and most detection/mapping fields are absent in 2 of the 3
report shapes (or null within one). Initialize every conditionally-checked
field to "" before the json{} filter.
Without these, DMARC-fail records would not be categorized AUTH_VIOLATION and
aggregate/TLS reports could fail parsing outright. README caveat and PR
validation steps updated accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adkim is the published policy's DKIM alignment mode (defaulted to "r" by
parsedmarc), an obscure thing to key detection on. Switch the aggregate
detector to "domain" -- the reported From-domain, a required element present
and non-empty in every aggregate record (2388/2388 sample rows) and unique to
aggregate (failure uses reported_domain, SMTP TLS uses policy_domain).
header_from is unsuitable: it can be empty when a record carries no
identifiers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SecOps-side custom parser (CBN) that maps parsedmarc's [syslog] JSON
events to the Unified Data Model. No library changes: parsedmarc already
emits structured JSON, so the DMARC->UDM mapping lives in the parser and a
downstream UDM schema change is a parser edit, not a parsedmarc release.
Covers all three report types:
- aggregate -> EMAIL_TRANSACTION
- failure -> EMAIL_TRANSACTION
- smtp_tls -> GENERIC_EVENT (noun from policy_domain, present on every row)
Built strictly against the official UDM and parser-syntax docs (cited
inline). Sets metadata.event_timestamp from the report window via date{},
maps disposition / auth-failure to security_result with valid action and
category enums (AUTH_VIOLATION on DMARC fail), uses real network.email
field names, and strips syslog framing before JSON parsing. Ships real
sample events generated from the project's sample reports for validation.
Not yet validated against a live SecOps tenant; caveats are documented in
the README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make Microsoft Graph connection activity observable
parsedmarc only configured its own logger, so all Graph connection
activity was silently dropped even with --debug: the mailbox layer
logs under mailsuite.mailbox.graph, token acquisition under
azure.identity (including the AADSTS error codes that distinguish a
local config problem from an Exchange Online / Entra ID one), and
HTTP traffic under httpx/msgraph — none of which had a handler or
level set. _main() also logged nothing around the MSGraphConnection
call, so a hang left no trace at all.
Three changes, all parsedmarc-side (no mailsuite changes needed):
- Log a redacted connection summary at INFO before connecting (auth
method, tenant ID, client ID, mailbox, Graph URL) plus a --debug
detail line with certificate path, token-file path, and set/not-set
flags for secrets. Secret values are never logged; a regression
test asserts they don't appear in captured output.
- Log a timing line after the connection object is initialized.
- Propagate parsedmarc's --verbose/--debug level and handlers to the
dependency loggers (mailsuite, azure, msgraph, httpx, httpcore) via
_configure_dependency_logging(), synced to exactly the parsedmarc
logger's handlers so SIGHUP log-file swaps neither duplicate output
nor write to closed handlers. At the default level dependency
loggers sit at WARNING, so their warnings keep surfacing (formatted)
without new noise.
All four new tests fail on the unfixed code (verified by stashing the
cli.py change).
Fixes https://github.com/domainaware/parsedmarc/issues/814.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Disable propagation on dependency loggers; document kiota's absence
Set propagate=False on the dependency loggers when syncing handlers, so
a stray logging.basicConfig() anywhere in the process cannot
double-print every dependency record through the root logger — the
function already owns these loggers' handler lists, and this makes that
ownership complete. Asserted alongside the existing level/handler checks.
kiota_http and its sibling packages were considered for
_DEPENDENCY_LOGGERS but verified to not use Python logging at all
(their observability is OpenTelemetry tracing), so a comment now
records why they are absent rather than leaving the omission to be
"fixed" later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* Stop system GeoIP files from shadowing the bundled IPinfo database
_get_ip_database_path() searched well-known system paths (including
/usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before
the database parsedmarc manages, so on any host with a distro GeoIP
package installed every lookup silently used a country-only — and
often years-old — database instead of the bundled IPinfo Lite one.
That disabled ASN enrichment entirely (asn/as_name/as_domain were None
for every IP) and with it the ASN-fallback path into the reverse-DNS
map, with no signal beyond a generic "IP database is more than a
month old" warning. Verified live on a Fedora host whose distro
GeoLite2-Country.mmdb dated to December 2019.
New precedence: explicit ip_db_path -> _IP_DB_PATH selected by
load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system
paths as a true last resort (only consulted when the bundled data
file is missing). The selected file is logged at debug level so a
--debug run shows which database answered.
The automatic system-path pickup was documented behavior, so
installation.md now tells MaxMind GeoLite2 users to set ip_db_path
explicitly, with a migration note.
Both new regression tests reproduce the shadowing portably via a decoy
CWD GeoLite2-Country.mmdb (the fallback list includes relative names),
and fail on the unfixed code (verified by stashing the source change).
Fixes https://github.com/domainaware/parsedmarc/issues/810.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: accurate fallback comment, dedup selection log, cover fallback tiers
- Correct the nothing-found-anywhere comment: the os.stat() age check
raises FileNotFoundError before the caller's open_database() would.
- Log "Using IP database at ..." only when the selected path changes
instead of on every uncached IP lookup, so --debug runs over large
batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in
the test fixture for order-independence.
- Cover the previously untested branches of _get_ip_database_path:
system-path fallback when the bundled database is missing, the
FileNotFoundError when nothing exists anywhere, the stale-database
warning, and the log-once-per-path behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks
arrival_date_utc is a UTC wall-clock string (generated in
parse_failure_report via an aware-UTC strftime), but elastic.py,
opensearch.py, and splunk.py parsed it back into a naive datetime and
called .timestamp(), which per the Python docs interprets naive values
as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp).
On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date
field, used in the failure-report dedup match query, and sent as the
Splunk HEC event time was therefore off by the host's UTC offset
(verified -3600 s under TZ=Europe/Warsaw in January).
Add an assume_utc keyword to human_timestamp_to_datetime() /
human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive
parses, and use it at the three arrival_date_utc call sites. Aware
inputs (explicit offsets) are unaffected; all other callers keep the
existing local-time semantics, whose round-trip with timestamp_to_human
is self-consistent on a single host (the broader local-time output
question is tracked separately in issue #811 bug 2).
The three new sink regression tests fail on the unfixed code
(verified by stashing the source changes) and force TZ=Europe/Warsaw
via time.tzset() so they catch the skew even on UTC CI runners.
Fixes half of https://github.com/domainaware/parsedmarc/issues/811.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring
Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from
the four timestamp regression tests into a shared tests/tzutil.py
force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s
docstring, which said the return type was float while the function
returns int.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* draft fix MS Graph
* Add ClientAssertion auth method support for MS Graph in cli.py
MSGraphConnection/AuthMethod (via mailsuite) already supported
ClientAssertion, but cli.py never parsed a client_assertion config
value or passed it through, so it was unusable from the CLI. Wire
config_msgraph.client_assertion through _parse_config and the
MSGraphConnection call, and document the auth method (including its
short-lived-JWT caveat vs. Certificate/ClientSecret for watch mode).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
requires-python stays ">=3.10" (already correct and matching the CI matrix
of 3.10-3.14); add the per-version Programming Language :: Python :: 3.10-3.14
classifiers and "3 :: Only" so the PyPI page lists supported versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Explain why a report is invalid instead of "Not a valid report"
The parser catches broadly so one malformed report can't crash a batch,
but every failure surfaced as the generic ParserError("Not a valid
report"), telling operators nothing about the cause.
parse_report_file() now keeps each format parser's specific error as it
tries aggregate XML -> SMTP TLS JSON -> report email, and when all three
reject the input it content-sniffs the leading byte to surface the single
relevant reason (e.g. "Invalid aggregate report: Missing field:
'org_name'", or "Not a recognized report format (...)"). The CLI already
logs str(error), so this reaches the user with no cli.py change.
Every parser catch site also re-raises with `raise ... from <original>`,
preserving the underlying ExpatError / JSONDecodeError / KeyError /
archive errors on __cause__ for library callers and tracebacks. The same
exception *types* are still raised.
Finally, the catch-all "unexpected error" branches append
`(raised at <file>:<line>)` from the deepest traceback frame, but only
when the parsedmarc logger is at DEBUG level (e.g. the CLI's --debug);
normal-level output is unchanged.
Bumps the in-progress version to 10.2.0 and documents all three in the
CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover the failure-report path in parse_report_file error tests
The reason-surfacing tests covered the aggregate and SMTP TLS branches but
not failure reports, which reach parse_report_file only via the email
path. Add a malformed multipart/report failure email (missing the required
Source-IP) and assert the message names the failure format and the missing
field rather than collapsing to "Not a valid report".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover the new error-reporting lines; drop one unreachable catch
Bring the lines added by this PR to full test coverage:
- Test _exc_origin() with a never-raised exception (no __traceback__) so the
"no frames" guard is exercised.
- Test _parse_smtp_tls_failure_details() with a non-dict, which raises
TypeError (not KeyError) and exercises the generic catch-all.
- Test parse_report_email() with an unparseable Date header, which trips the
initial mail-parse catch-all and becomes a ParserError.
Two dead lines are removed rather than hidden, per the project's "delete
unreachable branches, no # pragma: no cover" rule:
- _looks_like_email() looped with a `continue` for blank lines, but every
caller passes lstrip()-ed text, so the first line is never blank. Simplified
to inspect the first line directly.
- parse_report_email()'s `except Exception` after `except InvalidFailureReport`
was unreachable: parse_failure_report wraps its entire body and provably
raises only InvalidFailureReport, which the preceding handler already catches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix IndexError when backfilling envelope_from from SPF results
_parse_report_record() backfills a missing/empty envelope_from from the
last SPF auth result's domain. The "envelope_from is None" branch gated on
the raw auth_results["spf"] list but indexed the filtered
new_record["auth_results"]["spf"] list, which only holds results that have
a domain. A reporter sending an SPF result with no domain made the filtered
list empty while the raw list was non-empty, so [-1] raised IndexError and
the whole record failed to parse.
The two near-identical envelope_from backfill branches (missing identifier
vs. empty identifier) drifted apart -- only one was updated when the
filtered new_record list was introduced -- which is what let them disagree
on which list to read. Merge them into a single path, keyed on
dict.get("envelope_from") is None, that gates and indexes the same raw list
with the "domain" membership guard the missing-identifier branch already
used.
Regression test: envelope_from=None with an SPF result carrying no domain
now parses to envelope_from=None instead of raising. This is the bug that
motivated the surrounding error-reporting work in this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Trim comment
* Cover the touched error branches; drop a dead UnicodeDecodeError catch
Codecov flagged the pre-existing error branches this PR touched (adding
`from e` / `_exc_origin`) as changed-and-uncovered. Most are real
malformed-report paths, so add honest tests that drive them with realistic
inputs:
- parse_smtp_tls_report_json: nested missing key (date-range without
start-datetime) -> InvalidSMTPTLSReport chaining a KeyError.
- parse_aggregate_report_xml: non-structured report_metadata -> the
AttributeError branch ("Report missing required section").
- parse_report_email: valid legacy text/plain failure report (success path),
a text report missing its fields, a base64 attachment of malformed
aggregate XML, and one of invalid SMTP TLS JSON.
- parse_report_file: gzipped junk -> the str branch of the content sniff.
The `except UnicodeDecodeError` in extract_report is removed as dead code
(no `# pragma: no cover`, per the repo rule): str-mode streams are already
rejected by explicit isinstance checks, and every decode() uses
errors="ignore", so it can never fire. str-mode still raises ParserError.
Also rename the two new failure-report tests from "Forensic" to "Failure"
and add an AGENTS.md rule: RUF reports are "failure reports"; "forensic" is
reserved for the literal backward-compat alias identifiers only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix pyright errors in the new error-branch tests
CI runs pyright over the whole repo (tests included); these slipped through
because the local check only covered parsedmarc/__init__.py:
- _parse_smtp_tls_failure_details("not a dict") is a deliberate wrong-type
test -> targeted `# pyright: ignore[reportArgumentType]`.
- result["report"]["source"]["ip_address"] on a ParsedReport TypedDict ->
cast(FailureReport, result["report"]) first, matching existing tests.
This is what failed lint-docs-build (and, since `test` needs it, skipped the
Codecov upload) on the prior commits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The module-level try/except import is environment-dependent: with
psycopg installed the ImportError fallback never runs, and without it
(CI's test job) the successful-import arm never completes — so Codecov
flags one side or the other no matter where coverage is measured (it
flagged the import line on master right after #798 merged).
Exercise both arms explicitly: execute the module's source into a
fresh, throwaway module object (importlib.util.module_from_spec +
exec_module) under a patched sys.modules — a None entry forces
ImportError, fake module entries force the success path — and assert
on the psycopg / psycopg_json bindings each arm produces. The
throwaway-module approach (rather than importlib.reload) leaves the
canonical parsedmarc.postgres untouched, so the identity of
PostgreSQLError / AlreadySaved held by the rest of the test module is
preserved.
Verified covered in both environments: with the venv's real psycopg,
and with psycopg hidden via a PYTHONPATH shim to simulate CI; the
import block reports no missing lines either way.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Make the whole codebase pass pyright cleanly and enforce it in CI
Fix all 102 pyright (1.1.410, standard mode) errors across the library,
tests, and maps scripts, then pin and enforce the zero-errors bar:
- postgres.py: make the optional psycopg import TYPE_CHECKING-aware so
the module is properly typed while keeping the runtime install-hint
fallback; import psycopg.types.json explicitly as psycopg_json (the
old psycopg_types.json attribute access only worked because psycopg
imports the submodule eagerly); have _connect()/_ensure_connected()
return the live connection so save methods use a non-Optional local;
type the DDL list as list[LiteralString] to match psycopg's execute()
overloads.
- kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error
fallback statically via TYPE_CHECKING (kafka-python 3.0 removed
NoBrokersAvailable), which also fixes _BootstrapError's import
resolution in tests.
- syslog.py: go through getattr/setattr for SysLogHandler.socket
(absent from typeshed); type the save_* methods with the report
TypedDicts (single or list, matching cli.py call sites — gelf.py gets
the same signatures); raise ValueError when retry_attempts < 1
instead of falling through and registering a None handler (bug fix,
with a regression test and a CHANGELOG entry).
- elastic.py / opensearch.py: human_result params are Optional[str].
- maps scripts: sort_csv declared a return type but never returned
(now -> None); seen_sort_field_values was possibly unbound;
convert_to_utf8's src_encoding is Optional[str].
- tests: cast sample-report dict helpers to their TypedDicts; mark
deliberate wrong-type calls with targeted pyright ignores; add
narrowing asserts for Optional results; access the mocked
KafkaProducer through a cast helper; match the mailsuite
fetch_message base signature (**kwargs); patch the renamed
parsedmarc.postgres.psycopg_json in test_postgres's setUpModule.
Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc,
tests, docs; standard mode), pyright==1.1.410 pinned in the [build]
extra (pinned exactly so a new pyright release can't break CI without a
code change), and a "Check types" step in the lint CI job — which now
also runs ruff format --check and installs the [postgresql] extra so
the optional psycopg import resolves. Documented in AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Set session headers via update() instead of replacing the dict
requests 2.34 ships inline type annotations, and Session.headers is a
CaseInsensitiveDict[str] — assigning a plain dict fails pyright there
(the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid
it). headers.update() is correctly typed against both versions, and is
the documented requests idiom: it overrides User-Agent and the
client-specific headers while keeping the session's defaults
(Accept-Encoding, Connection) instead of wiping them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Declare backward-compatible method aliases inside class bodies
Assigning the legacy save_forensic_* aliases onto the classes after the
class body (KafkaClient.save_forensic_reports_to_kafka = ...) is invisible
to static type checkers, so Pylance/Pyright flagged every assignment and
every use with reportAttributeAccessIssue. Declaring the alias inside the
class body is statically visible — the IDE errors disappear and the
aliases get autocomplete and proper typing. Runtime behavior is identical
(same function object bound as a method), guarded by the existing
assertIs alias tests, whose type-ignore comments are now unnecessary.
Also add a pyright ignore on the NoBrokersAvailable import in
kafkaclient.py: the import is guarded by try/except ImportError for
kafka-python 2.x, but Pyright resolves against the installed 3.x where
the name no longer exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump version to 10.1.0
10.0.4 is tagged and released; CHANGELOG.md already documents the
in-progress 10.1.0 section that this release will ship.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
kafka-python-ng is archived and vulnerable to CVE-2026-10142 and
CVE-2026-10143, both fixed in upstream kafka-python 2.3.2.
kafka-python 3.0 removed the NoBrokersAvailable exception (a failed
producer bootstrap now raises KafkaTimeoutError), so kafkaclient.py
imports whichever the installed version provides via a compat shim,
keeping the >=2.3.2 range honest for both 2.x and 3.x. Verified against
kafka-python 3.0.0 (full test suite) and 2.3.2 (import shim resolution).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI
Previously SIGTERM (systemctl stop, docker stop, Kubernetes pod termination)
killed parsedmarc mid-batch, tearing output writes and silently dropping
buffered Kafka records. Shutdown is now cooperative:
- SIGTERM/SIGINT set a flag that is polled at safe boundaries. The one-shot
CLI checks it between batches; watch mode passes it as `config_reloading` so
the mailbox backend -- including the IMAP IDLE loop -- returns once the
current batch is fully processed. Either way the in-flight batch and its
output writes finish before the process exits 0.
- Ctrl-C is a double-tap: the first press is graceful, the second
short-circuits to os._exit(130).
- Output clients are now closed on every exit path (atexit plus a trailing
close in _main), fixing a long-standing leak where one-shot runs and
graceful shutdowns never flushed Kafka / closed Elasticsearch / S3 / etc.
Docs: the example systemd unit gains KillSignal=SIGTERM and TimeoutStopSec=60
(keep it above mailbox_check_timeout). Tests cover watch shutdown, the one-shot
between-batch stop, the SIGINT double-tap, and the output-client-close leak.
* test: cover the one-shot mbox-loop shutdown break
Extend the one-shot SIGTERM test to also pass an .mbox path so a single
run exercises both shutdown checkpoints: the file-batch loop break and the
subsequent mbox loop break (which Codecov flagged as the only uncovered
lines on PR #794). is_mbox is keyed by suffix and get_dmarc_reports_from_mbox
is asserted not called, since the mbox loop breaks before reaching it.
* test: narrow signal.getsignal() return before invoking in SIGINT test
signal.getsignal() is typed Callable | int | Handlers | None; calling it
directly fails pyright's callable check. Assert callable() first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Build multi-arch Docker images with PostgreSQL support
The prebuilt image now installs the `[postgresql]` extra, so the optional
PostgreSQL output backend (psycopg) works out of the box in the container
without a separate `pip install` (#792). The wheel path is resolved into a
variable before appending the extra so the shell doesn't treat
`*.whl[postgresql]` as a bracket glob.
The build workflow now sets up QEMU + Buildx and builds a multi-arch
manifest for `linux/amd64` and `linux/arm64`, so the image runs natively on
64-bit ARM hosts such as a Raspberry Pi (#789). Every compiled dependency
(psycopg[binary], lxml, maxminddb, cryptography) ships prebuilt aarch64
manylinux wheels, so the arm64 build adds no source-compilation step.
A `pull_request` trigger (scoped to the build inputs) and `workflow_dispatch`
are added so the multi-arch build can be validated on PRs and rebuilt on
demand; pushes are still gated on the release event, so neither pushes images.
Closes#789Closes#792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bump version to 10.0.4 to publish the new images
The docker workflow only pushes to the registry on a `release` event, so
shipping the multi-arch + PostgreSQL-enabled image requires cutting a
release. 10.0.3 is already tagged, so bump to 10.0.4 and document the
Docker changes in the changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Don't run the docker build on pull requests
The pull_request trigger (added to validate the multi-arch build) re-ran the
full ~10-minute amd64+arm64 build on every commit pushed to a docker-touching
PR, because the pull_request `paths` filter matches against the PR's entire
diff, not just the newest commit. That is wasteful once the build has been
validated.
Drop the pull_request trigger and rely on workflow_dispatch for on-demand
validation (plus the existing master-push and release triggers). Also gate the
registry login on the release event so that no non-release run authenticates
to ghcr at all — a build can only ever be pushed from a published release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 10.0.3 Reply-To header flattening (elastic.py / opensearch.py line 711)
has two branches: display-name present ("Name <addr>") and absent (bare
address). The existing test only exercised the former, leaving the
empty-display-name branch uncovered — the two lines Codecov flagged on the
10.0.3 patch. Add a failure report whose Reply-To has no display name and
assert sample.headers["reply-to"] flattens to the bare address.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.
This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump mailsuite to >=2.2.1; release 10.0.2
mailsuite 2.2.1 raises the transitive mail-parser floor to >=4.2.1, which
stops mail-parser from returning a phantom ('', '') entry for absent address
headers (verified against samples/failure/* with mail-parser 4.2.1: cc/bcc
now parse to [] instead of [{address: ""}]). parsedmarc reads the mail-parser
object directly via its own parse_email(), so this previously caused an empty
{address: ""} Cc/Bcc entry to be indexed for every failure-report sample in
Elasticsearch/OpenSearch and emitted in JSON/S3/Kafka output.
The Reply-To-always-empty behavior in parsedmarc's own parse_email() (a
hyphen-vs-underscore key mismatch, not an upstream issue) and the failure
dashboards are out of scope here and tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: note CVE-2023-27043 hardening from mail-parser 4.2.1 in 10.0.2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: import OpenSearch dashboards into the real Global tenant
dashboard-dev-bootstrap.sh sent `securitytenant: global_tenant`. The
OpenSearch security plugin reads that header as a tenant *name*, and
`global_tenant` is a sample custom tenant from the security demo config
-- not the shared Global tenant, whose token is the literal `global`.
The import therefore landed in a separate `global_tenant` tenant (its
own `.kibana_<hash>_globaltenant_1` index) and the dashboards were
invisible to anyone viewing the Global tenant in OpenSearch Dashboards.
Verified against the live dev cluster: `_find` under `securitytenant:
global` returned 26 objects and `.kibana_1` (the Global tenant index the
UI reads) went from 2 to 67 docs after re-importing with the fix. An
empty/omitted header read 0 from Global -- it falls back to the user's
configured default tenant -- so `global` is the only reliable token.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: don't drop report files whose names contain glob metacharacters
The CLI expanded every file argument with glob(), which treats [, ], *,
and ? as pattern syntax. A literal path like
"[Netease DMARC Failure Report] Rent Reminder.eml" -- the bracketed shape
many providers use for emailed failure reports -- was read as a character
class, matched nothing, and was dropped before reaching the parser, with
no error. File arguments that exist on disk are now taken literally; only
non-existent paths are globbed, so shell-style wildcards still expand.
Also adds "postgresql" to _KNOWN_SECTIONS so PARSEDMARC_POSTGRESQL_* env
vars (and their _FILE Docker-secret variants) resolve like every other
backend -- the PostgreSQL backend is new in 10.0.0, so this completes the
unreleased feature rather than fixing a released regression, and is
documented under the PostgreSQL enhancement, not Bug fixes.
Regression tests added for both. Verified end-to-end: all four
samples/failure/*.eml now index (the bracketed Netease report included).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dev: validate dashboards on OpenSearch 3.x and add PostgreSQL to the dev stack
The dev stack ran OpenSearch Dashboards 3.x against OpenSearch 2.x, an
unsupported cross-major pairing. Bump opensearch to :3 (validated on
3.6.0: OSD import into the Global tenant and all dashboards work).
Add a postgresql service plus bootstrap wiring so the new PostgreSQL
backend is exercised alongside the others: wait for PG, seed it via
PARSEDMARC_POSTGRESQL_* env vars on the same parsedmarc run, wipe it on
RESEED, create a Grafana grafana-postgresql-datasource (uid dmarc-pg),
and import dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json.
PG seeding is gated on psycopg being importable: parsedmarc aborts the
whole run (exit 1, nothing written to any backend) when a configured
output backend can't initialize, so wiring in PG without the optional
extra would silently zero ES/OS/Splunk too. When psycopg is absent the
script warns and skips PG, leaving the other backends seeded.
Also fix the Grafana admin password env: the container was given
GRAFANA_PASSWORD, which Grafana ignores -- it reads
GF_SECURITY_ADMIN_PASSWORD. Defaults to admin to match the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: list PostgreSQL on the premade-dashboards features bullet
PostgreSQL ships a premade Grafana dashboard
(dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json), so it belongs
on the "for use with premade dashboards" bullet alongside Elasticsearch,
OpenSearch, and Splunk rather than on the plain-output-destinations line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: clear stale org_email mapping conflict in the OpenSearch dashboards
The aggregate index pattern in dashboards/opensearch/opensearch_dashboards.ndjson
shipped a cached field-list snapshot where org_email was a text/object
conflict, plus leftover org_email.#text and org_email.#text.keyword
subfields. Those came from a cluster that had indexed a langAttrString
email dict ({"#text": ..., "@lang": ...}) before the parser unwrapped it.
org_email is mapped as Text() and parse_aggregate_report_xml now unwraps a
dict email to a plain string, so current data is consistently text -- a
clean cluster's _field_caps reports no conflict. Cleared the frozen
conflict and the two artifact subfields, leaving org_email (text) and
org_email.keyword, matching the live mapping.
Verified: re-importing the corrected ndjson yields an index pattern with
org_email as a plain text field and zero conflicts; only the aggregate
index-pattern line changed, all other saved objects byte-identical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dev: seed the RFC 9990 (dmarc-2.0) aggregate samples
samples/aggregate/rfc9990-sample.xml and rfc9990-example.net!...xml were
not in the bootstrap's SAMPLE_FILES, so the dev stack only ever indexed
RFC 7489 reports and the new DMARCbis fields (np, testing,
discovery_method, generator, xml_namespace) never appeared in the
OpenSearch/Kibana indices or were available to the dashboards.
Added both samples (one declares the urn:ietf:params:xml:ns:dmarc-2.0
namespace, the other is namespaceless RFC 9990-shaped, covering both
detection paths). Verified the seeded data now carries np/testing/
discovery_method/generator and xml_namespace=urn:ietf:params:xml:ns:dmarc-2.0;
OpenSearch Dashboards surfaces them on an index-pattern field-list refresh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dev: auto-resolve (or create) a venv for the seed and ensure psycopg
The seed previously required parsedmarc to be pre-installed and only
warned-and-skipped PostgreSQL when psycopg was missing. Resolve the seed
environment by precedence instead:
1. explicit PARSEDMARC_BIN -> used as-is, nothing installed
2. active $VIRTUAL_ENV
3. existing repo venv/ or .venv/
4. otherwise create $REPO_ROOT/venv
For cases 2-4, run `pip install -e .[postgresql]` only when the CLI or
psycopg is missing, so the dev stack can populate Postgres out of the box
without a manual install step. The explicit-PARSEDMARC_BIN path is left
untouched (and the psycopg seed guard still warns/skips if that env lacks
the extra).
Verified: a RESEED run resolves the active venv, seeds ES/OS/Splunk/PG
including the RFC 9990 fields, with no output-client errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Ignore data/export trees via cSpell.ignorePaths: parsedmarc/resources/**
(maps tooling holds thousands of intentional foreign-language classifier
keywords + bundled data), plus samples/** and dashboards/** (report
samples and dashboard exports). These are data, not whitelist vocabulary,
so excluding them keeps the editor quiet without bloating the word list.
- Add the remaining genuine false-positives across code, docs, CI
workflows, and editor config to cSpell.words (technical terms, library
names, SQL/identifier tokens, brand/operator and multilingual examples
from AGENTS.md, plus charliermarsh/junitxml/mktemp/pipefail/seanthegeek).
- Fix two genuine typos found while triaging rather than whitelisting
them: "maidir" -> "maildir" and "connexion" -> "connection".
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PostgreSQL dashboard shipped at the repo-root grafana/ by oversight;
every other dashboard source lives under dashboards/ (opensearch/,
grafana/, splunk/). Move it next to the existing Grafana dashboard, list
it in dashboards/README.md, and fix the CHANGELOG path reference.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: note DMARC RFC support in the features list
The features list only mentioned "draft and 1.0" aggregate reports. Spell
out the standards parsedmarc parses: RFC 7489 (legacy DMARC) and the final
DMARC standard RFC 9989 with RFC 9990 aggregate reports, RFC 6591 and
RFC 9991 failure reports, and RFC 8460 SMTP TLS reports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: align Python compatibility table pipes (MD060)
The emoji cells were padded for display width, leaving the source pipes
misaligned by character count and tripping markdownlint MD060. Re-pad so
every row's pipes line up by codepoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: list all optional output destinations; fix table emoji alignment
Expand the features list to cover every output sink: Elasticsearch,
OpenSearch, Splunk, and PostgreSQL (premade dashboards), plus Kafka,
Amazon S3, Azure Log Analytics (Microsoft Sentinel), Graylog (GELF),
syslog, and HTTP webhooks.
Also re-pad the Python compatibility table using display width (the
status emoji render two columns wide), which is what markdownlint MD060
measures — the previous codepoint-based padding still tripped the rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: separate PostgreSQL from the premade-dashboards clause
PostgreSQL is a storage target without bundled premade dashboards, so it
shouldn't sit inside the "for use with premade dashboards" phrase next to
Elasticsearch/OpenSearch/Splunk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: move PostgreSQL to the non-dashboard outputs line
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: use compact markdown tables
Switch the markdown tables (Python compatibility, env-var section mapping)
to compact single-space format. It reads cleanly in a text editor and
sidesteps the column-alignment churn that emoji/variable-width content
caused with padded tables (markdownlint MD060). The reStructuredText grid
table in dmarc.md is left as-is — it relies on multi-line cells markdown
can't express.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AGENTS.md notes get_dmarc_reports_from_mailbox was halted at low coverage
because honest testing needed a live IMAP server or mocks so deep they test
the mock. mailsuite's MaildirConnection is a real on-disk backend with no
network or credentials, so the fetch -> parse/classify -> route loop can now be
exercised for real in CI.
TestGetDmarcReportsFromMailboxMaildir delivers real sample reports (one
aggregate, failure, and SMTP-TLS email) plus an unparseable message into a
Maildir INBOX, runs get_dmarc_reports_from_mailbox offline, and asserts on
observable results — parsed report counts and which archive subfolder each
message physically lands in:
- each report type routed to Archive/{Aggregate,Failure,SMTP-TLS}, the junk
message to Archive/Invalid, INBOX drained
- delete=True removes processed messages instead of archiving them
- test=True parses and returns reports but moves nothing and creates no folders
setUp resets the module-global SEEN_AGGREGATE_REPORT_IDS dedup cache so test
order can't drop an already-"seen" aggregate report, and the maildir lives at a
fresh subpath so mailbox.Maildir(create=True) actually builds cur/new/tmp.
Lifts parsedmarc/__init__.py from 76% to 82%, honestly.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The forensic→failure rename (#659) left a few loose ends and one deliberate
hold-back. This closes them.
Leftover rename misses (broken paths / stale canonical names):
- CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/*
- dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml
→ dmarc_failure_dashboard.xml (the file was already renamed; the import path
and view name were not)
- docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example
- samples/parsedmarc.ini: save_forensic → save_failure
- pyproject.toml, README.md: canonical "failure" naming
(ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.)
Archive subfolder rename + on-startup migration:
- New failure reports now archive to <archive>/Failure (was <archive>/Forensic).
- _migrate_forensic_archive_folder() runs once on startup (best-effort):
renames Forensic→Failure when no Failure folder exists yet, merges the two
when both exist, no-ops when there's no legacy folder, and logs-and-skips a
mailbox it can't reorganize (warn, don't crash). This consolidates pre- and
post-rename failure reports into one folder, replacing the previously
documented decision to keep the folder named Forensic to avoid a split
archive. Uses the folder-management API (folder_exists / rename_folder /
merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0.
Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana):
- Forensic panel titles + the datasource label → Failure; the fo-column display
label and its linked byName field-override matcher both → "Failure Policy"
(changed together so the column-width override keeps matching).
- dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both
pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana
dashboards); RESEED wipe loop now also clears dmarc_failure* indices.
- Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an
orphan export accidentally committed in #736 and referenced by nothing.
Tests (tests/test_init.py):
- TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via
mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full
get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds).
- TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir
can't reproduce — a backend that raises mid-operation must warn, not crash.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a PostgreSQL output backend as a lighter-weight alternative to
Elasticsearch/OpenSearch, configured via a [postgresql] section
(host/port/user/password/database or a libpq connection_string). Tables
are created automatically on first run; a Grafana dashboard is included.
- psycopg is an optional extra (pip install parsedmarc[postgresql]); the
import is guarded so `import parsedmarc` works without it, and
PostgreSQLClient raises a clear install hint when constructed without
the driver. Binary wheels aren't available for every platform.
- Schema captures the RFC 9990 / DMARCbis aggregate fields: np, testing,
discovery_method, generator, xml_namespace, and per-result human_result
on the DKIM/SPF auth-result tables.
- forensic -> failure naming throughout (table dmarc_failure_report,
save_failure_report_to_postgresql, dashboard, docs) to match #659.
- Failure-report de-duplication mirrors the Elasticsearch backend exactly:
arrival date + From + To + Subject (NULL-safe via IS NOT DISTINCT FROM;
semantic JSONB equality). Aggregate and SMTP-TLS use ON CONFLICT.
- PostgreSQLClient.close() for clean CLI shutdown; comment documents why
the two timestamp helpers must stay distinct (report dates are local,
record/SMTP-TLS dates are UTC).
- CLI: config parse raises ConfigurationError on missing
host/connection_string; wired into _init_output_clients + save loops.
- Tests in tests/test_postgres.py (helpers, mocked-DB save assertions,
create_tables, connect/error wrapping, dedup, real-sample round trip)
and tests/test_cli.py (config parse + end-to-end save wiring incl.
AlreadySaved/PostgreSQLError handling). postgres.py at 99% line
coverage; only _main's output-client-init retry path is left.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Appending _FILE to any PARSEDMARC_{SECTION}_{KEY} env var reads the
value from the referenced file, with one trailing newline stripped.
This matches the Postgres/MariaDB/Redis container-image convention so
Docker Compose and Kubernetes secret mounts work without extra glue,
keeping credentials out of plain environment: blocks (and out of
docker inspect, container logs, and /proc/<pid>/environ).
When both the direct var and its _FILE companion are set, the file
wins. A missing or unreadable file raises ConfigurationError rather
than silently degrading to an empty credential. The four pre-existing
config keys whose own names end in _file ([general] log_file,
[msgraph] token_file, [gmail_api] credentials_file / token_file)
keep their direct-path semantics; pass their values via secret by
doubling the suffix (_FILE_FILE).
Sean WhalenandClaude Opus 4.8
Ivan The GeekandGitHub
DVBandGitHub
VincentandGitHub