mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-16 13:34:56 +00:00
01b85ab7def0d3d405596ba11af4c3c4e67e5379
1571
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
01b85ab7de |
Fix aggregate-report timestamp skew on non-UTC hosts (#819) (#821)
record["interval_begin"]/["interval_end"] are UTC wall-clock strings
(produced in __init__.py by strftime() on a datetime already converted
via human_timestamp_to_datetime(..., to_utc=True)), but the
Elasticsearch and OpenSearch per-record save loops and the Splunk HEC
aggregate-report event builder re-parsed them without assume_utc=True,
so on non-UTC hosts they were misinterpreted as local time and shifted
by the host's UTC offset -- shifting the stored date_begin/date_end,
the daily/monthly index date, and the Splunk event time. Verified by
parsing a real sample report under TZ=Europe/Warsaw vs TZ=UTC and by
reproducing the exact skew via human_timestamp_to_unix_timestamp().
This is the same class of bug fixed for arrival_date_utc in #811/#812
(commit
|
||
 
|
f33951c0bb |
Fix host-dependent flakiness in testMissingEverythingRaisesFileNotFoundError (#820)
The system-path fallback list in _get_ip_database_path() checks real absolute paths like /usr/share/GeoIP/GeoLite2-Country.mmdb. On a host that actually has one installed there, the fallback succeeds and no FileNotFoundError is raised, regardless of the test's mocked bundled path -- the code is behaving correctly, the test just wasn't isolated from the host filesystem. Patch os.path.exists to force every system path to look absent so the test is host-independent. Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
547e6d7113 |
chore: update IPinfo Lite MMDB (#818)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
18b38c991d |
Add verify skill and document model roles in CLAUDE.md (#817)
* Raise test coverage: utils, elastic, and opensearch to 100%
Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).
New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.
Bugs found by the new tests, fixed in the same PR per the testing
standards:
- parse_email() crashed with KeyError: 'Headers' on messages whose
From header is present but unparseable (e.g. a bare "From:" line):
the fallback read parsed_email["Headers"], but the parsed headers
are stored under lowercase "headers" (assigned a few lines up in
the same function), so the key never exists. At the CLI surface
this made any failure report whose embedded sample had an empty
From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
when the probe could not reach the API, contradicting its own
docstring ("other errors are logged and the token is still
accepted"): _ipinfo_api_lookup() returns None on network errors
instead of raising, so the probe's exception handler was
unreachable. The probe now checks the lookup result and warns on
failure; 401/403 still raises InvalidIPinfoAPIKey.
Dead code deleted rather than padded with tests:
- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
(the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
input branch assigns file_object or raises first (confirmed by
pyright narrowing with the guard removed).
Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use assertGreater for the reverse-DNS-map fallback size check
Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add verification claude skill for parsedmarc CLI usage and sample inputs
* Document model roles for feature work in CLAUDE.md
Codifies the plan-with-Fable / implement-with-Sonnet / review-with-Fable
split (Opus as fallback) for feature work and PR reviews.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
746da77de5 |
Raise test coverage: utils.py, elastic.py, and opensearch.py to 100% (#816)
* Raise test coverage: utils, elastic, and opensearch to 100%
Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).
New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.
Bugs found by the new tests, fixed in the same PR per the testing
standards:
- parse_email() crashed with KeyError: 'Headers' on messages whose
From header is present but unparseable (e.g. a bare "From:" line):
the fallback read parsed_email["Headers"], but the parsed headers
are stored under lowercase "headers" (assigned a few lines up in
the same function), so the key never exists. At the CLI surface
this made any failure report whose embedded sample had an empty
From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
when the probe could not reach the API, contradicting its own
docstring ("other errors are logged and the token is still
accepted"): _ipinfo_api_lookup() returns None on network errors
instead of raising, so the probe's exception handler was
unreachable. The probe now checks the lookup result and warns on
failure; 401/403 still raises InvalidIPinfoAPIKey.
Dead code deleted rather than padded with tests:
- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
(the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
input branch assigns file_object or raises first (confirmed by
pyright narrowing with the guard removed).
Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use assertGreater for the reverse-DNS-map fallback size check
Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Sean Whalen
|
a1da7b3420 | Bump version to 10.2.1 10.2.1 | ||
|
|
3fda55d385 |
Make Microsoft Graph connection activity observable (#815)
* Make Microsoft Graph connection activity observable parsedmarc only configured its own logger, so all Graph connection activity was silently dropped even with --debug: the mailbox layer logs under mailsuite.mailbox.graph, token acquisition under azure.identity (including the AADSTS error codes that distinguish a local config problem from an Exchange Online / Entra ID one), and HTTP traffic under httpx/msgraph — none of which had a handler or level set. _main() also logged nothing around the MSGraphConnection call, so a hang left no trace at all. Three changes, all parsedmarc-side (no mailsuite changes needed): - Log a redacted connection summary at INFO before connecting (auth method, tenant ID, client ID, mailbox, Graph URL) plus a --debug detail line with certificate path, token-file path, and set/not-set flags for secrets. Secret values are never logged; a regression test asserts they don't appear in captured output. - Log a timing line after the connection object is initialized. - Propagate parsedmarc's --verbose/--debug level and handlers to the dependency loggers (mailsuite, azure, msgraph, httpx, httpcore) via _configure_dependency_logging(), synced to exactly the parsedmarc logger's handlers so SIGHUP log-file swaps neither duplicate output nor write to closed handlers. At the default level dependency loggers sit at WARNING, so their warnings keep surfacing (formatted) without new noise. All four new tests fail on the unfixed code (verified by stashing the cli.py change). Fixes https://github.com/domainaware/parsedmarc/issues/814. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Disable propagation on dependency loggers; document kiota's absence Set propagate=False on the dependency loggers when syncing handlers, so a stray logging.basicConfig() anywhere in the process cannot double-print every dependency record through the root logger — the function already owns these loggers' handler lists, and this makes that ownership complete. Asserted alongside the existing level/handler checks. kiota_http and its sibling packages were considered for _DEPENDENCY_LOGGERS but verified to not use Python logging at all (their observability is OpenTelemetry tracing), so a comment now records why they are absent rather than leaving the omission to be "fixed" later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
|
7fed72798a |
Stop system GeoIP files from shadowing the bundled IPinfo database (#813)
* Stop system GeoIP files from shadowing the bundled IPinfo database _get_ip_database_path() searched well-known system paths (including /usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before the database parsedmarc manages, so on any host with a distro GeoIP package installed every lookup silently used a country-only — and often years-old — database instead of the bundled IPinfo Lite one. That disabled ASN enrichment entirely (asn/as_name/as_domain were None for every IP) and with it the ASN-fallback path into the reverse-DNS map, with no signal beyond a generic "IP database is more than a month old" warning. Verified live on a Fedora host whose distro GeoLite2-Country.mmdb dated to December 2019. New precedence: explicit ip_db_path -> _IP_DB_PATH selected by load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system paths as a true last resort (only consulted when the bundled data file is missing). The selected file is logged at debug level so a --debug run shows which database answered. The automatic system-path pickup was documented behavior, so installation.md now tells MaxMind GeoLite2 users to set ip_db_path explicitly, with a migration note. Both new regression tests reproduce the shadowing portably via a decoy CWD GeoLite2-Country.mmdb (the fallback list includes relative names), and fail on the unfixed code (verified by stashing the source change). Fixes https://github.com/domainaware/parsedmarc/issues/810. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: accurate fallback comment, dedup selection log, cover fallback tiers - Correct the nothing-found-anywhere comment: the os.stat() age check raises FileNotFoundError before the caller's open_database() would. - Log "Using IP database at ..." only when the selected path changes instead of on every uncached IP lookup, so --debug runs over large batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in the test fixture for order-independence. - Cover the previously untested branches of _get_ip_database_path: system-path fallback when the bundled database is missing, the FileNotFoundError when nothing exists anywhere, the stale-database warning, and the log-once-per-path behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
|
cdda5dae62 |
Fix failure-report timestamp skew on non-UTC hosts in ES/OpenSearch/Splunk outputs (#812)
* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks arrival_date_utc is a UTC wall-clock string (generated in parse_failure_report via an aware-UTC strftime), but elastic.py, opensearch.py, and splunk.py parsed it back into a naive datetime and called .timestamp(), which per the Python docs interprets naive values as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp). On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date field, used in the failure-report dedup match query, and sent as the Splunk HEC event time was therefore off by the host's UTC offset (verified -3600 s under TZ=Europe/Warsaw in January). Add an assume_utc keyword to human_timestamp_to_datetime() / human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive parses, and use it at the three arrival_date_utc call sites. Aware inputs (explicit offsets) are unaffected; all other callers keep the existing local-time semantics, whose round-trip with timestamp_to_human is self-consistent on a single host (the broader local-time output question is tracked separately in issue #811 bug 2). The three new sink regression tests fail on the unfixed code (verified by stashing the source changes) and force TZ=Europe/Warsaw via time.tzset() so they catch the skew even on UTC CI runners. Fixes half of https://github.com/domainaware/parsedmarc/issues/811. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from the four timestamp regression tests into a shared tests/tzutil.py force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s docstring, which said the return type was float while the function returns int. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean Whalen
|
15dd69cc28 | fix: update virtual environment setup to use python3 -m venv | ||
|
|
fa8faa6b39 |
MS Graph: case-insensitive auth_method + ClientAssertion support (#809)
* draft fix MS Graph * Add ClientAssertion auth method support for MS Graph in cli.py MSGraphConnection/AuthMethod (via mailsuite) already supported ClientAssertion, but cli.py never parsed a client_assertion config value or passed it through, so it was unusable from the CLI. Wire config_msgraph.client_assertion through _parse_config and the MSGraphConnection call, and document the auth method (including its short-lived-JWT caveat vs. Certificate/ClientSecret for watch mode). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Ivan The GeekandGitHub
|
a6241d537e |
Fix grammatical error in project maintenance note (#805)
The sponsors note read "This is a project is maintained by one developer" in both README.md and docs/source/index.md; drop the stray "is a". |
||
|
|
25638cc3cd |
chore: update IPinfo Lite MMDB (#807)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
2547422f40 |
chore: update IPinfo Lite MMDB (#804)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandGitHub
|
c423b8dfff | Modernize type hints to PEP 585 / PEP 604 syntax (#803) 10.2.0 | ||
|
Sean WhalenandClaude Opus 4.8
|
d13eb86782 |
Advertise supported Python versions via trove classifiers
requires-python stays ">=3.10" (already correct and matching the CI matrix of 3.10-3.14); add the per-version Programming Language :: Python :: 3.10-3.14 classifiers and "3 :: Only" so the PyPI page lists supported versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a67e8d3ebc |
10.2.0 - Explain why a report is invalid instead of "Not a valid report" (#802)
* Explain why a report is invalid instead of "Not a valid report"
The parser catches broadly so one malformed report can't crash a batch,
but every failure surfaced as the generic ParserError("Not a valid
report"), telling operators nothing about the cause.
parse_report_file() now keeps each format parser's specific error as it
tries aggregate XML -> SMTP TLS JSON -> report email, and when all three
reject the input it content-sniffs the leading byte to surface the single
relevant reason (e.g. "Invalid aggregate report: Missing field:
'org_name'", or "Not a recognized report format (...)"). The CLI already
logs str(error), so this reaches the user with no cli.py change.
Every parser catch site also re-raises with `raise ... from <original>`,
preserving the underlying ExpatError / JSONDecodeError / KeyError /
archive errors on __cause__ for library callers and tracebacks. The same
exception *types* are still raised.
Finally, the catch-all "unexpected error" branches append
`(raised at <file>:<line>)` from the deepest traceback frame, but only
when the parsedmarc logger is at DEBUG level (e.g. the CLI's --debug);
normal-level output is unchanged.
Bumps the in-progress version to 10.2.0 and documents all three in the
CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover the failure-report path in parse_report_file error tests
The reason-surfacing tests covered the aggregate and SMTP TLS branches but
not failure reports, which reach parse_report_file only via the email
path. Add a malformed multipart/report failure email (missing the required
Source-IP) and assert the message names the failure format and the missing
field rather than collapsing to "Not a valid report".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover the new error-reporting lines; drop one unreachable catch
Bring the lines added by this PR to full test coverage:
- Test _exc_origin() with a never-raised exception (no __traceback__) so the
"no frames" guard is exercised.
- Test _parse_smtp_tls_failure_details() with a non-dict, which raises
TypeError (not KeyError) and exercises the generic catch-all.
- Test parse_report_email() with an unparseable Date header, which trips the
initial mail-parse catch-all and becomes a ParserError.
Two dead lines are removed rather than hidden, per the project's "delete
unreachable branches, no # pragma: no cover" rule:
- _looks_like_email() looped with a `continue` for blank lines, but every
caller passes lstrip()-ed text, so the first line is never blank. Simplified
to inspect the first line directly.
- parse_report_email()'s `except Exception` after `except InvalidFailureReport`
was unreachable: parse_failure_report wraps its entire body and provably
raises only InvalidFailureReport, which the preceding handler already catches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix IndexError when backfilling envelope_from from SPF results
_parse_report_record() backfills a missing/empty envelope_from from the
last SPF auth result's domain. The "envelope_from is None" branch gated on
the raw auth_results["spf"] list but indexed the filtered
new_record["auth_results"]["spf"] list, which only holds results that have
a domain. A reporter sending an SPF result with no domain made the filtered
list empty while the raw list was non-empty, so [-1] raised IndexError and
the whole record failed to parse.
The two near-identical envelope_from backfill branches (missing identifier
vs. empty identifier) drifted apart -- only one was updated when the
filtered new_record list was introduced -- which is what let them disagree
on which list to read. Merge them into a single path, keyed on
dict.get("envelope_from") is None, that gates and indexes the same raw list
with the "domain" membership guard the missing-identifier branch already
used.
Regression test: envelope_from=None with an SPF result carrying no domain
now parses to envelope_from=None instead of raising. This is the bug that
motivated the surrounding error-reporting work in this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Trim comment
* Cover the touched error branches; drop a dead UnicodeDecodeError catch
Codecov flagged the pre-existing error branches this PR touched (adding
`from e` / `_exc_origin`) as changed-and-uncovered. Most are real
malformed-report paths, so add honest tests that drive them with realistic
inputs:
- parse_smtp_tls_report_json: nested missing key (date-range without
start-datetime) -> InvalidSMTPTLSReport chaining a KeyError.
- parse_aggregate_report_xml: non-structured report_metadata -> the
AttributeError branch ("Report missing required section").
- parse_report_email: valid legacy text/plain failure report (success path),
a text report missing its fields, a base64 attachment of malformed
aggregate XML, and one of invalid SMTP TLS JSON.
- parse_report_file: gzipped junk -> the str branch of the content sniff.
The `except UnicodeDecodeError` in extract_report is removed as dead code
(no `# pragma: no cover`, per the repo rule): str-mode streams are already
rejected by explicit isinstance checks, and every decode() uses
errors="ignore", so it can never fire. str-mode still raises ParserError.
Also rename the two new failure-report tests from "Forensic" to "Failure"
and add an AGENTS.md rule: RUF reports are "failure reports"; "forensic" is
reserved for the literal backward-compat alias identifiers only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix pyright errors in the new error-branch tests
CI runs pyright over the whole repo (tests included); these slipped through
because the local check only covered parsedmarc/__init__.py:
- _parse_smtp_tls_failure_details("not a dict") is a deliberate wrong-type
test -> targeted `# pyright: ignore[reportArgumentType]`.
- result["report"]["source"]["ip_address"] on a ParsedReport TypedDict ->
cast(FailureReport, result["report"]) first, matching existing tests.
This is what failed lint-docs-build (and, since `test` needs it, skipped the
Codecov upload) on the prior commits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
bc17e6afb2 |
chore: update IPinfo Lite MMDB (#801)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
d40447ea91 |
chore: update IPinfo Lite MMDB (#800)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean Whalen
|
8275665e11 |
10.1.1
- Require `mailsuite>=2.2.2.2` to honor config reloading in the IMAP `IDLE` loop10.1.1 |
||
|
|
8337b67351 |
Cover both arms of the optional psycopg import in postgres.py (#799)
The module-level try/except import is environment-dependent: with psycopg installed the ImportError fallback never runs, and without it (CI's test job) the successful-import arm never completes — so Codecov flags one side or the other no matter where coverage is measured (it flagged the import line on master right after #798 merged). Exercise both arms explicitly: execute the module's source into a fresh, throwaway module object (importlib.util.module_from_spec + exec_module) under a patched sys.modules — a None entry forces ImportError, fake module entries force the success path — and assert on the psycopg / psycopg_json bindings each arm produces. The throwaway-module approach (rather than importlib.reload) leaves the canonical parsedmarc.postgres untouched, so the identity of PostgreSQLError / AlreadySaved held by the rest of the test module is preserved. Verified covered in both environments: with the venv's real psycopg, and with psycopg hidden via a PYTHONPATH shim to simulate CI; the import block reports no missing lines either way. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>10.1.0 |
||
|
|
eaeea4f53d |
Make the whole codebase pass pyright cleanly and enforce it in CI (#798)
* Make the whole codebase pass pyright cleanly and enforce it in CI Fix all 102 pyright (1.1.410, standard mode) errors across the library, tests, and maps scripts, then pin and enforce the zero-errors bar: - postgres.py: make the optional psycopg import TYPE_CHECKING-aware so the module is properly typed while keeping the runtime install-hint fallback; import psycopg.types.json explicitly as psycopg_json (the old psycopg_types.json attribute access only worked because psycopg imports the submodule eagerly); have _connect()/_ensure_connected() return the live connection so save methods use a non-Optional local; type the DDL list as list[LiteralString] to match psycopg's execute() overloads. - kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error fallback statically via TYPE_CHECKING (kafka-python 3.0 removed NoBrokersAvailable), which also fixes _BootstrapError's import resolution in tests. - syslog.py: go through getattr/setattr for SysLogHandler.socket (absent from typeshed); type the save_* methods with the report TypedDicts (single or list, matching cli.py call sites — gelf.py gets the same signatures); raise ValueError when retry_attempts < 1 instead of falling through and registering a None handler (bug fix, with a regression test and a CHANGELOG entry). - elastic.py / opensearch.py: human_result params are Optional[str]. - maps scripts: sort_csv declared a return type but never returned (now -> None); seen_sort_field_values was possibly unbound; convert_to_utf8's src_encoding is Optional[str]. - tests: cast sample-report dict helpers to their TypedDicts; mark deliberate wrong-type calls with targeted pyright ignores; add narrowing asserts for Optional results; access the mocked KafkaProducer through a cast helper; match the mailsuite fetch_message base signature (**kwargs); patch the renamed parsedmarc.postgres.psycopg_json in test_postgres's setUpModule. Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc, tests, docs; standard mode), pyright==1.1.410 pinned in the [build] extra (pinned exactly so a new pyright release can't break CI without a code change), and a "Check types" step in the lint CI job — which now also runs ruff format --check and installs the [postgresql] extra so the optional psycopg import resolves. Documented in AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Set session headers via update() instead of replacing the dict requests 2.34 ships inline type annotations, and Session.headers is a CaseInsensitiveDict[str] — assigning a plain dict fails pyright there (the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid it). headers.update() is correctly typed against both versions, and is the documented requests idiom: it overrides User-Agent and the client-specific headers while keeping the session's defaults (Accept-Encoding, Connection) instead of wiping them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
0c456d44ed |
Declare backward-compatible method aliases inside class bodies (#797)
* Declare backward-compatible method aliases inside class bodies Assigning the legacy save_forensic_* aliases onto the classes after the class body (KafkaClient.save_forensic_reports_to_kafka = ...) is invisible to static type checkers, so Pylance/Pyright flagged every assignment and every use with reportAttributeAccessIssue. Declaring the alias inside the class body is statically visible — the IDE errors disappear and the aliases get autocomplete and proper typing. Runtime behavior is identical (same function object bound as a method), guarded by the existing assertIs alias tests, whose type-ignore comments are now unnecessary. Also add a pyright ignore on the NoBrokersAvailable import in kafkaclient.py: the import is guarded by try/except ImportError for kafka-python 2.x, but Pyright resolves against the installed 3.x where the name no longer exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump version to 10.1.0 10.0.4 is tagged and released; CHANGELOG.md already documents the in-progress 10.1.0 section that this release will ship. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
ebc6a55715 |
Switch from kafka-python-ng to kafka-python>=2.3.2 (#795) (#796)
kafka-python-ng is archived and vulnerable to CVE-2026-10142 and CVE-2026-10143, both fixed in upstream kafka-python 2.3.2. kafka-python 3.0 removed the NoBrokersAvailable exception (a failed producer bootstrap now raises KafkaTimeoutError), so kafkaclient.py imports whichever the installed version provides via a compat shim, keeping the >=2.3.2 range honest for both 2.x and 3.x. Verified against kafka-python 3.0.0 (full test suite) and 2.3.2 (import shim resolution). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Sean Whalen
|
a4be378e30 | Update the changelog | ||
|
d3510da3a6 |
feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI (#794)
* feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI Previously SIGTERM (systemctl stop, docker stop, Kubernetes pod termination) killed parsedmarc mid-batch, tearing output writes and silently dropping buffered Kafka records. Shutdown is now cooperative: - SIGTERM/SIGINT set a flag that is polled at safe boundaries. The one-shot CLI checks it between batches; watch mode passes it as `config_reloading` so the mailbox backend -- including the IMAP IDLE loop -- returns once the current batch is fully processed. Either way the in-flight batch and its output writes finish before the process exits 0. - Ctrl-C is a double-tap: the first press is graceful, the second short-circuits to os._exit(130). - Output clients are now closed on every exit path (atexit plus a trailing close in _main), fixing a long-standing leak where one-shot runs and graceful shutdowns never flushed Kafka / closed Elasticsearch / S3 / etc. Docs: the example systemd unit gains KillSignal=SIGTERM and TimeoutStopSec=60 (keep it above mailbox_check_timeout). Tests cover watch shutdown, the one-shot between-batch stop, the SIGINT double-tap, and the output-client-close leak. * test: cover the one-shot mbox-loop shutdown break Extend the one-shot SIGTERM test to also pass an .mbox path so a single run exercises both shutdown checkpoints: the file-batch loop break and the subsequent mbox loop break (which Codecov flagged as the only uncovered lines on PR #794). is_mbox is keyed by suffix and get_dmarc_reports_from_mbox is asserted not called, since the mbox loop breaks before reaching it. * test: narrow signal.getsignal() return before invoking in SIGINT test signal.getsignal() is typed Callable | int | Handlers | None; calling it directly fails pyright's callable check. Assert callable() first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
b869235224 |
Build multi-arch (amd64+arm64) Docker images with PostgreSQL support (#793)
* Build multi-arch Docker images with PostgreSQL support The prebuilt image now installs the `[postgresql]` extra, so the optional PostgreSQL output backend (psycopg) works out of the box in the container without a separate `pip install` (#792). The wheel path is resolved into a variable before appending the extra so the shell doesn't treat `*.whl[postgresql]` as a bracket glob. The build workflow now sets up QEMU + Buildx and builds a multi-arch manifest for `linux/amd64` and `linux/arm64`, so the image runs natively on 64-bit ARM hosts such as a Raspberry Pi (#789). Every compiled dependency (psycopg[binary], lxml, maxminddb, cryptography) ships prebuilt aarch64 manylinux wheels, so the arm64 build adds no source-compilation step. A `pull_request` trigger (scoped to the build inputs) and `workflow_dispatch` are added so the multi-arch build can be validated on PRs and rebuilt on demand; pushes are still gated on the release event, so neither pushes images. Closes #789 Closes #792 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Bump version to 10.0.4 to publish the new images The docker workflow only pushes to the registry on a `release` event, so shipping the multi-arch + PostgreSQL-enabled image requires cutting a release. 10.0.3 is already tagged, so bump to 10.0.4 and document the Docker changes in the changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Don't run the docker build on pull requests The pull_request trigger (added to validate the multi-arch build) re-ran the full ~10-minute amd64+arm64 build on every commit pushed to a docker-touching PR, because the pull_request `paths` filter matches against the PR's entire diff, not just the newest commit. That is wasteful once the build has been validated. Drop the pull_request trigger and rely on workflow_dispatch for on-demand validation (plus the existing master-push and release triggers). Also gate the registry login on the release event so that no non-release run authenticates to ghcr at all — a build can only ever be pushed from a published release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>10.0.4 |
||
|
|
8ec3cc9ce8 |
chore: update IPinfo Lite MMDB (#791)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
ec0a313669 |
chore: update IPinfo Lite MMDB (#788)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
f65eebc4d8 |
chore: update IPinfo Lite MMDB (#787)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
08db305e5a |
test: cover no-display-name Reply-To header flattening (#786)
The 10.0.3 Reply-To header flattening (elastic.py / opensearch.py line 711)
has two branches: display-name present ("Name <addr>") and absent (bare
address). The existing test only exercised the former, leaving the
empty-display-name branch uncovered — the two lines Codecov flagged on the
10.0.3 patch. Add a failure report whose Reply-To has no display name and
assert sample.headers["reply-to"] flattens to the bare address.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
e104f1118c |
Land 10.0.3 changes on master (#785)
PR #784 was stacked on the #783 branch and its base was never retargeted to master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master. master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes. This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the >=2.2.1 floor is already on master from 10.0.2). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>10.0.3 |
||
|
|
2c8b2c0f14 |
Bump mailsuite to >=2.2.1 (release 10.0.2) (#783)
* Bump mailsuite to >=2.2.1; release 10.0.2
mailsuite 2.2.1 raises the transitive mail-parser floor to >=4.2.1, which
stops mail-parser from returning a phantom ('', '') entry for absent address
headers (verified against samples/failure/* with mail-parser 4.2.1: cc/bcc
now parse to [] instead of [{address: ""}]). parsedmarc reads the mail-parser
object directly via its own parse_email(), so this previously caused an empty
{address: ""} Cc/Bcc entry to be indexed for every failure-report sample in
Elasticsearch/OpenSearch and emitted in JSON/S3/Kafka output.
The Reply-To-always-empty behavior in parsedmarc's own parse_email() (a
hyphen-vs-underscore key mismatch, not an upstream issue) and the failure
dashboards are out of scope here and tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: note CVE-2023-27043 hardening from mail-parser 4.2.1 in 10.0.2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sean Whalen
|
3f64e30f6f | Update version to 10.0.1 and bump mailsuite requirement to >=2.2.0 | ||
|
Sean Whalen
|
9e675bf43c | Update build command to use pytest with coverage reporting 10.0.0 | ||
|
Sean Whalen
|
d92593f2da | Add RFC 9990 fields to opensearch_dashboards.json | ||
|
|
180fc581fe |
fix: OSD Global-tenant import + dropped report files with glob metacharacters; validate dev stack on OpenSearch 3.x with PostgreSQL (#781)
* fix: import OpenSearch dashboards into the real Global tenant dashboard-dev-bootstrap.sh sent `securitytenant: global_tenant`. The OpenSearch security plugin reads that header as a tenant *name*, and `global_tenant` is a sample custom tenant from the security demo config -- not the shared Global tenant, whose token is the literal `global`. The import therefore landed in a separate `global_tenant` tenant (its own `.kibana_<hash>_globaltenant_1` index) and the dashboards were invisible to anyone viewing the Global tenant in OpenSearch Dashboards. Verified against the live dev cluster: `_find` under `securitytenant: global` returned 26 objects and `.kibana_1` (the Global tenant index the UI reads) went from 2 to 67 docs after re-importing with the fix. An empty/omitted header read 0 from Global -- it falls back to the user's configured default tenant -- so `global` is the only reliable token. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't drop report files whose names contain glob metacharacters The CLI expanded every file argument with glob(), which treats [, ], *, and ? as pattern syntax. A literal path like "[Netease DMARC Failure Report] Rent Reminder.eml" -- the bracketed shape many providers use for emailed failure reports -- was read as a character class, matched nothing, and was dropped before reaching the parser, with no error. File arguments that exist on disk are now taken literally; only non-existent paths are globbed, so shell-style wildcards still expand. Also adds "postgresql" to _KNOWN_SECTIONS so PARSEDMARC_POSTGRESQL_* env vars (and their _FILE Docker-secret variants) resolve like every other backend -- the PostgreSQL backend is new in 10.0.0, so this completes the unreleased feature rather than fixing a released regression, and is documented under the PostgreSQL enhancement, not Bug fixes. Regression tests added for both. Verified end-to-end: all four samples/failure/*.eml now index (the bracketed Netease report included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: validate dashboards on OpenSearch 3.x and add PostgreSQL to the dev stack The dev stack ran OpenSearch Dashboards 3.x against OpenSearch 2.x, an unsupported cross-major pairing. Bump opensearch to :3 (validated on 3.6.0: OSD import into the Global tenant and all dashboards work). Add a postgresql service plus bootstrap wiring so the new PostgreSQL backend is exercised alongside the others: wait for PG, seed it via PARSEDMARC_POSTGRESQL_* env vars on the same parsedmarc run, wipe it on RESEED, create a Grafana grafana-postgresql-datasource (uid dmarc-pg), and import dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json. PG seeding is gated on psycopg being importable: parsedmarc aborts the whole run (exit 1, nothing written to any backend) when a configured output backend can't initialize, so wiring in PG without the optional extra would silently zero ES/OS/Splunk too. When psycopg is absent the script warns and skips PG, leaving the other backends seeded. Also fix the Grafana admin password env: the container was given GRAFANA_PASSWORD, which Grafana ignores -- it reads GF_SECURITY_ADMIN_PASSWORD. Defaults to admin to match the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: list PostgreSQL on the premade-dashboards features bullet PostgreSQL ships a premade Grafana dashboard (dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json), so it belongs on the "for use with premade dashboards" bullet alongside Elasticsearch, OpenSearch, and Splunk rather than on the plain-output-destinations line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: clear stale org_email mapping conflict in the OpenSearch dashboards The aggregate index pattern in dashboards/opensearch/opensearch_dashboards.ndjson shipped a cached field-list snapshot where org_email was a text/object conflict, plus leftover org_email.#text and org_email.#text.keyword subfields. Those came from a cluster that had indexed a langAttrString email dict ({"#text": ..., "@lang": ...}) before the parser unwrapped it. org_email is mapped as Text() and parse_aggregate_report_xml now unwraps a dict email to a plain string, so current data is consistently text -- a clean cluster's _field_caps reports no conflict. Cleared the frozen conflict and the two artifact subfields, leaving org_email (text) and org_email.keyword, matching the live mapping. Verified: re-importing the corrected ndjson yields an index pattern with org_email as a plain text field and zero conflicts; only the aggregate index-pattern line changed, all other saved objects byte-identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: seed the RFC 9990 (dmarc-2.0) aggregate samples samples/aggregate/rfc9990-sample.xml and rfc9990-example.net!...xml were not in the bootstrap's SAMPLE_FILES, so the dev stack only ever indexed RFC 7489 reports and the new DMARCbis fields (np, testing, discovery_method, generator, xml_namespace) never appeared in the OpenSearch/Kibana indices or were available to the dashboards. Added both samples (one declares the urn:ietf:params:xml:ns:dmarc-2.0 namespace, the other is namespaceless RFC 9990-shaped, covering both detection paths). Verified the seeded data now carries np/testing/ discovery_method/generator and xml_namespace=urn:ietf:params:xml:ns:dmarc-2.0; OpenSearch Dashboards surfaces them on an index-pattern field-list refresh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: auto-resolve (or create) a venv for the seed and ensure psycopg The seed previously required parsedmarc to be pre-installed and only warned-and-skipped PostgreSQL when psycopg was missing. Resolve the seed environment by precedence instead: 1. explicit PARSEDMARC_BIN -> used as-is, nothing installed 2. active $VIRTUAL_ENV 3. existing repo venv/ or .venv/ 4. otherwise create $REPO_ROOT/venv For cases 2-4, run `pip install -e .[postgresql]` only when the CLI or psycopg is missing, so the dev stack can populate Postgres out of the box without a manual install step. The explicit-PARSEDMARC_BIN path is left untouched (and the psycopg seed guard still warns/skips if that env lacks the extra). Verified: a RESEED run resolves the active venv, seeds ES/OS/Splunk/PG including the RFC 9990 fields, with no output-client errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
411f5a8886 |
chore: tidy cSpell config; fix two doc typos (#779)
- Ignore data/export trees via cSpell.ignorePaths: parsedmarc/resources/** (maps tooling holds thousands of intentional foreign-language classifier keywords + bundled data), plus samples/** and dashboards/** (report samples and dashboard exports). These are data, not whitelist vocabulary, so excluding them keeps the editor quiet without bloating the word list. - Add the remaining genuine false-positives across code, docs, CI workflows, and editor config to cSpell.words (technical terms, library names, SQL/identifier tokens, brand/operator and multilingual examples from AGENTS.md, plus charliermarsh/junitxml/mktemp/pipefail/seanthegeek). - Fix two genuine typos found while triaging rather than whitelisting them: "maidir" -> "maildir" and "connexion" -> "connection". Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
2b3cd32b9c |
chore: move PostgreSQL Grafana dashboard into dashboards/grafana/ (#780)
The PostgreSQL dashboard shipped at the repo-root grafana/ by oversight; every other dashboard source lives under dashboards/ (opensearch/, grafana/, splunk/). Move it next to the existing Grafana dashboard, list it in dashboards/README.md, and fix the CHANGELOG path reference. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
caac8e68f0 |
docs: note DMARC RFC support in the features list (#778)
* docs: note DMARC RFC support in the features list The features list only mentioned "draft and 1.0" aggregate reports. Spell out the standards parsedmarc parses: RFC 7489 (legacy DMARC) and the final DMARC standard RFC 9989 with RFC 9990 aggregate reports, RFC 6591 and RFC 9991 failure reports, and RFC 8460 SMTP TLS reports. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: align Python compatibility table pipes (MD060) The emoji cells were padded for display width, leaving the source pipes misaligned by character count and tripping markdownlint MD060. Re-pad so every row's pipes line up by codepoint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: list all optional output destinations; fix table emoji alignment Expand the features list to cover every output sink: Elasticsearch, OpenSearch, Splunk, and PostgreSQL (premade dashboards), plus Kafka, Amazon S3, Azure Log Analytics (Microsoft Sentinel), Graylog (GELF), syslog, and HTTP webhooks. Also re-pad the Python compatibility table using display width (the status emoji render two columns wide), which is what markdownlint MD060 measures — the previous codepoint-based padding still tripped the rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: separate PostgreSQL from the premade-dashboards clause PostgreSQL is a storage target without bundled premade dashboards, so it shouldn't sit inside the "for use with premade dashboards" phrase next to Elasticsearch/OpenSearch/Splunk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: move PostgreSQL to the non-dashboard outputs line Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: use compact markdown tables Switch the markdown tables (Python compatibility, env-var section mapping) to compact single-space format. It reads cleanly in a text editor and sidesteps the column-alignment churn that emoji/variable-width content caused with padded tables (markdownlint MD060). The reStructuredText grid table in dmarc.md is left as-is — it relies on multi-line cells markdown can't express. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
ef2fb84cc0 |
test: cover parsedmarc's mailbox processing loop end-to-end on a real Maildir (#777)
AGENTS.md notes get_dmarc_reports_from_mailbox was halted at low coverage
because honest testing needed a live IMAP server or mocks so deep they test
the mock. mailsuite's MaildirConnection is a real on-disk backend with no
network or credentials, so the fetch -> parse/classify -> route loop can now be
exercised for real in CI.
TestGetDmarcReportsFromMailboxMaildir delivers real sample reports (one
aggregate, failure, and SMTP-TLS email) plus an unparseable message into a
Maildir INBOX, runs get_dmarc_reports_from_mailbox offline, and asserts on
observable results — parsed report counts and which archive subfolder each
message physically lands in:
- each report type routed to Archive/{Aggregate,Failure,SMTP-TLS}, the junk
message to Archive/Invalid, INBOX drained
- delete=True removes processed messages instead of archiving them
- test=True parses and returns reports but moves nothing and creates no folders
setUp resets the module-global SEEN_AGGREGATE_REPORT_IDS dedup cache so test
order can't drop an already-"seen" aggregate report, and the maildir lives at a
fresh subpath so mailbox.Maildir(create=True) actually builds cur/new/tmp.
Lifts parsedmarc/__init__.py from 76% to 82%, honestly.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
a6778707d7 |
Finish forensic→failure rename: archive-folder migration + dashboard/doc cleanup (#776)
The forensic→failure rename (#659) left a few loose ends and one deliberate hold-back. This closes them. Leftover rename misses (broken paths / stale canonical names): - CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/* - dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml → dmarc_failure_dashboard.xml (the file was already renamed; the import path and view name were not) - docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example - samples/parsedmarc.ini: save_forensic → save_failure - pyproject.toml, README.md: canonical "failure" naming (ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.) Archive subfolder rename + on-startup migration: - New failure reports now archive to <archive>/Failure (was <archive>/Forensic). - _migrate_forensic_archive_folder() runs once on startup (best-effort): renames Forensic→Failure when no Failure folder exists yet, merges the two when both exist, no-ops when there's no legacy folder, and logs-and-skips a mailbox it can't reorganize (warn, don't crash). This consolidates pre- and post-rename failure reports into one folder, replacing the previously documented decision to keep the folder named Forensic to avoid a split archive. Uses the folder-management API (folder_exists / rename_folder / merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0. Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana): - Forensic panel titles + the datasource label → Failure; the fo-column display label and its linked byName field-override matcher both → "Failure Policy" (changed together so the column-width override keeps matching). - dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana dashboards); RESEED wipe loop now also clears dmarc_failure* indices. - Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an orphan export accidentally committed in #736 and referenced by nothing. Tests (tests/test_init.py): - TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds). - TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir can't reproduce — a backend that raises mid-operation must warn, not crash. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
327fcff2b9 |
Add optional PostgreSQL storage backend (#667)
Adds a PostgreSQL output backend as a lighter-weight alternative to Elasticsearch/OpenSearch, configured via a [postgresql] section (host/port/user/password/database or a libpq connection_string). Tables are created automatically on first run; a Grafana dashboard is included. - psycopg is an optional extra (pip install parsedmarc[postgresql]); the import is guarded so `import parsedmarc` works without it, and PostgreSQLClient raises a clear install hint when constructed without the driver. Binary wheels aren't available for every platform. - Schema captures the RFC 9990 / DMARCbis aggregate fields: np, testing, discovery_method, generator, xml_namespace, and per-result human_result on the DKIM/SPF auth-result tables. - forensic -> failure naming throughout (table dmarc_failure_report, save_failure_report_to_postgresql, dashboard, docs) to match #659. - Failure-report de-duplication mirrors the Elasticsearch backend exactly: arrival date + From + To + Subject (NULL-safe via IS NOT DISTINCT FROM; semantic JSONB equality). Aggregate and SMTP-TLS use ON CONFLICT. - PostgreSQLClient.close() for clean CLI shutdown; comment documents why the two timestamp helpers must stay distinct (report dates are local, record/SMTP-TLS dates are UTC). - CLI: config parse raises ConfigurationError on missing host/connection_string; wired into _init_output_clients + save loops. - Tests in tests/test_postgres.py (helpers, mocked-DB save assertions, create_tables, connect/error wrapping, dedup, real-sample round trip) and tests/test_cli.py (config parse + end-to-end save wiring incl. AlreadySaved/PostgreSQLError handling). postgres.py at 99% line coverage; only _main's output-client-init retry path is left. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean WhalenandGitHub
|
0a703172de | Reorder enhancements in the changelog | ||
 DVBandGitHub
|
bf37ded688 | Add support for Elastic Cloud Serverless projects (#770) | ||
 VincentandGitHub
|
535d9db1ad |
cli: support _FILE suffix on PARSEDMARC_* env vars for Docker secrets (#772)
Appending _FILE to any PARSEDMARC_{SECTION}_{KEY} env var reads the
value from the referenced file, with one trailing newline stripped.
This matches the Postgres/MariaDB/Redis container-image convention so
Docker Compose and Kubernetes secret mounts work without extra glue,
keeping credentials out of plain environment: blocks (and out of
docker inspect, container logs, and /proc/<pid>/environ).
When both the direct var and its _FILE companion are set, the file
wins. A missing or unreadable file raises ConfigurationError rather
than silently degrading to an empty credential. The four pre-existing
config keys whose own names end in _file ([general] log_file,
[msgraph] token_file, [gmail_api] credentials_file / token_file)
keep their direct-path semantics; pass their values via secret by
doubling the suffix (_FILE_FILE).
|
||
|
|
b7b8383fa4 |
Expand honest test coverage from 59% to 83%; fix two latent bugs (#775)
* Expand honest test coverage from 59% to 83%; fix two latent bugs 271 new tests across the output modules, ES/OS clients, CLI config parsing, and the top-level parsing surface. Coverage measured against shipped code only (see [tool.coverage.run] source = ["parsedmarc"] omit = ["*/parsedmarc/resources/maps/*.py"] in pyproject.toml). Per-module results: s3.py 38% → 100% (also fixes SMTP-TLS-to-S3 bug below) gelf.py 40% → 100% syslog.py 46% → 100% kafkaclient.py 34% → 100% splunk.py 24% → 100% loganalytics.py 56% → 100% webhook.py 78% → 100% (also removes redundant try/except) elastic.py 36% → 99% opensearch.py 40% → 99% cli.py 52% → 69% __init__.py 74% → 76% (also fixes append_json bug below) utils.py 84% (unchanged in this PR) TOTAL 59% → 83% The remaining 17% is honest. The biggest unreached blocks are _main() in cli.py and the watch-mode mailbox iteration in __init__.py, both of which would require either standing up live subsystems (real Elasticsearch, real IMAP) or mocking deep enough that the test would verify the mock rather than the code. The PR-A AGENTS.md guidance — "if 90% requires faking it, ship 85% honestly" — applies here. Bugs fixed while writing tests: 1. parsedmarc/s3.py — SMTP-TLS-to-S3 was completely broken. save_report_to_s3 unconditionally read report["report_metadata"] when building S3 object metadata, but RFC 8460 §4.3 SMTP TLS reports are flat (no report_metadata sub-object). The CLI's surrounding try/except silently swallowed the KeyError, so every SMTP-TLS report quietly failed to upload. Also fixes a related issue: parse_smtp_tls_report_json stores begin_date as the raw ISO-8601 string from the report (per the SMTPTLSReport TypedDict and RFC 8460 §4.3), but the S3 code path assumed a datetime with .year / .month / .day attributes. Both fixed; the broken metadata-extraction branch now uses the flat-report fields, and the date branch normalizes via human_timestamp_to_datetime. 2. parsedmarc/__init__.py — append_json corrupted JSON output files on the second write. The original implementation opened files in "a+" mode, then seek()ed backwards to overwrite the trailing "]" with ",\n" before appending more elements. Python's docs are explicit (https://docs.python.org/3/library/functions.html#open): on POSIX, writes in "a"/"a+" mode always go to EOF regardless of seek() position. The result was that the second call produced [...]\n],\n[...] -style corrupted output instead of a single merged array. Replaced with a read-merge-write pattern: load the existing array (if any), append the new elements, rewrite the whole file. The CSV cousin append_csv was not affected — it doesn't seek backwards. 3. parsedmarc/webhook.py — removed redundant try/except blocks in save_aggregate_report_to_webhook / save_failure_report_to_webhook / save_smtp_tls_report_to_webhook. _send_to_webhook already catches every Exception itself, so the outer except blocks were unreachable dead code (covered nothing, defended against nothing, and inflated the source-line count without testing value). Testing approach: mocks at SDK boundaries (boto3 resource, kafka producer, requests session, opensearch/elasticsearch Document/Search, azure LogsIngestionClient). Tests verify the parsedmarc-side transformation logic — document/event construction, index/topic naming, dedup queries, error wrapping — rather than asserting on mock invocations as a proxy for behaviour. Where a branch is defensive against a caller that doesn't exist in the codebase, the test is omitted (commented in code rather than hidden behind a pragma). 547 tests total (was 276), all passing. ruff check + format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document the two bug fixes from this PR in the 10.0.0 changelog Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document testing standards in AGENTS.md Adds a "Testing standards" section covering the principles applied in PR-A (split) and PR-B (coverage expansion): - Coverage measures shipped code only — don't reintroduce tests/* to the scope, don't expand omit, don't use # pragma: no cover. - Honest tests assert on observable behaviour, not "the mock was called". Mock at SDK boundaries; parse the payload that gets sent. - "If 90% requires faking it, ship 85% honestly" — coverage is a tool, not a goal. PR-B's deliberate stops at cli.py 69% and __init__.py 76% are the documented precedent for when to halt. - Verify bug claims against the relevant RFC, internal types, installed SDK source, or upstream docs before changing code. Cite the source in the commit message and test docstring (RFC 8460 §4.3 and the Python open() docs for #775's two bug fixes are the pattern to follow). - Bugs found while writing tests are fixed in the same PR; the test doubles as the regression guard. - File layout (tests/test_<module>.py) is non-negotiable; module-level test loggers need fresh-handler setup so test ordering doesn't break assertLogs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Cover the corrupt-file fallback in append_json Codecov flagged 2 missing patch-coverage lines on PR #775: the except (json.JSONDecodeError, OSError) branch in append_json, which falls back to overwriting when the existing file isn't a parseable JSON array. Two new tests in tests/test_init.py:TestAppendJson exercise both paths: - test_corrupt_existing_file_is_overwritten_cleanly: existing file contains invalid JSON; append_json overwrites with the new array. - test_existing_file_with_non_list_root_is_overwritten: existing file parses as {"foo": ...} (dict, not list); the isinstance guard rejects it and we overwrite cleanly. Patch coverage now 100% on the bug fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
5b08627eaa |
Split tests.py into per-module tests/test_<module>.py (#774)
* Split tests.py into per-module tests/test_<module>.py The 5174-line tests.py monolith is split into per-module files under tests/, mirroring the checkdmarc layout: tests/test_init.py parsedmarc/__init__.py parsing surface tests/test_cli.py parsedmarc/cli.py + config / env-vars / SIGHUP tests/test_utils.py parsedmarc/utils.py (DNS, IP info, PSL, etc.) tests/test_webhook.py parsedmarc/webhook.py tests/test_kafkaclient.py parsedmarc/kafkaclient.py tests/test_splunk.py parsedmarc/splunk.py tests/test_syslog.py parsedmarc/syslog.py tests/test_loganalytics.py parsedmarc/loganalytics.py tests/test_gelf.py parsedmarc/gelf.py tests/test_s3.py parsedmarc/s3.py tests/test_maps.py parsedmarc/resources/maps/ maintainer scripts The split is purely a redistribution — no test bodies changed, no tests added or removed. All 276 existing tests pass under the new layout. The current tests.py contains two kitchen-sink classes (`Test` at line 54 and `TestEnvVarConfig` at line 2360) holding tests that span many modules. Their methods are routed to the correct per-module file by name prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx, TestSighupReload, etc.) move whole. Each target file gets its own `class Test(unittest.TestCase)` for the redistributed kitchen-sink methods, plus the thematic classes verbatim. Wiring updates: - `.github/workflows/python-tests.yml`: `pytest ... tests.py` → `python -m pytest ... tests/` (also switches to `python -m pytest` per the checkdmarc convention so cwd lands on the project root). - `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]` and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for `parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only batch tooling that ships out of the wheel; excluding them from coverage makes the headline number reflect only installed library code. Runtime coverage on the new layout is 59% (was 45% with maps counted), and PR-B will push it to 90%+. - `AGENTS.md`: documents the new layout and how to run individual files / tests; tells future contributors not to reintroduce a monolithic tests.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restore 66.9% coverage baseline (count tests/ + parsedmarc) Master's headline 66.9% number on Codecov includes the tests.py file itself (99.35% covered) being measured alongside parsedmarc/*. The original tests.py had no `[tool.coverage.run]` block, so coverage's default — "measure every file imported during the run" — counted the test code as if it were product code. The split commit added `source = ["parsedmarc"]` which suppressed measurement of the test files (correct in principle, since test files aren't shipped code), and that alone made the headline number drop by ~8 percentage points without any actual loss of testing. This commit swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]` so both halves are measured the way they were on master. Verified: 276 tests, 66.96% line coverage (effectively unchanged from master's 66.90%). If you want the shipped-code-only number (was the headline that this commit overrides), run `pytest --cov=parsedmarc tests/`. That number is currently 59% and is the focus of the upcoming coverage-expansion PR. Also adds junit.xml to .gitignore so the CI artefact doesn't get accidentally committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restrict coverage to shipped code (`source = ["parsedmarc"]`) Reverts the prior commit's `include = ["tests/*"]`. Counting the test files toward coverage was wrong — it conflates "shipped code exercised by tests" with "test code that pytest auto-runs", inflates the headline number, and rewards writing more tests rather than tests that verify more code. Master's apparent 66.9% was an artefact of the old monolithic tests.py having no [tool.coverage.run] block at all; coverage's default behaviour measured every imported file, including the test file itself at ~99% "covered", which added ~8 percentage points to the displayed number without any real testing signal. Restricting to `source = ["parsedmarc"]` plus the existing maps omit gives a meaningful baseline: 59% of shipped code is exercised by the test suite today. That's the number the next PR is targeting to lift to 90%+ before the 10.0.0 release; the Codecov "drop" here is a measurement correction, not a regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
ae1e5adb66 |
Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659)
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @
|
||
|
Sean WhalenandGitHub
|
8c5f63620c |
Fix Validate-dashboards CI: heredoc was redirecting itself to stdin (#773)
`echo "$response" | python3 - <<'PY' ... PY` redirected the heredoc to python3's stdin (where it was correctly read as the script body), but sys.stdin was then at EOF when the script called json.load(sys.stdin) — so the assertion blew up with 'Expecting value: line 1 column 1' even when Kibana's import had succeeded. Pass the response via env var instead. The OSD ndjson import itself was working all along (successCount: 26, success: true); only the assertion step was broken, so master has been showing a red Validate-dashboards run since the workflow was introduced. |