Commit Graph
1586 Commits
Author SHA1 Message Date
Sean Whalen 855d267650 Bump version to 10.2.4 and update changelog 10.2.4 2026-07-20 19:51:15 -04:00
supaeasyandGitHub d9f6532841 Default missing Feedback-Type and Authentication-Results in failure reports (#332) (#831)
Some Exim/cPanel-based gateways send DMARC failure reports without a
machine-readable message/feedback-report part. parse_report_email()'s
plain-text fallback synthesizes a minimal feedback report with only
Arrival-Date and Source-IP, but the Elasticsearch/OpenSearch outputs
access feedback_type and authentication_results with hard key lookups,
so every such report was archived but never indexed, failing with
"Failure report missing required field: 'feedback_type'".

parse_failure_report() now defaults feedback_type to auth-failure
(RFC 5965 3.1) and authentication_results to None (RFC 6591 3.1) with
logged warnings, matching the existing handling of the REQUIRED
Auth-Failure and Identity-Alignment fields. Adds a sanitized sample
and a regression test asserting the sink-required keys are present.
2026-07-20 15:21:55 -04:00
5dc83613e6 chore: update IPinfo Lite MMDB (#832)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-20 15:15:19 -04:00
1986835f51 Fix daily-cadence charts defaulting to sub-day time buckets (#828) (#830)
DMARC aggregate reports post one data point per reporting period
(typically daily), but several dashboard time-series panels used a
dynamically-computed bucket interval that scales with the viewed time
range/panel width instead of the data's actual cadence, producing a
spiked/gapped chart whenever that computed interval landed below a day:

- OpenSearch Dashboards: both date_histogram aggregations used
  "interval": "auto". Fixed to the "d" unit code (verified against
  OpenSearch-Dashboards' and Kibana's own _interval_options.ts /
  parse_interval.ts source — "day" is not a valid value and would throw
  at render time).
- Grafana + PostgreSQL: 7 query targets used
  $__timeGroup(col, $__interval); hardcoded to $__timeGroup(col, '1d'),
  matching the equivalent already-correct panels in the Grafana +
  Elasticsearch dashboard.
- Splunk: both `timechart` panels had no explicit span, so Splunk's
  100-bin auto-algorithm only coincidentally produced daily buckets for
  the dashboard's default 7-day view. Added span=1d per Splunk's own
  timechart documentation.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 20:17:57 -04:00
Sean WhalenandClaude Sonnet 5 3b53f836a6 Map two domain batches: MMDB ASN-coverage gap and uncategorized-sources export
Batch 1 — MMDB coverage-gap scan (find_unmapped_as_domains.py, default
4,096-IP floor): 132 candidates collected and classified. Manually
audited every auto-classified row rather than trusting the classifier
output directly, since it derives as_name straight from the MMDB and
will auto-classify off that single string even with no reachable
homepage — which fails the two-corroborating-sources bar. Demoted 10
rows to known-unknown on that basis (e.g. pellera.com's ASN registrant
is "Converge Technology Solutions", aerloop.in's is "Aerpace" — no
shared brand token, single source). Corrected several brand-quality
issues before they hit the map: sdtv.com.tw was about to ship as
"SDTV Webmail" (a login-page title) instead of "San Da Cable TV";
liyang.gov.cn was about to ship the hosting network's name instead of
the actual government operator (confirmed via Chinese-language WHOIS).
Also fixed a pre-existing bad entry noticed in passing: backwaves.net
was mapped under a garbled Chinese tagline instead of "Back Waves".

Result: 52 map rows, 82 known-unknown entries.

Batch 2 — plain-text uncategorized-sources export (842 source names via
find_unknown_base_reverse_dns.py -i, mix of raw MMDB as_name strings
and reverse-DNS domains) -> 377 unknown domains -> collected and
classified. This batch surfaced more data-quality traps: a parked
domain-registrar welcome page (serveur-vps.net) and a "Site under
maintenance" title (gou.go.ug, resolved via TLD instead) auto-classified
as if they were live operators; a WHOIS privacy-proxy service name
("GKG.NET Domain Proxy Service Administrator") slipped past the
privacy-org filter for atlasok.com; grohe.org.ru turned out to be a
Russian third-party reseller storefront, not the real Grohe corporation,
so it was held back from the map to avoid misattributing a reseller's
mail to the multinational. Refined the two-source test to distinguish
domain-WHOIS (ties directly to the domain; counts even without a
literal name match, e.g. portotelecom.net.br's WHOIS "Rondotech" backs
its self-described "Porto Telecom" brand) from MMDB as_name
(infrastructure-level signal, needs an actual lexical link, same
caution as IP-WHOIS). Used existing map entries as legitimate second
sources for redirects/aliases (vsys.host -> v-sys.org, metronet.net ->
metronet.com, mentari.net.id -> megahub.id) and cleaned up
megahub.id's own tagline-derived name to "Megahub" while there.

Result: 24 map rows, 353 known-unknown entries.

Also ignore the scratch/ working directory used to stage the
uncategorized-sources export for this pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 19:29:21 -04:00
7d87ba18be Ingest uncategorized-sources exports and add MMDB coverage scan with anti-poisoning guards (#829)
* Accept plain-text uncategorized-sources lists in find_unknown_base_reverse_dns.py

Dashboard exports of uncategorized email sources are plain-text lists of
one source name per line — a mix of raw MMDB as_name strings (when the
source IP had no PTR and resolved via the IPinfo Lite MMDB) and base
reverse-DNS domains. The script already translates as_names to their
as_domain and subtracts mapped/known-unknown entries, but only read a
hardcoded source_name-headed CSV.

Add -i/--input and -o/--output flags (defaults preserve current
behavior) and auto-detect the input format from the first line: a
source_name CSV header selects the existing DictReader path, anything
else is read as plain text with each line taken verbatim (never
comma-split, since as_names contain commas) and deduped
case-insensitively. Fix the missing-input error message, which
reported the map path instead of the input path.

Document the new entry point in the maps README and AGENTS.md, and make
explicit in the brand-quality triage rule that map display names must be
human-friendly operator names — never raw as_name strings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add MMDB coverage scan script with anti-poisoning guards

find_unmapped_as_domains.py turns the manual "Checking ASN-domain
coverage of the MMDB" recipe into a maintainer script: walk every IPv4
record in the bundled IPinfo Lite MMDB, aggregate routed footprint per
as_domain, subtract mapped/known-unknown keys, apply PSL folding and
the full-IP privacy filter, and emit domain,ipv4_count,as_name sorted
by footprint for the collector -> classifier pipeline.

Because ASN registration data is self-declared to the RIRs and
as_domain derives from registrant-controlled WHOIS, bulk-categorizing
the MMDB needs poisoning defenses:

- An IPv4-footprint floor (--min-ips, default 4096, a /20) keeps tiny
  self-described ASNs out of the auto-classification queue; dropped
  counts are always printed.
- A brand-collision guard in classify_unknown_domains.py loads the
  existing map (--map) and demotes any single-category candidate whose
  proposed display name matches an existing map name without a lexical
  relationship to that operator's keys into the ambiguous bucket
  (marked name-collision-with-existing-map-entry) for human review.
  HAND overrides bypass the guard. The guard protects the PTR-side
  flow as well as the MMDB-coverage flow.

Verified: scan yields 132 candidates at the default floor (1512
dropped); collector accepts the output directly; a fixture titled as
Comcast under an unrelated domain lands in ambiguous while a
comcast-rooted sibling auto-promotes. Also fix the maps README links
that still pointed at the root AGENTS.md for the classification
workflow after its extraction to maps/AGENTS.md, and correct the
classify_tsv docstring's return signature.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Enhance planning guidance in CLAUDE.md by specifying auto mode activation after user approval

* Codify triage flagging for identified operators with no fitting type

An operator confidently identified from two corroborating sources but
matching none of the README's type values should be flagged during
triage with a proposed new type for the reviewer, not force-fitted and
not silently recorded as known-unknown — KU means "we couldn't
identify this", which would bury completed research. Extends workflow
rule 7 and the LLM low-confidence list in the maps AGENTS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 18:21:47 -04:00
Sean Whalen 64ddb13f18 Clarify introductory guidance in AGENTS.md by removing redundant phrasing 2026-07-16 17:10:15 -04:00
Sean Whalen 51cd3bfa45 Fix file path reference in AGENTS.md for clarity 2026-07-16 17:09:37 -04:00
Sean Whalen 1f1a0f0a0e Refactor AGENTS.md to remove redundant file paths for clarity 2026-07-16 16:54:55 -04:00
Sean Whalen eb7f11798b Remove examples in privacy rule regarding full IPv4 addresses in AGENTS.md 2026-07-16 16:48:31 -04:00
Sean WhalenandClaude Fable 5 0e31dcc0b8 Extract reverse-DNS map guidance to a nested AGENTS.md
Move the "Maintaining the reverse DNS maps" section (two-thirds of the
root AGENTS.md) to parsedmarc/resources/maps/AGENTS.md, next to the
files and tooling it governs. Agents that follow the AGENTS.md spec
read the nearest file in the directory tree; a one-line CLAUDE.md
import alongside it covers Claude Code, which loads subdirectory
context lazily when files under maps/ are first read. A pointer stub
remains in the root file for agents that only read the repo root.

The moved content is verbatim except for one cross-reference that now
points at the root AGENTS.md instead of "earlier in this file".

Exclude AGENTS.md/CLAUDE.md from hatch build targets so the new files
don't ship in the wheel, matching the existing README.md exclusion
(verified with a scratch wheel build).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:42:59 -04:00
df9bf82e04 Post-review follow-ups for Graph send (#825/#826) and requests-to-httpx migration (#827)
Follow-ups from the review of PR #825 (whose implementation had already
landed on master via #826's stacked merge):

- Honor the documented [smtp] attachment and [smtp] message options.
  Both were parsed into opts but never passed to either summary-email
  transport (also broken in released 10.2.2), so a configured custom
  attachment filename or message body was silently ignored. Both the
  SMTP and Microsoft Graph transports now receive them, and the missing
  smtp_attachment Namespace default is added (also covers SIGHUP
  reload, which rebuilds opts from the CLI Namespace).
- Don't mislabel non-Graph mailbox errors as Microsoft Graph failures:
  the shared mailbox-fetch and watch handlers now log a generic
  "Mailbox Error" with traceback when the connection isn't Graph.
- Declare microsoft-kiota-abstractions as a direct dependency (imported
  directly in cli.py for Graph error handling; previously transitive).

Migrate all runtime HTTP from requests to httpx (webhook client, Splunk
HEC client, and the PSL-overrides / IP-database / reverse-DNS-map /
IPinfo-API fetches in utils.py):

- follow_redirects=True everywhere to preserve requests' default
  redirect-following; httpx does not follow redirects by default.
- The PSL-overrides and reverse-DNS-map fetches gain a 60s timeout
  (previously none), matching the IP-database fetch.
- response.ok -> response.is_success; requests.RequestException ->
  httpx.HTTPError; raw string bodies use content= (httpx's data= is
  form-encoding only); Splunk HEC verification moves to client
  construction (httpx has no per-request verify).
- requests drops out of [project] dependencies and moves to the [build]
  extra for the out-of-wheel maintainer script collect_domain_info.py,
  which deliberately stays on requests/urllib3 for its permissive-TLS
  adapter.
- Remove the requests-era module-level
  urllib3.disable_warnings(InsecureRequestWarning) in splunk.py; httpx
  doesn't route through urllib3, so its only remaining effect was
  globally silencing insecure-TLS warnings from other urllib3-based
  components as an import side effect. Nothing imports urllib3 directly
  anymore, so it also leaves [project] dependencies.

Tests: config-to-transport wiring for attachment/message on both
transports (including defaults), non-Graph errors keep the generic log
line, webhook/Splunk payload assertions moved to content=, and Splunk
verify asserted at httpx.Client construction. 736 passed; ruff and
pyright clean.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 16:11:47 -04:00
31c928d6fc Refresh Microsoft Graph docs: national clouds, examples, troubleshooting (#826)
* Send report summary via Microsoft Graph; make Graph failures observable

Two related fixes shipped together:

Send via Graph: the periodic DMARC summary email can now be sent
through the already-authenticated Microsoft Graph mailbox connection
(MSGraphConnection.send_message(), /users/{mailbox}/sendMail) instead
of only SMTP. Triggered when [msgraph] is configured and [smtp] has a
`to` value but no `host` -- SMTP is always preferred when `host` is
set, with no automatic fallback to Graph on SMTP failure. Reuses the
same connection used for reading; no new send-only config mode.
email_results()'s SMTP behavior is unchanged; a new
email_results_via_msgraph() shares its content-building logic via a
new _build_report_email_content() helper. Graph's sendMail always
sends as the authenticated mailbox, so [smtp] from is ignored on this
path -- documented, along with the required Mail.Send permissions and
a caveat that delegated auth flows (UsernamePassword/DeviceCode) don't
currently request that scope, so app-only auth is the supported path
for sending. Tracks #472.

Observable Graph failures: MSGraphConnection construction, mailbox
fetch, message send, and --watch failures now catch
ClientAuthenticationError/APIError/httpx.HTTPError specifically and
log one clear ERROR line naming the mailbox, tenant, auth method, and
the Graph request-id/client-request-id when available, instead of a
bare "MS Graph Error"/"Mailbox Error" with no context. Full traceback
still preserved at --debug. --watch previously had no Graph-specific
error handling at all -- a Graph error there crashed with a raw
uncaught traceback; it now exits the same way as the other three
sites. No new config options.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Refresh Microsoft Graph docs: national clouds, examples, troubleshooting

The [msgraph] docs were accurate but missed guidance the community has
been asking for:

- graph_url now lists the actual national/sovereign-cloud endpoint
  values (GCC High, DoD, China/21Vianet), with an explicit warning
  that setting it alone is not sufficient -- the Entra ID auth
  endpoint isn't independently configurable in parsedmarc or
  mailsuite, so it always hits the global login.microsoftonline.com.
- A minimal working [msgraph] example for every auth method
  (UsernamePassword, DeviceCode, ClientSecret, Certificate,
  ClientAssertion) -- previously only Certificate had one, entangled
  with the SMTP-sending example.
- A reading-permission matrix alongside the existing sending one, so
  every auth method x own/shared-mailbox combination is explicit in
  one place for both directions.
- An accurate note on the parsedmarc-named token cache: it's a
  deliberate backward-compatibility choice from the 9.11.0 mailsuite
  extraction (mailsuite's own default cache name differs), not a
  migration users need to act on.
- A troubleshooting table for four error scenarios, verified against
  source rather than assumed: admin consent and folder-resolution
  failures are still live and documented with real fixes; the
  event-loop and ISO-timestamp errors are historical, already fixed
  below this project's dependency/version floor.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-14 13:34:56 -04:00
40509f801b Migrate Elasticsearch output to the elasticsearch-py 8.x client (#822)
* Migrate Elasticsearch output to the elasticsearch-py 8.x client (#806)

The mandatory elasticsearch<7.14.0 + elasticsearch-dsl==7.4.0 pins
transitively forced urllib3<2 (EOL 1.26.x) onto every install. The old
<7.14.0 cap only existed to dodge the client product check that broke
OpenSearch users (#452, #653) — obsolete now that parsedmarc has a
dedicated [opensearch] backend on opensearch-py.

- Depend on elasticsearch>=8.18,<9 and drop elasticsearch-dsl entirely
  (the DSL ships inside the client as elasticsearch.dsl since 8.18.0).
  The 8.x client's elastic-transport allows urllib3>=1.26.2,<3, so
  installs can now resolve urllib3 2.x. The 8.x line supports both
  Elasticsearch 8.x and 9.x servers; ES 7.x servers are no longer
  supported, and OpenSearch users pointing [elasticsearch] at an
  OpenSearch cluster must switch to the [opensearch] section.
- set_hosts() now builds 8.x connection kwargs (scheme-qualified host
  URLs, request_timeout, basic_auth) while keeping the function
  signature and every INI option unchanged.
- migrate_indexes() is now a documented no-op kept for API
  compatibility: its only migration (re-typing published_policy.fo
  from long to text) applied exclusively to indices carrying the
  legacy ES 6-era "doc" mapping type, which cannot exist on any
  server the 8.x client can reach.
- The elasticsearch.dsl 8.x stubs use dataclass_transform and don't
  surface pre-8.x-style bare `name = Text()` fields as constructor
  parameters; each Document/InnerDoc class now carries a
  TYPE_CHECKING-only `__init__(*args, **kwargs)` declaration matching
  the real runtime signature, which also made nine pre-existing
  pyright ignores unnecessary.

Verified with ruff, pyright (0 errors/0 warnings), the full pytest
suite (718 passed), and a CLI run over the bundled samples; CI's live
elasticsearch:8.19.7 service exercises the new client end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Use pass instead of ... in TYPE_CHECKING __init__ stubs

CodeQL flags an ellipsis-only body as "Statement has no effect" (12
alerts on PR #822); pass is equivalent at runtime and to the type
checker and keeps the alerts from resurfacing on every future scan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 13:50:25 -04:00
Sean Whalen a1fd66de04 Bump version to 10.2.2 10.2.2 2026-07-13 10:08:51 -04:00
01b85ab7de Fix aggregate-report timestamp skew on non-UTC hosts (#819) (#821)
record["interval_begin"]/["interval_end"] are UTC wall-clock strings
(produced in __init__.py by strftime() on a datetime already converted
via human_timestamp_to_datetime(..., to_utc=True)), but the
Elasticsearch and OpenSearch per-record save loops and the Splunk HEC
aggregate-report event builder re-parsed them without assume_utc=True,
so on non-UTC hosts they were misinterpreted as local time and shifted
by the host's UTC offset -- shifting the stored date_begin/date_end,
the daily/monthly index date, and the Splunk event time. Verified by
parsing a real sample report under TZ=Europe/Warsaw vs TZ=UTC and by
reproducing the exact skew via human_timestamp_to_unix_timestamp().

This is the same class of bug fixed for arrival_date_utc in #811/#812
(commit cdda5da); the assume_utc keyword already exists on
human_timestamp_to_datetime()/human_timestamp_to_unix_timestamp() and
is reused here rather than reimplemented.

The issue also proposed changes to the report-level begin_date/end_date
parses in elastic.py/opensearch.py (~line 455) and to postgres.py.
Both were investigated and left unchanged: the report-level strings
are genuinely host-local time (from timestamp_to_human() ->
datetime.fromtimestamp()), so their existing no-assume_utc round-trip
is already correct on a single host -- postgres.py already gets this
right via two distinct helpers (_naive_local_to_timestamptz vs
_ensure_utc_suffix). Adding assume_utc to the report-level parses
would introduce a skew rather than fix one.

Fixes https://github.com/domainaware/parsedmarc/issues/819

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 09:59:35 -04:00
f33951c0bb Fix host-dependent flakiness in testMissingEverythingRaisesFileNotFoundError (#820)
The system-path fallback list in _get_ip_database_path() checks real
absolute paths like /usr/share/GeoIP/GeoLite2-Country.mmdb. On a host
that actually has one installed there, the fallback succeeds and no
FileNotFoundError is raised, regardless of the test's mocked bundled
path -- the code is behaving correctly, the test just wasn't isolated
from the host filesystem. Patch os.path.exists to force every system
path to look absent so the test is host-independent.

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 09:32:32 -04:00
547e6d7113 chore: update IPinfo Lite MMDB (#818)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-13 09:17:41 -04:00
18b38c991d Add verify skill and document model roles in CLAUDE.md (#817)
* Raise test coverage: utils, elastic, and opensearch to 100%

Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).

New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.

Bugs found by the new tests, fixed in the same PR per the testing
standards:

- parse_email() crashed with KeyError: 'Headers' on messages whose
  From header is present but unparseable (e.g. a bare "From:" line):
  the fallback read parsed_email["Headers"], but the parsed headers
  are stored under lowercase "headers" (assigned a few lines up in
  the same function), so the key never exists. At the CLI surface
  this made any failure report whose embedded sample had an empty
  From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
  when the probe could not reach the API, contradicting its own
  docstring ("other errors are logged and the token is still
  accepted"): _ipinfo_api_lookup() returns None on network errors
  instead of raising, so the probe's exception handler was
  unreachable. The probe now checks the lookup result and warns on
  failure; 401/403 still raises InvalidIPinfoAPIKey.

Dead code deleted rather than padded with tests:

- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
  (the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
  both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
  re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
  input branch assigns file_object or raises first (confirmed by
  pyright narrowing with the guard removed).

Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Use assertGreater for the reverse-DNS-map fallback size check

Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add verification claude skill for parsedmarc CLI usage and sample inputs

* Document model roles for feature work in CLAUDE.md

Codifies the plan-with-Fable / implement-with-Sonnet / review-with-Fable
split (Opus as fallback) for feature work and PR reviews.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 09:33:57 -04:00
746da77de5 Raise test coverage: utils.py, elastic.py, and opensearch.py to 100% (#816)
* Raise test coverage: utils, elastic, and opensearch to 100%

Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).

New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.

Bugs found by the new tests, fixed in the same PR per the testing
standards:

- parse_email() crashed with KeyError: 'Headers' on messages whose
  From header is present but unparseable (e.g. a bare "From:" line):
  the fallback read parsed_email["Headers"], but the parsed headers
  are stored under lowercase "headers" (assigned a few lines up in
  the same function), so the key never exists. At the CLI surface
  this made any failure report whose embedded sample had an empty
  From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
  when the probe could not reach the API, contradicting its own
  docstring ("other errors are logged and the token is still
  accepted"): _ipinfo_api_lookup() returns None on network errors
  instead of raising, so the probe's exception handler was
  unreachable. The probe now checks the lookup result and warns on
  failure; 401/403 still raises InvalidIPinfoAPIKey.

Dead code deleted rather than padded with tests:

- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
  (the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
  both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
  re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
  input branch assigns file_object or raises first (confirmed by
  pyright narrowing with the guard removed).

Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Use assertGreater for the reverse-DNS-map fallback size check

Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 18:21:04 -04:00
Sean Whalen a1da7b3420 Bump version to 10.2.1 10.2.1 2026-07-09 20:50:45 -04:00
3fda55d385 Make Microsoft Graph connection activity observable (#815)
* Make Microsoft Graph connection activity observable

parsedmarc only configured its own logger, so all Graph connection
activity was silently dropped even with --debug: the mailbox layer
logs under mailsuite.mailbox.graph, token acquisition under
azure.identity (including the AADSTS error codes that distinguish a
local config problem from an Exchange Online / Entra ID one), and
HTTP traffic under httpx/msgraph — none of which had a handler or
level set. _main() also logged nothing around the MSGraphConnection
call, so a hang left no trace at all.

Three changes, all parsedmarc-side (no mailsuite changes needed):

- Log a redacted connection summary at INFO before connecting (auth
  method, tenant ID, client ID, mailbox, Graph URL) plus a --debug
  detail line with certificate path, token-file path, and set/not-set
  flags for secrets. Secret values are never logged; a regression
  test asserts they don't appear in captured output.
- Log a timing line after the connection object is initialized.
- Propagate parsedmarc's --verbose/--debug level and handlers to the
  dependency loggers (mailsuite, azure, msgraph, httpx, httpcore) via
  _configure_dependency_logging(), synced to exactly the parsedmarc
  logger's handlers so SIGHUP log-file swaps neither duplicate output
  nor write to closed handlers. At the default level dependency
  loggers sit at WARNING, so their warnings keep surfacing (formatted)
  without new noise.

All four new tests fail on the unfixed code (verified by stashing the
cli.py change).

Fixes https://github.com/domainaware/parsedmarc/issues/814.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Disable propagation on dependency loggers; document kiota's absence

Set propagate=False on the dependency loggers when syncing handlers, so
a stray logging.basicConfig() anywhere in the process cannot
double-print every dependency record through the root logger — the
function already owns these loggers' handler lists, and this makes that
ownership complete. Asserted alongside the existing level/handler checks.

kiota_http and its sibling packages were considered for
_DEPENDENCY_LOGGERS but verified to not use Python logging at all
(their observability is OpenTelemetry tracing), so a comment now
records why they are absent rather than leaving the omission to be
"fixed" later.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
2026-07-09 20:46:39 -04:00
7fed72798a Stop system GeoIP files from shadowing the bundled IPinfo database (#813)
* Stop system GeoIP files from shadowing the bundled IPinfo database

_get_ip_database_path() searched well-known system paths (including
/usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before
the database parsedmarc manages, so on any host with a distro GeoIP
package installed every lookup silently used a country-only — and
often years-old — database instead of the bundled IPinfo Lite one.
That disabled ASN enrichment entirely (asn/as_name/as_domain were None
for every IP) and with it the ASN-fallback path into the reverse-DNS
map, with no signal beyond a generic "IP database is more than a
month old" warning. Verified live on a Fedora host whose distro
GeoLite2-Country.mmdb dated to December 2019.

New precedence: explicit ip_db_path -> _IP_DB_PATH selected by
load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system
paths as a true last resort (only consulted when the bundled data
file is missing). The selected file is logged at debug level so a
--debug run shows which database answered.

The automatic system-path pickup was documented behavior, so
installation.md now tells MaxMind GeoLite2 users to set ip_db_path
explicitly, with a migration note.

Both new regression tests reproduce the shadowing portably via a decoy
CWD GeoLite2-Country.mmdb (the fallback list includes relative names),
and fail on the unfixed code (verified by stashing the source change).

Fixes https://github.com/domainaware/parsedmarc/issues/810.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: accurate fallback comment, dedup selection log, cover fallback tiers

- Correct the nothing-found-anywhere comment: the os.stat() age check
  raises FileNotFoundError before the caller's open_database() would.
- Log "Using IP database at ..." only when the selected path changes
  instead of on every uncached IP lookup, so --debug runs over large
  batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in
  the test fixture for order-independence.
- Cover the previously untested branches of _get_ip_database_path:
  system-path fallback when the bundled database is missing, the
  FileNotFoundError when nothing exists anywhere, the stale-database
  warning, and the log-once-per-path behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
2026-07-09 20:07:00 -04:00
cdda5dae62 Fix failure-report timestamp skew on non-UTC hosts in ES/OpenSearch/Splunk outputs (#812)
* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks

arrival_date_utc is a UTC wall-clock string (generated in
parse_failure_report via an aware-UTC strftime), but elastic.py,
opensearch.py, and splunk.py parsed it back into a naive datetime and
called .timestamp(), which per the Python docs interprets naive values
as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp).
On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date
field, used in the failure-report dedup match query, and sent as the
Splunk HEC event time was therefore off by the host's UTC offset
(verified -3600 s under TZ=Europe/Warsaw in January).

Add an assume_utc keyword to human_timestamp_to_datetime() /
human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive
parses, and use it at the three arrival_date_utc call sites. Aware
inputs (explicit offsets) are unaffected; all other callers keep the
existing local-time semantics, whose round-trip with timestamp_to_human
is self-consistent on a single host (the broader local-time output
question is tracked separately in issue #811 bug 2).

The three new sink regression tests fail on the unfixed code
(verified by stashing the source changes) and force TZ=Europe/Warsaw
via time.tzset() so they catch the skew even on UTC CI runners.

Fixes half of https://github.com/domainaware/parsedmarc/issues/811.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring

Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from
the four timestamp regression tests into a shared tests/tzutil.py
force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s
docstring, which said the return type was float while the function
returns int.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
2026-07-09 19:45:34 -04:00
Sean Whalen 15dd69cc28 fix: update virtual environment setup to use python3 -m venv 2026-07-09 19:42:10 -04:00
fa8faa6b39 MS Graph: case-insensitive auth_method + ClientAssertion support (#809)
* draft fix MS Graph

* Add ClientAssertion auth method support for MS Graph in cli.py

MSGraphConnection/AuthMethod (via mailsuite) already supported
ClientAssertion, but cli.py never parsed a client_assertion config
value or passed it through, so it was unusable from the CLI. Wire
config_msgraph.client_assertion through _parse_config and the
MSGraphConnection call, and document the auth method (including its
short-lived-JWT caveat vs. Certificate/ClientSecret for watch mode).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 13:46:36 -04:00
Ivan The GeekandGitHub a6241d537e Fix grammatical error in project maintenance note (#805)
The sponsors note read "This is a project is maintained by one
developer" in both README.md and docs/source/index.md; drop the
stray "is a".
2026-07-06 09:33:14 -04:00
25638cc3cd chore: update IPinfo Lite MMDB (#807)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-06 09:29:52 -04:00
2547422f40 chore: update IPinfo Lite MMDB (#804)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-29 12:54:42 -04:00
Sean WhalenandGitHub c423b8dfff Modernize type hints to PEP 585 / PEP 604 syntax (#803) 10.2.0 2026-06-25 15:21:48 -04:00
Sean WhalenandClaude Opus 4.8 d13eb86782 Advertise supported Python versions via trove classifiers
requires-python stays ">=3.10" (already correct and matching the CI matrix
of 3.10-3.14); add the per-version Programming Language :: Python :: 3.10-3.14
classifiers and "3 :: Only" so the PyPI page lists supported versions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 15:07:50 -04:00
a67e8d3ebc 10.2.0 - Explain why a report is invalid instead of "Not a valid report" (#802)
* Explain why a report is invalid instead of "Not a valid report"

The parser catches broadly so one malformed report can't crash a batch,
but every failure surfaced as the generic ParserError("Not a valid
report"), telling operators nothing about the cause.

parse_report_file() now keeps each format parser's specific error as it
tries aggregate XML -> SMTP TLS JSON -> report email, and when all three
reject the input it content-sniffs the leading byte to surface the single
relevant reason (e.g. "Invalid aggregate report: Missing field:
'org_name'", or "Not a recognized report format (...)"). The CLI already
logs str(error), so this reaches the user with no cli.py change.

Every parser catch site also re-raises with `raise ... from <original>`,
preserving the underlying ExpatError / JSONDecodeError / KeyError /
archive errors on __cause__ for library callers and tracebacks. The same
exception *types* are still raised.

Finally, the catch-all "unexpected error" branches append
`(raised at <file>:<line>)` from the deepest traceback frame, but only
when the parsedmarc logger is at DEBUG level (e.g. the CLI's --debug);
normal-level output is unchanged.

Bumps the in-progress version to 10.2.0 and documents all three in the
CHANGELOG.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Cover the failure-report path in parse_report_file error tests

The reason-surfacing tests covered the aggregate and SMTP TLS branches but
not failure reports, which reach parse_report_file only via the email
path. Add a malformed multipart/report failure email (missing the required
Source-IP) and assert the message names the failure format and the missing
field rather than collapsing to "Not a valid report".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Cover the new error-reporting lines; drop one unreachable catch

Bring the lines added by this PR to full test coverage:

- Test _exc_origin() with a never-raised exception (no __traceback__) so the
  "no frames" guard is exercised.
- Test _parse_smtp_tls_failure_details() with a non-dict, which raises
  TypeError (not KeyError) and exercises the generic catch-all.
- Test parse_report_email() with an unparseable Date header, which trips the
  initial mail-parse catch-all and becomes a ParserError.

Two dead lines are removed rather than hidden, per the project's "delete
unreachable branches, no # pragma: no cover" rule:

- _looks_like_email() looped with a `continue` for blank lines, but every
  caller passes lstrip()-ed text, so the first line is never blank. Simplified
  to inspect the first line directly.
- parse_report_email()'s `except Exception` after `except InvalidFailureReport`
  was unreachable: parse_failure_report wraps its entire body and provably
  raises only InvalidFailureReport, which the preceding handler already catches.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Fix IndexError when backfilling envelope_from from SPF results

_parse_report_record() backfills a missing/empty envelope_from from the
last SPF auth result's domain. The "envelope_from is None" branch gated on
the raw auth_results["spf"] list but indexed the filtered
new_record["auth_results"]["spf"] list, which only holds results that have
a domain. A reporter sending an SPF result with no domain made the filtered
list empty while the raw list was non-empty, so [-1] raised IndexError and
the whole record failed to parse.

The two near-identical envelope_from backfill branches (missing identifier
vs. empty identifier) drifted apart -- only one was updated when the
filtered new_record list was introduced -- which is what let them disagree
on which list to read. Merge them into a single path, keyed on
dict.get("envelope_from") is None, that gates and indexes the same raw list
with the "domain" membership guard the missing-identifier branch already
used.

Regression test: envelope_from=None with an SPF result carrying no domain
now parses to envelope_from=None instead of raising. This is the bug that
motivated the surrounding error-reporting work in this PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Trim comment

* Cover the touched error branches; drop a dead UnicodeDecodeError catch

Codecov flagged the pre-existing error branches this PR touched (adding
`from e` / `_exc_origin`) as changed-and-uncovered. Most are real
malformed-report paths, so add honest tests that drive them with realistic
inputs:

- parse_smtp_tls_report_json: nested missing key (date-range without
  start-datetime) -> InvalidSMTPTLSReport chaining a KeyError.
- parse_aggregate_report_xml: non-structured report_metadata -> the
  AttributeError branch ("Report missing required section").
- parse_report_email: valid legacy text/plain failure report (success path),
  a text report missing its fields, a base64 attachment of malformed
  aggregate XML, and one of invalid SMTP TLS JSON.
- parse_report_file: gzipped junk -> the str branch of the content sniff.

The `except UnicodeDecodeError` in extract_report is removed as dead code
(no `# pragma: no cover`, per the repo rule): str-mode streams are already
rejected by explicit isinstance checks, and every decode() uses
errors="ignore", so it can never fire. str-mode still raises ParserError.

Also rename the two new failure-report tests from "Forensic" to "Failure"
and add an AGENTS.md rule: RUF reports are "failure reports"; "forensic" is
reserved for the literal backward-compat alias identifiers only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Fix pyright errors in the new error-branch tests

CI runs pyright over the whole repo (tests included); these slipped through
because the local check only covered parsedmarc/__init__.py:

- _parse_smtp_tls_failure_details("not a dict") is a deliberate wrong-type
  test -> targeted `# pyright: ignore[reportArgumentType]`.
- result["report"]["source"]["ip_address"] on a ParsedReport TypedDict ->
  cast(FailureReport, result["report"]) first, matching existing tests.

This is what failed lint-docs-build (and, since `test` needs it, skipped the
Codecov upload) on the prior commits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 14:53:56 -04:00
bc17e6afb2 chore: update IPinfo Lite MMDB (#801)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-23 13:46:35 -04:00
d40447ea91 chore: update IPinfo Lite MMDB (#800)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-15 08:54:43 -04:00
Sean Whalen 8275665e11 10.1.1
- Require `mailsuite>=2.2.2.2` to  honor config reloading in the IMAP `IDLE` loop
10.1.1
2026-06-13 20:46:57 -04:00
8337b67351 Cover both arms of the optional psycopg import in postgres.py (#799)
The module-level try/except import is environment-dependent: with
psycopg installed the ImportError fallback never runs, and without it
(CI's test job) the successful-import arm never completes — so Codecov
flags one side or the other no matter where coverage is measured (it
flagged the import line on master right after #798 merged).

Exercise both arms explicitly: execute the module's source into a
fresh, throwaway module object (importlib.util.module_from_spec +
exec_module) under a patched sys.modules — a None entry forces
ImportError, fake module entries force the success path — and assert
on the psycopg / psycopg_json bindings each arm produces. The
throwaway-module approach (rather than importlib.reload) leaves the
canonical parsedmarc.postgres untouched, so the identity of
PostgreSQLError / AlreadySaved held by the rest of the test module is
preserved.

Verified covered in both environments: with the venv's real psycopg,
and with psycopg hidden via a PYTHONPATH shim to simulate CI; the
import block reports no missing lines either way.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
10.1.0
2026-06-12 21:48:20 -04:00
eaeea4f53d Make the whole codebase pass pyright cleanly and enforce it in CI (#798)
* Make the whole codebase pass pyright cleanly and enforce it in CI

Fix all 102 pyright (1.1.410, standard mode) errors across the library,
tests, and maps scripts, then pin and enforce the zero-errors bar:

- postgres.py: make the optional psycopg import TYPE_CHECKING-aware so
  the module is properly typed while keeping the runtime install-hint
  fallback; import psycopg.types.json explicitly as psycopg_json (the
  old psycopg_types.json attribute access only worked because psycopg
  imports the submodule eagerly); have _connect()/_ensure_connected()
  return the live connection so save methods use a non-Optional local;
  type the DDL list as list[LiteralString] to match psycopg's execute()
  overloads.
- kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error
  fallback statically via TYPE_CHECKING (kafka-python 3.0 removed
  NoBrokersAvailable), which also fixes _BootstrapError's import
  resolution in tests.
- syslog.py: go through getattr/setattr for SysLogHandler.socket
  (absent from typeshed); type the save_* methods with the report
  TypedDicts (single or list, matching cli.py call sites — gelf.py gets
  the same signatures); raise ValueError when retry_attempts < 1
  instead of falling through and registering a None handler (bug fix,
  with a regression test and a CHANGELOG entry).
- elastic.py / opensearch.py: human_result params are Optional[str].
- maps scripts: sort_csv declared a return type but never returned
  (now -> None); seen_sort_field_values was possibly unbound;
  convert_to_utf8's src_encoding is Optional[str].
- tests: cast sample-report dict helpers to their TypedDicts; mark
  deliberate wrong-type calls with targeted pyright ignores; add
  narrowing asserts for Optional results; access the mocked
  KafkaProducer through a cast helper; match the mailsuite
  fetch_message base signature (**kwargs); patch the renamed
  parsedmarc.postgres.psycopg_json in test_postgres's setUpModule.

Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc,
tests, docs; standard mode), pyright==1.1.410 pinned in the [build]
extra (pinned exactly so a new pyright release can't break CI without a
code change), and a "Check types" step in the lint CI job — which now
also runs ruff format --check and installs the [postgresql] extra so
the optional psycopg import resolves. Documented in AGENTS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Set session headers via update() instead of replacing the dict

requests 2.34 ships inline type annotations, and Session.headers is a
CaseInsensitiveDict[str] — assigning a plain dict fails pyright there
(the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid
it). headers.update() is correctly typed against both versions, and is
the documented requests idiom: it overrides User-Agent and the
client-specific headers while keeping the session's defaults
(Accept-Encoding, Connection) instead of wiping them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 21:33:01 -04:00
0c456d44ed Declare backward-compatible method aliases inside class bodies (#797)
* Declare backward-compatible method aliases inside class bodies

Assigning the legacy save_forensic_* aliases onto the classes after the
class body (KafkaClient.save_forensic_reports_to_kafka = ...) is invisible
to static type checkers, so Pylance/Pyright flagged every assignment and
every use with reportAttributeAccessIssue. Declaring the alias inside the
class body is statically visible — the IDE errors disappear and the
aliases get autocomplete and proper typing. Runtime behavior is identical
(same function object bound as a method), guarded by the existing
assertIs alias tests, whose type-ignore comments are now unnecessary.

Also add a pyright ignore on the NoBrokersAvailable import in
kafkaclient.py: the import is guarded by try/except ImportError for
kafka-python 2.x, but Pyright resolves against the installed 3.x where
the name no longer exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Bump version to 10.1.0

10.0.4 is tagged and released; CHANGELOG.md already documents the
in-progress 10.1.0 section that this release will ship.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:50:47 -04:00
ebc6a55715 Switch from kafka-python-ng to kafka-python>=2.3.2 (#795) (#796)
kafka-python-ng is archived and vulnerable to CVE-2026-10142 and
CVE-2026-10143, both fixed in upstream kafka-python 2.3.2.

kafka-python 3.0 removed the NoBrokersAvailable exception (a failed
producer bootstrap now raises KafkaTimeoutError), so kafkaclient.py
imports whichever the installed version provides via a compat shim,
keeping the >=2.3.2 range honest for both 2.x and 3.x. Verified against
kafka-python 3.0.0 (full test suite) and 2.3.2 (import shim resolution).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:25:35 -04:00
Sean Whalen a4be378e30 Update the changelog 2026-06-12 20:14:02 -04:00
d3510da3a6 feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI (#794)
* feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI

Previously SIGTERM (systemctl stop, docker stop, Kubernetes pod termination)
killed parsedmarc mid-batch, tearing output writes and silently dropping
buffered Kafka records. Shutdown is now cooperative:

- SIGTERM/SIGINT set a flag that is polled at safe boundaries. The one-shot
  CLI checks it between batches; watch mode passes it as `config_reloading` so
  the mailbox backend -- including the IMAP IDLE loop -- returns once the
  current batch is fully processed. Either way the in-flight batch and its
  output writes finish before the process exits 0.
- Ctrl-C is a double-tap: the first press is graceful, the second
  short-circuits to os._exit(130).
- Output clients are now closed on every exit path (atexit plus a trailing
  close in _main), fixing a long-standing leak where one-shot runs and
  graceful shutdowns never flushed Kafka / closed Elasticsearch / S3 / etc.

Docs: the example systemd unit gains KillSignal=SIGTERM and TimeoutStopSec=60
(keep it above mailbox_check_timeout). Tests cover watch shutdown, the one-shot
between-batch stop, the SIGINT double-tap, and the output-client-close leak.

* test: cover the one-shot mbox-loop shutdown break

Extend the one-shot SIGTERM test to also pass an .mbox path so a single
run exercises both shutdown checkpoints: the file-batch loop break and the
subsequent mbox loop break (which Codecov flagged as the only uncovered
lines on PR #794). is_mbox is keyed by suffix and get_dmarc_reports_from_mbox
is asserted not called, since the mbox loop breaks before reaching it.

* test: narrow signal.getsignal() return before invoking in SIGINT test

signal.getsignal() is typed Callable | int | Handlers | None; calling it
directly fails pyright's callable check. Assert callable() first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:00:32 -04:00
b869235224 Build multi-arch (amd64+arm64) Docker images with PostgreSQL support (#793)
* Build multi-arch Docker images with PostgreSQL support

The prebuilt image now installs the `[postgresql]` extra, so the optional
PostgreSQL output backend (psycopg) works out of the box in the container
without a separate `pip install` (#792). The wheel path is resolved into a
variable before appending the extra so the shell doesn't treat
`*.whl[postgresql]` as a bracket glob.

The build workflow now sets up QEMU + Buildx and builds a multi-arch
manifest for `linux/amd64` and `linux/arm64`, so the image runs natively on
64-bit ARM hosts such as a Raspberry Pi (#789). Every compiled dependency
(psycopg[binary], lxml, maxminddb, cryptography) ships prebuilt aarch64
manylinux wheels, so the arm64 build adds no source-compilation step.

A `pull_request` trigger (scoped to the build inputs) and `workflow_dispatch`
are added so the multi-arch build can be validated on PRs and rebuilt on
demand; pushes are still gated on the release event, so neither pushes images.

Closes #789
Closes #792

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Bump version to 10.0.4 to publish the new images

The docker workflow only pushes to the registry on a `release` event, so
shipping the multi-arch + PostgreSQL-enabled image requires cutting a
release. 10.0.3 is already tagged, so bump to 10.0.4 and document the
Docker changes in the changelog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Don't run the docker build on pull requests

The pull_request trigger (added to validate the multi-arch build) re-ran the
full ~10-minute amd64+arm64 build on every commit pushed to a docker-touching
PR, because the pull_request `paths` filter matches against the PR's entire
diff, not just the newest commit. That is wasteful once the build has been
validated.

Drop the pull_request trigger and rely on workflow_dispatch for on-demand
validation (plus the existing master-push and release triggers). Also gate the
registry login on the release event so that no non-release run authenticates
to ghcr at all — a build can only ever be pushed from a published release.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
10.0.4
2026-06-08 17:42:00 -04:00
8ec3cc9ce8 chore: update IPinfo Lite MMDB (#791)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-08 17:02:09 -04:00
ec0a313669 chore: update IPinfo Lite MMDB (#788)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-01 13:26:09 -04:00
f65eebc4d8 chore: update IPinfo Lite MMDB (#787)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-05-26 14:37:59 -04:00
08db305e5a test: cover no-display-name Reply-To header flattening (#786)
The 10.0.3 Reply-To header flattening (elastic.py / opensearch.py line 711)
has two branches: display-name present ("Name <addr>") and absent (bare
address). The existing test only exercised the former, leaving the
empty-display-name branch uncovered — the two lines Codecov flagged on the
10.0.3 patch. Add a failure report whose Reply-To has no display name and
assert sample.headers["reply-to"] flattens to the bare address.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 14:12:24 -04:00
e104f1118c Land 10.0.3 changes on master (#785)
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.

This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
10.0.3
2026-05-24 13:54:40 -04:00
2c8b2c0f14 Bump mailsuite to >=2.2.1 (release 10.0.2) (#783)
* Bump mailsuite to >=2.2.1; release 10.0.2

mailsuite 2.2.1 raises the transitive mail-parser floor to >=4.2.1, which
stops mail-parser from returning a phantom ('', '') entry for absent address
headers (verified against samples/failure/* with mail-parser 4.2.1: cc/bcc
now parse to [] instead of [{address: ""}]). parsedmarc reads the mail-parser
object directly via its own parse_email(), so this previously caused an empty
{address: ""} Cc/Bcc entry to be indexed for every failure-report sample in
Elasticsearch/OpenSearch and emitted in JSON/S3/Kafka output.

The Reply-To-always-empty behavior in parsedmarc's own parse_email() (a
hyphen-vs-underscore key mismatch, not an upstream issue) and the failure
dashboards are out of scope here and tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: note CVE-2023-27043 hardening from mail-parser 4.2.1 in 10.0.2

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:57:50 -04:00
Sean Whalen 3f64e30f6f Update version to 10.0.1 and bump mailsuite requirement to >=2.2.0 2026-05-23 22:08:34 -04:00
Sean Whalen 9e675bf43c Update build command to use pytest with coverage reporting 10.0.0 2026-05-21 17:15:29 -04:00