amorfo77/parsedmarc
1
0 Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-11-06 16:59:23 +00:00
Code Issues Projects Releases Wiki Activity
1,678 Commits 6 Branches 71 Tags
48ea47adee7ed2f014d770b79d742fde7bb03098
Commit Graph
17 Commits
Author SHA1 Message Date
Sean WhalenClaude Opus 5.5Sean Whalen
 		48ea47adee Merge commit from fork 
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv)

With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init
lines and opensearch.set_hosts() logged configured URLs verbatim, so
userinfo passwords reached the console and log_file (CWE-532).

Triage found more paths of the same class:
- the httpx "HTTP Request:" line, forwarded at --verbose/--debug,
  contains the full URL (str(httpx.URL) keeps userinfo and query), so
  webhook and HEC URL credentials and the IPinfo ?token= were logged
  on every request;
- the PSL overrides, IP database and reverse DNS map downloaders logged
  their URL at DEBUG and str(HTTPStatusError), which embeds the full
  URL, at WARNING;
- libpq echoes the DSN or a password fragment for some malformed
  PostgreSQL connection strings, logged as an output-client error.

Add redact_url_credentials() (string methods only; urlsplit misparses
scheme-less "user:pw@host:port") and use it at each log site; add a
filter on the httpx logger that masks userinfo and the token query
value; build download-failure warnings from the status code and the
redacted request URL instead of the exception text; replace psycopg
ProgrammingError detail from a connection_string with a fixed message.
Docs now recommend separate user/password settings over URL
credentials.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Mask credentials in the Loki URL in the log and LokiError text

The Loki output (#926) landed on master after this fix was written. Its
url keeps userinfo (copy_with(path=...) preserves it), and the --debug
"Initializing Loki client" line and both LokiError messages, which the
CLI logs on a failed push, included it as-is: the same CWE-532 path as
the Splunk HEC line. Mask it with redact_url_credentials() at all three
sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers

Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532):

- httpx.InvalidURL quotes the component it rejected, so a password with
  an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there)
  leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats
  it. Loki, Splunk HEC, the webhook and the three download loaders now
  report a fixed "details withheld" message raised from None.
  InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url
  crashed startup; the loaders now treat it as a download failure.
- redact_url_credentials() masks through the last "@" when the authority
  has no "@" but one follows it, so the --debug lines no longer print
  such passwords; the userinfo lookup is shared with the new nameserver
  masking.
- dnspython 2.9 DoH logs its request line on the httpx2 logger, which was
  forwarded but not filtered; the filter now covers httpx2 and
  httpx2.URL.
- [general] nameservers / --nameservers credentials are masked in the
  DNS pre-flight error, reverse DNS debug errors and per-report errors,
  and unusable https:// / tls:// ports are rejected before any query.
- Docs recommend token/username settings for [loki] and [splunk_hec];
  AGENTS.md gains rules for keeping credentials out of logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
 2026-10-31 17:14:00 -04:00
Sean WhalenandClaude Opus 5.5 70f96d0734 Fix DNS over HTTPS under dnspython 2.9 and the misleading pre-flight error (#935) (#936) 
* Fix DNS over HTTPS under dnspython 2.9 and the pre-flight error (#935)

dnspython 2.9 moved its [doh] extra from httpx to httpx2, and
dns.query.https() rejects a session of any other class before sending
anything (dnspython 2.9.1 dns/query.py:551: `if session and not
isinstance(session, httpx2.Client): raise ValueError("session parameter
must be an httpx2.Client")`; 2.8.0 checks httpx.Client at :566).
_get_doh_session() always built an httpx.Client, so every https://
nameserver failed and the resolver retried it until LifetimeTimeout.

The shared DoH client is now built from the Client class of the module
dnspython itself imported (dns.query.httpx2 on 2.9+, dns.query.httpx on
2.7/2.8), so it matches dnspython's isinstance check even when httpx2 is
installed beside dnspython 2.8. httpx2 keeps httpx's
`allow_env_proxies = trust_env and transport is None` (_client.py:658),
so the #880 proxy behavior carries over.

The nameservers pre-flight check called get_reverse_dns(), which turns
every DNS error into None, so the failure above was reported as "no PTR
record for 1.1.1.1". It now calls query_dns() directly and reports
"no PTR record" only for NXDOMAIN/NoAnswer, and the resolver's actual
error otherwise.

Reported by florianutz.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review of the DNS over HTTPS fix (#935)

- Add the httpx2 and httpcore2 loggers to the dependency loggers so
  --debug shows DoH traffic under dnspython 2.9+.
- Report an invalid dns_test_address by name and value in the
  pre-flight check instead of "Text input is malformed.", without
  attempting a query.
- Rewrap an over-long docstring line in _nameservers_to_resolver_input.
- Soften the test docstring about the no-[doh]-extra fallback.
- Correct the pyproject comment on what the [doh] extra installs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rewrap the dnspython [doh] comment in pyproject.toml (#935)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-31 15:16:26 -04:00
Sean WhalenandClaude Opus 5.5 6b7c203126 Merge commit from fork 
* Fix three security advisories (DOCTYPE amplification, poison mailbox message, byte-length filenames)

GHSA-m47j-wp78-fqjg: reject any DOCTYPE in aggregate report XML. An
internal subset's <!ATTLIST> default attributes are applied by expat to
every matching element, so 64 KB of XML (K=1000, N=10000) took 4.0 s and
1.4 GB here. xmltodict now gets an expat shim whose parsers raise on
StartDoctypeDecl; the lxml recovery path checks docinfo and serializes
only the root element.

GHSA-qxwv-2vwj-v828: parse_report_email() now raises only ParserError
subclasses (HeaderWriteError / RecursionError from re-serializing a part
used to escape). Content errors raised while fetching a message
(email.errors.MessageError, RecursionError) are returned as ParserError
and the message is disposed of as invalid; transport errors still
propagate. A failed invalid-message move/delete is logged instead of
aborting the batch, because mailsuite's Maildir move_message()
re-serializes the message too. mbox reads use get_bytes() instead of the
re-serializing get_string().

GHSA-j53g-34vj-g25q: get_filename_safe_string() bounds names to 100
UTF-8 bytes as well as 100 characters; filesystems limit a name to 255
bytes, so 84 CJK characters + ".eml" failed with ENAMETOOLONG.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review: recursion margins, mbox wording, DOCTYPE check order

- Nest the RecursionError poison 400 levels deep (POISON_NESTING_LEVELS)
  instead of 250: the smallest failing depth is ~247 from a shallow
  stack, 245 in a fork worker and ~222 at +100 frames, so 250 left a
  margin of a few levels. The shallow case uses 50 levels and now asserts
  InvalidDMARCReport ("is not a valid report"); the deep case asserts
  "recursion" in the message.
- Poison-loop tests skip a benign poison per message (continue) and
  assert at least one poison was exercised, so they cannot pass vacuously.
- lxml recovery: check for a DOCTYPE before serializing the root, and
  comment why the str() around tostring(encoding="unicode") stays
  (pyright, no lxml stubs).
- CHANGELOG: describe what the walk actually serializes, drop the
  depth-dependent 245 figure, and say plainly that an mbox run still
  stops at such a message, now with a ParserError naming it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Make the Maildir poison-message test interpreter-independent

Whether MaildirConnection.move_message() can store the control-character
message depends on the interpreter: upstream CPython (3.12.9) runs the
CVE-2024-6923 generated-header check only in the text Generator used by
as_string(), so mailbox.Maildir.add()'s BytesGenerator stores it, while
Ubuntu's patched 3.12.3 also checks in BytesGenerator._write_headers and
the move fails. The deeply nested message fails the move everywhere.

The test now decides per message, by trying the same Maildir round trip,
whether the message lands in Archive/Invalid with no mailbox error logged
or stays in INBOX with the move error logged; both branches assert the
batch completed with the aggregate archived. CHANGELOG and the
_dispose_invalid_message docstring now say which messages stay put.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 22:05:36 -04:00
Sean WhalenandClaude Fable 5.1 16a9016e2a Neutralize CSV formulas and escape control characters in logged strings (#931) 
* Neutralize CSV formulas and escape control characters in logged strings

CSV fields starting with =, +, -, @, tab, or CR get a leading apostrophe
(OWASP CSV Injection, CWE-1236). Sender-controlled subject/From and a few
aggregate-parser values are passed through the new sanitize_for_log()
before they reach error messages and logs.

Closes #929

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Sanitize more sender-controlled log values; fix sanitize_for_log docs

Escape the SMTP TLS policy-type in its error and org_name/report_id in the
duplicate-report debug lines, correct the sanitize_for_log docstring, and
drop the unreachable return in parse_report_email().

Closes #929

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Test the From-header escaping with a raw control character

mail-parser 4.8.0, which CI resolves, no longer decodes an RFC 2047
encoded word in the From header (4.6.2 did), so the previous test saw the
encoded word rather than the CR LF it was meant to observe. A raw ESC in
the display name survives both versions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-26 21:34:23 -04:00
Sean WhalenandClaude Fable 5.1 14aa287dc1 Fix ReDoS in failure-report, schema-tag, and timestamp regexes (11.0.4) (#921) 
Three regular expressions that run on attacker-controlled input
backtracked super-linearly (CWE-1333):

GHSA-7grx-mj6g-qxmc (affects 8.2.0 through 11.0.3, reported by
mjatin2007): text_report_regex, used by the Exim/cPanel plain-text
failure-report fallback, was \s*([a-zA-Z\s]+):\s(.+). The leading \s*
and the \s inside the class overlap and the pattern is unanchored, so a
whitespace run is retried from every offset and every split: cubic,
about 8x per doubling (400 spaces 0.06 s, 1600 spaces 3.3 s on CPython
3.12). A ~100 KB message would stall the watch loop for days and be
re-parsed after every restart. The pattern is now anchored to a line
start with disjoint classes.

GHSA-4h9r-c435-j834 (affects 4.1.7 through 11.0.3, found while auditing
every regex in the package): xml_schema_regex was </??xs:schema.*>,
whose greedy .*> rescans to end of line for every <xs:schema
occurrence: quadratic, 1 MB in 19 s. It runs on every attachment that
decodes to "<" after extract_report() has inflated it up to the 100 MiB
cap, which 194 KB of gzip reaches. Now </?xs:schema[^<>]*>. This also
fixes the greedy match deleting a minified single-line report from the
schema tag to the last ">" on the line.

human_timestamp_to_datetime() stripped RFC 5322 comments with
\s*\(.*\)\s*, quadratic on whitespace (200,000 spaces 18.6 s) and
reached through every Date header; filed as a bug fix because MTAs cap
header size. Comments are now removed by a depth-counting scan, which is
linear and handles the nested comments and quoted-pairs RFC 5322
section 3.2.2 allows (a character-class regex would have regressed
them).

Adds regression tests sized so the old patterns fail the time bound in
minutes rather than hanging CI, a "Regular expressions on untrusted
input" section in AGENTS.md, and the 11.0.4 changelog and version bump.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-18 12:01:32 -04:00
Sean WhalenandClaude Opus 5 77045c2df0 chore: address CodeQL code-quality findings (no behavior change) (#900) 
- py/empty-except: add a short comment to each of the 10 flagged
  `except <Type>: pass` blocks explaining why swallowing the
  exception is correct there (best-effort cleanup/close, or
  intentional fall-through to the next report format).
- py/unnecessary-lambda: replace `map(lambda x: parse_email_address(x), ...)`
  with `map(parse_email_address, ...)` at the 5 flagged sites in
  parsedmarc/utils.py; parse_email_address takes exactly one
  positional argument, so this is behavior-preserving. Ran
  `ruff format` afterward, which collapsed 3 of the now-shorter
  calls onto single lines.
- py/imprecise-assert: replace `assertTrue(a > b)` / `assertTrue(a >= b)`
  with `assertGreater(a, b)` / `assertGreaterEqual(a, b)` at the 9
  flagged test sites; none carried a custom assertion message.
- .vscode/settings.json: drop three cSpell whitelist entries that are
  pure misspellings ("passsword", "httpasswd", "unparasable") and
  appear nowhere else in the tracked tree, so a recurrence of the typo
  they used to hide (see #888) would be caught again. The correctly
  spelled "htpasswd" and "unparseable" entries are untouched.

Verified clean: ruff check, ruff format --check, pyright (0
errors/warnings), and pytest tests/ (989 tests collected and passing
before and after, GITHUB_ACTIONS=true).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-09 19:06:34 -04:00
Sean WhalenandClaude Fable 5.1 09c88ca2a3 11.0.1 release: harden failure-report sample filenames and cap decompressed report size (#895) 
* Harden failure-report sample filenames and cap decompressed report size

Fixes the two open security advisories.

GHSA-c284-w5m6-jhjm (path traversal, affects 9.0.6 through 11.0.0):
save_output() named each failure report's message sample after the
sample's Subject header, falling back to the raw subject whenever
sanitizing it produced an empty string. A subject of only path
separators and dots -- "../../../" or "/" -- sanitizes to "", so the raw
value reached os.path.join() and the .eml landed outside the samples
directory or at an absolute path. That subject comes from a message that
failed authentication, so any sender a monitored mailbox accepts
controls it. The name is now sanitized at write time and falls back to
"sample", and the caller-supplied filename_safe_subject key is no longer
trusted. get_filename_safe_string() also strips NUL (which would
otherwise make open() raise ValueError: embedded null byte and hold back
the whole mailbox batch), truncates before stripping trailing
characters, and strips trailing spaces along with trailing dots, since
Windows drops both when creating a file; its docstring now states the
guarantees callers depend on.

GHSA-43qf-f35w-2x4r (unbounded decompression, affects all versions
through 11.0.0): extract_report() inflated gzip with one unbounded
zlib.decompress() and read zip members with an unbounded .read(). The
attachment's content is chosen by its sender, and deflate reaches about
1000:1 on degenerate input, so a 100 KB attachment inflated to 100 MB
with a ~209 MiB peak. Extraction now stops at
MAX_DECOMPRESSED_REPORT_SIZE (100 MiB) and raises ParserError. The gzip
path moves to a zlib.decompressobj() bounded by max_length, which does
not raise on a stream that ends early the way the one-shot call did, so
the helper checks decompressor.eof itself; a stream with trailing bytes
after the gzip member still extracts, matching the old behavior. Both
behaviors were observed against the unmodified code first and are
pinned by tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Strip NUL from CSV fields so Python 3.10's csv writer accepts them

Python 3.10's csv writer raises _csv.Error: need to escape, but no
escapechar set on any field containing NUL (CPython issue 97503, a 3.10
regression fixed in 3.11+). Failure report text fields (subject, user
agent, authentication results, addresses, etc.) come from untrusted
mail, so a NUL byte in one made parsed_failure_reports_to_csv() -- and
therefore save_output() -- raise on 3.10, which the CLI's
except (OSError, ValueError) around save_output does not catch. NUL is
now stripped from every CSV field on all Python versions via a shared
_csv_safe() helper, applied in all three CSV writers, so output is
identical regardless of interpreter version.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* 11.0.1 release: bump version and finalize changelog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-03 15:51:44 -04:00
Sean WhalenClaude Fable 5Copilot
 		07bca1ad28 Make the output and mailbox integrations optional extras (#888) 
* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-08-28 17:33:18 -04:00
Sean WhalenandClaude Fable 5 cd864f4cfc Add DNS over HTTPS and DNS over TLS support via the nameservers option (#886) 
Each entry in the existing nameservers option now selects its own
transport (#880): an IP address means plain DNS on port 53 exactly as
before, an https:// URL means DoH, and tls://ip[:port][#hostname] means
DoT, with the optional #hostname naming the TLS certificate identity
(systemd-resolved syntax). Forms can be mixed in one list, and no new
configuration option is involved.

DoH queries go through a shared per-process httpx client passed to
dns.query.https as session=, which is what makes them honor HTTP_PROXY/
HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE — the motivating proxy-only
corporate network case. dnspython's stock DoH path cannot do this: it
builds its httpx client around a custom transport, and httpx only reads
proxy environment variables when no transport is supplied
(allow_env_proxies = trust_env and transport is None). The client is
rebuilt when the PID changes so fork-based worker pools never share a
parent's sockets.

The dnspython requirement becomes dnspython[doh]>=2.7.0 — the extra
supplies the httpx/h2 floors DoH needs, and 2.7.0 is the floor verified
against the dns.nameserver and dns.query.https(session=...) APIs used.

The startup DNS pre-flight check now exercises whichever transports are
configured, so a malformed DoH/DoT entry raises ConfigurationError
before any mailbox work begins. Malformed tls:// entries — including
the plausible slash-for-# typo tls://9.9.9.9/dns.quad9.net, which would
otherwise silently drop the certificate identity — are rejected at
configuration time naming the entry.

Verified live: DoH A/PTR queries against Cloudflare and DoT against
Quad9 (tls://9.9.9.9#dns.quad9.net), plus a full CLI run over a sample
report with encrypted-DNS-only nameservers.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-28 15:03:51 -04:00
Sean WhalenandClaude Fable 5 4e80047e68 Centralize configuration handling with ParserConfig (#503) (#851) 
* Centralize configuration handling with ParserConfig (#503)

Add parsedmarc/config.py with ParserConfig, a frozen dataclass carrying
every parsing/enrichment option plus the three shared caches (IP address
info, seen aggregate report IDs, reverse DNS map). All eight public
parsing/mailbox functions accept a keyword-only config= argument; when
provided, the individual option keyword arguments are ignored in favor
of the config's values, and every existing per-option keyword argument
keeps working unchanged. The three hand-copied parse_kwargs dicts and
the dns_timeout<->timeout rename chain are gone; the CLI builds one
ParserConfig per run (rebuilt on SIGHUP) and passes it everywhere.

Explicitly constructed configs own fresh isolated caches;
dataclasses.replace() shares them; pickling drops cache contents and
rebinds the unpickling process's module defaults, preserving the
per-worker cache behavior of n_procs parallel parsing. The module
globals IP_ADDRESS_CACHE / SEEN_AGGREGATE_REPORT_IDS / REVERSE_DNS_MAP
remain, identity-preserved, as re-exports of the default caches.

Bug fixes that ride along, each with a regression test:

- One-shot mailbox runs now honor [general] dns_timeout/dns_retries;
  the CLI call site never forwarded dns_timeout, so the library's
  stray 6.0 default silently applied.
- Lazily-triggered reverse DNS map loads (get_ip_address_info /
  get_service_from_reverse_dns_base_domain, including in n_procs
  workers) now thread psl_overrides_path/psl_overrides_url through to
  load_reverse_dns_map instead of clobbering operator-configured PSL
  overrides with the bundled defaults.
- get_dmarc_reports_from_mailbox() and watch_inbox() dns_timeout
  defaults unified to DEFAULT_DNS_TIMEOUT (2.0s) from a stray 6.0, and
  normalize_timespan_threshold_hours to the float 24.0 used everywhere
  else.

Closes #503

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address CI and review feedback on #851

- Wrap the IMAPConnection example in usage.md so ruff format is clean
  over the docs code blocks (CI runs ruff format --check on the whole
  repo; the local runs were scoped to parsedmarc/ and tests/ and
  missed it).
- Fix the pre-existing "URL ro a reverse DNS map" docstring typo in
  get_service_from_reverse_dns_base_domain, caught by Copilot on the
  adjacent hunk.
- Import parsedmarc.config once, as an aliased plain import, in
  tests/test_config.py instead of mixing import and import-from of the
  same module (flagged by code quality scanning); the aliased module
  import also keeps pyright able to resolve the submodule attribute
  access.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address second Copilot review round on #851

- Add parse_aggregate_report_file() to the library entry-point list in
  usage.md; the following paragraph describes the config= contract for
  "each of these functions", so the list must name all eight
  config-accepting entry points.
- ParserConfig.__setstate__ now initializes every non-cache field to
  its class default before applying the pickled state, so a config
  serialized by an older parsedmarc version (whose state predates
  fields added later) unpickles with the newer fields at their
  defaults instead of unset entirely (__init__ never runs during
  unpickling, so an absent field would raise AttributeError on first
  access). Covered by a regression test that feeds __setstate__ a
  partial state dict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-25 19:27:09 -04:00
Sean WhalenandClaude Fable 5 df9bf82e04 Post-review follow-ups for Graph send (#825/#826) and requests-to-httpx migration (#827) 
Follow-ups from the review of PR #825 (whose implementation had already
landed on master via #826's stacked merge):

- Honor the documented [smtp] attachment and [smtp] message options.
  Both were parsed into opts but never passed to either summary-email
  transport (also broken in released 10.2.2), so a configured custom
  attachment filename or message body was silently ignored. Both the
  SMTP and Microsoft Graph transports now receive them, and the missing
  smtp_attachment Namespace default is added (also covers SIGHUP
  reload, which rebuilds opts from the CLI Namespace).
- Don't mislabel non-Graph mailbox errors as Microsoft Graph failures:
  the shared mailbox-fetch and watch handlers now log a generic
  "Mailbox Error" with traceback when the connection isn't Graph.
- Declare microsoft-kiota-abstractions as a direct dependency (imported
  directly in cli.py for Graph error handling; previously transitive).

Migrate all runtime HTTP from requests to httpx (webhook client, Splunk
HEC client, and the PSL-overrides / IP-database / reverse-DNS-map /
IPinfo-API fetches in utils.py):

- follow_redirects=True everywhere to preserve requests' default
  redirect-following; httpx does not follow redirects by default.
- The PSL-overrides and reverse-DNS-map fetches gain a 60s timeout
  (previously none), matching the IP-database fetch.
- response.ok -> response.is_success; requests.RequestException ->
  httpx.HTTPError; raw string bodies use content= (httpx's data= is
  form-encoding only); Splunk HEC verification moves to client
  construction (httpx has no per-request verify).
- requests drops out of [project] dependencies and moves to the [build]
  extra for the out-of-wheel maintainer script collect_domain_info.py,
  which deliberately stays on requests/urllib3 for its permissive-TLS
  adapter.
- Remove the requests-era module-level
  urllib3.disable_warnings(InsecureRequestWarning) in splunk.py; httpx
  doesn't route through urllib3, so its only remaining effect was
  globally silencing insecure-TLS warnings from other urllib3-based
  components as an import side effect. Nothing imports urllib3 directly
  anymore, so it also leaves [project] dependencies.

Tests: config-to-transport wiring for attachment/message on both
transports (including defaults), non-Graph errors keep the generic log
line, webhook/Splunk payload assertions moved to content=, and Splunk
verify asserted at httpx.Client construction. 736 passed; ruff and
pyright clean.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-14 16:11:47 -04:00
KiliMISAPOR LABClaude Sonnet 5
 		f33951c0bb Fix host-dependent flakiness in testMissingEverythingRaisesFileNotFoundError (#820) 
The system-path fallback list in _get_ip_database_path() checks real
absolute paths like /usr/share/GeoIP/GeoLite2-Country.mmdb. On a host
that actually has one installed there, the fallback succeeds and no
FileNotFoundError is raised, regardless of the test's mocked bundled
path -- the code is behaving correctly, the test just wasn't isolated
from the host filesystem. Patch os.path.exists to force every system
path to look absent so the test is host-independent.

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
 2026-07-13 09:32:32 -04:00
Sean WhalenandClaude Fable 5 746da77de5 Raise test coverage: utils.py, elastic.py, and opensearch.py to 100% (#816) 
* Raise test coverage: utils, elastic, and opensearch to 100%

Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).

New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.

Bugs found by the new tests, fixed in the same PR per the testing
standards:

- parse_email() crashed with KeyError: 'Headers' on messages whose
  From header is present but unparseable (e.g. a bare "From:" line):
  the fallback read parsed_email["Headers"], but the parsed headers
  are stored under lowercase "headers" (assigned a few lines up in
  the same function), so the key never exists. At the CLI surface
  this made any failure report whose embedded sample had an empty
  From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
  when the probe could not reach the API, contradicting its own
  docstring ("other errors are logged and the token is still
  accepted"): _ipinfo_api_lookup() returns None on network errors
  instead of raising, so the probe's exception handler was
  unreachable. The probe now checks the lookup result and warns on
  failure; 401/403 still raises InvalidIPinfoAPIKey.

Dead code deleted rather than padded with tests:

- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
  (the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
  both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
  re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
  input branch assigns file_object or raises first (confirmed by
  pyright narrowing with the guard removed).

Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Use assertGreater for the reverse-DNS-map fallback size check

Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-11 18:21:04 -04:00
KiliClaude Fable 5MISAPOR LABSean Whalen
 		7fed72798a Stop system GeoIP files from shadowing the bundled IPinfo database (#813) 
* Stop system GeoIP files from shadowing the bundled IPinfo database

_get_ip_database_path() searched well-known system paths (including
/usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before
the database parsedmarc manages, so on any host with a distro GeoIP
package installed every lookup silently used a country-only — and
often years-old — database instead of the bundled IPinfo Lite one.
That disabled ASN enrichment entirely (asn/as_name/as_domain were None
for every IP) and with it the ASN-fallback path into the reverse-DNS
map, with no signal beyond a generic "IP database is more than a
month old" warning. Verified live on a Fedora host whose distro
GeoLite2-Country.mmdb dated to December 2019.

New precedence: explicit ip_db_path -> _IP_DB_PATH selected by
load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system
paths as a true last resort (only consulted when the bundled data
file is missing). The selected file is logged at debug level so a
--debug run shows which database answered.

The automatic system-path pickup was documented behavior, so
installation.md now tells MaxMind GeoLite2 users to set ip_db_path
explicitly, with a migration note.

Both new regression tests reproduce the shadowing portably via a decoy
CWD GeoLite2-Country.mmdb (the fallback list includes relative names),
and fail on the unfixed code (verified by stashing the source change).

Fixes https://github.com/domainaware/parsedmarc/issues/810.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: accurate fallback comment, dedup selection log, cover fallback tiers

- Correct the nothing-found-anywhere comment: the os.stat() age check
  raises FileNotFoundError before the caller's open_database() would.
- Log "Using IP database at ..." only when the selected path changes
  instead of on every uncached IP lookup, so --debug runs over large
  batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in
  the test fixture for order-independence.
- Cover the previously untested branches of _get_ip_database_path:
  system-path fallback when the bundled database is missing, the
  FileNotFoundError when nothing exists anywhere, the stale-database
  warning, and the log-once-per-path behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-07-09 20:07:00 -04:00
KiliClaude Fable 5MISAPOR LABSean Whalen
 		cdda5dae62 Fix failure-report timestamp skew on non-UTC hosts in ES/OpenSearch/Splunk outputs (#812) 
* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks

arrival_date_utc is a UTC wall-clock string (generated in
parse_failure_report via an aware-UTC strftime), but elastic.py,
opensearch.py, and splunk.py parsed it back into a naive datetime and
called .timestamp(), which per the Python docs interprets naive values
as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp).
On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date
field, used in the failure-report dedup match query, and sent as the
Splunk HEC event time was therefore off by the host's UTC offset
(verified -3600 s under TZ=Europe/Warsaw in January).

Add an assume_utc keyword to human_timestamp_to_datetime() /
human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive
parses, and use it at the three arrival_date_utc call sites. Aware
inputs (explicit offsets) are unaffected; all other callers keep the
existing local-time semantics, whose round-trip with timestamp_to_human
is self-consistent on a single host (the broader local-time output
question is tracked separately in issue #811 bug 2).

The three new sink regression tests fail on the unfixed code
(verified by stashing the source changes) and force TZ=Europe/Warsaw
via time.tzset() so they catch the skew even on UTC CI runners.

Fixes half of https://github.com/domainaware/parsedmarc/issues/811.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring

Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from
the four timestamp regression tests into a shared tests/tzutil.py
force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s
docstring, which said the return type was float while the function
returns int.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-07-09 19:45:34 -04:00
Sean WhalenandClaude Opus 4.7 e104f1118c Land 10.0.3 changes on master (#785) 
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.

This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 13:54:40 -04:00
Sean WhalenandClaude Opus 4.7 5b08627eaa Split tests.py into per-module tests/test_<module>.py (#774) 
* Split tests.py into per-module tests/test_<module>.py

The 5174-line tests.py monolith is split into per-module files under
tests/, mirroring the checkdmarc layout:

  tests/test_init.py          parsedmarc/__init__.py parsing surface
  tests/test_cli.py           parsedmarc/cli.py + config / env-vars / SIGHUP
  tests/test_utils.py         parsedmarc/utils.py (DNS, IP info, PSL, etc.)
  tests/test_webhook.py       parsedmarc/webhook.py
  tests/test_kafkaclient.py   parsedmarc/kafkaclient.py
  tests/test_splunk.py        parsedmarc/splunk.py
  tests/test_syslog.py        parsedmarc/syslog.py
  tests/test_loganalytics.py  parsedmarc/loganalytics.py
  tests/test_gelf.py          parsedmarc/gelf.py
  tests/test_s3.py            parsedmarc/s3.py
  tests/test_maps.py          parsedmarc/resources/maps/ maintainer scripts

The split is purely a redistribution — no test bodies changed, no tests
added or removed. All 276 existing tests pass under the new layout.

The current tests.py contains two kitchen-sink classes (`Test` at line 54
and `TestEnvVarConfig` at line 2360) holding tests that span many
modules. Their methods are routed to the correct per-module file by name
prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx,
TestSighupReload, etc.) move whole. Each target file gets its own
`class Test(unittest.TestCase)` for the redistributed kitchen-sink
methods, plus the thematic classes verbatim.

Wiring updates:
- `.github/workflows/python-tests.yml`: `pytest ... tests.py` →
  `python -m pytest ... tests/` (also switches to `python -m pytest` per
  the checkdmarc convention so cwd lands on the project root).
- `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]`
  and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for
  `parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only
  batch tooling that ships out of the wheel; excluding them from
  coverage makes the headline number reflect only installed library
  code. Runtime coverage on the new layout is 59% (was 45% with maps
  counted), and PR-B will push it to 90%+.
- `AGENTS.md`: documents the new layout and how to run individual files
  / tests; tells future contributors not to reintroduce a monolithic
  tests.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Restore 66.9% coverage baseline (count tests/ + parsedmarc)

Master's headline 66.9% number on Codecov includes the tests.py file
itself (99.35% covered) being measured alongside parsedmarc/*.  The
original tests.py had no `[tool.coverage.run]` block, so coverage's
default — "measure every file imported during the run" — counted the
test code as if it were product code.

The split commit added `source = ["parsedmarc"]` which suppressed
measurement of the test files (correct in principle, since test files
aren't shipped code), and that alone made the headline number drop by
~8 percentage points without any actual loss of testing.  This commit
swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]`
so both halves are measured the way they were on master.  Verified:
276 tests, 66.96% line coverage (effectively unchanged from master's
66.90%).

If you want the shipped-code-only number (was the headline that this
commit overrides), run `pytest --cov=parsedmarc tests/`.  That number
is currently 59% and is the focus of the upcoming coverage-expansion PR.

Also adds junit.xml to .gitignore so the CI artefact doesn't get
accidentally committed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Restrict coverage to shipped code (`source = ["parsedmarc"]`)

Reverts the prior commit's `include = ["tests/*"]`. Counting the test
files toward coverage was wrong — it conflates "shipped code exercised
by tests" with "test code that pytest auto-runs", inflates the headline
number, and rewards writing more tests rather than tests that verify
more code. Master's apparent 66.9% was an artefact of the old
monolithic tests.py having no [tool.coverage.run] block at all; coverage's
default behaviour measured every imported file, including the test file
itself at ~99% "covered", which added ~8 percentage points to the
displayed number without any real testing signal.

Restricting to `source = ["parsedmarc"]` plus the existing maps omit
gives a meaningful baseline: 59% of shipped code is exercised by the
test suite today. That's the number the next PR is targeting to lift
to 90%+ before the 10.0.0 release; the Codecov "drop" here is a
measurement correction, not a regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 19:29:09 -04:00