mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-10 19:11:00 +00:00
Sean WhalenandClaude Fable 5.1
e8292bef70
* Wrap the failure-report duplicate search in the output error The aggregate and SMTP TLS saves turn a failed duplicate-check search into ElasticsearchError or OpenSearchError; the failure-report saves let the search library's exception escape instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Refuse date strings too long to be dates human_timestamp_to_datetime() had no length limit; a digit run of a million characters took 12 s in dateutil. Add a 128 character limit (ValueError) and map it to the matching Invalid*Report at the failure Arrival-Date, the message Date header and the failure-report sample Date in the Elasticsearch and OpenSearch saves. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Skip failure sample headers with an empty name The Elasticsearch and OpenSearch failure saves normalise each header name; a name that is empty after that is not a valid field name, so leave the header out and save the rest. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add a persisted store of unfinished parse attempts A message whose parse kills the process (the out-of-memory killer, a container memory cap) is never moved out of the reports folder, so every later run parses it and dies again. parsedmarc.attempts keeps a small JSON file counting, per message, the parse attempts that started but never finished, so the mailbox loop can set such a message aside after MAX_PARSE_ATTEMPTS. The key hashes the first 64 KiB of the message because mailsuite's IMAP backend does not expose UIDVALIDITY, so a reused UID must not inherit an old count. Writes are atomic and owner only, and a failed write turns the store off instead of stopping a run. Refs GHSA-qcc2-4x3v-v4g7, GHSA-q7gm-8cj9-94hr, GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Set aside mailbox messages that keep killing the parse A message whose parse ends the process stays in the reports folder, so every one-shot run or service restart parsed it again and died before reaching the reports behind it. The new [mailbox] attempts_file option records each message in an AttemptStore before it is parsed and clears the entry once the parse returns a report or raises ParserError, so an entry left behind marks a parse that never finished; a message with MAX_PARSE_ATTEMPTS of them is disposed of as invalid without parsing. The entry is cleared before the save, so a failed save is never counted as a crash, and only after a successful move, so a failed move is retried. With n_procs > 1 a dead run had a whole submission window in flight, so counting every message submitted beside the culprit would set innocent reports aside on the third run. A message with one unfinished attempt is therefore parsed alone in the calling process after the rest of the batch, and a second death counts against it only. PARSEDMARC_MAILBOX_ATTEMPTS_FILE names the file directly, so the key joins _DIRECT_FILE_KEYS; without it the variable was read as a Docker-secret reference. Refs GHSA-qcc2-4x3v-v4g7, GHSA-q7gm-8cj9-94hr, GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stop a parallel_map job that runs past a timeout The parse limits added for GHSA-qcc2-4x3v-v4g7 bound every input known to parse slowly, but nothing bounds a parse they do not cover: with n_procs = 1 the rest of the mailbox waits behind it. parallel_map() now takes parse_timeout and timeout_result. A job that runs that long gets timeout_result(job, parse_timeout) in its place; since a ProcessPoolExecutor cannot stop one job, every worker is killed, a new pool is started and the other unfinished jobs are resubmitted. A job's clock starts when its worker reports that it began, through a queue the pool initializer hands each worker. Future.running() is not enough: the executor marks one job more than it has workers as running, so a job queued behind a slow neighbour would be charged for the wait. ParseTimeoutError subclasses InvalidDMARCReport so the mbox and mailbox loops treat a timed-out message as invalid without changes. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add parse_timeout to the mbox, mailbox and watch functions get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox() and watch_inbox() take parse_timeout and pass it to parallel_map() with the email job's timeout value. A timeout set with n_procs = 1 also takes the parallel_map() branch, since only a parse in a worker process can be stopped. The mailbox function and watch_inbox() reject an invalid value before fetching or watching, as they do for max_unsaved_retries: raised inside a watch check, the error would be retried on every check. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the [general] parse_timeout option The CLI reads parse_timeout as seconds, rejects a negative or non-numeric value as a ConfigurationError, defaults it to 0 (no limit), and passes it to the report-file, mbox, mailbox and watch calls. The watch call reads it from opts, so a SIGHUP reload takes a new value. The default stays 0 because the parse limits already bound the known inputs and a timeout routes every n_procs = 1 run through a worker process; the documentation suggests 300 seconds. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Document the attempts_file option Describe [mailbox] attempts_file in the option list and in a new usage section on messages that crash parsedmarc, say in the Unsaved note that the failed-save counts are the in-memory ones, keep the file in a state directory in the systemd example and in a named volume in the Docker Compose example, and add the Security changelog entry for the retry-loop backstop the three advisories deferred. Refs GHSA-qcc2-4x3v-v4g7, GHSA-q7gm-8cj9-94hr, GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * List parse_timeout and attempts_file as orchestration options ParserConfig's docstring and the usage page's ParserConfig notes name the keyword arguments that stay outside the dataclass. The two new options are such arguments, so a library caller looking for them on ParserConfig is told where they go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound a suspect message's parse by parse_timeout With an attempts file, a message that was being parsed when an earlier run died is parsed alone after the rest of its batch, so that a second death is charged to it and not to the messages parsed beside it. That parse ran in the main process, where parse_timeout cannot stop it, so the one message most likely to be slow or fatal was the one the timeout did not cover. With a timeout set, each suspect now gets a worker pool of its own, one message per pool so a crash is still attributed to it alone, and its result is taken the same way as the batch's. Without a timeout the suspect is still parsed in this process. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore default signal handlers in parsing workers Under the fork start method a worker inherits the parent's signal handlers, and the CLI installs SIGTERM and SIGINT handlers that only set a flag. When a worker dies, ProcessPoolExecutor terminates the others with SIGTERM and joins them; a worker running the inherited handler ignores that signal, so the teardown waited forever and the run hung instead of failing. The pool initializer now resets both signals to their defaults. The logging tests that call the initializer in the test process put the process's own handlers back afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Order the Unreleased changelog and tie its entries together The three branches each added to the Unreleased entry. The GHSA-q7gm bullet still promised a follow-up for a persisted failed-attempt marker, which the attempts_file option now is, so it names the option. The attempts_file bullet now says that a suspect message is parsed in a worker of its own when parse_timeout is set, matching the code. The #942 fix moves up beside the other fix already on master, and the worker signal-handler fix gets its own line. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ignore SIGINT in workers and end the batch when one dies Resetting SIGINT to its default in the workers made a Ctrl-C, which the terminal sends to the whole process group, kill every worker, so the run ended in BrokenProcessPool where the CLI's handler meant to let the batch finish. Workers now ignore SIGINT and keep the default SIGTERM action, which ProcessPoolExecutor needs to tear down a broken pool. A worker that dies for another reason (the out-of-memory killer, a crash) raised BrokenProcessPool out of parallel_map, which no caller handled: the file and mbox paths ended in a traceback. parallel_map now kills the remaining workers, closes the start queue and raises ParserError from it, and the CLI file and mbox paths log it and exit 1 as the mailbox paths already did. The in-flight messages keep their attempts-file entries, so the next run parses each alone. The systemd example gains KillMode=mixed so stopping the service signals only the main process and the batch in progress finishes. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * State the parse_timeout bounds the code actually gives A slow job running beside another that expires loses its partial run and starts again, so P slow jobs cost one more round of parse_timeout than the docstring said, plus the bounded joins of the killed workers, during which no heartbeat is sent. The clocks are also checked only while the parent waits for a result, so "at most one second past the timeout" was not true while the caller is busy between results; the docstring, usage.md and the changelog now say a parse is stopped within one wait slice of the parent next waiting. ParseTimeoutError's docstring says it is the value for every timed-out input, SMTP TLS and unrecognised ones included. A new test closes the results iterator mid-run and checks the workers are killed and the start queue closed; the zero-timeout test now observes which pool ran the job instead of patching _DeadlinePool. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct the attempts-file wording and test the parallel test mode Suspect isolation happens whenever the mailbox takes the parallel path, which parse_timeout selects even with n_procs = 1; the docstring, comment, usage.md and changelog now say so. The set-aside log said a message "has crashed parsedmarc {count-1} time(s)", which overstates the crashes once a failed move has added attempts; it now reports the unfinished attempts against the cap. usage.md lists attempts_file among the _file keys read directly from the environment and among the settings a SIGHUP reloads, and says that a failed write clearing an entry after a good parse can lead to a healthy message being set aside after two such failures. The test-mode set-aside test now runs the parallel path too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Measure a date's length after its comments are removed The length guard in human_timestamp_to_datetime() ran before the RFC 5322 comments were stripped. RFC 5322 section 3.2.2 puts no limit on a comment's length, so a valid Date header with a long comment was refused, and through the sample Date path the whole report email with it. The check now runs on the stripped value, which is linear to produce; a long digit run survives stripping, so it is still refused before dateutil sees it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Save parsed reports before exiting on a dead worker A ParserError from parallel_map (a worker process that died) or from an mbox made the CLI exit at once, discarding every report parsed so far; report files already moved under archive_directory were then saved nowhere. The CLI now stops reading input, saves what it gathered, and exits with status 1. Workers ignore SIGINT, so the plain pool path's executor shutdown on an early exit (KeyboardInterrupt, closing the generator) waited for the whole submission window. Any early exit now kills the workers, as the timed path already did. The Ctrl-C test ran under fork only, where a worker inherits the test's own harmless handler, so it passed without the SIGINT line. It now runs under every available start method and signals each worker once it has started a job. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stop counting a message that is already set aside A message at the attempt limit that stayed in the folder (a failed move, test mode) was recorded again on every later run, so the logged count grew without bound. It is now set aside without being recorded, and the log names the stored count and the limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe worker death per mode and the systemd stop window A dead worker ends a one-shot run with status 1 after saving, but in watch mode the IMAP and Maildir watchers log it and check again, while Graph and Gmail exit. KillMode=mixed lets the batch finish only within TimeoutStopSec. Also say that queued messages keep attempt entries too, and re-wrap a parse_timeout note line. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Clear the seen report IDs around the worker-death CLI tests The new save-before-exit test parses a sample in the parent, which records it in the shared dedup set, so a later test of the same sample saw it as a duplicate and sent no results email. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Save file reports when a mailbox fetch ends in a ParserError A worker that died during the one-shot mailbox fetch reached the ParserError handler, which exited before the reports from report files and mbox inputs were saved; files already moved under archive_directory were then kept nowhere. The handler now saves them first. A second test covers the input-error path: no mailbox is opened and the parsed file stays under archive_directory. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say what a mailbox run keeps when a worker dies Earlier batches are saved and the batch in progress stays in the mailbox. Also say that a set-aside message is not recorded again, so its entry ages out after 30 days, and rewrap two paragraphs. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which set-aside entries age out of the attempts file The 30-day drop applies only to an entry that stays: a failed move, or a message left in place in test mode. The sentence read as if every set-aside message kept an entry, contradicting the line before it. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Carry the test attempts path in the message, not the environment test_parallel_path_records_each_attempt_before_submitting_it passed the attempts file's path to its worker job through os.environ. A forkserver worker forks from a server started before the patch, so it never saw the variable and the job raised KeyError (Python 3.14's Linux default). The path now travels in a header line of the message itself, which reaches the worker under every start method. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the worker-side helpers and the save-failure exits Codecov listed added lines no test reached: the timed pool's initializer and job wrapper, which run only in workers; _kill_workers on an executor without _processes; the plain path's heartbeat; the CLI's progress bar; the mbox loop's stop after a file worker dies; and a failed save before exiting, on the file/mbox path and the mailbox path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read start tokens every round, not only while waiting With parse_timeout set, the parent read the workers' start tokens only inside its wait loop. When results were already done at harvest time it never waited, so each finished job stayed in the pool's token map with its whole input, and the map grew with the run past the submission window. The tokens are now read at the top of every round, and close() clears the map. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Kill the parsing workers on a second Ctrl-C The second Ctrl-C ends parsedmarc with os._exit(130), which skips the process pool's cleanup. Workers now ignore SIGINT, so under every start method a running worker survived as an orphan and kept parsing, without end when parse_timeout is off. The handler now kills the active child processes first; Python runs it in the main thread between bytecodes, so active_children() and Process.kill() are safe there. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hand timed results back through files, not the pool's pipe With parse_timeout set, a timeout kills every worker, possibly one that is sending a large result. The executor's management thread then waits for the rest of that message forever; the bounded join returns with the thread alive, and because it is not a daemon the interpreter hangs at exit. Reproduced under fork, forkserver and spawn with a 256-400 MiB result. Workers of the timed pool now pickle each outcome, error values included, to a file in a private directory the pool owns and return None; every message left on a pipe is far below PIPE_BUF, which POSIX writes atomically. The pool without a timeout never kills a running worker and is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Report an unwritable temporary directory as a ParserError With parse_timeout set the results travel through a temporary directory. If it could not be created, or a worker could not write its result file, a raw OSError escaped parallel_map, which the CLI does not handle. The pool now raises ParserError naming the cause; a worker that cannot write its file returns the error text through the executor, still a small message, and the parent raises the same ParserError. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep every hand-back failure a short ParserError A result the worker could not pickle (PicklingError, TypeError, RecursionError, MemoryError) escaped through the executor with a traceback of any size, breaking the short-message guarantee the timed pool relies on, and a result file that did not unpickle raised a raw exception in the parent. The worker now catches any failure to write its outcome and returns a short marker, its text cut to 200 characters so the message stays below the 512-byte minimum PIPE_BUF; the parent turns it, and any failure to load a result file, into a ParserError. Also corrects the PIPE_BUF figure in the module docstring, names the temporary directory as TMPDIR, and avoids a second 128 MiB object in a test. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cap the hand-back error text in bytes and survive a failing __str__ The pipe's atomic-write bound is in bytes, so a 200-character text of multibyte characters could exceed it; the cap now applies to the UTF-8 encoding. An exception whose __str__ raises is named by its type alone, so nothing in the worker's error path can raise through the executor. A failed write's .tmp file is removed with the pool's result directory; the docstring says so. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Match the pickler's message across Python versions in a test The unpicklable-result test asserted "Can't get local object", which is the wording of Python 3.12 and 3.13 only; 3.10, 3.11 and 3.14 say "Can't pickle local object". Assert the part they share. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Cover the worker-side hand-back failures in the test process The three failure branches of _run_timed_job and _take_outcome run in worker processes in the pool tests, where coverage cannot see them; call them directly here with an unwritable directory, an unpicklable result and an unreadable result file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Assert the unreadable-result message by its text, and format The temporary-directory error carries the OSError's text, not its class name, so the test asserts "Is a directory"; one long test line is wrapped as ruff format wants. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Narrow the marker before unpacking it in two tests pyright cannot narrow Optional through assertIsNotNone; a plain assert does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
parsedmarc
parsedmarc is a Python module and CLI utility for parsing DMARC
reports. When used with Elasticsearch and Kibana (or Splunk), or with
OpenSearch and Grafana, it works as a self-hosted open-source
alternative to commercial DMARC report processing services such as
Agari Brand Protection, Dmarcian, OnDMARC, ProofPoint Email Fraud
Defense, and Valimail.
Note
Domain-based Message Authentication, Reporting, and Conformance (DMARC) is an email authentication protocol.
Sponsors
This project is maintained by one developer. Please consider sponsoring my work if you or your organization benefit from it.
Features
- Parses aggregate/rua DMARC reports: the legacy draft and 1.0 schemas (RFC 7489) and the new RFC 9990 schema for the final DMARC standard (RFC 9989)
- Parses failure/ruf DMARC reports (RFC 6591 and RFC 9991; formerly called forensic reports)
- Parses reports from SMTP TLS Reporting (TLS-RPT, RFC 8460)
- Can parse reports from an inbox over IMAP, Microsoft Graph, or Gmail API
- Transparently handles gzip or zip compressed reports
- Consistent data structures
- Simple JSON and/or CSV output
- Optionally email the results
- Optionally send the results to Elasticsearch, OpenSearch, Splunk, or PostgreSQL, for use with premade dashboards
- Optionally send the results to Apache Kafka, Amazon S3, Azure Log Analytics (Microsoft Sentinel), a Graylog (GELF) endpoint, a syslog server, Grafana Loki, or an HTTP webhook
Python Compatibility
This project supports the following Python versions, which are either actively maintained or are the default versions for RHEL or Debian.
| Version | Supported | Reason |
|---|---|---|
| < 3.6 | ❌ | End of Life (EOL) |
| 3.6 | ❌ | Used in RHEL 8, but not supported by project dependencies |
| 3.7 | ❌ | End of Life (EOL) |
| 3.8 | ❌ | End of Life (EOL) |
| 3.9 | ❌ | Used in Debian 11 and RHEL 9, but not supported by project dependencies |
| 3.10 | ✅ | Actively maintained |
| 3.11 | ✅ | Actively maintained; supported until June 2028 (Debian 12) |
| 3.12 | ✅ | Actively maintained; supported until May 2035 (RHEL 10) |
| 3.13 | ✅ | Actively maintained; supported until June 2030 (Debian 13) |
| 3.14 | ✅ | Supported (requires imapclient>=3.1.0) |