parsedmarc/CHANGELOG.md

Changelog

Unreleased

Security

• Crafted email messages no longer take seconds to minutes and hundreds of megabytes to parse (GHSA-qcc2-4x3v-v4g7, affects 11.2.1 and earlier). parse_report_email() (mailbox, mbox and file input) and the failure-sample parser parsedmarc.utils.parse_email() gave the whole message to mail-parser and Python's email package with no limit on blank lines, header lines, header field size, MIME parts or nested multiparts. Parsing builds one Message per blank line after a message/delivery-status part (512 KiB of blank lines took 6.1 s and 283 MiB of extra memory on 11.2.1 under Python 3.12) and tests every line against every open boundary of nested multiparts (900 levels around 30,000 parts, 0.3 MB, took 10.9 s); a 4 MiB flood of header lines took 1.4 s, and a 1 MB To: address group of 250,000 members took 1.9 s, or 44 s through mailsuite's own parse_email(), which parsed the group whole. With n_procs = 1 the rest of the mailbox waited behind such a message, and a process with a memory cap was killed before the message was moved, so it was parsed again on the next run. Both parsers now call mailsuite.utils.check_message_limits(), new in mailsuite 2.3.5 (GHSA-748w-c9w9-79xp), before mail-parser sees the message. It parses the message with the email package's own feed parser under the compat32 policy, the parser mail-parser and email.message_from_string() use, and stops as soon as the message has more than 10,000 parts (the message itself, its MIME parts, the bodies of message/* parts and each block of a message/delivery-status body), nests them more than 100 levels deep, or makes the parser test a line against an open boundary more than 20,000,000 times. It also stops when the message's header blocks hold more than 100,000 lines in all (folded lines and lines the parser keeps only as defects included). It then rejects a parsed message with a header field longer than 65,536 characters (name, colon and raw value, folded lines included), or whose From, Sender, Reply-To, To, Cc, Bcc and Delivered-To headers hold more than 262,144 characters in all, counted in every part; that limit bounds mail-parser's own address parsing (it parses From, Reply-To, To, Cc, Bcc and Delivered-To), which does work for each malformed address item. When a part lacks its start boundary, mail-parser parses the top-level epilogue again, and that text is checked with the same limits. parse_report_email() also no longer renders every part it walks as text: it renders a part only where it reads it, because rendering each level of a deep nesting rendered everything inside it again (98 nested multiparts around a 20 MiB leaf took 26.6 s, a chain of 99 message/rfc822 parts around a 10 MiB leaf 11.3 s, and a chain of 99 message/feedback-report parts around a 10 MiB leaf 11.6 s, all within the limits; each now takes under a second). An over-limit email raises InvalidDMARCReport("Message rejected: Message exceeds <limit> (<value>)"), which the mailbox loop files under Invalid; an over-limit failure-report sample raises InvalidFailureReport("Sample message rejected: …"). Each crafted message from the report is now rejected in under half a second (the 4 MiB header flood, the slowest, in 0.2 s). mailsuite 2.3.5 or later is now required. The floor applies to the base dependency and to the msgraph, gmail and all extras. mailsuite 2.3.5 also removes RFC 5322 group syntax from address header values over 8 KiB before parsing them, caps the address header value its own parse_email() reads at 65,536 characters, and fixes a quadratic url_regex; all only affect callers of mailsuite's parse_email(), which parsedmarc does not use. A follow-up issue tracks a per-job timeout in parallel_map(). Compatibility: the limits are fixed; there is no new option. A message with more than 10,000 parts or parts nested more than 100 levels deep is now rejected as invalid instead of being parsed or, when nested past Python's recursion limit, failing with a ParserError; in mbox input such a message (the nested message of GHSA-qxwv-2vwj-v828 is one) is now skipped with a warning instead of stopping the run. A message/rfc822 attached inside a failure report's sample no longer replaces the sample, since a feedback report has one sample (RFC 6591 §3.1), and only the first message/feedback-report part of a message is read, since a feedback report has one (RFC 6591 §3). Reported by helixnebulatech.
• Deeply nested or very wide aggregate report XML no longer exhausts memory (GHSA-q7gm-8cj9-94hr, affects 11.2.1 and earlier). parse_aggregate_report_xml() passed the decompressed report, up to the 100 MiB decompression cap, to xmltodict.parse() with no limit on nesting depth or element count. xmltodict keeps one stack entry per open element and expat reports missing end tags only at the end of the input, so a gzip attachment of about 30 KB holding 30 MiB of unclosed <a> tags needed more than 2 GB of memory (2 MiB of them cost 255 MiB on 11.2.1). The out-of-memory killer ended the process before the message was moved, so every restart parsed it again. A document of millions of sibling elements did the same at a smaller scale, and a document whose first parse failed was parsed again through lxml recovery and a second xmltodict pass. While expat reads the document, before xmltodict adds each element to its tree, parsedmarc now counts nesting depth and nodes (elements plus attributes); a report deeper than 100 levels (MAX_XML_DEPTH) or with more than 1,000,000 nodes (MAX_XML_NODES) is rejected as InvalidAggregateReport("Invalid XML: …") without trying lxml recovery. Before any parser sees the document, a tag longer than 65,536 characters (MAX_XML_TAG_LENGTH) is rejected too, because expat and libxml2 build an element's whole attribute list before its attributes can be counted (one element with 900,000 attributes, 9.3 MiB, within the node limit, took 1.1 s and +200 MiB; it is now rejected in under 0.1 s). Because a quoted attribute value may hold a raw >, which splits a tag for that scan, a document with more = characters than the node limit is rejected first, in one pass (2,000,000 x=">" attributes, 23.7 MiB, took 1.4 s and +310 MiB before the node limit rejected them; they are now rejected in under 0.1 s). The decompression cap for a report attachment is lowered from 100 MiB to 32 MiB (MAX_DECOMPRESSED_REPORT_SIZE), and an uncompressed report of more than 32 MiB of UTF-8 is rejected the same way, as is a report file of that size, which is no longer tried as a report email: at the node limit a legitimate report is at most about 28 MB, and with that cap a report's text is bounded along with its node count. When the first parse fails and lxml recovery runs, a parser target that builds no tree first counts the document's elements, attributes and namespace declarations against the same node limit (a 22 MiB wide document behind a malformed XML declaration took 5.0 s and 1,532 MiB of extra memory before, 0.57 s and 23 MiB now), and the recovered XML then goes through the same depth and node limits in the second xmltodict pass. A follow-up issue tracks a persisted failed-attempt marker, so that a message that kills the process before it is filed is moved to Invalid after a number of runs. Compatibility: there is no new option. A real report is fewer than 10 levels deep and the largest sample (909 KB) has 32,019 nodes, so the node limit is a report of about 28 MB of XML; a larger report is now rejected. A report at the size cap needs about 400 MB of memory on the lxml recovery path and about 350 MB otherwise when its text is parsed directly, and about 500 MB and 450 MB through parse_report_file(), which also holds the file's bytes, where a report under the old 100 MiB decompression cap could need gigabytes. An aggregate or SMTP TLS report of more than 32 MiB of UTF-8, compressed or not, is now rejected, where 100 MiB was accepted (an uncompressed SMTP TLS report had no cap: 60 MiB of JSON cost +1,505 MiB). parse_report_file() no longer tries a file that begins with < as SMTP TLS JSON or a report email, nor one that begins with { as a report email; its error message is unchanged, and the exception's cause now matches the format the file begins with. It also keeps only each failed attempt's message while it tries the next format, so a file at the size cap on the lxml recovery path takes +500 MiB there, where it took +1,046 MiB. The XML parse step is about 20% slower, and a malformed document that needs lxml recovery is now parsed twice by lxml (once to count, once to build the tree). Reported by helixnebulatech.
• An aggregate report with a huge date range can no longer expand into millions of records or fill an Elasticsearch or OpenSearch cluster with indexes (GHSA-p85v-578x-3wcf, affects 11.2.1 and earlier). An aggregate report whose <date_range> was longer than normalize_timespan_threshold_hours was split into one record per calendar day, with no limit on the number of days. A report under 2 KB declaring a range of millions of days with a large <count> expanded one <record> into millions of records (200,000 days gave 200,000 records in 0.8 s; the reporter's 2.9 million days exhausted 2 GB of memory). With Elasticsearch or OpenSearch output each of those days created its own dated index of one shard until the cluster reached cluster.max_shards_per_node (1,000 by default), after which it refused every new index for every application using it. A report whose date range is negative or longer than 366 days (MAX_REPORT_SPAN_DAYS) is now rejected with InvalidAggregateReport. A report whose split would produce more than 250,000 records (records times days, MAX_NORMALIZED_RECORDS) is not split: its records are kept whole over the report's own date range, and one warning is logged (10,000 records over 366 days, each with a large <count>, produced 3,670,000 records in 13.5 s and 1.7 GB of memory before). In Elasticsearch and OpenSearch a single report creates at most 31 dated indexes (MAX_INDEXES_PER_REPORT): any further records are saved to the report's first index with their own dates, and one warning is logged for the report. Dated index names still come from each report's own dates, so reports with many different dates still create one index per date; data streams (use_data_streams) are affected by neither. Compatibility: a genuine report spanning more than 366 days is now rejected; a report too large to split has timespan_requires_normalization: false and records with normalized_timespan: false; there is no new option. Reported by helixnebulatech.
• One crafted report can no longer make an output fail and stop every run on the same mailbox (GHSA-4272-4r8g-6c5h, affects 11.2.1 and earlier). Some reports made a configured output raise an exception that process_reports() did not catch. It escaped the mailbox save callback, parsedmarc logged "Mailbox Error" and exited with status 1, and the batch stayed in the mailbox, so every one-shot run failed on the same message, outputs that do not deduplicate received the batch's legitimate reports again on each attempt, and watch mode finally moved the whole batch to Unsaved. The triggers were a failure report whose sample has an empty From:, To: or Reply-To: header (a legitimate To: undisclosed-recipients:; too) with Elasticsearch or OpenSearch output (IndexError); an SMTP TLS report with a field of the wrong type, such as "policy-string": [1] with file output (TypeError in the CSV writer) or "start-datetime": 5 with Elasticsearch, OpenSearch, Splunk HEC or PostgreSQL output; and an aggregate report whose policy override <reason> has no <type>, or a <type> with an attribute or given twice, with file output (KeyError, TypeError). Every output destination in process_reports() now records any exception as an output error, as the aggregate Elasticsearch and OpenSearch blocks already did (the failure and SMTP TLS Elasticsearch and OpenSearch blocks, every PostgreSQL and Splunk HEC block and the file output lacked that catch-all), so the batch is reported unsaved and fail_on_output_error still applies. SMTP TLS reports are now type-checked against RFC 8460 §4.4 and rejected as InvalidSMTPTLSReport otherwise: a string for organization-name, report-id, policy-domain, policy-type, result-type and the optional failure-detail strings, a string or a list of strings for contact-info, a list of strings for policy-string, an integer or null for each session count (true and false are rejected), and a list of objects for failure-details. start-datetime and end-datetime must convert with the same functions the outputs use and be at most 64 characters long, because dateutil took 12.4 s to fail on a date of 1,000,000 digits. A string holding a lone surrogate (JSON "\ud800"), which I-JSON (RFC 7493 §2.1) forbids, is rejected too: it passed every type check and made printing the results to standard output raise UnicodeEncodeError, which process_reports() now also records as an output error; a report that cannot be printed to the console (a non-UTF-8 console, a closed pipe) is now recorded as an output error instead of ending the run. A policy override reason's type and comment are normalised to a string or null (an element with an attribute to its text, a repeated element to null), so the CSV writer and the Elasticsearch and OpenSearch saves no longer fail on them. An address header with no address is left out of the Elasticsearch and OpenSearch failure document and of the duplicate query. The Splunk HEC SMTP TLS save raises SplunkError on a date it cannot convert, and the PostgreSQL date helper passes a non-string value through unchanged. Compatibility: an SMTP TLS report with a field of the wrong type that used to parse is now invalid and goes to Invalid; a policy-string that is not a list is now rejected (it was silently dropped), while a null one is still accepted as absent. A policy override reason without a type now has type: null and is written as none in CSV; one whose type has an attribute is stored as its text, and a repeated one as null. Reported by helixnebulatech.
• One crafted report can no longer change the field types of an Elasticsearch or OpenSearch index and get every later report rejected (GHSA-qpcg-29vr-jwm3, affects the Elasticsearch and OpenSearch outputs in 11.2.1 and earlier, data streams included). Indexes and the data-stream index template were created without field mappings, so the cluster typed each field from the first document that held it, with date detection on, and a dated index is named from the report's own begin date. One crafted report could make report_id a date field (<report_id>2026-11-09</report_id>), turn sample.headers.subject into an object (with a sample header named Subject.x), or use up index.mapping.total_fields.limit (1,000 fields) with invented sample header names. Every later legitimate report for that index, or for that data stream until its next rollover, was then rejected with a mapping error, retried and moved to Unsaved; this was confirmed against Elasticsearch 8.19.7 and OpenSearch 2.19.3. New indexes and data-stream templates are now created with date detection off and the date fields declared as dates (@timestamp, date_range, date_begin and date_end; for failure reports @timestamp, arrival_date and sample.date); when an aggregate report is parsed, every XML attribute is dropped and every field the schema types as a string is stored as a string or null, so a report cannot change a field's type (an element with an attribute or a child element, such as <p a="1">none</p> or <p><x>1</x></p>, became an object that mapped the field as one). In failure documents sample.headers is mapped with dynamic: false and indexes only 29 headers, those the shipped dashboards reference plus cc; every header stays in _source except a From, To or Reply-To header that holds no address (see GHSA-4272-4r8g-6c5h above), and a . in a header name is stored as _. create_indexes() and create_data_streams() take a new optional mappings argument. Compatibility: existing dated indexes are not changed; the protection applies to dated indexes created by 11.2.2 or later, and, because parsedmarc puts the data stream template again in every process, to an existing data stream's next backing index after a rollover. On those, sample headers outside the list are no longer searchable. The new "Field mappings" sections of the Elasticsearch and OpenSearch documentation show how to add one through the <stream>@custom component template (Elasticsearch data streams), an index template (dated indexes) or the put-mapping API (OpenSearch data streams, repeated after each rollover). For string fields, an aggregate report element with text and a stray child element now yields its text, where it yielded an empty string or an object before: samples/aggregate/invalid_xml.xml, whose <header_from>bad<xml.net</header_from> lxml recovery turns into mixed content, now gives the header-from domain bad. A <begin> or <end> with mixed content is still rejected. This narrows, but does not replace, the explicit mappings proposed in #782. Reported by helixnebulatech.

Bug fixes

• A failure report's RFC 6591 feedback report and sample can no longer be replaced by legacy report text. parse_report_email() also reads the legacy text format, a text/plain part beginning "A message claiming to be from you has failed" that carries the source IP and the sample, and a later such part replaced the message/feedback-report part and the sample already read. A text/plain part inside the sample, which is the spoofed message's own text, could therefore set the report's Source-IP and sample. Once a message/feedback-report part is read, legacy text parts are ignored, and the RFC 6591 part and its sample win over a legacy part that came before it; a legacy part followed by a message/feedback-report part with no sample after it (RFC 6591 §3.1 puts the sample after the report part) is now rejected as InvalidDMARCReport, where a mix of the two was returned. Parts inside the sample, which are the reported message's own attachments, are no longer read as reports either: an SVG, zipped HTML or application/json attachment there made the whole email invalid, and a gzipped aggregate report or SMTP TLS JSON there was returned in place of the failure report. This holds for a second sample part too, which is not read as the sample; a report attached beside the sample, outside it, is still read.
• SMTP TLS reports keep their MX host patterns (#942). RFC 8460 §4.4 keys a policy's MX host patterns mx-host; mx-host-pattern is the name of the value's type ("a JSON array of strings"), and the example in Appendix B gives a plain string. parsedmarc read only mx-host-pattern, so every RFC-conformant report lost the field. mx-host is now read first, as a list of strings or a plain string (which becomes a one-element list); mx-host-pattern is kept as a fallback, and a plain string there is now kept too. The output key mx_host_patterns and its PostgreSQL TEXT[] column are unchanged, and samples/smtp_tls/rfc8460.json now yields ["*.mail.company-y.example"]. Reported by kowax.

11.2.1

Security

• Credentials are no longer sent in plain text when an attacker strips STARTTLS from the SMTP or IMAP server's reply (GHSA-3588-2mr7-vrcm, affects 11.2.0 and earlier). With [smtp] ssl = False (the default; the default port is 25), email_results() opened a plain-text connection and upgraded it with STARTTLS only if the server's EHLO reply listed STARTTLS; otherwise it logged a warning and authenticated anyway. That reply arrives before any encryption, so an on-path attacker who removed STARTTLS from it received the SMTP username and password and the emailed results (CWE-319). The CLI requires [smtp] user and password whenever [smtp] host is set, so every deployment that emailed results with the default settings used this path. [imap] ssl = False (the default is True) had the same weakness: parsedmarc logged in over plain text when the server did not advertise STARTTLS. mailsuite 2.3.4 (GHSA-2x4j-7g3q-wr6f) now refuses a plain-text connection that was not upgraded before any credentials are sent, so the run fails with "Failed to email results" or "IMAP Error" instead. mailsuite 2.3.4 or later is now required. ssl = True (implicit TLS), the Microsoft Graph and Gmail API connections, and servers that advertise STARTTLS are unchanged. Compatibility: a server that offers no TLS at all now fails. Set the new [imap] or [smtp] starttls = False option to allow plain-text authentication again (recommended only for a server on the same host, such as DavMail on the loopback address; a warning is logged at startup when it is set and ssl = False), or switch to ssl = True. Library callers can pass the new starttls argument to email_results() (None by default, which requires STARTTLS before credentials are sent; True always requires it; False never upgrades). A call to email_results() without username and password still sends the message in plain text when the server does not advertise STARTTLS, with a warning. Reported by helixnebulatech.

11.2.0

New features

• Grafana Loki output. A new [loki] section pushes every aggregate, failure, and SMTP TLS report record as one JSON log line to the Loki push API (/loki/api/v1/push), using the same flat records as the syslog and GELF outputs plus report_type and domain. Streams are labeled job=parsedmarc and host=<FQDN>, overridable and extendable with labels, plus the record fields named in label_fields (default report_type, domain). Entries are timestamped at ingest by default, or at the begin of the reported period with timestamp = report_begin. Basic authentication, a bearer token, and a tenant ID (X-Scope-OrgID) are supported, and large reports are split into pushes of batch_size lines. Redirects are not followed, so a 3xx response counts as a failed push. A failed push raises LokiError, so it counts as a failed save and holds the batch's mailbox messages back for a retry. Needs only httpx, so it is part of the base install.
• Grafana dashboard for the Loki output. dashboards/grafana/Grafana-DMARC_Reports-Loki.json mirrors the PostgreSQL dashboard's layout with LogQL queries against the Loki output's lines, filtered by the domain stream label. The dashboard dev stack now runs Loki, seeds it with the sample reports, and imports the dashboard.
• Elasticsearch and OpenSearch output can now write to data streams (#842, relates to #476). The new use_data_streams option in [elasticsearch] and [opensearch] (default False) saves each report type to one data stream named like today's indexes without the date suffix (dmarc_aggregate, dmarc_failure, smtp_tls, with index_prefix, index_suffix, or the index_prefix_domain_map prefix applied as before), instead of creating an index per day or month. That gives each report type one stable name, keeps the shard count down, and lets the cluster roll over and delete old data. parsedmarc puts an index template named after each stream, whose index_patterns is the exact stream name so it never applies to the dated indexes, and creates the stream; it does not create or manage retention policies. On Elasticsearch, attach an ILM policy through an optional <stream>@custom component template, which the template includes; on OpenSearch, an ISM policy attaches itself through its ism_template. docs/source/elasticsearch.md and docs/source/opensearch.md have example policies. Existing dated indexes are left alone and keep matching the dashboards' dmarc_aggregate*, dmarc_f*, and smtp_tls* patterns next to the stream, and the duplicate check searches both, so a report saved before the switch is not saved again after it. monthly_indexes is ignored when use_data_streams is on, and a warning says so. If a concrete index or alias already holds a stream's name, as an undated index from a release before 5.0.0 might, saving fails with an error that says so instead of writing into it. Every saved document now also carries an @timestamp field, in both modes: each record's begin date (date_begin) for aggregate reports, which is the report's begin date unless parsedmarc split a report spanning more than a day into daily records; the arrival date for failure reports; and the report's begin date for SMTP TLS reports. These are the same dates that pick the dated index. Documents are now saved with op_type=create, which data streams require; without a document ID this is the same as the default for dated indexes.

Security

• S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report (GHSA-7wqr-f6g9-cm29, affects 11.1.0 and earlier). S3Client.save_report_to_s3() put the sender-controlled report ID into the object key as-is, and sent org_name and the other metadata values as-is. A report ID such as ../../x added path segments to the key, an ID could carry \, %, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit (AWS, "Naming Amazon S3 objects"). botocore rejects non-ASCII metadata with ParamValidationError, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise HTTPClientError: Invalid header value before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the x-amz-meta-* header; the 2 KB metadata limit (AWS, "Working with object metadata") was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are /, every AWS "character to avoid" (\ { } ^ [ ] " < > ~ # | ` and %), control characters, and non-ASCII characters, so % appears in a key only as the escape character. The punctuation real report IDs use, such as :, @, ., and _, is kept. Each metadata value is percent-encoded (%, control characters, and non-ASCII characters) so urllib.parse.unquote() reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in _<hash>.json, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
• Credentials in URLs and the IPinfo API token are no longer written to the log (GHSA-hjr3-rx4c-24cv, affects 11.1.0 and earlier). The [elasticsearch] and [opensearch] hosts settings and the [splunk_hec] url setting accept URLs, and a URL can carry credentials (https://user:secret@host:9200). With --debug, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to log_file. With --verbose or --debug, parsedmarc also forwards the httpx library's HTTP Request: line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in [general] nameservers (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the ipinfo_api_token was logged on every IPinfo lookup as the token query parameter. The password, or the whole userinfo when it has no :, is now replaced with *** in these lines (https://user:***@host:9200), and so is the token query value. Credentials in a [general] nameservers entry are also masked in the DNS pre-flight error, and credentials in either that setting or a --nameservers entry are masked in the --debug reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable https:// or tls:// entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded /, ?, or # (see below), is now rejected before any query is sent. The download URLs ipinfo_url, reverse_dns_map_url, and psl_overrides_url ([general]) can carry credentials too: they were logged as-is at --debug in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with --warnings, --verbose, or --debug) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by ***. A malformed [postgresql] connection_string could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors ([loki] url, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded /, ?, or # ends the URL's authority early (RFC 3986, section 3.2), so httpx reads the text after the colon as the port and raises InvalidURL("Invalid port: 'p4ssw0rd'"), with a chained ValueError that repeats that part of the password. That text reached the log from [loki] url (at startup and on a SIGHUP reload), from [splunk_hec] url (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from ipinfo_url (in the "Failed to save IP database" warning). A malformed psl_overrides_url crashed startup with the traceback, because InvalidURL is not an httpx.HTTPError. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed psl_overrides_url now logs a warning and falls back to the local_psl_overrides_path file when that is set, or else to the bundled file, and a malformed reverse_dns_map_url or ipinfo_url is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no @ in its authority, so the --debug "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last @ (https://***@host:3100). A URL with no credentials but an @ in its path or query is masked the same way in these lines (https://host/path@file is logged as https://***@file). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: user and password (or api_key) for [elasticsearch] and [opensearch], user and password for [postgresql], username and password (or token) for [loki], and token for [splunk_hec]. Reported by helixnebulatech.
• mailsuite 2.3.3 or later is now required. mailsuite 2.3.3 (GHSA-9jx3-p8wm-g8vw) makes the Maildir connection return a message's stored bytes from fetch_message() and rename the file in move_message(), instead of re-serializing it. A report email that cannot be re-serialized (the GHSA-qxwv-2vwj-v828 messages) is therefore now moved to the Invalid folder on Maildir too, instead of staying in the inbox and being logged on every run.

Bug fixes

• Failure reports are now saved to S3. save_report_to_s3() read report["report_metadata"] for every report type except SMTP TLS, but failure reports have no report_metadata (see FailureReport in parsedmarc/types.py), so every failure report raised KeyError since S3 output was added in 7.0.0. The error was counted as an S3 output error, so the batch holding the report was held back. A failure report is now saved under <path>/failure/year=YYYY/month=MM/day=DD/<reported domain>_<hash>.json, dated by its UTC arrival date, with reported_domain and arrival_date metadata. Its hash covers the report type, the arrival date, reported domain, original mail from, original recipient, original envelope ID, source IP address, and the message sample.
• DNS over HTTPS works again under dnspython 2.9 (#935). dnspython 2.9.0 moved DNS over HTTPS from httpx to httpx2, and dns.query.https() now raises ValueError: session parameter must be an httpx2.Client when it is given an httpx.Client. parsedmarc always passed an httpx.Client, so every https:// nameserver failed, and the resolver retried it until the lookup timed out. The shared DoH client is now built from the HTTP library dnspython itself imported: httpx2 on dnspython 2.9+, httpx on 2.7 and 2.8. The nameservers pre-flight check also reported that failure as no PTR record for 1.1.1.1, because get_reverse_dns() returns None for every DNS error. The check now queries the PTR record directly, so it reports no PTR record only when the nameserver answers that the record does not exist, and otherwise shows the resolver's actual error. The httpx2 and httpcore2 loggers are now included in --debug dependency logging, since dnspython 2.9 DoH traffic is logged under those names. An invalid dns_test_address is now reported by name in the pre-flight error. Reported by florianutz.
• Keep distinct aggregate reports when one reporter reuses a Report-ID for different policy domains. RFC 9990 §3.5.1 scopes Report-ID uniqueness to the same domain. Direct report files, mbox files, and mailbox ingestion now all include the policy domain in one shared dedup key, using a JSON array to avoid ambiguous field separators. ASCII domain case differences still count as the same domain (RFC 4343 §3), without changing the returned report. Same-domain repeats remain suppressed — including a report passed both as a file and inside an mbox in one CLI run — and mailbox keys are still committed only after a successful save callback.

11.1.0

Security

• Aggregate reports containing a DOCTYPE declaration are now rejected, which fixes XML default-attribute amplification (GHSA-m47j-wp78-fqjg, affects 11.0.5 and earlier). Entity expansion was already disabled (xmltodict's disable_entities=True), but an internal DTD subset can also declare default attributes: <!ATTLIST z a0 CDATA "v" ... a999 CDATA "v"> followed by 10,000 <z/> elements makes expat report 1,000 attributes on each element, and xmltodict builds 10 million dict entries from 64 KB of XML. Locally that took 4.0 s and 1.4 GB of memory per report; the reporter measured 29 s and 1.4 GB from a 3.8 KB gzip email, and anyone who can send mail to a monitored mailbox can send it. Real aggregate reports never carry a DOCTYPE, so parse_aggregate_report_xml() now aborts with InvalidAggregateReport as soon as expat reaches one, before it reads the subset; the same document is rejected in under a millisecond. The lxml recovery path for malformed XML, which a document reaches when its first parse fails before the DOCTYPE (for example on a malformed XML declaration), applies the same check and now serializes only the root element, so no internal subset reaches the second parse. Reported by helixnebulatech.
• A report email that cannot be re-serialized no longer aborts the whole mailbox batch (GHSA-qxwv-2vwj-v828, affects 11.0.5 and earlier). While walking the message, parse_report_email() serializes the first child of each part, which re-renders any message/rfc822 sub-message, and the Maildir backend re-serializes each message while fetching it, so a message/rfc822 part with a \x1c control character in a header (email.errors.HeaderWriteError) or multiparts nested about 250 levels deep (RecursionError; a shallower nesting is rejected as not a report) raised an exception that the mailbox loop's per-message except ParserError did not catch. The batch aborted before the message could be moved to the Invalid folder, so every later run, and every watch-mode check, aborted on the same message and no report behind it was ever processed. parse_report_email() now raises only ParserError subclasses for any input. A content error raised while fetching a message (email.errors.MessageError or RecursionError) is treated like a parse failure, so the message is moved to the Invalid folder (or deleted with delete_invalid); transport errors still propagate, so a dropped connection cannot get a legitimate report filed as invalid. A move or delete of an invalid message that the backend rejects is now logged and skipped, as it already was for archiving parsed reports, instead of aborting the batch: the Maildir backend moves a message by storing it again through the binary generator, which fails for a message nested that deeply on every Python and, on interpreters whose binary generator also verifies generated headers (e.g. some distribution builds; upstream CPython verifies them only when serializing to text), for the control-character one too. Such a message stays in the inbox, logged on each run, until it is removed by hand or delete_invalid is set. get_dmarc_reports_from_mbox() now reads each message's stored bytes instead of a re-serialized string, so these messages reach the parser and fail with a ParserError naming the message rather than an unrelated exception; an mbox run still stops at such a message, as it does at any message that raises a ParserError other than InvalidDMARCReport. Reported by helixnebulatech.
• Failure report sample filenames are now limited by UTF-8 bytes as well as characters (GHSA-j53g-34vj-g25q, affects 11.0.5 and earlier). get_filename_safe_string() truncated the sample's Subject to 100 characters, but filesystems limit a name to 255 bytes (eCryptfs: 143), so a subject of 84 CJK characters or 63 emoji plus .eml made a 256-byte name. save_output() then failed with ENAMETOOLONG after the batch's JSON and CSV rows had already been appended, so a one-shot run retried the batch indefinitely and watch mode moved it, legitimate reports included, to the Unsaved folder, duplicating rows on each attempt; email_results() failed the same way. Names are now cut to at most 100 UTF-8 bytes without splitting a character, which leaves room for a (N) suffix and the extension under every common limit; ASCII names are unchanged. Reported by helixnebulatech.

Changes

• CSV output now neutralizes spreadsheet formulas (CWE-1236, OWASP CSV Injection). Every text field in the aggregate, failure, and SMTP TLS CSV output comes from a report that a third party sent, and a cell beginning with =, +, -, or @ is evaluated as a formula by Excel, LibreOffice, and Google Sheets, so a reporter could set an org_name such as =HYPERLINK("http://evil","x") and have it run when an analyst opens the CSV. _csv_safe() now prefixes a string field that starts with =, +, -, @, a tab, or a carriage return with a single ', the OWASP-recommended fix, after the existing NUL strip; non-string values such as ASNs and counts are untouched. Compatibility: only cells starting with one of those six characters change, and real report fields almost never do. The JSON output and the data sent to the other output integrations are unchanged. (Closes #929)
• Control characters in sender-controlled strings are escaped before they reach logs and error messages. An RFC 2047 encoded word in a Subject or From header can decode to a raw CR LF (=?utf-8?q?line1=0D=0AINJECTED?=), and parse_report_email() interpolated the decoded subject into its error messages and the From header into its Parsing mail from info line, so a sender could forge extra log lines in the mailbox watch loop's warnings. The new parsedmarc.utils.sanitize_for_log() replaces every non-printable character with its unicode_escape form (\r\n, \x1b, \u2028) rather than stripping it, so the log still shows that the characters were there; the ASCII space, letters in any script, and most emoji pass through, while non-ASCII spaces such as NBSP and U+3000 and format characters such as the zero-width joiner are escaped. It is applied to the subject and From header in parse_report_email(), the SMTP TLS policy-type in its "Invalid policy type" error, the RFC 9990 np, testing, and discovery_method values and the per-record parse error that the aggregate parser logs, and the org_name and report_id in the "Skipping duplicate aggregate report" debug lines in __init__.py and cli.py. (Closes #929)

Bug fixes

• Pool workers now use the IPinfo API token and IP database the parent process selected, whatever the multiprocessing start method. parsedmarc.utils keeps the IPinfo Lite REST API token (_IPINFO_API_TOKEN, set by configure_ipinfo_api()) and the selected IP database path (_IP_DB_PATH, set by load_ip_db()) in process-global variables, and only the parent process sets them. Under the fork start method (the Linux default through Python 3.13) a pool worker inherits them ("All resources of the parent are inherited by the child process", Python multiprocessing docs), but under spawn (the macOS and Windows default) and forkserver (the Linux default from Python 3.14) it does not ("The child process will only inherit those resources necessary to run the process object's run() method"). A worker there started from scratch: its IPinfo API lookups were silently skipped (_ipinfo_api_lookup() returns None when no token is set), and _get_ip_database_path() fell back to the bundled MMDB instead of the freshly downloaded or cached copy the parent had selected, so enrichment (country, ASN) could differ from what the operator configured and from the same run under fork. The CLI parses file inputs through the process pool even with -n 1 (cli.py), so every file-based CLI run on such platforms was affected. parallel_map() now snapshots both globals in the parent and restores them in every worker via its pool initializer (which previously only reconstructed logging, and is renamed _init_worker accordingly), so workers make the same IPinfo API and MMDB selections the parent configured under every start method. The module-level psl_overrides list had a smaller gap: a fresh worker loads the bundled psl_overrides.txt at import time, and only switches to the operator's psl_overrides_path or URL when its first reverse DNS map load reloads the overrides from the config, so base-domain lookups made before that load (the first aggregate report's org_name and the first source IP's reverse DNS name) used the bundled list; the pool initializer now restores the parent's list in place as well. Reported by helixnebulatech. (Closes #928)

11.0.5

Bug fixes

• The PostgreSQL schema now indexes its foreign-key columns, so the PostgreSQL Grafana dashboard's "Aggregate Report Details" table no longer times out. PostgreSQL indexes the primary key a REFERENCES column points at, but not the referencing column itself ("the declaration of a foreign key constraint does not automatically create an index on the referencing columns", PostgreSQL docs, "Foreign Keys"), and create_tables() indexed only two of the schema's seven foreign-key columns (dmarc_aggregate_record.report_id and smtp_tls_policy.report_id). The panel looks up each record's DKIM and SPF results with a correlated subquery on dmarc_aggregate_record_dkim.record_id and dmarc_aggregate_record_spf.record_id, so every row of the panel ran two sequential scans of those tables. With 413,237 aggregate records (5.5 years of reports, PostgreSQL 16), 500 records took 46.1 s, about 92 ms each; a one-year time range of 43,568 records would have taken over an hour, and Grafana returned a 504 long before that. create_tables() now also creates idx_agg_record_dkim_record_id, idx_agg_record_spf_record_id, idx_agg_record_override_record_id, idx_failure_sample_address_report_id, and idx_smtp_tls_failure_detail_policy_id, covering every foreign-key column in the schema; the same 500 records take 15.8 ms and the one-year panel query 3.2 s. create_tables() runs on every start with CREATE INDEX IF NOT EXISTS, so existing databases get the indexes on their next start; the two largest indexes took about 0.3 s each to build on the database above, and the indexed tables reject writes until the startup DDL transaction commits.

11.0.4

Security

• Fixed catastrophic regex backtracking in the plain-text failure report fallback (GHSA-7grx-mj6g-qxmc, affects 8.2.0 through 11.0.3). A text/plain message containing A message claiming to be from you has failed is parsed as an Exim/cPanel failure notice by text_report_regex, \s*([a-zA-Z\s]+):\s(.+). Its leading \s* and the \s inside the field-name class can each consume the same whitespace, and the pattern is unanchored, so a run of whitespace is retried from every starting position and every way of splitting it: the cost grows cubically, about 8x per doubling — on CPython 3.12 (x86-64), 400 spaces took 0.06 s and 1600 spaces took 3.3 s. Anyone who can send mail to a monitored mailbox controls the body, so a message of about 100 KB would stall the mailbox watch loop for days, and since the message is never marked processed it would be re-parsed after every restart. The pattern is now anchored to the start of a line and its classes no longer overlap, so it runs in linear time. The pattern now requires a field name to start a line, with its words separated by single spaces, which is what Exim emits; notices with a tab, doubled space, or line break inside a field name were already rejected, because such names never produced the keys the parser looks up. Reported by mjatin2007.
• Fixed quadratic regex backtracking when stripping XML schema tags from aggregate reports (GHSA-4h9r-c435-j834, affects 4.1.7 through 11.0.3). parse_aggregate_report_xml() removed <xs:schema> tags with </??xs:schema.*>, whose unanchored greedy .*> scanned to the end of the line for every <xs:schema it found, so a line of repeated <xs:schema took quadratic time — 1 MB took 19 s. The pattern runs on report attachments after gzip and zip are inflated, up to the 100 MiB decompression cap, and about 194 KB of gzip inflates to that cap. The pattern now stops at the first > (</?xs:schema[^<>]*>) and runs in linear time. This also fixes a correctness bug: on a minified report with the whole document on one line, the greedy .*> deleted everything from the schema tag to the last > on the line, discarding the report.

Bug fixes

• Parsing a timestamp no longer takes quadratic time on whitespace. human_timestamp_to_datetime() stripped parenthesized comments with the regex \s*\(.*\)\s*, whose leading \s* retried every whitespace run from each starting position. The function runs on every message's Date header, so a long folded header was enough: 200,000 spaces took 18.6 s. This is the same weakness as the two entries above, but a lesser one: the input is a header that mail servers cap in size (Postfix's default is 100 KB, about 5 s) rather than an attachment inflated to the 100 MiB decompression cap, so it is filed as a bug fix without an advisory. Comments are now removed by a single depth-counting pass, which is linear and also handles the nested comments and quoted-pairs that RFC 5322 section 3.2.2 allows and that the old regex only stripped by accident.
• The Grafana dashboard's "Message sources by name and type" table no longer comes up empty when a message source has a name but no type. Whenever a source's reverse DNS base domain is not in base_reverse_dns_map.csv, get_service_from_reverse_dns_base_domain() names the source after that base domain and leaves its type None; a source with no PTR record whose AS domain is not in the map is likewise named after its AS name with no type. For every such source the saved aggregate record has a source_name but no source_type. The panel nests a terms aggregation on source_type.keyword under one on source_name.keyword, without a missing value, so such a source produced a source_name bucket with no source_type sub-bucket, and Grafana's Elasticsearch data source then failed to build the table from the response. With Grafana 13.2.4 (bundled Elasticsearch data source 12.9.2) and the reports in samples/aggregate/, 12 sources had a name but no type and the table showed "No data"; with missing set, it lists all 27 name/type rows. The source_type.keyword aggregation now sets missing to -, the placeholder most of the dashboard's other tables already use for absent values, so those sources are listed with - as their type.

11.0.3

Bug fixes

• An SMTP TLS (RFC 8460) failure report whose failure-details entry sends an empty string for an optional field — e.g. "sending-mta-ip": "" and "receiving-ip": "", which real reporters send for results like sts-policy-fetch-error — no longer crashes the PostgreSQL save. _parse_smtp_tls_failure_details() copied every optional field verbatim whenever the JSON key was present, so an empty string reached the PostgreSQL sending_mta_ip/receiving_ip INET columns and failed with invalid input syntax for type inet: "", discarding the whole report. The same blank values would also have crashed the Elasticsearch and OpenSearch saves: those outputs' sending_mta_ip/receiving_ip fields are IP-typed (Ip() in elastic.py/opensearch.py), and Document.save()'s validation pass rejects an empty string for an Ip() field with a bare ValueError — '' does not appear to be an IPv4 or IPv6 address — raised before any network call, so the whole report was lost rather than indexed with an empty IP. A None or blank/whitespace-only value for any optional failure-detail field (sending-mta-ip, receiving-ip, receiving-mx-hostname, receiving-mx-helo, the additional-information URI, failure-reason-code) is now treated as absent, so it is stored as NULL rather than ''. (Closes #915)
• The RFC 8460 §4.4 additional-information failure-details key is now read. The JSON Report Schema names the key additional-information (its value is described by the placeholder additional-info-uri), but the parser only read the non-RFC key additional-info-uri, so a conformant reporter's URI was silently dropped — including in the RFC's own worked example, samples/smtp_tls/rfc8460.json. The non-RFC additional-info-uri key is still read as a fallback for backward compatibility, and is used when the RFC key is absent or blank.

11.0.2

Changes

• Upgrade the mailsuite requirement to >=2.3.2, which sets the IMAPClient version to >=4.0.0, fixing support for Python 3.14 (Closes #838)

• The prebuilt Docker image (ghcr.io/domainaware/parsedmarc) is roughly 40% smaller to pull (#893). The runtime stage copied the built wheel out of the build stage and deleted it again at the end of the next RUN, but a RUN can only write a whiteout over a layer an earlier instruction already committed: the wheel shipped in every published image and every docker pull downloaded it (10,713,473 bytes of the 11.0.0 image, on both architectures). The wheel is now bind-mounted from the build stage instead, and a bind mount is never committed to a layer. pip install also runs with --no-cache-dir, which drops a further ~99 MB of pip's download cache that the image had been carrying in the same layer as site-packages. Measured on linux/amd64: 272,138,724 compressed bytes across six layers before, 163,757,439 across five after.

• parse_report_file(), extract_report(), and parse_aggregate_report_file() no longer close a file object supplied by the caller, on success or failure. A path input is still opened and closed by the function, and a bytes/bytearray/memoryview input is still wrapped in an internally-created BytesIO that the function closes; but a file-like object the caller passed in may be seeked, tell()'d, or reused afterward, and closing it out from under the caller (as these functions previously did on the success path) makes that impossible. parse_aggregate_report_file() forwards its input straight to extract_report(), so it inherits the fix without changes of its own. extract_report()'s handling of a caller-supplied non-seekable stream is unaffected: its contents are still copied into an internally-owned buffer that the function closes, since that buffer was never the caller's own handle.

• Bare exit(...) calls are now sys.exit(...) throughout the CLI and the maintainer map-tooling scripts. exit is installed into builtins by the site module, not just for interactive sessions, so it isn't guaranteed to exist under python -S or an embedded interpreter that skips site — the failure paths that called it would raise NameError instead of actually exiting. sys.exit is always available and was already used elsewhere in cli.py and find_unknown_base_reverse_dns.py. Also removed the dead, immediately-recomputed index_date stores in the Elasticsearch and OpenSearch aggregate-report savers; behavior is unchanged.

Bug fixes

• find_unknown_base_reverse_dns.py's missing-file checks for base_reverse_dns_map.csv and the known_unknown/PSL-override lists printed a clean error message but fell through into an unhandled FileNotFoundError traceback instead of exiting.
• parse_report_file() now closes the file handle it opens itself for a path input if reading it raises. When input_ is a path, the function opened the file, read it, and closed it with no exception handling in between; an exception raised by read() (e.g. an OSError from the underlying storage) skipped the close, so the descriptor was left to be released only when Python's garbage collector eventually finalized the object — CPython's io.IOBase.__del__ closes an unclosed file on finalization (https://docs.python.org/3/library/io.html) — rather than being closed deterministically. This is the pattern CodeQL's py/file-not-closed query flags, found in a local code-quality scan. The path branch now opens the file with a with block, so the handle is closed on both the success and exception paths. (A caller-supplied file-like object's closing behavior changed again later in this same Unreleased version — see the Changes section above.)
• A SIGHUP configuration reload no longer breaks every subsequent save to Elasticsearch or OpenSearch. With [elasticsearch] or [opensearch] configured in watch mode, reloading the configuration re-registered the search client under the client library's default connection alias and then closed the previous run's clients. The close step re-resolved that alias instead of remembering the client it was created for, so it closed the newly built client and deleted the default alias outright — after which every report save failed with KeyError: "There is no connection with alias 'default'." until parsedmarc was restarted, and the original client was left open. Each backend's handle now closes the exact client it was created for and gives up the alias only while the alias still points at that client. Both the Elasticsearch and OpenSearch backends were affected.
• A configuration reload that fails part-way no longer leaves reports being written to the new Elasticsearch or OpenSearch hosts under the old configuration. The search client is registered under the client library's process-wide default connection alias as soon as it is constructed — before the index migration runs, and before the outputs configured after it are created. When a later step then failed on a SIGHUP reload — for example an [opensearch] section that cannot build its client (an unsupported auth_type, awssigv4 without an aws_region, AWS credentials that will not load) — parsedmarc logged Config reload failed, continuing with previous config and kept the old configuration. But the alias had already been handed to the new client, so every subsequent save resolved it to the new hosts while the index prefixes, suffixes, and index_prefix_domain_map still came from the old configuration. Reports were silently written to a destination that was never successfully configured, with the log saying nothing had changed. The half-built client was never closed either, nor were the other clients (S3, Kafka, PostgreSQL, and so on) built earlier in the same failed reload — the same leak occurred on every attempt of the startup retry loop, which calls the same function. Building the output clients is now all-or-nothing: if any step fails, everything built so far is closed and the module-level state the search backends keep is put back — each configured backend's default alias restored to exactly the client it named beforehand, and the Elasticsearch serverless flag (which decides whether number_of_shards/number_of_replicas are sent when an index is created) to its old value — so a failed reload no longer changes where reports are written, or how indexes are created.
• A SIGHUP configuration reload that fails after the replacement output clients are built now really does keep the previous configuration, instead of leaving the new clients live and the old ones closed. The reload built the replacement clients, then immediately closed the old ones and swapped them in — and only afterwards reloaded the reverse DNS map and the PSL overrides, reloaded the IP database, re-applied the IPinfo API token, copied the new values onto opts, rebuilt the ParserConfig, and converted the watch parameters. A failure in any of those (a typo in local_reverse_dns_map_path or local_psl_overrides_path, an unreadable map or overrides file, an IP database that cannot be resolved) was caught and logged as Config reload failed, continuing with previous config, which was not true: the new output clients were already receiving every report, the old clients were closed and could not be brought back, opts could be half-applied, and load_reverse_dns_map()/load_psl_overrides() — both of which empty their target before reading anything — had left one of the two empty. A bad map path emptied the reverse DNS map, so reverse DNS lookups stopped resolving to known services until some later uncached lookup happened to refill the map from the previous configuration's paths (get_service_from_reverse_dns_base_domain() and get_ip_address_info() both reload a map they find empty). A bad PSL overrides path emptied the override list without ever touching the map, so nothing triggered that lazy reload and base domains were folded without the overrides until parsedmarc was restarted. The whole reload is now staged before anything live is touched: the map is loaded into a fresh dict, the parsedmarc.utils globals the loaders assign are snapshotted, the replacement log_file is opened during staging, and the commit is a run of plain assignments followed by the logging refresh, whose one remaining failure point — closing the replaced log file — is caught and logged rather than allowed to abort a reload that is already live. A log_file that cannot be opened now aborts the reload with the previous log file still attached and still receiving logs; previously the old handler had already been removed, the failed open was only a warning, file logging stayed dead, and a later reload of the same path was a no-op that never retried it. Relatedly, a log_file that could not be opened at startup — a warning there, since there is no previous configuration to keep — is now retried by the next reload even when the setting is unchanged; startup used to record the unopened file as the active one, so the reload saw nothing to do. On failure the replacement clients are closed, the log file opened for the reload is closed (best-effort, so a close error cannot skip the restores that follow it), the search backends' default connection alias is handed back to the old client, and the utils globals are restored, so the log message is accurate. The clients the reload replaces are now closed last, after the new configuration is live, and a failure to close one is logged without disturbing the reload. #906 fixed the other half of this — building the clients themselves is all-or-nothing.

11.0.1

Security

• Fixed a path traversal in the failure report sample filenames written by --output (GHSA-c284-w5m6-jhjm, affects 9.0.6 through 11.0.0). save_output() named each failure report's message sample after the sample's Subject header, falling back to the raw, unsanitized subject whenever sanitizing it produced an empty string. A subject consisting only of path separators and dots — ../../../ or / — sanitizes to nothing, so the raw value reached os.path.join() and the .eml file was written outside the samples directory, or at an absolute path. The subject comes from a message that failed authentication, so anyone who can send mail to a monitored mailbox controls it. The filename is now sanitized at write time and falls back to sample, and the filename_safe_subject key that a library caller may supply alongside the subject is no longer trusted.
• Fixed unbounded decompression of report attachments (GHSA-43qf-f35w-2x4r, affects all versions through 11.0.0). extract_report() inflated gzip and zip attachments with no limit on the output size. The attachment's content is chosen by whoever sent it, and deflate reaches compression ratios of about 1000:1 on degenerate input, so a 100 KB attachment from any sender a monitored mailbox accepts inflated to 100 MB, with a peak of about twice that. Extraction now stops at 100 MiB of decompressed data and raises ParserError. The limit is deliberately not configurable: real DMARC aggregate, failure, and SMTP TLS reports are orders of magnitude smaller.

Bug fixes

• CSV output (failure.csv, and the other CSV renderers) no longer raises _csv.Error on Python 3.10 when a report field contains a NUL character. Python 3.10's csv writer rejects any field containing NUL unless an escapechar is set (CPython issue 97503, a 3.10 regression fixed in 3.11+). Failure report text fields (subject, user agent, authentication results, addresses, and more) come from untrusted mail, so a NUL byte anywhere in one made parsed_failure_reports_to_csv() — and therefore save_output() — raise on 3.10, uncaught by the CLI's except (OSError, ValueError) around save_output. The character is now stripped from CSV fields on every Python version, matching the sanitizing already applied to sample filenames.

11.0.0

Changes

• Breaking: the output and mailbox integrations are now optional extras, so pip install parsedmarc installs the parsing core and a working core CLI instead of every SDK (#883). A 10.x install pulled in the Elasticsearch, OpenSearch, Kafka, AWS (boto3), Azure, Gmail, and Microsoft Graph client libraries whether or not a deployment used any of them — roughly 1 GB of site-packages against roughly 300 MB without — which is a lot to ask of the motivating case, running parsedmarc as a parsing library on a small mail host. The base install now covers the parsing library plus the CLI reading from files, IMAP, Maildir, and mbox, and writing CSV/JSON, Splunk HEC, webhook, and syslog output; those outputs need only httpx and the standard library, so they deliberately have no extra of their own. Everything else moves behind an extra: elastic (Elasticsearch), opensearch (OpenSearch, including the boto3 SigV4 signer), kafka, s3, gelf, loganalytics (Azure Monitor), msgraph (Microsoft 365 mailboxes), and gmail (Gmail API mailboxes), joining the postgresql extra that already existed. CLI users upgrading from 10.x should switch their upgrade command to pip install -U "parsedmarc[all]" — the new umbrella extra — to keep every integration available; add postgresql to the list (parsedmarc[all,postgresql]) if the PostgreSQL backend is in use. all deliberately excludes postgresql because psycopg's prebuilt binary wheels do not exist for every platform, and pip install parsedmarc[all] must not fail on a platform they do not cover. Users of the prebuilt Docker image (ghcr.io/domainaware/parsedmarc) see no change at all: the image now installs [all,postgresql], so it still bundles every integration. A configuration section whose extra is missing no longer fails with an ImportError traceback at startup; it fails fast with a ConfigurationError naming both the section and the command that fixes it, e.g. The [elasticsearch] configuration section requires the elastic extra: pip install parsedmarc[elastic]. Finally, the never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which parsedmarc.utils actually imports and which used to arrive only transitively through dateparser.
• The CLI now accepts --dns-timeout as an alias of --dns_timeout, which is kept for backward compatibility (public since 6.0.0); --dns-retries already used the hyphenated form.
• Failure and SMTP TLS reports are now sent to Kafka as one message per report, matching the aggregate saver's long-documented per-record behavior. save_failure_reports_to_kafka and save_smtp_tls_reports_to_kafka documented per-record sends in every released version, but the code actually sent the entire report list as a single Kafka message (an unreleased docstring pass in #888 had briefly aligned the wording to the buggy code); a large batch could exceed Kafka's default 1MB message limit, and failure reports in particular carry message samples that make that more likely. Both savers now send/flush one message per report, mirroring save_aggregate_reports_to_kafka's existing per-slice shape. This is consumer-visible: consumers now receive individual report objects on these topics rather than one JSON array per batch.
• dashboard-dev-bootstrap.sh (the contributor dashboard dev-stack bootstrap) now works with Podman as well as Docker — it auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (docker compose/docker-compose, podman compose/podman-compose), selectable explicitly with --backend docker|podman or the CONTAINER_BACKEND environment variable. Contributor tooling only; no behavior change for the parsedmarc package itself.

Bug fixes

• [elasticsearch]/[opensearch] number_of_replicas is no longer ignored when number_of_shards is not also set — the parser only read number_of_replicas inside the number_of_shards branch (accidental nesting dating to the 6.4.0-era code), while the documentation lists the two options independently and the client code accepts them independently.
• The CLI now logs a warning for each file path argument that matches no files, instead of silently succeeding with empty results. _expand_file_path_args glob-expands each file_path argument, so a non-existent plain path (e.g. a typo in a cron job) previously vanished as a zero-match glob with no message of any kind; a mailbox-only run that passes no file arguments still logs nothing.

10.5.0

New features

• DNS over HTTPS (DoH) and DNS over TLS (DoT) are now supported through the existing nameservers option (#880). No new configuration option is involved: each entry in the comma-separated list now picks its own transport, and the forms can be mixed in one list. An IP address means plain DNS over UDP and TCP port 53, exactly as before; an https:// URL means DoH; and tls://ip[:port][#hostname] means DoT, where the port defaults to 853 and the optional #hostname supplies the TLS certificate identity (SNI) to verify the server against, matching systemd-resolved's syntax. DoH queries go through a per-process httpx client with its environment trust left at the default, so they honor the standard HTTP_PROXY, HTTPS_PROXY, and NO_PROXY variables — the motivating case, a corporate network that blocks outbound DNS but offers an HTTP proxy, where the proxy also resolves the DoH server's own hostname, so no access to UDP port 53 is needed at all — as well as SSL_CERT_FILE/SSL_CERT_DIR, for a private CA in front of a TLS-inspecting proxy. dnspython's own DoH support cannot do this, because it builds its HTTP client with a custom transport, which is precisely the condition under which httpx ignores proxy environment variables. DoT connections are made directly to TCP port 853 and do not use a proxy. A nameservers list of plain IP addresses — including the default — behaves exactly as it did before, and the pre-flight DNS check at startup now exercises whichever transports are configured, so a bad DoH or DoT entry is reported before any mailbox is opened. The dnspython requirement is now dnspython[doh]>=2.7.0.

Bug fixes

• Failure report MIME parts are now decoded according to their Content-Transfer-Encoding (#882). parse_report_email() read every part's payload without asking the standard library to decode it, so any transfer encoding was left in place. Two things went wrong as a result. A quoted-printable text/rfc822-headers sample part kept its RFC 2045 §6.7 soft line breaks — a = followed by a line ending, with none of the RFC 5322 folding whitespace a header parser needs — which split long headers mid-value; the sample's From header became unparseable, and since the report itself carried no Reported-Domain field, the whole failure report was discarded with TypeError: 'NoneType' object is not subscriptable. Separately, a quoted-printable message/feedback-report part parsed "successfully" with silently corrupted field values, such as an authentication_results of mx.example.com; dmarc=3Dfail. Decoding is applied only to the message/feedback-report part and to the sample part; all other branches (attachments, application/tlsrpt+*, text/plain) still receive the raw payload, since they decode base64 and sniff zip/gzip magic themselves. Parts declaring no transfer encoding, or a 7bit/8bit one, are likewise left alone: there is nothing to undo, and running such a part through the standard library's decoder would round-trip its already-correct text through raw-unicode-escape bytes and replace every non-ASCII character with U+FFFD. Note that a composite part carrying a real transfer encoding is illegal per RFC 2045 §6.4, but reporters send them anyway, and the standard library nests the still-encoded text as a child message rather than decoding it — so those parts are decoded by hand. That nested parse needs one repair first: a soft line break splits a long field onto a continuation line with no colon and no leading whitespace, so the parser treats the remainder as a message body and re-serializing inserts a blank line the encoded text never had, which would otherwise truncate the value at its first split. Any unexpected failure in the new decoding step falls back to the previous behavior, so no report that parsed before can start failing because of it.
• Feedback report field values no longer keep the carriage return of a CRLF line ending. A message/feedback-report part is a MIME body part, so its lines end with CRLF per RFC 5322 §2.1, but the field regex matched the value with .+ under re.MULTILINE, where $ matches before the LF and not before the CR — so every value ended in a stray \r. The bug was masked because the base64 branch above decoded through str(bytes) and then stripped the resulting literal \r escape sequences; now that parts are properly transfer-decoded, real CRLF reaches the regex, and the regex excludes the line ending explicitly.
• A failure report with no Reported-Domain field whose sample has no usable From header is now rejected with a clear message instead of crashing with Unexpected error: 'NoneType' object is not subscriptable. reported_domain is a required string in the FailureReport type and is read directly by the Elasticsearch and OpenSearch outputs, so it cannot be defaulted to None; such a report is unusable and is reported as invalid, naming both the missing field and the unparseable From header. As part of this, InvalidFailureReport raised inside parse_failure_report() now propagates unchanged instead of being caught by the function's catch-all handler and re-wrapped, so pre-existing messages such as "Failure sample is not a valid email" are no longer prefixed with Unexpected error:.

10.4.3

Changes

• Bumped the mailsuite floor to >=2.3.1, picking up the fix for mailsuite.utils.parse_email() raising TypeError instead of ValueError("Not an email") on unparseable (non-email) input under mail-parser 4.6.2 and later (seanthegeek/mailsuite#61) — a potential crash for anything reading non-email files through that API. mail-parser 4.6.2 changed its contract for unparseable input (it now returns a header-less parse result instead of the input string), so mailsuite's not-an-email detection never fired and parsing crashed on the missing From header. Every install of the previous parsedmarc release shipped the affected combination, because mailsuite 2.3.0 — the floor since parsedmarc 10.4.2 — itself requires mail-parser >=4.6.2. parsedmarc's own CLI and library report parsing never call that mailsuite API; they use parsedmarc's separate parse_email implementation, which rejects non-email input gracefully and behaves identically under both mailsuite versions, so this bump removes the affected combination from installs rather than fixing a parsedmarc crash.

10.4.2

Changes

• Bumped the mailsuite floor to >=2.3.0, which raises the transitive mail-parser floor to >=4.6.2 and the transitive cryptography floor to >=50.0.0.
• Releases and docs deployment are now automated by a tag-triggered GitHub Actions workflow: CI-gated build, PyPI publishing via Trusted Publishing, a GitHub Release with attached distributions, a Docker image push to ghcr.io, and a GitHub Pages docs deploy. The manual build.sh and publish-docs.sh scripts are removed.

10.4.1

Bug fixes

• The automatic dkim_results_combined/spf_results_combined and policies_combined/failure_details_combined backfills now cover the indexes named by index_prefix_domain_map and by a previous index_suffix (#868). The startup migration derived its Elasticsearch/OpenSearch index names from the [elasticsearch]/[opensearch] index_prefix/index_suffix options alone, while the save path honors index_prefix_domain_map as well. In a multi-tenant deployment whose per-tenant prefixes come from that map — with no index_prefix set, as the multi-tenant documentation instructs — the guard count therefore ran against dmarc_aggregate*/smtp_tls*, patterns that match none of the real <tenant>_dmarc_aggregate-* indexes. Since the query passes allow_no_indices=True, a zero-match wildcard returns a count of 0, which is indistinguishable from "already backfilled", so the backfill was skipped silently, with no log line at any level, leaving the 10.4.0 dashboards' "DKIM details" table empty and every "SPF details" row labelled none. The migration now targets one index name per tenant prefix in the map — normalized by the same helper the save path uses, so the two cannot drift — plus the unprefixed name, since aggregate and failure reports for a domain that is absent from the map are still saved without a prefix and every report type may have indexes predating the map. A configured index_prefix still wins outright and suppresses the fan-out, matching save-time precedence, so such a deployment never submits an _update_by_query against an index pattern it does not write to. The index_suffix axis is widened the same way: the unsuffixed pattern is now targeted alongside the suffixed one, so documents indexed before the suffix was configured are backfilled too — note that on a shared cluster the unsuffixed pattern also matches other deployments' suffixes. The resolved target index names are logged at debug level, and a SIGHUP configuration reload re-resolves them, so a newly onboarded tenant is covered without a restart. Finally, an index_prefix_domain_map YAML file that is not a mapping of tenant names to lists of domain names, all strings, is now rejected at startup with a ConfigurationError. Every other shape failed silently rather than loudly: a non-string key raised mid-save, a scalar value matched the wrong domains (in on a str is a substring test, so "example.co" in "example.com" is True), and a non-string list item such as tenant_a: [42] simply never compared equal to any domain.
• The legacy published_policy.fo index migration works again, and is back for Elasticsearch as well as OpenSearch. parsedmarc releases before 5.0.0 declared that field as an integer, so their indexes mapped it as long, which cannot hold the multi-value fo settings reports carry (0:1, d:s); 5.0.0 both fixed the declaration and added a migration that rebuilds such an index as a -v2 index with the text/keyword shape. That migration worked against the Elasticsearch 6 clusters of the day, but has been unable to complete since mapping types were removed from Elasticsearch 7 — and they never existed in OpenSearch — for two reasons: it read the field mapping in the old response shape that nests fields under a mapping type name, so the type check always fell through; and its put_mapping() call passed a doc_type argument that neither current client accepts, which would have raised TypeError if the check had ever passed. It now reads either response shape and uses each client's current put_mapping() signature. It is also retry-safe: an attempt interrupted between creating the -v2 index and deleting the original used to leave debris that made every later startup fail on "resource already exists", so the index was never migrated; a leftover target is now discarded first, which is safe precisely because the original is only deleted once the reindex has succeeded. The whole migration is verified end to end against live Elasticsearch 8.19 and OpenSearch 3 clusters, including that recovery path. The index names it checks are also corrected: it takes exact names rather than wildcard patterns, because 5.0.0 introduced date-suffixed index names in the same release that fixed the declaration, so an affected index has no date component — but it can be prefixed or suffixed, since both options date back to 4.1.0, so the configured index_prefix/index_suffix are applied. As with the backfill, an index_suffix also gets the unsuffixed name checked, for history predating the suffix. index_prefix_domain_map prefixes are deliberately not applied, since that option arrived in 8.19.0, long after the mapping was fixed; this migration renames the index it rebuilds — documents are reindexed into the -v2 index before the original is deleted, so none are lost, but the name changes — and so it must not be pointed at indexes belonging to a tenant it is not migrating. The Elasticsearch copy of the migration, removed as unreachable during the #806 client migration, is restored now that the cause is understood. Note that Elasticsearch 8 refuses to open an index created before 7.0, so an affected index reaches a supported cluster only by being carried forward through a reindex, which preserves the old mapping whenever the destination index is pre-created from it — the standard reindex procedure.

10.4.0

New features

• n_procs parallel parsing now covers messages from mbox files and mailbox connections (#147), not just report files passed directly as CLI arguments: IMAP, Microsoft Graph, Gmail API, and Maildir connections, including watch mode. get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox(), and watch_inbox() all gained an n_procs keyword argument. Parallel workers are now a reused process pool for the whole run, with a bounded submission window that keeps at most roughly 2 * n_procs messages in flight at a time so memory stays bounded even for huge mboxes; the main process sends periodic IMAP keepalives while workers parse.
• Added ParserConfig, a single frozen dataclass carrying every parsing/enrichment option (#503): offline mode, IP database path, reverse DNS map and PSL overrides paths/URLs, DNS nameservers/timeout/retries, strip_attachment_payloads, normalize_timespan_threshold_hours, and the three caches the parser and enrichment code share across calls (IP address info, seen aggregate report IDs, and the reverse DNS map). parse_aggregate_report_xml(), parse_aggregate_report_file(), parse_failure_report(), parse_report_email(), parse_report_file(), get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox(), and watch_inbox() all accept a keyword-only config= argument; when it's provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument continues to work unchanged when config= is omitted. Every explicitly constructed ParserConfig owns fresh, isolated caches; omitting config= falls back to the existing process-wide shared default caches, unchanged and identity-preserved (parsedmarc.IP_ADDRESS_CACHE, parsedmarc.SEEN_AGGREGATE_REPORT_IDS, parsedmarc.REVERSE_DNS_MAP). Caches never cross multiprocessing worker boundaries: ParserConfig.__getstate__ drops the three cache fields when a config is pickled for a worker, and each worker accumulates its own from that point on. keep_alive and n_procs remain separate keyword arguments — they control process/worker orchestration, not parsing or enrichment behavior, so they're deliberately not ParserConfig fields. See the new "Using parsedmarc as a library" section of the usage docs for an example.
• Added a "Domain policy" dropdown filter to the Splunk aggregate DMARC dashboard (#854): filters every panel on the published DMARC policy (published_policy.p) or subdomain policy (published_policy.sp), making it easy to see which domains with a none policy are ready to move to quarantine or reject. It sits immediately before the "Message disposition" dropdown and offers the same choices (any/none/quarantine/reject, defaulting to any).
• New [mailbox] options delete_aggregate, delete_failure, delete_smtp_tls, and delete_invalid (#256): control message deletion per report type instead of all-or-nothing. Each defaults to the value of the overall delete option, so existing configurations behave exactly as before; set one to override just that type — for example delete = True with delete_failure = False deletes processed aggregate and SMTP TLS report messages while archiving failure report messages. delete_invalid covers messages that could not be parsed, so they can be kept in the Invalid archive subfolder for debugging even when everything else is deleted. get_dmarc_reports_from_mailbox() and watch_inbox() gained matching delete_aggregate, delete_failure, delete_smtp_tls, and delete_invalid keyword arguments, where None (the default) means "inherit delete". The mutual exclusion between deletion and test mode is now evaluated against the effective per-type flags: any type that would be deleted still raises a ValueError alongside test, but delete = True with all four per-type options explicitly False is now valid with test enabled, since nothing would be deleted.
• New [mailbox] option max_unsaved_retries (default 2): how many times a batch of messages whose reports could not be saved is retried before its messages are moved to the Unsaved archive subfolder and stop being retried. 0 moves messages on the first failed save; negative values are rejected with a ValueError. Failures are counted in memory, per message and per process, so the cap applies across watch-mode checks within one long-running process rather than across separate one-shot runs. See the mailbox-persistence bug fix below for the full behavior; the cap is deliberately low because each retry re-delivers the batch to every output destination that does not deduplicate. get_dmarc_reports_from_mailbox() and watch_inbox() gained a matching max_unsaved_retries keyword argument.
• New [general] option archive_directory: move successfully processed local report files into a dated archive tree (<archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/); files that fail to parse as a report go to <archive_directory>/Invalid/, while files that fail for other reasons (e.g. transient I/O errors) are left in place so a later run can retry them; existing files are never overwritten (numeric suffix); files already under the archive directory are excluded from later runs. Applies only to report files passed directly as local path arguments (#570).

Changes

• The DMARC pass/fail pie chart and over-time chart are now named for DMARC compliance in the OpenSearch Dashboards/Kibana and Splunk dashboards and the Elasticsearch-backed Grafana dashboard (the PostgreSQL Grafana variant has no pass/fail charts and is unchanged): "DMARC compliance" and "DMARC compliance over time" ("DMARC Compliance" / "DMARC Compliance Over Time" in Grafana, which title-cases panel names), replacing the previous mix of "Passed DMARC", "DMARC Passage", and "DMARC passage over time". This matches the existing "% DMARC Compliant" table column and the "Message volume and DMARC compliance by from domain" panels. The Splunk dashboard's pass/fail filter dropdown label was renamed to "DMARC compliant" to match.
• The Grafana dashboard's Guide panel walkthrough now matches the current panel layout, directing readers to the "Top 2000 Message Sources by Reverse DNS" and "Message volume and DMARC compliance by from domain" tables and describing the compliance percentage they show, replacing stale references to a from-domain list "on the right" and a "Message From Header" table that no longer exists.
• get_dmarc_reports_from_mailbox() gained an optional keyword-only save_callback parameter, invoked once per fetched batch with a ParsingResults dict holding only that batch's newly parsed reports, after parsing but before any of the batch's messages are deleted or moved out of reports_folder. Returning False reports the batch as unsaved: its messages stay in place for retry and the aggregate-report dedup cache is not updated for that batch, so the retry reparses those reports rather than skipping them as duplicates. Raising counts as an unsaved batch too — the same retention and max_unsaved_retries bookkeeping runs — and the exception is then re-raised, so a callback that fails by raising (the CLI's does, under fail_on_output_error) is still bounded by the retry cap even when the caller's watch loop swallows the exception, as mailsuite's IMAP and Maildir backends do. Any other return value, including None, commits the batch exactly as before, which is what save_callback=None (the default) does for every batch. As a related, strictly safer side effect of staging the dedup keys per batch, a mid-batch crash now leaves SEEN_AGGREGATE_REPORT_IDS unmodified for that batch instead of partially populated.
• watch_inbox()'s callback is now passed through as save_callback, so it runs once per batch before that batch is archived or deleted, with only that batch's reports, rather than once afterward with the whole check's accumulated results. It may return False to defer a batch to the next check.
• A single-shot CLI run configured with both file/mbox inputs and a mailbox connection now emits output in two passes instead of one: mailbox-derived reports are saved through the new save_callback before get_dmarc_reports_from_mailbox() returns, and file/mbox-derived reports are saved in a second pass afterward. append_json/append_csv are already append-safe and watch mode already called its callback repeatedly, so this changes how often output happens in the single-shot case, not what is written. When a mailbox connection is configured and the file/mbox inputs yielded no reports (none given, or none parsed), the second pass is skipped entirely rather than printing an empty JSON document.

Bug fixes

• Mailbox messages are no longer archived or deleted until the output destinations confirm the reports were saved (#242). Previously get_dmarc_reports_from_mailbox() archived or deleted every successfully parsed message before the CLI wrote anything to Elasticsearch/OpenSearch/Splunk/S3/Kafka/PostgreSQL/etc., so an output outage meant the report was never persisted anywhere while its source message was already gone from the reports folder, with no way to recover it. Each batch is now written to every configured destination first; if any destination fails, the whole batch's messages stay in the reports folder and are retried on the next run or watch-mode check. This is all-or-nothing per batch (archiving on partial success would still permanently lose the data for whichever destination failed) and applies independently of fail_on_output_error, which only controls the process exit code. Unparseable messages carry no report data and are still filed under Invalid (or deleted per delete_invalid) as before. To bound re-delivery to destinations that do not deduplicate, a message whose batch has failed max_unsaved_retries + 1 times (three by default) is moved to <archive_folder>/Unsaved and stops being retried — never deleted, whatever the delete options say. Retries can produce duplicates in Kafka, Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the --output JSON/CSV files; Elasticsearch, OpenSearch, and PostgreSQL dedupe via AlreadySaved, and S3 overwrites the same key. Thanks to @mkilijanek for the original implementation in #823.
• extract_report() now accepts plain uncompressed JSON. The JSON magic-byte constant was written as b"\7b", which Python reads as the octal escape \7 (BEL, 0x07) followed by a literal b — not 0x7B, the { every RFC 8259 JSON object begins with — so the JSON branch never matched and plain JSON input was rejected as "Not a valid zip, gzip, json, or xml file". Every in-tree caller pre-guarded with its own zip/gzip or { check, which masked the dead branch; the practical impact was on extract_report() as a public API and on application/tlsrpt+gzip email attachments whose payload is actually uncompressed JSON. Now b"\x7b".
• An SMTP TLS report with an empty policies list no longer crashes the CLI when index_prefix_domain_map is configured. parse_smtp_tls_report_json() accepts an empty list, but get_index_prefix() unconditionally indexed policies[0], raising IndexError. Such a report has no domain to map, so it is now treated as unmappable — excluded from prefix-mapped output like any other unmapped-domain report — instead of crashing.
• A failed --output file write no longer escapes uncaught. save_output() was the only output destination the CLI did not wrap in a try/except, so a full disk or an unwritable output path crashed the run instead of being recorded like every other destination's failure. It is now caught and recorded as a File output error, which also means it correctly blocks mailbox archiving like any other failure.
• The emailed summary (smtp_host / Microsoft Graph) once again respects index_prefix_domain_map filtering of SMTP TLS reports. process_reports() applies that filter in place to the dict it is given, and it is now given each mailbox batch and the file-derived snapshot rather than the combined results — so the combined dict passed to email_results() no longer inherited the filtering. It is filtered explicitly instead, so a configuration combining index_prefix_domain_map and an email destination excludes unmapped-domain SMTP TLS reports from the emailed summary, matching what was saved.
• Fixed the broken "XML files" link on the Splunk docs page: it pointed at the old splunk/ repository path instead of dashboards/splunk/, where the dashboards have lived since the dashboards/ directory was introduced.
• A report file whose parsing raised an unexpected non-parser exception no longer hangs the CLI forever. The old direct-file parallel implementation spawned a fresh child process per file and blocked the parent on a pipe read; a child that crashed with anything other than a ParserError never wrote to that pipe, so the parent waited indefinitely. The new implementation returns the error as a value and logs Failed to parse <path> instead.
• Direct-file parallel parsing no longer stalls a whole batch on one slow file, and no longer spawns a fresh interpreter per file. The old implementation processed files in hard batches of n_procs, so a single slow file delayed every other file in its batch; the new pooled-worker implementation streams files through a reused pool instead.
• A one-shot mailbox run (no --watch) now honors [general] dns_timeout and dns_retries (#503). The CLI's call to get_dmarc_reports_from_mailbox() never forwarded those two options, so every one-shot mailbox run silently used the library's own hardcoded default (dns_timeout=6.0, dns_retries=0) regardless of what the operator configured; watch-mode runs were unaffected since the watch_inbox() call site did pass them. The CLI now builds a single ParserConfig per run (via the new _build_parser_config() helper) from parsed options and passes it as config= to every parsing entry point — the direct-file, mbox, one-shot mailbox, and watch call sites, plus the SIGHUP config-reload path — instead of forwarding option keyword arguments by hand at each call site.
• Lazily-triggered reverse DNS map loads no longer silently clobber configured PSL overrides with the bundled defaults (#503). get_ip_address_info() and get_service_from_reverse_dns_base_domain() load the reverse DNS map on first use rather than eagerly (this also happens independently in each n_procs worker process, since a worker starts with an empty map), and load_reverse_dns_map() reloads psl_overrides.txt at the same time so map entries that depend on the current overrides fold correctly. That reload previously called load_psl_overrides() with no path/URL arguments, so the first lazy map load in a run overwrote any operator-configured PSL overrides file with the bundled defaults. get_ip_address_info() and get_service_from_reverse_dns_base_domain() now accept psl_overrides_path/psl_overrides_url parameters and thread them through to load_reverse_dns_map(), so a lazy load applies the same configured overrides an eager one would.
• get_dmarc_reports_from_mailbox() and watch_inbox()'s dns_timeout default changed from a stray 6.0 to DEFAULT_DNS_TIMEOUT (2.0 seconds, parsedmarc/constants.py), matching every other parsing entry point; normalize_timespan_threshold_hours also now defaults to the float 24.0 on both, rather than the int 24. This is a minor behavior change for library callers who invoke either function directly without passing dns_timeout: DNS queries now time out after 2 seconds by default instead of 6, matching parse_report_file() and the CLI's own default.

10.3.0

New features

• Added a per-domain DMARC compliance percentage to the aggregate dashboards of every provider (#112): OpenSearch Dashboards/Kibana, Grafana Elasticsearch, Grafana PostgreSQL, and Splunk. The from-domain volume table on each dashboard is now "Message volume and DMARC compliance by from domain", with columns for From Domain, Messages, and % DMARC Compliant.
  • On OpenSearch Dashboards/Kibana, the table is now a TSVB visualization using a Filter Ratio metric (passed messages over total messages per header_from), since the previous agg-based data table can't compute a per-domain ratio. Editing the imported visualization on Kibana 8.x requires first enabling the metrics:allowStringIndices advanced setting.
• Directory paths are now accepted as file_path CLI arguments (#397): a directory expands to the report files inside it using shell-glob semantics (dotfile entries excluded, subdirectories skipped by default), and the new -r/--recursive flag descends into subdirectories and enables ** recursion in glob patterns.

Changes

• Updated the pinned dev-tooling versions and converted the codebase to f-strings: ruff 0.15.21 → 0.16.0 and pyright 1.1.410 → 1.1.411 in the [build] extra. All str.format() calls were converted to f-strings (ruff rules UP030/UP032), except where the f-string form would require Python 3.12's quote reuse inside expressions — those keep .format() with the redundant positional indices removed. Because ruff 0.16.0 greatly expanded its default lint rule set (adding rule families such as BLE, SIM, C4, DTZ, and import sorting, some of which conflict with deliberate house style — e.g. BLE001 flags the parser's intentional broad catches), [tool.ruff.lint] now selects its rule set explicitly: the pre-0.16 defaults (E4/E7/E9/F) plus the modern-type-hint and f-string UP rules. Adopting any of the newly-default rule families is left as a deliberate future per-family decision. Alongside the conversion, a few pre-existing string defects the conversion surfaced were cleaned up: log/exception messages that embedded runs of indentation whitespace via backslash line continuations (and one missing sentence separator in the since-option warning) now read cleanly, and the Splunk HEC output builds its newline-delimited payload by joining a list instead of repeated string concatenation. No functional behavior changes.

Bug fixes

• The Elasticsearch/OpenSearch aggregate dashboards' over-time charts (and the Grafana ES dashboard's summary pies and time series) bucketed on the multi-valued date_range field; a date histogram counts a report once per value, double-counting any report whose begin and end dates fall in different buckets. All date histograms and time-range filters now use the single-valued date_begin, matching the report-begin semantics of the PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
• Aggregate-report policy and authentication result words are now normalized to lowercase (#288): reporters that emit mixed-case values such as Pass no longer create duplicate result categories in outputs and dashboards.
• The results email (SMTP and Microsoft Graph) is no longer sent when no reports were parsed (#200): previously an empty run — an empty inbox, or one where every message was invalid — still emailed a zip of headers-only CSVs. The email step is now skipped with an INFO log when the run produced no aggregate, failure, or SMTP TLS reports.
• The DKIM/SPF alignment-detail tables on the Kibana/OpenSearch Dashboards, Grafana (Elasticsearch), and Splunk aggregate dashboards no longer show a selector × domain × result cross-product (#169). Elasticsearch/OpenSearch flatten the dkim_results/spf_results object arrays (they are dynamic-mapped as object, not nested), so stacking terms aggregations on their subfields produced every combination of selector/domain/result across a report's signatures, each repeating the full message count. Aggregate documents now also carry dkim_results_combined and spf_results_combined — one "selector / domain / result" ("scope / domain / result") string per auth result — and the dashboards aggregate those instead; the Splunk detail panels now pair the values with mvzip/mvexpand. Documents saved by older versions are now backfilled automatically at startup (a non-blocking, idempotent background task); the documented _update_by_query command (see the Elasticsearch docs page) remains available for running the backfill manually. The Grafana "DKIM Alignment Details" panel's dmarcian.com DKIM-checker data link was removed because it required the separate domain/selector columns. The SMTP TLS visualizations had the same class of defect; that fix is described in its own entry below.
• The Kibana/OpenSearch Dashboards aggregate dashboard now includes an "Auth result filters" control panel above the SPF/DKIM details tables, with dropdowns for DKIM selector/domain/result and SPF scope/domain/result, so results can still be filtered by individual components alongside the combined per-signature columns.
• SMTP TLS visualizations had the same cross-product defect as the DKIM/SPF alignment tables (#169): policies is an object array, and each policy's failure_details is itself an object array inside it, so stacking terms aggregations on their subfields cross-products the same way. SMTP TLS documents now also carry policies_combined — one "policy_domain / policy_type" string per policy — and failure_details_combined — one "policy_domain / policy_type / result_type / sending_mta_ip / receiving_ip / receiving_mx_hostname" string per failure detail — and documents saved by older versions are backfilled automatically at startup the same non-blocking, idempotent way as the DKIM/SPF backfill; the equivalent manual _update_by_query command is documented on the Elasticsearch docs page. Also fixed two adjacent dead fields found while making this change: _SMTPTLSFailureDetailsDoc declared additional_information_uri, but add_failure_details passed it to the constructor as additional_information, so it was never actually populated on the declared field; and receiving_mx_hostname, which add_failure_details has always stored, had no field declaration at all. Both are now correctly wired in the Elasticsearch and OpenSearch outputs.
• The Grafana "Map of Message Source Countries" panel's markers now scale with message volume and use a higher-contrast style. Previously the markers were fixed-size 5 px dots at 50% opacity in dark green, which were nearly invisible on the dark basemap.
• Corrected the dead _SPFResult.results (plural) field declaration to result, matching what was always written to it.
• An mbox-only run no longer shows a misleading, permanently-stuck 0it progress bar, and mbox parsing now shows a real per-message progress bar on interactive terminals (#147). The CLI's progress bar only tracks report files passed directly as arguments — n_procs parallel parsing also only applies to those — so runs whose only input was an mbox file displayed an empty bar while messages were parsed sequentially with no visible progress (per-message progress is logged at INFO, hidden in --silent/config-file runs). The empty bar is no longer created, get_dmarc_reports_from_mbox() now wraps its message loop in a tqdm bar that auto-disables on non-TTY output, and the n_procs documentation now states the direct-file-arguments-only scope explicitly.

10.2.4

Bug fixes

• Failure reports without a message/feedback-report part are no longer silently dropped by the Elasticsearch and OpenSearch outputs (#332). Some Exim/cPanel-based gateways send DMARC failure reports as multipart/report without a report-type=feedback parameter and without a machine-readable message/feedback-report part — only a plain-text summary starting with "A message claiming to be from you has failed". parse_report_email() has a dedicated fallback for this format that synthesizes a minimal feedback report containing only Arrival-Date and Source-IP, and parse_failure_report() already defaulted most missing fields — but not feedback_type or authentication_results, both of which the Elasticsearch/OpenSearch save paths access with hard key lookups. The result: every such report was parsed, archived to the failure folder, and then rejected at the sink with Failure report missing required field: 'feedback_type', so it never reached the index. parse_failure_report() now defaults feedback_type to auth-failure (RFC 5965 §3.1) and authentication_results to None (RFC 6591 §3.1), each with a logged warning naming the offending omission — the same treatment the REQUIRED Auth-Failure and Identity-Alignment fields already receive. A sanitized sample (samples/failure/exim_plain_text_only_no_arf_part.eml) and a regression test asserting the sink-required keys are present cover the fallback path.
• The "Aggregate DMARC passage over time" and "Aggregate DMARC message disposition over time" OpenSearch Dashboards visualizations no longer default to a 12-hour X-axis interval (#828). Both date_histogram aggregations were configured with "interval": "auto", which OpenSearch Dashboards sizes off the currently viewed time range rather than the data's actual cadence. Aggregate DMARC reports post one date_range-bucketed data point per reporting period (typically daily), so an auto-computed sub-day bucket size produced empty buckets and a misleading sawtooth/spike pattern. Both visualizations in dashboards/opensearch/opensearch_dashboards.ndjson now hardcode "interval": "d", matching the fix the reporter applied manually in the visualization editor (the "Day" option in the interval picker). Per OpenSearch-Dashboards' _interval_options.ts and parse_interval.ts (src/plugins/data/common/search/aggs/...), the stored interval value must be a short duration-unit code (ms/s/m/h/d/w/M/y) or the literal auto; a spelled-out word like "day" fails parseInterval's regex and throws "day" is not a valid interval. at render time, so "d" is required, not "day".
• The same auto-interval time-bucketing bug from #828 also affected the PostgreSQL Grafana dashboard's "Over Time" panels (dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json): "SPF Results Over Time", "DKIM Results Over Time", "DMARC Pass/Fail Over Time", "Disposition Over Time", "SMTP TLS Sessions Over Time", and one unlabeled summary stat panel all grouped rows with $__timeGroup(<column>, $__interval), where $__interval is a Grafana-computed value that scales with the panel's pixel width and the selected time range, not the reports' actual daily cadence (per the Grafana PostgreSQL query-editor macro docs). All 7 occurrences now hardcode the bucket width to '1d' ($__timeGroup(rpt.begin_date, '1d') / $__timeGroup(tr.begin_date, '1d')), matching the fixed_interval: "1d" already used by the equivalent panels in the companion Elasticsearch/Grafana dashboard (Grafana-DMARC_Reports.json), which was already correct and needed no change. The Elasticsearch/Grafana dashboard's non-time-series pie/stat/table panels that still use a date_histogram with "fixed_interval": "auto" were left as-is: they sum every bucket into a single value or list raw hits, so the bucket width has no visible effect there.
• The same bug also affected the Splunk "DMARC passage over time" and "Message disposition over time" panels (dashboards/splunk/dmarc_aggregate_dashboard.xml). Both | timechart calls had no explicit span, so Splunk auto-computes a bin size targeting 100 buckets across whatever time range is selected; per Splunk's timechart documentation this only coincidentally lands on a 1-day span for the dashboard's default "last 7 days" time-range input; any other selected range (e.g. "last 24 hours" → 30 minutes) reproduces the same gapped/spiked chart. Both queries now set span=1d explicitly, per the Splunk docs' own guidance to use span=1d (not span=24h/span=86400s/span=1440m) for calendar-day boundaries.

10.2.3

Changes

• Migrated the Elasticsearch output to the elasticsearch-py 8.x client (#806): the elasticsearch-dsl dependency is gone (the DSL is bundled in the client as elasticsearch.dsl since 8.18), and the new pin elasticsearch>=8.18,<9 no longer forces urllib3<2 — installs can now resolve urllib3 2.x. Elasticsearch 7.x servers are no longer supported (the 8.x client supports ES 8.x and 9.x servers). OpenSearch users who were pointing the [elasticsearch] config section at an OpenSearch cluster must switch to the [opensearch] section (the 8.x client's product check rejects OpenSearch). Also removed the dead ES 6-era published_policy.fo index migration in migrate_indexes() (unreachable via the 8.x client; the function remains as a no-op for API compatibility).
• The periodic summary email can now be sent via Microsoft Graph (tracking #472). Previously the summary email required [smtp] host, forcing M365 tenants that block legacy SMTP AUTH to stand up a separate SMTP relay just to send from the same mailbox they already read reports from. When [smtp] host is omitted and [msgraph] is configured, the summary is now sent through the same already-authenticated Graph mailbox connection (/users/{mailbox}/sendMail), saved to Sent Items. SMTP is preferred whenever [smtp] host is set, with no fallback to Graph on SMTP failure. [smtp] host/user/password/from are now conditionally required — only when host is present; to keeps its existing requirement either way. A new public email_results_via_msgraph() function shares its subject/message/zip-building logic with the existing email_results() via an extracted _build_report_email_content() helper, so both transports stay in lockstep. Note [smtp] from has no effect on the Graph path — the message's From is always the [msgraph] mailbox — and sending requires the Graph Mail.Send permission (Mail.Send.Shared for a shared mailbox under delegated auth); see the "Sending the summary email via Microsoft Graph" docs section for the full permission matrix.
• Microsoft Graph connection/fetch/send failures now log a single clear ERROR line instead of a bare logger.exception() that hid the actual Azure/Graph error. The line identifies the mailbox, tenant ID, and auth method, and includes the Graph request-id/client-request-id when available (from the OData inner error or, failing that, the raw response headers) — details that matter when contacting Microsoft support. The full traceback is still preserved under --debug.
• Refreshed the [msgraph] documentation: national/sovereign-cloud graph_url values with an explicit warning that the Entra ID auth endpoint isn't independently configurable, a minimal example config for every auth method, a reading-permission matrix alongside the existing sending one, an accurate note on the parsedmarc-named token cache (no migration needed — it's a deliberate backward-compatibility choice from the 9.11.0 mailsuite extraction, not something users have to act on), and a troubleshooting table distinguishing still-live failure modes (admin consent, uninitialized-mailbox folder resolution) from historical ones already fixed at this project's dependency floor (Event loop is closed, invalid ISO timestamps).
• All runtime HTTP calls now use httpx instead of requests, and the dependency set changed accordingly: requests is no longer a runtime dependency (it moved to the [build] extra, where it's still used by the out-of-wheel maintainer script parsedmarc/resources/maps/collect_domain_info.py), and httpx and microsoft-kiota-abstractions are now declared dependencies (httpx for all runtime HTTP calls, microsoft-kiota-abstractions for the Microsoft Graph error handling above). The migration covers the webhook output client, the Splunk HEC client, and the PSL-overrides / IP-database / reverse-DNS-map / IPinfo-API fetches in utils.py. Redirect-following is preserved everywhere (requests follows redirects by default; httpx requires follow_redirects=True, which is now passed explicitly). The PSL-overrides and reverse-DNS-map fetches, which previously had no timeout, now time out after 60 seconds, matching the existing IP-database fetch. Also removed the module-level urllib3.disable_warnings(InsecureRequestWarning) call in the Splunk HEC output: httpx doesn't route through urllib3, so it no longer affected the HEC client, and its only remaining effect was globally silencing insecure-TLS warnings from every other urllib3-based component (Elasticsearch, OpenSearch, boto3) as an import side effect. With that gone, nothing imports urllib3 directly anymore, so it's also no longer a declared dependency (it remains transitively installed).

Bug fixes

• --watch no longer crashes with a raw uncaught traceback on a Microsoft Graph error. The continuous-mode loop previously caught only FileExistsError/ParserError; a Graph auth, API, or transport error during a long-running watch — arguably the most likely real-world failure point, since that's where token/certificate expiry actually surfaces — crashed uncaught. It now gets the same single formatted ERROR line as the other Graph call sites and exits cleanly.
• The documented [smtp] attachment and [smtp] message options are now honored. Both were parsed into opts.smtp_attachment/opts.smtp_message but never passed to either summary-email transport, so a configured custom attachment filename or message body was silently ignored. Both the SMTP (email_results()) and Microsoft Graph (email_results_via_msgraph()) transports now receive them. Visible side effect: the default email body for SMTP summaries is now the long-documented default Please see the attached DMARC results. instead of the previously hardcoded DMARC results for <date>.

10.2.2

Changes

• Removed dead code found while extending test coverage: the unused _SMTPTLSReportDoc.add_policy() helpers in the Elasticsearch and OpenSearch outputs (the save paths construct policy documents directly), a no-op failure_indexes loop in both migrate_indexes() implementations (the parameter is still accepted; no failure-index migrations are currently needed), an unreachable importlib.resources ImportError fallback in parsedmarc.utils (it re-imported the same module, and importlib.resources.files always exists on the supported Python ≥3.10), and an unreachable "Invalid report content" guard in extract_report() (every input branch either assigns the file object or raises first, confirmed by pyright narrowing).

Bug fixes

• parse_email() no longer crashes with KeyError: 'Headers' on messages whose From header is present but empty/unparseable (e.g. a bare From: line). mailparser omits "from" from mail_json for such messages, and the fallback read parsed_email["Headers"] — a key that is never set; the parsed headers are stored under lowercase "headers" (see the assignment at the top of parse_email()). The fallback now reads the correct key and treats an empty parsed header list the same as a missing header, yielding from=None.
• A failed IPinfo API token probe no longer logs "IPinfo API configured". configure_ipinfo_api(..., probe=True) documents that non-fatal probe errors are logged as warnings with the token still accepted, but _ipinfo_api_lookup() returns None on network errors instead of raising, so the probe's exception handler never fired and a probe that couldn't reach the API logged the success message. The probe now checks the lookup result and logs a warning when verification failed. Invalid tokens (401/403) still raise InvalidIPinfoAPIKey.
• Aggregate-report record timestamps are no longer skewed by the host's UTC offset in the Elasticsearch, OpenSearch, and Splunk HEC outputs (#819). A record's interval_begin/interval_end are UTC wall-clock strings (converted to UTC at parse time), but the ES/OpenSearch save paths re-parsed them as local time, shifting the stored date_begin/date_end fields and the daily/monthly index date by the host's UTC offset on any non-UTC host, and the Splunk HEC output computed the aggregate event time the same way. The three call sites now pass assume_utc=True, the same treatment arrival_date_utc received in the #811 fix. The report-level begin_date/end_date parses (ES/OpenSearch dedup query, PostgreSQL output) were investigated and left unchanged — those strings are genuinely local time, so their existing round-trip is already correct.

10.2.1

Changes

• Microsoft Graph connections are now observable (#814). Previously --debug showed nothing about Graph connection activity: the mailbox layer logs under mailsuite.mailbox.graph, token acquisition under azure.identity, and HTTP traffic under httpx/msgraph — none of which parsedmarc configured, so everything below WARNING was silently dropped, and _main() logged nothing around the connection attempt itself. Now:
  • A redacted connection summary is logged at INFO before connecting (auth method, tenant ID, client ID, mailbox, Graph URL), with a --debug detail line adding the certificate path, token-file path, and set/not-set flags for secrets. Secret values (passwords, client secrets, certificate passwords, client assertions) are never logged, and a regression test asserts they don't appear in captured output.
  • A timing line (Microsoft Graph connection initialized in N seconds) is logged after the connection object is created.
  • parsedmarc's --verbose/--debug level now propagates to the dependency loggers (mailsuite, azure, msgraph, httpx, httpcore), so --debug surfaces azure-identity's token-endpoint activity — including the AADSTS error codes that distinguish a local config problem from an Exchange Online / Entra ID one — and the Graph SDK's HTTP requests. At the default level, dependency loggers sit at WARNING so their warnings keep surfacing (now formatted) without new noise. This also benefits the IMAP backend, which lives in mailsuite too.

Bug fixes

• Failure-report timestamps are no longer skewed by the host's UTC offset in the Elasticsearch, OpenSearch, and Splunk HEC outputs (#811, bug 1). arrival_date_utc is a UTC wall-clock string, but the three sinks parsed it into a naive datetime and called .timestamp(), which per the Python docs interprets naive values as local time — so on any non-UTC host, the epoch stored as the ES/OpenSearch arrival_date field, used in the failure-report dedup query, and sent as the Splunk HEC event time was off by the host's UTC offset (1–2 h for most of Europe). human_timestamp_to_datetime() / human_timestamp_to_unix_timestamp() gained an assume_utc keyword that attaches timezone.utc to naive parses, and the arrival_date_utc consumers now use it.
• A system GeoIP file no longer silently overrides the bundled IPinfo database (#810). The IP database path resolution searched standard system paths (e.g. /usr/share/GeoIP/GeoLite2-Country.mmdb, plus CWD-relative names) before the database managed by parsedmarc, so on any host with a distro GeoIP package installed — often as a stale dependency of an unrelated package — every lookup used a country-only (and possibly years-old) database instead of the bundled IPinfo Lite database. That silently disabled ASN enrichment (asn, as_name, as_domain were None for every IP) and with it the ASN-fallback path into the reverse-DNS map. The precedence is now: explicit ip_db_path → the database selected by load_ip_db() (downloaded/cached/bundled) → the bundled copy → system paths as a true last resort. The selected database file is logged at debug level. If you deliberately relied on the automatic system-path pickup to use MaxMind GeoLite2, set ip_db_path explicitly — see the "Using MaxMind GeoLite2" section in the installation docs.

10.2.0

Changes

• Invalid reports now explain why they are invalid. The parser still catches broadly so one malformed report can never crash a batch, but the diagnostics are no longer a bare Not a valid report. parse_report_file() keeps each format parser's specific error as it tries aggregate XML → SMTP TLS JSON → report email, and when all three reject the input it sniffs the content shape to surface the single relevant reason — e.g. Invalid aggregate report: Missing field: 'org_name', Invalid SMTP TLS report: Missing required field: date-range, or Not a recognized report format (not a DMARC aggregate XML report, an SMTP TLS JSON report, or a DMARC report email). This is the message the CLI already logs (Failed to parse <file> - <reason>), so operators see the cause without a code change on their end.
• Original exceptions are now chained. Every parser catch site re-raises with raise … from <original>, so the underlying ExpatError, json.JSONDecodeError, KeyError, and archive/decompression errors are preserved on __cause__ for library callers and tracebacks instead of being flattened into a string. Behavior is otherwise unchanged — the same exception types are still raised.
• Unexpected errors cite their origin in debug mode. When a catch-all branch turns an unforeseen exception into a generic Unexpected error: … / archive / mail-parse failure, the message now appends (raised at <file>:<line>) pinpointing the deepest traceback frame — but only when the parsedmarc logger is at DEBUG level (e.g. the CLI's --debug). Normal-level output is unchanged.
• Added per-version Programming Language :: Python :: 3.10–3.14 (and 3 :: Only) trove classifiers so the PyPI page advertises the supported Python versions. No change to requires-python (>=3.10).

Bug fixes

• Fixed an IndexError when backfilling envelope_from from SPF results. In _parse_report_record(), when an aggregate report's <identifiers> carried an empty envelope_from, the fallback checked the raw auth_results["spf"] list for length but then indexed the filtered new_record["auth_results"]["spf"] list (which only contains results that have a domain). A reporter that sent an SPF auth result with no domain made the filtered list empty while the raw list was non-empty, so the [-1] index raised IndexError and the whole record failed to parse. The two near-identical envelope_from backfill branches (one for a missing identifier, one for an empty one) are now a single code path that gates and indexes the same list, matching the already-correct missing-identifier branch.

10.1.1

Changes

• Require mailsuite>=2.2.2.2 to honor config reloading in the IMAP IDLE loop

10.1.0

Changes

• Graceful shutdown on SIGTERM / SIGINT. Previously SIGTERM (sent by systemctl stop, docker stop, and Kubernetes pod termination) killed the process mid-batch, tearing output writes and silently dropping Kafka producer buffers. parsedmarc now sets a shutdown flag that is polled at safe boundaries: the one-shot CLI checks it between batches, and the watcher passes it as config_reloading so the mailbox backend — including the IMAP IDLE loop — returns once the current batch is processed. Either way the current batch and its output writes finish before the process exits. Ctrl-C is a double-tap: the first press is graceful, the second short-circuits to os._exit(130). This requires the config_reloading-aware IMAP IDLE loop from the pinned mailsuite fork; with a stock mailsuite the IDLE watcher cannot be interrupted cleanly. (PR #794)
  • Output clients are now closed on every exit path via atexit plus a trailing close at the end of _main(), fixing a long-standing leak where one-shot CLI runs and graceful shutdowns never flushed Kafka / closed Elasticsearch / S3 / etc. clients.
  • Example systemd unit in docs/source/usage.md now sets KillSignal=SIGTERM and TimeoutStopSec=60 so systemd waits long enough for the watcher to drain (keep it above mailbox_check_timeout).
• Switch the Kafka client dependency from kafka-python-ng back to kafka-python>=2.3.2 (#795). kafka-python-ng was a fork created while kafka-python was unmaintained; upstream kafka-python is active again, and the now-archived fork is vulnerable to CVE-2026-10142 and CVE-2026-10143, both fixed in kafka-python 2.3.2. Both packages install the same kafka module, so if you are upgrading an existing environment in place with pip, run pip uninstall kafka-python-ng before upgrading parsedmarc so the two distributions don't conflict with each other's files.
  • parsedmarc is compatible with both kafka-python 2.3.2+ and 3.x: kafka-python 3.0 removed the NoBrokersAvailable exception (a failed bootstrap now raises KafkaTimeoutError), and parsedmarc handles whichever the installed version provides.
• The whole codebase (library and tests) now passes pyright with zero errors and warnings, and CI enforces this (plus ruff format --check) on every push and pull request. Pyright is configured in pyproject.toml ([tool.pyright]) and pinned in the [build] extra. The fixes are annotation-level only (Optional parameters, TypedDict-aware signatures on the syslog/GELF save methods, TYPE_CHECKING-aware optional imports for psycopg and the kafka-python 2.x/3.x bootstrap-error fallback) — runtime behavior is unchanged apart from the SyslogClient fix below. Builds on the class-body alias declarations from #797.

Bug fixes

• SyslogClient: constructing a TCP/TLS client with retry_attempts < 1 now raises ValueError instead of silently skipping the connection loop and registering a broken (None) log handler.

10.0.4

Docker

• PostgreSQL backend in the prebuilt image (#792): the image now installs parsedmarc[postgresql], so the optional PostgreSQL output backend (psycopg) works out of the box in the container without a separate pip install.
• arm64 images (#789): the release workflow now builds and publishes a multi-arch manifest for linux/amd64 and linux/arm64, so ghcr.io/domainaware/parsedmarc runs natively on 64-bit ARM hosts such as a Raspberry Pi. All compiled dependencies (psycopg[binary], lxml, maxminddb, cryptography) ship prebuilt aarch64 wheels, so the build adds no source-compilation step.

10.0.3

Bug fixes

• Fix Reply-To (and Delivered-To) addresses being dropped from failure-report samples. parse_email() looked up mailparser's underscored reply_to / delivered_to keys, but mail_json names those headers reply-to / delivered-to, so the lookup always missed and parsed_sample["reply_to"] was always [] regardless of the message. Failure samples now carry their parsed Reply-To addresses through to JSON/CSV output and the Elasticsearch/OpenSearch nested sample.reply_to field.

Dashboard fixes

All failure (RUF) dashboards now render every displayed address (From, To, Reply-To) the same way: Display Name <addr>, or the bare address when there is no display name. The format is assembled at query time from fields (display_name / address) that already exist on previously-indexed reports, so the panels work on historical data, not only on reports stored after upgrading — with one unavoidable exception: a report's Reply-To only appears for reports parsed by 10.0.3 or later. Earlier versions discarded it at parse time (the bug above), so it is absent from older stored reports; recovering it requires re-parsing the original samples.

• Splunk failure dashboard: the email-samples panel showed empty from and reply_to columns — it renamed parsed_sample.headers.from{}{} / parsed_sample.headers.reply-to{}{}, which are mis-cased (the header keys are From / Reply-To) and array-of-array shaped. The panel now builds from and reply_to with an eval that coalesces display_name <address> down to the bare address when there is no display name. (A multi-address Reply-To falls back to addresses-only — a Splunk multi-value-rendering limitation, not a data-loss one.)
• OpenSearch failure dashboard: the column labelled reply_to aggregated sample.headers.in-reply-to.keyword — the In-Reply-To threading header, not the Reply-To address. It now aggregates sample.headers.reply-to.keyword, and that field was added to the dmarc_f* index pattern. To support it, the Elasticsearch/OpenSearch failure writer now flattens the Reply-To header into a display string on sample.headers["reply-to"], mirroring the existing From / To handling. (Re-import the dashboards, or refresh the dmarc_f* index pattern, to pick up the new field.)
• Grafana (Elasticsearch) dashboard: the Failure Samples panel already read sample.headers.reply-to.keyword, but that field previously held the raw [[name, address]] array (split into separate name/address terms). The failure-writer flattening above makes the existing ReplyTo column render a clean Name <address> string — no dashboard change required.
• Grafana (PostgreSQL) dashboard: the Failure Reports panel did not surface the message From header or Reply-To at all (it showed only the envelope Mail From / Rcpt To). Added From (from sample_from) and Reply To (aggregated from dmarc_failure_sample_address) columns.

10.0.2

Changes

• Bump the mailsuite requirement to >=2.2.1, which raises the transitive mail-parser floor to >=4.2.1. This pulls in two upstream fixes:

  • mail-parser 4.2.1 stops returning a phantom ('', '') entry for absent address headers, so parsedmarc no longer indexes an empty Cc/Bcc address ({address: ""}) for every DMARC failure-report sample in Elasticsearch/OpenSearch — and no longer emits it in JSON, S3, or Kafka output.
  • mail-parser 4.2.1 also adopts the stricter address parsing that hardens against CVE-2023-27043 — a Python email-module flaw where an RFC 2822 header containing a special character has the wrong portion identified as the addr-spec, which can let a crafted address bypass email-domain verification.

  (The Reply-To parsing for failure samples and the failure dashboards are tracked separately.)

10.0.1

Changes

• Bump mailsuite requirement to >-2.2.0 to fix an upstream Reply-To header parsing bug for failure samples

10.0.0

Enhancements

Support for RFC 9989 / RFC 9990 / RFC 9991 reports

Adds parsing support for the final DMARC specification (RFC 9989), the new aggregate-report schema (RFC 9990), and the new failure-report format (RFC 9991), while preserving full RFC 7489 / RFC 6591 backward compatibility.

New aggregate-report fields surfaced from the RFC 9990 XSD — added to types, parsing, CSV output, and Elasticsearch/OpenSearch mappings:

• np — non-existent subdomain policy (none/quarantine/reject)
• testing — testing mode flag (n/y); reports whether the published DMARC record sets t=y. It is a new field, not a replacement for pct; the pct mechanism was removed entirely by RFC 9989 Appendix A.6 with no per-message replacement.
• discovery_method — policy discovery method (psl/treewalk)
• generator — report generator software identifier, in report_metadata
• human_result — optional descriptive text on DKIM/SPF auth results (langAttrString; a possible lang attribute is automatically unwrapped)
• xml_namespace — the XML namespace declared on the <feedback> root, if any. RFC 9990 reports declare urn:ietf:params:xml:ns:dmarc-2.0.

pct is no longer present in RFC 9990's PolicyPublishedType and parses as None when absent. fo is still part of RFC 9990 and is preserved when set; it parses as None only when the reporter omits it.

The parser detects an RFC 9990 report from the dmarc-2.0 XML namespace or the presence of any RFC 9990-only field, so namespaceless reports that follow the RFC 9990 shape still receive RFC 9990-aware validation warnings (missing required DKIM selector, removed-in-RFC-9990 policy-override types forwarded / sampled_out). RFC 9990 also added policy_test_mode to the policy-override enumeration; it is parsed and stored unchanged.

For failure reports (RFC 9991), Identity-Alignment and Auth-Failure are split on CFWS-aware commas (whitespace is stripped from each token, per the RFC 9991 ABNF) and a warning is logged when either REQUIRED field is missing.

Several elements that became langAttrString in RFC 9990 (extra_contact_info, error, comment, human_result) are now safely unwrapped when the reporter sends them with a lang attribute.

Backwards compatibility to RFC 7489 is maintained.

PostgreSQL storage backend

New optional PostgreSQL output backend as a lighter-weight alternative to Elasticsearch/OpenSearch, configured via a [postgresql] section (host/port/user/password/database or a libpq connection_string), or equivalently through PARSEDMARC_POSTGRESQL_* environment variables and their _FILE Docker-secret variants like every other backend. Tables are created automatically on first run, and the schema captures the RFC 9990 aggregate fields (np, testing, discovery_method, generator, xml_namespace, and per-result human_result). A Grafana dashboard (dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json) is included. Aggregate and SMTP-TLS reports are de-duplicated via ON CONFLICT; failure reports via an arrival-date / From / To / Subject check mirroring the Elasticsearch backend.

The backend is opt-in: install it with pip install parsedmarc[postgresql] (it pulls in psycopg). It is not a mandatory dependency because the prebuilt psycopg binary wheels are not available for every platform.

Docker-secret support via _FILE env vars

Any PARSEDMARC_{SECTION}_{KEY} environment variable can now also be supplied via a file by appending _FILE to its name (e.g. PARSEDMARC_IMAP_PASSWORD_FILE=/run/secrets/imap_password). The file's contents (with trailing CR/LF stripped) are used as the value. This is the same convention used by the official Postgres, MariaDB, and Redis container images, so credentials no longer have to appear in plain environment: blocks where docker inspect, container logs, and /proc/<pid>/environ would expose them.

When both the direct var and its _FILE companion are set, the file wins. A missing or unreadable file raises ConfigurationError rather than silently falling back to an empty value. The four pre-existing *_file config keys ([general] log_file, [msgraph] token_file, [gmail_api] credentials_file, [gmail_api] token_file) keep their direct-path semantics; wrap them in a Docker secret by doubling the suffix (PARSEDMARC_GMAIL_API_CREDENTIALS_FILE_FILE).

Elastic Cloud Serverless compatibility

New [elasticsearch] serverless config flag (env var PARSEDMARC_ELASTICSEARCH_SERVERLESS). Elastic Cloud Serverless manages sharding and replication itself and rejects the number_of_shards / number_of_replicas index settings with HTTP 400 — previously every write into a Serverless project failed at index-creation time. With the flag set, create_indexes strips those two keys from the settings sent to Elasticsearch and passes any other settings (e.g. refresh_interval) through unchanged. Non-Serverless deployments are unaffected.

Bug fixes

• save_smtp_tls_report_to_s3 was completely broken. parsedmarc/s3.py:save_report_to_s3 unconditionally read report["report_metadata"] when assembling S3 object metadata, but SMTP TLS reports are flat per RFC 8460 §4.3 — they have no report_metadata sub-object — and parse_smtp_tls_report_json correctly stores begin_date as the raw ISO-8601 string from the report. The S3 path branch also assumed begin_date was a datetime and did .year / .month / .day on it. The CLI's surrounding try/except silently swallowed the resulting KeyError, so every SMTP-TLS report quietly failed to upload to S3 in production. Both issues are fixed: SMTP-TLS metadata is now built from the flat report fields directly, and the date is normalized via human_timestamp_to_datetime.
• append_json corrupted JSON output files on the second write. The original implementation opened files in "a+" mode, then seek()ed backwards to overwrite the trailing ] with ,\n before appending more elements. Python's docs are explicit: on POSIX, writes in "a"/"a+" mode always go to EOF regardless of seek position. The result was that every second call onto an existing file produced [...]\n],\n[...]-style corrupted output instead of a single merged JSON array. Anyone running parsedmarc in watch mode with JSON output enabled had aggregate.json / failure.json / smtp_tls.json quietly turning into invalid JSON after the first overlap. Replaced with a read-merge-write pattern: load the existing array (if any), append the new elements, rewrite the whole file. append_csv was not affected — it doesn't seek backwards.
• Removed redundant try/except in parsedmarc/webhook.py. save_aggregate_report_to_webhook / save_failure_report_to_webhook / save_smtp_tls_report_to_webhook each wrapped self._send_to_webhook(...) in a try/except, but _send_to_webhook already catches every Exception itself, so the outer except blocks were unreachable dead code.
• Report files whose names contain glob metacharacters were silently skipped. The CLI expanded every file argument with glob() (parsedmarc/cli.py), which interprets [, ], *, and ? as pattern syntax (see the glob docs). A literal path such as [Netease DMARC Failure Report] Rent Reminder.eml — the bracketed shape many providers use for emailed failure reports — was treated as a character class, matched nothing, and was dropped before reaching the parser, with no error. File arguments that already exist on disk are now taken literally; only non-existent paths are treated as glob patterns, so shell-style wildcards (samples/*.xml) still expand.
• OpenSearch Dashboards reported a mapping conflict on the aggregate index pattern's org_email field. The shipped dashboards/opensearch/opensearch_dashboards.ndjson froze a cached field-list snapshot in which org_email was a text / object conflict, alongside leftover org_email.#text and org_email.#text.keyword subfields — artifacts of a cluster that had once indexed a langAttrString email dict ({"#text": …, "@lang": …}) before the parser unwrapped it. org_email is mapped as Text() and the parser now unwraps a dict email to a plain string, so live data is consistent; cleared the stale conflict and the two artifact subfields from the index pattern, leaving org_email (text) and org_email.keyword so importers no longer see the warning.
• dashboard-dev-bootstrap.sh imported the OpenSearch Dashboards saved objects into the wrong tenant. The script sent securitytenant: global_tenant, but the OpenSearch security plugin reads that header as a tenant name, and global_tenant is a sample custom tenant shipped in the security demo config — not the shared Global tenant, whose token is the literal global. The import succeeded into a separate global_tenant tenant (its own .kibana_<hash>_globaltenant_1 index), so the dashboards were invisible to anyone viewing the Global tenant in OpenSearch Dashboards. Changed the default OSD_TENANT to global. (An empty/omitted securitytenant header is not equivalent — it falls back to the user's configured default tenant, not Global.) This affects the contributor dev stack only, not the shipped dashboards.

Breaking changes

Forensic reports have been renamed to failure reports

Forensic reports have been renamed to failure reports throughout the project to reflect the proper naming of the reports since RFC 7489.

• Core: types.py, __init__.py — ForensicReport→FailureReport, parse_forensic_report→parse_failure_report, report type "failure"
• Output modules: elastic.py, opensearch.py, splunk.py, kafkaclient.py, syslog.py, gelf.py, webhook.py, loganalytics.py, s3.py
• CLI: cli.py — args, config keys, index names (dmarc_failure)
• Docs & dashboards: all markdown, Grafana JSON, OpenSearch NDJSON, Splunk XML

Backward compatibility

• Old function/type names preserved as aliases: parse_forensic_report = parse_failure_report, ForensicReport = FailureReport, etc.
• CLI config accepts both old (save_forensic, forensic_topic) and new keys (save_failure, failure_topic)
• The archive subfolder for failure reports is now Failure (under archive_folder), renamed from Forensic. To avoid a split archive across Forensic/ and Failure/, parsedmarc migrates an existing Forensic subfolder into Failure automatically on startup (best-effort): it renames the folder when no Failure folder exists yet, merges the two when both already exist, and logs-and-skips any mailbox it cannot reorganize (warn, don't crash). This consolidation uses the folder-management API (folder_exists / rename_folder / merge_folders) added in mailsuite 2.1.0, so the required mailsuite version is now >=2.1.0.
• RFC 7489 reports parse with None for RFC 9990-only fields
• Updated dashboards with queries are backward compatible: queries match data indexed under both old (dmarc_forensic* / dmarc:forensic) and new (dmarc_failure* / dmarc:failure) names, so dashboards show data from before and after the rename:
  • OpenSearch Dashboards: Index pattern uses dmarc_f* to match both dmarc_forensic* and dmarc_failure*
  • Splunk: Base search queries (sourcetype="dmarc:failure" OR sourcetype="dmarc:forensic")
  • Elasticsearch/OpenSearch: Duplicate-check searches query across both dmarc_failure* and dmarc_forensic* index patterns

9.11.2

Changes

• base_reverse_dns_types.txt removed; sortlists.py now reads the authoritative type list directly from parsedmarc/resources/maps/README.md. The README's industry list (between new <!-- types-list:start --> / <!-- types-list:end --> HTML-comment markers) is now the single source of truth, eliminating the drift risk between the data file and the documented list. Before validating the map, sortlists.py also normalizes the README block in place: trims whitespace, deduplicates case-insensitively (errors on case-conflicting entries), and sorts entries alphabetically — so adding a new type is just inserting a - New Type line anywhere inside the markers. Also fixes a pre-existing typo in the precedence rules where rule 4 said Web Hosting but the canonical type used in 4,176 map rows is Web Host.
• Maintenance tooling no longer ships in the wheel/sdist. The Python scripts under parsedmarc/resources/maps/ (collect_domain_info.py, classify_unknown_domains.py, detect_psl_overrides.py, detect_rebrands.py, sortlists.py, plus the previously-already-excluded find_bad_utf8.py and find_unknown_base_reverse_dns.py) are maintainer-only batch tooling, not parsedmarc runtime code. They have always been in the repository for convenience but were unnecessarily included in distributions, pulling reviewer attention and contributing nothing to end-user functionality. The build now excludes any .py file under parsedmarc/resources/maps/ whose name doesn't start with an underscore via a single glob pattern (parsedmarc/resources/maps/[!_]*.py), so future maintainer scripts added to that directory are excluded automatically while __init__.py continues to ship. The directory's __init__.py and the runtime data files (base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, psl_overrides.txt) continue to ship — they're loaded at runtime via importlib.resources.files(parsedmarc.resources.maps).

9.11.1

Fixed

• Bump required mailsuite version to >=2.0.2 to address RuntimeError: Event loop is closed failures in the Microsoft Graph mailbox backend (#742).

9.11.0

Changes

• Mailbox backends now live in mailsuite>=2.0.0. The IMAPConnection, MSGraphConnection, GmailConnection, MaildirConnection, and MailboxConnection implementations were extracted into mailsuite 2.0.0 so other projects can reuse the same provider-agnostic interface. parsedmarc's parsedmarc.mail package is now a thin re-export of mailsuite.mailbox; existing imports (from parsedmarc.mail import IMAPConnection, etc.) continue to work unchanged. The CLI passes token_cache_name="parsedmarc" and forwards the existing [msgraph] graph_url config knob so cached AuthenticationRecords and tokens carry over without re-prompting.
• MSGraph backend rewritten on msgraph-sdk (kiota-based). mailsuite 2.0 replaces the retired msgraph-core==0.2.2 REST wrapper with the supported msgraph-sdk client. End-user behavior is unchanged.
• Direct dependencies on msgraph-core, imapclient, google-api-core, google-api-python-client, google-auth-httplib2, google-auth-oauthlib, and google-auth removed. They are now installed transitively via mailsuite[gmail,msgraph]>=2.0.0, which is included as a non-optional dependency so Gmail and Microsoft Graph support remain available out of the box.

9.10.3

Fixed

• Bundled OSD aggregate dashboard reported source-row counts as message volume. Pies, tables, and the choropleth aggregated with count instead of sum(message_count), so panels titled "Message volume…", "Reporting organizations", etc. counted distinct sources rather than emails. Bug present since the dashboard shipped in 9.4.0. Line-chart timeseries, SMTP TLS, and forensic panels were already correct.
• Splunk aggregate "Map of message sources by country" widget had the same count-instead-of-sum(message_count) bug.
• Splunk forensic-samples table dropped events with null From/To/Subject because the base search required those fields to exist (field=*). Replaced with a null-tolerant filter pattern.
• Splunk SMTP TLS Failure details panel returned no rows; Splunk doesn't evaluate field>0 against multivalued JSON-array paths at search time. Switched to a presence filter plus post-stats where failed_sessions>0.

Changes

• Aligned the Splunk dashboards with the OSD source-of-truth: new "Message sources by Autonomous System" panel; added missing dkim_aligned column to DKIM details; green/red colors for true/false on alignment pies and the DMARC-passage timechart; forensic dashboard simplified to OSD's two-panel layout (markdown + samples table); policy_type bucket added to SMTP TLS Domains; minor column / title alignments throughout.

Upgrade notes

Action required — re-import the dashboards. Stored saved objects don't auto-update on parsedmarc upgrade.

• OSD: Stack Management → Saved Objects → Import the new dashboards/opensearch/opensearch_dashboards.ndjson. Switch the import mode from the default "Create new objects with unique IDs" to "Check for existing objects" and enable "Automatically overwrite conflicts". The default mode would import the corrected viz under fresh UUIDs and leave the buggy originals in place, so the dashboards would keep rendering the wrong numbers.
• Splunk: paste each XML in dashboards/splunk/ into the corresponding dashboard's Source editor.

9.10.2

Fixed

• MaildirConnection.fetch_message() now marks messages as read after reading them (sets the S flag and moves the file from new/ to cur/), unless --test is in effect. Previously, a message was processed but its on-disk maildir state was unchanged, so an MUA scanning the same maildir kept showing it as unread. Mirrors the existing mark_read=not test pattern used for MSGraphConnection.
• get_ip_address_info() no longer caches weak-fallback attributions (no PTR + no ASN-domain map match → raw as_name used as source_name, source_type left null). get_reverse_dns() swallows every DNSException as None, so a transient PTR lookup failure (timeout, SERVFAIL, socket error) is indistinguishable from a genuine no-PTR case at that layer — caching the weak result would poison the 4-hour cache with a misattribution that persisted even after the PTR became resolvable again. PTR-backed matches and ASN-domain matches (both stable attributions) are still cached as before; only the specific reverse_dns=None AND type=None AND name=as_name state skips the cache write so the next lookup retries.

9.10.1

Fixed

• Stripped speculative behavior from the IPinfo Lite REST API integration shipped in 9.10.0 after auditing the code against the Lite API docs. The docs state the Lite API has "no daily or monthly limit and provides unlimited access" and document ?token= query-parameter auth only; nothing else removed here is documented for Lite. Removed: the 429 rate-limit and 402 quota-exhausted handling, Retry-After parsing, cooldown state, and the associated warning/recovery logging; the https://ipinfo.io/me account-info probe that expected plan/limit/remaining fields (that endpoint isn't a Lite account endpoint); and the Authorization: Bearer header. Auth is now the documented ?token= query param; the startup probe is a single /lite/1.1.1.1 lookup that logs IPinfo API configured at info level. Retained behavior: 401/403 remains a fatal InvalidIPinfoAPIKey, and any other non-2xx or network error falls back to the bundled/cached MMDB per request.

9.10.0

Changes

• Renamed [general] ip_db_url to ipinfo_url to reflect what it actually overrides (the bundled IPinfo Lite MMDB download URL). The old name is still accepted as a deprecated alias and logs a warning on use; the env-var equivalent is now PARSEDMARC_GENERAL_IPINFO_URL, with PARSEDMARC_GENERAL_IP_DB_URL also still honored.
• Added an optional IPinfo Lite REST API path for country + ASN lookups, so deployments that want the freshest data can query the API directly instead of waiting for the next MMDB release. Configure [general] ipinfo_api_token (or PARSEDMARC_GENERAL_IPINFO_API_TOKEN) and every IP lookup hits https://api.ipinfo.io/lite/<ip> first. At startup the https://ipinfo.io/me account endpoint is hit once to validate the token and log the plan, month-to-date usage, and remaining quota at info level (e.g. IPinfo API configured — plan: Lite, usage: 12345/50000 this month, 37655 remaining). An invalid token exits the process with a fatal error. Rate-limit (HTTP 429) and quota-exhausted (HTTP 402) responses put the API in a cooldown (honoring Retry-After, with a 5-minute / 1-hour default) and fall through to the bundled/cached MMDB; the first event is logged once at warning level and recovery is logged once at info level when the next lookup succeeds. Transient network errors fall through per-request without triggering a cooldown. The API token is never logged.
• Renamed the ASN name and domain fields to match the IPinfo Lite MMDB's native schema: asn_name → as_name and asn_domain → as_domain on every source record (JSON output), and source_asn_name → source_as_name / source_asn_domain → source_as_domain in CSV output (aggregate + failure) and the Elasticsearch / OpenSearch / Splunk integrations. The integer asn / source_asn field is unchanged. The emitted order is asn, as_name, as_domain.

Upgrade notes

• CSV / JSON / Elasticsearch / OpenSearch / Splunk consumers that query the 9.9.0 field names (asn_name, asn_domain, source_asn_name, source_asn_domain) must switch to as_name, as_domain, source_as_name, source_as_domain. Elasticsearch / OpenSearch will add the new mappings on next document write; existing documents indexed under the old names will stay in place until reindexed.

9.9.0

Changes

• Source attribution now has an ASN fallback. Every IP source record carries three new fields — asn (integer, e.g. 15169), asn_name ("Google LLC"), and asn_domain ("google.com") — sourced from the bundled IPinfo Lite MMDB. When an IP has no reverse DNS, get_ip_address_info() uses asn_domain as a lookup into the same reverse_dns_map, and if that misses, falls back to the raw asn_name. reverse_dns and base_domain stay null on ASN-derived rows so consumers can still distinguish PTR-derived from ASN-derived attribution.
• Added source_asn, source_asn_name, source_asn_domain to CSV output (aggregate + forensic), JSON output, and the Elasticsearch / OpenSearch / Splunk integrations. source_asn is mapped as Integer at the schema level so consumers can do range queries and numeric sorts; dashboards can prepend "AS" at display time.
• Expanded base_reverse_dns_map.csv with 500 ASN-domain aliases for the most-routed IPv4 ranges. IPv4-weighted coverage of the bundled ipinfo_lite.mmdb went from ~34% of routed space matching a map entry via ASN domain to ~85%. Every alias is a brand that was already in the map under a different rDNS-base key (e.g. adding comcast.com alongside the existing comcast.net), plus a small number of large operators that previously had no entry. 11 entries were also promoted out of known_unknown_base_reverse_dns.txt because ASN context made their identity unambiguous.
• Added get_ip_address_db_record() in parsedmarc.utils, a single-open MMDB reader that returns country + ASN fields together. get_ip_address_country() is now a thin wrapper. Supports both IPinfo Lite's schema (country_code, asn as "AS15169", as_name, as_domain) and MaxMind's (country.iso_code, autonomous_system_number as int, autonomous_system_organization) in one pass; ASN is normalized to a plain int from either. MaxMind users who drop in their own ASN MMDB get asn + asn_name populated; asn_domain stays null because MaxMind doesn't carry it.

Fixed

• get_ip_address_info() now caches entries for IPs without reverse DNS. Previously the cache write was inside the if reverse_dns is not None branch, so every no-PTR IP re-did the MMDB read and DNS attempt on every call.
• Fixed three bugs in parsedmarc/resources/maps/sortlists.py that silently disabled the type-column validator and sorted the map case-sensitively, contrary to its documented behavior:
  • Validator allowed-values map was keyed on "Type" (capital T), but the CSV header is "type" (lowercase), so every row bypassed validation.
  • Types were read with trailing newlines via f.readlines(), so comparisons would not have matched even if the column name had been right.
  • sort_csv() was called without case_insensitive_sort=True, which moved the sole mixed-case key (United-domains.de) to the top of the file instead of into its alphabetical position.
• Fixed eight pre-existing map rows with invalid or inconsistent type values that the now-working validator surfaced: casing corrections for dhl.com (logistics → Logistics), ghm-grenoble.fr (healthcare → Healthcare), and regusnet.com (Real estate → Real Estate); reclassified lodestonegroup.com from the nonexistent Insurance type to Finance; added missing Religion and Utilities entries to base_reverse_dns_types.txt so it matches the README's industry list.
• Fixed the rt.ru map entry: was classified as RT,Government Media, which conflated Rostelecom (the Russian telco that owns and uses rt.ru) with RT / Russia Today (which uses rt.com). Corrected to Rostelecom,ISP.

Upgrade notes

• Output schema change: CSV, JSON, Elasticsearch, OpenSearch, and Splunk all gain three new fields per row (source_asn, source_asn_name, source_asn_domain). Existing queries and dashboards keep working; dashboards that want to consume the new fields will need to be updated. Elasticsearch / OpenSearch will add the new mappings on next document write.
• Rows for IPs without reverse DNS now populate source_name / source_type via ASN fallback. If downstream dashboards treated "null source_name" as a signal for "no rDNS", switch to checking source_reverse_dns IS NULL instead — that remains the unambiguous signal.

9.8.0

Changes

• Replaced the bundled DB-IP Country Lite database with the IPinfo Lite database (parsedmarc/resources/ipinfo/ipinfo_lite.mmdb, under the Creative Commons Attribution-ShareAlike 4.0 License) for greater IP-to-country lookup accuracy. The download URL / cached filename / packaged module path have all moved from dbip/dbip-country-lite.mmdb to ipinfo/ipinfo_lite.mmdb.
• get_ip_address_country() now reads MMDBs with maxminddb directly and handles both schemas — the IPinfo flat-top-level country_code field and the MaxMind/DBIP nested country.iso_code field — so users who drop in their own MMDB from any of these providers continue to work. The in-disk search list for user-supplied files still includes ipinfo_lite.mmdb, GeoLite2-Country.mmdb, and dbip-country-lite*.mmdb.
• Dropped the geoip2 dependency (its only use was the .country() helper, which is incompatible with the IPinfo schema). Added maxminddb as a direct dependency — it was already installed transitively through geoip2, so this is a no-op for most environments.

Upgrade notes

• Callers that imported parsedmarc.resources.dbip directly need to switch to parsedmarc.resources.ipinfo. The parsedmarc.resources.dbip module has been removed.
• Callers that imported geoip2 only because parsedmarc depended on it will need to add it to their own requirements. parsedmarc itself no longer depends on geoip2.
• The auto-update download URL used by previous parsedmarc versions (.../dbip/dbip-country-lite.mmdb) is no longer hosted on master; those versions will fail to download and fall back to their bundled copy, which is the documented behavior of load_ip_db().

9.7.1

Changes

• Ported DNS lookup reliability improvements from checkdmarc 5.15.x:
  • Per-query UDP timeout is now capped at min(1.0, timeout) in query_dns(), so a single dropped UDP datagram no longer consumes the entire lifetime budget — dnspython retries UDP within the lifetime window (mirroring dig's default +tries=3). With multiple nameservers configured, the same cap also makes a slow or broken nameserver fall through to the next quickly.
  • With multiple nameservers configured, the resolver lifetime is now timeout × len(nameservers) so each nameserver gets its own timeout budget for failover rather than sharing one overall deadline.
  • New retries kwarg on query_dns(), get_reverse_dns(), and get_ip_address_info() retries the whole query on transient errors (LifetimeTimeout, NoNameservers/SERVFAIL, and OSError during TCP fallback). NXDOMAIN and NoAnswer remain non-retryable. Default is 0 (no behavior change for existing callers).
  • Threaded dns_retries through the parser API (parse_report_file, parse_aggregate_report_xml, parse_forensic_report, parse_report_email, get_dmarc_reports_from_mbox, get_dmarc_reports_from_mailbox, watch_inbox).
• Added --dns-retries N CLI flag and dns_retries INI option ([general] section, also surfaced via PARSEDMARC_GENERAL_DNS_RETRIES env var).
• Centralized DNS defaults in parsedmarc.constants: DEFAULT_DNS_TIMEOUT, DEFAULT_DNS_MAX_RETRIES, and RECOMMENDED_DNS_NAMESERVERS (a cross-provider mix — ("1.1.1.1", "8.8.8.8") — for callers that want public-resolver failover). The existing default nameservers (all-Cloudflare) are preserved for backward compatibility; callers opt in by passing nameservers=RECOMMENDED_DNS_NAMESERVERS.

9.7.0

Changes

• psl_overrides.txt is now automatically downloaded at startup (and on SIGHUP in watch mode) by load_psl_overrides() in parsedmarc.utils, with the same URL / local-file / offline fallback pattern as the reverse DNS map. It is also reloaded whenever load_reverse_dns_map() runs, so base_reverse_dns_map.csv entries that depend on a recent overrides entry resolve correctly without requiring a new parsedmarc release.
• Added the local_psl_overrides_path and psl_overrides_url configuration options ([general] section, also surfaced via PARSEDMARC_GENERAL_* env vars) to override the default PSL overrides source.
• Expanded base_reverse_dns_map.csv substantially in this release, following a multi-pass classification pass across the unknown/known-unknown lists (net ~+1,000 entries).
• Added Religion and Utilities to the allowed type values in base_reverse_dns_types.txt and documented them in parsedmarc/resources/maps/README.md.
• Added parsedmarc/resources/maps/collect_domain_info.py — a bulk enrichment collector that runs WHOIS, a size-capped HTTP GET, and A/AAAA + IP-WHOIS for every unmapped reverse-DNS base domain, writing a compact TSV suitable for a single classification pass. Respects psl_overrides.txt and skips full-IP entries.
• Added parsedmarc/resources/maps/detect_psl_overrides.py — scans unknown_base_reverse_dns.csv for IP-containing entries that share a brand suffix, auto-appends the suffix to psl_overrides.txt, folds affected entries in all three list files, and removes any remaining full-IP entries for privacy.
• find_unknown_base_reverse_dns.py now drops full-IP entries at ingest so customer IPs never enter the pipeline.
• Documented the full map-maintenance workflow (privacy rule, auto-override detection, conservative classification, known-unknown handling) in the top-level AGENTS.md.

Fixed

• Reverse-DNS base domains containing a full IPv4 address (four dotted or dashed octets) are now blocked from entering base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, and unknown_base_reverse_dns.csv. Customer IPs were previously possible in these lists as part of ISP-generated reverse-DNS subdomain patterns. The filter is enforced in find_unknown_base_reverse_dns.py, collect_domain_info.py, and detect_psl_overrides.py. The existing lists were swept and all pre-existing IP-containing entries removed.

9.6.0

Changes

• The included DB-IP Country Lite database is now automatically updated at startup (and on SIGHUP in watch mode) by downloading the latest copy from GitHub, unless the offline flag is set. Falls back to a previously cached copy or the bundled database on failure. This allows the IP-to-country database to stay current without requiring a new package release.
• Updated the included DB-IP Country Lite database to the 2026-04 release.
• Added the ip_db_url configuration option (PARSEDMARC_GENERAL_IP_DB_URL env var) to override the default download URL for the IP-to-country database.

9.5.5

Fixed

• Output client initialization now retries up to 4 times with exponential backoff before exiting. This fixes persistent Connection refused errors in Docker when OpenSearch or Elasticsearch is momentarily unavailable at startup.
• Use tuple format for http_auth in OpenSearch and Elasticsearch connections, matching the documented convention and avoiding potential issues if the password contains a colon.
• Fix current_time format for MSGraphConnection (current-time) (PR #708)

Changes

• Added debug logging to all output client initialization (S3, syslog, Splunk HEC, Kafka, GELF, webhook, Elasticsearch, OpenSearch).
• DEBUG=true and PARSEDMARC_DEBUG=true are now accepted as short aliases for PARSEDMARC_GENERAL_DEBUG=true.

9.5.4

Fixed

• Maildir fetch_messages now respects the reports_folder argument. Previously it always read from the top-level Maildir, ignoring the configured reports folder. fetch_message, delete_message, and move_message now also operate on the correct active folder.
• Config key aliases for env var compatibility: [maildir] create and path are now accepted as aliases for maildir_create and maildir_path, and [msgraph] url for graph_url. This allows natural env var names like PARSEDMARC_MAILDIR_CREATE to work without the redundant PARSEDMARC_MAILDIR_MAILDIR_CREATE.

9.5.3

Fixed

• Fixed FileNotFoundError when using Maildir with Docker volume mounts. Python's mailbox.Maildir(create=True) only creates cur/new/tmp subdirectories when the top-level directory doesn't exist; Docker volume mounts pre-create the directory as empty, skipping subdirectory creation. parsedmarc now explicitly creates the subdirectories when maildir_create is enabled.
• Maildir UID mismatch no longer crashes the process. In Docker containers where volume ownership differs from the container UID, parsedmarc now logs a warning instead of raising an exception. Also handles os.setuid failures gracefully in containers without CAP_SETUID.
• Token file writes (MS Graph and Gmail) now create parent directories automatically, preventing FileNotFoundError when the token path points to a directory that doesn't yet exist.
• File paths from config (token_file, credentials_file, cert_path, log_file, output, ip_db_path, maildir_path, syslog cert paths, etc.) now expand ~ and $VAR references via os.path.expanduser/os.path.expandvars.

9.5.2

Fixed

• Fixed ValueError: invalid interpolation syntax when config values (from env vars or INI files) contain % characters, such as in passwords. Disabled ConfigParser's %-based string interpolation.

9.5.1

Changes

• Correct ISO format for MSGraphConnection timestamps (PR #706)

9.5.0

Added

• Environment variable configuration support: any config option can now be set via PARSEDMARC_{SECTION}_{KEY} environment variables (e.g. PARSEDMARC_IMAP_PASSWORD, PARSEDMARC_SPLUNK_HEC_TOKEN). Environment variables override config file values but are overridden by CLI arguments.
• PARSEDMARC_CONFIG_FILE environment variable to specify the config file path without the -c flag.
• Env-only mode: parsedmarc can now run without a config file when PARSEDMARC_* environment variables are set, enabling fully file-less Docker deployments.
• Explicit read permission check on config file, giving a clear error message when the container UID cannot read the file (e.g. chmod 600 with a UID mismatch).

9.4.0

Added

• Extracted load_reverse_dns_map() utility function in utils.py for loading the reverse DNS map independently of individual IP lookups.
• SIGHUP reload now re-downloads/reloads the reverse DNS map, so changes take effect without restarting.
• Add premade OpenSearch index patterns, visualizations, and dashboards

Changed

• When index_prefix_domain_map is configured, SMTP TLS reports for domains not in the map are now silently dropped instead of being output. Unlike DMARC, TLS-RPT has no DNS authorization records, so this filtering prevents processing reports for unrelated domains.
• Bump OpenSearch support to < 4

Fixed

• Fixed get_index_prefix using wrong key (domain instead of policy_domain) for SMTP TLS reports, which prevented domain map matching from working for TLS reports.
• Domain matching in get_index_prefix now lowercases the domain for case-insensitive comparison.

9.3.1

Breaking changes

• Elasticsearch and OpenSearch now verify SSL certificates by default when ssl = True, even without a cert_path
• Added skip_certificate_verification option to the elasticsearch and opensearch configuration sections for consistency with splunk_hec

Fixed

• Splunk HEC skip_certificate_verification now works correctly
• SMTP TLS reports no longer fail when saving to multiple output targets (e.g. Elasticsearch and OpenSearch) due to in-place mutation of the report dict
• Output client initialization errors now identify which module failed (e.g. "OpenSearch: ConnectionError..." instead of generic "Output client error")

9.3.0

Added

• SIGHUP-based configuration reload for watch mode — update output destinations, DNS/GeoIP settings, processing flags, and log level without restarting the service or interrupting in-progress report processing.
  • Use systemctl reload parsedmarc when running under systemd.
  • On a successful reload, old output clients are closed and recreated.
  • On a failed reload, the previous configuration remains fully active.
• close() methods on GelfClient, KafkaClient, SyslogClient, WebhookClient, HECClient, and S3Client for clean resource teardown on reload.
• config_reloading parameter on all MailboxConnection.watch() implementations and watch_inbox() to ensure SIGHUP never triggers a new email batch mid-reload.
• Elasticsearch and OpenSearch connections are now tracked and cleaned up on reload via _close_output_clients().
• Extracted _parse_config_file() and _init_output_clients() from _main() in cli.py to support config reload and reduce code duplication.

Fixed

• get_index_prefix() crashed on failure reports with TypeError due to report() instead of report[] dict access.
• Missing exit(1) after IMAP user/password validation failure allowed execution to continue with None credentials.

9.2.1

Added

• Better checking of msgraph configuration (PR #695)

Changed

• Updated dbip-country-lite database to version 2026-03
• DNS query error logging level from warning to debug

9.2.0

Added

• OpenSearch AWS SigV4 authentication support (PR #673)
• IMAP move/delete compatibility fallbacks (PR #671)
• fail_on_output_error CLI option for sink failures (PR #672)
• Gmail service account auth mode for non-interactive runs (PR #676)
• Microsoft Graph certificate authentication support (PRs #692 and #693)
• Microsoft Graph well-known folder fallback for root listing failures (PR #618 and #684 close #609)

Fixed

• Pass mailbox since filter through watch_inbox callback (PR #670 closes issue #581)
• parsedmarc.mail.gmail.GmailConnection.delete_message now properly calls the Gmail API (PR #668)
• Avoid extra mailbox fetch in batch and test mode (PR #691 closes #533)

9.1.2

Fixes

• Fix duplicate detection for normalized aggregate reports in Elasticsearch/OpenSearch (PR #666 fixes issue #665)

9.1.1

Fixes

• Fix the use of Elasticsearch and OpenSearch API keys (PR #660 fixes issue #653)

Changes

• Drop support for Python 3.9 (PR #661)

9.1.0

Enhancements

• Add TCP and TLS support for syslog output. (#656)
• Skip DNS lookups in GitHub Actions to prevent DNS timeouts during tests timeouts. (#657)
• Remove microseconds from DMARC aggregate report time ranges before parsing them.

9.0.10

• Support Python 3.14+

9.0.9

Fixes

• Validate that a string is base64-encoded before trying to base64 decode it. (PRs #648 and #649)

9.0.8

Fixes

• Fix logging configuration not propagating to child parser processes (#646).
• Update mailsuite dependency to ?=1.11.1 to solve issues with iCloud IMAP (#493).

9.0.7

Fixes

• Fix IMAP since option (#PR 645 closes issues #581 and #643).

9.0.6

Fixes

• Fix #638.
• Fix/clarify report extraction and parsing behavior for multiple input types (bytes, base64 strings, and file-like objects).
• Fix type mismatches that could cause runtime issues in SMTP emailing and CLI option handling.

Improvements

• Improve type hints across the library (Pylance/Pyright friendliness) and reduce false-positive linter errors.
• Emails in Microsoft 365 are now marked read as they are read. This provides constancy with other mailbox types, and gives you a indication of when emails are being read as they are processed in batches. (Close #625)

Compatibility / Dependencies

• Set Python requirement to >=3.9,<3.14.
• Bump mailsuite requirement to >=1.11.0.

9.0.5

Fixes

• Fix report type detection introduced in 9.0.4.

9.0.4 (Yanked)

Fixes

• Fix saving reports to OpenSearch (#637)
• Fix parsing certain DMARC failure/forensic reports
• Some fixes to type hints (incomplete, but published as-is due to the above bugs)

9.0.3

Fixes

• Set requires-python to >=3.9, <3.14 to avoid this bug

9.0.2

Improvements

• Type hinting is now used properly across the entire library. (#445)

Fixes

• Decompress report files as needed when passed via the CLI.
• Fixed incomplete removal of the ability for parsedmarc.utils.extract_report to accept a file path directly in 8.15.0.

Breaking changes

This version of the library requires consumers to pass certain arguments as keyword-only. Internally, the API uses a bare * in the function signature. This is standard per PEP 3102 and as documented in the Python Language Reference.

9.0.1

Fixes

• Allow multiple records for the same aggregate DMARC report in Elasticsearch and Opensearch

9.0.0 (yanked)

• Normalize aggregate DMARC report volumes when a report timespan exceeds 24 hours

8.19.1

• Ignore HTML content type in report email parsing (#626)

8.19.0

• Add multi-tenant support via an index-prefix domain mapping file
• PSL overrides so that services like AWS are correctly identified
• Additional improvements to report type detection
• Fix webhook timeout parsing (PR #623)
• Output to STDOUT when the new general config boolean silent is set to False (Close #614)
• Additional services added to base_reverse_dns_map.csv

8.18.9

• Complete fix for #687 and more robust report type detection

8.18.8

• Fix parsing emails with an uncompressed aggregate report attachment (Closes #607)
• Add --no-prettify-json CLI option (PR #617)

8.18.7

Removed improper spaces from base_reverse_dns_map.csv (Closes #612)

8.18.6

• Fix since option to correctly work with weeks (PR #604)
• Add 183 entries to base_reverse_dns_map.csv
• Add 57 entries to known_unknown_base_reverse_dns.txt
• Check for invalid UTF-8 bytes in base_reverse_dns_map.csv at build
• Exclude unneeded items from the parsedmarc.resources module at build

8.18.5

• Fix CSV download

8.18.4

• Fix webhooks

8.18.3

• Move __version__ to parsedmarc.constants
• Create a constant USER_AGENT
• Use the HTTP User-Agent header value parsedmarc/version for all HTTP requests

8.18.2

• Merged PR #603
  • Fixes issue #595 - CI test fails for Elasticsearch
    • Moved Elasticsearch to a separate Docker service container for CI testing
    • Dropped Python 3.8 from CI testing
  • Fixes lookup and saving of DMARC forensic reports in Elasticsearch and OpenSearch
• Updated fallback base_reverse_dns_map.csv, which now includes over 1,400 lines
• Updated included dbip-country-lite.mmdb to the June 2025 release
• Automatically fall back to the internal base_reverse_dns_map.csv if the received file is not valid (Fixes #602)
  • Print the received data to the debug log

8.18.1

• Add missing https:// to the default Microsoft Graph URL

8.18.0

• Add support for Microsoft national clouds via Graph API base URL (PR #590)
• Avoid stopping processing when an invalid DMARC report is encountered (PR #587)
• Increase http.client._MAXHEADERS from 100 to 200 to avoid errors connecting to Elasticsearch/OpenSearch (PR #589)

8.17.0

• Ignore duplicate aggregate DMARC reports with the same org_name and report_id seen within the same hour (Fixes #535)
• Fix saving SMTP TLS reports to OpenSearch (PR #585 closed issue #576)
• Add 303 entries to base_reverse_dns_map.csv

8.16.1

• Failed attempt to ignore aggregate DMARC reports seen within a period of one hour (#535)

8.16.0

• Add a since option to only search for emails since a certain time (PR #527)

8.15.4

• Fix crash if aggregate report timespan is > 24 hours

8.15.3

• Ignore aggregate reports with a timespan of > 24 hours (Fixes #282)

8.15.2

• Require mailsuite>=1.9.18
  • Pins mail-parser version at 3.15.0 due to a parsing regression in mail-parser 4.0.0
  • Parse aggregate reports with empty <auth_results>
  • Do not overwrite the log on each run (PR #569 fixes issue #565)

8.15.1

• Proper IMAP namespace fix (Closes issue #557 and issue #563)
  • Require mailsuite>=1.9.17
  • Revert PR #552
• Add pre-flight check for nameservers (PR #562 closes issue #543)
• Reformat code with ruff

8.15.0

• Fix processing of SMTP-TLS reports (#549), which broke in commit 410663d(PR #530)
  • This PR enforced a stricter check for base64-encoded strings, which SMTP TLS reports from Google did not pass
  • Removing the check introduced its own issue, because some file paths were treated as base64-encoded strings
• Create a separate extract_report_from_file_path() function for processioning reports based on a file path
• Remove report extraction based on a file path from extract_report()

8.14.2

• Update base_reverse_dns_map.csv to fix over-replacement on f3a5f10 (PR #553)

8.14.1

• Failed attempt to fix processing of SMTP-TLS reports (#549)

8.14.0

• Skip invalid aggregate report rows without calling the whole report invalid
  • Some providers such as GoDaddy will send reports with some rows missing a source IP address, while other rows are fine
• Fix Dovecot support by using the separator provided by the IMAP namespace when possible (PR #552 closes #551)
• Only download base_reverse_dns_map.csv once (fixes #542)
• Update included base_reverse_dns_map.csv
  • Replace University category with Education to be more inclusive
• Update included dbip-country-lite.mmdb

8.13.0

• Add Elastic/OpenSearch index prefix option (PR #531 closes #159)
• Add GELF output support (PR #532)

8.12.0

• Fix for deadlock with large report (#508)
• Build: move to kafka-python-ng (#510)
• Fix new config variables previously not propagated in the code (#524)
• Fixes for kafka integration (#522)
• Fix if base_domain is None before get_service_from_reverse_dns_base_domain (#514)
• Update base_reverse_dns_map.csv

8.11.0

• Actually save source_type and source_name to Elasticsearch and OpenSearch
• Reverse-lookup cache improvements (PR #501 closes issue #498)
• Update the included dbip-country-lite.mmdb to the 2024-03 version
• Update base_reverse_dns_map.csv
• Add new general config options (closes issue #500)
  • always_use_local_files - Disables the download of the reverse DNS map
  • local_reverse_dns_map_path - Overrides the default local file path to use for the reverse DNS map
  • reverse_dns_map_url - Overrides the default download URL for the reverse DNS map

8.10.3

• Fix flaws in base_reverse_dns_map.csv

8.10.2

• Fix flaws in base_reverse_dns_map.csv

8.10.1

• Fix flaws in base_reverse_dns_map.csv

8.10.0

• Fix MSGraph UsernamePassword Authentication (PR #497)
• Attempt to download an updated base_reverse_dns_map.csv at runtime
• Update included base_reverse_dns_map.csv

8.9.4

• Update base_reverse_dns_map.csv

8.9.3

• Revert change in 8.9.2

8.9.2

• Use Uncategorized instead of None as the service type when a service cannot be identified

8.9.1

• Fix broken CLI by removing obsolete parameter from cli_parse call (PR #496 closes issue #495)

8.9.0

• Fix broken cache (PR #494)
• Add source name and type information based on static mapping of the reverse DNS base domain
  • See this documentation for more information, and to learn how to help!
• Replace multiprocessing.Pool with Pipe + Process (PR #491 closes issue #489)
• Remove unused parallel arguments (PR #492 closes issue #490)

8.8.0

• Add support for OpenSearch (PR #481 closes #480)
• Fix SMTP TLS reporting to Elasticsearch (PR #470)

8.7.0

• Add support for SMTP TLS reports (PR #453 closes issue #71)
• Do not replace content in forensic samples (fix #403)
• Pin msgraph-core dependency at version 0.2.2 until Microsoft provides better documentation (PR #466 Close #464)
• Properly handle base64-encoded email attachments (PR #453)
• Do not crash when attempting to parse invalid email content (PR #453)
• Ignore errors when parsing text-based forensic reports (PR #460)
• Add email date to email processing debug logs (PR #462)
• Set default batch size to 10 to match the documentation (PR #465)
• Properly handle none values (PR #468)
• Add Gmail pagination (PR #469)
• Use the correct msgraph scope (PR #471)

8.6.4

• Properly process aggregate reports that incorrectly call identifiers identities
• Ignore SPF results in aggregate report records if the domain is not provided

8.6.3

• Add an error message instead of raising an exception when an aggregate report time span is greater than 24 hours

8.6.2

• Use zlib instead of Gzip to decompress more .gz files, including the ones supplied by Mimecast (Based on #430 closes #429)

8.6.1

• Fix handling of non-domain organization names (PR #411 fixes issue #410)
• Skip processing of aggregate reports with a date range that is too long to be valid (PR #408 fixes issue #282)
• Better error handling for Elasticsearch queries and file parsing (PR #417)

8.6.0

• Replace publicsuffix2 with publicsuffixlist

8.5.0

• Add support for Azure Log Analytics (PR #394)
• Fix a bug in the Microsoft Graph integration that caused a crash when an inbox has 10+ folders (PR #398)
• Documentation fixes

8.4.2

• Only initialize the syslog, S3 and Kafka clients once (PR #386 closes issues #289 and #380)

8.4.1

• Fix bug introduced in 8.3.1 that caused No such file or directory errors if output files didn't exist (PR #385 closes issues #358 and #382)
• Make the --silent CLI option only print errors. Add the --warnings options to also print warnings (PR #383)

8.4.0

• Provide a warning when no file is located at the path specified by the ip_db_path option (based on PR #369 with improvements in grammar)
• Add allow_unencrypted_storage to possible msgraph settings. See documentation for details. (PR #375)
• Use the check_timeout value in the event of an IMAP connection error, instead of a static 5 second value (PR #377)
• Update the included DBIP IP to Country Lite database to the December 2022 release

8.3.2

• Improvements to the Microsoft Graph integration (PR #352)

8.3.1

• Handle unexpected XML parsing errors more gracefully (PR #349)
• Migrate build from setuptools to hatch

8.3.0

• Support MFA for Microsoft Graph (PR #320 closes issue #319)
• Add more options for S3 export (PR #328)
• Provide a helpful error message when the log file cannot be created (closes issue #317)

8.2.0

• Support non-standard, text-based forensic reports sent by some mail hosts
• Set forensic report version to None (null in JSON) if the report was in a non-standard format and/or is missing a version number
• The default value of the mailbox batch_size option is now 10 (use 0 for no limit)

8.1.1

• Fix marking messages as read via Microsoft Graph

8.1.0

• Restore compatibility with <8.0.0 configuration files (with deprecation warnings)
• Set default reports_folder to Inbox (rather than INBOX) when msgraph is configured
• Mark a message as read when fetching a message from Microsoft Graph

8.0.3

• Fix IMAP callback for IDLE connections (PR #313 closes issue #311)
• Add warnings in documentation and log output for IMAP configuration changes introduced in 8.0.0 (Closes issue #309)
• Actually pin the elasticsearch Python library version at <7.14.0 (Closes issue #315)
• Separate version numbers in __init__.py and setup.py to allow pip to install directly from git
• Update dateparser to 1.1.1 (closes issue #273)

8.0.2 (yanked)

• Strip leading and trailing whitespaces from Gmail scopes (Closes issue #310)

8.0.1 (yanked)

• Fix ModuleNotFoundError by adding parsedmarc.mail to the list of packages in setup.py (PR #308)

8.0.0 (yanked)

• Update included copy of dbip-country-lite.mmdb to the 2022-04 release
• Add support for Microsoft/Office 365 via Microsoft Graph API (PR #301 closes issue #111)
• Pin elasticsearch-dsl version at >=7.2.0<7.14.0 (PR #297 closes issue #296)
• Properly initialize ip_dp_path (PR #294 closes issue #286)
• Remove usage of logging.basicConfig (PR #285)
• Add support for the Gmail API (PR #284 and PR #307 close issue #96)

7.1.1

• Actually include dbip-country-lite.mmdb file in the parsedmarc.resources package (PR #281)
• Update dbip-country-lite.mmdb to the 2022-01 release

7.1.0

• A static copy of the DBIP Country Lite database is now included for use when a copy of the MaxMind GeoLite2 Country database is not installed (Closes #275)
• Add ip_db_path to as a parameter and general setting for a custom IP geolocation database location (Closes #184)
• Search default Homebrew path when searching for a copy of the MaxMind GeoLite2 Country database (Closes #272)
• Fix log messages written to root logger (PR #276)
• Fix --offline option in CLI not being passed as a boolean (PR #265)
• Set Elasticsearch shard replication to 0 (PR #274)
• Add support for syslog output (PR #263 closes #227)
• Do not print TQDDM progress bar when running in a no-interactive TTY (PR #264)

7.0.1

• Fix startup error (PR #254)

7.0.0

• Fix issue #221: Crash when handling invalid reports without root node (PR #248)
• Use UTC datetime objects for Elasticsearch output (PR #245)
• Fix issues #219, #155, and #103: IMAP connections break on large emails (PR #241)
• Add support for saving reports to S3 buckets (PR #223)
• Pass offline parameter to wait_inbox() (PR #216)
• Add more details to logging (PR #220)
• Add options customizing the names of output files (Modifications based on PR #225)
• Wait for 5 seconds before attempting to reconnect to an IMAP server (PR #217)
• Add option to process messages in batches (PR #222)

6.12.0

• Limit output filename length to 100 characters (PR #199)
• Add basic auth support for Elasticsearch (PR #191)
• Fix Windows paths when searching for the GeoIP database (PR #190)
• Remove six requirement
• Require mailsuite>=1.6.1
• Require dnspython>=2.0.0
  • Drop Python 3.5 support

6.11.0

• Fix parsing failure for some valid forensic reports (PR #170)
• Fix double count of messages in the Grafana dashboard (PR #182)
• Add begin and end date fields for aggregate DMARC reports in Elasticsearch (PR #183 fixes issue #162)
• Fix crash on IMAP timeout (PR #186 fixes issue #163)
• Fix IMAP debugging output
• Fix User-Agent string

6.10.0

• Ignore unknown forensic report fields when generating CSVs (Closes issue #148)
• Fix crash on IMAP timeout (PR #164 - closes issue #163)
• Use SMTP port from the config file when sending emails (PR #151)
• Add support for Elasticsearch 7.0 (PR #161 - closes issue #149)
• Remove temporary workaround for DMARC aggregate report records missing a SPF domain fields

6.9.0

• Use system nameservers instead of Cloudflare by default
• Parse aggregate report records with missing SPF domains

6.8.2

• Require mailsuite>=1.5.4

6.8.1

• Use match_phrase instead of match when looking for existing strings in Elasticsearch

6.8.0

• Display warning when GeoLite2-Country.mmdb is missing, instead of trying to download it
• Add documentation for MaxMind geoipupdate changes on January 30th, 2019 (closes issues #137 and #139)
• Require mail-parser>=3.11.0

6.7.4

• Update dependencies

6.7.3

• Make dkim_aligned and spf_aligned case-insensitive (PR #132)

6.7.2

• Fix SPF results field in CSV output (closes issue #128)

6.7.1

• Parse forensic email samples with non-standard date headers
• Graceful handling of a failure to download the GeoIP database (issue #123)

6.7.0

• Fix typos (PR #119)
• Make CSV output match JSON output (Issue # 22)
• Graceful processing of invalid aggregate DMARC reports (PR #122)
• Remove Python 3.4 support

6.6.1

• Close files after reading them

6.6.0

• Set a configurable default IMAP timeout of 30 seconds
• Set a configurable maximum of 4 IMAP timeout retry attempts
• Add support for reading MBOX files
• Set a configurable Elasticsearch timeout of 60 seconds

6.5.5

• Set minimum publicsuffix2 version

6.5.4

• Bump required mailsuite version to 1.2.1

6.5.3

• Fix typos in the CLI documentation
• Bump required mailsuite version to 1.1.1

6.5.2

• Merge PR #100 from michaeldavie
  • Correct a bug introduced in 6.5.1 that caused only the last record's data to be used for each row in an aggregate report's CSV version.
• Use mailsuite 1.1.0 to fix issues with some IMAP servers (closes issue 103)
  • Always use / as the folder hierarchy separator, and convert to the server's hierarchy separator in the background
  • Always remove folder name characters that conflict with the server's hierarchy separators
  • Prepend the namespace to the folder path when required

6.5.1

• Merge PR #98 from michaeldavie
  • Add functions
    • parsed_aggregate_reports_to_csv_row(reports)
    • parsed_forensic_reports_to_csv_row(reports)
• Require dnspython>=1.16.0

6.5.0

• Move mail processing functions to the mailsuite package
• Add offline option (closes issue #90)
• Use UDP instead of TCP, and properly set the timeout when querying DNS (closes issue #79 and #92)
• Log the current file path being processed when --debug is used (closes issue #95)

6.4.2

• Do not attempt to convert org_name to a base domain if org_name contains a space (closes issue #94)
• Always lowercase the header_from
• Provide a more helpful warning message when GeoLite2-Country.mmdb is missing

6.4.1

• Raise utils.DownloadError exception when a GeoIP database or Public Suffix List (PSL) download fails (closes issue #73)

6.4.0

• Add number_of_shards and number_of_replicas as possible options in the elasticsearch configuration file section (closes issue #78)

6.3.7

• Work around some unexpected IMAP responses reported in issue #75

6.3.6

• Work around some unexpected IMAP responses reported in issue #70
• Show correct destination folder in debug logs when moving aggregate reports

6.3.5

• Normalize Delivery-Result value in forensic/failure reports (issue #76) Thanks Freddie Leeman of URIports for the troubleshooting assistance

6.3.4

• Fix Elasticsearch index creation (closes issue #74)

6.3.3

• Set number_of_shards and number_of_replicas to 1 when creating indexes
• Fix dependency conflict

6.3.2

• Fix the monthly_indexes option in the elasticsearch configuration section

6.3.1

• Fix strip_attachment_payloads option

6.3.0

• Fix IMAP IDLE response processing for some mail servers (#67)
• Exit with a critical error when required settings are missing (#68)
• XML parsing fixes (#69)
• Add IMAP responses to debug logging
• Add smtp option skip_certificate_verification
• Add kafka option skip_certificate_verification
• Suppress mailparser logging output
• Suppress msgconvert warnings

6.2.2

• Fix crash when trying to save forensic reports with missing fields to Elasticsearch

6.2.1

• Add missing tqdm dependency to setup.py

6.2.0

• Add support for multiprocess parallelized processing via CLI (Thanks zscholl - PR #62)
• Save sha256 hashes of attachments in forensic samples to Elasticsearch

6.1.8

• Actually fix GeoIP lookups

6.1.7

• Fix GeoIP lookups

6.1.6

• Better GeoIP error handling

6.1.5

• Always use Cloudflare's nameservers by default instead of Google's
• Avoid re-downloading the Geolite2 database (and tripping their DDoS protection)
• Add geoipupdate to install instructions

6.1.4

• Actually package requirements

6.1.3

• Fix package requirements

6.1.2

• Use local Public Suffix List file instead of downloading it
• Fix argument name for send_email() (closes issue #60)

6.1.1

• Fix aggregate report processing
• Check for the existence of a configuration file if a path is supplied
• Replace publicsuffix with publicsuffix2
• Add minimum versions to requirements

6.1.0

• Fix aggregate report email parsing regression introduced in 6.0.3 (closes issue #57)
• Fix Davmail support (closes issue #56)

6.0.3

• Don't assume the report is the last part of the email message (issue #55)

6.0.2

• IMAP connectivity improvements (issue #53)
• Use a temp directory for temp files (issue #54)

6.0.1

• Fix Elasticsearch output (PR #50 - andrewmcgilvray)

6.0.0

• Move options from CLI to a config file (see updated installation documentation)
• Refactoring to make argument names consistent

5.3.0

• Fix crash on invalid forensic report sample (Issue #47)
• Fix DavMail support (Issue #45)

5.2.1

• Remove unnecessary debugging code

5.2.0

• Add filename and line number to logging output

• Improved IMAP error handling

• Add CLI options

  --elasticsearch-use-ssl
                        Use SSL when connecting to Elasticsearch
  --elasticsearch-ssl-cert-path ELASTICSEARCH_SSL_CERT_PATH
                        Path to the Elasticsearch SSL certificate
  --elasticsearch-monthly-indexes
                        Use monthly Elasticsearch indexes instead of daily
                        indexes
  --log-file LOG_FILE   output logging to a file
  

5.1.3

• Remove urllib3 version upper limit

5.1.2

• Workaround unexpected Office 365/Exchange IMAP responses

5.1.1

• Bugfix: Crash when parsing invalid forensic report samples (#38)
• Bugfix: Crash when IMAP connection is lost
• Increase default Splunk HEC response timeout to 60 seconds

5.1.0

• Bugfix: Submit aggregate dates to Elasticsearch as lists, not tuples
• Support elasticsearch-dsl<=6.3.0
• Add support for TLS/SSL and username/password auth to Kafka

5.0.2

• Revert to using publicsuffix instead of publicsuffix2

5.0.1

• Use publixsuffix2 (closes issue #4)
• Add Elasticsearch to automated testing
• Lock elasticsearch-dsl required version to 6.2.1 (closes issue #25)

5.0.0

Note: Re-importing kibana_saved_objects.json in Kibana is required when upgrading to this version!

• Bugfix: Reindex the aggregate report index field published_policy.fo as text instead of long (Closes issue #31)
• Bugfix: IDLE email processing in Gmail/G-Suite accounts (closes issue #33)
• Bugfix: Fix inaccurate DNS timeout in CLI documentation (closes issue #34)
• Bugfix: Forensic report processing via CLI
• Bugfix: Duplicate aggregate report Elasticsearch query broken
• Bugfix: Crash when Arrival-Date header is missing in a forensic/failure/ruf report
• IMAP reliability improvements
• Save data in separate indexes each day to make managing data retention easier
• Cache DNS queries in memory

4.4.1

• Don't crash if Elasticsearch returns an unexpected result (workaround for issue #31)

4.4.0

• Packaging fixes

4.3.9

• Kafka output improvements
  • Moved some key values (report_id, org_email, org_name) higher in the JSON structure
  • Recreated the date_range values from the ES client for easier parsing.
  • Started sending individual record slices. Kafka default message size is 1 MB, some aggregate reports were exceeding this. Now it appends meta-data and sends record by record.

4.3.8

• Fix decoding of attachments inside forensic samples
• Add CLI option --imap-skip-certificate-verification
• Add optional ssl_context argument for get_dmarc_reports_from_inbox() and watch_inbox()
• Debug logging improvements

4.3.7

• When checking an inbox, always recheck for messages when processing is complete

4.3.6

• Be more forgiving for forensic reports with missing fields

4.3.5

• Fix base64 attachment decoding (#26)

4.3.4

• Fix crash on empty aggregate report comments (brakhane - #25)
• Add SHA256 hashes of attachments to output
• Add strip_attachment_payloads option to functions and --strip-attachment-payloads option to the CLI (#23)
• Set urllib3 version requirements to match requests

4.3.3

• Fix forensic report email processing

4.3.2

• Fix normalization of the forensic sample from address

4.3.1

• Fix parsing of some emails
• Fix duplicate forensic report search for Elasticsearch

4.3.0

• Fix bug where parsedmarc would always try to save to Elastic search, even if only --hec was used
• Add options to save reports as a Kafka topic (mikesiegel - #21)
• Major refactoring of functions
• Support parsing forensic reports generated by Brightmail
• Make sample_headers_only flag more reliable
• Functions that might be useful to other projects are now stored in parsedmarc.utils:
  • get_base_domain(domain)
  • get_filename_safe_string(string)
  • get_ip_address_country(ip_address)
  • get_ip_address_info(ip_address, nameservers=None, timeout=2.0)
  • get_reverse_dns(ip_address, nameservers=None, timeout=2.0)
  • human_timestamp_to_datetime(human_timestamp)
  • human_timestamp_to_timestamp(human_timestamp)
  • parse_email(data)

4.2.0

• Save each aggregate report record as a separate Splunk event
• Fix IMAP delete action (#20)
• Suppress Splunk SSL validation warnings
• Change default logging level to WARNING

4.1.9

• Workaround for forensic/ruf reports that are missing Arrival-Date and/or Reported-Domain

4.1.8

• Be more forgiving of weird XML

4.1.7

• Remove any invalid XML schema tags before parsing the XML (#18)

4.1.6

• Fix typo in CLI parser

4.1.5

• Only move or delete IMAP emails after they all have been parsed
• Move/delete messages one at a time - do not exit on error
• Reconnect to IMAP if connection is broken during get_dmarc_reports_from_inbox()
• Add--imap-port and --imap-no-ssl CLI options

4.1.4

• Change default logging level to ERROR

4.1.3

• Fix crash introduced in 4.1.0 when creating Elasticsearch indexes (Issue #15)

4.1.2

• Fix packaging bug

4.1.1

• Add splunk instructions
• Reconnect reset IMAP connections when watching a folder

4.1.0

• Add options for Elasticsearch prefixes and suffixes
• If an aggregate report has the invalid disposition value pass, change it to none

4.0.2

• Use report timestamps for Splunk timestamps

4.0.1

• When saving aggregate reports in Elasticsearch store domain in published_policy
• Rename policy_published to published_policywhen saving aggregate reports to Splunk

4.0.0

• Add support for sending DMARC reports to a Splunk HTTP Events Collector (HEC)
• Use a browser-like User-Agent when downloading the Public Suffix List and GeoIP DB to avoid being blocked by security proxies
• Reduce default DNS timeout to 2.0 seconds
• Add alignment booleans to JSON output
• Fix .msg parsing CLI exception when msgconvert is not found in the system path
• Add --outgoing-port and --outgoing-ssl options
• Fall back to plain text SMTP if --outgoing-ssl is not used and STARTTLS is not supported by the server
• Always use as the newline when generating CSVs
• Workaround for random Exchange/Office 365 Server Unavailable IMAP errors

3.9.7

• Completely reset IMAP connection when a broken pipe is encountered

3.9.6

• Finish incomplete broken pipe fix

3.9.5

• Refactor to use a shared IMAP connection for inbox watching and message downloads

• Gracefully recover from broken pipes in IMAP

3.9.4

• Fix moving/deleting emails

3.9.3

• Fix crash when forensic reports are missing Arrival-Date

3.9.2

• Fix PEP 8 spacing
• Update build script to fail when CI tests fail

3.9.1

• Use COPY and delete if an IMAP server does not support MOVE (closes issue #9)

3.9.0

• Reduce IMAP IDLE refresh rate to 5 minutes to avoid session timeouts in Gmail
• Fix parsing of some forensic/failure/ruf reports
• Include email subject in all warning messages
• Fix example NGINX configuration in the installation documentation (closes issue #6)

3.8.2

• Fix nameservers option (mikesiegel)
• Move or delete invalid report emails in an IMAP inbox (closes issue #7)

3.8.1

• Better handling of .msg files when msgconvert is not installed

3.8.0

• Use . instead of / as the IMAP folder hierarchy separator when / does not work - fixes dovecot support (#5)
• Fix parsing of base64-encoded forensic report data

3.7.3

• Fix saving attachment from forensic sample to Elasticsearch

3.7.2

• Change uses of the DocType class to Document, to properly support elasticsearch-dsl 6.2.0 (this also fixes use in pypy)
• Add documentation for installation under pypy

3.7.1

• Require elasticsearch>=6.2.1,<7.0.0 and elasticsearch-dsl>=6.2.1,<7.0.0
• Update for class changes in elasticsearch-dsl 6.2.0

3.7.0

• Fix bug where PSL would be called before it was downloaded if the PSL was older than 24 Hours

3.6.1

• Parse aggregate reports with missing SPF domain

3.6.0

• Much more robust error handling

3.5.1

• Fix dashboard message counts for source IP addresses visualizations
• Improve dashboard loading times
• Improve dashboard layout
• Add country rankings to the dashboards
• Fix crash when parsing report with empty <auth_results></auth_results>

3.5.0

• Use Cloudflare's public DNS resolvers by default instead of Google's
• Fix installation from virtualenv
• Fix documentation typos

3.4.1

• Documentation fixes
• Fix console output

3.4.0

• Maintain IMAP IDLE state when watching the inbox
• The -i/--idle CLI option is now -w/--watch
• Improved Exception handling and documentation

3.3.0

• Fix errors when saving to Elasticsearch

3.2.0

• Fix existing aggregate report error message

3.1.0

• Fix existing aggregate report query

3.0.0

New features

• Add option to select the IMAP folder where reports are stored
• Add options to send data to Elasticsearch

Changes

• Use Google's public nameservers (8.8.8.8 and 4.4.4.4) by default
• Detect aggregate report email attachments by file content rather than file extension
• If an aggregate report's org_name is a FQDN, the base is used
• Normalize aggregate report IDs

2.1.2

• Rename parsed_dmarc_forensic_reports_to_csv() to parsed_forensic_reports_to_csv() to match other functions
• Rename parsed_aggregate_report_to_csv() to parsed_aggregate_reports_to_csv() to match other functions
• Use local time when generating the default email subject

2.1.1

• Documentation fixes

2.1.0

• Add get_report_zip() and email_results()
• Add support for sending report emails via the command line

2.0.1

• Fix documentation
• Remove Python 2 code

2.0.0

New features

• Parse forensic reports
• Parse reports from IMAP inbox

Changes

• Drop support for Python 2
• Command line output is always a JSON object containing the lists aggregate_reports and forensic_reports
• -o/--output option is now a path to an output directory, instead of an output file

1.1.0

• Add extract_xml() and human_timestamp_to_datetime methods

1.0.5

• Prefix public suffix and GeoIP2 database filenames with .
• Properly format errors list in CSV output

1.0.3

• Fix documentation formatting

1.0.2

• Fix more packaging flaws

1.0.1

• Fix packaging flaw

1.0.0

• Initial release