Merge commit from fork

* chore: update IPinfo Lite MMDB (#941)

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Cap report date range and indexes created per report

A report whose <date_range> spans millions of days was normalized into one
record per day, exhausting memory, and with Elasticsearch or OpenSearch
output created one dated index (one shard) per day, exhausting the
cluster's shard budget.

Reject a range longer than MAX_REPORT_SPAN_DAYS (366) or negative with
InvalidAggregateReport. In both save functions, once a report has used
MAX_INDEXES_PER_REPORT (31) distinct dated indexes, route further records
to the report's first index, keeping their own dates, and log one warning.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Catch output-stage exceptions and validate SMTP TLS report types

One report sent to a monitored mailbox could make a configured output
raise an exception that process_reports() did not catch. It escaped
through the mailbox save callback, _main logged "Mailbox Error" and
exited 1, and the batch stayed in the mailbox, so every one-shot run
died on the same message and duplicated the legitimate reports in
non-deduplicating outputs. Three sender-controlled inputs did it:

- A failure report whose sample has an empty From:, To: or Reply-To:
  (To: undisclosed-recipients:; included): mail-parser gives an empty
  address list, and the Elasticsearch and OpenSearch saves indexed
  [0] (IndexError).
- An SMTP TLS report with a wrong-typed field: "policy-string": [1]
  made the CSV writer raise TypeError; "start-datetime": 5 made the
  Elasticsearch, OpenSearch and Splunk saves raise AttributeError and
  _ensure_utc_suffix() in the PostgreSQL save raise TypeError.
- An aggregate report whose <reason> has no <type> (KeyError in the
  CSV writer) or a <type> that xmltodict parses to a dict or a list
  (TypeError in its join).

Fix:

- Every output destination block in process_reports() now ends with
  the catch-all the aggregate Elasticsearch block had: the failure and
  SMTP TLS Elasticsearch/OpenSearch blocks, every PostgreSQL and
  Splunk HEC block, and the file output (its OSError/ValueError branch
  is kept). The error goes through log_output_error(), so the batch is
  reported unsaved and fail_on_output_error still raises ParserError.
- parse_smtp_tls_report_json() and its helpers check each field
  against the type RFC 8460 section 4.4 gives it and raise
  InvalidSMTPTLSReport otherwise: strings for organization-name,
  report-id, policy-domain, policy-type, result-type and the optional
  failure-details fields; contact-info a string or a list of strings;
  policy-string and a list mx-host-pattern lists of strings; the
  session counts integers or null; failure-details a list of objects;
  start-datetime and end-datetime strings of at most 64 characters that
  the outputs' own conversions accept. dateutil took 12.4 s to fail on
  a 1,000,000-digit date, hence the length bound.
- _parse_report_record() makes a reason's type and comment a string or
  None (a dict unwrapped with _text(), a list as its str()), and the
  aggregate CSV writer reads both with .get().
- The Elasticsearch and OpenSearch failure saves flatten From, To and
  Reply-To with a helper that returns None for an empty list, and
  leave such a header out of the document and the duplicate query.
- The Splunk SMTP TLS save converts begin_date inside its try, so a bad
  value raises SplunkError; _ensure_utc_suffix() returns a non-string
  unchanged.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Limit XML nesting depth and node count in aggregate reports

parse_aggregate_report_xml() gave the decompressed report (up to the
100 MiB decompression cap) to xmltodict with no limit on depth or element
count. xmltodict keeps one stack entry per open element and expat reports
missing end tags only at the end of the input, so a few KB of gzipped
unclosed <a> tags grew to gigabytes of RSS and the kernel OOM killer
ended the process before the message was moved; a wide document of
millions of siblings did the same at a smaller scale. A failed first
parse also re-parsed the document through lxml recovery, which has no
node limit, and a second xmltodict pass.

_NoDoctypeExpat.ParserCreate() now returns a proxy around the expat
parser. pyexpat parsers cannot be subclassed and xmltodict assigns its
handlers after ParserCreate() returns, so the proxy's __setattr__ wraps
StartElementHandler to count depth and nodes (elements plus attributes)
and EndElementHandler to decrement depth, and forwards every other
attribute. Over MAX_XML_DEPTH (100) or MAX_XML_NODES (5,000,000) it
raises before xmltodict's handler runs. Both xmltodict.parse() call
sites turn that into InvalidAggregateReport naming the limit; the first
does so instead of trying lxml recovery.

The lxml recovery path now runs a counting pass first, with a parser
target that builds no tree and counts elements, attributes and
namespace declarations against MAX_XML_NODES; the tree is built only
for a document within the limit. Depth there is bounded by libxml2's
own limit of 256 (huge_tree stays off) and then by the second
xmltodict pass. A 22 MiB wide document behind a malformed XML
declaration peaked at +1532 MiB before this pass and +23 MiB after.

The largest aggregate sample (909 KB) has 32,019 nodes, so a legitimate
report at the 100 MiB cap stays near 3.6 million.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Create Elasticsearch/OpenSearch indexes with mapping guards

create_indexes() and create_data_streams() created indexes and the data
stream template with settings only, so the cluster typed every report
field from the first document that held it, with date_detection on. A
sender could be first into a day's index (its name comes from the
report) and poison it: a report_id of "2026-11-09" mapped report_id as a
date, a failure sample header named "Subject.x" mapped
sample.headers.subject as an object, and samples with invented header
names used up index.mapping.total_fields.limit. Every later legitimate
report for that index was then rejected and ended up in Unsaved; in data
stream mode this lasted until a rollover.

Both functions now take a mappings body, and the three save functions
pass one per report type. Each turns date_detection off and declares the
fields that hold dates (@timestamp, date_range, date_begin, date_end,
arrival_date, sample.date) as dates, the type date detection gave them
before; on the sample reports every other field keeps its old type. The failure mapping
maps sample.headers as an object with dynamic: false and declares, as
text with a keyword sub-field (the dynamic string default), every header
the shipped dashboards reference plus date, message-id and cc; all
headers still stay in _source. A "." in a header name is replaced with
"_" before the header is stored. The index is now created through the
client's indices.create with the settings and mappings body, because
opensearch-py's Index.create cannot take a mappings argument; the data
stream template merges the mappings with its @timestamp date mapping.

Existing indexes are not changed. The other report fields stay dynamic
and object-mapped (no nested), which the dashboards need (issue #169).

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Read the RFC 8460 mx-host key in SMTP TLS reports

RFC 8460 section 4.4 keys a policy's MX host patterns "mx-host"; its
schema line is "mx-host": mx-host-pattern, where mx-host-pattern is the
value type, described as "a JSON array of strings", and the RFC's
Appendix B example gives the value as a plain string. The parser read
only "mx-host-pattern", so every RFC-conformant report lost the field
(samples/smtp_tls/rfc8460.json, the Appendix B example, among them).

_parse_smtp_tls_report_policy() now reads "mx-host" as a string or a
list of strings, a string becoming a one-element list, and falls back
to "mx-host-pattern" (same rules) when "mx-host" is absent, null or
empty. Any other type raises InvalidSMTPTLSReport. The output key is
still mx_host_patterns, so the PostgreSQL TEXT[] column, the
Elasticsearch/OpenSearch field and the CSV column are unchanged.

Refs #942

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject over-limit messages with the mailsuite guard

parse_report_email() and parse_email() handed every message to
mail-parser and the email package with no bound on blank lines, header
lines, header field size, MIME parts or nested multiparts, so a small
crafted message took seconds to minutes and gigabytes to parse while the
rest of the mailbox waited.

Both now call mailsuite.utils.check_message_limits() first, one linear
pass. parse_report_email() turns MessageLimitExceeded into
InvalidDMARCReport so the mailbox loop files the message under Invalid;
parse_failure_report() turns one from the sample into InvalidFailureReport.
The mailsuite floor rises to 2.3.5, which adds the guard.

Messages nested deeper than 100 multiparts are now rejected by the guard
instead of reaching the recursion handler; the tests that used the
400-level message expect the new message.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Cite RFC 7489 §6.3 for the daily report interval

The comment on MAX_REPORT_SPAN_DAYS cited RFC 7489 Appendix A for the
one-day aggregate report. The daily interval comes from §6.3: the "ri"
tag requests the interval between aggregate reports, and its default is
86400 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add the 11.2.2 changelog: five advisories and #942

One Security bullet per advisory (GHSA-qcc2, GHSA-q7gm, GHSA-p85v,
GHSA-4272, GHSA-qpcg) and a Bug fixes bullet for the RFC 8460 mx-host
key. The version in constants.py is bumped by the release pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the date range rejection message and pyright's analysis

With the XML limits and the date range limit merged,
parse_aggregate_report_xml() became too complex for pyright to analyze
("Code is too complex to analyze"), and the date range rejection,
raised inside the function's catch-all, reached the caller as
"Unexpected error: Report date range spans ...".

The date range check moves to _check_report_span() and the second
xmltodict pass to _parse_limited_xml(), which raises the XML limit
breach as InvalidAggregateReport; the catch-all now re-raises an
InvalidAggregateReport as it is. The tests assert the message is not
re-wrapped.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Put the over-limit failure samples in a delivery-status body

mailsuite 2.3.5's check_message_limits() follows the MIME structure and
counts blank lines only inside message/delivery-status bodies, the shape
whose parse cost is per blank line; blank lines elsewhere in a body are
not limited. The two failure-sample tests used blank lines in a plain
body, which the final check accepts, so they now put them in a
delivery-status body. parsedmarc's call with the default limits is
unchanged.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Describe the final mailsuite check and the index cap in the docs

The GHSA-qcc2 changelog bullet now describes check_message_limits() as
mailsuite 2.3.5 ships it: limits follow the MIME structure, and body
lines outside message/delivery-status bodies are not limited. The
changelog section moves to Unreleased, as parsedmarc's AGENTS.md has
fix pull requests write it; the release pull request names the version.

The Elasticsearch and OpenSearch pages now say that an address header
with no address is left out of the failure document, and that a report
spanning more than 31 days saves its later records to its first dated
index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound records times days in normalization and fix the span message

The date range limit bounds how far one record is split, but not how
many records are split: 10,000 records over 366 days, each with a count
of 1,000,000 (3.3 MB of XML), produced 3,670,000 records in 13.5 s and
1.7 GB, and 30,000 of them (a 44 KB gzip) ran out of memory and reached
the caller as "Unexpected error: ". A report whose split would exceed
MAX_NORMALIZED_RECORDS (250,000) now keeps its records whole over its own
range, with timespan_requires_normalization false and one warning; a
large legitimate multi-day report is kept rather than rejected. The
record loop moves to _parse_report_records() so pyright can still
analyze parse_aggregate_report_xml().

The span message used float division, which raises OverflowError for an
<end> of 4,001 digits; it now uses integer division. The index-routing
warning escapes the sender's report ID with sanitize_for_log(). The
changelog and the Elasticsearch and OpenSearch pages say the index cap
is per report, that dated index names still come from report dates, and
that data streams are affected by neither.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Drop XML attributes from aggregate reports before they are used

xmltodict turns an element with an attribute, such as <p a="1">none</p>,
into {"@a": "1", "#text": "none"}. Such objects passed through the
parser into Elasticsearch and OpenSearch documents for version, p, fo,
disposition, envelope_to and other scalar fields, so the first one mapped
the field as an object and every later report to that index failed, and
hundreds of attributes on one element used up total_fields.limit. The
parsed report now goes through one linear pass that replaces an element
with only attributes and text by its text and drops attributes from the
others; the existing _text() calls now receive strings.

The changelog and the Elasticsearch and OpenSearch pages also say that
an existing data stream's next backing index gets the guarded mappings,
since parsedmarc puts the template again in every process, and the
indexed-header comment and changelog name cc as the one header added to
those the dashboards reference.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject lone surrogates in SMTP TLS reports and guard the stdout print

An SMTP TLS string field holding a lone surrogate (JSON "\ud800")
passed every type check and survived json.dumps(ensure_ascii=False), and
print(output_str) in process_reports(), outside every try, raised
UnicodeEncodeError: parsedmarc exited 1, and in mailbox mode the error
escaped the save callback. RFC 8460 section 4 reports are I-JSON, whose RFC
7493 section 2.1 forbids surrogate code points, so the string and
string-list checks now reject one as InvalidSMTPTLSReport, and the print
records a UnicodeEncodeError as a "Standard output" output error.

The _smtp_tls_datetime() docstring no longer says a parsed date cannot
fail in any output: dateutil accepts partial dates that PostgreSQL may
reject, which is contained as PostgreSQLError. The _reason_text()
docstring says a dict with child elements gives None.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Describe mailsuite 2.3.5's address group handling in the changelog

The final mailsuite 2.3.5 no longer rejects an address header by its
comma count; its parse_email() removes RFC 5322 group syntax from address
header values over 8 KiB before parsing them. The boundary limit
counts lines that are the boundary of an open multipart. parsedmarc's
call with the default limits is unchanged.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test the keep-alive call of the extracted record loop

_parse_report_records() holds the record loop moved out of
parse_aggregate_report_xml(); its keep-alive branch had no test.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test a policy override type with child elements

With XML attributes dropped at parse time, a dict reaches _reason_text()
only for an element with child elements; the test pins its None result
and the CSV's "none".

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Check plain-string contact-info and mx-host for lone surrogates

The surrogate check covered the string fields and string lists, but a
contact-info or mx-host given as one string took a branch that skipped
it, and a lone surrogate there still made the stdout print raise.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say which sample headers the failure document leaves out

The GHSA-qpcg changelog bullet said every sample header stays in
_source, but the failure save leaves out a From, To or Reply-To header
that holds no address, as the Elasticsearch and OpenSearch pages say.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep sender-derived numbers out of the date range message

The rejection message printed the report's span in days, so an <end> of
thousands of digits put a number of thousands of digits into the
exception text and the log. It now says only that the range is negative
or longer than MAX_REPORT_SPAN_DAYS.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Store every string-typed aggregate field as a string or null

Dropping attributes did not cover child elements: <version><x>1</x></version>,
<p><x>1</x></p> or <p>none<x/></p> still reached the parsed report, and the
Elasticsearch and OpenSearch documents, as a dict, and the first such
document mapped the field as an object. _scalar() now makes each field the
schema types as a string a string (an element's own text) or None, applied
by _scalarize_report() to version, the report_metadata strings and every
policy_published child, and by _parse_report_record() to source_ip,
disposition, dkim, spf, the identifiers and every auth_results string. A
<count> with no text of its own takes the ValueError path of a non-numeric
count. _drop_xml_attributes() stays, so a field missed here still loses
its attribute keys.

The <email> dict unwrap is gone, since the field can no longer be a dict;
a text-only report_metadata still raises the AttributeError that reports
a missing section. Comments that described attribute dicts now say the
dict shape comes only from child elements. The namespace test, which the
new code could not fail, is replaced by one that detects RFC 9990 through
an attributed np field.

Tests walk the to_dict() of a mocked Elasticsearch and OpenSearch save and
allow lists and objects only in the known object fields, so a scalar field
added later is covered too.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record a closed stdout pipe as an output error and fix two comments

The standard-output guard in process_reports() caught only
UnicodeEncodeError, so a closed pipe (BrokenPipeError, an OSError) still
ended the run; it now records both as a "Standard output" output error.

_reason_text()'s docstring said a dict gives None because the element has
no text of its own, but mixed content (<type>forwarded<x/></type>) gives
the text beside the children; the docstring now says what the code does,
and a test pins it. The comment above the reason normalization no longer
describes attribute dicts, which no longer reach it.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test the normalization limit at its boundary and reword the index cap

The suite passed with the limit's <= changed to < and with
_days_touched()'s ceiling changed to a floor. New tests: 1,000 records
over 250 days (exactly MAX_NORMALIZED_RECORDS) are split; 683 records
over 366 days (249,978) are split and 684 (250,344) are not; 682 records
over 366 days starting 1 s after midnight touch 367 days (250,294) and
are not split. Each mutant was run against these tests and fails: the <
mutant fails the exact-limit test, the floor mutant the after-midnight
test.

The Elasticsearch and OpenSearch pages now say the index cap applies
when a report's records would use more than 31 dated indexes, which is
what it counts (with monthly_indexes the span limit keeps a report under
it).

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Match the tests and changelog to mailsuite 2.3.5's final check

mailsuite 2.3.5's check_message_limits() now parses the message with
the email package's own feed parser under compat32 and counts the parts
it creates (each message/delivery-status block among them), their
nesting depth and its boundary tests, then limits header field length
and count on the parsed tree; the limit names changed. The over-limit
failure samples now trip max_parts, and the deeply nested poison message
max_depth; the tests assert those texts. The GHSA-qcc2 changelog bullet
describes the final check, the epilogue mail-parser parses again, and
the changes that affect only mailsuite's own parse_email().

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test a repeated policy_published and note mixed-content behaviour

No test failed when _scalarize_report() stopped coercing the fields of a
repeated <policy_published> (the list case): with that mutant, a report
with two policy_published elements and <p><x>1</x></p> puts a dict into
published_policy.p. The parse test and the Elasticsearch and OpenSearch
document-walk tests now include that report; each fails on the mutant.

The changelog says that an element with text and a stray child element
now yields its text (samples/aggregate/invalid_xml.xml gives header_from
"bad" where it gave ""). AGENTS.md and the _normalize_result_word() and
_scalarize_report() docstrings describe what runs now: attributes are
dropped as soon as the XML is parsed, and _scalar() coerces the string
fields, among them the RFC 7489 and RFC 9990 policy_published fields.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Coerce policy override reasons with the same rule as other strings

_reason_text() turned a repeated <type> or <comment> into the str() of
the list, while _scalar() turns every other repeated string field into
None. One class of input now has one rule: the reason's type and comment
go through _scalar(), so a repeated one is None ("none" in CSV), and
_reason_text() is removed. The test and the GHSA-4272 changelog bullet
say so.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test an aggregate <error> with a child element

With attributes dropped and reasons coerced by _scalar(), only <error>,
which may repeat, still reaches _text() with a dict; its branch had no
test. An <error> with text beside a child element keeps the text, and
one with no text is dropped.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Describe mailsuite's six message limits in the changelog

mailsuite 2.3.5's check now counts every line of every header block
(folded lines and lines the parser keeps only as defects included) and
stops the parse early, and limits the total length of the address
headers mail-parser parses to 262,144 characters. The GHSA-qcc2 bullet
names the six limits.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render a report email's parts only where their text is read

parse_report_email() did str(payload_obj[0]) for every part walk()
visits, before the multipart/ continue. str() of a composite part
renders its whole subtree, and walk() visits every level, so a nesting
within mailsuite's limits rendered its leaf once per level: 98 nested
multiparts around a 20 MiB one-line leaf took 26.6 s, and a chain of 99
message/rfc822 parts around a 10 MiB leaf 11.3 s. A part is now rendered
only in the branches that read it; other message/* bodies are skipped,
since walk() visits their parts. Both shapes now take under 0.5 s, and
every sample under samples/failure/ parses to the same output.

A feedback report's sample is taken from the first sample part only: a
message/rfc822 nested inside the sample used to replace it, but a
feedback report has one sample (RFC 6591 section 3.1). Multipart nesting
alone no longer reaches the recursion limit, so the GHSA-qxwv test now
reaches the recursion guard through a deeply nested sample, which is
still rendered.

The changelog describes this and the address headers mailsuite 2.3.5
counts (From, Sender, Reply-To, To, Cc, Bcc, Delivered-To, in every
part; mail-parser parses all but Sender).

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Parse aggregate XML without attributes and lower the node limit

_drop_xml_attributes() rebuilt the parsed tree while the original was
alive, and the counting wrappers of _LimitedExpatParser referred to the
proxy, closing a reference cycle through the expat parser that kept
xmltodict's tree alive until the cyclic garbage collector ran. Both
xmltodict passes now use xml_attribs=False, so no attribute key is ever
built (expat still hands the attributes to the start handler, so they
are still counted: a test shows one element with 1,000 attributes is
1,001 nodes), _drop_xml_attributes() is removed, and the counts live in
an _XMLCounts object the wrappers share, so a parse leaves no cycle (a
test asserts gc.collect() finds nothing after one).

Peak extra memory for 2,000,000 distinct sibling elements: +999 MiB
before, +386 MiB after (master +391 MiB); for one element with 2,000,000
attributes: +1,274 MiB before, +445 MiB after (master +734 MiB).

At the old limit of 5,000,000 nodes a report could still need about 3 GB.
Measured after the change, the worst shape is distinct elements with
text behind a malformed XML declaration (the lxml recovery path): 559
bytes per node, +533 MiB for 999,000 nodes and +1,023 MiB for 2,000,000;
on the expat path about 220 bytes per node. MAX_XML_NODES is now
1,000,000, so a report at the limit needs at most about 550 MiB; the
largest sample has 32,019 nodes (909 KB), and the limit is a report of
about 28 MB. The constant's comment and the changelog state the bound.

AGENTS.md describes the attribute handling as it now is, and says that
<error>, which may repeat, goes through _text() and stays a list.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Qualify the mixed-content compatibility note to string fields

An element with text beside a stray child element yields its text only
for the fields the schema types as strings; a <begin> or <end> with mixed
content is still rejected, as before.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject over-long XML tags before any parser reads them

expat and libxml2 build a start tag's whole attribute list before they
report the element, so the node counter only sees the attributes after
they cost memory: one element with 4,999,000 attributes (56 MiB) took
5.25 s and +767 MiB before the node limit rejected it, and a single
element is bounded only by the 100 MiB decompression cap.

parse_aggregate_report_xml() now scans the decoded text once, before the
first parse and so before the lxml recovery path too, and rejects a
document in which any text between a "<" and the next ">" (or the end
of the text) is longer than MAX_XML_TAG_LENGTH (65,536 characters). The
scan uses no regex. It splits the text on ">" one 64 KiB chunk at a time,
carrying the unfinished piece from its last "<": a single split of the
whole text makes one string per tag, which peaked at +1,318 MiB (1.9 s)
for 100 MiB of "<a/>"; the chunked scan needs about 2 MiB and takes
0.6 s on that text, and a test bounds its memory and covers a tag across
a chunk boundary. The longest tag in the samples is 108 characters. The same shape
is now rejected in under 0.1 s, with a tracemalloc peak under 200 MB in the
test, behind a well-formed or a malformed declaration.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render a report email's feedback-report part only once

The message/* skip exempted message/feedback-report, which was rendered
with str() every time walk() reached one: a chain of 99 nested
message/feedback-report parts around a 10 MiB leaf passed mailsuite's
check in 0.02 s and then took 11.6 s. A feedback report has one
message/feedback-report part (RFC 6591 section 3), so, as with the
sample, only the first is read and rendered; the chain now takes 0.56 s.
The failure samples parse to byte-identical output.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound attribute counts by their = signs and cap reports at 32 MiB

XML allows a raw > inside a quoted attribute value, so attributes such as
x=">" cut a tag into short pieces that pass the tag scan: 4,999,000 of
them cost 5.61 s and +781 MiB, and 8,000,000 (98 MiB) 8.9 s and
+1,215 MiB, before the node limit fired. Every attribute needs an =, so
a document with more = characters than MAX_XML_NODES is now rejected in
one str.count() pass, before the tag scan; the samples hold at most 3.
The tag scan stays for long tags, and its docstring says it does not
bound attribute counts.

The stated memory bound was false: text per node is bounded only by the
size of the report, so at the node limit with 80 characters per element
(94 MiB) the recovery path reached +827 MiB. MAX_DECOMPRESSED_REPORT_SIZE
falls from 100 MiB to 32 MiB (the node limit already means a legitimate
report is at most about 28 MB, at the samples' 35 nodes per KB, and the
largest sample is 909 KB), and parse_aggregate_report_xml() holds an
uncompressed report to the same length, so the bound holds for every
input. Measured at the cap, 999,000 distinct elements with 14 characters
of text (31 MiB) need +587 MiB on the recovery path and +302 MiB on the
expat path; the constant comment and the changelog state it.

When the first parse fails, the exception's traceback kept the partial
xmltodict tree alive while the recovery parse ran; it is cleared first
(+603 to +587 MiB, and +263 to +241 MiB for 330,000 elements of 80
characters). A test shows the recovery path leaves the same 18 lxml
helper objects for a document ten times larger, so no tree survives a
parse there either.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Let the RFC 6591 feedback part win over legacy report text

The guard that reads only the first message/feedback-report part tested
feedback_report, which the legacy text/plain branch ("A message claiming
to be from you has failed ... detected.") also sets, so a legacy text
part followed by a real feedback-report part and sample now took the
legacy part and never read the ARF, where master read the ARF. The
guard now tests is_feedback_report, and reading the feedback-report part
discards a sample a legacy part set before it.

The same rule closes an older gap: a text/plain part inside the sample,
which is the spoofed message's own text, could hold the legacy phrase and
replace the report's Source-IP and its sample. Once a feedback-report
part is read, the legacy branch is skipped; a legacy report has no such
part. Tests cover both orders and the in-sample case; the failure
samples parse to byte-identical output.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hold every report input to the size cap and free failed parses

parse_report_file() kept the aggregate parser's exception, whose
traceback holds the parser's frames and the parsed tree, while it tried
SMTP TLS and then the email parser: a 31 MiB file at the cap cost
+1,046 MiB there against +587 MiB parsing it directly. It now keeps only
each failure's text and raises from a fresh exception of the relevant
format, so the chain survives without the tree. A file that begins with
"<" is not tried as JSON (json.loads() decoded it all) or as an email,
one that begins with "{" is not tried as an email (an RFC 822 message
begins with a header line), and text over the size cap is never handed
to the email parser, which has none: a 200 MiB <x/> file reached
+2,801 MiB in 4.6 s and now costs +232 MiB (reading the file) in 0.05 s.
extract_report() reads an uncompressed report at most one byte past the
cap instead of decoding it whole.

The cap counted characters for text and bytes for compressed reports,
so a non-ASCII report could be four times the cap in bytes: 999,000
elements of 14 emoji (31.2 Mi characters, 71 MiB of UTF-8) cost +649 MiB.
_exceeds_size_cap() measures text as UTF-8 (ASCII by length, other text
1 MiB of characters at a time, without a full copy), and bytes are
checked before they are decoded.

Clearing e.__traceback__ inside the except block did nothing on Python
3.10, which keeps the handled exception's traceback while the handler
runs. The lxml recovery now runs after the except block, in
_recover_xml(), so the first parse's frames and partial tree are
unreachable when it starts on every supported Python; a test holds a
weak reference to the first parse's counter and fails on the old code
under Python 3.10.19 and, with the clearing removed, under 3.14.

The changelog's examples are re-measured under the 32 MiB cap, and the
test docstrings cite the same figures.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip the parts inside a failure report's sample

Once the sample part was taken, walk() still visited its descendants,
and they reached the SMTP TLS and base64 attachment branches: a sample
whose spoofed message carried an SVG or zipped HTML attachment made the
email rejected ("Missing field: 'feedback'", "Unable to parse"), and a
gzipped aggregate report or a tlsrpt+json attachment inside it was
returned as that report instead of the failure report. The sample is
the reported message, so after it is taken every part inside it is
skipped (by the id() of each part its walk() yields). Tests cover SVG,
zipped HTML, JSON, gzipped aggregate XML, tlsrpt+json and PDF
attachments; the failure samples parse to byte-identical output.

The Bug fixes bullet also says that a legacy text part followed by a
feedback-report part with no sample after it is now invalid, with a
test.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cap SMTP TLS reports and test the format skips

parse_smtp_tls_report_json() decoded and json.loads()-ed a report of
any size; only the gzipped path was capped, and 60 MiB of "[]," cost
+1,565 MiB. It now rejects bytes or text over MAX_DECOMPRESSED_REPORT_SIZE
first, as InvalidSMTPTLSReport, so every report type is held to the cap.

parse_report_file() names the actual reason it skips the email parser
(the leading "<" or "{", or the size cap) rather than a combined one,
and a size message for bytes no longer says "of UTF-8". New tests kill
the mutants the review found surviving: removing the "<" skip for JSON,
narrowing the email skip to "<", the bytes check >= for >, and
removing the SMTP TLS cap.

The constant comment and the changelog state the memory at the cap as
measured on the current code: +345 MiB on the lxml recovery path and
+302 MiB on expat parsing the text directly, +500 MiB and +452 MiB
through parse_report_file(). The changelog says only the exception's
cause changed for files skipped by format, not the message.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip every sample candidate and correct the stated measurements

sample_parts held only the first sample candidate's parts, so a later
message/rfc822 (or one after a text/rfc822-headers sample) was dropped
as a sample but its parts were still walked, and a gzipped aggregate
report inside it was returned in place of the failure report. Every
candidate's parts are now skipped. Tests cover both shapes, and a test
that a gzipped aggregate report beside the sample is still returned
pins the rule (inside the sample: skipped; beside it: read) and fails
when the set is built from the whole message.

parse_report_file() no longer builds email reasons for content that
begins with "<" or "{", which its message never shows, and the size
reason reads "Invalid report email: the report is larger than 32 MiB".
The memory figures are re-measured in a fresh process with the input
read from disk before the baseline: +399 MiB on the recovery path and
+338 to +355 MiB on expat for the 31 MiB shape, +1,505 MiB for 60 MiB of
JSON before the SMTP TLS cap; the constant comment, the changelog and the
test docstrings state them. RFC 6591 section 3.1 is cited for the sample's
place, and application/json is named among the attachments that made a
failure report invalid.

Refs GHSA-qcc2-4x3v-v4g7
Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Scale the timed tests' bounds to the machine running them

The mailsuite fix's first CI run on GitHub's shared runners failed six
timed tests: the suite ran four to six times slower there than on a
workstation, and checks doing real work took longer than their fixed
bounds. Several bounds here have far less margin (0.2 s on a test that
takes 0.18 s whole), so they would fail the same way.

tests/timing.py adds budget(seconds), which measures a fixed reference
parse twice just before the timed call and multiplies the bound by the
slowdown against a reference machine; a faster machine keeps the written
bound. Every timed assert now compares against budget(<written seconds>)
computed before the timer starts. No written bound changed and no assert
was removed.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

AGENTS.md

@@ -78,7 +78,7 @@ The parser detects an RFC 9990 report from the `urn:ietf:params:xml:ns:dmarc-2.0
 
 RFC 9990's `PolicyOverrideType` enumeration is `{local_policy, mailing_list, other, policy_test_mode, trusted_forwarder}`. `policy_test_mode` is new (emitted when `t=y` suppresses enforcement); `forwarded` and `sampled_out` were removed. Override types are stored as-is and warned about on mismatch.
 
-Several elements (`extra_contact_info`, `error`, `comment`, `human_result`) are `langAttrString` in RFC 9990 — i.e. xs:string with an optional `lang` attribute. When the reporter sends the attribute, xmltodict turns the element into `{"#text": "...", "@lang": "en"}`; the parser unwraps that to a plain string via `_text()`.
+Several elements (`extra_contact_info`, `error`, `comment`, `human_result`) are `langAttrString` in RFC 9990 — i.e. xs:string with an optional `lang` attribute. xmltodict would turn such an element into `{"#text": "...", "@lang": "en"}`, but the report is parsed with `xml_attribs=False`, so no attribute reaches the parser, and `_scalar()` coerces every single-valued field the schema types as a string (`extra_contact_info`, `comment`, `human_result` among them) to a string or `None` (a child element's dict gives its `#text`, a repeated element `None`), so no dict or list reaches the outputs in place of a string and changes a field's mapping (GHSA-qpcg-29vr-jwm3). `error` may repeat, so it goes through `_text()` item by item instead: a repeated one stays a list of strings, and `_text()` now only meets dicts made by child elements.
 
 Failure reports (RFC 9991): `Identity-Alignment` and `Auth-Failure` are split on CFWS-aware commas (each token stripped per the RFC 9991 ABNF), and a warning is logged when either REQUIRED field is missing.
 
@@ -126,7 +126,7 @@ IP address info cached for 4 hours, seen aggregate report IDs cached for 1 hour
 
 ### Regular expressions on untrusted input
 
-Assume every regex in the package runs on content a remote sender controls (message bodies, headers, attachments inflated to the 100 MiB decompression cap), so a pattern that backtracks super-linearly is a remote CPU-exhaustion bug, not a style nit (GHSA-7grx-mj6g-qxmc: a cubic pattern on whitespace; GHSA-4h9r-c435-j834: a quadratic `.*>` that a 194 KB gzip attachment could turn into days of CPU). Before adding or editing a pattern:
+Assume every regex in the package runs on content a remote sender controls (message bodies, headers, attachments inflated to the 32 MiB decompression cap), so a pattern that backtracks super-linearly is a remote CPU-exhaustion bug, not a style nit (GHSA-7grx-mj6g-qxmc: a cubic pattern on whitespace; GHSA-4h9r-c435-j834: a quadratic `.*>` that a 194 KB gzip attachment could turn into days of CPU). Before adding or editing a pattern:
 
 - **Adjacent quantifiers must not overlap.** `\s*([a-zA-Z\s]+)` lets both parts consume the same whitespace, so every split is retried. Make neighbouring classes disjoint (`[ \t]*` next to `[A-Za-z]`).
 - **A repeated class must exclude the delimiter it stops on.** Write `\([^()]*\)` and `<tag[^<>]*>`, never `\(.*\)` or `<tag.*>`: a greedy `.*` scans to end of line and backtracks for every occurrence, which is quadratic whenever the closer is missing, and also matches past the *first* closer (the old schema-tag regex deleted a whole minified report that way).

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## Unreleased
+
+### Security
+
+- **Crafted email messages no longer take seconds to minutes and hundreds of megabytes to parse** ([GHSA-qcc2-4x3v-v4g7](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-qcc2-4x3v-v4g7), affects 11.2.1 and earlier). `parse_report_email()` (mailbox, mbox and file input) and the failure-sample parser `parsedmarc.utils.parse_email()` gave the whole message to mail-parser and Python's `email` package with no limit on blank lines, header lines, header field size, MIME parts or nested multiparts. Parsing builds one `Message` per blank line after a `message/delivery-status` part (512 KiB of blank lines took 6.1 s and 283 MiB of extra memory on 11.2.1 under Python 3.12) and tests every line against every open boundary of nested multiparts (900 levels around 30,000 parts, 0.3 MB, took 10.9 s); a 4 MiB flood of header lines took 1.4 s, and a 1 MB `To:` address group of 250,000 members took 1.9 s, or 44 s through mailsuite's own `parse_email()`, which parsed the group whole. With `n_procs = 1` the rest of the mailbox waited behind such a message, and a process with a memory cap was killed before the message was moved, so it was parsed again on the next run. Both parsers now call `mailsuite.utils.check_message_limits()`, new in mailsuite 2.3.5 ([GHSA-748w-c9w9-79xp](https://github.com/seanthegeek/mailsuite/security/advisories/GHSA-748w-c9w9-79xp)), before mail-parser sees the message. It parses the message with the `email` package's own feed parser under the `compat32` policy, the parser mail-parser and `email.message_from_string()` use, and stops as soon as the message has more than 10,000 parts (the message itself, its MIME parts, the bodies of `message/*` parts and each block of a `message/delivery-status` body), nests them more than 100 levels deep, or makes the parser test a line against an open boundary more than 20,000,000 times. It also stops when the message's header blocks hold more than 100,000 lines in all (folded lines and lines the parser keeps only as defects included). It then rejects a parsed message with a header field longer than 65,536 characters (name, colon and raw value, folded lines included), or whose `From`, `Sender`, `Reply-To`, `To`, `Cc`, `Bcc` and `Delivered-To` headers hold more than 262,144 characters in all, counted in every part; that limit bounds mail-parser's own address parsing (it parses `From`, `Reply-To`, `To`, `Cc`, `Bcc` and `Delivered-To`), which does work for each malformed address item. When a part lacks its start boundary, mail-parser parses the top-level epilogue again, and that text is checked with the same limits. `parse_report_email()` also no longer renders every part it walks as text: it renders a part only where it reads it, because rendering each level of a deep nesting rendered everything inside it again (98 nested multiparts around a 20 MiB leaf took 26.6 s, a chain of 99 `message/rfc822` parts around a 10 MiB leaf 11.3 s, and a chain of 99 `message/feedback-report` parts around a 10 MiB leaf 11.6 s, all within the limits; each now takes under a second). An over-limit email raises `InvalidDMARCReport("Message rejected: Message exceeds <limit> (<value>)")`, which the mailbox loop files under `Invalid`; an over-limit failure-report sample raises `InvalidFailureReport("Sample message rejected: …")`. Each crafted message from the report is now rejected in under half a second (the 4 MiB header flood, the slowest, in 0.2 s). **mailsuite 2.3.5 or later is now required.** The floor applies to the base dependency and to the `msgraph`, `gmail` and `all` extras. mailsuite 2.3.5 also removes RFC 5322 group syntax from address header values over 8 KiB before parsing them, caps the address header value its own `parse_email()` reads at 65,536 characters, and fixes a quadratic `url_regex`; all only affect callers of mailsuite's `parse_email()`, which parsedmarc does not use. A follow-up issue tracks a per-job timeout in `parallel_map()`. **Compatibility:** the limits are fixed; there is no new option. A message with more than 10,000 parts or parts nested more than 100 levels deep is now rejected as invalid instead of being parsed or, when nested past Python's recursion limit, failing with a `ParserError`; in mbox input such a message (the nested message of GHSA-qxwv-2vwj-v828 is one) is now skipped with a warning instead of stopping the run. A `message/rfc822` attached inside a failure report's sample no longer replaces the sample, since a feedback report has one sample ([RFC 6591 §3.1](https://www.rfc-editor.org/rfc/rfc6591#section-3.1)), and only the first `message/feedback-report` part of a message is read, since a feedback report has one ([RFC 6591 §3](https://www.rfc-editor.org/rfc/rfc6591#section-3)). Reported by helixnebulatech.
+- **Deeply nested or very wide aggregate report XML no longer exhausts memory** ([GHSA-q7gm-8cj9-94hr](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-q7gm-8cj9-94hr), affects 11.2.1 and earlier). `parse_aggregate_report_xml()` passed the decompressed report, up to the 100 MiB decompression cap, to `xmltodict.parse()` with no limit on nesting depth or element count. xmltodict keeps one stack entry per open element and expat reports missing end tags only at the end of the input, so a gzip attachment of about 30 KB holding 30 MiB of unclosed `<a>` tags needed more than 2 GB of memory (2 MiB of them cost 255 MiB on 11.2.1). The out-of-memory killer ended the process before the message was moved, so every restart parsed it again. A document of millions of sibling elements did the same at a smaller scale, and a document whose first parse failed was parsed again through lxml recovery and a second xmltodict pass. While expat reads the document, before xmltodict adds each element to its tree, parsedmarc now counts nesting depth and nodes (elements plus attributes); a report deeper than 100 levels (`MAX_XML_DEPTH`) or with more than 1,000,000 nodes (`MAX_XML_NODES`) is rejected as `InvalidAggregateReport("Invalid XML: …")` without trying lxml recovery. Before any parser sees the document, a tag longer than 65,536 characters (`MAX_XML_TAG_LENGTH`) is rejected too, because expat and libxml2 build an element's whole attribute list before its attributes can be counted (one element with 900,000 attributes, 9.3 MiB, within the node limit, took 1.1 s and +200 MiB; it is now rejected in under 0.1 s). Because a quoted attribute value may hold a raw `>`, which splits a tag for that scan, a document with more `=` characters than the node limit is rejected first, in one pass (2,000,000 `x=">"` attributes, 23.7 MiB, took 1.4 s and +310 MiB before the node limit rejected them; they are now rejected in under 0.1 s). The decompression cap for a report attachment is lowered from 100 MiB to 32 MiB (`MAX_DECOMPRESSED_REPORT_SIZE`), and an uncompressed report of more than 32 MiB of UTF-8 is rejected the same way, as is a report file of that size, which is no longer tried as a report email: at the node limit a legitimate report is at most about 28 MB, and with that cap a report's text is bounded along with its node count. When the first parse fails and lxml recovery runs, a parser target that builds no tree first counts the document's elements, attributes and namespace declarations against the same node limit (a 22 MiB wide document behind a malformed XML declaration took 5.0 s and 1,532 MiB of extra memory before, 0.57 s and 23 MiB now), and the recovered XML then goes through the same depth and node limits in the second xmltodict pass. A follow-up issue tracks a persisted failed-attempt marker, so that a message that kills the process before it is filed is moved to `Invalid` after a number of runs. **Compatibility:** there is no new option. A real report is fewer than 10 levels deep and the largest sample (909 KB) has 32,019 nodes, so the node limit is a report of about 28 MB of XML; a larger report is now rejected. A report at the size cap needs about 400 MB of memory on the lxml recovery path and about 350 MB otherwise when its text is parsed directly, and about 500 MB and 450 MB through `parse_report_file()`, which also holds the file's bytes, where a report under the old 100 MiB decompression cap could need gigabytes. An aggregate or SMTP TLS report of more than 32 MiB of UTF-8, compressed or not, is now rejected, where 100 MiB was accepted (an uncompressed SMTP TLS report had no cap: 60 MiB of JSON cost +1,505 MiB). `parse_report_file()` no longer tries a file that begins with `<` as SMTP TLS JSON or a report email, nor one that begins with `{` as a report email; its error message is unchanged, and the exception's cause now matches the format the file begins with. It also keeps only each failed attempt's message while it tries the next format, so a file at the size cap on the lxml recovery path takes +500 MiB there, where it took +1,046 MiB. The XML parse step is about 20% slower, and a malformed document that needs lxml recovery is now parsed twice by lxml (once to count, once to build the tree). Reported by helixnebulatech.
+- **An aggregate report with a huge date range can no longer expand into millions of records or fill an Elasticsearch or OpenSearch cluster with indexes** ([GHSA-p85v-578x-3wcf](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-p85v-578x-3wcf), affects 11.2.1 and earlier). An aggregate report whose `<date_range>` was longer than `normalize_timespan_threshold_hours` was split into one record per calendar day, with no limit on the number of days. A report under 2 KB declaring a range of millions of days with a large `<count>` expanded one `<record>` into millions of records (200,000 days gave 200,000 records in 0.8 s; the reporter's 2.9 million days exhausted 2 GB of memory). With Elasticsearch or OpenSearch output each of those days created its own dated index of one shard until the cluster reached `cluster.max_shards_per_node` (1,000 by default), after which it refused every new index for every application using it. A report whose date range is negative or longer than 366 days (`MAX_REPORT_SPAN_DAYS`) is now rejected with `InvalidAggregateReport`. A report whose split would produce more than 250,000 records (records times days, `MAX_NORMALIZED_RECORDS`) is not split: its records are kept whole over the report's own date range, and one warning is logged (10,000 records over 366 days, each with a large `<count>`, produced 3,670,000 records in 13.5 s and 1.7 GB of memory before). In Elasticsearch and OpenSearch a single report creates at most 31 dated indexes (`MAX_INDEXES_PER_REPORT`): any further records are saved to the report's first index with their own dates, and one warning is logged for the report. Dated index names still come from each report's own dates, so reports with many different dates still create one index per date; data streams (`use_data_streams`) are affected by neither. **Compatibility:** a genuine report spanning more than 366 days is now rejected; a report too large to split has `timespan_requires_normalization: false` and records with `normalized_timespan: false`; there is no new option. Reported by helixnebulatech.
+- **One crafted report can no longer make an output fail and stop every run on the same mailbox** ([GHSA-4272-4r8g-6c5h](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-4272-4r8g-6c5h), affects 11.2.1 and earlier). Some reports made a configured output raise an exception that `process_reports()` did not catch. It escaped the mailbox save callback, parsedmarc logged "Mailbox Error" and exited with status 1, and the batch stayed in the mailbox, so every one-shot run failed on the same message, outputs that do not deduplicate received the batch's legitimate reports again on each attempt, and watch mode finally moved the whole batch to `Unsaved`. The triggers were a failure report whose sample has an empty `From:`, `To:` or `Reply-To:` header (a legitimate `To: undisclosed-recipients:;` too) with Elasticsearch or OpenSearch output (`IndexError`); an SMTP TLS report with a field of the wrong type, such as `"policy-string": [1]` with file output (`TypeError` in the CSV writer) or `"start-datetime": 5` with Elasticsearch, OpenSearch, Splunk HEC or PostgreSQL output; and an aggregate report whose policy override `<reason>` has no `<type>`, or a `<type>` with an attribute or given twice, with file output (`KeyError`, `TypeError`). Every output destination in `process_reports()` now records any exception as an output error, as the aggregate Elasticsearch and OpenSearch blocks already did (the failure and SMTP TLS Elasticsearch and OpenSearch blocks, every PostgreSQL and Splunk HEC block and the file output lacked that catch-all), so the batch is reported unsaved and `fail_on_output_error` still applies. SMTP TLS reports are now type-checked against RFC 8460 §4.4 and rejected as `InvalidSMTPTLSReport` otherwise: a string for `organization-name`, `report-id`, `policy-domain`, `policy-type`, `result-type` and the optional failure-detail strings, a string or a list of strings for `contact-info`, a list of strings for `policy-string`, an integer or `null` for each session count (`true` and `false` are rejected), and a list of objects for `failure-details`. `start-datetime` and `end-datetime` must convert with the same functions the outputs use and be at most 64 characters long, because dateutil took 12.4 s to fail on a date of 1,000,000 digits. A string holding a lone surrogate (JSON `"\ud800"`), which I-JSON ([RFC 7493 §2.1](https://www.rfc-editor.org/rfc/rfc7493#section-2.1)) forbids, is rejected too: it passed every type check and made printing the results to standard output raise `UnicodeEncodeError`, which `process_reports()` now also records as an output error; a report that cannot be printed to the console (a non-UTF-8 console, a closed pipe) is now recorded as an output error instead of ending the run. A policy override reason's `type` and `comment` are normalised to a string or `null` (an element with an attribute to its text, a repeated element to null), so the CSV writer and the Elasticsearch and OpenSearch saves no longer fail on them. An address header with no address is left out of the Elasticsearch and OpenSearch failure document and of the duplicate query. The Splunk HEC SMTP TLS save raises `SplunkError` on a date it cannot convert, and the PostgreSQL date helper passes a non-string value through unchanged. **Compatibility:** an SMTP TLS report with a field of the wrong type that used to parse is now invalid and goes to `Invalid`; a `policy-string` that is not a list is now rejected (it was silently dropped), while a `null` one is still accepted as absent. A policy override reason without a `type` now has `type: null` and is written as `none` in CSV; one whose `type` has an attribute is stored as its text, and a repeated one as null. Reported by helixnebulatech.
+- **One crafted report can no longer change the field types of an Elasticsearch or OpenSearch index and get every later report rejected** ([GHSA-qpcg-29vr-jwm3](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-qpcg-29vr-jwm3), affects the Elasticsearch and OpenSearch outputs in 11.2.1 and earlier, data streams included). Indexes and the data-stream index template were created without field mappings, so the cluster typed each field from the first document that held it, with date detection on, and a dated index is named from the report's own begin date. One crafted report could make `report_id` a `date` field (`<report_id>2026-11-09</report_id>`), turn `sample.headers.subject` into an object (with a sample header named `Subject.x`), or use up `index.mapping.total_fields.limit` (1,000 fields) with invented sample header names. Every later legitimate report for that index, or for that data stream until its next rollover, was then rejected with a mapping error, retried and moved to `Unsaved`; this was confirmed against Elasticsearch 8.19.7 and OpenSearch 2.19.3. New indexes and data-stream templates are now created with date detection off and the date fields declared as dates (`@timestamp`, `date_range`, `date_begin` and `date_end`; for failure reports `@timestamp`, `arrival_date` and `sample.date`); when an aggregate report is parsed, every XML attribute is dropped and every field the schema types as a string is stored as a string or null, so a report cannot change a field's type (an element with an attribute or a child element, such as `<p a="1">none</p>` or `<p><x>1</x></p>`, became an object that mapped the field as one). In failure documents `sample.headers` is mapped with `dynamic: false` and indexes only 29 headers, those the shipped dashboards reference plus `cc`; every header stays in `_source` except a `From`, `To` or `Reply-To` header that holds no address (see GHSA-4272-4r8g-6c5h above), and a `.` in a header name is stored as `_`. `create_indexes()` and `create_data_streams()` take a new optional `mappings` argument. **Compatibility:** existing dated indexes are not changed; the protection applies to dated indexes created by 11.2.2 or later, and, because parsedmarc puts the data stream template again in every process, to an existing data stream's next backing index after a rollover. On those, sample headers outside the list are no longer searchable. The new "Field mappings" sections of the Elasticsearch and OpenSearch documentation show how to add one through the `<stream>@custom` component template (Elasticsearch data streams), an index template (dated indexes) or the put-mapping API (OpenSearch data streams, repeated after each rollover). For string fields, an aggregate report element with text and a stray child element now yields its text, where it yielded an empty string or an object before: `samples/aggregate/invalid_xml.xml`, whose `<header_from>bad<xml.net</header_from>` lxml recovery turns into mixed content, now gives the header-from domain `bad`. A `<begin>` or `<end>` with mixed content is still rejected. This narrows, but does not replace, the explicit mappings proposed in [#782](https://github.com/domainaware/parsedmarc/issues/782). Reported by helixnebulatech.
+
+### Bug fixes
+
+- **A failure report's RFC 6591 feedback report and sample can no longer be replaced by legacy report text.** `parse_report_email()` also reads the legacy text format, a `text/plain` part beginning "A message claiming to be from you has failed" that carries the source IP and the sample, and a later such part replaced the `message/feedback-report` part and the sample already read. A `text/plain` part inside the sample, which is the spoofed message's own text, could therefore set the report's Source-IP and sample. Once a `message/feedback-report` part is read, legacy text parts are ignored, and the RFC 6591 part and its sample win over a legacy part that came before it; a legacy part followed by a `message/feedback-report` part with no sample after it (RFC 6591 §3.1 puts the sample after the report part) is now rejected as `InvalidDMARCReport`, where a mix of the two was returned. Parts inside the sample, which are the reported message's own attachments, are no longer read as reports either: an SVG, zipped HTML or `application/json` attachment there made the whole email invalid, and a gzipped aggregate report or SMTP TLS JSON there was returned in place of the failure report. This holds for a second sample part too, which is not read as the sample; a report attached beside the sample, outside it, is still read.
+- **SMTP TLS reports keep their MX host patterns** ([#942](https://github.com/domainaware/parsedmarc/issues/942)). RFC 8460 §4.4 keys a policy's MX host patterns `mx-host`; `mx-host-pattern` is the name of the value's type ("a JSON array of strings"), and the example in Appendix B gives a plain string. parsedmarc read only `mx-host-pattern`, so every RFC-conformant report lost the field. `mx-host` is now read first, as a list of strings or a plain string (which becomes a one-element list); `mx-host-pattern` is kept as a fallback, and a plain string there is now kept too. The output key `mx_host_patterns` and its PostgreSQL `TEXT[]` column are unchanged, and `samples/smtp_tls/rfc8460.json` now yields `["*.mail.company-y.example"]`. Reported by kowax.
+
 ## 11.2.1
 
 ### Security

docs/source/elasticsearch.md

@@ -413,6 +413,13 @@ the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
 easy to comply with records retention regulations such as GDPR: delete
 the indexes for the dates you no longer keep. For more information,
 check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
+A single aggregate report creates at most 31 dated indexes
+(`MAX_INDEXES_PER_REPORT`): when its records would use more than 31
+dated indexes, the later records are saved to the report's first dated
+index with their own dates, and a warning is logged. Dated index names still come
+from each report's own dates, so reports with many different dates
+still create one index per date; data streams (`use_data_streams`) are
+affected by neither.
 
 Set `use_data_streams = True` in the `[elasticsearch]` section to write
 to [data streams] instead. Each report type then goes to one data stream
@@ -436,7 +443,8 @@ exist yet. The template:
   reports, which is the report's begin date unless parsedmarc split a
   report spanning more than a day into daily records; the arrival date
   for failure reports; and the report's begin date for SMTP TLS reports.
-  The other fields are mapped dynamically, as in the dated indexes;
+  The other fields get the same [field mappings](#field-mappings) as the
+  dated indexes;
 - includes a component template named `<stream>@custom`, such as
   `dmarc_aggregate@custom`, if one exists.
 
@@ -519,6 +527,36 @@ PUT _data_stream/dmarc_aggregate/_lifecycle
 }
 ```
 
+### Field mappings
+
+Indexes and data stream templates created by parsedmarc 11.2.2 or later
+turn date detection off and declare the date fields (`date_range`,
+`date_begin`, `date_end`, `arrival_date`, `@timestamp`, and
+`sample.date`), so a report
+cannot make a text field such as `report_id` a date and block every
+later report for that index. In failure reports, only these sample
+headers are indexed: `authentication-results`, `auto-submitted`, `cc`,
+`content-transfer-encoding`, `content-type`, `date`, `from`,
+`in-reply-to`, `message-id`, `mime-version`, `received`, `references`,
+`reply-to`, `return-path`, `subject`, `thread-index`, `thread-topic`,
+`to`, `x-auto-response-suppress`, `x-exclaimer-md-config`,
+`x-linkedin-fe`, `x-mailer`, `x-ms-exchange-generated-message-source`,
+`x-ms-exchange-inbox-rules-loop`, `x-ms-exchange-parent-message-id`,
+`x-ms-exchange-transport-fromentityheader`, `x-ms-has-attach`,
+`x-ms-tnef-correlator`, and `x-onpremexternalip`. Every header is still
+kept in `_source` under its lower-cased name, with any `.` replaced by
+`_`, except a `From`, `To` or `Reply-To` header that holds no address,
+which is left out of the document. To make another header searchable,
+declare it under
+`sample.headers.properties` as `text` with a `keyword` sub-field: in the
+`dmarc_failure@custom` component template for a data stream, or in an
+index template whose `index_patterns` match the dated indexes
+(`dmarc_failure-*`). Either applies only to indexes created after it.
+Dated indexes created by earlier releases keep their mappings unchanged.
+parsedmarc puts the data stream template again in every process
+before its first save to the stream, so after upgrading, an existing
+stream's next backing index after a rollover gets these mappings too.
+
 ### Switching an existing installation to data streams
 
 Dated indexes that already exist are left alone. They keep working next

docs/source/opensearch.md

@@ -14,6 +14,13 @@ index for each day (or each month, with `monthly_indexes`), named after
 the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
 easy to comply with records retention regulations such as GDPR: delete
 the indexes for the dates you no longer keep.
+A single aggregate report creates at most 31 dated indexes
+(`MAX_INDEXES_PER_REPORT`): when its records would use more than 31
+dated indexes, the later records are saved to the report's first dated
+index with their own dates, and a warning is logged. Dated index names still come
+from each report's own dates, so reports with many different dates
+still create one index per date; data streams (`use_data_streams`) are
+affected by neither.
 
 Set `use_data_streams = True` in the `[opensearch]` section to write to
 [data streams] instead. Each report type then goes to one data stream
@@ -39,7 +46,8 @@ exist yet. The template:
   reports, which is the report's begin date unless parsedmarc split a
   report spanning more than a day into daily records; the arrival date
   for failure reports; and the report's begin date for SMTP TLS reports.
-  The other fields are mapped dynamically, as in the dated indexes.
+  The other fields get the same [field mappings](#field-mappings) as the
+  dated indexes.
 
 parsedmarc puts the template again in every process, so changes made to
 it directly do not last.
@@ -109,6 +117,38 @@ was created, whatever dates the reports in it cover. If you need
 retention by report date, keep using dated indexes and delete them by
 name.
 
+### Field mappings
+
+Indexes and data stream templates created by parsedmarc 11.2.2 or later
+turn date detection off and declare the date fields (`date_range`,
+`date_begin`, `date_end`, `arrival_date`, `@timestamp`, and
+`sample.date`), so a report
+cannot make a text field such as `report_id` a date and block every
+later report for that index. In failure reports, only these sample
+headers are indexed: `authentication-results`, `auto-submitted`, `cc`,
+`content-transfer-encoding`, `content-type`, `date`, `from`,
+`in-reply-to`, `message-id`, `mime-version`, `received`, `references`,
+`reply-to`, `return-path`, `subject`, `thread-index`, `thread-topic`,
+`to`, `x-auto-response-suppress`, `x-exclaimer-md-config`,
+`x-linkedin-fe`, `x-mailer`, `x-ms-exchange-generated-message-source`,
+`x-ms-exchange-inbox-rules-loop`, `x-ms-exchange-parent-message-id`,
+`x-ms-exchange-transport-fromentityheader`, `x-ms-has-attach`,
+`x-ms-tnef-correlator`, and `x-onpremexternalip`. Every header is still
+kept in `_source` under its lower-cased name, with any `.` replaced by
+`_`, except a `From`, `To` or `Reply-To` header that holds no address,
+which is left out of the document. To make another header searchable
+in the dated indexes, declare it
+under `sample.headers.properties` as `text` with a `keyword` sub-field
+in an index template whose `index_patterns` match them
+(`dmarc_failure-*`); it applies only to indexes created after it. The
+data stream template composes no component template, so for a data
+stream add the header to its backing indices with the update mapping
+API (`PUT dmarc_failure/_mapping`), and again after each rollover.
+Dated indexes created by earlier releases keep their mappings unchanged.
+parsedmarc puts the data stream template again in every process
+before its first save to the stream, so after upgrading, an existing
+stream's next backing index after a rollover gets these mappings too.
+
 ### Switching an existing installation to data streams
 
 Dated indexes that already exist are left alone. They keep working next

parsedmarc/cli.py

@@ -2656,7 +2656,13 @@ def _main():
         )
 
         if not opts.silent:
-            print(output_str)
+            try:
+                print(output_str)
+            except (UnicodeEncodeError, OSError) as error_:
+                # A report string the console's encoding cannot write, or a
+                # closed pipe, must not escape process_reports()
+                # (GHSA-4272-4r8g-6c5h).
+                log_output_error("Standard output exception", error_.__str__())
         if opts.output:
             try:
                 save_output(
@@ -2670,12 +2676,18 @@ def _main():
                     smtp_tls_csv_filename=opts.smtp_tls_csv_filename,
                 )
             except (OSError, ValueError) as error_:
-                # The only output destination that was not already caught:
-                # a full disk or an unwritable directory used to crash the
+                # A full disk or an unwritable directory used to crash the
                 # run outright, and now that a failed save holds mailbox
                 # messages back it also has to be recorded like any other
                 # destination's failure rather than escaping.
                 log_output_error("File output", str(error_))
+            except Exception as error_:
+                # Every destination below ends with the same catch-all: a
+                # report a sender shaped to make one writer raise (a
+                # wrong-typed field reaching the CSV writer, say) must mark
+                # the batch unsaved, not escape process_reports() and stop
+                # the whole mailbox run (GHSA-4272-4r8g-6c5h).
+                log_output_error("File output exception", error_.__str__())
 
         kafka_client = clients.get("kafka_client")
         s3_client = clients.get("s3_client")
@@ -2755,6 +2767,8 @@ def _main():
                     logger.warning(warning.__str__())
                 except postgres.PostgreSQLError as error_:
                     log_output_error("PostgreSQL", error_.__str__())
+                except Exception as error_:
+                    log_output_error("PostgreSQL exception", error_.__str__())
 
                 try:
                     if syslog_client:
@@ -2790,6 +2804,8 @@ def _main():
                         hec_client.save_aggregate_reports_to_splunk(aggregate_reports_)
                 except splunk.SplunkError as e:
                     log_output_error("Splunk HEC", e.__str__())
+                except Exception as error_:
+                    log_output_error("Splunk HEC exception", error_.__str__())
 
         if opts.save_failure:
             for report in reports_["failure_reports"]:
@@ -2813,6 +2829,8 @@ def _main():
                     log_output_error("Elasticsearch", error_.__str__())
                 except InvalidDMARCReport as error_:
                     log_output_error("Invalid DMARC report", error_.__str__())
+                except Exception as error_:
+                    log_output_error("Elasticsearch exception", error_.__str__())
 
                 try:
                     shards = opts.opensearch_number_of_shards
@@ -2834,6 +2852,8 @@ def _main():
                     log_output_error("OpenSearch", error_.__str__())
                 except InvalidDMARCReport as error_:
                     log_output_error("Invalid DMARC report", error_.__str__())
+                except Exception as error_:
+                    log_output_error("OpenSearch exception", error_.__str__())
 
                 try:
                     if kafka_client:
@@ -2856,6 +2876,8 @@ def _main():
                     logger.warning(warning.__str__())
                 except postgres.PostgreSQLError as error_:
                     log_output_error("PostgreSQL", error_.__str__())
+                except Exception as error_:
+                    log_output_error("PostgreSQL exception", error_.__str__())
 
                 try:
                     if syslog_client:
@@ -2891,6 +2913,8 @@ def _main():
                         hec_client.save_failure_reports_to_splunk(failure_reports_)
                 except splunk.SplunkError as e:
                     log_output_error("Splunk HEC", e.__str__())
+                except Exception as error_:
+                    log_output_error("Splunk HEC exception", error_.__str__())
 
         if opts.save_smtp_tls:
             for report in reports_["smtp_tls_reports"]:
@@ -2914,6 +2938,8 @@ def _main():
                     log_output_error("Elasticsearch", error_.__str__())
                 except InvalidDMARCReport as error_:
                     log_output_error("Invalid DMARC report", error_.__str__())
+                except Exception as error_:
+                    log_output_error("Elasticsearch exception", error_.__str__())
 
                 try:
                     shards = opts.opensearch_number_of_shards
@@ -2935,6 +2961,8 @@ def _main():
                     log_output_error("OpenSearch", error_.__str__())
                 except InvalidDMARCReport as error_:
                     log_output_error("Invalid DMARC report", error_.__str__())
+                except Exception as error_:
+                    log_output_error("OpenSearch exception", error_.__str__())
 
                 try:
                     if kafka_client:
@@ -2957,6 +2985,8 @@ def _main():
                     logger.warning(warning.__str__())
                 except postgres.PostgreSQLError as error_:
                     log_output_error("PostgreSQL", error_.__str__())
+                except Exception as error_:
+                    log_output_error("PostgreSQL exception", error_.__str__())
 
                 try:
                     if syslog_client:
@@ -2992,6 +3022,8 @@ def _main():
                         hec_client.save_smtp_tls_reports_to_splunk(smtp_tls_reports_)
                 except splunk.SplunkError as e:
                     log_output_error("Splunk HEC", e.__str__())
+                except Exception as error_:
+                    log_output_error("Splunk HEC exception", error_.__str__())
 
         if opts.la_dce:
             try:

parsedmarc/constants.py

@@ -10,3 +10,35 @@ DEFAULT_DNS_MAX_RETRIES = 0
 # operator's anycast outage or authoritative-server incompatibility falls
 # through to a different provider within one resolve() call.
 RECOMMENDED_DNS_NAMESERVERS = ("1.1.1.1", "8.8.8.8")
+
+# Limits on the XML of an aggregate report, enforced while expat reads it and
+# before xmltodict builds the tree (GHSA-q7gm-8cj9-94hr). A real report is
+# fewer than 10 elements deep. Nodes are elements plus attributes; the
+# number of "=" characters bounds the attributes, and the samples hold at
+# most 3 (namespace declarations). The largest sample has 32,019 nodes
+# (909 KB), so the limit is a report of about 28 MB. With the 32 MiB size cap
+# (MAX_DECOMPRESSED_REPORT_SIZE), 999,000 distinct elements with 14
+# characters of text each (31 MiB, read from disk before the baseline) need
+# +399 MiB on the lxml recovery path and +338 to +355 MiB on expat when the
+# text is parsed directly, and +500 MiB and +452 MiB through
+# parse_report_file(), which also holds the file's bytes.
+MAX_XML_DEPTH = 100
+MAX_XML_NODES = 1_000_000
+# Longest text between a "<" and the next ">" in an aggregate report. expat
+# and libxml2 build a start tag's whole attribute list before reporting it,
+# so attributes cannot be counted before they cost memory. The longest tag
+# in the samples is the root element's namespace declarations, 108
+# characters (a 234-character comment is the longest construct).
+MAX_XML_TAG_LENGTH = 65_536
+
+# A report's <date_range> longer than this many days is rejected. A real
+# aggregate report covers a day (RFC 7489 §6.3: the "ri" tag's default
+# interval is 86400 seconds); a year leaves room for mistakes, and
+# normalization emits at most one record per day.
+MAX_REPORT_SPAN_DAYS = 366
+# Most distinct dated indexes one report may create in Elasticsearch or
+# OpenSearch (one shard each); further records go to the report's first index.
+MAX_INDEXES_PER_REPORT = 31
+# Most records per-day normalization may produce from one report (records
+# times days). A larger report keeps its records whole, over its own range.
+MAX_NORMALIZED_RECORDS = 250_000

parsedmarc/elastic.py

@@ -24,8 +24,9 @@ from elasticsearch.dsl import (
 from elasticsearch.helpers import reindex
 
 from parsedmarc import InvalidFailureReport
+from parsedmarc.constants import MAX_INDEXES_PER_REPORT
 from parsedmarc.log import logger
-from parsedmarc.utils import human_timestamp_to_datetime
+from parsedmarc.utils import human_timestamp_to_datetime, sanitize_for_log
 
 
 class ElasticsearchError(Exception):
@@ -59,6 +60,102 @@ _READY_DATA_STREAMS: set[str] = set()
 # exact stream name, so an overlap is unlikely in the first place.
 _DATA_STREAM_TEMPLATE_PRIORITY = 501
 
+# Mappings ``create_indexes`` and ``create_data_streams`` install on the
+# indexes and templates they create. Without them the cluster infers every
+# field's type from the first document that holds it, with date detection
+# on, so a crafted report could make a string field a date (report_id
+# "2026-11-09") or use up index.mapping.total_fields.limit with invented
+# sample header names, and every later report into that index would then
+# be rejected (GHSA-qpcg-29vr-jwm3). Date detection is off; the fields that
+# hold dates are declared as dates instead, the type they got before.
+# Everything else stays dynamic and object-mapped, which the shipped
+# dashboards need (see create_indexes).
+_DATE_FIELD: dict[str, Any] = {"type": "date"}
+
+# The type Elasticsearch's dynamic mapping gives a JSON string, so a header
+# declared here has the same type as in indexes created before.
+_DYNAMIC_STRING_FIELD: dict[str, Any] = {
+    "type": "text",
+    "fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
+}
+
+# The failure sample headers indexed for search: every header the shipped
+# dashboards (dashboards/) reference, plus cc.
+# Header names are stored lower-cased. The other headers stay in _source
+# but are not indexed (``dynamic: False``).
+_FAILURE_SAMPLE_INDEXED_HEADERS = (
+    "authentication-results",
+    "auto-submitted",
+    "cc",
+    "content-transfer-encoding",
+    "content-type",
+    "date",
+    "from",
+    "in-reply-to",
+    "message-id",
+    "mime-version",
+    "received",
+    "references",
+    "reply-to",
+    "return-path",
+    "subject",
+    "thread-index",
+    "thread-topic",
+    "to",
+    "x-auto-response-suppress",
+    "x-exclaimer-md-config",
+    "x-linkedin-fe",
+    "x-mailer",
+    "x-ms-exchange-generated-message-source",
+    "x-ms-exchange-inbox-rules-loop",
+    "x-ms-exchange-parent-message-id",
+    "x-ms-exchange-transport-fromentityheader",
+    "x-ms-has-attach",
+    "x-ms-tnef-correlator",
+    "x-onpremexternalip",
+)
+
+_AGGREGATE_MAPPINGS: dict[str, Any] = {
+    "date_detection": False,
+    "properties": {
+        "@timestamp": _DATE_FIELD,
+        "date_range": _DATE_FIELD,
+        "date_begin": _DATE_FIELD,
+        "date_end": _DATE_FIELD,
+    },
+}
+
+_FAILURE_MAPPINGS: dict[str, Any] = {
+    "date_detection": False,
+    "properties": {
+        "@timestamp": _DATE_FIELD,
+        "arrival_date": _DATE_FIELD,
+        "sample": {
+            "properties": {
+                "date": _DATE_FIELD,
+                "headers": {
+                    "type": "object",
+                    "dynamic": False,
+                    "properties": {
+                        name: _DYNAMIC_STRING_FIELD
+                        for name in _FAILURE_SAMPLE_INDEXED_HEADERS
+                    },
+                },
+            }
+        },
+    },
+}
+
+_SMTP_TLS_MAPPINGS: dict[str, Any] = {
+    "date_detection": False,
+    "properties": {
+        "@timestamp": _DATE_FIELD,
+        "date_range": _DATE_FIELD,
+        "date_begin": _DATE_FIELD,
+        "date_end": _DATE_FIELD,
+    },
+}
+
 # Guard query for the dkim_results_combined/spf_results_combined backfill
 # (see ``migrate_indexes``). Matches only documents that have at least one
 # DKIM or SPF auth result and are missing the corresponding combined field.
@@ -677,15 +774,45 @@ def _effective_index_settings(settings: dict[str, Any] | None) -> dict[str, Any]
     return dict(settings)
 
 
-def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
+def _with_timestamp_mapping(mappings: dict[str, Any] | None) -> dict[str, Any]:
+    """Return a copy of ``mappings`` that also maps ``@timestamp`` as a date.
+
+    A data stream rejects documents unless ``@timestamp`` is a date field,
+    so the data stream template always declares it.
+
+    Args:
+        mappings (dict | None): A mappings body, or ``None``.
+
+    Returns:
+        dict: The mappings body with ``properties["@timestamp"]`` set; the
+        argument is not modified.
+    """
+    result = dict(mappings) if mappings else {}
+    result["properties"] = {
+        **result.get("properties", {}),
+        "@timestamp": {"type": "date"},
+    }
+    return result
+
+
+def create_indexes(
+    names: list[str],
+    settings: dict[str, Any] | None = None,
+    mappings: dict[str, Any] | None = None,
+):
     """
     Create Elasticsearch indexes
 
+    An index that already exists is left as it is, mappings included.
+
     Args:
         names (list): A list of index names
         settings (dict): Index settings. In Serverless mode, keys in
             ``_SERVERLESS_REJECTED_SETTINGS`` are filtered out and the
             remaining keys are passed through; defaults are skipped entirely.
+        mappings (dict | None): The mappings body to create each index
+            with, such as ``_AGGREGATE_MAPPINGS``; ``None`` leaves every
+            field to dynamic mapping.
     """
     effective_settings = _effective_index_settings(settings)
 
@@ -698,22 +825,30 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
             # fields, Vega can run nested aggregations but does not render
             # tables, and Grafana's nested bucket aggregation (9.4+) lacks
             # reverse_nested for parent-level metrics like message_count —
-            # so the dynamic `object` mapping produced by a bare create is
-            # load-bearing for the shipped dashboards. _AggregateReportDoc
-            # still declares dkim_results/spf_results with Nested(...), but
-            # that is only the DSL's in-memory shape for building documents
-            # — it is never installed as a mapping. See issue #169 and the
+            # so the dynamic `object` mapping of those fields (the mappings
+            # body declares none of them) is load-bearing for the shipped
+            # dashboards. _AggregateReportDoc still declares
+            # dkim_results/spf_results with Nested(...), but that is only
+            # the DSL's in-memory shape for building documents — it is
+            # never installed as a mapping. See issue #169 and the
             # *_combined fields on _AggregateReportDoc.
             if not index.exists():
                 logger.debug(f"Creating Elasticsearch index: {name}")
+                body: dict[str, Any] = {}
                 if effective_settings:
-                    index.settings(**effective_settings)
-                index.create()
+                    body["settings"] = effective_settings
+                if mappings:
+                    body["mappings"] = mappings
+                connections.get_connection().indices.create(index=name, body=body)
         except Exception as e:
             raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
 
 
-def create_data_streams(names: list[str], settings: dict[str, Any] | None = None):
+def create_data_streams(
+    names: list[str],
+    settings: dict[str, Any] | None = None,
+    mappings: dict[str, Any] | None = None,
+):
     """
     Create Elasticsearch data streams and their index templates
 
@@ -727,14 +862,17 @@ def create_data_streams(names: list[str], settings: dict[str, Any] | None = None
     the stream, so edits made to that template directly do not last. Each
     name is set up once per process; ``set_hosts`` resets that.
 
-    The template maps only ``@timestamp``. Like ``create_indexes``, it
-    deliberately installs no mappings for the report fields themselves (see
-    issue #169), so they keep their dynamic ``object`` mapping.
+    The template maps ``@timestamp`` as a date, on top of ``mappings``.
+    Like ``create_indexes``, it declares no ``nested`` fields (see issue
+    #169), so the report fields keep their ``object`` mapping.
 
     Args:
         names (list): A list of data stream names
         settings (dict): Index settings for the stream's backing indices,
             adjusted for Serverless the same way ``create_indexes`` does.
+        mappings (dict | None): The mappings body for the stream's backing
+            indices, such as ``_AGGREGATE_MAPPINGS``; ``None`` maps only
+            ``@timestamp``.
 
     Raises:
         ElasticsearchError: A concrete index or alias already holds one of
@@ -742,6 +880,7 @@ def create_data_streams(names: list[str], settings: dict[str, Any] | None = None
             is not remembered, so the next save retries it.
     """
     effective_settings = _effective_index_settings(settings)
+    template_mappings = _with_timestamp_mapping(mappings)
 
     for name in names:
         if name in _READY_DATA_STREAMS:
@@ -773,7 +912,7 @@ def create_data_streams(names: list[str], settings: dict[str, Any] | None = None
                     priority=_DATA_STREAM_TEMPLATE_PRIORITY,
                     template={
                         "settings": effective_settings,
-                        "mappings": {"properties": {"@timestamp": {"type": "date"}}},
+                        "mappings": template_mappings,
                     },
                     meta={
                         "description": (
@@ -1119,6 +1258,10 @@ def save_aggregate_report_to_elasticsearch(
         discovery_method=aggregate_report["policy_published"].get("discovery_method"),
     )
 
+    # Distinct dated indexes this report has used, in first-use order.
+    report_indexes: dict[str, None] = {}
+    routed_records = 0
+
     for record in aggregate_report["records"]:
         begin_date = human_timestamp_to_datetime(
             record["interval_begin"], to_utc=True, assume_utc=True
@@ -1202,14 +1345,24 @@ def save_aggregate_report_to_elasticsearch(
             number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
         )
         if use_data_streams:
-            create_data_streams([index], index_settings)
+            create_data_streams([index], index_settings, _AGGREGATE_MAPPINGS)
         else:
             if monthly_indexes:
                 index_date = begin_date.strftime("%Y-%m")
             else:
                 index_date = begin_date.strftime("%Y-%m-%d")
             index = f"{index}-{index_date}"
-            create_indexes([index], index_settings)
+            if (
+                index not in report_indexes
+                and len(report_indexes) >= MAX_INDEXES_PER_REPORT
+            ):
+                # Each new index costs a shard; send the overflow to the
+                # report's first index, keeping the record's own dates.
+                index = next(iter(report_indexes))
+                routed_records += 1
+            else:
+                report_indexes[index] = None
+                create_indexes([index], index_settings, _AGGREGATE_MAPPINGS)
         agg_doc.meta.index = index  # pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]
 
         try:
@@ -1219,6 +1372,34 @@ def save_aggregate_report_to_elasticsearch(
         except Exception as e:
             raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
 
+    if routed_records:
+        logged_report_id = sanitize_for_log(
+            str(aggregate_report["report_metadata"]["report_id"])
+        )
+        logger.warning(
+            f"{routed_records} records of report {logged_report_id} were saved to "
+            f"{next(iter(report_indexes))} instead of their own dated index: a "
+            f"report may create at most {MAX_INDEXES_PER_REPORT} indexes"
+        )
+
+
+def _flatten_address_header(addresses: Any) -> str | None:
+    """Flatten a failure sample's address-list header to its first address.
+
+    mail-parser gives ``From``, ``To`` and ``Reply-To`` as a list of
+    ``[display name, address]`` pairs, and as an empty list when the header
+    holds no address: an empty ``From:``, or ``To: undisclosed-recipients:;``
+    (an RFC 5322 §3.4 group with no members). Returns ``"Name <address>"``,
+    the bare address when the display name is empty, or ``None`` for an
+    empty list.
+    """
+    if not addresses:
+        return None
+    first = addresses[0]
+    if first[0] == "":
+        return first[1]
+    return " <".join(first) + ">"
+
 
 def save_failure_report_to_elasticsearch(
     failure_report: dict[str, Any],
@@ -1261,7 +1442,11 @@ def save_failure_report_to_elasticsearch(
     original_headers = failure_report["parsed_sample"]["headers"]
     headers: dict[str, Any] = {}
     for original_header in original_headers:
-        headers[original_header.lower()] = original_headers[original_header]
+        # A "." in a field name is an object path to the cluster, so a
+        # header named "Subject.x" would turn sample.headers.subject into an
+        # object and reject every later sample with a Subject header.
+        header_name = original_header.lower().replace(".", "_")
+        headers[header_name] = original_headers[original_header]
 
     # arrival_date_utc is a UTC wall-clock string; without assume_utc the
     # naive .timestamp() below would interpret it as local time and skew
@@ -1287,40 +1472,26 @@ def save_failure_report_to_elasticsearch(
     from_ = None
     to_ = None
     subject = None
+    # From, To and Reply-To are flattened to their first address so they
+    # can be displayed and aggregated as strings. A header with no address
+    # is left out of the document; From and To also feed the dedup query.
+    for address_header in ("from", "to", "reply-to"):
+        if address_header in headers:
+            flattened = _flatten_address_header(headers[address_header])
+            if flattened is None:
+                del headers[address_header]
+            else:
+                headers[address_header] = flattened
     if "from" in headers:
-        # We convert the FROM header from a string list to a flat string.
-        headers["from"] = headers["from"][0]
-        if headers["from"][0] == "":
-            headers["from"] = headers["from"][1]
-        else:
-            headers["from"] = " <".join(headers["from"]) + ">"
-
         from_ = dict()
         from_["sample.headers.from"] = headers["from"]
         from_query = Q(dict(match_phrase=from_))  # pyright: ignore[reportArgumentType]
         q = q & from_query
     if "to" in headers:
-        # We convert the TO header from a string list to a flat string.
-        headers["to"] = headers["to"][0]
-        if headers["to"][0] == "":
-            headers["to"] = headers["to"][1]
-        else:
-            headers["to"] = " <".join(headers["to"]) + ">"
-
         to_ = dict()
         to_["sample.headers.to"] = headers["to"]
         to_query = Q(dict(match_phrase=to_))  # pyright: ignore[reportArgumentType]
         q = q & to_query
-    if "reply-to" in headers:
-        # Flatten the Reply-To header to a string so it can be displayed
-        # and aggregated like From/To. Only the first address is used,
-        # matching the From/To handling above. Not part of the dedup
-        # query.
-        headers["reply-to"] = headers["reply-to"][0]
-        if headers["reply-to"][0] == "":
-            headers["reply-to"] = headers["reply-to"][1]
-        else:
-            headers["reply-to"] = " <".join(headers["reply-to"]) + ">"
     if "subject" in headers:
         subject = headers["subject"]
         subject_query = {"match_phrase": {"sample.headers.subject": subject}}
@@ -1401,14 +1572,14 @@ def save_failure_report_to_elasticsearch(
             number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
         )
         if use_data_streams:
-            create_data_streams([index], index_settings)
+            create_data_streams([index], index_settings, _FAILURE_MAPPINGS)
         else:
             if monthly_indexes:
                 index_date = arrival_date.strftime("%Y-%m")
             else:
                 index_date = arrival_date.strftime("%Y-%m-%d")
             index = f"{index}-{index_date}"
-            create_indexes([index], index_settings)
+            create_indexes([index], index_settings, _FAILURE_MAPPINGS)
         failure_doc.meta.index = index  # pyright: ignore[reportAttributeAccessIssue, reportOptionalMemberAccess]
         try:
             # Data streams accept only op_type=create. No _id is set, so
@@ -1599,9 +1770,9 @@ def save_smtp_tls_report_to_elasticsearch(
         smtp_tls_doc.policies.append(policy_doc)
 
     if use_data_streams:
-        create_data_streams([index], index_settings)
+        create_data_streams([index], index_settings, _SMTP_TLS_MAPPINGS)
     else:
-        create_indexes([index], index_settings)
+        create_indexes([index], index_settings, _SMTP_TLS_MAPPINGS)
     smtp_tls_doc.meta.index = index  # pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]
 
     try:

parsedmarc/opensearch.py

@@ -28,8 +28,13 @@ from opensearchpy import (
 from opensearchpy.helpers import reindex
 
 from parsedmarc import InvalidFailureReport
+from parsedmarc.constants import MAX_INDEXES_PER_REPORT
 from parsedmarc.log import logger
-from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
+from parsedmarc.utils import (
+    human_timestamp_to_datetime,
+    redact_url_credentials,
+    sanitize_for_log,
+)
 
 
 class OpenSearchError(Exception):
@@ -52,6 +57,102 @@ _READY_DATA_STREAMS: set[str] = set()
 # another template is unlikely in the first place.
 _DATA_STREAM_TEMPLATE_PRIORITY = 501
 
+# Mappings ``create_indexes`` and ``create_data_streams`` install on the
+# indexes and templates they create. Without them the cluster infers every
+# field's type from the first document that holds it, with date detection
+# on, so a crafted report could make a string field a date (report_id
+# "2026-11-09") or use up index.mapping.total_fields.limit with invented
+# sample header names, and every later report into that index would then
+# be rejected (GHSA-qpcg-29vr-jwm3). Date detection is off; the fields that
+# hold dates are declared as dates instead, the type they got before.
+# Everything else stays dynamic and object-mapped, which the shipped
+# dashboards need (see create_indexes).
+_DATE_FIELD: dict[str, Any] = {"type": "date"}
+
+# The type OpenSearch's dynamic mapping gives a JSON string, so a header
+# declared here has the same type as in indexes created before.
+_DYNAMIC_STRING_FIELD: dict[str, Any] = {
+    "type": "text",
+    "fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
+}
+
+# The failure sample headers indexed for search: every header the shipped
+# dashboards (dashboards/) reference, plus cc.
+# Header names are stored lower-cased. The other headers stay in _source
+# but are not indexed (``dynamic: False``).
+_FAILURE_SAMPLE_INDEXED_HEADERS = (
+    "authentication-results",
+    "auto-submitted",
+    "cc",
+    "content-transfer-encoding",
+    "content-type",
+    "date",
+    "from",
+    "in-reply-to",
+    "message-id",
+    "mime-version",
+    "received",
+    "references",
+    "reply-to",
+    "return-path",
+    "subject",
+    "thread-index",
+    "thread-topic",
+    "to",
+    "x-auto-response-suppress",
+    "x-exclaimer-md-config",
+    "x-linkedin-fe",
+    "x-mailer",
+    "x-ms-exchange-generated-message-source",
+    "x-ms-exchange-inbox-rules-loop",
+    "x-ms-exchange-parent-message-id",
+    "x-ms-exchange-transport-fromentityheader",
+    "x-ms-has-attach",
+    "x-ms-tnef-correlator",
+    "x-onpremexternalip",
+)
+
+_AGGREGATE_MAPPINGS: dict[str, Any] = {
+    "date_detection": False,
+    "properties": {
+        "@timestamp": _DATE_FIELD,
+        "date_range": _DATE_FIELD,
+        "date_begin": _DATE_FIELD,
+        "date_end": _DATE_FIELD,
+    },
+}
+
+_FAILURE_MAPPINGS: dict[str, Any] = {
+    "date_detection": False,
+    "properties": {
+        "@timestamp": _DATE_FIELD,
+        "arrival_date": _DATE_FIELD,
+        "sample": {
+            "properties": {
+                "date": _DATE_FIELD,
+                "headers": {
+                    "type": "object",
+                    "dynamic": False,
+                    "properties": {
+                        name: _DYNAMIC_STRING_FIELD
+                        for name in _FAILURE_SAMPLE_INDEXED_HEADERS
+                    },
+                },
+            }
+        },
+    },
+}
+
+_SMTP_TLS_MAPPINGS: dict[str, Any] = {
+    "date_detection": False,
+    "properties": {
+        "@timestamp": _DATE_FIELD,
+        "date_range": _DATE_FIELD,
+        "date_begin": _DATE_FIELD,
+        "date_end": _DATE_FIELD,
+    },
+}
+
 
 # Guard query for the dkim_results_combined/spf_results_combined backfill
 # (see ``migrate_indexes``). Matches only documents that have at least one
@@ -593,13 +694,44 @@ def set_hosts(
     return connections.create_connection(**conn_params)
 
 
-def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
+def _with_timestamp_mapping(mappings: dict[str, Any] | None) -> dict[str, Any]:
+    """Return a copy of ``mappings`` that also maps ``@timestamp`` as a date.
+
+    A data stream rejects documents unless ``@timestamp`` is a date field,
+    so the data stream template always declares it.
+
+    Args:
+        mappings (dict | None): A mappings body, or ``None``.
+
+    Returns:
+        dict: The mappings body with ``properties["@timestamp"]`` set; the
+        argument is not modified.
+    """
+    result = dict(mappings) if mappings else {}
+    result["properties"] = {
+        **result.get("properties", {}),
+        "@timestamp": {"type": "date"},
+    }
+    return result
+
+
+def create_indexes(
+    names: list[str],
+    settings: dict[str, Any] | None = None,
+    mappings: dict[str, Any] | None = None,
+):
     """
     Create OpenSearch indexes
 
+    An index that already exists is left as it is, mappings included.
+
     Args:
         names (list): A list of index names
-        settings (dict): Index settings
+        settings (dict): Index settings; one shard and no replicas when
+            ``None``
+        mappings (dict | None): The mappings body to create each index
+            with, such as ``_AGGREGATE_MAPPINGS``; ``None`` leaves every
+            field to dynamic mapping.
 
     """
     for name in names:
@@ -611,24 +743,32 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
             # fields, Vega can run nested aggregations but does not render
             # tables, and Grafana's nested bucket aggregation (9.4+) lacks
             # reverse_nested for parent-level metrics like message_count —
-            # so the dynamic `object` mapping produced by a bare create is
-            # load-bearing for the shipped dashboards. _AggregateReportDoc
-            # still declares dkim_results/spf_results with Nested(...), but
-            # that is only the DSL's in-memory shape for building documents
-            # — it is never installed as a mapping. See issue #169 and the
+            # so the dynamic `object` mapping of those fields (the mappings
+            # body declares none of them) is load-bearing for the shipped
+            # dashboards. _AggregateReportDoc still declares
+            # dkim_results/spf_results with Nested(...), but that is only
+            # the DSL's in-memory shape for building documents — it is
+            # never installed as a mapping. See issue #169 and the
             # *_combined fields on _AggregateReportDoc.
             if not index.exists():
                 logger.debug(f"Creating OpenSearch index: {name}")
+                body: dict[str, Any] = {}
                 if settings is None:
-                    index.settings(number_of_shards=1, number_of_replicas=0)
-                else:
-                    index.settings(**settings)
-                index.create()
+                    body["settings"] = {"number_of_shards": 1, "number_of_replicas": 0}
+                elif settings:
+                    body["settings"] = dict(settings)
+                if mappings:
+                    body["mappings"] = mappings
+                connections.get_connection().indices.create(index=name, body=body)
         except Exception as e:
             raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
 
 
-def create_data_streams(names: list[str], settings: dict[str, Any] | None = None):
+def create_data_streams(
+    names: list[str],
+    settings: dict[str, Any] | None = None,
+    mappings: dict[str, Any] | None = None,
+):
     """
     Create OpenSearch data streams and their index templates
 
@@ -642,15 +782,18 @@ def create_data_streams(names: list[str], settings: dict[str, Any] | None = None
     ``ism_template`` instead. Each name is set up once per process;
     ``set_hosts`` resets that.
 
-    The template maps only ``@timestamp``. Like ``create_indexes``, it
-    deliberately installs no mappings for the report fields themselves (see
-    issue #169), so they keep their dynamic ``object`` mapping.
+    The template maps ``@timestamp`` as a date, on top of ``mappings``.
+    Like ``create_indexes``, it declares no ``nested`` fields (see issue
+    #169), so the report fields keep their ``object`` mapping.
 
     Args:
         names (list): A list of data stream names
         settings (dict): Index settings for the stream's backing indices;
             one shard and no replicas when ``None``, as in
             ``create_indexes``.
+        mappings (dict | None): The mappings body for the stream's backing
+            indices, such as ``_AGGREGATE_MAPPINGS``; ``None`` maps only
+            ``@timestamp``.
 
     Raises:
         OpenSearchError: A concrete index or alias already holds one of the
@@ -664,6 +807,7 @@ def create_data_streams(names: list[str], settings: dict[str, Any] | None = None
         }
     else:
         effective_settings = dict(settings)
+    template_mappings = _with_timestamp_mapping(mappings)
 
     for name in names:
         if name in _READY_DATA_STREAMS:
@@ -695,9 +839,7 @@ def create_data_streams(names: list[str], settings: dict[str, Any] | None = None
                         "priority": _DATA_STREAM_TEMPLATE_PRIORITY,
                         "template": {
                             "settings": effective_settings,
-                            "mappings": {
-                                "properties": {"@timestamp": {"type": "date"}}
-                            },
+                            "mappings": template_mappings,
                         },
                         "_meta": {
                             "description": (
@@ -1051,6 +1193,10 @@ def save_aggregate_report_to_opensearch(
         discovery_method=aggregate_report["policy_published"].get("discovery_method"),
     )
 
+    # Distinct dated indexes this report has used, in first-use order.
+    report_indexes: dict[str, None] = {}
+    routed_records = 0
+
     for record in aggregate_report["records"]:
         begin_date = human_timestamp_to_datetime(
             record["interval_begin"], to_utc=True, assume_utc=True
@@ -1134,14 +1280,24 @@ def save_aggregate_report_to_opensearch(
             number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
         )
         if use_data_streams:
-            create_data_streams([index], index_settings)
+            create_data_streams([index], index_settings, _AGGREGATE_MAPPINGS)
         else:
             if monthly_indexes:
                 index_date = begin_date.strftime("%Y-%m")
             else:
                 index_date = begin_date.strftime("%Y-%m-%d")
             index = f"{index}-{index_date}"
-            create_indexes([index], index_settings)
+            if (
+                index not in report_indexes
+                and len(report_indexes) >= MAX_INDEXES_PER_REPORT
+            ):
+                # Each new index costs a shard; send the overflow to the
+                # report's first index, keeping the record's own dates.
+                index = next(iter(report_indexes))
+                routed_records += 1
+            else:
+                report_indexes[index] = None
+                create_indexes([index], index_settings, _AGGREGATE_MAPPINGS)
         agg_doc.meta.index = index
 
         try:
@@ -1151,6 +1307,34 @@ def save_aggregate_report_to_opensearch(
         except Exception as e:
             raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
 
+    if routed_records:
+        logged_report_id = sanitize_for_log(
+            str(aggregate_report["report_metadata"]["report_id"])
+        )
+        logger.warning(
+            f"{routed_records} records of report {logged_report_id} were saved to "
+            f"{next(iter(report_indexes))} instead of their own dated index: a "
+            f"report may create at most {MAX_INDEXES_PER_REPORT} indexes"
+        )
+
+
+def _flatten_address_header(addresses: Any) -> str | None:
+    """Flatten a failure sample's address-list header to its first address.
+
+    mail-parser gives ``From``, ``To`` and ``Reply-To`` as a list of
+    ``[display name, address]`` pairs, and as an empty list when the header
+    holds no address: an empty ``From:``, or ``To: undisclosed-recipients:;``
+    (an RFC 5322 §3.4 group with no members). Returns ``"Name <address>"``,
+    the bare address when the display name is empty, or ``None`` for an
+    empty list.
+    """
+    if not addresses:
+        return None
+    first = addresses[0]
+    if first[0] == "":
+        return first[1]
+    return " <".join(first) + ">"
+
 
 def save_failure_report_to_opensearch(
     failure_report: dict[str, Any],
@@ -1193,7 +1377,11 @@ def save_failure_report_to_opensearch(
     original_headers = failure_report["parsed_sample"]["headers"]
     headers: dict[str, Any] = {}
     for original_header in original_headers:
-        headers[original_header.lower()] = original_headers[original_header]
+        # A "." in a field name is an object path to the cluster, so a
+        # header named "Subject.x" would turn sample.headers.subject into an
+        # object and reject every later sample with a Subject header.
+        header_name = original_header.lower().replace(".", "_")
+        headers[header_name] = original_headers[original_header]
 
     # arrival_date_utc is a UTC wall-clock string; without assume_utc the
     # naive .timestamp() below would interpret it as local time and skew
@@ -1219,40 +1407,26 @@ def save_failure_report_to_opensearch(
     from_ = None
     to_ = None
     subject = None
+    # From, To and Reply-To are flattened to their first address so they
+    # can be displayed and aggregated as strings. A header with no address
+    # is left out of the document; From and To also feed the dedup query.
+    for address_header in ("from", "to", "reply-to"):
+        if address_header in headers:
+            flattened = _flatten_address_header(headers[address_header])
+            if flattened is None:
+                del headers[address_header]
+            else:
+                headers[address_header] = flattened
     if "from" in headers:
-        # We convert the FROM header from a string list to a flat string.
-        headers["from"] = headers["from"][0]
-        if headers["from"][0] == "":
-            headers["from"] = headers["from"][1]
-        else:
-            headers["from"] = " <".join(headers["from"]) + ">"
-
         from_ = dict()
         from_["sample.headers.from"] = headers["from"]
         from_query = Q(dict(match_phrase=from_))
         q = q & from_query
     if "to" in headers:
-        # We convert the TO header from a string list to a flat string.
-        headers["to"] = headers["to"][0]
-        if headers["to"][0] == "":
-            headers["to"] = headers["to"][1]
-        else:
-            headers["to"] = " <".join(headers["to"]) + ">"
-
         to_ = dict()
         to_["sample.headers.to"] = headers["to"]
         to_query = Q(dict(match_phrase=to_))
         q = q & to_query
-    if "reply-to" in headers:
-        # Flatten the Reply-To header to a string so it can be displayed
-        # and aggregated like From/To. Only the first address is used,
-        # matching the From/To handling above. Not part of the dedup
-        # query.
-        headers["reply-to"] = headers["reply-to"][0]
-        if headers["reply-to"][0] == "":
-            headers["reply-to"] = headers["reply-to"][1]
-        else:
-            headers["reply-to"] = " <".join(headers["reply-to"]) + ">"
     if "subject" in headers:
         subject = headers["subject"]
         subject_query = {"match_phrase": {"sample.headers.subject": subject}}
@@ -1331,14 +1505,14 @@ def save_failure_report_to_opensearch(
             number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
         )
         if use_data_streams:
-            create_data_streams([index], index_settings)
+            create_data_streams([index], index_settings, _FAILURE_MAPPINGS)
         else:
             if monthly_indexes:
                 index_date = arrival_date.strftime("%Y-%m")
             else:
                 index_date = arrival_date.strftime("%Y-%m-%d")
             index = f"{index}-{index_date}"
-            create_indexes([index], index_settings)
+            create_indexes([index], index_settings, _FAILURE_MAPPINGS)
         failure_doc.meta.index = index
         try:
             # Data streams accept only op_type=create. No _id is set, so
@@ -1529,9 +1703,9 @@ def save_smtp_tls_report_to_opensearch(
         smtp_tls_doc.policies.append(policy_doc)
 
     if use_data_streams:
-        create_data_streams([index], index_settings)
+        create_data_streams([index], index_settings, _SMTP_TLS_MAPPINGS)
     else:
-        create_indexes([index], index_settings)
+        create_indexes([index], index_settings, _SMTP_TLS_MAPPINGS)
     smtp_tls_doc.meta.index = index
 
     try:

parsedmarc/postgres.py

@@ -45,9 +45,16 @@ def _ensure_utc_suffix(value: str | None) -> str | None:
     Several parsers produce ``YYYY-MM-DD HH:MM:SS`` format strings that
     are known to be UTC but lack an explicit offset.  PostgreSQL
     ``TIMESTAMPTZ`` columns need the offset to avoid interpreting the
-    value in the session timezone.
+    value in the session timezone. A value that is not a string is
+    returned unchanged, for PostgreSQL to accept or reject.
     """
-    if value and "+" not in value and "-" not in value[10:] and "Z" not in value:
+    if (
+        isinstance(value, str)
+        and value
+        and "+" not in value
+        and "-" not in value[10:]
+        and "Z" not in value
+    ):
         return value + "+00"
     return value
 

parsedmarc/splunk.py

@@ -215,16 +215,18 @@ class HECClient(object):
 
         data = self._common_data.copy()
         json_lines: list[str] = []
-        for report in reports:
-            data["sourcetype"] = "smtp:tls"
-            timestamp = human_timestamp_to_unix_timestamp(report["begin_date"])
-            data["time"] = timestamp
-            data["event"] = report.copy()
-            json_lines.append(f"{json.dumps(data)}\n")
-
         if not self.verify:
             logger.debug("Skipping certificate verification for Splunk HEC")
         try:
+            # Inside the try so a report date that does not convert raises
+            # SplunkError like any other failed save.
+            for report in reports:
+                data["sourcetype"] = "smtp:tls"
+                timestamp = human_timestamp_to_unix_timestamp(report["begin_date"])
+                data["time"] = timestamp
+                data["event"] = report.copy()
+                json_lines.append(f"{json.dumps(data)}\n")
+
             response = self.session.post(
                 self.url, content="".join(json_lines), timeout=self.timeout
             )

parsedmarc/utils.py

@@ -21,6 +21,7 @@ from urllib.parse import SplitResult, urlsplit
 
 import mailparser
 from expiringdict import ExpiringDict
+from mailsuite.utils import check_message_limits
 
 from importlib.resources import files
 
@@ -1762,12 +1763,17 @@ def parse_email(data: bytes | str, *, strip_attachment_payloads: bool = False) -
 
     Returns:
         dict: Parsed email data
+
+    Raises:
+        mailsuite.utils.MessageLimitExceeded: The message's structure exceeds
+            the limits of ``mailsuite.utils.check_message_limits``
     """
 
     if isinstance(data, bytes):
         if is_outlook_msg(data):
             data = convert_outlook_msg(data)
         data = data.decode("utf-8", errors="replace")
+    check_message_limits(data)
     parsed_email = mailparser.parse_from_string(data)
     headers = json.loads(parsed_email.headers_json).copy()
     parsed_email = json.loads(parsed_email.mail_json).copy()

pyproject.toml

@@ -59,7 +59,7 @@ dependencies = [
     # No extras: the base mailsuite supplies IMAP and Maildir connections
     # plus mail-parser. Gmail and Microsoft Graph come from the gmail and
     # msgraph extras below.
-    "mailsuite>=2.3.4",
+    "mailsuite>=2.3.5",
     "maxminddb>=2.0.0",
     "publicsuffixlist>=0.10.0",
     # Imported directly at utils.py:39 (dateutil.parser). It used to
@@ -96,13 +96,13 @@ loganalytics = [
     "azure-monitor-ingestion>=1.0.0",
 ]
 msgraph = [
-    "mailsuite[msgraph]>=2.3.4",
+    "mailsuite[msgraph]>=2.3.5",
     # Imported directly in cli.py for Graph error handling; otherwise
     # only a transitive dep of mailsuite[msgraph] -> msgraph-sdk.
     "microsoft-kiota-abstractions>=1.8.0",
 ]
 gmail = [
-    "mailsuite[gmail]>=2.3.4",
+    "mailsuite[gmail]>=2.3.5",
 ]
 # `all` must stay the union of every extra above — and must keep
 # excluding `postgresql`: psycopg's prebuilt binary wheels do not exist
@@ -118,7 +118,7 @@ all = [
     "boto3>=1.16.63",
     "elasticsearch>=8.18,<9",
     "kafka-python>=2.3.2",
-    "mailsuite[gmail,msgraph]>=2.3.4",
+    "mailsuite[gmail,msgraph]>=2.3.5",
     "microsoft-kiota-abstractions>=1.8.0",
     "opensearch-py>=2.4.2,<=4.0.0",
     "pygelf>=0.4.2",

tests/test_cli.py

@@ -21,7 +21,7 @@ import unittest
 import zipfile
 from collections.abc import Iterator, Sequence
 from configparser import ConfigParser
-from contextlib import contextmanager
+from contextlib import ExitStack, contextmanager
 from email.mime.application import MIMEApplication
 from email.mime.multipart import MIMEMultipart
 from tempfile import NamedTemporaryFile
@@ -10536,5 +10536,261 @@ tenant_id = fosdem
         self.assertIsInstance(context.exception.__cause__, ValueError)
 
 
+class TestOutputDestinationExceptionsMarkTheBatchUnsaved(unittest.TestCase):
+    """GHSA-4272-4r8g-6c5h: the failure and SMTP TLS Elasticsearch and
+    OpenSearch blocks of process_reports(), every Splunk HEC and PostgreSQL
+    block, and the file output caught only their own client's error. Any
+    other exception a report made a saver raise (an IndexError on an empty
+    From: list, a TypeError in the CSV writer) escaped through the save
+    callback, and _main logged "Mailbox Error" and exited 1 on every run,
+    leaving the message in place. Each block now ends with the catch-all the
+    aggregate Elasticsearch block has: the error is recorded through
+    log_output_error() and the batch is reported unsaved."""
+
+    _AGGREGATE = {"policy_published": {"domain": "example.com"}}
+    _FAILURE = {"reported_domain": "example.com"}
+    _SMTP_TLS = {"policies": [{"policy_domain": "example.com"}]}
+
+    def setUp(self):
+        parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear()
+        self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear)
+
+    def _run(self, sections, batch, *patches, general="", silent=True):
+        """Run _main() on one mailbox batch.
+
+        Returns the save callback's verdicts, the ERROR log lines, the exit
+        code (``None`` when _main returned) and the mocks ``patches`` made.
+        """
+        with NamedTemporaryFile("w", suffix=".ini", delete=False) as config_file:
+            config_file.write(
+                "[general]\n"
+                "save_aggregate = true\n"
+                "save_failure = true\n"
+                "save_smtp_tls = true\n"
+                f"silent = {str(silent).lower()}\n"
+                f"{general}\n"
+                "[imap]\n"
+                "host = imap.example.com\n"
+                "user = test-user\n"
+                "password = test-password\n\n"
+                f"{sections}"
+            )
+            config_path = config_file.name
+        self.addCleanup(lambda: os.path.exists(config_path) and os.remove(config_path))
+        full_batch = {
+            "aggregate_reports": [],
+            "failure_reports": [],
+            "smtp_tls_reports": [],
+        }
+        full_batch.update(batch)
+        verdicts = []
+
+        def _fetch_and_save(**kwargs):
+            verdicts.append(kwargs["save_callback"](full_batch))
+            return full_batch
+
+        with ExitStack() as stack:
+            stack.enter_context(
+                patch(
+                    "parsedmarc.cli.get_dmarc_reports_from_mailbox",
+                    side_effect=_fetch_and_save,
+                )
+            )
+            stack.enter_context(
+                patch("parsedmarc.cli.IMAPConnection", return_value=object())
+            )
+            mocks = [stack.enter_context(patcher) for patcher in patches]
+            stack.enter_context(
+                patch.object(sys, "argv", ["parsedmarc", "-c", config_path])
+            )
+            cm = stack.enter_context(self.assertLogs("parsedmarc.log", level="ERROR"))
+            exit_code = None
+            try:
+                parsedmarc.cli._main()
+            except SystemExit as system_exit:
+                exit_code = system_exit.code
+        return verdicts, cm.output, exit_code, mocks
+
+    def _assert_unsaved(self, result, destination, message):
+        verdicts, output, exit_code, _mocks = result
+        self.assertIsNone(exit_code)
+        self.assertEqual(verdicts, [False])
+        expected = f"{destination} exception Error: {message}"
+        self.assertTrue(any(expected in line for line in output), output)
+        self.assertFalse(any("Mailbox Error" in line for line in output), output)
+
+    def _search_backend_patches(self, module, saver, error):
+        return (
+            patch(f"parsedmarc.cli.{module}.set_hosts"),
+            patch(f"parsedmarc.cli.{module}.migrate_indexes"),
+            patch(f"parsedmarc.cli.{module}.{saver}", side_effect=error),
+        )
+
+    def testStandardOutputEncodingErrorIsRecorded(self):
+        """A report string the console cannot encode (a lone surrogate) made
+        print() in process_reports() raise UnicodeEncodeError outside every
+        try."""
+        report = {
+            "organization_name": "Comp\ud800any",
+            "begin_date": "2024-01-01T00:00:00Z",
+            "end_date": "2024-01-01T23:59:59Z",
+            "contact_info": "tls@attacker.example",
+            "report_id": "poc",
+            "policies": [],
+        }
+        stdout = io.TextIOWrapper(io.BytesIO(), encoding="utf-8")
+        with patch.object(sys, "stdout", stdout):
+            result = self._run("", {"smtp_tls_reports": [report]}, silent=False)
+        self._assert_unsaved(
+            result, "Standard output", "'utf-8' codec can't encode character"
+        )
+
+    def testStandardOutputClosedPipeIsRecorded(self):
+        """A closed pipe on standard output (BrokenPipeError) is recorded the
+        same way as an encoding error."""
+
+        class _ClosedPipe(io.StringIO):
+            def write(self, text):
+                raise BrokenPipeError(32, "Broken pipe")
+
+        report = {
+            "organization_name": "Reporter Inc.",
+            "begin_date": "2024-01-01T00:00:00Z",
+            "end_date": "2024-01-01T23:59:59Z",
+            "contact_info": "tls@reporter.example",
+            "report_id": "pipe",
+            "policies": [],
+        }
+        with patch.object(sys, "stdout", _ClosedPipe()):
+            result = self._run("", {"smtp_tls_reports": [report]}, silent=False)
+        self._assert_unsaved(result, "Standard output", "[Errno 32] Broken pipe")
+
+    def testFileOutputExceptionIsRecorded(self):
+        """A real TypeError from the SMTP TLS CSV writer (a policy_strings
+        list holding an integer), not a mock."""
+        output_dir = tempfile.mkdtemp()
+        self.addCleanup(lambda: shutil.rmtree(output_dir, ignore_errors=True))
+        report = {
+            "organization_name": "Attacker Inc.",
+            "begin_date": "2024-01-01T00:00:00Z",
+            "end_date": "2024-01-01T23:59:59Z",
+            "contact_info": "tls@attacker.example",
+            "report_id": "poc",
+            "policies": [
+                {
+                    "policy_domain": "example.com",
+                    "policy_type": "sts",
+                    "policy_strings": [1],
+                    "successful_session_count": 1,
+                    "failed_session_count": 0,
+                }
+            ],
+        }
+        result = self._run(
+            "", {"smtp_tls_reports": [report]}, general=f"output = {output_dir}"
+        )
+        self._assert_unsaved(
+            result,
+            "File output",
+            "sequence item 0: expected str instance, int found",
+        )
+
+    def testSearchBackendFailureAndSmtpTlsExceptionsAreRecorded(self):
+        cases = (
+            ("elastic", "Elasticsearch", "elasticsearch"),
+            ("opensearch", "OpenSearch", "opensearch"),
+        )
+        report_types = (
+            ("failure", "failure_reports", self._FAILURE),
+            ("smtp_tls", "smtp_tls_reports", self._SMTP_TLS),
+        )
+        for module, destination, section in cases:
+            for report_type, batch_key, report in report_types:
+                saver = f"save_{report_type}_report_to_{section}"
+                with self.subTest(saver=saver):
+                    result = self._run(
+                        f"[{section}]\nhosts = localhost\n",
+                        {batch_key: [report]},
+                        *self._search_backend_patches(
+                            module, saver, IndexError("list index out of range")
+                        ),
+                    )
+                    self.assertIs(result[3][2].call_args.args[0], report)
+                    self._assert_unsaved(result, destination, "list index out of range")
+
+    def testPostgreSQLExceptionsAreRecorded(self):
+        report_types = (
+            ("aggregate", "aggregate_reports", self._AGGREGATE),
+            ("failure", "failure_reports", self._FAILURE),
+            ("smtp_tls", "smtp_tls_reports", self._SMTP_TLS),
+        )
+        for report_type, batch_key, report in report_types:
+            saver = f"save_{report_type}_report_to_postgresql"
+            message = "argument of type 'int' is not a container or iterable"
+            client_cls = MagicMock()
+            getattr(client_cls.return_value, saver).side_effect = TypeError(message)
+            with self.subTest(saver=saver):
+                result = self._run(
+                    "[postgresql]\nhost = db.example.com\n",
+                    {batch_key: [report]},
+                    patch("parsedmarc.cli.postgres.PostgreSQLClient", client_cls),
+                )
+                client = client_cls.return_value
+                getattr(client, saver).assert_called_once_with(report)
+                self._assert_unsaved(result, "PostgreSQL", message)
+
+    def testSplunkHecExceptionsAreRecorded(self):
+        report_types = (
+            ("aggregate", "aggregate_reports", self._AGGREGATE),
+            ("failure", "failure_reports", self._FAILURE),
+            ("smtp_tls", "smtp_tls_reports", self._SMTP_TLS),
+        )
+        for report_type, batch_key, report in report_types:
+            saver = f"save_{report_type}_reports_to_splunk"
+            message = "'int' object has no attribute 'replace'"
+            client_cls = MagicMock()
+            getattr(client_cls.return_value, saver).side_effect = AttributeError(
+                message
+            )
+            with self.subTest(saver=saver):
+                result = self._run(
+                    "[splunk_hec]\n"
+                    "url = https://splunk.example.com:8088\n"
+                    "token = test-token\n"
+                    "index = dmarc\n",
+                    {batch_key: [report]},
+                    patch("parsedmarc.cli.splunk.HECClient", client_cls),
+                )
+                client = client_cls.return_value
+                getattr(client, saver).assert_called_once_with([report])
+                self._assert_unsaved(result, "Splunk HEC", message)
+
+    def testFailOnOutputErrorStillExitsThroughParserError(self):
+        """With fail_on_output_error the recorded exception is raised as
+        ParserError, as for any other output error, so _main exits 1 with
+        the output-error message rather than as a mailbox error."""
+        verdicts, output, exit_code, _mocks = self._run(
+            "[elasticsearch]\nhosts = localhost\n",
+            {"failure_reports": [self._FAILURE]},
+            *self._search_backend_patches(
+                "elastic",
+                "save_failure_report_to_elasticsearch",
+                IndexError("list index out of range"),
+            ),
+            general="fail_on_output_error = true",
+        )
+        self.assertEqual(exit_code, 1)
+        self.assertEqual(verdicts, [])
+        self.assertTrue(
+            any(
+                "Output destination failures detected" in line
+                and "Elasticsearch exception Error: list index out of range" in line
+                for line in output
+            ),
+            output,
+        )
+        self.assertFalse(any("Mailbox Error" in line for line in output), output)
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)

tests/test_elastic.py

@@ -8,18 +8,27 @@ construction, index naming, deduplication queries, error wrapping —
 without needing a running Elasticsearch cluster.
 """
 
+import copy
+import json
+import re
 import time
 import unittest
 from collections.abc import Iterator
 from contextlib import contextmanager
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
 from typing import cast
 from unittest.mock import MagicMock, call, patch
 
 from elasticsearch import NotFoundError
 
 import parsedmarc.elastic as elastic_module
-from parsedmarc import InvalidFailureReport, parse_report_file
+from parsedmarc import (
+    InvalidFailureReport,
+    parse_aggregate_report_xml,
+    parse_report_email,
+    parse_report_file,
+)
 from parsedmarc.elastic import (
     AlreadySaved,
     ElasticsearchError,
@@ -320,32 +329,40 @@ class TestSetHosts(unittest.TestCase):
 
 class TestCreateIndexes(unittest.TestCase):
     def test_creates_missing_index_with_default_settings(self):
-        with patch("parsedmarc.elastic.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
+        with (
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = False
             create_indexes(["dmarc_aggregate-2024-01-15"])
-        mock_index.settings.assert_called_once_with(
-            number_of_shards=1, number_of_replicas=0
+        mock_get_conn.return_value.indices.create.assert_called_once_with(
+            index="dmarc_aggregate-2024-01-15",
+            body={"settings": {"number_of_shards": 1, "number_of_replicas": 0}},
         )
-        mock_index.create.assert_called_once()
 
     def test_creates_with_custom_settings(self):
-        with patch("parsedmarc.elastic.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
+        with (
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = False
             create_indexes(
                 ["idx"], settings={"number_of_shards": 3, "refresh_interval": "5s"}
             )
-        mock_index.settings.assert_called_once_with(
-            number_of_shards=3, refresh_interval="5s"
+        body = mock_get_conn.return_value.indices.create.call_args.kwargs["body"]
+        self.assertEqual(
+            body, {"settings": {"number_of_shards": 3, "refresh_interval": "5s"}}
         )
 
     def test_skips_existing_index(self):
-        with patch("parsedmarc.elastic.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = True
-            create_indexes(["idx"])
-        mock_index.create.assert_not_called()
+        with (
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = True
+            create_indexes(["idx"], mappings={"date_detection": False})
+        mock_get_conn.return_value.indices.create.assert_not_called()
+        mock_get_conn.return_value.indices.put_mapping.assert_not_called()
 
     def test_wraps_sdk_error(self):
         with patch("parsedmarc.elastic.Index") as mock_index_cls:
@@ -372,37 +389,36 @@ class TestCreateIndexesServerless(unittest.TestCase):
     def tearDown(self):
         elastic_module._SERVERLESS = self._original
 
+    def _created_body(self, **kwargs):
+        with (
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            create_indexes(["idx"], **kwargs)
+        create = mock_get_conn.return_value.indices.create
+        create.assert_called_once()
+        return create.call_args.kwargs["body"]
+
     def test_serverless_default_skips_settings_entirely(self):
-        with patch("parsedmarc.elastic.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
-            create_indexes(["idx"])
-        mock_index.settings.assert_not_called()
-        mock_index.create.assert_called_once()
+        self.assertEqual(self._created_body(), {})
 
     def test_serverless_filters_rejected_keys_and_passes_others_through(self):
-        with patch("parsedmarc.elastic.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
-            create_indexes(
-                ["idx"],
-                settings={
-                    "number_of_shards": 3,
-                    "number_of_replicas": 2,
-                    "refresh_interval": "5s",
-                },
-            )
-        mock_index.settings.assert_called_once_with(refresh_interval="5s")
+        body = self._created_body(
+            settings={
+                "number_of_shards": 3,
+                "number_of_replicas": 2,
+                "refresh_interval": "5s",
+            },
+        )
+        self.assertEqual(body, {"settings": {"refresh_interval": "5s"}})
 
     def test_serverless_skips_settings_when_only_rejected_keys(self):
-        with patch("parsedmarc.elastic.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
-            create_indexes(
-                ["idx"], settings={"number_of_shards": 3, "number_of_replicas": 2}
-            )
-        mock_index.settings.assert_not_called()
-        mock_index.create.assert_called_once()
+        body = self._created_body(
+            settings={"number_of_shards": 3, "number_of_replicas": 2},
+            mappings={"date_detection": False},
+        )
+        self.assertEqual(body, {"mappings": {"date_detection": False}})
 
 
 # ---------------------------------------------------------------------------
@@ -1629,7 +1645,7 @@ class TestCreateDataStreams(_DataStreamTestCase):
             name="dmarc_aggregate"
         )
         # No concrete index is created for a stream.
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         self.assertIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
 
     def test_default_settings_are_one_shard_no_replicas(self):
@@ -1803,7 +1819,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
         client.indices.create_data_stream.assert_called_once_with(
             name="cust_dmarc_aggregate_tenant_a"
         )
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         # The dedup search still uses the wildcard, which matches the
         # stream as well as dated indexes written before the switch.
         self.assertEqual(
@@ -1824,7 +1840,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
             index_call["body"]["@timestamp"],
             datetime(2024, 1, 15, tzinfo=timezone.utc),
         )
-        mock_index_cls.return_value.create.assert_called_once()
+        client.indices.create.assert_called_once()
         client.indices.put_index_template.assert_not_called()
         client.indices.create_data_stream.assert_not_called()
 
@@ -1843,7 +1859,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
             datetime(2024, 1, 1, tzinfo=timezone.utc),
         )
         client.indices.create_data_stream.assert_called_once_with(name="dmarc_failure")
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         self.assertEqual(
             mock_search_cls.call_args.kwargs["index"],
             "dmarc_failure*,dmarc_forensic*",
@@ -1877,7 +1893,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
             datetime(2024, 2, 3, tzinfo=timezone.utc),
         )
         client.indices.create_data_stream.assert_called_once_with(name="cust_smtp_tls")
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         self.assertEqual(mock_search_cls.call_args.kwargs["index"], "cust_smtp_tls*")
 
     def test_smtp_tls_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
@@ -1964,9 +1980,558 @@ class TestBackwardCompatAlias(unittest.TestCase):
         )
 
 
+class TestMappingGuards(_DataStreamTestCase):
+    """New indexes and data stream templates carry explicit mappings
+    (GHSA-qpcg-29vr-jwm3).
+
+    Before 11.2.2 they were created with settings only, so the cluster
+    inferred every field's type from the first document, with
+    ``date_detection`` on: an aggregate report with ``report_id``
+    ``2026-11-09`` mapped ``report_id`` as a date for its day, a failure
+    sample header ``Subject.x`` mapped ``sample.headers.subject`` as an
+    object, and invented header names used up the 1,000-field limit, after
+    which legitimate reports for that index were rejected.
+    """
+
+    def _save_dated(self, save_function, report):
+        with (
+            _registered_client() as client,
+            patch("parsedmarc.elastic.Search") as mock_search_cls,
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+        ):
+            mock_search_cls.return_value.execute.return_value = []
+            mock_index_cls.return_value.exists.return_value = False
+            save_function(report)
+        client.indices.create.assert_called_once()
+        return client
+
+    def _created_mappings(self, save_function, report):
+        client = self._save_dated(save_function, report)
+        return client.indices.create.call_args.kwargs["body"]["mappings"]
+
+    def _template_mappings(self, save_function, report):
+        with (
+            _registered_client() as client,
+            patch("parsedmarc.elastic.Search") as mock_search_cls,
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+        ):
+            mock_search_cls.return_value.execute.return_value = []
+            client.indices.get_data_stream.side_effect = _not_found()
+            mock_index_cls.return_value.exists.return_value = False
+            save_function(report, use_data_streams=True)
+        template = client.indices.put_index_template.call_args.kwargs
+        return template["template"]["mappings"]
+
+    def _assert_dates_declared(self, mappings, names):
+        for name in names:
+            self.assertEqual(mappings["properties"][name], {"type": "date"}, name)
+
+    def test_aggregate_index_turns_date_detection_off(self):
+        mappings = self._created_mappings(
+            save_aggregate_report_to_elasticsearch, _aggregate_report()
+        )
+        self.assertIs(mappings["date_detection"], False)
+        self._assert_dates_declared(
+            mappings, ["@timestamp", "date_range", "date_begin", "date_end"]
+        )
+        self.assertNotIn("sample", mappings["properties"])
+        # report_id stays dynamic: with date detection off a date-shaped
+        # value maps it as a string, not a date.
+        self.assertNotIn("report_id", mappings["properties"])
+
+    def test_smtp_tls_index_turns_date_detection_off(self):
+        mappings = self._created_mappings(
+            save_smtp_tls_report_to_elasticsearch, _smtp_tls_report()
+        )
+        self.assertIs(mappings["date_detection"], False)
+        self._assert_dates_declared(
+            mappings, ["@timestamp", "date_range", "date_begin", "date_end"]
+        )
+        self.assertNotIn("sample", mappings["properties"])
+
+    def test_failure_index_maps_sample_headers_without_dynamic_fields(self):
+        mappings = self._created_mappings(
+            save_failure_report_to_elasticsearch, _failure_report()
+        )
+        self.assertIs(mappings["date_detection"], False)
+        self._assert_dates_declared(mappings, ["@timestamp", "arrival_date"])
+        sample = mappings["properties"]["sample"]["properties"]
+        self.assertEqual(sample["date"], {"type": "date"})
+        headers = sample["headers"]
+        self.assertEqual(headers["type"], "object")
+        self.assertIs(headers["dynamic"], False)
+        string_field = {
+            "type": "text",
+            "fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
+        }
+        for name in ("from", "to", "subject", "date", "message-id", "cc"):
+            self.assertEqual(headers["properties"][name], string_field, name)
+        for name in headers["properties"]:
+            # Stored names are lower-cased with "." replaced, so a declared
+            # name must be in that form or no stored header can match it.
+            self.assertEqual(name, name.lower().replace(".", "_"), name)
+
+    def test_failure_mapping_declares_every_header_the_dashboards_query(self):
+        dashboards = Path(__file__).resolve().parent.parent / "dashboards"
+        referenced = set()
+        for path in dashboards.rglob("*"):
+            if path.is_file() and path.suffix in (".json", ".ndjson", ".xml"):
+                text = path.read_text(encoding="utf-8")
+                for match in re.finditer(r"sample\.headers\.([A-Za-z0-9_-]+)", text):
+                    referenced.add(match.group(1).lower())
+        self.assertIn("x-mailer", referenced)
+        mappings = self._created_mappings(
+            save_failure_report_to_elasticsearch, _failure_report()
+        )
+        declared = mappings["properties"]["sample"]["properties"]["headers"][
+            "properties"
+        ]
+        self.assertLessEqual(referenced, set(declared))
+
+    def test_data_stream_templates_carry_the_mappings(self):
+        for save_function, report, has_sample in (
+            (save_aggregate_report_to_elasticsearch, _aggregate_report(), False),
+            (save_failure_report_to_elasticsearch, _failure_report(), True),
+            (save_smtp_tls_report_to_elasticsearch, _smtp_tls_report(), False),
+        ):
+            case = save_function.__name__
+            elastic_module._READY_DATA_STREAMS.clear()
+            mappings = self._template_mappings(save_function, report)
+            self.assertIs(mappings["date_detection"], False, case)
+            self.assertEqual(
+                mappings["properties"]["@timestamp"], {"type": "date"}, case
+            )
+            self.assertEqual("sample" in mappings["properties"], has_sample, case)
+            if has_sample:
+                headers = mappings["properties"]["sample"]["properties"]["headers"]
+                self.assertIs(headers["dynamic"], False, case)
+                self.assertIn("subject", headers["properties"], case)
+
+    def test_data_stream_template_adds_timestamp_without_changing_the_argument(
+        self,
+    ):
+        mappings = {"date_detection": False, "properties": {"x": {"type": "long"}}}
+        with (
+            _registered_client() as client,
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+        ):
+            client.indices.get_data_stream.side_effect = _not_found()
+            mock_index_cls.return_value.exists.return_value = False
+            create_data_streams(["s"], mappings=mappings)
+        template = client.indices.put_index_template.call_args.kwargs
+        self.assertEqual(
+            template["template"]["mappings"],
+            {
+                "date_detection": False,
+                "properties": {"x": {"type": "long"}, "@timestamp": {"type": "date"}},
+            },
+        )
+        self.assertEqual(
+            mappings, {"date_detection": False, "properties": {"x": {"type": "long"}}}
+        )
+
+    def test_dotted_header_name_is_stored_with_an_underscore(self):
+        report = _failure_report()
+        report["parsed_sample"] = dict(report["parsed_sample"])
+        report["parsed_sample"]["headers"] = {
+            "From": [["", "sender@example.com"]],
+            "Subject.x": "v",
+            "X.Y.Z": "w",
+        }
+        client = self._save_dated(save_failure_report_to_elasticsearch, report)
+        headers = client.index.call_args.kwargs["body"]["sample"]["headers"]
+        self.assertEqual(headers["subject_x"], "v")
+        self.assertEqual(headers["x_y_z"], "w")
+        self.assertNotIn("subject.x", headers)
+        self.assertNotIn("subject", headers)
+        # The address headers are still flattened under their usual keys.
+        self.assertEqual(headers["from"], "sender@example.com")
+
+
 # Silence unused-import lint in the test module preamble.
 _ = call
 
 
+class TestAggregateIndexCap(unittest.TestCase):
+    """One report may create at most MAX_INDEXES_PER_REPORT dated indexes;
+    the overflow records are saved to the report's first index."""
+
+    DAYS = 40
+
+    def _report(self):
+        report = _aggregate_report()
+        template = report["records"][0]
+        report["records"] = []
+        for day in range(self.DAYS):
+            record = copy.deepcopy(template)
+            begin = datetime(2024, 1, 1, tzinfo=timezone.utc) + timedelta(days=day)
+            end = begin + timedelta(days=1)
+            record["interval_begin"] = begin.strftime("%Y-%m-%d %H:%M:%S")
+            record["interval_end"] = end.strftime("%Y-%m-%d %H:%M:%S")
+            report["records"].append(record)
+        return report
+
+    def _save(self, **kwargs):
+        """Save the report; return created index names, the (index,
+        date_begin) of each saved document and the logger mock."""
+        saved = []
+
+        def capture(doc, *args, **save_kwargs):
+            saved.append((doc.meta.index, doc.date_begin))
+
+        with (
+            patch("parsedmarc.elastic.Search", return_value=_empty_search()),
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection") as mock_get_conn,
+            patch("parsedmarc.elastic.logger") as mock_logger,
+            patch("parsedmarc.elastic.create_data_streams") as mock_streams,
+            patch.object(
+                elastic_module._AggregateReportDoc,
+                "save",
+                autospec=True,
+                side_effect=capture,
+            ),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_elasticsearch(self._report(), **kwargs)
+            created = [
+                c.kwargs["index"]
+                for c in mock_get_conn.return_value.indices.create.call_args_list
+            ]
+        return created, saved, mock_logger, mock_streams
+
+    def test_daily_indexes_are_capped_and_overflow_goes_to_the_first(self):
+        created, saved, mock_logger, _ = self._save()
+        expected = [f"dmarc_aggregate-2024-01-{d:02d}" for d in range(1, 32)]
+        self.assertEqual(sorted(set(created)), expected)
+        self.assertEqual(len(saved), self.DAYS)
+        self.assertEqual([index for index, _ in saved[:31]], expected)
+        self.assertEqual(
+            [index for index, _ in saved[31:]], ["dmarc_aggregate-2024-01-01"] * 9
+        )
+        # The overflow records keep their own dates.
+        begins = [begin for _, begin in saved]
+        self.assertEqual(begins[31], datetime(2024, 2, 1, tzinfo=timezone.utc))
+        self.assertEqual(begins[39], datetime(2024, 2, 9, tzinfo=timezone.utc))
+        mock_logger.warning.assert_called_once()
+        self.assertIn("9 records", mock_logger.warning.call_args.args[0])
+
+    def test_routing_warning_escapes_the_report_id(self):
+        """The report ID is sender-controlled; a CR/LF in it must not forge
+        a log line."""
+        report = self._report()
+        report["report_metadata"]["report_id"] = "poc\r\nforged"
+        with (
+            patch("parsedmarc.elastic.Search", return_value=_empty_search()),
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection"),
+            patch("parsedmarc.elastic.logger") as mock_logger,
+            patch.object(elastic_module._AggregateReportDoc, "save", autospec=True),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_elasticsearch(report)
+        message = mock_logger.warning.call_args.args[0]
+        self.assertIn("poc\\r\\nforged", message)
+        self.assertNotIn("\n", message)
+
+    def test_monthly_indexes_are_not_affected(self):
+        created, saved, mock_logger, _ = self._save(monthly_indexes=True)
+        self.assertEqual(
+            sorted(set(created)), ["dmarc_aggregate-2024-01", "dmarc_aggregate-2024-02"]
+        )
+        self.assertEqual(
+            [index for index, _ in saved],
+            ["dmarc_aggregate-2024-01"] * 31 + ["dmarc_aggregate-2024-02"] * 9,
+        )
+        mock_logger.warning.assert_not_called()
+
+    def test_data_streams_are_not_affected(self):
+        created, saved, mock_logger, mock_streams = self._save(use_data_streams=True)
+        self.assertEqual(created, [])
+        self.assertEqual({index for index, _ in saved}, {"dmarc_aggregate"})
+        self.assertEqual(len(saved), self.DAYS)
+        self.assertEqual(mock_streams.call_count, self.DAYS)
+        mock_logger.warning.assert_not_called()
+
+
+class TestSaveFailureReportEmptyAddressHeaders(unittest.TestCase):
+    """GHSA-4272-4r8g-6c5h: a failure sample whose From, To or Reply-To
+    header holds no address made the save index ``headers[...][0]`` of
+    mail-parser's empty list and raise ``IndexError``, which escaped the
+    CLI and stopped the mailbox run. The header is now left out of the
+    document and of the duplicate query."""
+
+    # An empty From: and Reply-To:, and To: undisclosed-recipients:; (an
+    # RFC 5322 §3.4 group with no members, which legitimate mail uses).
+    ARF = (
+        "From: dmarc@reporter.example\n"
+        "To: ruf@example.com\n"
+        "Date: Mon, 01 Jan 2024 00:00:00 +0000\n"
+        "Subject: DMARC Failure Report\n"
+        "MIME-Version: 1.0\n"
+        'Content-Type: multipart/report; report-type=feedback-report; boundary="b"\n'
+        "\n"
+        "--b\n"
+        "Content-Type: text/plain\n"
+        "\n"
+        "Authentication failure report.\n"
+        "--b\n"
+        "Content-Type: message/feedback-report\n"
+        "\n"
+        "Feedback-Type: auth-failure\n"
+        "User-Agent: test/1.0\n"
+        "Version: 1\n"
+        "Original-Mail-From: user@example.com\n"
+        "Arrival-Date: Mon, 01 Jan 2024 00:00:00 +0000\n"
+        "Source-IP: 192.0.2.1\n"
+        "Reported-Domain: example.com\n"
+        "Authentication-Results: mx; dmarc=fail\n"
+        "\n"
+        "--b\n"
+        "Content-Type: text/rfc822-headers\n"
+        "\n"
+        "From:\n"
+        "To: undisclosed-recipients:;\n"
+        "Reply-To:\n"
+        "Subject: spoofed\n"
+        "Date: Mon, 01 Jan 2024 00:00:00 +0000\n"
+        "--b--\n"
+    )
+
+    def test_empty_address_headers_are_left_out(self):
+        parsed = parse_report_email(self.ARF, offline=True)
+        self.assertEqual(parsed["report_type"], "failure")
+        report = cast(dict, parsed["report"])
+        headers = report["parsed_sample"]["headers"]
+        self.assertEqual(
+            (headers["From"], headers["To"], headers["Reply-To"]), ([], [], [])
+        )
+        search = _empty_search()
+        with (
+            patch("parsedmarc.elastic.Search", return_value=search),
+            patch("parsedmarc.elastic.Index"),
+            patch.object(
+                elastic_module._FailureReportDoc, "save", autospec=True
+            ) as mock_save,
+        ):
+            save_failure_report_to_elasticsearch(report)
+        doc = mock_save.call_args.args[0]
+        self.assertEqual(doc.sample.headers["subject"], "spoofed")
+        for header in ("from", "to", "reply-to"):
+            self.assertNotIn(header, doc.sample.headers)
+        query = json.dumps(search.query.to_dict())
+        self.assertIn("sample.headers.subject", query)
+        self.assertNotIn("sample.headers.from", query)
+        self.assertNotIn("sample.headers.to", query)
+
+
+class TestAggregateAttributesDoNotReachTheDocument(unittest.TestCase):
+    """GHSA-qpcg-29vr-jwm3: an XML attribute on a scalar element reached the
+    document as an object and mapped the field as one."""
+
+    def test_document_holds_no_object_for_attributed_fields(self):
+        attribute = ' a="1"'
+        xml = (
+            f"<feedback><version{attribute}>1.0</version><report_metadata>"
+            "<org_name>example.net</org_name><email>a@example.net</email>"
+            "<report_id>r</report_id><date_range><begin>1704067200</begin>"
+            "<end>1704153599</end></date_range></report_metadata>"
+            "<policy_published><domain>example.com</domain>"
+            f"<p{attribute}>none</p><fo{attribute}>1</fo></policy_published>"
+            "<record><row><source_ip>192.0.2.1</source_ip><count>1</count>"
+            f"<policy_evaluated><disposition{attribute}>none</disposition>"
+            "<dkim>pass</dkim><spf>pass</spf></policy_evaluated></row>"
+            "<identifiers><header_from>example.com</header_from>"
+            f"<envelope_to{attribute}>example.org</envelope_to></identifiers>"
+            "<auth_results><spf><domain>example.com</domain><result>pass</result>"
+            "</spf></auth_results></record></feedback>"
+        )
+        report = parse_aggregate_report_xml(xml, offline=True)
+        docs = []
+        with (
+            patch("parsedmarc.elastic.Search", return_value=_empty_search()),
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection"),
+            patch.object(
+                elastic_module._AggregateReportDoc,
+                "save",
+                autospec=True,
+                side_effect=lambda doc, *args, **kwargs: docs.append(doc.to_dict()),
+            ),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_elasticsearch(cast(dict, report))
+        doc = docs[0]
+        self.assertEqual(doc["xml_schema"], "1.0")
+        self.assertEqual(doc["disposition"], "none")
+        self.assertEqual(doc["envelope_to"], "example.org")
+        self.assertEqual(doc["published_policy"]["p"], "none")
+        self.assertEqual(doc["published_policy"]["fo"], "1")
+
+
+# Every element of an aggregate report that the schema types as a string,
+# with its plain value; {name} placeholders take a variant.
+_SCALAR_FIELD_VALUES = {
+    "version": "1.0",
+    "org_name": "reporter.example",
+    "email": "dmarc@reporter.example",
+    "extra_contact_info": "https://reporter.example/dmarc",
+    "report_id": "scalar-test",
+    "generator": "reporter/1.0",
+    "domain": "example.com",
+    "adkim": "r",
+    "aspf": "s",
+    "p": "none",
+    "sp": "quarantine",
+    "pct": "100",
+    "fo": "1",
+    "np": "reject",
+    "testing": "n",
+    "discovery_method": "psl",
+    "source_ip": "192.0.2.1",
+    "count": "3",
+    "disposition": "none",
+    "dkim": "pass",
+    "spf": "fail",
+    "header_from": "example.com",
+    "envelope_from": "bounce.example.com",
+    "envelope_to": "example.org",
+    "dkim_domain": "example.com",
+    "selector": "s1",
+    "dkim_result": "pass",
+    "dkim_human_result": "good signature",
+    "spf_domain": "bounce.example.com",
+    "scope": "mfrom",
+    "spf_result": "fail",
+    "spf_human_result": "not listed",
+}
+_SCALAR_FIELD_TEMPLATE = (
+    "<feedback><version>{version}</version><report_metadata>"
+    "<org_name>{org_name}</org_name><email>{email}</email>"
+    "<extra_contact_info>{extra_contact_info}</extra_contact_info>"
+    "<report_id>{report_id}</report_id><generator>{generator}</generator>"
+    "<date_range><begin>1704067200</begin><end>1704153599</end></date_range>"
+    "</report_metadata><policy_published><domain>{domain}</domain>"
+    "<adkim>{adkim}</adkim><aspf>{aspf}</aspf><p>{p}</p><sp>{sp}</sp>"
+    "<pct>{pct}</pct><fo>{fo}</fo><np>{np}</np><testing>{testing}</testing>"
+    "<discovery_method>{discovery_method}</discovery_method></policy_published>"
+    "<record><row><source_ip>{source_ip}</source_ip><count>{count}</count>"
+    "<policy_evaluated><disposition>{disposition}</disposition>"
+    "<dkim>{dkim}</dkim><spf>{spf}</spf>"
+    "<reason><type>other</type><comment>c</comment></reason>"
+    "</policy_evaluated></row><identifiers><header_from>{header_from}</header_from>"
+    "<envelope_from>{envelope_from}</envelope_from>"
+    "<envelope_to>{envelope_to}</envelope_to></identifiers><auth_results>"
+    "<dkim><domain>{dkim_domain}</domain><selector>{selector}</selector>"
+    "<result>{dkim_result}</result><human_result>{dkim_human_result}</human_result>"
+    "</dkim><spf><domain>{spf_domain}</domain><scope>{scope}</scope>"
+    "<result>{spf_result}</result><human_result>{spf_human_result}</human_result>"
+    "</spf></auth_results></record></feedback>"
+)
+
+
+def _scalar_field_xml(**variants: str) -> str:
+    """The report with each named field's value replaced by its variant."""
+    return _SCALAR_FIELD_TEMPLATE.format(**{**_SCALAR_FIELD_VALUES, **variants})
+
+
+# Fields whose element can hold child elements without the report being
+# rejected.
+_OPTIONAL_SCALAR_FIELDS = tuple(
+    name
+    for name in _SCALAR_FIELD_VALUES
+    if name
+    not in (
+        "org_name",
+        "report_id",
+        "domain",
+        "p",
+        "source_ip",
+        "count",
+        "header_from",
+        "dkim_domain",
+        "spf_domain",
+    )
+)
+
+
+# The fields of an aggregate report document that hold objects or lists;
+# "[]" stands for a list item. Every other field must be a scalar.
+_AGGREGATE_DOC_CONTAINERS = {
+    "date_range",
+    "published_policy",
+    "policy_overrides",
+    "policy_overrides[]",
+    "dkim_results",
+    "dkim_results[]",
+    "dkim_results_combined",
+    "spf_results",
+    "spf_results[]",
+    "spf_results_combined",
+    "errors",
+}
+
+
+def _container_paths(value, path=""):
+    """Yield the path of every dict and list inside ``value``."""
+    if isinstance(value, dict):
+        if path:
+            yield path
+        for key, item in value.items():
+            yield from _container_paths(item, f"{path}.{key}" if path else key)
+    elif isinstance(value, list):
+        yield path
+        for item in value:
+            yield from _container_paths(item, f"{path}[]")
+
+
+# A second <policy_published> makes the field a list; the first one's <p>
+# holds a child element.
+_REPEATED_POLICY_VARIANTS = {
+    "p": "<x>1</x>",
+    "discovery_method": (
+        "psl</discovery_method></policy_published><policy_published>"
+        "<domain>example.com</domain><p>none</p><discovery_method>psl"
+    ),
+}
+
+
+class TestAggregateScalarFieldsReachTheDocumentAsScalars(unittest.TestCase):
+    """GHSA-qpcg-29vr-jwm3: a child element under a field the schema types
+    as a string reached the document as an object and mapped the field as
+    one. The walk covers every field, so a scalar field added later is
+    checked too."""
+
+    def _document(self, **variants):
+        report = parse_aggregate_report_xml(_scalar_field_xml(**variants), offline=True)
+        docs = []
+        with (
+            patch("parsedmarc.elastic.Search", return_value=_empty_search()),
+            patch("parsedmarc.elastic.Index") as mock_index_cls,
+            patch("parsedmarc.elastic.connections.get_connection"),
+            patch.object(
+                elastic_module._AggregateReportDoc,
+                "save",
+                autospec=True,
+                side_effect=lambda doc, *args, **kwargs: docs.append(doc.to_dict()),
+            ),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_elasticsearch(cast(dict, report))
+        return docs[0]
+
+    def test_no_object_outside_the_object_fields(self):
+        mixed = {name: f"{value}<x/>" for name, value in _SCALAR_FIELD_VALUES.items()}
+        child = {name: "<x>1</x>" for name in _OPTIONAL_SCALAR_FIELDS}
+        for label, variants in (
+            ("mixed", mixed),
+            ("child", child),
+            ("repeated policy_published", _REPEATED_POLICY_VARIANTS),
+        ):
+            paths = set(_container_paths(self._document(**variants)))
+            self.assertIn("published_policy", paths, msg=label)
+            self.assertLessEqual(paths, _AGGREGATE_DOC_CONTAINERS, msg=label)
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)

tests/test_opensearch.py

@@ -8,18 +8,27 @@ construction, index naming, deduplication queries, error wrapping —
 without needing a running OpenSearch cluster.
 """
 
+import copy
+import json
+import re
 import time
 import unittest
 from collections.abc import Iterator
 from contextlib import contextmanager
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
 from typing import cast
 from unittest.mock import MagicMock, call, patch
 
 from opensearchpy import NotFoundError
 
 import parsedmarc.opensearch as opensearch_module
-from parsedmarc import InvalidFailureReport, parse_report_file
+from parsedmarc import (
+    InvalidFailureReport,
+    parse_aggregate_report_xml,
+    parse_report_email,
+    parse_report_file,
+)
 from parsedmarc.opensearch import (
     AlreadySaved,
     OpenSearchError,
@@ -366,32 +375,50 @@ class TestSetHosts(unittest.TestCase):
 
 class TestCreateIndexes(unittest.TestCase):
     def test_creates_missing_index_with_default_settings(self):
-        with patch("parsedmarc.opensearch.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
+        with (
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = False
             create_indexes(["dmarc_aggregate-2024-01-15"])
-        mock_index.settings.assert_called_once_with(
-            number_of_shards=1, number_of_replicas=0
+        mock_get_conn.return_value.indices.create.assert_called_once_with(
+            index="dmarc_aggregate-2024-01-15",
+            body={"settings": {"number_of_shards": 1, "number_of_replicas": 0}},
         )
-        mock_index.create.assert_called_once()
 
     def test_creates_with_custom_settings(self):
-        with patch("parsedmarc.opensearch.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = False
+        with (
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = False
             create_indexes(
                 ["idx"], settings={"number_of_shards": 3, "refresh_interval": "5s"}
             )
-        mock_index.settings.assert_called_once_with(
-            number_of_shards=3, refresh_interval="5s"
+        body = mock_get_conn.return_value.indices.create.call_args.kwargs["body"]
+        self.assertEqual(
+            body, {"settings": {"number_of_shards": 3, "refresh_interval": "5s"}}
         )
 
+    def test_empty_settings_send_mappings_only(self):
+        with (
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            create_indexes(["idx"], settings={}, mappings={"date_detection": False})
+        body = mock_get_conn.return_value.indices.create.call_args.kwargs["body"]
+        self.assertEqual(body, {"mappings": {"date_detection": False}})
+
     def test_skips_existing_index(self):
-        with patch("parsedmarc.opensearch.Index") as mock_index_cls:
-            mock_index = mock_index_cls.return_value
-            mock_index.exists.return_value = True
-            create_indexes(["idx"])
-        mock_index.create.assert_not_called()
+        with (
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection") as mock_get_conn,
+        ):
+            mock_index_cls.return_value.exists.return_value = True
+            create_indexes(["idx"], mappings={"date_detection": False})
+        mock_get_conn.return_value.indices.create.assert_not_called()
+        mock_get_conn.return_value.indices.put_mapping.assert_not_called()
 
     def test_wraps_sdk_error(self):
         with patch("parsedmarc.opensearch.Index") as mock_index_cls:
@@ -1635,7 +1662,7 @@ class TestCreateDataStreams(_DataStreamTestCase):
             name="dmarc_aggregate"
         )
         # No concrete index is created for a stream.
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         self.assertIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
 
     def test_default_settings_are_one_shard_no_replicas(self):
@@ -1791,7 +1818,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
         client.indices.create_data_stream.assert_called_once_with(
             name="cust_dmarc_aggregate_tenant_a"
         )
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         # The dedup search still uses the wildcard, which matches the
         # stream as well as dated indexes written before the switch.
         self.assertEqual(
@@ -1812,7 +1839,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
             index_call["body"]["@timestamp"],
             datetime(2024, 1, 15, tzinfo=timezone.utc),
         )
-        mock_index_cls.return_value.create.assert_called_once()
+        client.indices.create.assert_called_once()
         client.indices.put_index_template.assert_not_called()
         client.indices.create_data_stream.assert_not_called()
 
@@ -1831,7 +1858,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
             datetime(2024, 1, 1, tzinfo=timezone.utc),
         )
         client.indices.create_data_stream.assert_called_once_with(name="dmarc_failure")
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         self.assertEqual(
             mock_search_cls.call_args.kwargs["index"],
             "dmarc_failure*,dmarc_forensic*",
@@ -1863,7 +1890,7 @@ class TestSaveToDataStreams(_DataStreamTestCase):
             datetime(2024, 2, 3, tzinfo=timezone.utc),
         )
         client.indices.create_data_stream.assert_called_once_with(name="cust_smtp_tls")
-        mock_index_cls.return_value.create.assert_not_called()
+        client.indices.create.assert_not_called()
         self.assertEqual(mock_search_cls.call_args.kwargs["index"], "cust_smtp_tls*")
 
     def test_smtp_tls_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
@@ -1950,9 +1977,558 @@ class TestBackwardCompatAlias(unittest.TestCase):
         )
 
 
+class TestMappingGuards(_DataStreamTestCase):
+    """New indexes and data stream templates carry explicit mappings
+    (GHSA-qpcg-29vr-jwm3).
+
+    Before 11.2.2 they were created with settings only, so the cluster
+    inferred every field's type from the first document, with
+    ``date_detection`` on: an aggregate report with ``report_id``
+    ``2026-11-09`` mapped ``report_id`` as a date for its day, a failure
+    sample header ``Subject.x`` mapped ``sample.headers.subject`` as an
+    object, and invented header names used up the 1,000-field limit, after
+    which legitimate reports for that index were rejected.
+    """
+
+    def _save_dated(self, save_function, report):
+        with (
+            _registered_client() as client,
+            patch("parsedmarc.opensearch.Search") as mock_search_cls,
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+        ):
+            mock_search_cls.return_value.execute.return_value = []
+            mock_index_cls.return_value.exists.return_value = False
+            save_function(report)
+        client.indices.create.assert_called_once()
+        return client
+
+    def _created_mappings(self, save_function, report):
+        client = self._save_dated(save_function, report)
+        return client.indices.create.call_args.kwargs["body"]["mappings"]
+
+    def _template_mappings(self, save_function, report):
+        with (
+            _registered_client() as client,
+            patch("parsedmarc.opensearch.Search") as mock_search_cls,
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+        ):
+            mock_search_cls.return_value.execute.return_value = []
+            client.indices.get_data_stream.side_effect = _not_found()
+            mock_index_cls.return_value.exists.return_value = False
+            save_function(report, use_data_streams=True)
+        template = client.indices.put_index_template.call_args.kwargs["body"]
+        return template["template"]["mappings"]
+
+    def _assert_dates_declared(self, mappings, names):
+        for name in names:
+            self.assertEqual(mappings["properties"][name], {"type": "date"}, name)
+
+    def test_aggregate_index_turns_date_detection_off(self):
+        mappings = self._created_mappings(
+            save_aggregate_report_to_opensearch, _aggregate_report()
+        )
+        self.assertIs(mappings["date_detection"], False)
+        self._assert_dates_declared(
+            mappings, ["@timestamp", "date_range", "date_begin", "date_end"]
+        )
+        self.assertNotIn("sample", mappings["properties"])
+        # report_id stays dynamic: with date detection off a date-shaped
+        # value maps it as a string, not a date.
+        self.assertNotIn("report_id", mappings["properties"])
+
+    def test_smtp_tls_index_turns_date_detection_off(self):
+        mappings = self._created_mappings(
+            save_smtp_tls_report_to_opensearch, _smtp_tls_report()
+        )
+        self.assertIs(mappings["date_detection"], False)
+        self._assert_dates_declared(
+            mappings, ["@timestamp", "date_range", "date_begin", "date_end"]
+        )
+        self.assertNotIn("sample", mappings["properties"])
+
+    def test_failure_index_maps_sample_headers_without_dynamic_fields(self):
+        mappings = self._created_mappings(
+            save_failure_report_to_opensearch, _failure_report()
+        )
+        self.assertIs(mappings["date_detection"], False)
+        self._assert_dates_declared(mappings, ["@timestamp", "arrival_date"])
+        sample = mappings["properties"]["sample"]["properties"]
+        self.assertEqual(sample["date"], {"type": "date"})
+        headers = sample["headers"]
+        self.assertEqual(headers["type"], "object")
+        self.assertIs(headers["dynamic"], False)
+        string_field = {
+            "type": "text",
+            "fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
+        }
+        for name in ("from", "to", "subject", "date", "message-id", "cc"):
+            self.assertEqual(headers["properties"][name], string_field, name)
+        for name in headers["properties"]:
+            # Stored names are lower-cased with "." replaced, so a declared
+            # name must be in that form or no stored header can match it.
+            self.assertEqual(name, name.lower().replace(".", "_"), name)
+
+    def test_failure_mapping_declares_every_header_the_dashboards_query(self):
+        dashboards = Path(__file__).resolve().parent.parent / "dashboards"
+        referenced = set()
+        for path in dashboards.rglob("*"):
+            if path.is_file() and path.suffix in (".json", ".ndjson", ".xml"):
+                text = path.read_text(encoding="utf-8")
+                for match in re.finditer(r"sample\.headers\.([A-Za-z0-9_-]+)", text):
+                    referenced.add(match.group(1).lower())
+        self.assertIn("x-mailer", referenced)
+        mappings = self._created_mappings(
+            save_failure_report_to_opensearch, _failure_report()
+        )
+        declared = mappings["properties"]["sample"]["properties"]["headers"][
+            "properties"
+        ]
+        self.assertLessEqual(referenced, set(declared))
+
+    def test_data_stream_templates_carry_the_mappings(self):
+        for save_function, report, has_sample in (
+            (save_aggregate_report_to_opensearch, _aggregate_report(), False),
+            (save_failure_report_to_opensearch, _failure_report(), True),
+            (save_smtp_tls_report_to_opensearch, _smtp_tls_report(), False),
+        ):
+            case = save_function.__name__
+            opensearch_module._READY_DATA_STREAMS.clear()
+            mappings = self._template_mappings(save_function, report)
+            self.assertIs(mappings["date_detection"], False, case)
+            self.assertEqual(
+                mappings["properties"]["@timestamp"], {"type": "date"}, case
+            )
+            self.assertEqual("sample" in mappings["properties"], has_sample, case)
+            if has_sample:
+                headers = mappings["properties"]["sample"]["properties"]["headers"]
+                self.assertIs(headers["dynamic"], False, case)
+                self.assertIn("subject", headers["properties"], case)
+
+    def test_data_stream_template_adds_timestamp_without_changing_the_argument(
+        self,
+    ):
+        mappings = {"date_detection": False, "properties": {"x": {"type": "long"}}}
+        with (
+            _registered_client() as client,
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+        ):
+            client.indices.get_data_stream.side_effect = _not_found()
+            mock_index_cls.return_value.exists.return_value = False
+            create_data_streams(["s"], mappings=mappings)
+        template = client.indices.put_index_template.call_args.kwargs["body"]
+        self.assertEqual(
+            template["template"]["mappings"],
+            {
+                "date_detection": False,
+                "properties": {"x": {"type": "long"}, "@timestamp": {"type": "date"}},
+            },
+        )
+        self.assertEqual(
+            mappings, {"date_detection": False, "properties": {"x": {"type": "long"}}}
+        )
+
+    def test_dotted_header_name_is_stored_with_an_underscore(self):
+        report = _failure_report()
+        report["parsed_sample"] = dict(report["parsed_sample"])
+        report["parsed_sample"]["headers"] = {
+            "From": [["", "sender@example.com"]],
+            "Subject.x": "v",
+            "X.Y.Z": "w",
+        }
+        client = self._save_dated(save_failure_report_to_opensearch, report)
+        headers = client.index.call_args.kwargs["body"]["sample"]["headers"]
+        self.assertEqual(headers["subject_x"], "v")
+        self.assertEqual(headers["x_y_z"], "w")
+        self.assertNotIn("subject.x", headers)
+        self.assertNotIn("subject", headers)
+        # The address headers are still flattened under their usual keys.
+        self.assertEqual(headers["from"], "sender@example.com")
+
+
 # Silence unused-import lint in the test module preamble.
 _ = call
 
 
+class TestAggregateIndexCap(unittest.TestCase):
+    """One report may create at most MAX_INDEXES_PER_REPORT dated indexes;
+    the overflow records are saved to the report's first index."""
+
+    DAYS = 40
+
+    def _report(self):
+        report = _aggregate_report()
+        template = report["records"][0]
+        report["records"] = []
+        for day in range(self.DAYS):
+            record = copy.deepcopy(template)
+            begin = datetime(2024, 1, 1, tzinfo=timezone.utc) + timedelta(days=day)
+            end = begin + timedelta(days=1)
+            record["interval_begin"] = begin.strftime("%Y-%m-%d %H:%M:%S")
+            record["interval_end"] = end.strftime("%Y-%m-%d %H:%M:%S")
+            report["records"].append(record)
+        return report
+
+    def _save(self, **kwargs):
+        """Save the report; return created index names, the (index,
+        date_begin) of each saved document and the logger mock."""
+        saved = []
+
+        def capture(doc, *args, **save_kwargs):
+            saved.append((doc.meta.index, doc.date_begin))
+
+        with (
+            patch("parsedmarc.opensearch.Search", return_value=_empty_search()),
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection") as mock_get_conn,
+            patch("parsedmarc.opensearch.logger") as mock_logger,
+            patch("parsedmarc.opensearch.create_data_streams") as mock_streams,
+            patch.object(
+                opensearch_module._AggregateReportDoc,
+                "save",
+                autospec=True,
+                side_effect=capture,
+            ),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_opensearch(self._report(), **kwargs)
+            created = [
+                c.kwargs["index"]
+                for c in mock_get_conn.return_value.indices.create.call_args_list
+            ]
+        return created, saved, mock_logger, mock_streams
+
+    def test_daily_indexes_are_capped_and_overflow_goes_to_the_first(self):
+        created, saved, mock_logger, _ = self._save()
+        expected = [f"dmarc_aggregate-2024-01-{d:02d}" for d in range(1, 32)]
+        self.assertEqual(sorted(set(created)), expected)
+        self.assertEqual(len(saved), self.DAYS)
+        self.assertEqual([index for index, _ in saved[:31]], expected)
+        self.assertEqual(
+            [index for index, _ in saved[31:]], ["dmarc_aggregate-2024-01-01"] * 9
+        )
+        # The overflow records keep their own dates.
+        begins = [begin for _, begin in saved]
+        self.assertEqual(begins[31], datetime(2024, 2, 1, tzinfo=timezone.utc))
+        self.assertEqual(begins[39], datetime(2024, 2, 9, tzinfo=timezone.utc))
+        mock_logger.warning.assert_called_once()
+        self.assertIn("9 records", mock_logger.warning.call_args.args[0])
+
+    def test_routing_warning_escapes_the_report_id(self):
+        """The report ID is sender-controlled; a CR/LF in it must not forge
+        a log line."""
+        report = self._report()
+        report["report_metadata"]["report_id"] = "poc\r\nforged"
+        with (
+            patch("parsedmarc.opensearch.Search", return_value=_empty_search()),
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection"),
+            patch("parsedmarc.opensearch.logger") as mock_logger,
+            patch.object(opensearch_module._AggregateReportDoc, "save", autospec=True),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_opensearch(report)
+        message = mock_logger.warning.call_args.args[0]
+        self.assertIn("poc\\r\\nforged", message)
+        self.assertNotIn("\n", message)
+
+    def test_monthly_indexes_are_not_affected(self):
+        created, saved, mock_logger, _ = self._save(monthly_indexes=True)
+        self.assertEqual(
+            sorted(set(created)), ["dmarc_aggregate-2024-01", "dmarc_aggregate-2024-02"]
+        )
+        self.assertEqual(
+            [index for index, _ in saved],
+            ["dmarc_aggregate-2024-01"] * 31 + ["dmarc_aggregate-2024-02"] * 9,
+        )
+        mock_logger.warning.assert_not_called()
+
+    def test_data_streams_are_not_affected(self):
+        created, saved, mock_logger, mock_streams = self._save(use_data_streams=True)
+        self.assertEqual(created, [])
+        self.assertEqual({index for index, _ in saved}, {"dmarc_aggregate"})
+        self.assertEqual(len(saved), self.DAYS)
+        self.assertEqual(mock_streams.call_count, self.DAYS)
+        mock_logger.warning.assert_not_called()
+
+
+class TestSaveFailureReportEmptyAddressHeaders(unittest.TestCase):
+    """GHSA-4272-4r8g-6c5h: a failure sample whose From, To or Reply-To
+    header holds no address made the save index ``headers[...][0]`` of
+    mail-parser's empty list and raise ``IndexError``, which escaped the
+    CLI and stopped the mailbox run. The header is now left out of the
+    document and of the duplicate query."""
+
+    # An empty From: and Reply-To:, and To: undisclosed-recipients:; (an
+    # RFC 5322 §3.4 group with no members, which legitimate mail uses).
+    ARF = (
+        "From: dmarc@reporter.example\n"
+        "To: ruf@example.com\n"
+        "Date: Mon, 01 Jan 2024 00:00:00 +0000\n"
+        "Subject: DMARC Failure Report\n"
+        "MIME-Version: 1.0\n"
+        'Content-Type: multipart/report; report-type=feedback-report; boundary="b"\n'
+        "\n"
+        "--b\n"
+        "Content-Type: text/plain\n"
+        "\n"
+        "Authentication failure report.\n"
+        "--b\n"
+        "Content-Type: message/feedback-report\n"
+        "\n"
+        "Feedback-Type: auth-failure\n"
+        "User-Agent: test/1.0\n"
+        "Version: 1\n"
+        "Original-Mail-From: user@example.com\n"
+        "Arrival-Date: Mon, 01 Jan 2024 00:00:00 +0000\n"
+        "Source-IP: 192.0.2.1\n"
+        "Reported-Domain: example.com\n"
+        "Authentication-Results: mx; dmarc=fail\n"
+        "\n"
+        "--b\n"
+        "Content-Type: text/rfc822-headers\n"
+        "\n"
+        "From:\n"
+        "To: undisclosed-recipients:;\n"
+        "Reply-To:\n"
+        "Subject: spoofed\n"
+        "Date: Mon, 01 Jan 2024 00:00:00 +0000\n"
+        "--b--\n"
+    )
+
+    def test_empty_address_headers_are_left_out(self):
+        parsed = parse_report_email(self.ARF, offline=True)
+        self.assertEqual(parsed["report_type"], "failure")
+        report = cast(dict, parsed["report"])
+        headers = report["parsed_sample"]["headers"]
+        self.assertEqual(
+            (headers["From"], headers["To"], headers["Reply-To"]), ([], [], [])
+        )
+        search = _empty_search()
+        with (
+            patch("parsedmarc.opensearch.Search", return_value=search),
+            patch("parsedmarc.opensearch.Index"),
+            patch.object(
+                opensearch_module._FailureReportDoc, "save", autospec=True
+            ) as mock_save,
+        ):
+            save_failure_report_to_opensearch(report)
+        doc = mock_save.call_args.args[0]
+        self.assertEqual(doc.sample.headers["subject"], "spoofed")
+        for header in ("from", "to", "reply-to"):
+            self.assertNotIn(header, doc.sample.headers)
+        query = json.dumps(search.query.to_dict())
+        self.assertIn("sample.headers.subject", query)
+        self.assertNotIn("sample.headers.from", query)
+        self.assertNotIn("sample.headers.to", query)
+
+
+class TestAggregateAttributesDoNotReachTheDocument(unittest.TestCase):
+    """GHSA-qpcg-29vr-jwm3: an XML attribute on a scalar element reached the
+    document as an object and mapped the field as one."""
+
+    def test_document_holds_no_object_for_attributed_fields(self):
+        attribute = ' a="1"'
+        xml = (
+            f"<feedback><version{attribute}>1.0</version><report_metadata>"
+            "<org_name>example.net</org_name><email>a@example.net</email>"
+            "<report_id>r</report_id><date_range><begin>1704067200</begin>"
+            "<end>1704153599</end></date_range></report_metadata>"
+            "<policy_published><domain>example.com</domain>"
+            f"<p{attribute}>none</p><fo{attribute}>1</fo></policy_published>"
+            "<record><row><source_ip>192.0.2.1</source_ip><count>1</count>"
+            f"<policy_evaluated><disposition{attribute}>none</disposition>"
+            "<dkim>pass</dkim><spf>pass</spf></policy_evaluated></row>"
+            "<identifiers><header_from>example.com</header_from>"
+            f"<envelope_to{attribute}>example.org</envelope_to></identifiers>"
+            "<auth_results><spf><domain>example.com</domain><result>pass</result>"
+            "</spf></auth_results></record></feedback>"
+        )
+        report = parse_aggregate_report_xml(xml, offline=True)
+        docs = []
+        with (
+            patch("parsedmarc.opensearch.Search", return_value=_empty_search()),
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection"),
+            patch.object(
+                opensearch_module._AggregateReportDoc,
+                "save",
+                autospec=True,
+                side_effect=lambda doc, *args, **kwargs: docs.append(doc.to_dict()),
+            ),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_opensearch(cast(dict, report))
+        doc = docs[0]
+        self.assertEqual(doc["xml_schema"], "1.0")
+        self.assertEqual(doc["disposition"], "none")
+        self.assertEqual(doc["envelope_to"], "example.org")
+        self.assertEqual(doc["published_policy"]["p"], "none")
+        self.assertEqual(doc["published_policy"]["fo"], "1")
+
+
+# Every element of an aggregate report that the schema types as a string,
+# with its plain value; {name} placeholders take a variant.
+_SCALAR_FIELD_VALUES = {
+    "version": "1.0",
+    "org_name": "reporter.example",
+    "email": "dmarc@reporter.example",
+    "extra_contact_info": "https://reporter.example/dmarc",
+    "report_id": "scalar-test",
+    "generator": "reporter/1.0",
+    "domain": "example.com",
+    "adkim": "r",
+    "aspf": "s",
+    "p": "none",
+    "sp": "quarantine",
+    "pct": "100",
+    "fo": "1",
+    "np": "reject",
+    "testing": "n",
+    "discovery_method": "psl",
+    "source_ip": "192.0.2.1",
+    "count": "3",
+    "disposition": "none",
+    "dkim": "pass",
+    "spf": "fail",
+    "header_from": "example.com",
+    "envelope_from": "bounce.example.com",
+    "envelope_to": "example.org",
+    "dkim_domain": "example.com",
+    "selector": "s1",
+    "dkim_result": "pass",
+    "dkim_human_result": "good signature",
+    "spf_domain": "bounce.example.com",
+    "scope": "mfrom",
+    "spf_result": "fail",
+    "spf_human_result": "not listed",
+}
+_SCALAR_FIELD_TEMPLATE = (
+    "<feedback><version>{version}</version><report_metadata>"
+    "<org_name>{org_name}</org_name><email>{email}</email>"
+    "<extra_contact_info>{extra_contact_info}</extra_contact_info>"
+    "<report_id>{report_id}</report_id><generator>{generator}</generator>"
+    "<date_range><begin>1704067200</begin><end>1704153599</end></date_range>"
+    "</report_metadata><policy_published><domain>{domain}</domain>"
+    "<adkim>{adkim}</adkim><aspf>{aspf}</aspf><p>{p}</p><sp>{sp}</sp>"
+    "<pct>{pct}</pct><fo>{fo}</fo><np>{np}</np><testing>{testing}</testing>"
+    "<discovery_method>{discovery_method}</discovery_method></policy_published>"
+    "<record><row><source_ip>{source_ip}</source_ip><count>{count}</count>"
+    "<policy_evaluated><disposition>{disposition}</disposition>"
+    "<dkim>{dkim}</dkim><spf>{spf}</spf>"
+    "<reason><type>other</type><comment>c</comment></reason>"
+    "</policy_evaluated></row><identifiers><header_from>{header_from}</header_from>"
+    "<envelope_from>{envelope_from}</envelope_from>"
+    "<envelope_to>{envelope_to}</envelope_to></identifiers><auth_results>"
+    "<dkim><domain>{dkim_domain}</domain><selector>{selector}</selector>"
+    "<result>{dkim_result}</result><human_result>{dkim_human_result}</human_result>"
+    "</dkim><spf><domain>{spf_domain}</domain><scope>{scope}</scope>"
+    "<result>{spf_result}</result><human_result>{spf_human_result}</human_result>"
+    "</spf></auth_results></record></feedback>"
+)
+
+
+def _scalar_field_xml(**variants: str) -> str:
+    """The report with each named field's value replaced by its variant."""
+    return _SCALAR_FIELD_TEMPLATE.format(**{**_SCALAR_FIELD_VALUES, **variants})
+
+
+# Fields whose element can hold child elements without the report being
+# rejected.
+_OPTIONAL_SCALAR_FIELDS = tuple(
+    name
+    for name in _SCALAR_FIELD_VALUES
+    if name
+    not in (
+        "org_name",
+        "report_id",
+        "domain",
+        "p",
+        "source_ip",
+        "count",
+        "header_from",
+        "dkim_domain",
+        "spf_domain",
+    )
+)
+
+
+# The fields of an aggregate report document that hold objects or lists;
+# "[]" stands for a list item. Every other field must be a scalar.
+_AGGREGATE_DOC_CONTAINERS = {
+    "date_range",
+    "published_policy",
+    "policy_overrides",
+    "policy_overrides[]",
+    "dkim_results",
+    "dkim_results[]",
+    "dkim_results_combined",
+    "spf_results",
+    "spf_results[]",
+    "spf_results_combined",
+    "errors",
+}
+
+
+def _container_paths(value, path=""):
+    """Yield the path of every dict and list inside ``value``."""
+    if isinstance(value, dict):
+        if path:
+            yield path
+        for key, item in value.items():
+            yield from _container_paths(item, f"{path}.{key}" if path else key)
+    elif isinstance(value, list):
+        yield path
+        for item in value:
+            yield from _container_paths(item, f"{path}[]")
+
+
+# A second <policy_published> makes the field a list; the first one's <p>
+# holds a child element.
+_REPEATED_POLICY_VARIANTS = {
+    "p": "<x>1</x>",
+    "discovery_method": (
+        "psl</discovery_method></policy_published><policy_published>"
+        "<domain>example.com</domain><p>none</p><discovery_method>psl"
+    ),
+}
+
+
+class TestAggregateScalarFieldsReachTheDocumentAsScalars(unittest.TestCase):
+    """GHSA-qpcg-29vr-jwm3: a child element under a field the schema types
+    as a string reached the document as an object and mapped the field as
+    one. The walk covers every field, so a scalar field added later is
+    checked too."""
+
+    def _document(self, **variants):
+        report = parse_aggregate_report_xml(_scalar_field_xml(**variants), offline=True)
+        docs = []
+        with (
+            patch("parsedmarc.opensearch.Search", return_value=_empty_search()),
+            patch("parsedmarc.opensearch.Index") as mock_index_cls,
+            patch("parsedmarc.opensearch.connections.get_connection"),
+            patch.object(
+                opensearch_module._AggregateReportDoc,
+                "save",
+                autospec=True,
+                side_effect=lambda doc, *args, **kwargs: docs.append(doc.to_dict()),
+            ),
+        ):
+            mock_index_cls.return_value.exists.return_value = False
+            save_aggregate_report_to_opensearch(cast(dict, report))
+        return docs[0]
+
+    def test_no_object_outside_the_object_fields(self):
+        mixed = {name: f"{value}<x/>" for name, value in _SCALAR_FIELD_VALUES.items()}
+        child = {name: "<x>1</x>" for name in _OPTIONAL_SCALAR_FIELDS}
+        for label, variants in (
+            ("mixed", mixed),
+            ("child", child),
+            ("repeated policy_published", _REPEATED_POLICY_VARIANTS),
+        ):
+            paths = set(_container_paths(self._document(**variants)))
+            self.assertIn("published_policy", paths, msg=label)
+            self.assertLessEqual(paths, _AGGREGATE_DOC_CONTAINERS, msg=label)
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)

tests/test_postgres.py

@@ -1012,5 +1012,20 @@ class TestPostgreSQLWithSamples(unittest.TestCase):
         self.assertGreater(saved, 0, "Expected at least one SMTP TLS sample")
 
 
+class TestEnsureUtcSuffixNonString(unittest.TestCase):
+    """GHSA-4272-4r8g-6c5h: ``_ensure_utc_suffix(5)`` raised ``TypeError``
+    (``"+" in 5``) outside the save's ``psycopg.Error`` handler, so an SMTP
+    TLS report with ``"start-datetime": 5`` escaped the CLI. A non-string
+    is now returned unchanged, for PostgreSQL to accept or reject."""
+
+    def test_non_string_is_returned_unchanged(self):
+        for value in (5, 1.5, ["2024-01-01"]):
+            with self.subTest(value=value):
+                self.assertEqual(
+                    _ensure_utc_suffix(value),  # pyright: ignore[reportArgumentType]
+                    value,
+                )
+
+
 if __name__ == "__main__":
     unittest.main()

tests/test_splunk.py

@@ -538,5 +538,24 @@ class TestSplunkBackwardCompatAlias(unittest.TestCase):
         )
 
 
+class TestSmtpTlsTimestampErrorIsSplunkError(unittest.TestCase):
+    """GHSA-4272-4r8g-6c5h: the SMTP TLS save converted ``begin_date``
+    outside its ``try``, so a wrong-typed date (``"start-datetime": 5`` in
+    the report) raised a bare ``AttributeError`` that the CLI did not catch.
+    Parsing now rejects such a report; the save still raises
+    ``SplunkError`` for a value that does not convert."""
+
+    def test_unconvertible_begin_date_raises_splunk_error(self):
+        client = _client()
+        client.session = MagicMock()
+        for begin_date in (5, "not a date"):
+            report = _smtp_tls_report()
+            report["begin_date"] = begin_date
+            with self.subTest(begin_date=begin_date):
+                with self.assertRaises(SplunkError):
+                    client.save_smtp_tls_reports_to_splunk([report])
+        client.session.post.assert_not_called()
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)

tests/test_utils.py

@@ -27,6 +27,7 @@ from expiringdict import ExpiringDict
 import parsedmarc
 import parsedmarc.resources.ipinfo
 import parsedmarc.utils
+from tests.timing import budget
 from tests.tzutil import force_tz
 
 
@@ -293,11 +294,12 @@ class Test(unittest.TestCase):
         comment is left unclosed so the old pattern can never match: with a
         closing parenthesis its first attempt succeeded and the backtracking
         never happened."""
+        limit = budget(5)
         start = time.perf_counter()
         # dateutil's ParserError subclasses ValueError.
         with self.assertRaises(ValueError):
             parsedmarc.utils.human_timestamp_to_datetime(" " * 200_000 + "(x")
-        self.assertLess(time.perf_counter() - start, 5)
+        self.assertLess(time.perf_counter() - start, limit)
 
     def testHumanTimestampToDatetimeNegativeZero(self):
         """-0000 timezone is handled"""
@@ -628,9 +630,10 @@ class TestRedactUrlCredentials(unittest.TestCase):
             "/@" * 500_000,
         ):
             with self.subTest(prefix=value[:8]):
+                limit = budget(5)
                 started = time.monotonic()
                 parsedmarc.utils.redact_url_credentials(value)
-                self.assertLess(time.monotonic() - started, 5)
+                self.assertLess(time.monotonic() - started, limit)
 
 
 class TestSanitizeForLog(unittest.TestCase):
@@ -1325,6 +1328,27 @@ class TestUtilsIpDbPaths(unittest.TestCase):
 class TestUtilsParseEmail(unittest.TestCase):
     """Tests for parse_email edge cases"""
 
+    def testOverLimitSampleIsRejected(self):
+        """parse_email raises MessageLimitExceeded for a sample whose
+        message/delivery-status body holds too many blank-line-separated
+        blocks, each a part (GHSA-qcc2-4x3v-v4g7), and still parses a normal
+        one"""
+        from mailsuite.utils import MessageLimitExceeded
+
+        sample = (
+            "From: test@example.com\nSubject: Test\n"
+            "Content-Type: message/delivery-status\n\n" + "\n" * 200_000
+        )
+        with self.assertRaises(MessageLimitExceeded) as ctx:
+            parsedmarc.utils.parse_email(sample)
+        self.assertEqual(str(ctx.exception), "Message exceeds max_parts (10000)")
+        with self.assertRaises(MessageLimitExceeded):
+            parsedmarc.utils.parse_email(sample.encode())
+        result = parsedmarc.utils.parse_email(
+            "From: test@example.com\nSubject: Ok\n\nhi"
+        )
+        self.assertEqual(result["subject"], "Ok")
+
     def testMinimalEmail(self):
         """parse_email handles email with minimal headers"""
         email_str = """From: test@example.com

tests/timing.py

@@ -0,0 +1,29 @@
+"""Timing bounds that scale with the machine running the suite."""
+
+import email
+import time
+
+_REFERENCE_MESSAGE = (
+    "From: sender@example.net\r\nTo: dmarc@example.com\r\nSubject: x\r\n\r\n"
+    + "a\r\n" * 300_000
+)
+
+
+def budget(seconds: float) -> float:
+    """Return ``seconds``, scaled up on a machine slower than the reference.
+
+    Timed checks do real work at a limit, so their bound must scale with the
+    machine running them. The reference is ``_REFERENCE_MESSAGE`` parsed by the
+    email package, 0.04 s on the workstation the bounds were written on
+    (Python 3.12). It is measured just before the timed call, twice, and the
+    slower run sets the scale, so a brief fast moment in the reference is not
+    taken as the machine's speed; a slowdown during the timed call itself is
+    not covered. A faster machine keeps the written bound, and the scale is
+    capped at ten, so a stall during the reference cannot hide a regression.
+    """
+    reference = 0.0
+    for _ in range(2):
+        start = time.perf_counter()
+        email.message_from_string(_REFERENCE_MESSAGE)
+        reference = max(reference, time.perf_counter() - start)
+    return seconds * min(max(1.0, reference / 0.04), 10.0)