amorfo77/parsedmarc
1
0 Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-11-11 03:11:33 +00:00
Code Issues Projects Releases Wiki Activity
1,694 Commits 1 Branch 73 Tags
14c04d78b049dbc357256e2c7ef76d12b0cd8308
Commit Graph
18 Commits
Author SHA1 Message Date
Sean WhalenandClaude Fable 5.2 d9193eafea Reject reports dated in the future and bound what parse_report_file() reads (#954) 
* Reject reports whose index-naming date is more than two days in the future (#947)

The date that names a report's dated Elasticsearch/OpenSearch index (an
aggregate report's begin date, a failure report's Arrival-Date, an SMTP
TLS report's start-datetime) was never compared with the present, so a
report dated 2100 named an index for 2100. Each parser now rejects such a
report with its own Invalid*Report; past dates stay unbounded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Refuse an input larger than the report size cap before parse_report_file() reads it (#949)

The 32 MiB cap was applied only after the whole input was read. A path is
now checked with os.path.getsize(), bytes-like input with len(), and a
file object is read at most one byte past the cap. The error is the
ParserError with the message the content check already gives. Three
existing tests that fed over-cap content now see the earlier rejection.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Measure a memoryview input by its byte size

len() of a memoryview counts items, which are not bytes for every format,
so the size cap in parse_report_file() compares nbytes instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Bound the aggregate end date and read file objects to their end

Address the Copilot review of #954. A normalized report's per-day records
name their indexes from their own interval starts, so the report's end date
is bounded like its begin date. A raw stream may return fewer bytes than
asked for before its end, so parse_report_file() now reads in a loop until
end of file or one byte past the cap.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Accumulate bounded reads in one buffer

Address the second Copilot review of #954. Keeping every chunk in a list
cost one bytes object per read, about 85 bytes per byte for a stream that
returns a byte at a time (22.7 MiB for 256 KiB of input). One bytearray
keeps memory proportional to the bytes read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-11-10 17:09:23 -05:00
Sean WhalenClaude Sonnet 5.5github-actions[bot]
 		2ee209005b Merge commit from fork 
* chore: update IPinfo Lite MMDB (#941)

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Cap report date range and indexes created per report

A report whose <date_range> spans millions of days was normalized into one
record per day, exhausting memory, and with Elasticsearch or OpenSearch
output created one dated index (one shard) per day, exhausting the
cluster's shard budget.

Reject a range longer than MAX_REPORT_SPAN_DAYS (366) or negative with
InvalidAggregateReport. In both save functions, once a report has used
MAX_INDEXES_PER_REPORT (31) distinct dated indexes, route further records
to the report's first index, keeping their own dates, and log one warning.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Catch output-stage exceptions and validate SMTP TLS report types

One report sent to a monitored mailbox could make a configured output
raise an exception that process_reports() did not catch. It escaped
through the mailbox save callback, _main logged "Mailbox Error" and
exited 1, and the batch stayed in the mailbox, so every one-shot run
died on the same message and duplicated the legitimate reports in
non-deduplicating outputs. Three sender-controlled inputs did it:

- A failure report whose sample has an empty From:, To: or Reply-To:
  (To: undisclosed-recipients:; included): mail-parser gives an empty
  address list, and the Elasticsearch and OpenSearch saves indexed
  [0] (IndexError).
- An SMTP TLS report with a wrong-typed field: "policy-string": [1]
  made the CSV writer raise TypeError; "start-datetime": 5 made the
  Elasticsearch, OpenSearch and Splunk saves raise AttributeError and
  _ensure_utc_suffix() in the PostgreSQL save raise TypeError.
- An aggregate report whose <reason> has no <type> (KeyError in the
  CSV writer) or a <type> that xmltodict parses to a dict or a list
  (TypeError in its join).

Fix:

- Every output destination block in process_reports() now ends with
  the catch-all the aggregate Elasticsearch block had: the failure and
  SMTP TLS Elasticsearch/OpenSearch blocks, every PostgreSQL and
  Splunk HEC block, and the file output (its OSError/ValueError branch
  is kept). The error goes through log_output_error(), so the batch is
  reported unsaved and fail_on_output_error still raises ParserError.
- parse_smtp_tls_report_json() and its helpers check each field
  against the type RFC 8460 section 4.4 gives it and raise
  InvalidSMTPTLSReport otherwise: strings for organization-name,
  report-id, policy-domain, policy-type, result-type and the optional
  failure-details fields; contact-info a string or a list of strings;
  policy-string and a list mx-host-pattern lists of strings; the
  session counts integers or null; failure-details a list of objects;
  start-datetime and end-datetime strings of at most 64 characters that
  the outputs' own conversions accept. dateutil took 12.4 s to fail on
  a 1,000,000-digit date, hence the length bound.
- _parse_report_record() makes a reason's type and comment a string or
  None (a dict unwrapped with _text(), a list as its str()), and the
  aggregate CSV writer reads both with .get().
- The Elasticsearch and OpenSearch failure saves flatten From, To and
  Reply-To with a helper that returns None for an empty list, and
  leave such a header out of the document and the duplicate query.
- The Splunk SMTP TLS save converts begin_date inside its try, so a bad
  value raises SplunkError; _ensure_utc_suffix() returns a non-string
  unchanged.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Limit XML nesting depth and node count in aggregate reports

parse_aggregate_report_xml() gave the decompressed report (up to the
100 MiB decompression cap) to xmltodict with no limit on depth or element
count. xmltodict keeps one stack entry per open element and expat reports
missing end tags only at the end of the input, so a few KB of gzipped
unclosed <a> tags grew to gigabytes of RSS and the kernel OOM killer
ended the process before the message was moved; a wide document of
millions of siblings did the same at a smaller scale. A failed first
parse also re-parsed the document through lxml recovery, which has no
node limit, and a second xmltodict pass.

_NoDoctypeExpat.ParserCreate() now returns a proxy around the expat
parser. pyexpat parsers cannot be subclassed and xmltodict assigns its
handlers after ParserCreate() returns, so the proxy's __setattr__ wraps
StartElementHandler to count depth and nodes (elements plus attributes)
and EndElementHandler to decrement depth, and forwards every other
attribute. Over MAX_XML_DEPTH (100) or MAX_XML_NODES (5,000,000) it
raises before xmltodict's handler runs. Both xmltodict.parse() call
sites turn that into InvalidAggregateReport naming the limit; the first
does so instead of trying lxml recovery.

The lxml recovery path now runs a counting pass first, with a parser
target that builds no tree and counts elements, attributes and
namespace declarations against MAX_XML_NODES; the tree is built only
for a document within the limit. Depth there is bounded by libxml2's
own limit of 256 (huge_tree stays off) and then by the second
xmltodict pass. A 22 MiB wide document behind a malformed XML
declaration peaked at +1532 MiB before this pass and +23 MiB after.

The largest aggregate sample (909 KB) has 32,019 nodes, so a legitimate
report at the 100 MiB cap stays near 3.6 million.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Create Elasticsearch/OpenSearch indexes with mapping guards

create_indexes() and create_data_streams() created indexes and the data
stream template with settings only, so the cluster typed every report
field from the first document that held it, with date_detection on. A
sender could be first into a day's index (its name comes from the
report) and poison it: a report_id of "2026-11-09" mapped report_id as a
date, a failure sample header named "Subject.x" mapped
sample.headers.subject as an object, and samples with invented header
names used up index.mapping.total_fields.limit. Every later legitimate
report for that index was then rejected and ended up in Unsaved; in data
stream mode this lasted until a rollover.

Both functions now take a mappings body, and the three save functions
pass one per report type. Each turns date_detection off and declares the
fields that hold dates (@timestamp, date_range, date_begin, date_end,
arrival_date, sample.date) as dates, the type date detection gave them
before; on the sample reports every other field keeps its old type. The failure mapping
maps sample.headers as an object with dynamic: false and declares, as
text with a keyword sub-field (the dynamic string default), every header
the shipped dashboards reference plus date, message-id and cc; all
headers still stay in _source. A "." in a header name is replaced with
"_" before the header is stored. The index is now created through the
client's indices.create with the settings and mappings body, because
opensearch-py's Index.create cannot take a mappings argument; the data
stream template merges the mappings with its @timestamp date mapping.

Existing indexes are not changed. The other report fields stay dynamic
and object-mapped (no nested), which the dashboards need (issue #169).

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Read the RFC 8460 mx-host key in SMTP TLS reports

RFC 8460 section 4.4 keys a policy's MX host patterns "mx-host"; its
schema line is "mx-host": mx-host-pattern, where mx-host-pattern is the
value type, described as "a JSON array of strings", and the RFC's
Appendix B example gives the value as a plain string. The parser read
only "mx-host-pattern", so every RFC-conformant report lost the field
(samples/smtp_tls/rfc8460.json, the Appendix B example, among them).

_parse_smtp_tls_report_policy() now reads "mx-host" as a string or a
list of strings, a string becoming a one-element list, and falls back
to "mx-host-pattern" (same rules) when "mx-host" is absent, null or
empty. Any other type raises InvalidSMTPTLSReport. The output key is
still mx_host_patterns, so the PostgreSQL TEXT[] column, the
Elasticsearch/OpenSearch field and the CSV column are unchanged.

Refs #942

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject over-limit messages with the mailsuite guard

parse_report_email() and parse_email() handed every message to
mail-parser and the email package with no bound on blank lines, header
lines, header field size, MIME parts or nested multiparts, so a small
crafted message took seconds to minutes and gigabytes to parse while the
rest of the mailbox waited.

Both now call mailsuite.utils.check_message_limits() first, one linear
pass. parse_report_email() turns MessageLimitExceeded into
InvalidDMARCReport so the mailbox loop files the message under Invalid;
parse_failure_report() turns one from the sample into InvalidFailureReport.
The mailsuite floor rises to 2.3.5, which adds the guard.

Messages nested deeper than 100 multiparts are now rejected by the guard
instead of reaching the recursion handler; the tests that used the
400-level message expect the new message.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Cite RFC 7489 §6.3 for the daily report interval

The comment on MAX_REPORT_SPAN_DAYS cited RFC 7489 Appendix A for the
one-day aggregate report. The daily interval comes from §6.3: the "ri"
tag requests the interval between aggregate reports, and its default is
86400 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add the 11.2.2 changelog: five advisories and #942

One Security bullet per advisory (GHSA-qcc2, GHSA-q7gm, GHSA-p85v,
GHSA-4272, GHSA-qpcg) and a Bug fixes bullet for the RFC 8460 mx-host
key. The version in constants.py is bumped by the release pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the date range rejection message and pyright's analysis

With the XML limits and the date range limit merged,
parse_aggregate_report_xml() became too complex for pyright to analyze
("Code is too complex to analyze"), and the date range rejection,
raised inside the function's catch-all, reached the caller as
"Unexpected error: Report date range spans ...".

The date range check moves to _check_report_span() and the second
xmltodict pass to _parse_limited_xml(), which raises the XML limit
breach as InvalidAggregateReport; the catch-all now re-raises an
InvalidAggregateReport as it is. The tests assert the message is not
re-wrapped.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Put the over-limit failure samples in a delivery-status body

mailsuite 2.3.5's check_message_limits() follows the MIME structure and
counts blank lines only inside message/delivery-status bodies, the shape
whose parse cost is per blank line; blank lines elsewhere in a body are
not limited. The two failure-sample tests used blank lines in a plain
body, which the final check accepts, so they now put them in a
delivery-status body. parsedmarc's call with the default limits is
unchanged.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Describe the final mailsuite check and the index cap in the docs

The GHSA-qcc2 changelog bullet now describes check_message_limits() as
mailsuite 2.3.5 ships it: limits follow the MIME structure, and body
lines outside message/delivery-status bodies are not limited. The
changelog section moves to Unreleased, as parsedmarc's AGENTS.md has
fix pull requests write it; the release pull request names the version.

The Elasticsearch and OpenSearch pages now say that an address header
with no address is left out of the failure document, and that a report
spanning more than 31 days saves its later records to its first dated
index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound records times days in normalization and fix the span message

The date range limit bounds how far one record is split, but not how
many records are split: 10,000 records over 366 days, each with a count
of 1,000,000 (3.3 MB of XML), produced 3,670,000 records in 13.5 s and
1.7 GB, and 30,000 of them (a 44 KB gzip) ran out of memory and reached
the caller as "Unexpected error: ". A report whose split would exceed
MAX_NORMALIZED_RECORDS (250,000) now keeps its records whole over its own
range, with timespan_requires_normalization false and one warning; a
large legitimate multi-day report is kept rather than rejected. The
record loop moves to _parse_report_records() so pyright can still
analyze parse_aggregate_report_xml().

The span message used float division, which raises OverflowError for an
<end> of 4,001 digits; it now uses integer division. The index-routing
warning escapes the sender's report ID with sanitize_for_log(). The
changelog and the Elasticsearch and OpenSearch pages say the index cap
is per report, that dated index names still come from report dates, and
that data streams are affected by neither.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Drop XML attributes from aggregate reports before they are used

xmltodict turns an element with an attribute, such as <p a="1">none</p>,
into {"@a": "1", "#text": "none"}. Such objects passed through the
parser into Elasticsearch and OpenSearch documents for version, p, fo,
disposition, envelope_to and other scalar fields, so the first one mapped
the field as an object and every later report to that index failed, and
hundreds of attributes on one element used up total_fields.limit. The
parsed report now goes through one linear pass that replaces an element
with only attributes and text by its text and drops attributes from the
others; the existing _text() calls now receive strings.

The changelog and the Elasticsearch and OpenSearch pages also say that
an existing data stream's next backing index gets the guarded mappings,
since parsedmarc puts the template again in every process, and the
indexed-header comment and changelog name cc as the one header added to
those the dashboards reference.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject lone surrogates in SMTP TLS reports and guard the stdout print

An SMTP TLS string field holding a lone surrogate (JSON "\ud800")
passed every type check and survived json.dumps(ensure_ascii=False), and
print(output_str) in process_reports(), outside every try, raised
UnicodeEncodeError: parsedmarc exited 1, and in mailbox mode the error
escaped the save callback. RFC 8460 section 4 reports are I-JSON, whose RFC
7493 section 2.1 forbids surrogate code points, so the string and
string-list checks now reject one as InvalidSMTPTLSReport, and the print
records a UnicodeEncodeError as a "Standard output" output error.

The _smtp_tls_datetime() docstring no longer says a parsed date cannot
fail in any output: dateutil accepts partial dates that PostgreSQL may
reject, which is contained as PostgreSQLError. The _reason_text()
docstring says a dict with child elements gives None.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Describe mailsuite 2.3.5's address group handling in the changelog

The final mailsuite 2.3.5 no longer rejects an address header by its
comma count; its parse_email() removes RFC 5322 group syntax from address
header values over 8 KiB before parsing them. The boundary limit
counts lines that are the boundary of an open multipart. parsedmarc's
call with the default limits is unchanged.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test the keep-alive call of the extracted record loop

_parse_report_records() holds the record loop moved out of
parse_aggregate_report_xml(); its keep-alive branch had no test.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test a policy override type with child elements

With XML attributes dropped at parse time, a dict reaches _reason_text()
only for an element with child elements; the test pins its None result
and the CSV's "none".

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Check plain-string contact-info and mx-host for lone surrogates

The surrogate check covered the string fields and string lists, but a
contact-info or mx-host given as one string took a branch that skipped
it, and a lone surrogate there still made the stdout print raise.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say which sample headers the failure document leaves out

The GHSA-qpcg changelog bullet said every sample header stays in
_source, but the failure save leaves out a From, To or Reply-To header
that holds no address, as the Elasticsearch and OpenSearch pages say.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep sender-derived numbers out of the date range message

The rejection message printed the report's span in days, so an <end> of
thousands of digits put a number of thousands of digits into the
exception text and the log. It now says only that the range is negative
or longer than MAX_REPORT_SPAN_DAYS.

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Store every string-typed aggregate field as a string or null

Dropping attributes did not cover child elements: <version><x>1</x></version>,
<p><x>1</x></p> or <p>none<x/></p> still reached the parsed report, and the
Elasticsearch and OpenSearch documents, as a dict, and the first such
document mapped the field as an object. _scalar() now makes each field the
schema types as a string a string (an element's own text) or None, applied
by _scalarize_report() to version, the report_metadata strings and every
policy_published child, and by _parse_report_record() to source_ip,
disposition, dkim, spf, the identifiers and every auth_results string. A
<count> with no text of its own takes the ValueError path of a non-numeric
count. _drop_xml_attributes() stays, so a field missed here still loses
its attribute keys.

The <email> dict unwrap is gone, since the field can no longer be a dict;
a text-only report_metadata still raises the AttributeError that reports
a missing section. Comments that described attribute dicts now say the
dict shape comes only from child elements. The namespace test, which the
new code could not fail, is replaced by one that detects RFC 9990 through
an attributed np field.

Tests walk the to_dict() of a mocked Elasticsearch and OpenSearch save and
allow lists and objects only in the known object fields, so a scalar field
added later is covered too.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record a closed stdout pipe as an output error and fix two comments

The standard-output guard in process_reports() caught only
UnicodeEncodeError, so a closed pipe (BrokenPipeError, an OSError) still
ended the run; it now records both as a "Standard output" output error.

_reason_text()'s docstring said a dict gives None because the element has
no text of its own, but mixed content (<type>forwarded<x/></type>) gives
the text beside the children; the docstring now says what the code does,
and a test pins it. The comment above the reason normalization no longer
describes attribute dicts, which no longer reach it.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test the normalization limit at its boundary and reword the index cap

The suite passed with the limit's <= changed to < and with
_days_touched()'s ceiling changed to a floor. New tests: 1,000 records
over 250 days (exactly MAX_NORMALIZED_RECORDS) are split; 683 records
over 366 days (249,978) are split and 684 (250,344) are not; 682 records
over 366 days starting 1 s after midnight touch 367 days (250,294) and
are not split. Each mutant was run against these tests and fails: the <
mutant fails the exact-limit test, the floor mutant the after-midnight
test.

The Elasticsearch and OpenSearch pages now say the index cap applies
when a report's records would use more than 31 dated indexes, which is
what it counts (with monthly_indexes the span limit keeps a report under
it).

Refs GHSA-p85v-578x-3wcf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Match the tests and changelog to mailsuite 2.3.5's final check

mailsuite 2.3.5's check_message_limits() now parses the message with
the email package's own feed parser under compat32 and counts the parts
it creates (each message/delivery-status block among them), their
nesting depth and its boundary tests, then limits header field length
and count on the parsed tree; the limit names changed. The over-limit
failure samples now trip max_parts, and the deeply nested poison message
max_depth; the tests assert those texts. The GHSA-qcc2 changelog bullet
describes the final check, the epilogue mail-parser parses again, and
the changes that affect only mailsuite's own parse_email().

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test a repeated policy_published and note mixed-content behaviour

No test failed when _scalarize_report() stopped coercing the fields of a
repeated <policy_published> (the list case): with that mutant, a report
with two policy_published elements and <p><x>1</x></p> puts a dict into
published_policy.p. The parse test and the Elasticsearch and OpenSearch
document-walk tests now include that report; each fails on the mutant.

The changelog says that an element with text and a stray child element
now yields its text (samples/aggregate/invalid_xml.xml gives header_from
"bad" where it gave ""). AGENTS.md and the _normalize_result_word() and
_scalarize_report() docstrings describe what runs now: attributes are
dropped as soon as the XML is parsed, and _scalar() coerces the string
fields, among them the RFC 7489 and RFC 9990 policy_published fields.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Coerce policy override reasons with the same rule as other strings

_reason_text() turned a repeated <type> or <comment> into the str() of
the list, while _scalar() turns every other repeated string field into
None. One class of input now has one rule: the reason's type and comment
go through _scalar(), so a repeated one is None ("none" in CSV), and
_reason_text() is removed. The test and the GHSA-4272 changelog bullet
say so.

Refs GHSA-4272-4r8g-6c5h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test an aggregate <error> with a child element

With attributes dropped and reasons coerced by _scalar(), only <error>,
which may repeat, still reaches _text() with a dict; its branch had no
test. An <error> with text beside a child element keeps the text, and
one with no text is dropped.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Describe mailsuite's six message limits in the changelog

mailsuite 2.3.5's check now counts every line of every header block
(folded lines and lines the parser keeps only as defects included) and
stops the parse early, and limits the total length of the address
headers mail-parser parses to 262,144 characters. The GHSA-qcc2 bullet
names the six limits.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render a report email's parts only where their text is read

parse_report_email() did str(payload_obj[0]) for every part walk()
visits, before the multipart/ continue. str() of a composite part
renders its whole subtree, and walk() visits every level, so a nesting
within mailsuite's limits rendered its leaf once per level: 98 nested
multiparts around a 20 MiB one-line leaf took 26.6 s, and a chain of 99
message/rfc822 parts around a 10 MiB leaf 11.3 s. A part is now rendered
only in the branches that read it; other message/* bodies are skipped,
since walk() visits their parts. Both shapes now take under 0.5 s, and
every sample under samples/failure/ parses to the same output.

A feedback report's sample is taken from the first sample part only: a
message/rfc822 nested inside the sample used to replace it, but a
feedback report has one sample (RFC 6591 section 3.1). Multipart nesting
alone no longer reaches the recursion limit, so the GHSA-qxwv test now
reaches the recursion guard through a deeply nested sample, which is
still rendered.

The changelog describes this and the address headers mailsuite 2.3.5
counts (From, Sender, Reply-To, To, Cc, Bcc, Delivered-To, in every
part; mail-parser parses all but Sender).

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Parse aggregate XML without attributes and lower the node limit

_drop_xml_attributes() rebuilt the parsed tree while the original was
alive, and the counting wrappers of _LimitedExpatParser referred to the
proxy, closing a reference cycle through the expat parser that kept
xmltodict's tree alive until the cyclic garbage collector ran. Both
xmltodict passes now use xml_attribs=False, so no attribute key is ever
built (expat still hands the attributes to the start handler, so they
are still counted: a test shows one element with 1,000 attributes is
1,001 nodes), _drop_xml_attributes() is removed, and the counts live in
an _XMLCounts object the wrappers share, so a parse leaves no cycle (a
test asserts gc.collect() finds nothing after one).

Peak extra memory for 2,000,000 distinct sibling elements: +999 MiB
before, +386 MiB after (master +391 MiB); for one element with 2,000,000
attributes: +1,274 MiB before, +445 MiB after (master +734 MiB).

At the old limit of 5,000,000 nodes a report could still need about 3 GB.
Measured after the change, the worst shape is distinct elements with
text behind a malformed XML declaration (the lxml recovery path): 559
bytes per node, +533 MiB for 999,000 nodes and +1,023 MiB for 2,000,000;
on the expat path about 220 bytes per node. MAX_XML_NODES is now
1,000,000, so a report at the limit needs at most about 550 MiB; the
largest sample has 32,019 nodes (909 KB), and the limit is a report of
about 28 MB. The constant's comment and the changelog state the bound.

AGENTS.md describes the attribute handling as it now is, and says that
<error>, which may repeat, goes through _text() and stays a list.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Qualify the mixed-content compatibility note to string fields

An element with text beside a stray child element yields its text only
for the fields the schema types as strings; a <begin> or <end> with mixed
content is still rejected, as before.

Refs GHSA-qpcg-29vr-jwm3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject over-long XML tags before any parser reads them

expat and libxml2 build a start tag's whole attribute list before they
report the element, so the node counter only sees the attributes after
they cost memory: one element with 4,999,000 attributes (56 MiB) took
5.25 s and +767 MiB before the node limit rejected it, and a single
element is bounded only by the 100 MiB decompression cap.

parse_aggregate_report_xml() now scans the decoded text once, before the
first parse and so before the lxml recovery path too, and rejects a
document in which any text between a "<" and the next ">" (or the end
of the text) is longer than MAX_XML_TAG_LENGTH (65,536 characters). The
scan uses no regex. It splits the text on ">" one 64 KiB chunk at a time,
carrying the unfinished piece from its last "<": a single split of the
whole text makes one string per tag, which peaked at +1,318 MiB (1.9 s)
for 100 MiB of "<a/>"; the chunked scan needs about 2 MiB and takes
0.6 s on that text, and a test bounds its memory and covers a tag across
a chunk boundary. The longest tag in the samples is 108 characters. The same shape
is now rejected in under 0.1 s, with a tracemalloc peak under 200 MB in the
test, behind a well-formed or a malformed declaration.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render a report email's feedback-report part only once

The message/* skip exempted message/feedback-report, which was rendered
with str() every time walk() reached one: a chain of 99 nested
message/feedback-report parts around a 10 MiB leaf passed mailsuite's
check in 0.02 s and then took 11.6 s. A feedback report has one
message/feedback-report part (RFC 6591 section 3), so, as with the
sample, only the first is read and rendered; the chain now takes 0.56 s.
The failure samples parse to byte-identical output.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound attribute counts by their = signs and cap reports at 32 MiB

XML allows a raw > inside a quoted attribute value, so attributes such as
x=">" cut a tag into short pieces that pass the tag scan: 4,999,000 of
them cost 5.61 s and +781 MiB, and 8,000,000 (98 MiB) 8.9 s and
+1,215 MiB, before the node limit fired. Every attribute needs an =, so
a document with more = characters than MAX_XML_NODES is now rejected in
one str.count() pass, before the tag scan; the samples hold at most 3.
The tag scan stays for long tags, and its docstring says it does not
bound attribute counts.

The stated memory bound was false: text per node is bounded only by the
size of the report, so at the node limit with 80 characters per element
(94 MiB) the recovery path reached +827 MiB. MAX_DECOMPRESSED_REPORT_SIZE
falls from 100 MiB to 32 MiB (the node limit already means a legitimate
report is at most about 28 MB, at the samples' 35 nodes per KB, and the
largest sample is 909 KB), and parse_aggregate_report_xml() holds an
uncompressed report to the same length, so the bound holds for every
input. Measured at the cap, 999,000 distinct elements with 14 characters
of text (31 MiB) need +587 MiB on the recovery path and +302 MiB on the
expat path; the constant comment and the changelog state it.

When the first parse fails, the exception's traceback kept the partial
xmltodict tree alive while the recovery parse ran; it is cleared first
(+603 to +587 MiB, and +263 to +241 MiB for 330,000 elements of 80
characters). A test shows the recovery path leaves the same 18 lxml
helper objects for a document ten times larger, so no tree survives a
parse there either.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Let the RFC 6591 feedback part win over legacy report text

The guard that reads only the first message/feedback-report part tested
feedback_report, which the legacy text/plain branch ("A message claiming
to be from you has failed ... detected.") also sets, so a legacy text
part followed by a real feedback-report part and sample now took the
legacy part and never read the ARF, where master read the ARF. The
guard now tests is_feedback_report, and reading the feedback-report part
discards a sample a legacy part set before it.

The same rule closes an older gap: a text/plain part inside the sample,
which is the spoofed message's own text, could hold the legacy phrase and
replace the report's Source-IP and its sample. Once a feedback-report
part is read, the legacy branch is skipped; a legacy report has no such
part. Tests cover both orders and the in-sample case; the failure
samples parse to byte-identical output.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hold every report input to the size cap and free failed parses

parse_report_file() kept the aggregate parser's exception, whose
traceback holds the parser's frames and the parsed tree, while it tried
SMTP TLS and then the email parser: a 31 MiB file at the cap cost
+1,046 MiB there against +587 MiB parsing it directly. It now keeps only
each failure's text and raises from a fresh exception of the relevant
format, so the chain survives without the tree. A file that begins with
"<" is not tried as JSON (json.loads() decoded it all) or as an email,
one that begins with "{" is not tried as an email (an RFC 822 message
begins with a header line), and text over the size cap is never handed
to the email parser, which has none: a 200 MiB <x/> file reached
+2,801 MiB in 4.6 s and now costs +232 MiB (reading the file) in 0.05 s.
extract_report() reads an uncompressed report at most one byte past the
cap instead of decoding it whole.

The cap counted characters for text and bytes for compressed reports,
so a non-ASCII report could be four times the cap in bytes: 999,000
elements of 14 emoji (31.2 Mi characters, 71 MiB of UTF-8) cost +649 MiB.
_exceeds_size_cap() measures text as UTF-8 (ASCII by length, other text
1 MiB of characters at a time, without a full copy), and bytes are
checked before they are decoded.

Clearing e.__traceback__ inside the except block did nothing on Python
3.10, which keeps the handled exception's traceback while the handler
runs. The lxml recovery now runs after the except block, in
_recover_xml(), so the first parse's frames and partial tree are
unreachable when it starts on every supported Python; a test holds a
weak reference to the first parse's counter and fails on the old code
under Python 3.10.19 and, with the clearing removed, under 3.14.

The changelog's examples are re-measured under the 32 MiB cap, and the
test docstrings cite the same figures.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip the parts inside a failure report's sample

Once the sample part was taken, walk() still visited its descendants,
and they reached the SMTP TLS and base64 attachment branches: a sample
whose spoofed message carried an SVG or zipped HTML attachment made the
email rejected ("Missing field: 'feedback'", "Unable to parse"), and a
gzipped aggregate report or a tlsrpt+json attachment inside it was
returned as that report instead of the failure report. The sample is
the reported message, so after it is taken every part inside it is
skipped (by the id() of each part its walk() yields). Tests cover SVG,
zipped HTML, JSON, gzipped aggregate XML, tlsrpt+json and PDF
attachments; the failure samples parse to byte-identical output.

The Bug fixes bullet also says that a legacy text part followed by a
feedback-report part with no sample after it is now invalid, with a
test.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cap SMTP TLS reports and test the format skips

parse_smtp_tls_report_json() decoded and json.loads()-ed a report of
any size; only the gzipped path was capped, and 60 MiB of "[]," cost
+1,565 MiB. It now rejects bytes or text over MAX_DECOMPRESSED_REPORT_SIZE
first, as InvalidSMTPTLSReport, so every report type is held to the cap.

parse_report_file() names the actual reason it skips the email parser
(the leading "<" or "{", or the size cap) rather than a combined one,
and a size message for bytes no longer says "of UTF-8". New tests kill
the mutants the review found surviving: removing the "<" skip for JSON,
narrowing the email skip to "<", the bytes check >= for >, and
removing the SMTP TLS cap.

The constant comment and the changelog state the memory at the cap as
measured on the current code: +345 MiB on the lxml recovery path and
+302 MiB on expat parsing the text directly, +500 MiB and +452 MiB
through parse_report_file(). The changelog says only the exception's
cause changed for files skipped by format, not the message.

Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip every sample candidate and correct the stated measurements

sample_parts held only the first sample candidate's parts, so a later
message/rfc822 (or one after a text/rfc822-headers sample) was dropped
as a sample but its parts were still walked, and a gzipped aggregate
report inside it was returned in place of the failure report. Every
candidate's parts are now skipped. Tests cover both shapes, and a test
that a gzipped aggregate report beside the sample is still returned
pins the rule (inside the sample: skipped; beside it: read) and fails
when the set is built from the whole message.

parse_report_file() no longer builds email reasons for content that
begins with "<" or "{", which its message never shows, and the size
reason reads "Invalid report email: the report is larger than 32 MiB".
The memory figures are re-measured in a fresh process with the input
read from disk before the baseline: +399 MiB on the recovery path and
+338 to +355 MiB on expat for the 31 MiB shape, +1,505 MiB for 60 MiB of
JSON before the SMTP TLS cap; the constant comment, the changelog and the
test docstrings state them. RFC 6591 section 3.1 is cited for the sample's
place, and application/json is named among the attachments that made a
failure report invalid.

Refs GHSA-qcc2-4x3v-v4g7
Refs GHSA-q7gm-8cj9-94hr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Scale the timed tests' bounds to the machine running them

The mailsuite fix's first CI run on GitHub's shared runners failed six
timed tests: the suite ran four to six times slower there than on a
workstation, and checks doing real work took longer than their fixed
bounds. Several bounds here have far less margin (0.2 s on a test that
takes 0.18 s whole), so they would fail the same way.

tests/timing.py adds budget(seconds), which measures a fixed reference
parse twice just before the timed call and multiplies the bound by the
slowdown against a reference machine; a faster machine keeps the written
bound. Every timed assert now compares against budget(<written seconds>)
computed before the timer starts. No written bound changed and no assert
was removed.

Refs GHSA-qcc2-4x3v-v4g7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-11-09 14:10:30 -05:00
Sean WhalenandClaude Opus 5.5 56e5a09923 Add opt-in Elasticsearch/OpenSearch data streams (#937) 
* Add opt-in Elasticsearch/OpenSearch data streams (#842)

New use_data_streams option in [elasticsearch] and [opensearch] saves
each report type to a data stream named like today's index without the
date suffix, instead of daily or monthly indexes. parsedmarc puts an
exact-name index template (priority 501, @timestamp mapped as date,
<stream>@custom component template on Elasticsearch) and creates the
stream; retention is left to operator-managed ILM/ISM policies, with
documented examples. Dated indexes are untouched and still searched for
duplicates and matched by the dashboards. Every document now carries
@timestamp and is saved with op_type=create in both modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix data stream prose found by cross-check review (layer 1)

Aggregate @timestamp is each record's begin date, not the report's; the
cluster only rolls backing indices over under an attached policy; move
the Elasticsearch 8.7 note after the template it refers to; clarify the
name-collision paragraph, OpenSearch-side comments, the migrate_indexes
docstring, and the test-module headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tighten names and wiring tests found by cross-check review (layer 2)

Split the use_data_streams CLI passthrough test per backend so a call
site reading the other section's flag fails; name the dated-mode tests
for both changes they assert; compute index_date only in dated mode in
every save function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Align data stream docs with Elastic/OpenSearch references (layer 3)

Say the stream must exist before reindexing into it (ES reindex API:
auto-creation needs a data-stream template); an existing stream's write
index needs the ILM policy attached directly since nothing rolls it over
otherwise; target the stream name in the _settings example; note
OpenSearch's leading-dot rule for stream names and Fleet's template
priorities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cover the Loki client-initialization error wrapper

_parse_config accepts a [loki] token alongside username/password; only
the LokiClient constructor rejects the combination. Exercise that path
through _init_output_clients with the real constructor so the
RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-31 15:37:37 -04:00
Sean WhalenClaude Fable 5Copilot
 		07bca1ad28 Make the output and mailbox integrations optional extras (#888) 
* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-08-28 17:33:18 -04:00
Sean WhalenandClaude Opus 5 b304cf8639 Cover index_prefix_domain_map and index_suffix in index migrations (#868) (#869) 
* Cover index_prefix_domain_map and index_suffix in index migrations (#868)

The startup Elasticsearch/OpenSearch index migrations built their target
index names from the [elasticsearch]/[opensearch] index_prefix and
index_suffix options alone, while the save path also honors
general.index_prefix_domain_map. A multi-tenant deployment therefore ran
the backfill guard `count` against dmarc_aggregate*/smtp_tls*, patterns
matching none of its real <tenant>_dmarc_aggregate-* indexes. The query
passes allow_no_indices=True, so a zero-match wildcard returns count 0 --
indistinguishable from "already backfilled" -- and the backfill was
skipped silently, with no log line at any level.

Resolve migration index names through a new _migration_index_names()
helper that widens both configurable axes: one name per tenant prefix in
the map plus the unprefixed name (unmapped domains are still saved
unprefixed), and, when an index_suffix is set, the unsuffixed name
alongside the suffixed one so history predating the suffix is covered. A
configured index_prefix still wins outright and suppresses the map
fan-out, matching save-time precedence. The key normalization is now
shared with get_index_prefix() via _normalize_index_prefix(), so the
names parsedmarc migrates cannot drift from the ones it writes. The
resolved lists are logged at DEBUG, and the SIGHUP reload path passes the
freshly parsed map, so a newly onboarded tenant is covered without a
restart.

Also repair the legacy published_policy.fo migration, which has been
unable to complete since mapping types were removed in Elasticsearch 7
(and never existed in OpenSearch): it read the field mapping in the
type-keyed response shape, so the check always fell through, and its
put_mapping() call passed a doc_type argument neither current client
accepts. It now reads either response shape, uses each client's current
signature, and takes its index names in a separate legacy_fo_indexes
argument -- exact names, since 5.0.0 introduced date-suffixed index names
in the same release that fixed the fo declaration, but prefixed and
suffixed where configured, since both options date back to 4.1.0. Tenant
prefixes are excluded: index_prefix_domain_map arrived in 8.19.0, and
this migration renames the index it rebuilds. The Elasticsearch copy,
removed as unreachable during the #806 client migration, is restored now
that the cause is understood.

Finally, reject an index_prefix_domain_map YAML file that is not a
mapping of string tenant names to lists of domains, instead of raising
mid-save on a non-string key or silently matching the wrong domains on a
scalar value -- `in` on a str is a substring test, so "example.co"
matches "example.com"
(https://docs.python.org/3/reference/expressions.html#membership-test-operations).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix OpenSearch capitalization and spacing in the usage docs

Copilot review of #869: the `index_prefix_domain_map` option line spelled
"OpenSearch" as "Opensearch" and had a doubled space before the type. Both
predate this branch but sit inside a hunk it rewrites. Also wrapped the line
to match the continuation-indent style of every other option in the list, and
corrected the same misspelling in the multi-tenant section a few lines above
the paragraph this branch added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Require index_prefix_domain_map domain lists to hold strings

Copilot review of #869: the shape check verified that each value was a
list but not what the list contained, so `tenant_a: [42]` passed and then
never compared equal to any domain the save path looks up -- the same
silent-misbehavior class the check exists to reject, and a contradiction
of the "any other shape is rejected at startup" claim in the docs.

Check the list's items too, and reword the error message, comment,
CHANGELOG and docs to state the rule the check now enforces: a mapping of
tenant names to lists of domain names, all strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make the legacy fo migration retry-safe, and nest its mapping body

Copilot review of #869, verified against live Elasticsearch 8.19 and
OpenSearch 3 containers rather than mocks.

Retry safety (a real defect): an attempt interrupted between creating the
-v2 index and deleting the original left debris that made create() fail
with "resource already exists" on every later startup, inside the same try
that swallows the error -- so the index was never migrated and the debris
document survived. Reproduced on a live cluster. Discard a leftover target
first; that is safe precisely because reaching this point means the
original still holds the data, since it is deleted only once the reindex
has succeeded.

Mapping body: the reviewer's concern that a dotted key under `properties`
risks a runtime failure does not hold -- both clusters accept it and
produce a byte-identical mapping, with the dot expanded into
published_policy -> properties -> fo. Switch to the nested object form
anyway, since dot expansion is conditional on the object's `subobjects`
setting and this shape never is, and derive the object/leaf names from the
dotted constant so the write cannot drift from the field the read looks up.

Both fo-migration suites now build per-name Index mocks. A single shared
mock cannot express "the original exists but its migration target does
not", which is the ordinary case and the one the retry fix turns on; the
happy path now also asserts the target index is never deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-07-27 20:48:00 -04:00
Sean WhalenandClaude Fable 5 23c5ea9ad6 Fix DKIM/SPF and SMTP TLS detail-table cross-products in dashboards (#169) (#839) 
* Fix DKIM/SPF alignment detail cross-product in dashboards (#169)

Elasticsearch and OpenSearch dynamic-map the dkim_results/spf_results
object arrays as `object` (create_indexes never registers the DSL
document mappings), so Lucene flattens each array into independent
multi-valued fields and stacked terms aggregations on
dkim_results.selector/.domain/.result return every combination of
values across a report's signatures — each phantom row repeating the
full message count.

Aggregate documents now also carry dkim_results_combined and
spf_results_combined: one "selector / domain / result"
("scope / domain / result") string per auth result, composed in
add_dkim_result/add_spf_result. The Kibana/OpenSearch Dashboards and
Grafana (Elasticsearch) alignment-detail tables aggregate those
instead, and the Splunk detail panels pair the values with
mvzip/mvexpand. A documented idempotent _update_by_query backfills
documents saved by older versions; the query matches only documents
that have auth results and lack the combined fields, because an
`exists` query cannot see an empty array.

Also corrects the dead _SPFResult.results (plural) declaration to
`result` (the save path always wrote the singular key), fixes the
result parameter annotations on add_dkim_result/add_spf_result, and
removes the Grafana dmarcian.com DKIM-checker data link, which
required the separate domain/selector columns.

The SMTP TLS visualizations have the same class of defect and are
tracked separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review findings on #839

Reword the combined-field regression test docstrings: the DKIM/SPF
auth results are dynamic-mapped as plain `object`, not the `nested`
mapping type the previous wording implied — the distinction is the
crux of the fix. Also drop the inert renameByName entries Copilot
flagged on the Grafana Overview and DKIM Alignment Details panels,
which referenced fields those panels' queries no longer (or never)
produced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add PR #839 review lessons to AGENTS.md

Extend the "Review passes cover prose" section with two rules from
the #839 Copilot findings: docstrings/comments get the same
text-level review pass as docs and dashboard labels (with suspicion
for dual-use terms like "nested" near Elasticsearch code), and
inert config entries inside hunks a PR already rewrites should be
cleaned rather than preserved to minimize the diff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Reflow create_indexes comments flagged by Copilot

The line wrap placed "#169" directly after the comment marker, so the
raw source read "# #169;". Reword so the issue reference stays on one
line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Extend the hunk-proofreading rule with rendered-text wraps

Fold the PR #839 second-round Copilot lesson into the existing rule:
proofread how wrapped lines render (comment markers, punctuation at
wrap points), not just the wording itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Rename Overview combined-result column labels (Copilot round 3)

The Overview table's "DKIM Auth Result" / "SPF Auth Result" labels
were kept when the columns switched to the combined
"selector / domain / result" values, leaving the headers misleading.
Rename them to match the detail panels' convention and retarget the
byName width overrides that matched the old labels, widening them for
the longer values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Document per-signature row semantics in the alignment tables

A message carrying multiple DKIM signatures appears once per signature
in the details tables, so summing the messages column across rows can
exceed the total message count. State that explicitly rather than
leaving readers to infer it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot round-4 findings on dashboards

Fix three pre-existing saved-object title typos in the OpenSearch
ndjson (leading space on "Aggregate DMARC passed DMARC", trailing
space on "Aggregate DMARC reporting organizations", double space in
"map  of message sources by country"), in both the top-level title
and the embedded visState title.

Normalize the Splunk DKIM details placeholders: the base search's
fillnull renders wholly-missing DKIM fields as the literal string
"null", so unsigned mail showed "null / null / null" while the SPF
panel shows "none". Rewrite the values to "none" after the signature
split, where the fields are single-valued and the mvzip pairing
cannot be disturbed. Verified against the dev Splunk that no
truncation or mis-pairing occurs either way, since fillnull
guarantees the fields are never actually null.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Backfill combined DKIM/SPF fields automatically at startup

migrate_indexes() now backfills dkim_results_combined and
spf_results_combined on aggregate documents saved by older versions,
so ES/OS users get historical data in the reworked alignment tables
without running the documented _update_by_query by hand. The backfill
is submitted as a non-blocking background task
(wait_for_completion=false, conflicts=proceed) guarded by a cheap
count query, making repeated startups a fast no-op once an index is
backfilled; any cluster error is logged as a warning and retried at
the next startup rather than raised. The manual command remains
documented for users who upgrade dashboards without pointing the new
parsedmarc at the cluster or who want to control write-load timing.

The legacy published_policy.fo long-to-text reindex migration in the
OpenSearch module is kept ahead of the new backfill, for clusters
upgraded from very old data.

Verified end-to-end against the live dev environment: a real CLI
startup backfilled 9 stripped OpenSearch documents (logged with task
ID) while the already-backfilled Elasticsearch side stayed silent,
and a second startup was silent on both engines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Match backfill guard on either domain or result subfield

End-to-end upgrade testing (real parsedmarc 10.2.4 ingest, then a
branch startup) surfaced that an exists query cannot see an empty
string: a text field with no tokens is invisible to exists. The
parsers we audited never store an auth result with an empty or
missing domain — they drop such entries entirely, so the previous
domain-only guard was sufficient for their data — but the storage
shape of every historical parsedmarc version can't be audited, so
the guard (and the documented manual command) now matches either
the domain or the result subfield per protocol. Matching either
costs nothing and cannot skip a document that has something to
backfill.

Verified by recomputing expected combined values from _source for
all 2,299 documents on both engines: every document with stored
auth results has exactly the recomputed pairs, and the legacy fo
migration correctly did not fire on typeless indexes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add auth-result filter controls to the Kibana/OSD aggregate dashboard

The combined per-signature columns fixed the #169 cross-product but left
no way to click-filter by an individual selector, domain, or result. Add
an "Aggregate DMARC auth result filters" input_control_vis panel above
the SPF/DKIM details tables with six option-list dropdowns (DKIM
selector/domain/result, SPF scope/domain/result) that emit ordinary
dashboard-wide filter pills. Works on both Kibana 8.19 and OpenSearch
Dashboards 3, verified by driving the controls in both UIs against the
issue's two-signature repro report.

Documented in kibana.md, including the flat-mapping caveat: combining
two component filters matches documents where any signature satisfies
each condition individually.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Scale Grafana source-country map markers with message volume

The "Map of Message Source Countries" panel drew fixed 5 px dark-green
markers at 50% opacity — nearly invisible on the dark basemap, so the
panel read as empty even when data was flowing (verified via the query
API). Markers now scale with Sum(message_count) (min 4, max 30 px) at
0.8 opacity in a higher-contrast green. Pre-existing issue; the
identically-styled failure-dashboard map panel is intentionally left
untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Correct the nested-mapping rationale in the create_indexes comments

The comments claimed Kibana/OSD/Grafana "cannot terms-aggregate fields
inside a nested mapping" — too absolute. Fact-checked empirically and
against primary docs: Kibana/OSD visual editors (Lens and classic
Visualize) do not support nested fields, but Vega panels can run nested
aggregations (they just cannot render tables, per Elastic's docs), and
Grafana >= 9.4 has a nested bucket aggregation (grafana/grafana#62301)
but no reverse_nested, so parent-level metrics like Sum(message_count)
return 0 inside per-signature buckets (reproduced live). Conclusion
unchanged: the dynamic object mapping stays load-bearing for the
shipped dashboards. PR #839's body was updated with the same
correction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Pin ruff exactly, matching the existing pyright pin rationale

CI installs the [build] extra fresh on every run, and ruff was the one
lint tool left unpinned. ruff 0.16.0 (released this week) began
flagging this codebase's str.format() house style, so every PR started
failing lint on lines it never touched. Pin to 0.15.21 — the version
the codebase is clean under — with the same bump-deliberately comment
pyright carries. Upgrading to 0.16 and converting to f-strings can be
its own PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot findings: harden migrate_indexes, normalize panel titles

Three unresolved review threads, all verified against cli.py's
re-raising init handler before fixing:

- elastic.py/opensearch.py: connections.get_connection() sat outside
  migrate_indexes()'s try/except, so a connection-registration failure
  would abort startup despite the docstring's promise that migration
  errors are caught and logged. Now caught, logged, and skipped until
  the next startup.
- opensearch.py: the legacy published_policy.fo migration loop did
  unguarded network I/O (exists/get_field_mapping/reindex/delete), so a
  transient cluster error aborted startup on the OpenSearch path while
  the identical situation on the Elasticsearch path was logged and
  survived. Each index's migration attempt is now wrapped, warns, and
  moves on.
- opensearch_dashboards.ndjson: normalized two pre-existing panel
  titles in the aggregate dashboard's panelsJSON ("Reporting
  organizations " trailing space, "Map  of message sources by country"
  double space).

Regression tests assert migrate_indexes never propagates connection or
per-index cluster errors on either backend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Clarify that Nested() on auth-result fields is in-memory shape only

Copilot flagged that _AggregateReportDoc declares dkim_results and
spf_results with Nested(...) while the create_indexes comment insists
the stored mapping must stay dynamic `object`. Both are true: the
Nested declaration only shapes the DSL's in-memory document building
and is never installed as a mapping, because create_indexes skips
Index.document() registration. Say so at both sites, in both backends,
so nobody "fixes" the mismatch by registering the mapping — which
would install real nested mappings and blank the dashboards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Refer to the filter panel by its displayed title in docs and CHANGELOG

The dashboard convention is a short panel display title backed by a
long-form saved-object name ("SPF details" / "Aggregate DMARC SPF
details"), and the new controls panel follows it. The docs and
CHANGELOG named the panel by its saved-object title, which is not what
a user sees on the dashboard; use the displayed "Auth result filters"
instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix misspelled column label in the failure email samples table

The "DMARC failure email samples" visualization labeled its
authentication_results column "autentication_results". The underlying
field reference was already correct; only the user-facing customLabel
was misspelled. A sweep of every title and customLabel in the ndjson
found no other misspellings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Extend the combined-field fix to SMTP TLS documents

SMTP TLS reports have the same cross-product defect as the DKIM/SPF
alignment tables (issue #169), one level deeper: policies is an object
array and each policy's failure_details is an object array inside it,
so stacked terms aggregations on their subfields fabricate rows.

Documents now also carry policies_combined ("domain / type" per policy)
and failure_details_combined ("domain / type / result / sending mta /
receiving ip / mx" per failure detail), composed at save time with the
same "none" fallbacks as the aggregate fields. migrate_indexes() gains
smtp_tls_indexes and backfills old documents with the same guarded,
non-blocking update_by_query pattern; cli.py wires the index name in on
both backends, and the manual _update_by_query command is documented.

Also fixes two adjacent dead fields: add_failure_details stored
additional_information_uri under the wrong constructor kwarg
(additional_information), and receiving_mx_hostname had no declaration
despite always being stored.

Verified live on ES 8.19 and OpenSearch 3: a two-policy repro report
yields exactly 2 policy rows and 2 failure-detail rows via the combined
fields where the old stacked aggregations return 4 of each; the startup
backfill converted the 4 pre-existing sample documents on both engines
with zero recompute mismatches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Rework the SMTP TLS dashboards onto the combined fields

Kibana/OSD: "SMTP TLS domains" replaces its stacked policy_domain ×
policy_type terms with one terms agg on policies_combined.keyword;
"SMTP TLS failure details" replaces six stacked terms spanning both
array levels with one on failure_details_combined.keyword; the
smtp_tls* index-pattern field cache gains the new fields. The
"reporting organizations" table only buckets on doc-level org_name and
needed no change.

Splunk: the base search now expands policies at the JSON level (spath +
mvexpand) so policy fields are scalars per event, and the failure
details panel expands the second level the same way — sums are the
detail's own failed_session_count, correctly paired. Verified via the
search REST API: a two-policy repro returns exactly one row per real
failure detail with per-detail counts.

kibana.md documents the per-policy/per-detail row semantics and the
honest caveat that session-count sums remain per report document.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Persist additional_info_uri from parsed SMTP TLS failure details

Copilot caught that the savers read additional_information_uri from
the parsed failure-detail dict, but the parser's key is
additional_info_uri (SMTPTLSFailureDetailsOptional in types.py, set in
parse_smtp_tls_report_json), so the URI was never persisted — the
read-side half of the dead-field bug whose write-side half (wrong
constructor kwarg) was fixed earlier. Read the parser's key first,
keeping the long-form key as a fallback for dicts built by other
callers. Regression test proven to fail on the unfixed savers.

Also restructured the expected combined-string test values into named
locals so no implicit string concatenation sits inside a list literal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Say "inner doc", not "nested doc", in the singular-key test docstrings

Final review sweep: in this codebase "nested" is reserved for the
Elasticsearch mapping type, and these InnerDoc-serialization
docstrings used it colloquially — the same dual-use-term trap
documented in AGENTS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-25 12:29:40 -04:00
Sean WhalenandClaude Fable 5 40509f801b Migrate Elasticsearch output to the elasticsearch-py 8.x client (#822) 
* Migrate Elasticsearch output to the elasticsearch-py 8.x client (#806)

The mandatory elasticsearch<7.14.0 + elasticsearch-dsl==7.4.0 pins
transitively forced urllib3<2 (EOL 1.26.x) onto every install. The old
<7.14.0 cap only existed to dodge the client product check that broke
OpenSearch users (#452, #653) — obsolete now that parsedmarc has a
dedicated [opensearch] backend on opensearch-py.

- Depend on elasticsearch>=8.18,<9 and drop elasticsearch-dsl entirely
  (the DSL ships inside the client as elasticsearch.dsl since 8.18.0).
  The 8.x client's elastic-transport allows urllib3>=1.26.2,<3, so
  installs can now resolve urllib3 2.x. The 8.x line supports both
  Elasticsearch 8.x and 9.x servers; ES 7.x servers are no longer
  supported, and OpenSearch users pointing [elasticsearch] at an
  OpenSearch cluster must switch to the [opensearch] section.
- set_hosts() now builds 8.x connection kwargs (scheme-qualified host
  URLs, request_timeout, basic_auth) while keeping the function
  signature and every INI option unchanged.
- migrate_indexes() is now a documented no-op kept for API
  compatibility: its only migration (re-typing published_policy.fo
  from long to text) applied exclusively to indices carrying the
  legacy ES 6-era "doc" mapping type, which cannot exist on any
  server the 8.x client can reach.
- The elasticsearch.dsl 8.x stubs use dataclass_transform and don't
  surface pre-8.x-style bare `name = Text()` fields as constructor
  parameters; each Document/InnerDoc class now carries a
  TYPE_CHECKING-only `__init__(*args, **kwargs)` declaration matching
  the real runtime signature, which also made nine pre-existing
  pyright ignores unnecessary.

Verified with ruff, pyright (0 errors/0 warnings), the full pytest
suite (718 passed), and a CLI run over the bundled samples; CI's live
elasticsearch:8.19.7 service exercises the new client end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Use pass instead of ... in TYPE_CHECKING __init__ stubs

CodeQL flags an ellipsis-only body as "Statement has no effect" (12
alerts on PR #822); pass is equivalent at runtime and to the type
checker and keeps the alerts from resurfacing on every future scan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-13 13:50:25 -04:00
Sean WhalenandClaude Opus 4.7 411f5a8886 chore: tidy cSpell config; fix two doc typos (#779) 
- Ignore data/export trees via cSpell.ignorePaths: parsedmarc/resources/**
  (maps tooling holds thousands of intentional foreign-language classifier
  keywords + bundled data), plus samples/** and dashboards/** (report
  samples and dashboard exports). These are data, not whitelist vocabulary,
  so excluding them keeps the editor quiet without bloating the word list.
- Add the remaining genuine false-positives across code, docs, CI
  workflows, and editor config to cSpell.words (technical terms, library
  names, SQL/identifier tokens, brand/operator and multilingual examples
  from AGENTS.md, plus charliermarsh/junitxml/mktemp/pipefail/seanthegeek).
- Fix two genuine typos found while triaging rather than whitelisting
  them: "maidir" -> "maildir" and "connexion" -> "connection".

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 13:52:52 -04:00
CopilotClaude Opus 4.7Sean Whalen
 		ae1e5adb66 Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659) 
* Add DMARCbis report support; rename forensic→failure project-wide

Rebased on top of master @ 2cda5bf (9.9.0), which added the ASN
source attribution work (#712, #713, #714, #715). Individual Copilot
iteration commits squashed into this single commit — the per-commit
history on the feature branch was iterative (add tests, fix lint,
move field, revert, etc.) and not worth preserving; GitHub squash-
merges PRs anyway.

New fields from the DMARCbis XSD, plumbed through types, parsing, CSV
output, and the Elasticsearch / OpenSearch mappings:

- ``np`` — non-existent subdomain policy (``none`` / ``quarantine`` /
  ``reject``)
- ``testing`` — testing mode flag (``n`` / ``y``), replaces RFC 7489
  ``pct``
- ``discovery_method`` — policy discovery method (``psl`` /
  ``treewalk``)
- ``generator`` — report generator software identifier (metadata)
- ``human_result`` — optional descriptive text on DKIM / SPF results

RFC 7489 reports parse with ``None`` for DMARCbis-only fields.

Forensic reports have been renamed to failure reports throughout the
project to reflect the proper naming since RFC 7489.

- Core: ``types.py``, ``__init__.py`` — ``ForensicReport`` →
  ``FailureReport``, ``parse_forensic_report`` →
  ``parse_failure_report``, report type ``"failure"``.
- Output modules: ``elastic.py``, ``opensearch.py``, ``splunk.py``,
  ``kafkaclient.py``, ``syslog.py``, ``gelf.py``, ``webhook.py``,
  ``loganalytics.py``, ``s3.py``.
- CLI: ``cli.py`` — args, config keys, index names
  (``dmarc_failure``).
- Docs + dashboards: all markdown, Grafana JSON, Kibana NDJSON,
  Splunk XML.

Backward compatibility preserved: old function / type names remain as
aliases (``parse_forensic_report = parse_failure_report``,
``ForensicReport = FailureReport``, etc.), CLI accepts both the old
(``save_forensic``, ``forensic_topic``) and new (``save_failure``,
``failure_topic``) config keys, and updated dashboards query both
old and new index / sourcetype names so data from before and after
the rename appears together.

Merge conflicts resolved in ``parsedmarc/constants.py`` (took bis's
10.0.0 bump), ``parsedmarc/__init__.py`` (combined bis's "failure"
wording with master's IPinfo MMDB mention), ``parsedmarc/elastic.py``
and ``parsedmarc/opensearch.py`` (kept master's ``source_asn`` /
``source_asn_name`` / ``source_asn_domain`` on the failure doc path
while renaming ``forensic_report`` → ``failure_report``), and
``CHANGELOG.md`` (10.0.0 entry now sits above the 9.9.0 entry).

All 324 tests pass; ``ruff check`` / ``ruff format --check`` clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Apply post-RFC review fixes: RFC 9990 detection, langAttrString, CFWS-aware RUF parsing

Aligns the implementation with the final RFCs (9989/9990/9991) instead of
inferring DMARCbis support from the version element or the namespace alone.

Aggregate parsing (RFC 9990):
- _text() helper unwraps langAttrString values (extra_contact_info, error,
  comment, human_result, generator) — when reporters include the lang
  attribute, xmltodict yields {"#text": ..., "@lang": ...} dicts instead
  of strings; the parser now stores the text payload in both shapes.
- New xml_namespace field on AggregateReport records the declared XML
  namespace (urn:ietf:params:xml:ns:dmarc-2.0 for RFC 9990 reports).
- RFC 9990 detection accepts namespaceless reports that follow the
  RFC 9990 shape (presence of np / testing / discovery_method / generator),
  so reporters that don't declare the namespace still receive RFC 9990-
  aware validation.
- Warnings: missing DKIM <selector> (REQUIRED in RFC 9990); legacy
  forwarded / sampled_out policy-override types (removed by RFC 9990);
  unknown policy-override types per the RFC 9990 enumeration.
- xml_namespace added to Elasticsearch and OpenSearch document mappings.

Failure parsing (RFC 9991):
- Identity-Alignment and Auth-Failure are split on commas with CFWS
  whitespace stripped per the RFC 9991 ABNF; previously "dkim, spf"
  yielded ["dkim", " spf"] with a leading space on the second token.
- Warnings logged when either REQUIRED field is missing.

Terminology: every reference to "DMARCbis" in code, tests, sample
filenames, AGENTS.md, and CHANGELOG.md is replaced with the appropriate
RFC number (9989 for the policy spec, 9990 for aggregate reports, 9991
for failure reports). Sample contents are unchanged.

Docs: corrects the prior claim that fo was dropped from RFC 9990 (only
pct was), reframes testing as a new field (not a pct replacement, since
RFC 9989 Appendix A.6 removed pct with no per-message substitute), and
documents the policy_override_reason enum changes (added policy_test_mode;
removed forwarded / sampled_out).

Tests: 8 new tests covering xml_namespace capture, RFC 9990 detection
from field shape, missing-DKIM-selector warning, legacy-override-type
warning, langAttrString unwrapping across all four affected elements,
and CFWS-aware Identity-Alignment / Auth-Failure parsing plus their
missing-field warnings. 276 tests total, all passing; ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:51:08 -04:00
Sean WhalenClaude Opus 4.7Sean Whalen
 		4e8c28bbc0 Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737) 
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth

OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.

Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.

Net effects for Kibana users on import:

- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
  the choropleth now sum(message_count) instead of counting OS docs,
  giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
  name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
  intro + samples table) — drops the Kibana-only IP-address and
  country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
  Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
  visualization that snuck in via a community contribution.

Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file

Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.

Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 01:30:48 -04:00
Sean WhalenClaude Opus 4.7Sean Whalen
 		826e78c390 Fix DMARC dashboard metrics (OSD + Splunk) and add dashboard-dev bootstrap (#736) 
* OSD: fix aggregate dashboard metrics to sum(message_count)

13 panels on the DMARC aggregate dashboard were aggregating with `count`
(number of OSD docs) when they should have been summing `message_count`.
Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from
the XML and carries an integer message_count, so doc-counting reports
"distinct sources" rather than "messages". Panels with titles like "Message
volume by header from", "DMARC passage over time", etc. were producing
misleading numbers.

Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by
reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and
DKIM details. (DMARC failure email samples kept count — one OSD doc per
RUF sample, so it's correct. SMTP TLS panels untouched — they sum the
right session-count fields.)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Splunk: align dashboards with OSD and fix query bugs

Aggregate dashboard:
- Add "Message sources by Autonomous System" panel (source_asn / as_name /
  as_domain), formatted "AS<n>" at render with eval, matching the OSD addition.
- DKIM details: add the missing dkim_aligned column.
- SPF details: reorder columns to OSD order (spf_aligned at end).
- Map / country titles renamed to match OSD ("Map of message sources by
  country", "Message sources by country").
- Map widget: stats count by Country -> stats sum(message_count) by
  Country, so the choropleth shades by message volume not record count.
- fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain,
  source_country to mirror OSD's missing-bucket labels.
- charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC
  pies and the DMARC-passage timechart.

Forensic dashboard:
- Restructure to match OSD's two-panel layout (markdown + samples table).
- Drop the country map / IP table / country-ISO table panels (not in OSD).
- Samples table columns aligned to OSD: arrival_date_utc, source.ip_address,
  from, subject, reply_to, authentication_results.
- Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=*
  required field to exist; LinkedIn RUF sample with null From was filtered out).

SMTP TLS dashboard:
- Reorder metrics to OSD order (successful before failed).
- Domains panel: add policy_type bucket.
- Failure details: replace search-time `failed_session_count>0` (which
  doesn't evaluate against multivalued JSON paths in Splunk) with
  `result_type=*` for presence + post-stats `where failed_sessions>0`.
  Drop _time/successful_sessions columns; reorder to match OSD.
- Wire the existing policy_type input into all three searches.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add dashboard-dev bootstrap script and VSCode task

dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml,
seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini,
and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana,
and Splunk. Idempotent: existence checks skip provisioning that's already
done; only the dashboard imports re-run unconditionally on every invocation
(that's the point of running it after a dashboard edit).

Notable provisioning quirks the script handles:
- Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships
  with indexes=[] and index=default; rewrites it to allow the email index.
- ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily
  parsedmarc indexes via _cat/indices and deletes one at a time.
- Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1
  deletes and recreates the email index (then re-applies the HEC token).
- OSD security plugin tenants: imports target global_tenant explicitly
  via the securitytenant header so they're visible to the shared workspace
  rather than landing in the API user's private tenant. Override with
  OSD_TENANT=<name>.
- Splunk ships an in-product announcement view (scheduled_export_dashboard)
  with sharing=global; the script narrows it to sharing=app so it stops
  showing up in every app's dashboards list.

Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs
the script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Bump version to 9.10.3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* CHANGELOG: warn against the "Create new objects with unique IDs" import mode

OSD's import dialog has two modes: the default "Check for existing objects"
(which honors saved-object IDs and overwrites in place when "Automatically
overwrite conflicts" is on) and "Create new objects with unique IDs" (which
imports under fresh UUIDs and leaves the buggy originals untouched). Picking
the second one means the dashboards keep rendering the wrong numbers because
the originals are never replaced. Spell that out so users don't fall into
the trap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* OSD: label the metric column "messages" instead of "Sum of message_count"

OSD's table column header defaults to "Sum of message_count" when the
metric agg has no customLabel. "messages" reads better and matches what
the panels are actually counting.

Applies to all 15 aggregate-DMARC visualizations that use sum(message_count).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable

Trim the verbose technical exposition; lead each fix with the user-visible
symptom. Move the action-required call out to its own header in upgrade
notes so the re-import instructions don't get lost in a wall of text.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Move per-tool dashboard exports under a single dashboards/ directory

Consolidates the four sibling top-level folders (kibana/, opensearch/,
grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/.
Updates the only path references in tracked files: bootstrap script (5
lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in
docs/source/elasticsearch.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* OSD: restore the "DKIM alignment" panel title on the aggregate dashboard

The DKIM alignment panel had no title override in panelsJSON, so OSD fell
back to the visualization's own name ("Aggregate DMARC DKIM alignment").
Every other pie/table on the same dashboard sets a clean title (SPF
alignment, Passed DMARC, etc.) — this was a stray regression. Set the
panel title to "DKIM alignment" to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Splunk: color the message-disposition timechart by severity

Reject is red, quarantine is yellow, none is green — same semantic
mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage
timechart, applied via charting.fieldColors. Matches OSD's existing
color overrides on the equivalent viz.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* CHANGELOG: clarify that "Create new objects with unique IDs" is the default

The OSD import dialog defaults to that mode — users have to actively
switch away from it, not just avoid picking it. Reword the upgrade note
to lead with the switch and explain why the default would silently
preserve the bug.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 00:40:01 -04:00
Anael Mobilia 3a1360a47a Always use systemctl command (#482) 
In order to align with the rest of the documentation
 2024-03-17 11:44:55 -04:00
Anael MobiliaandAnael Mobilia 3bf0bea710 Fix typo on documentation (#443) 
* Doc cleanup/improvement

* Fix typos

---------

Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-12-16 10:08:06 -05:00
Anael MobiliaandAnael Mobilia 732547e622 Doc cleanup/improvement (#432) 
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-10-11 17:24:50 -04:00
Anael MobiliaandAnael Mobilia bf69ea8ccc Fix typos (#413) 
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-05-14 18:07:07 -04:00
Anael Mobilia 062d6ea821 Use a local network connection to elasticsearch (#407) 2023-05-09 08:48:17 -04:00
Sean Whalen cd475255c5 Documentation cleanup 2023-05-03 16:44:15 -04:00
Ben CompanjenandSean Whalen 2b35b785c6 Split and Organise documentation files (#404) 
* Set global TOC collapse to false

* Split documentation

I tried to split the index.md file into logical parts, not changing the contents.
I did add a space and change one HTTP URL to HTTPS.

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2023-05-03 16:11:58 -04:00