mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-11 11:21:32 +00:00
e53b0f7f76295ce00aece47eb8631ec48bc64e1e
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
2ee209005b |
Merge commit from fork
* chore: update IPinfo Lite MMDB (#941) Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Cap report date range and indexes created per report A report whose <date_range> spans millions of days was normalized into one record per day, exhausting memory, and with Elasticsearch or OpenSearch output created one dated index (one shard) per day, exhausting the cluster's shard budget. Reject a range longer than MAX_REPORT_SPAN_DAYS (366) or negative with InvalidAggregateReport. In both save functions, once a report has used MAX_INDEXES_PER_REPORT (31) distinct dated indexes, route further records to the report's first index, keeping their own dates, and log one warning. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Catch output-stage exceptions and validate SMTP TLS report types One report sent to a monitored mailbox could make a configured output raise an exception that process_reports() did not catch. It escaped through the mailbox save callback, _main logged "Mailbox Error" and exited 1, and the batch stayed in the mailbox, so every one-shot run died on the same message and duplicated the legitimate reports in non-deduplicating outputs. Three sender-controlled inputs did it: - A failure report whose sample has an empty From:, To: or Reply-To: (To: undisclosed-recipients:; included): mail-parser gives an empty address list, and the Elasticsearch and OpenSearch saves indexed [0] (IndexError). - An SMTP TLS report with a wrong-typed field: "policy-string": [1] made the CSV writer raise TypeError; "start-datetime": 5 made the Elasticsearch, OpenSearch and Splunk saves raise AttributeError and _ensure_utc_suffix() in the PostgreSQL save raise TypeError. - An aggregate report whose <reason> has no <type> (KeyError in the CSV writer) or a <type> that xmltodict parses to a dict or a list (TypeError in its join). Fix: - Every output destination block in process_reports() now ends with the catch-all the aggregate Elasticsearch block had: the failure and SMTP TLS Elasticsearch/OpenSearch blocks, every PostgreSQL and Splunk HEC block, and the file output (its OSError/ValueError branch is kept). The error goes through log_output_error(), so the batch is reported unsaved and fail_on_output_error still raises ParserError. - parse_smtp_tls_report_json() and its helpers check each field against the type RFC 8460 section 4.4 gives it and raise InvalidSMTPTLSReport otherwise: strings for organization-name, report-id, policy-domain, policy-type, result-type and the optional failure-details fields; contact-info a string or a list of strings; policy-string and a list mx-host-pattern lists of strings; the session counts integers or null; failure-details a list of objects; start-datetime and end-datetime strings of at most 64 characters that the outputs' own conversions accept. dateutil took 12.4 s to fail on a 1,000,000-digit date, hence the length bound. - _parse_report_record() makes a reason's type and comment a string or None (a dict unwrapped with _text(), a list as its str()), and the aggregate CSV writer reads both with .get(). - The Elasticsearch and OpenSearch failure saves flatten From, To and Reply-To with a helper that returns None for an empty list, and leave such a header out of the document and the duplicate query. - The Splunk SMTP TLS save converts begin_date inside its try, so a bad value raises SplunkError; _ensure_utc_suffix() returns a non-string unchanged. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Limit XML nesting depth and node count in aggregate reports parse_aggregate_report_xml() gave the decompressed report (up to the 100 MiB decompression cap) to xmltodict with no limit on depth or element count. xmltodict keeps one stack entry per open element and expat reports missing end tags only at the end of the input, so a few KB of gzipped unclosed <a> tags grew to gigabytes of RSS and the kernel OOM killer ended the process before the message was moved; a wide document of millions of siblings did the same at a smaller scale. A failed first parse also re-parsed the document through lxml recovery, which has no node limit, and a second xmltodict pass. _NoDoctypeExpat.ParserCreate() now returns a proxy around the expat parser. pyexpat parsers cannot be subclassed and xmltodict assigns its handlers after ParserCreate() returns, so the proxy's __setattr__ wraps StartElementHandler to count depth and nodes (elements plus attributes) and EndElementHandler to decrement depth, and forwards every other attribute. Over MAX_XML_DEPTH (100) or MAX_XML_NODES (5,000,000) it raises before xmltodict's handler runs. Both xmltodict.parse() call sites turn that into InvalidAggregateReport naming the limit; the first does so instead of trying lxml recovery. The lxml recovery path now runs a counting pass first, with a parser target that builds no tree and counts elements, attributes and namespace declarations against MAX_XML_NODES; the tree is built only for a document within the limit. Depth there is bounded by libxml2's own limit of 256 (huge_tree stays off) and then by the second xmltodict pass. A 22 MiB wide document behind a malformed XML declaration peaked at +1532 MiB before this pass and +23 MiB after. The largest aggregate sample (909 KB) has 32,019 nodes, so a legitimate report at the 100 MiB cap stays near 3.6 million. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Create Elasticsearch/OpenSearch indexes with mapping guards create_indexes() and create_data_streams() created indexes and the data stream template with settings only, so the cluster typed every report field from the first document that held it, with date_detection on. A sender could be first into a day's index (its name comes from the report) and poison it: a report_id of "2026-11-09" mapped report_id as a date, a failure sample header named "Subject.x" mapped sample.headers.subject as an object, and samples with invented header names used up index.mapping.total_fields.limit. Every later legitimate report for that index was then rejected and ended up in Unsaved; in data stream mode this lasted until a rollover. Both functions now take a mappings body, and the three save functions pass one per report type. Each turns date_detection off and declares the fields that hold dates (@timestamp, date_range, date_begin, date_end, arrival_date, sample.date) as dates, the type date detection gave them before; on the sample reports every other field keeps its old type. The failure mapping maps sample.headers as an object with dynamic: false and declares, as text with a keyword sub-field (the dynamic string default), every header the shipped dashboards reference plus date, message-id and cc; all headers still stay in _source. A "." in a header name is replaced with "_" before the header is stored. The index is now created through the client's indices.create with the settings and mappings body, because opensearch-py's Index.create cannot take a mappings argument; the data stream template merges the mappings with its @timestamp date mapping. Existing indexes are not changed. The other report fields stay dynamic and object-mapped (no nested), which the dashboards need (issue #169). Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read the RFC 8460 mx-host key in SMTP TLS reports RFC 8460 section 4.4 keys a policy's MX host patterns "mx-host"; its schema line is "mx-host": mx-host-pattern, where mx-host-pattern is the value type, described as "a JSON array of strings", and the RFC's Appendix B example gives the value as a plain string. The parser read only "mx-host-pattern", so every RFC-conformant report lost the field (samples/smtp_tls/rfc8460.json, the Appendix B example, among them). _parse_smtp_tls_report_policy() now reads "mx-host" as a string or a list of strings, a string becoming a one-element list, and falls back to "mx-host-pattern" (same rules) when "mx-host" is absent, null or empty. Any other type raises InvalidSMTPTLSReport. The output key is still mx_host_patterns, so the PostgreSQL TEXT[] column, the Elasticsearch/OpenSearch field and the CSV column are unchanged. Refs #942 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject over-limit messages with the mailsuite guard parse_report_email() and parse_email() handed every message to mail-parser and the email package with no bound on blank lines, header lines, header field size, MIME parts or nested multiparts, so a small crafted message took seconds to minutes and gigabytes to parse while the rest of the mailbox waited. Both now call mailsuite.utils.check_message_limits() first, one linear pass. parse_report_email() turns MessageLimitExceeded into InvalidDMARCReport so the mailbox loop files the message under Invalid; parse_failure_report() turns one from the sample into InvalidFailureReport. The mailsuite floor rises to 2.3.5, which adds the guard. Messages nested deeper than 100 multiparts are now rejected by the guard instead of reaching the recursion handler; the tests that used the 400-level message expect the new message. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Cite RFC 7489 §6.3 for the daily report interval The comment on MAX_REPORT_SPAN_DAYS cited RFC 7489 Appendix A for the one-day aggregate report. The daily interval comes from §6.3: the "ri" tag requests the interval between aggregate reports, and its default is 86400 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the 11.2.2 changelog: five advisories and #942 One Security bullet per advisory (GHSA-qcc2, GHSA-q7gm, GHSA-p85v, GHSA-4272, GHSA-qpcg) and a Bug fixes bullet for the RFC 8460 mx-host key. The version in constants.py is bumped by the release pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the date range rejection message and pyright's analysis With the XML limits and the date range limit merged, parse_aggregate_report_xml() became too complex for pyright to analyze ("Code is too complex to analyze"), and the date range rejection, raised inside the function's catch-all, reached the caller as "Unexpected error: Report date range spans ...". The date range check moves to _check_report_span() and the second xmltodict pass to _parse_limited_xml(), which raises the XML limit breach as InvalidAggregateReport; the catch-all now re-raises an InvalidAggregateReport as it is. The tests assert the message is not re-wrapped. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Put the over-limit failure samples in a delivery-status body mailsuite 2.3.5's check_message_limits() follows the MIME structure and counts blank lines only inside message/delivery-status bodies, the shape whose parse cost is per blank line; blank lines elsewhere in a body are not limited. The two failure-sample tests used blank lines in a plain body, which the final check accepts, so they now put them in a delivery-status body. parsedmarc's call with the default limits is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe the final mailsuite check and the index cap in the docs The GHSA-qcc2 changelog bullet now describes check_message_limits() as mailsuite 2.3.5 ships it: limits follow the MIME structure, and body lines outside message/delivery-status bodies are not limited. The changelog section moves to Unreleased, as parsedmarc's AGENTS.md has fix pull requests write it; the release pull request names the version. The Elasticsearch and OpenSearch pages now say that an address header with no address is left out of the failure document, and that a report spanning more than 31 days saves its later records to its first dated index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound records times days in normalization and fix the span message The date range limit bounds how far one record is split, but not how many records are split: 10,000 records over 366 days, each with a count of 1,000,000 (3.3 MB of XML), produced 3,670,000 records in 13.5 s and 1.7 GB, and 30,000 of them (a 44 KB gzip) ran out of memory and reached the caller as "Unexpected error: ". A report whose split would exceed MAX_NORMALIZED_RECORDS (250,000) now keeps its records whole over its own range, with timespan_requires_normalization false and one warning; a large legitimate multi-day report is kept rather than rejected. The record loop moves to _parse_report_records() so pyright can still analyze parse_aggregate_report_xml(). The span message used float division, which raises OverflowError for an <end> of 4,001 digits; it now uses integer division. The index-routing warning escapes the sender's report ID with sanitize_for_log(). The changelog and the Elasticsearch and OpenSearch pages say the index cap is per report, that dated index names still come from report dates, and that data streams are affected by neither. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop XML attributes from aggregate reports before they are used xmltodict turns an element with an attribute, such as <p a="1">none</p>, into {"@a": "1", "#text": "none"}. Such objects passed through the parser into Elasticsearch and OpenSearch documents for version, p, fo, disposition, envelope_to and other scalar fields, so the first one mapped the field as an object and every later report to that index failed, and hundreds of attributes on one element used up total_fields.limit. The parsed report now goes through one linear pass that replaces an element with only attributes and text by its text and drops attributes from the others; the existing _text() calls now receive strings. The changelog and the Elasticsearch and OpenSearch pages also say that an existing data stream's next backing index gets the guarded mappings, since parsedmarc puts the template again in every process, and the indexed-header comment and changelog name cc as the one header added to those the dashboards reference. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject lone surrogates in SMTP TLS reports and guard the stdout print An SMTP TLS string field holding a lone surrogate (JSON "\ud800") passed every type check and survived json.dumps(ensure_ascii=False), and print(output_str) in process_reports(), outside every try, raised UnicodeEncodeError: parsedmarc exited 1, and in mailbox mode the error escaped the save callback. RFC 8460 section 4 reports are I-JSON, whose RFC 7493 section 2.1 forbids surrogate code points, so the string and string-list checks now reject one as InvalidSMTPTLSReport, and the print records a UnicodeEncodeError as a "Standard output" output error. The _smtp_tls_datetime() docstring no longer says a parsed date cannot fail in any output: dateutil accepts partial dates that PostgreSQL may reject, which is contained as PostgreSQLError. The _reason_text() docstring says a dict with child elements gives None. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe mailsuite 2.3.5's address group handling in the changelog The final mailsuite 2.3.5 no longer rejects an address header by its comma count; its parse_email() removes RFC 5322 group syntax from address header values over 8 KiB before parsing them. The boundary limit counts lines that are the boundary of an open multipart. parsedmarc's call with the default limits is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the keep-alive call of the extracted record loop _parse_report_records() holds the record loop moved out of parse_aggregate_report_xml(); its keep-alive branch had no test. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a policy override type with child elements With XML attributes dropped at parse time, a dict reaches _reason_text() only for an element with child elements; the test pins its None result and the CSV's "none". Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check plain-string contact-info and mx-host for lone surrogates The surrogate check covered the string fields and string lists, but a contact-info or mx-host given as one string took a branch that skipped it, and a lone surrogate there still made the stdout print raise. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which sample headers the failure document leaves out The GHSA-qpcg changelog bullet said every sample header stays in _source, but the failure save leaves out a From, To or Reply-To header that holds no address, as the Elasticsearch and OpenSearch pages say. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep sender-derived numbers out of the date range message The rejection message printed the report's span in days, so an <end> of thousands of digits put a number of thousands of digits into the exception text and the log. It now says only that the range is negative or longer than MAX_REPORT_SPAN_DAYS. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Store every string-typed aggregate field as a string or null Dropping attributes did not cover child elements: <version><x>1</x></version>, <p><x>1</x></p> or <p>none<x/></p> still reached the parsed report, and the Elasticsearch and OpenSearch documents, as a dict, and the first such document mapped the field as an object. _scalar() now makes each field the schema types as a string a string (an element's own text) or None, applied by _scalarize_report() to version, the report_metadata strings and every policy_published child, and by _parse_report_record() to source_ip, disposition, dkim, spf, the identifiers and every auth_results string. A <count> with no text of its own takes the ValueError path of a non-numeric count. _drop_xml_attributes() stays, so a field missed here still loses its attribute keys. The <email> dict unwrap is gone, since the field can no longer be a dict; a text-only report_metadata still raises the AttributeError that reports a missing section. Comments that described attribute dicts now say the dict shape comes only from child elements. The namespace test, which the new code could not fail, is replaced by one that detects RFC 9990 through an attributed np field. Tests walk the to_dict() of a mocked Elasticsearch and OpenSearch save and allow lists and objects only in the known object fields, so a scalar field added later is covered too. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record a closed stdout pipe as an output error and fix two comments The standard-output guard in process_reports() caught only UnicodeEncodeError, so a closed pipe (BrokenPipeError, an OSError) still ended the run; it now records both as a "Standard output" output error. _reason_text()'s docstring said a dict gives None because the element has no text of its own, but mixed content (<type>forwarded<x/></type>) gives the text beside the children; the docstring now says what the code does, and a test pins it. The comment above the reason normalization no longer describes attribute dicts, which no longer reach it. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the normalization limit at its boundary and reword the index cap The suite passed with the limit's <= changed to < and with _days_touched()'s ceiling changed to a floor. New tests: 1,000 records over 250 days (exactly MAX_NORMALIZED_RECORDS) are split; 683 records over 366 days (249,978) are split and 684 (250,344) are not; 682 records over 366 days starting 1 s after midnight touch 367 days (250,294) and are not split. Each mutant was run against these tests and fails: the < mutant fails the exact-limit test, the floor mutant the after-midnight test. The Elasticsearch and OpenSearch pages now say the index cap applies when a report's records would use more than 31 dated indexes, which is what it counts (with monthly_indexes the span limit keeps a report under it). Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Match the tests and changelog to mailsuite 2.3.5's final check mailsuite 2.3.5's check_message_limits() now parses the message with the email package's own feed parser under compat32 and counts the parts it creates (each message/delivery-status block among them), their nesting depth and its boundary tests, then limits header field length and count on the parsed tree; the limit names changed. The over-limit failure samples now trip max_parts, and the deeply nested poison message max_depth; the tests assert those texts. The GHSA-qcc2 changelog bullet describes the final check, the epilogue mail-parser parses again, and the changes that affect only mailsuite's own parse_email(). Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a repeated policy_published and note mixed-content behaviour No test failed when _scalarize_report() stopped coercing the fields of a repeated <policy_published> (the list case): with that mutant, a report with two policy_published elements and <p><x>1</x></p> puts a dict into published_policy.p. The parse test and the Elasticsearch and OpenSearch document-walk tests now include that report; each fails on the mutant. The changelog says that an element with text and a stray child element now yields its text (samples/aggregate/invalid_xml.xml gives header_from "bad" where it gave ""). AGENTS.md and the _normalize_result_word() and _scalarize_report() docstrings describe what runs now: attributes are dropped as soon as the XML is parsed, and _scalar() coerces the string fields, among them the RFC 7489 and RFC 9990 policy_published fields. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Coerce policy override reasons with the same rule as other strings _reason_text() turned a repeated <type> or <comment> into the str() of the list, while _scalar() turns every other repeated string field into None. One class of input now has one rule: the reason's type and comment go through _scalar(), so a repeated one is None ("none" in CSV), and _reason_text() is removed. The test and the GHSA-4272 changelog bullet say so. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test an aggregate <error> with a child element With attributes dropped and reasons coerced by _scalar(), only <error>, which may repeat, still reaches _text() with a dict; its branch had no test. An <error> with text beside a child element keeps the text, and one with no text is dropped. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe mailsuite's six message limits in the changelog mailsuite 2.3.5's check now counts every line of every header block (folded lines and lines the parser keeps only as defects included) and stops the parse early, and limits the total length of the address headers mail-parser parses to 262,144 characters. The GHSA-qcc2 bullet names the six limits. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render a report email's parts only where their text is read parse_report_email() did str(payload_obj[0]) for every part walk() visits, before the multipart/ continue. str() of a composite part renders its whole subtree, and walk() visits every level, so a nesting within mailsuite's limits rendered its leaf once per level: 98 nested multiparts around a 20 MiB one-line leaf took 26.6 s, and a chain of 99 message/rfc822 parts around a 10 MiB leaf 11.3 s. A part is now rendered only in the branches that read it; other message/* bodies are skipped, since walk() visits their parts. Both shapes now take under 0.5 s, and every sample under samples/failure/ parses to the same output. A feedback report's sample is taken from the first sample part only: a message/rfc822 nested inside the sample used to replace it, but a feedback report has one sample (RFC 6591 section 3.1). Multipart nesting alone no longer reaches the recursion limit, so the GHSA-qxwv test now reaches the recursion guard through a deeply nested sample, which is still rendered. The changelog describes this and the address headers mailsuite 2.3.5 counts (From, Sender, Reply-To, To, Cc, Bcc, Delivered-To, in every part; mail-parser parses all but Sender). Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Parse aggregate XML without attributes and lower the node limit _drop_xml_attributes() rebuilt the parsed tree while the original was alive, and the counting wrappers of _LimitedExpatParser referred to the proxy, closing a reference cycle through the expat parser that kept xmltodict's tree alive until the cyclic garbage collector ran. Both xmltodict passes now use xml_attribs=False, so no attribute key is ever built (expat still hands the attributes to the start handler, so they are still counted: a test shows one element with 1,000 attributes is 1,001 nodes), _drop_xml_attributes() is removed, and the counts live in an _XMLCounts object the wrappers share, so a parse leaves no cycle (a test asserts gc.collect() finds nothing after one). Peak extra memory for 2,000,000 distinct sibling elements: +999 MiB before, +386 MiB after (master +391 MiB); for one element with 2,000,000 attributes: +1,274 MiB before, +445 MiB after (master +734 MiB). At the old limit of 5,000,000 nodes a report could still need about 3 GB. Measured after the change, the worst shape is distinct elements with text behind a malformed XML declaration (the lxml recovery path): 559 bytes per node, +533 MiB for 999,000 nodes and +1,023 MiB for 2,000,000; on the expat path about 220 bytes per node. MAX_XML_NODES is now 1,000,000, so a report at the limit needs at most about 550 MiB; the largest sample has 32,019 nodes (909 KB), and the limit is a report of about 28 MB. The constant's comment and the changelog state the bound. AGENTS.md describes the attribute handling as it now is, and says that <error>, which may repeat, goes through _text() and stays a list. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qualify the mixed-content compatibility note to string fields An element with text beside a stray child element yields its text only for the fields the schema types as strings; a <begin> or <end> with mixed content is still rejected, as before. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject over-long XML tags before any parser reads them expat and libxml2 build a start tag's whole attribute list before they report the element, so the node counter only sees the attributes after they cost memory: one element with 4,999,000 attributes (56 MiB) took 5.25 s and +767 MiB before the node limit rejected it, and a single element is bounded only by the 100 MiB decompression cap. parse_aggregate_report_xml() now scans the decoded text once, before the first parse and so before the lxml recovery path too, and rejects a document in which any text between a "<" and the next ">" (or the end of the text) is longer than MAX_XML_TAG_LENGTH (65,536 characters). The scan uses no regex. It splits the text on ">" one 64 KiB chunk at a time, carrying the unfinished piece from its last "<": a single split of the whole text makes one string per tag, which peaked at +1,318 MiB (1.9 s) for 100 MiB of "<a/>"; the chunked scan needs about 2 MiB and takes 0.6 s on that text, and a test bounds its memory and covers a tag across a chunk boundary. The longest tag in the samples is 108 characters. The same shape is now rejected in under 0.1 s, with a tracemalloc peak under 200 MB in the test, behind a well-formed or a malformed declaration. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render a report email's feedback-report part only once The message/* skip exempted message/feedback-report, which was rendered with str() every time walk() reached one: a chain of 99 nested message/feedback-report parts around a 10 MiB leaf passed mailsuite's check in 0.02 s and then took 11.6 s. A feedback report has one message/feedback-report part (RFC 6591 section 3), so, as with the sample, only the first is read and rendered; the chain now takes 0.56 s. The failure samples parse to byte-identical output. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound attribute counts by their = signs and cap reports at 32 MiB XML allows a raw > inside a quoted attribute value, so attributes such as x=">" cut a tag into short pieces that pass the tag scan: 4,999,000 of them cost 5.61 s and +781 MiB, and 8,000,000 (98 MiB) 8.9 s and +1,215 MiB, before the node limit fired. Every attribute needs an =, so a document with more = characters than MAX_XML_NODES is now rejected in one str.count() pass, before the tag scan; the samples hold at most 3. The tag scan stays for long tags, and its docstring says it does not bound attribute counts. The stated memory bound was false: text per node is bounded only by the size of the report, so at the node limit with 80 characters per element (94 MiB) the recovery path reached +827 MiB. MAX_DECOMPRESSED_REPORT_SIZE falls from 100 MiB to 32 MiB (the node limit already means a legitimate report is at most about 28 MB, at the samples' 35 nodes per KB, and the largest sample is 909 KB), and parse_aggregate_report_xml() holds an uncompressed report to the same length, so the bound holds for every input. Measured at the cap, 999,000 distinct elements with 14 characters of text (31 MiB) need +587 MiB on the recovery path and +302 MiB on the expat path; the constant comment and the changelog state it. When the first parse fails, the exception's traceback kept the partial xmltodict tree alive while the recovery parse ran; it is cleared first (+603 to +587 MiB, and +263 to +241 MiB for 330,000 elements of 80 characters). A test shows the recovery path leaves the same 18 lxml helper objects for a document ten times larger, so no tree survives a parse there either. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let the RFC 6591 feedback part win over legacy report text The guard that reads only the first message/feedback-report part tested feedback_report, which the legacy text/plain branch ("A message claiming to be from you has failed ... detected.") also sets, so a legacy text part followed by a real feedback-report part and sample now took the legacy part and never read the ARF, where master read the ARF. The guard now tests is_feedback_report, and reading the feedback-report part discards a sample a legacy part set before it. The same rule closes an older gap: a text/plain part inside the sample, which is the spoofed message's own text, could hold the legacy phrase and replace the report's Source-IP and its sample. Once a feedback-report part is read, the legacy branch is skipped; a legacy report has no such part. Tests cover both orders and the in-sample case; the failure samples parse to byte-identical output. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold every report input to the size cap and free failed parses parse_report_file() kept the aggregate parser's exception, whose traceback holds the parser's frames and the parsed tree, while it tried SMTP TLS and then the email parser: a 31 MiB file at the cap cost +1,046 MiB there against +587 MiB parsing it directly. It now keeps only each failure's text and raises from a fresh exception of the relevant format, so the chain survives without the tree. A file that begins with "<" is not tried as JSON (json.loads() decoded it all) or as an email, one that begins with "{" is not tried as an email (an RFC 822 message begins with a header line), and text over the size cap is never handed to the email parser, which has none: a 200 MiB <x/> file reached +2,801 MiB in 4.6 s and now costs +232 MiB (reading the file) in 0.05 s. extract_report() reads an uncompressed report at most one byte past the cap instead of decoding it whole. The cap counted characters for text and bytes for compressed reports, so a non-ASCII report could be four times the cap in bytes: 999,000 elements of 14 emoji (31.2 Mi characters, 71 MiB of UTF-8) cost +649 MiB. _exceeds_size_cap() measures text as UTF-8 (ASCII by length, other text 1 MiB of characters at a time, without a full copy), and bytes are checked before they are decoded. Clearing e.__traceback__ inside the except block did nothing on Python 3.10, which keeps the handled exception's traceback while the handler runs. The lxml recovery now runs after the except block, in _recover_xml(), so the first parse's frames and partial tree are unreachable when it starts on every supported Python; a test holds a weak reference to the first parse's counter and fails on the old code under Python 3.10.19 and, with the clearing removed, under 3.14. The changelog's examples are re-measured under the 32 MiB cap, and the test docstrings cite the same figures. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip the parts inside a failure report's sample Once the sample part was taken, walk() still visited its descendants, and they reached the SMTP TLS and base64 attachment branches: a sample whose spoofed message carried an SVG or zipped HTML attachment made the email rejected ("Missing field: 'feedback'", "Unable to parse"), and a gzipped aggregate report or a tlsrpt+json attachment inside it was returned as that report instead of the failure report. The sample is the reported message, so after it is taken every part inside it is skipped (by the id() of each part its walk() yields). Tests cover SVG, zipped HTML, JSON, gzipped aggregate XML, tlsrpt+json and PDF attachments; the failure samples parse to byte-identical output. The Bug fixes bullet also says that a legacy text part followed by a feedback-report part with no sample after it is now invalid, with a test. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cap SMTP TLS reports and test the format skips parse_smtp_tls_report_json() decoded and json.loads()-ed a report of any size; only the gzipped path was capped, and 60 MiB of "[]," cost +1,565 MiB. It now rejects bytes or text over MAX_DECOMPRESSED_REPORT_SIZE first, as InvalidSMTPTLSReport, so every report type is held to the cap. parse_report_file() names the actual reason it skips the email parser (the leading "<" or "{", or the size cap) rather than a combined one, and a size message for bytes no longer says "of UTF-8". New tests kill the mutants the review found surviving: removing the "<" skip for JSON, narrowing the email skip to "<", the bytes check >= for >, and removing the SMTP TLS cap. The constant comment and the changelog state the memory at the cap as measured on the current code: +345 MiB on the lxml recovery path and +302 MiB on expat parsing the text directly, +500 MiB and +452 MiB through parse_report_file(). The changelog says only the exception's cause changed for files skipped by format, not the message. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip every sample candidate and correct the stated measurements sample_parts held only the first sample candidate's parts, so a later message/rfc822 (or one after a text/rfc822-headers sample) was dropped as a sample but its parts were still walked, and a gzipped aggregate report inside it was returned in place of the failure report. Every candidate's parts are now skipped. Tests cover both shapes, and a test that a gzipped aggregate report beside the sample is still returned pins the rule (inside the sample: skipped; beside it: read) and fails when the set is built from the whole message. parse_report_file() no longer builds email reasons for content that begins with "<" or "{", which its message never shows, and the size reason reads "Invalid report email: the report is larger than 32 MiB". The memory figures are re-measured in a fresh process with the input read from disk before the baseline: +399 MiB on the recovery path and +338 to +355 MiB on expat for the 31 MiB shape, +1,505 MiB for 60 MiB of JSON before the SMTP TLS cap; the constant comment, the changelog and the test docstrings state them. RFC 6591 section 3.1 is cited for the sample's place, and application/json is named among the attachments that made a failure report invalid. Refs GHSA-qcc2-4x3v-v4g7 Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Scale the timed tests' bounds to the machine running them The mailsuite fix's first CI run on GitHub's shared runners failed six timed tests: the suite ran four to six times slower there than on a workstation, and checks doing real work took longer than their fixed bounds. Several bounds here have far less margin (0.2 s on a test that takes 0.18 s whole), so they would fail the same way. tests/timing.py adds budget(seconds), which measures a fixed reference parse twice just before the timed call and multiplies the bound by the slowdown against a reference machine; a faster machine keeps the written bound. Every timed assert now compares against budget(<written seconds>) computed before the timer starts. No written bound changed and no assert was removed. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
48ea47adee |
Merge commit from fork
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv) With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init lines and opensearch.set_hosts() logged configured URLs verbatim, so userinfo passwords reached the console and log_file (CWE-532). Triage found more paths of the same class: - the httpx "HTTP Request:" line, forwarded at --verbose/--debug, contains the full URL (str(httpx.URL) keeps userinfo and query), so webhook and HEC URL credentials and the IPinfo ?token= were logged on every request; - the PSL overrides, IP database and reverse DNS map downloaders logged their URL at DEBUG and str(HTTPStatusError), which embeds the full URL, at WARNING; - libpq echoes the DSN or a password fragment for some malformed PostgreSQL connection strings, logged as an output-client error. Add redact_url_credentials() (string methods only; urlsplit misparses scheme-less "user:pw@host:port") and use it at each log site; add a filter on the httpx logger that masks userinfo and the token query value; build download-failure warnings from the status code and the redacted request URL instead of the exception text; replace psycopg ProgrammingError detail from a connection_string with a fixed message. Docs now recommend separate user/password settings over URL credentials. Reported by helixnebulatech. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Mask credentials in the Loki URL in the log and LokiError text The Loki output (#926) landed on master after this fix was written. Its url keeps userinfo (copy_with(path=...) preserves it), and the --debug "Initializing Loki client" line and both LokiError messages, which the CLI logs on a failed push, included it as-is: the same CWE-532 path as the Splunk HEC line. Mask it with redact_url_credentials() at all three sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532): - httpx.InvalidURL quotes the component it rejected, so a password with an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there) leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats it. Loki, Splunk HEC, the webhook and the three download loaders now report a fixed "details withheld" message raised from None. InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url crashed startup; the loaders now treat it as a download failure. - redact_url_credentials() masks through the last "@" when the authority has no "@" but one follows it, so the --debug lines no longer print such passwords; the userinfo lookup is shared with the new nameserver masking. - dnspython 2.9 DoH logs its request line on the httpx2 logger, which was forwarded but not filtered; the filter now covers httpx2 and httpx2.URL. - [general] nameservers / --nameservers credentials are masked in the DNS pre-flight error, reverse DNS debug errors and per-report errors, and unusable https:// / tls:// ports are rejected before any query. - Docs recommend token/username settings for [loki] and [splunk_hec]; AGENTS.md gains rules for keeping credentials out of logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Fable 5
|
df9bf82e04 |
Post-review follow-ups for Graph send (#825/#826) and requests-to-httpx migration (#827)
Follow-ups from the review of PR #825 (whose implementation had already landed on master via #826's stacked merge): - Honor the documented [smtp] attachment and [smtp] message options. Both were parsed into opts but never passed to either summary-email transport (also broken in released 10.2.2), so a configured custom attachment filename or message body was silently ignored. Both the SMTP and Microsoft Graph transports now receive them, and the missing smtp_attachment Namespace default is added (also covers SIGHUP reload, which rebuilds opts from the CLI Namespace). - Don't mislabel non-Graph mailbox errors as Microsoft Graph failures: the shared mailbox-fetch and watch handlers now log a generic "Mailbox Error" with traceback when the connection isn't Graph. - Declare microsoft-kiota-abstractions as a direct dependency (imported directly in cli.py for Graph error handling; previously transitive). Migrate all runtime HTTP from requests to httpx (webhook client, Splunk HEC client, and the PSL-overrides / IP-database / reverse-DNS-map / IPinfo-API fetches in utils.py): - follow_redirects=True everywhere to preserve requests' default redirect-following; httpx does not follow redirects by default. - The PSL-overrides and reverse-DNS-map fetches gain a 60s timeout (previously none), matching the IP-database fetch. - response.ok -> response.is_success; requests.RequestException -> httpx.HTTPError; raw string bodies use content= (httpx's data= is form-encoding only); Splunk HEC verification moves to client construction (httpx has no per-request verify). - requests drops out of [project] dependencies and moves to the [build] extra for the out-of-wheel maintainer script collect_domain_info.py, which deliberately stays on requests/urllib3 for its permissive-TLS adapter. - Remove the requests-era module-level urllib3.disable_warnings(InsecureRequestWarning) in splunk.py; httpx doesn't route through urllib3, so its only remaining effect was globally silencing insecure-TLS warnings from other urllib3-based components as an import side effect. Nothing imports urllib3 directly anymore, so it also leaves [project] dependencies. Tests: config-to-transport wiring for attachment/message on both transports (including defaults), non-Graph errors keep the generic log line, webhook/Splunk payload assertions moved to content=, and Splunk verify asserted at httpx.Client construction. 736 passed; ruff and pyright clean. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Sonnet 5
|
01b85ab7de |
Fix aggregate-report timestamp skew on non-UTC hosts (#819) (#821)
record["interval_begin"]/["interval_end"] are UTC wall-clock strings
(produced in __init__.py by strftime() on a datetime already converted
via human_timestamp_to_datetime(..., to_utc=True)), but the
Elasticsearch and OpenSearch per-record save loops and the Splunk HEC
aggregate-report event builder re-parsed them without assume_utc=True,
so on non-UTC hosts they were misinterpreted as local time and shifted
by the host's UTC offset -- shifting the stored date_begin/date_end,
the daily/monthly index date, and the Splunk event time. Verified by
parsing a real sample report under TZ=Europe/Warsaw vs TZ=UTC and by
reproducing the exact skew via human_timestamp_to_unix_timestamp().
This is the same class of bug fixed for arrival_date_utc in #811/#812
(commit
|
||
 
|
cdda5dae62 |
Fix failure-report timestamp skew on non-UTC hosts in ES/OpenSearch/Splunk outputs (#812)
* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks arrival_date_utc is a UTC wall-clock string (generated in parse_failure_report via an aware-UTC strftime), but elastic.py, opensearch.py, and splunk.py parsed it back into a naive datetime and called .timestamp(), which per the Python docs interprets naive values as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp). On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date field, used in the failure-report dedup match query, and sent as the Splunk HEC event time was therefore off by the host's UTC offset (verified -3600 s under TZ=Europe/Warsaw in January). Add an assume_utc keyword to human_timestamp_to_datetime() / human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive parses, and use it at the three arrival_date_utc call sites. Aware inputs (explicit offsets) are unaffected; all other callers keep the existing local-time semantics, whose round-trip with timestamp_to_human is self-consistent on a single host (the broader local-time output question is tracked separately in issue #811 bug 2). The three new sink regression tests fail on the unfixed code (verified by stashing the source changes) and force TZ=Europe/Warsaw via time.tzset() so they catch the skew even on UTC CI runners. Fixes half of https://github.com/domainaware/parsedmarc/issues/811. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from the four timestamp regression tests into a shared tests/tzutil.py force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s docstring, which said the return type was float while the function returns int. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Fable 5
|
0c456d44ed |
Declare backward-compatible method aliases inside class bodies (#797)
* Declare backward-compatible method aliases inside class bodies Assigning the legacy save_forensic_* aliases onto the classes after the class body (KafkaClient.save_forensic_reports_to_kafka = ...) is invisible to static type checkers, so Pylance/Pyright flagged every assignment and every use with reportAttributeAccessIssue. Declaring the alias inside the class body is statically visible — the IDE errors disappear and the aliases get autocomplete and proper typing. Runtime behavior is identical (same function object bound as a method), guarded by the existing assertIs alias tests, whose type-ignore comments are now unnecessary. Also add a pyright ignore on the NoBrokersAvailable import in kafkaclient.py: the import is guarded by try/except ImportError for kafka-python 2.x, but Pyright resolves against the installed 3.x where the name no longer exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump version to 10.1.0 10.0.4 is tagged and released; CHANGELOG.md already documents the in-progress 10.1.0 section that this release will ship. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 4.7
|
b7b8383fa4 |
Expand honest test coverage from 59% to 83%; fix two latent bugs (#775)
* Expand honest test coverage from 59% to 83%; fix two latent bugs 271 new tests across the output modules, ES/OS clients, CLI config parsing, and the top-level parsing surface. Coverage measured against shipped code only (see [tool.coverage.run] source = ["parsedmarc"] omit = ["*/parsedmarc/resources/maps/*.py"] in pyproject.toml). Per-module results: s3.py 38% → 100% (also fixes SMTP-TLS-to-S3 bug below) gelf.py 40% → 100% syslog.py 46% → 100% kafkaclient.py 34% → 100% splunk.py 24% → 100% loganalytics.py 56% → 100% webhook.py 78% → 100% (also removes redundant try/except) elastic.py 36% → 99% opensearch.py 40% → 99% cli.py 52% → 69% __init__.py 74% → 76% (also fixes append_json bug below) utils.py 84% (unchanged in this PR) TOTAL 59% → 83% The remaining 17% is honest. The biggest unreached blocks are _main() in cli.py and the watch-mode mailbox iteration in __init__.py, both of which would require either standing up live subsystems (real Elasticsearch, real IMAP) or mocking deep enough that the test would verify the mock rather than the code. The PR-A AGENTS.md guidance — "if 90% requires faking it, ship 85% honestly" — applies here. Bugs fixed while writing tests: 1. parsedmarc/s3.py — SMTP-TLS-to-S3 was completely broken. save_report_to_s3 unconditionally read report["report_metadata"] when building S3 object metadata, but RFC 8460 §4.3 SMTP TLS reports are flat (no report_metadata sub-object). The CLI's surrounding try/except silently swallowed the KeyError, so every SMTP-TLS report quietly failed to upload. Also fixes a related issue: parse_smtp_tls_report_json stores begin_date as the raw ISO-8601 string from the report (per the SMTPTLSReport TypedDict and RFC 8460 §4.3), but the S3 code path assumed a datetime with .year / .month / .day attributes. Both fixed; the broken metadata-extraction branch now uses the flat-report fields, and the date branch normalizes via human_timestamp_to_datetime. 2. parsedmarc/__init__.py — append_json corrupted JSON output files on the second write. The original implementation opened files in "a+" mode, then seek()ed backwards to overwrite the trailing "]" with ",\n" before appending more elements. Python's docs are explicit (https://docs.python.org/3/library/functions.html#open): on POSIX, writes in "a"/"a+" mode always go to EOF regardless of seek() position. The result was that the second call produced [...]\n],\n[...] -style corrupted output instead of a single merged array. Replaced with a read-merge-write pattern: load the existing array (if any), append the new elements, rewrite the whole file. The CSV cousin append_csv was not affected — it doesn't seek backwards. 3. parsedmarc/webhook.py — removed redundant try/except blocks in save_aggregate_report_to_webhook / save_failure_report_to_webhook / save_smtp_tls_report_to_webhook. _send_to_webhook already catches every Exception itself, so the outer except blocks were unreachable dead code (covered nothing, defended against nothing, and inflated the source-line count without testing value). Testing approach: mocks at SDK boundaries (boto3 resource, kafka producer, requests session, opensearch/elasticsearch Document/Search, azure LogsIngestionClient). Tests verify the parsedmarc-side transformation logic — document/event construction, index/topic naming, dedup queries, error wrapping — rather than asserting on mock invocations as a proxy for behaviour. Where a branch is defensive against a caller that doesn't exist in the codebase, the test is omitted (commented in code rather than hidden behind a pragma). 547 tests total (was 276), all passing. ruff check + format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document the two bug fixes from this PR in the 10.0.0 changelog Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document testing standards in AGENTS.md Adds a "Testing standards" section covering the principles applied in PR-A (split) and PR-B (coverage expansion): - Coverage measures shipped code only — don't reintroduce tests/* to the scope, don't expand omit, don't use # pragma: no cover. - Honest tests assert on observable behaviour, not "the mock was called". Mock at SDK boundaries; parse the payload that gets sent. - "If 90% requires faking it, ship 85% honestly" — coverage is a tool, not a goal. PR-B's deliberate stops at cli.py 69% and __init__.py 76% are the documented precedent for when to halt. - Verify bug claims against the relevant RFC, internal types, installed SDK source, or upstream docs before changing code. Cite the source in the commit message and test docstring (RFC 8460 §4.3 and the Python open() docs for #775's two bug fixes are the pattern to follow). - Bugs found while writing tests are fixed in the same PR; the test doubles as the regression guard. - File layout (tests/test_<module>.py) is non-negotiable; module-level test loggers need fresh-handler setup so test ordering doesn't break assertLogs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Cover the corrupt-file fallback in append_json Codecov flagged 2 missing patch-coverage lines on PR #775: the except (json.JSONDecodeError, OSError) branch in append_json, which falls back to overwriting when the existing file isn't a parseable JSON array. Two new tests in tests/test_init.py:TestAppendJson exercise both paths: - test_corrupt_existing_file_is_overwritten_cleanly: existing file contains invalid JSON; append_json overwrites with the new array. - test_existing_file_with_non_list_root_is_overwritten: existing file parses as {"foo": ...} (dict, not list); the isinstance guard rejects it and we overwrite cleanly. Patch coverage now 100% on the bug fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 4.7
|
5b08627eaa |
Split tests.py into per-module tests/test_<module>.py (#774)
* Split tests.py into per-module tests/test_<module>.py The 5174-line tests.py monolith is split into per-module files under tests/, mirroring the checkdmarc layout: tests/test_init.py parsedmarc/__init__.py parsing surface tests/test_cli.py parsedmarc/cli.py + config / env-vars / SIGHUP tests/test_utils.py parsedmarc/utils.py (DNS, IP info, PSL, etc.) tests/test_webhook.py parsedmarc/webhook.py tests/test_kafkaclient.py parsedmarc/kafkaclient.py tests/test_splunk.py parsedmarc/splunk.py tests/test_syslog.py parsedmarc/syslog.py tests/test_loganalytics.py parsedmarc/loganalytics.py tests/test_gelf.py parsedmarc/gelf.py tests/test_s3.py parsedmarc/s3.py tests/test_maps.py parsedmarc/resources/maps/ maintainer scripts The split is purely a redistribution — no test bodies changed, no tests added or removed. All 276 existing tests pass under the new layout. The current tests.py contains two kitchen-sink classes (`Test` at line 54 and `TestEnvVarConfig` at line 2360) holding tests that span many modules. Their methods are routed to the correct per-module file by name prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx, TestSighupReload, etc.) move whole. Each target file gets its own `class Test(unittest.TestCase)` for the redistributed kitchen-sink methods, plus the thematic classes verbatim. Wiring updates: - `.github/workflows/python-tests.yml`: `pytest ... tests.py` → `python -m pytest ... tests/` (also switches to `python -m pytest` per the checkdmarc convention so cwd lands on the project root). - `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]` and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for `parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only batch tooling that ships out of the wheel; excluding them from coverage makes the headline number reflect only installed library code. Runtime coverage on the new layout is 59% (was 45% with maps counted), and PR-B will push it to 90%+. - `AGENTS.md`: documents the new layout and how to run individual files / tests; tells future contributors not to reintroduce a monolithic tests.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restore 66.9% coverage baseline (count tests/ + parsedmarc) Master's headline 66.9% number on Codecov includes the tests.py file itself (99.35% covered) being measured alongside parsedmarc/*. The original tests.py had no `[tool.coverage.run]` block, so coverage's default — "measure every file imported during the run" — counted the test code as if it were product code. The split commit added `source = ["parsedmarc"]` which suppressed measurement of the test files (correct in principle, since test files aren't shipped code), and that alone made the headline number drop by ~8 percentage points without any actual loss of testing. This commit swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]` so both halves are measured the way they were on master. Verified: 276 tests, 66.96% line coverage (effectively unchanged from master's 66.90%). If you want the shipped-code-only number (was the headline that this commit overrides), run `pytest --cov=parsedmarc tests/`. That number is currently 59% and is the focus of the upcoming coverage-expansion PR. Also adds junit.xml to .gitignore so the CI artefact doesn't get accidentally committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restrict coverage to shipped code (`source = ["parsedmarc"]`) Reverts the prior commit's `include = ["tests/*"]`. Counting the test files toward coverage was wrong — it conflates "shipped code exercised by tests" with "test code that pytest auto-runs", inflates the headline number, and rewards writing more tests rather than tests that verify more code. Master's apparent 66.9% was an artefact of the old monolithic tests.py having no [tool.coverage.run] block at all; coverage's default behaviour measured every imported file, including the test file itself at ~99% "covered", which added ~8 percentage points to the displayed number without any real testing signal. Restricting to `source = ["parsedmarc"]` plus the existing maps omit gives a meaningful baseline: 59% of shipped code is exercised by the test suite today. That's the number the next PR is targeting to lift to 90%+ before the 10.0.0 release; the Codecov "drop" here is a measurement correction, not a regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |