mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-10 19:11:00 +00:00
fix-postgresql-output-test
1686
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Sean WhalenandClaude Sonnet 5.5
|
8c00e1a695 |
Make the PostgreSQL output test independent of psycopg
testPostgreSQLExceptionsAreRecorded replaces PostgreSQLClient with a mock, but _init_output_clients() first checks postgres.psycopg and raises ConfigurationError when the extra is missing, so the client was never built and the saver mocks were never called. Patch psycopg with a stand-in so the test passes whether or not the postgresql extra is installed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
2ee209005b |
Merge commit from fork
* chore: update IPinfo Lite MMDB (#941) Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Cap report date range and indexes created per report A report whose <date_range> spans millions of days was normalized into one record per day, exhausting memory, and with Elasticsearch or OpenSearch output created one dated index (one shard) per day, exhausting the cluster's shard budget. Reject a range longer than MAX_REPORT_SPAN_DAYS (366) or negative with InvalidAggregateReport. In both save functions, once a report has used MAX_INDEXES_PER_REPORT (31) distinct dated indexes, route further records to the report's first index, keeping their own dates, and log one warning. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Catch output-stage exceptions and validate SMTP TLS report types One report sent to a monitored mailbox could make a configured output raise an exception that process_reports() did not catch. It escaped through the mailbox save callback, _main logged "Mailbox Error" and exited 1, and the batch stayed in the mailbox, so every one-shot run died on the same message and duplicated the legitimate reports in non-deduplicating outputs. Three sender-controlled inputs did it: - A failure report whose sample has an empty From:, To: or Reply-To: (To: undisclosed-recipients:; included): mail-parser gives an empty address list, and the Elasticsearch and OpenSearch saves indexed [0] (IndexError). - An SMTP TLS report with a wrong-typed field: "policy-string": [1] made the CSV writer raise TypeError; "start-datetime": 5 made the Elasticsearch, OpenSearch and Splunk saves raise AttributeError and _ensure_utc_suffix() in the PostgreSQL save raise TypeError. - An aggregate report whose <reason> has no <type> (KeyError in the CSV writer) or a <type> that xmltodict parses to a dict or a list (TypeError in its join). Fix: - Every output destination block in process_reports() now ends with the catch-all the aggregate Elasticsearch block had: the failure and SMTP TLS Elasticsearch/OpenSearch blocks, every PostgreSQL and Splunk HEC block, and the file output (its OSError/ValueError branch is kept). The error goes through log_output_error(), so the batch is reported unsaved and fail_on_output_error still raises ParserError. - parse_smtp_tls_report_json() and its helpers check each field against the type RFC 8460 section 4.4 gives it and raise InvalidSMTPTLSReport otherwise: strings for organization-name, report-id, policy-domain, policy-type, result-type and the optional failure-details fields; contact-info a string or a list of strings; policy-string and a list mx-host-pattern lists of strings; the session counts integers or null; failure-details a list of objects; start-datetime and end-datetime strings of at most 64 characters that the outputs' own conversions accept. dateutil took 12.4 s to fail on a 1,000,000-digit date, hence the length bound. - _parse_report_record() makes a reason's type and comment a string or None (a dict unwrapped with _text(), a list as its str()), and the aggregate CSV writer reads both with .get(). - The Elasticsearch and OpenSearch failure saves flatten From, To and Reply-To with a helper that returns None for an empty list, and leave such a header out of the document and the duplicate query. - The Splunk SMTP TLS save converts begin_date inside its try, so a bad value raises SplunkError; _ensure_utc_suffix() returns a non-string unchanged. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Limit XML nesting depth and node count in aggregate reports parse_aggregate_report_xml() gave the decompressed report (up to the 100 MiB decompression cap) to xmltodict with no limit on depth or element count. xmltodict keeps one stack entry per open element and expat reports missing end tags only at the end of the input, so a few KB of gzipped unclosed <a> tags grew to gigabytes of RSS and the kernel OOM killer ended the process before the message was moved; a wide document of millions of siblings did the same at a smaller scale. A failed first parse also re-parsed the document through lxml recovery, which has no node limit, and a second xmltodict pass. _NoDoctypeExpat.ParserCreate() now returns a proxy around the expat parser. pyexpat parsers cannot be subclassed and xmltodict assigns its handlers after ParserCreate() returns, so the proxy's __setattr__ wraps StartElementHandler to count depth and nodes (elements plus attributes) and EndElementHandler to decrement depth, and forwards every other attribute. Over MAX_XML_DEPTH (100) or MAX_XML_NODES (5,000,000) it raises before xmltodict's handler runs. Both xmltodict.parse() call sites turn that into InvalidAggregateReport naming the limit; the first does so instead of trying lxml recovery. The lxml recovery path now runs a counting pass first, with a parser target that builds no tree and counts elements, attributes and namespace declarations against MAX_XML_NODES; the tree is built only for a document within the limit. Depth there is bounded by libxml2's own limit of 256 (huge_tree stays off) and then by the second xmltodict pass. A 22 MiB wide document behind a malformed XML declaration peaked at +1532 MiB before this pass and +23 MiB after. The largest aggregate sample (909 KB) has 32,019 nodes, so a legitimate report at the 100 MiB cap stays near 3.6 million. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Create Elasticsearch/OpenSearch indexes with mapping guards create_indexes() and create_data_streams() created indexes and the data stream template with settings only, so the cluster typed every report field from the first document that held it, with date_detection on. A sender could be first into a day's index (its name comes from the report) and poison it: a report_id of "2026-11-09" mapped report_id as a date, a failure sample header named "Subject.x" mapped sample.headers.subject as an object, and samples with invented header names used up index.mapping.total_fields.limit. Every later legitimate report for that index was then rejected and ended up in Unsaved; in data stream mode this lasted until a rollover. Both functions now take a mappings body, and the three save functions pass one per report type. Each turns date_detection off and declares the fields that hold dates (@timestamp, date_range, date_begin, date_end, arrival_date, sample.date) as dates, the type date detection gave them before; on the sample reports every other field keeps its old type. The failure mapping maps sample.headers as an object with dynamic: false and declares, as text with a keyword sub-field (the dynamic string default), every header the shipped dashboards reference plus date, message-id and cc; all headers still stay in _source. A "." in a header name is replaced with "_" before the header is stored. The index is now created through the client's indices.create with the settings and mappings body, because opensearch-py's Index.create cannot take a mappings argument; the data stream template merges the mappings with its @timestamp date mapping. Existing indexes are not changed. The other report fields stay dynamic and object-mapped (no nested), which the dashboards need (issue #169). Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read the RFC 8460 mx-host key in SMTP TLS reports RFC 8460 section 4.4 keys a policy's MX host patterns "mx-host"; its schema line is "mx-host": mx-host-pattern, where mx-host-pattern is the value type, described as "a JSON array of strings", and the RFC's Appendix B example gives the value as a plain string. The parser read only "mx-host-pattern", so every RFC-conformant report lost the field (samples/smtp_tls/rfc8460.json, the Appendix B example, among them). _parse_smtp_tls_report_policy() now reads "mx-host" as a string or a list of strings, a string becoming a one-element list, and falls back to "mx-host-pattern" (same rules) when "mx-host" is absent, null or empty. Any other type raises InvalidSMTPTLSReport. The output key is still mx_host_patterns, so the PostgreSQL TEXT[] column, the Elasticsearch/OpenSearch field and the CSV column are unchanged. Refs #942 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject over-limit messages with the mailsuite guard parse_report_email() and parse_email() handed every message to mail-parser and the email package with no bound on blank lines, header lines, header field size, MIME parts or nested multiparts, so a small crafted message took seconds to minutes and gigabytes to parse while the rest of the mailbox waited. Both now call mailsuite.utils.check_message_limits() first, one linear pass. parse_report_email() turns MessageLimitExceeded into InvalidDMARCReport so the mailbox loop files the message under Invalid; parse_failure_report() turns one from the sample into InvalidFailureReport. The mailsuite floor rises to 2.3.5, which adds the guard. Messages nested deeper than 100 multiparts are now rejected by the guard instead of reaching the recursion handler; the tests that used the 400-level message expect the new message. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Cite RFC 7489 §6.3 for the daily report interval The comment on MAX_REPORT_SPAN_DAYS cited RFC 7489 Appendix A for the one-day aggregate report. The daily interval comes from §6.3: the "ri" tag requests the interval between aggregate reports, and its default is 86400 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the 11.2.2 changelog: five advisories and #942 One Security bullet per advisory (GHSA-qcc2, GHSA-q7gm, GHSA-p85v, GHSA-4272, GHSA-qpcg) and a Bug fixes bullet for the RFC 8460 mx-host key. The version in constants.py is bumped by the release pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the date range rejection message and pyright's analysis With the XML limits and the date range limit merged, parse_aggregate_report_xml() became too complex for pyright to analyze ("Code is too complex to analyze"), and the date range rejection, raised inside the function's catch-all, reached the caller as "Unexpected error: Report date range spans ...". The date range check moves to _check_report_span() and the second xmltodict pass to _parse_limited_xml(), which raises the XML limit breach as InvalidAggregateReport; the catch-all now re-raises an InvalidAggregateReport as it is. The tests assert the message is not re-wrapped. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Put the over-limit failure samples in a delivery-status body mailsuite 2.3.5's check_message_limits() follows the MIME structure and counts blank lines only inside message/delivery-status bodies, the shape whose parse cost is per blank line; blank lines elsewhere in a body are not limited. The two failure-sample tests used blank lines in a plain body, which the final check accepts, so they now put them in a delivery-status body. parsedmarc's call with the default limits is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe the final mailsuite check and the index cap in the docs The GHSA-qcc2 changelog bullet now describes check_message_limits() as mailsuite 2.3.5 ships it: limits follow the MIME structure, and body lines outside message/delivery-status bodies are not limited. The changelog section moves to Unreleased, as parsedmarc's AGENTS.md has fix pull requests write it; the release pull request names the version. The Elasticsearch and OpenSearch pages now say that an address header with no address is left out of the failure document, and that a report spanning more than 31 days saves its later records to its first dated index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound records times days in normalization and fix the span message The date range limit bounds how far one record is split, but not how many records are split: 10,000 records over 366 days, each with a count of 1,000,000 (3.3 MB of XML), produced 3,670,000 records in 13.5 s and 1.7 GB, and 30,000 of them (a 44 KB gzip) ran out of memory and reached the caller as "Unexpected error: ". A report whose split would exceed MAX_NORMALIZED_RECORDS (250,000) now keeps its records whole over its own range, with timespan_requires_normalization false and one warning; a large legitimate multi-day report is kept rather than rejected. The record loop moves to _parse_report_records() so pyright can still analyze parse_aggregate_report_xml(). The span message used float division, which raises OverflowError for an <end> of 4,001 digits; it now uses integer division. The index-routing warning escapes the sender's report ID with sanitize_for_log(). The changelog and the Elasticsearch and OpenSearch pages say the index cap is per report, that dated index names still come from report dates, and that data streams are affected by neither. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop XML attributes from aggregate reports before they are used xmltodict turns an element with an attribute, such as <p a="1">none</p>, into {"@a": "1", "#text": "none"}. Such objects passed through the parser into Elasticsearch and OpenSearch documents for version, p, fo, disposition, envelope_to and other scalar fields, so the first one mapped the field as an object and every later report to that index failed, and hundreds of attributes on one element used up total_fields.limit. The parsed report now goes through one linear pass that replaces an element with only attributes and text by its text and drops attributes from the others; the existing _text() calls now receive strings. The changelog and the Elasticsearch and OpenSearch pages also say that an existing data stream's next backing index gets the guarded mappings, since parsedmarc puts the template again in every process, and the indexed-header comment and changelog name cc as the one header added to those the dashboards reference. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject lone surrogates in SMTP TLS reports and guard the stdout print An SMTP TLS string field holding a lone surrogate (JSON "\ud800") passed every type check and survived json.dumps(ensure_ascii=False), and print(output_str) in process_reports(), outside every try, raised UnicodeEncodeError: parsedmarc exited 1, and in mailbox mode the error escaped the save callback. RFC 8460 section 4 reports are I-JSON, whose RFC 7493 section 2.1 forbids surrogate code points, so the string and string-list checks now reject one as InvalidSMTPTLSReport, and the print records a UnicodeEncodeError as a "Standard output" output error. The _smtp_tls_datetime() docstring no longer says a parsed date cannot fail in any output: dateutil accepts partial dates that PostgreSQL may reject, which is contained as PostgreSQLError. The _reason_text() docstring says a dict with child elements gives None. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe mailsuite 2.3.5's address group handling in the changelog The final mailsuite 2.3.5 no longer rejects an address header by its comma count; its parse_email() removes RFC 5322 group syntax from address header values over 8 KiB before parsing them. The boundary limit counts lines that are the boundary of an open multipart. parsedmarc's call with the default limits is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the keep-alive call of the extracted record loop _parse_report_records() holds the record loop moved out of parse_aggregate_report_xml(); its keep-alive branch had no test. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a policy override type with child elements With XML attributes dropped at parse time, a dict reaches _reason_text() only for an element with child elements; the test pins its None result and the CSV's "none". Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check plain-string contact-info and mx-host for lone surrogates The surrogate check covered the string fields and string lists, but a contact-info or mx-host given as one string took a branch that skipped it, and a lone surrogate there still made the stdout print raise. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which sample headers the failure document leaves out The GHSA-qpcg changelog bullet said every sample header stays in _source, but the failure save leaves out a From, To or Reply-To header that holds no address, as the Elasticsearch and OpenSearch pages say. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep sender-derived numbers out of the date range message The rejection message printed the report's span in days, so an <end> of thousands of digits put a number of thousands of digits into the exception text and the log. It now says only that the range is negative or longer than MAX_REPORT_SPAN_DAYS. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Store every string-typed aggregate field as a string or null Dropping attributes did not cover child elements: <version><x>1</x></version>, <p><x>1</x></p> or <p>none<x/></p> still reached the parsed report, and the Elasticsearch and OpenSearch documents, as a dict, and the first such document mapped the field as an object. _scalar() now makes each field the schema types as a string a string (an element's own text) or None, applied by _scalarize_report() to version, the report_metadata strings and every policy_published child, and by _parse_report_record() to source_ip, disposition, dkim, spf, the identifiers and every auth_results string. A <count> with no text of its own takes the ValueError path of a non-numeric count. _drop_xml_attributes() stays, so a field missed here still loses its attribute keys. The <email> dict unwrap is gone, since the field can no longer be a dict; a text-only report_metadata still raises the AttributeError that reports a missing section. Comments that described attribute dicts now say the dict shape comes only from child elements. The namespace test, which the new code could not fail, is replaced by one that detects RFC 9990 through an attributed np field. Tests walk the to_dict() of a mocked Elasticsearch and OpenSearch save and allow lists and objects only in the known object fields, so a scalar field added later is covered too. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record a closed stdout pipe as an output error and fix two comments The standard-output guard in process_reports() caught only UnicodeEncodeError, so a closed pipe (BrokenPipeError, an OSError) still ended the run; it now records both as a "Standard output" output error. _reason_text()'s docstring said a dict gives None because the element has no text of its own, but mixed content (<type>forwarded<x/></type>) gives the text beside the children; the docstring now says what the code does, and a test pins it. The comment above the reason normalization no longer describes attribute dicts, which no longer reach it. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the normalization limit at its boundary and reword the index cap The suite passed with the limit's <= changed to < and with _days_touched()'s ceiling changed to a floor. New tests: 1,000 records over 250 days (exactly MAX_NORMALIZED_RECORDS) are split; 683 records over 366 days (249,978) are split and 684 (250,344) are not; 682 records over 366 days starting 1 s after midnight touch 367 days (250,294) and are not split. Each mutant was run against these tests and fails: the < mutant fails the exact-limit test, the floor mutant the after-midnight test. The Elasticsearch and OpenSearch pages now say the index cap applies when a report's records would use more than 31 dated indexes, which is what it counts (with monthly_indexes the span limit keeps a report under it). Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Match the tests and changelog to mailsuite 2.3.5's final check mailsuite 2.3.5's check_message_limits() now parses the message with the email package's own feed parser under compat32 and counts the parts it creates (each message/delivery-status block among them), their nesting depth and its boundary tests, then limits header field length and count on the parsed tree; the limit names changed. The over-limit failure samples now trip max_parts, and the deeply nested poison message max_depth; the tests assert those texts. The GHSA-qcc2 changelog bullet describes the final check, the epilogue mail-parser parses again, and the changes that affect only mailsuite's own parse_email(). Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a repeated policy_published and note mixed-content behaviour No test failed when _scalarize_report() stopped coercing the fields of a repeated <policy_published> (the list case): with that mutant, a report with two policy_published elements and <p><x>1</x></p> puts a dict into published_policy.p. The parse test and the Elasticsearch and OpenSearch document-walk tests now include that report; each fails on the mutant. The changelog says that an element with text and a stray child element now yields its text (samples/aggregate/invalid_xml.xml gives header_from "bad" where it gave ""). AGENTS.md and the _normalize_result_word() and _scalarize_report() docstrings describe what runs now: attributes are dropped as soon as the XML is parsed, and _scalar() coerces the string fields, among them the RFC 7489 and RFC 9990 policy_published fields. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Coerce policy override reasons with the same rule as other strings _reason_text() turned a repeated <type> or <comment> into the str() of the list, while _scalar() turns every other repeated string field into None. One class of input now has one rule: the reason's type and comment go through _scalar(), so a repeated one is None ("none" in CSV), and _reason_text() is removed. The test and the GHSA-4272 changelog bullet say so. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test an aggregate <error> with a child element With attributes dropped and reasons coerced by _scalar(), only <error>, which may repeat, still reaches _text() with a dict; its branch had no test. An <error> with text beside a child element keeps the text, and one with no text is dropped. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe mailsuite's six message limits in the changelog mailsuite 2.3.5's check now counts every line of every header block (folded lines and lines the parser keeps only as defects included) and stops the parse early, and limits the total length of the address headers mail-parser parses to 262,144 characters. The GHSA-qcc2 bullet names the six limits. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render a report email's parts only where their text is read parse_report_email() did str(payload_obj[0]) for every part walk() visits, before the multipart/ continue. str() of a composite part renders its whole subtree, and walk() visits every level, so a nesting within mailsuite's limits rendered its leaf once per level: 98 nested multiparts around a 20 MiB one-line leaf took 26.6 s, and a chain of 99 message/rfc822 parts around a 10 MiB leaf 11.3 s. A part is now rendered only in the branches that read it; other message/* bodies are skipped, since walk() visits their parts. Both shapes now take under 0.5 s, and every sample under samples/failure/ parses to the same output. A feedback report's sample is taken from the first sample part only: a message/rfc822 nested inside the sample used to replace it, but a feedback report has one sample (RFC 6591 section 3.1). Multipart nesting alone no longer reaches the recursion limit, so the GHSA-qxwv test now reaches the recursion guard through a deeply nested sample, which is still rendered. The changelog describes this and the address headers mailsuite 2.3.5 counts (From, Sender, Reply-To, To, Cc, Bcc, Delivered-To, in every part; mail-parser parses all but Sender). Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Parse aggregate XML without attributes and lower the node limit _drop_xml_attributes() rebuilt the parsed tree while the original was alive, and the counting wrappers of _LimitedExpatParser referred to the proxy, closing a reference cycle through the expat parser that kept xmltodict's tree alive until the cyclic garbage collector ran. Both xmltodict passes now use xml_attribs=False, so no attribute key is ever built (expat still hands the attributes to the start handler, so they are still counted: a test shows one element with 1,000 attributes is 1,001 nodes), _drop_xml_attributes() is removed, and the counts live in an _XMLCounts object the wrappers share, so a parse leaves no cycle (a test asserts gc.collect() finds nothing after one). Peak extra memory for 2,000,000 distinct sibling elements: +999 MiB before, +386 MiB after (master +391 MiB); for one element with 2,000,000 attributes: +1,274 MiB before, +445 MiB after (master +734 MiB). At the old limit of 5,000,000 nodes a report could still need about 3 GB. Measured after the change, the worst shape is distinct elements with text behind a malformed XML declaration (the lxml recovery path): 559 bytes per node, +533 MiB for 999,000 nodes and +1,023 MiB for 2,000,000; on the expat path about 220 bytes per node. MAX_XML_NODES is now 1,000,000, so a report at the limit needs at most about 550 MiB; the largest sample has 32,019 nodes (909 KB), and the limit is a report of about 28 MB. The constant's comment and the changelog state the bound. AGENTS.md describes the attribute handling as it now is, and says that <error>, which may repeat, goes through _text() and stays a list. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qualify the mixed-content compatibility note to string fields An element with text beside a stray child element yields its text only for the fields the schema types as strings; a <begin> or <end> with mixed content is still rejected, as before. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject over-long XML tags before any parser reads them expat and libxml2 build a start tag's whole attribute list before they report the element, so the node counter only sees the attributes after they cost memory: one element with 4,999,000 attributes (56 MiB) took 5.25 s and +767 MiB before the node limit rejected it, and a single element is bounded only by the 100 MiB decompression cap. parse_aggregate_report_xml() now scans the decoded text once, before the first parse and so before the lxml recovery path too, and rejects a document in which any text between a "<" and the next ">" (or the end of the text) is longer than MAX_XML_TAG_LENGTH (65,536 characters). The scan uses no regex. It splits the text on ">" one 64 KiB chunk at a time, carrying the unfinished piece from its last "<": a single split of the whole text makes one string per tag, which peaked at +1,318 MiB (1.9 s) for 100 MiB of "<a/>"; the chunked scan needs about 2 MiB and takes 0.6 s on that text, and a test bounds its memory and covers a tag across a chunk boundary. The longest tag in the samples is 108 characters. The same shape is now rejected in under 0.1 s, with a tracemalloc peak under 200 MB in the test, behind a well-formed or a malformed declaration. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render a report email's feedback-report part only once The message/* skip exempted message/feedback-report, which was rendered with str() every time walk() reached one: a chain of 99 nested message/feedback-report parts around a 10 MiB leaf passed mailsuite's check in 0.02 s and then took 11.6 s. A feedback report has one message/feedback-report part (RFC 6591 section 3), so, as with the sample, only the first is read and rendered; the chain now takes 0.56 s. The failure samples parse to byte-identical output. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound attribute counts by their = signs and cap reports at 32 MiB XML allows a raw > inside a quoted attribute value, so attributes such as x=">" cut a tag into short pieces that pass the tag scan: 4,999,000 of them cost 5.61 s and +781 MiB, and 8,000,000 (98 MiB) 8.9 s and +1,215 MiB, before the node limit fired. Every attribute needs an =, so a document with more = characters than MAX_XML_NODES is now rejected in one str.count() pass, before the tag scan; the samples hold at most 3. The tag scan stays for long tags, and its docstring says it does not bound attribute counts. The stated memory bound was false: text per node is bounded only by the size of the report, so at the node limit with 80 characters per element (94 MiB) the recovery path reached +827 MiB. MAX_DECOMPRESSED_REPORT_SIZE falls from 100 MiB to 32 MiB (the node limit already means a legitimate report is at most about 28 MB, at the samples' 35 nodes per KB, and the largest sample is 909 KB), and parse_aggregate_report_xml() holds an uncompressed report to the same length, so the bound holds for every input. Measured at the cap, 999,000 distinct elements with 14 characters of text (31 MiB) need +587 MiB on the recovery path and +302 MiB on the expat path; the constant comment and the changelog state it. When the first parse fails, the exception's traceback kept the partial xmltodict tree alive while the recovery parse ran; it is cleared first (+603 to +587 MiB, and +263 to +241 MiB for 330,000 elements of 80 characters). A test shows the recovery path leaves the same 18 lxml helper objects for a document ten times larger, so no tree survives a parse there either. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let the RFC 6591 feedback part win over legacy report text The guard that reads only the first message/feedback-report part tested feedback_report, which the legacy text/plain branch ("A message claiming to be from you has failed ... detected.") also sets, so a legacy text part followed by a real feedback-report part and sample now took the legacy part and never read the ARF, where master read the ARF. The guard now tests is_feedback_report, and reading the feedback-report part discards a sample a legacy part set before it. The same rule closes an older gap: a text/plain part inside the sample, which is the spoofed message's own text, could hold the legacy phrase and replace the report's Source-IP and its sample. Once a feedback-report part is read, the legacy branch is skipped; a legacy report has no such part. Tests cover both orders and the in-sample case; the failure samples parse to byte-identical output. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold every report input to the size cap and free failed parses parse_report_file() kept the aggregate parser's exception, whose traceback holds the parser's frames and the parsed tree, while it tried SMTP TLS and then the email parser: a 31 MiB file at the cap cost +1,046 MiB there against +587 MiB parsing it directly. It now keeps only each failure's text and raises from a fresh exception of the relevant format, so the chain survives without the tree. A file that begins with "<" is not tried as JSON (json.loads() decoded it all) or as an email, one that begins with "{" is not tried as an email (an RFC 822 message begins with a header line), and text over the size cap is never handed to the email parser, which has none: a 200 MiB <x/> file reached +2,801 MiB in 4.6 s and now costs +232 MiB (reading the file) in 0.05 s. extract_report() reads an uncompressed report at most one byte past the cap instead of decoding it whole. The cap counted characters for text and bytes for compressed reports, so a non-ASCII report could be four times the cap in bytes: 999,000 elements of 14 emoji (31.2 Mi characters, 71 MiB of UTF-8) cost +649 MiB. _exceeds_size_cap() measures text as UTF-8 (ASCII by length, other text 1 MiB of characters at a time, without a full copy), and bytes are checked before they are decoded. Clearing e.__traceback__ inside the except block did nothing on Python 3.10, which keeps the handled exception's traceback while the handler runs. The lxml recovery now runs after the except block, in _recover_xml(), so the first parse's frames and partial tree are unreachable when it starts on every supported Python; a test holds a weak reference to the first parse's counter and fails on the old code under Python 3.10.19 and, with the clearing removed, under 3.14. The changelog's examples are re-measured under the 32 MiB cap, and the test docstrings cite the same figures. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip the parts inside a failure report's sample Once the sample part was taken, walk() still visited its descendants, and they reached the SMTP TLS and base64 attachment branches: a sample whose spoofed message carried an SVG or zipped HTML attachment made the email rejected ("Missing field: 'feedback'", "Unable to parse"), and a gzipped aggregate report or a tlsrpt+json attachment inside it was returned as that report instead of the failure report. The sample is the reported message, so after it is taken every part inside it is skipped (by the id() of each part its walk() yields). Tests cover SVG, zipped HTML, JSON, gzipped aggregate XML, tlsrpt+json and PDF attachments; the failure samples parse to byte-identical output. The Bug fixes bullet also says that a legacy text part followed by a feedback-report part with no sample after it is now invalid, with a test. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cap SMTP TLS reports and test the format skips parse_smtp_tls_report_json() decoded and json.loads()-ed a report of any size; only the gzipped path was capped, and 60 MiB of "[]," cost +1,565 MiB. It now rejects bytes or text over MAX_DECOMPRESSED_REPORT_SIZE first, as InvalidSMTPTLSReport, so every report type is held to the cap. parse_report_file() names the actual reason it skips the email parser (the leading "<" or "{", or the size cap) rather than a combined one, and a size message for bytes no longer says "of UTF-8". New tests kill the mutants the review found surviving: removing the "<" skip for JSON, narrowing the email skip to "<", the bytes check >= for >, and removing the SMTP TLS cap. The constant comment and the changelog state the memory at the cap as measured on the current code: +345 MiB on the lxml recovery path and +302 MiB on expat parsing the text directly, +500 MiB and +452 MiB through parse_report_file(). The changelog says only the exception's cause changed for files skipped by format, not the message. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip every sample candidate and correct the stated measurements sample_parts held only the first sample candidate's parts, so a later message/rfc822 (or one after a text/rfc822-headers sample) was dropped as a sample but its parts were still walked, and a gzipped aggregate report inside it was returned in place of the failure report. Every candidate's parts are now skipped. Tests cover both shapes, and a test that a gzipped aggregate report beside the sample is still returned pins the rule (inside the sample: skipped; beside it: read) and fails when the set is built from the whole message. parse_report_file() no longer builds email reasons for content that begins with "<" or "{", which its message never shows, and the size reason reads "Invalid report email: the report is larger than 32 MiB". The memory figures are re-measured in a fresh process with the input read from disk before the baseline: +399 MiB on the recovery path and +338 to +355 MiB on expat for the 31 MiB shape, +1,505 MiB for 60 MiB of JSON before the SMTP TLS cap; the constant comment, the changelog and the test docstrings state them. RFC 6591 section 3.1 is cited for the sample's place, and application/json is named among the attachments that made a failure report invalid. Refs GHSA-qcc2-4x3v-v4g7 Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Scale the timed tests' bounds to the machine running them The mailsuite fix's first CI run on GitHub's shared runners failed six timed tests: the suite ran four to six times slower there than on a workstation, and checks doing real work took longer than their fixed bounds. Several bounds here have far less margin (0.2 s on a test that takes 0.18 s whole), so they would fail the same way. tests/timing.py adds budget(seconds), which measures a fixed reference parse twice just before the timed call and multiplies the bound by the slowdown against a reference machine; a faster machine keeps the written bound. Every timed assert now compares against budget(<written seconds>) computed before the timer starts. No written bound changed and no assert was removed. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]andseanthegeek
|
19512d7c8b |
chore: update IPinfo Lite MMDB (#943)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
github-actions[bot]andseanthegeek
|
bc17e81c59 |
chore: update IPinfo Lite MMDB (#941)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
7bbc405c01 |
Release 11.2.1 (#940)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>11.2.1 |
||
|
Sean WhalenandClaude Opus 5.5
|
33ca4ecf71 |
Merge commit from fork
With [smtp] ssl = False (the default) or [imap] ssl = False, mailsuite upgraded the connection only if the server's plain-text EHLO/CAPABILITY reply advertised STARTTLS, and otherwise authenticated in plain text. An on-path attacker who stripped STARTTLS received the credentials and the emailed results (CWE-319). - Require mailsuite >= 2.3.4 (GHSA-2x4j-7g3q-wr6f), whose default refuses to authenticate over a connection that was not upgraded. - Add [imap] and [smtp] starttls (unset = require STARTTLS; false = explicit plain-text opt-out, for loopback servers such as DavMail), with a startup warning when the opt-out is set and ssl is False. - Add a starttls argument to email_results(). - Document both options; set starttls = False in the DavMail example. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
dc3b14dc0e |
Release 11.2.0 (#938)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>11.2.0 |
||
|
Sean WhalenandClaude Opus 5.5
|
2986c58b0f |
Redact httpx2 URLs whenever httpx2 is installed (#939)
_HTTP_URL_CLASSES took httpx2 from dns.query, so it only matched httpx2.URL when dnspython 2.9+ had imported httpx2. dnspython 2.9 requires Python >= 3.11, so on Python 3.10 CI gets dnspython 2.8 while httpx2 is still installed (hatch in the [build] extra requires it), and httpx2.URL arguments on the httpx2 logger went unmasked. Three TestHttpx2LogRedaction tests failed on 3.10 as a result. httpx2 is now imported directly behind a ModuleNotFoundError guard. The new tests re-execute cli.py with dns.query.httpx2 removed and with httpx2 unimportable, so the regression is caught on every dnspython version. Also drops the import of dns.query, which nothing uses now. Follow-up to GHSA-hjr3-rx4c-24cv (unreleased). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
48ea47adee |
Merge commit from fork
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv) With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init lines and opensearch.set_hosts() logged configured URLs verbatim, so userinfo passwords reached the console and log_file (CWE-532). Triage found more paths of the same class: - the httpx "HTTP Request:" line, forwarded at --verbose/--debug, contains the full URL (str(httpx.URL) keeps userinfo and query), so webhook and HEC URL credentials and the IPinfo ?token= were logged on every request; - the PSL overrides, IP database and reverse DNS map downloaders logged their URL at DEBUG and str(HTTPStatusError), which embeds the full URL, at WARNING; - libpq echoes the DSN or a password fragment for some malformed PostgreSQL connection strings, logged as an output-client error. Add redact_url_credentials() (string methods only; urlsplit misparses scheme-less "user:pw@host:port") and use it at each log site; add a filter on the httpx logger that masks userinfo and the token query value; build download-failure warnings from the status code and the redacted request URL instead of the exception text; replace psycopg ProgrammingError detail from a connection_string with a fixed message. Docs now recommend separate user/password settings over URL credentials. Reported by helixnebulatech. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Mask credentials in the Loki URL in the log and LokiError text The Loki output (#926) landed on master after this fix was written. Its url keeps userinfo (copy_with(path=...) preserves it), and the --debug "Initializing Loki client" line and both LokiError messages, which the CLI logs on a failed push, included it as-is: the same CWE-532 path as the Splunk HEC line. Mask it with redact_url_credentials() at all three sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532): - httpx.InvalidURL quotes the component it rejected, so a password with an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there) leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats it. Loki, Splunk HEC, the webhook and the three download loaders now report a fixed "details withheld" message raised from None. InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url crashed startup; the loaders now treat it as a download failure. - redact_url_credentials() masks through the last "@" when the authority has no "@" but one follows it, so the --debug lines no longer print such passwords; the userinfo lookup is shared with the new nameserver masking. - dnspython 2.9 DoH logs its request line on the httpx2 logger, which was forwarded but not filtered; the filter now covers httpx2 and httpx2.URL. - [general] nameservers / --nameservers credentials are masked in the DNS pre-flight error, reverse DNS debug errors and per-report errors, and unusable https:// / tls:// ports are rejected before any query. - Docs recommend token/username settings for [loki] and [splunk_hec]; AGENTS.md gains rules for keeping credentials out of logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
56e5a09923 |
Add opt-in Elasticsearch/OpenSearch data streams (#937)
* Add opt-in Elasticsearch/OpenSearch data streams (#842) New use_data_streams option in [elasticsearch] and [opensearch] saves each report type to a data stream named like today's index without the date suffix, instead of daily or monthly indexes. parsedmarc puts an exact-name index template (priority 501, @timestamp mapped as date, <stream>@custom component template on Elasticsearch) and creates the stream; retention is left to operator-managed ILM/ISM policies, with documented examples. Dated indexes are untouched and still searched for duplicates and matched by the dashboards. Every document now carries @timestamp and is saved with op_type=create in both modes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix data stream prose found by cross-check review (layer 1) Aggregate @timestamp is each record's begin date, not the report's; the cluster only rolls backing indices over under an attached policy; move the Elasticsearch 8.7 note after the template it refers to; clarify the name-collision paragraph, OpenSearch-side comments, the migrate_indexes docstring, and the test-module headers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tighten names and wiring tests found by cross-check review (layer 2) Split the use_data_streams CLI passthrough test per backend so a call site reading the other section's flag fails; name the dated-mode tests for both changes they assert; compute index_date only in dated mode in every save function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Align data stream docs with Elastic/OpenSearch references (layer 3) Say the stream must exist before reindexing into it (ES reindex API: auto-creation needs a data-stream template); an existing stream's write index needs the ILM policy attached directly since nothing rolls it over otherwise; target the stream name in the _settings example; note OpenSearch's leading-dot rule for stream names and Fleet's template priorities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cover the Loki client-initialization error wrapper _parse_config accepts a [loki] token alongside username/password; only the LokiClient constructor rejects the combination. Exercise that path through _init_output_clients with the real constructor so the RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
70f96d0734 |
Fix DNS over HTTPS under dnspython 2.9 and the misleading pre-flight error (#935) (#936)
* Fix DNS over HTTPS under dnspython 2.9 and the pre-flight error (#935) dnspython 2.9 moved its [doh] extra from httpx to httpx2, and dns.query.https() rejects a session of any other class before sending anything (dnspython 2.9.1 dns/query.py:551: `if session and not isinstance(session, httpx2.Client): raise ValueError("session parameter must be an httpx2.Client")`; 2.8.0 checks httpx.Client at :566). _get_doh_session() always built an httpx.Client, so every https:// nameserver failed and the resolver retried it until LifetimeTimeout. The shared DoH client is now built from the Client class of the module dnspython itself imported (dns.query.httpx2 on 2.9+, dns.query.httpx on 2.7/2.8), so it matches dnspython's isinstance check even when httpx2 is installed beside dnspython 2.8. httpx2 keeps httpx's `allow_env_proxies = trust_env and transport is None` (_client.py:658), so the #880 proxy behavior carries over. The nameservers pre-flight check called get_reverse_dns(), which turns every DNS error into None, so the failure above was reported as "no PTR record for 1.1.1.1". It now calls query_dns() directly and reports "no PTR record" only for NXDOMAIN/NoAnswer, and the resolver's actual error otherwise. Reported by florianutz. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review of the DNS over HTTPS fix (#935) - Add the httpx2 and httpcore2 loggers to the dependency loggers so --debug shows DoH traffic under dnspython 2.9+. - Report an invalid dns_test_address by name and value in the pre-flight check instead of "Text input is malformed.", without attempting a query. - Rewrap an over-long docstring line in _nameservers_to_resolver_input. - Soften the test docstring about the no-[doh]-extra fallback. - Correct the pyproject comment on what the [doh] extra installs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rewrap the dnspython [doh] comment in pyproject.toml (#935) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
768bf75480 |
Bastischubert/loki output (#926)
* feat: add grafana loki output Push aggregate, failure and SMTP TLS reports as JSON log lines to the Loki push API. Static labels and labels taken from report fields are configurable, host defaults to the FQDN. The timestamp is either the ingest time or the report begin time. Supports basic auth, bearer tokens, X-Scope-OrgID and batching. Push errors are reported as output errors so messages are not archived. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * feat(dashboards): add grafana dashboard for the loki output Signed-off-by: Sebastian Schubert <basti@schubert.digital> * Address Copilot review on the Loki output Responds to Copilot's review of #926: - Treat any push status other than 200/204 as a failure. Loki returns block_ingestion_status_code (260 by default) when block_ingestion_until blocks ingestion, and the logs are discarded (Loki configuration reference); the old >= 300 check counted that as a successful save, so the CLI could archive mailbox messages whose reports were never stored. - Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate CSV rows carry the record's interval_begin, which the parser builds as naive UTC wall-clock time (to_utc=True); reading it as local time shifted entries by the host's UTC offset (+4h on America/New_York). - Group the Published Policies dashboard dedup by org_name as well as report_id: report IDs are only unique per reporter (the PostgreSQL key is UNIQUE (org_name, report_id, ...)). - Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback with the automatic step yields overlapping rolling totals, so the sum counted the same sessions repeatedly. Adds regression tests for each, including a non-UTC timezone test against a real sample report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address second Copilot review on the Loki output Responds to Copilot's second review of #926 (previously-missed findings): - Strip each [loki] label_fields entry. _str_to_list() only lstrips, so "report_type , domain" produced "report_type " and LokiClient refused to start with "Invalid Loki label name 'report_type '". - Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat inside the ES-has-docs guard, so after the ES container was recreated a reseed appended a second copy of the samples to Loki, which does not deduplicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address third Copilot review on the Loki output Responds to Copilot's third review of #926 (previously-missed findings): - Keep a base URL's query string when appending the push path. The path was string-concatenated, so https://gw.example.org?route=logs became ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted query. The path is now set on the parsed httpx.URL. - Push each report's entries in timestamp order. With timestamp = report_begin, a normalized multi-day aggregate report yields rows source-major (every day of one source, then day 1 of the next), all in one stream. Loki rejects entries older than the newest line minus half of max_chunk_age (1h by default) with too_far_behind (Loki configuration docs, "Accept out-of-order writes"), so later sources were partly rejected. The whole entry list is now sorted before batching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(loki): do not follow redirects and drop forensic alias httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError. Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * fix(dashboards): use the existing Loki datasource uid in the dev bootstrap When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead. Signed-off-by: Sebastian Schubert <basti@schubert.digital> --------- Signed-off-by: Sebastian Schubert <basti@schubert.digital> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
2f89507e5d |
Fix aggregate report deduplication across policy domains (#934)
* Fix aggregate report deduplication across policy domains Scope aggregate-report identity to the reporting organization, Report-ID, and policy domain, matching RFC 9990 section 3.5.1. Encode the fields as a JSON array and compare ASCII domain letters without changing returned report data. Preserve accepted empty-domain reports and save-confirmed cache semantics. Add real MIME/mbox and dry-run Maildir regressions for distinct domains, same-domain repeats, shared configurations, boundary collisions, parallel parsing, and false/raising save callbacks. The implementation and tests were prepared with AI assistance, including a separate review of the scoped changes and test evidence. * Share the domain-scoped dedup key with the CLI direct-file path Responds to Copilot review on #934: the CLI's direct-file path still wrote `{org_name}_{report_id}` keys into the same seen-aggregate-report cache that the mbox/mailbox classifier now reads with a JSON [org, report_id, policy_domain] key, so a report passed both as a file and inside an mbox in one run was saved twice, and direct files from one reporter sharing a Report-ID for different domains (RFC 9990 §3.5.1) were still collapsed. Extract the key into `_aggregate_report_dedup_key()` and use it from both paths; the direct-file path now writes through `parser_config.seen_aggregate_report_ids` like the mbox path. Adds end-to-end `_main()` regression tests for mixed file+mbox input and for domain-scoped direct-file dedup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
7d4661c2af |
Merge commit from fork
save_report_to_s3() put the sender-controlled report ID into the object
key and org_name/org_email/report_id into user-defined metadata verbatim.
botocore rejects non-ASCII metadata client-side (ParamValidationError) and
a bare CR/LF with "Invalid header value", which the CLI counts as an S3
output error, so one report held back its whole mailbox batch. "/" and
".." in the report ID added key segments, and keys had no sender identity.
- Percent-encode the report ID into one key segment ("/", the AWS
"characters to avoid", "%", control and non-ASCII characters) and
shorten it so the key stays within S3's 1,024-byte limit.
- Percent-encode metadata values (reversible with unquote) and cap each at
256 characters, under S3's 2 KB metadata limit.
- Append the first 16 hex characters of a SHA-256 over the report type and
its identifying, report-sourced fields (never enrichment), so reporters
that reuse one report ID no longer overwrite their own reports and a
retry still overwrites the same object.
- Fix failure reports, which raised KeyError on report_metadata (absent
from FailureReport in types.py) since S3 output was added in 7.0.0.
Sources: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html,
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html,
botocore handlers.validate_ascii_metadata.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Sean WhalenandClaude Sonnet 5.5
|
7908907f2c |
Require mailsuite >= 2.3.3 and simplify the Maildir poison-message test (#933)
mailsuite 2.3.3 (GHSA-9jx3-p8wm-g8vw) returns stored bytes from fetch_message() and renames the file in move_message(), so poison messages now always land in the Invalid folder. Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Fable 5.1
|
5a6cf2175f |
11.1.0 release: bump version and finalize changelog (#932)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>11.1.0 |
||
|
Sean WhalenandClaude Opus 5.5
|
6b7c203126 |
Merge commit from fork
* Fix three security advisories (DOCTYPE amplification, poison mailbox message, byte-length filenames) GHSA-m47j-wp78-fqjg: reject any DOCTYPE in aggregate report XML. An internal subset's <!ATTLIST> default attributes are applied by expat to every matching element, so 64 KB of XML (K=1000, N=10000) took 4.0 s and 1.4 GB here. xmltodict now gets an expat shim whose parsers raise on StartDoctypeDecl; the lxml recovery path checks docinfo and serializes only the root element. GHSA-qxwv-2vwj-v828: parse_report_email() now raises only ParserError subclasses (HeaderWriteError / RecursionError from re-serializing a part used to escape). Content errors raised while fetching a message (email.errors.MessageError, RecursionError) are returned as ParserError and the message is disposed of as invalid; transport errors still propagate. A failed invalid-message move/delete is logged instead of aborting the batch, because mailsuite's Maildir move_message() re-serializes the message too. mbox reads use get_bytes() instead of the re-serializing get_string(). GHSA-j53g-34vj-g25q: get_filename_safe_string() bounds names to 100 UTF-8 bytes as well as 100 characters; filesystems limit a name to 255 bytes, so 84 CJK characters + ".eml" failed with ENAMETOOLONG. Reported by helixnebulatech. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: recursion margins, mbox wording, DOCTYPE check order - Nest the RecursionError poison 400 levels deep (POISON_NESTING_LEVELS) instead of 250: the smallest failing depth is ~247 from a shallow stack, 245 in a fork worker and ~222 at +100 frames, so 250 left a margin of a few levels. The shallow case uses 50 levels and now asserts InvalidDMARCReport ("is not a valid report"); the deep case asserts "recursion" in the message. - Poison-loop tests skip a benign poison per message (continue) and assert at least one poison was exercised, so they cannot pass vacuously. - lxml recovery: check for a DOCTYPE before serializing the root, and comment why the str() around tostring(encoding="unicode") stays (pyright, no lxml stubs). - CHANGELOG: describe what the walk actually serializes, drop the depth-dependent 245 figure, and say plainly that an mbox run still stops at such a message, now with a ParserError naming it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make the Maildir poison-message test interpreter-independent Whether MaildirConnection.move_message() can store the control-character message depends on the interpreter: upstream CPython (3.12.9) runs the CVE-2024-6923 generated-header check only in the text Generator used by as_string(), so mailbox.Maildir.add()'s BytesGenerator stores it, while Ubuntu's patched 3.12.3 also checks in BytesGenerator._write_headers and the move fails. The deeply nested message fails the move everywhere. The test now decides per message, by trying the same Maildir round trip, whether the message lands in Archive/Invalid with no mailbox error logged or stays in INBOX with the move error logged; both branches assert the batch completed with the aggregate archived. CHANGELOG and the _dispose_invalid_message docstring now say which messages stay put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 
|
c7a9d294b5 |
fix: restore parent IP-lookup state in pool workers under spawn/forkserver (#930)
* fix: restore parent IP-lookup state in pool workers under spawn/forkserver
parsedmarc.utils keeps the IPinfo Lite REST API token
(_IPINFO_API_TOKEN, configure_ipinfo_api) and the selected IP database
path (_IP_DB_PATH, load_ip_db) in process-global variables, and only
the parent process sets them. Under the fork start method a pool worker
inherits them ("All resources of the parent are inherited by the child
process", https://docs.python.org/3/library/multiprocessing.html), but
under spawn (macOS/Windows default) and forkserver (the Linux default
from Python 3.14) it does not ("The child process will only inherit
those resources necessary to run the process object's run() method").
A worker there started from scratch: its IPinfo API lookups were
silently skipped, and _get_ip_database_path() fell back to the bundled
MMDB instead of the downloaded or cached copy the parent selected, so
enrichment differed from the operator's configuration and from the same
run under fork. The CLI parses file inputs through the process pool
even with -n 1, so every file-based CLI run on such platforms was
affected.
parallel_map() now snapshots both globals in the parent and restores
them in every worker via its pool initializer, so IP enrichment matches
the parent's configuration under every start method. The initializer
previously only reconstructed logging and is renamed _init_worker
accordingly.
Fixes #928
Co-authored-by: ZCode Scout <noreply@zcode.local>
Signed-off-by: NefelibateLG <3453510037@qq.com>
* Maintainer edits: Unreleased changelog, PSL overrides, forkserver coverage
- CHANGELOG: move the entry under "## Unreleased" (feature/fix PRs never
pick a version number) and describe the psl_overrides gap.
- Restore parsedmarc.utils.psl_overrides in pool workers as a separate
initarg, in place ([:] =), never rebound. A fresh spawn/forkserver
worker loads the bundled psl_overrides.txt at import and only switches
to the operator's path/URL on its first reverse DNS map load, so
get_base_domain() calls before that load (the first aggregate report's
org_name, the first source IP's reverse DNS name) used the bundled list.
- Cross-reference _init_worker and cli._utils_globals_snapshot(), which
enumerate the same three utils globals.
- Extend TestWorkerIPState to forkserver (where available) and to the
PSL overrides list; without the restore, both subtests fail with the
worker reporting the bundled list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Signed-off-by: NefelibateLG <3453510037@qq.com>
Co-authored-by: ZCode Scout <noreply@zcode.local>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
Sean WhalenandClaude Fable 5.1
|
16a9016e2a |
Neutralize CSV formulas and escape control characters in logged strings (#931)
* Neutralize CSV formulas and escape control characters in logged strings CSV fields starting with =, +, -, @, tab, or CR get a leading apostrophe (OWASP CSV Injection, CWE-1236). Sender-controlled subject/From and a few aggregate-parser values are passed through the new sanitize_for_log() before they reach error messages and logs. Closes #929 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Sanitize more sender-controlled log values; fix sanitize_for_log docs Escape the SMTP TLS policy-type in its error and org_name/report_id in the duplicate-report debug lines, correct the sanitize_for_log docstring, and drop the unreachable return in parse_report_email(). Closes #929 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Test the From-header escaping with a raw control character mail-parser 4.8.0, which CI resolves, no longer decodes an RFC 2047 encoded word in the From header (4.6.2 did), so the previous test saw the encoded word rather than the CR LF it was meant to observe. A raw ESC in the display name survives both versions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
github-actions[bot]andseanthegeek
|
c1e7662e9e |
chore: update IPinfo Lite MMDB (#927)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
050fa9b0ca |
11.0.5 release: bump version and finalize changelog (#925)
Bump __version__ in parsedmarc/constants.py from 11.0.4 to 11.0.5 and rename the "## Unreleased" CHANGELOG heading to "## 11.0.5". The section holds one bug fix: the PostgreSQL schema now indexes every foreign-key column, fixing the Grafana "Aggregate Report Details" panel timeout (#923). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>11.0.5 |
||
 David UandClaude Opus 5.5
|
65a7bfb7f9 |
Fix "Aggregate Report Details" Grafana panel timing out on PostgreSQL (#923)
PostgreSQL does not create an index on the referencing side of a foreign key (docs, "Foreign Keys"), and create_tables() indexed only two of the schema's seven foreign-key columns. The panel's per-record DKIM/SPF subqueries on record_id therefore ran two sequential scans per row and timed out on a 413k-record database. Index the five remaining foreign-key columns, and add a test that every REFERENCES column in the schema has an index. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]andseanthegeek
|
4d3afac0c9 |
chore: update IPinfo Lite MMDB (#922)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Fable 5.1
|
14aa287dc1 |
Fix ReDoS in failure-report, schema-tag, and timestamp regexes (11.0.4) (#921)
Three regular expressions that run on attacker-controlled input backtracked super-linearly (CWE-1333): GHSA-7grx-mj6g-qxmc (affects 8.2.0 through 11.0.3, reported by mjatin2007): text_report_regex, used by the Exim/cPanel plain-text failure-report fallback, was \s*([a-zA-Z\s]+):\s(.+). The leading \s* and the \s inside the class overlap and the pattern is unanchored, so a whitespace run is retried from every offset and every split: cubic, about 8x per doubling (400 spaces 0.06 s, 1600 spaces 3.3 s on CPython 3.12). A ~100 KB message would stall the watch loop for days and be re-parsed after every restart. The pattern is now anchored to a line start with disjoint classes. GHSA-4h9r-c435-j834 (affects 4.1.7 through 11.0.3, found while auditing every regex in the package): xml_schema_regex was </??xs:schema.*>, whose greedy .*> rescans to end of line for every <xs:schema occurrence: quadratic, 1 MB in 19 s. It runs on every attachment that decodes to "<" after extract_report() has inflated it up to the 100 MiB cap, which 194 KB of gzip reaches. Now </?xs:schema[^<>]*>. This also fixes the greedy match deleting a minified single-line report from the schema tag to the last ">" on the line. human_timestamp_to_datetime() stripped RFC 5322 comments with \s*\(.*\)\s*, quadratic on whitespace (200,000 spaces 18.6 s) and reached through every Date header; filed as a bug fix because MTAs cap header size. Comments are now removed by a depth-counting scan, which is linear and handles the nested comments and quoted-pairs RFC 5322 section 3.2.2 allows (a character-class regex would have regressed them). Adds regression tests sized so the old patterns fail the time bound in minutes rather than hanging CI, a "Regular expressions on untrusted input" section in AGENTS.md, and the 11.0.4 changelog and version bump. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>11.0.4 |
||
 SBINFOandClaude Opus 5.5
|
1c335d951f |
Fix empty "Message sources by name and type" Grafana panel (#920)
The panel nests a terms aggregation on source_type.keyword under one on
source_name.keyword, with no `missing` value on the inner aggregation.
parsedmarc saves a source_name without a source_type for every source
outside the reverse DNS map: get_service_from_reverse_dns_base_domain()
returns {"name": base_domain, "type": None} when the base domain is not
in base_reverse_dns_map.csv (parsedmarc/utils.py), and the no-PTR
fallback names a source after its AS name with no type. Such a source
yields a source_name bucket with no source_type sub-bucket, and
Grafana's Elasticsearch data source then fails to build the table:
with Grafana 13.2.4 (bundled Elasticsearch data source 12.9.2) and the
reports in samples/aggregate/, the panel showed "No data" while 12
sources had no type; with `missing` set it lists all 27 name/type rows.
Set `missing` to "-" on the source_type.keyword aggregation, the
placeholder most of the dashboard's other tables use. Hand-edited
rather than re-exported, to keep the diff to one value in this
Grafana 7.4-era export.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
github-actions[bot]andseanthegeek
|
d6e3ef8393 |
chore: update IPinfo Lite MMDB (#919)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
github-actions[bot]andseanthegeek
|
af63675aeb |
chore: update IPinfo Lite MMDB (#918)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
github-actions[bot]andseanthegeek
|
4a6b467586 |
chore: update IPinfo Lite MMDB (#917)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
e077136b60 |
Fix blank SMTP TLS failure-detail fields crashing saves; read RFC 8460 additional-information (11.0.3) (#916)
* Treat blank SMTP TLS failure-detail fields as absent; read RFC 8460 additional-information
Some reporters send optional failure-details fields as empty strings
(e.g. "sending-mta-ip": "" on sts-policy-fetch-error). The parser copied
them verbatim, so the PostgreSQL save failed on the INET columns
("invalid input syntax for type inet") and the Elasticsearch/OpenSearch
saves failed Ip() validation, discarding the whole report. Blank or
whitespace-only optional values are now omitted.
RFC 8460 section 4.4 names the URI key "additional-information"; the
parser only read "additional-info-uri" (the schema's value placeholder),
so conformant reports lost the URI. The RFC key is now read, with the old
key kept as a fallback.
Bump version to 11.0.3.
Closes #915
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Import parse_report_file directly in the ES/OpenSearch tests
Addresses github-code-quality's "Module is imported with 'import' and
'import from'" findings on tests/test_elastic.py and
tests/test_opensearch.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
11.0.3
|
||
|
github-actions[bot]andseanthegeek
|
6648be7d02 |
chore: update IPinfo Lite MMDB (#914)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
52edca0245 |
Block a release while a code scanning alert is open (#913)
A tag push ran python-tests.yml — lint, docs build, the test matrix — and nothing else. CodeQL runs as GitHub default setup rather than from that workflow, so release.yml never saw it, and a pull request's own CodeQL check reports only what that diff introduces, which means an alert already open on master passed every check a release went through. release.yml now has a code-scanning job that build waits on, so a release stops before it publishes anything rather than after. Everything else hangs off build, so gating build gates the PyPI upload, the GitHub release, the Docker image, and the docs deploy with it. Reading the alert list is only meaningful if CodeQL has looked at the code being released, and the tag is usually pushed moments after the commit lands on master, while its analyses are still running. So the job first waits, up to 15 minutes, for an analysis of the tagged commit in every language CodeQL covers. That set is read back from the analyses already published for the default branch instead of hard-coded, so it cannot drift out of step with the repository configuration, and a commit CodeQL never analyzed fails rather than passing against a scan of different code. A false positive is dismissed in the Security tab: a dismissed alert has state "dismissed", not "open", so it stops blocking without needing an exception here. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
github-actions[bot]andseanthegeek
|
edcfd0063b |
chore: update IPinfo Lite MMDB (#912)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Fable 5.1
|
da2beb5855 |
Bump version to 11.0.2 (#911)
The release commit (
11.0.2
|
||
|
Sean Whalen
|
f7c94ae758 | Release 11.0.2 | ||
|
Sean Whalen
|
281a3cf3b1 | Merge branch 'master' of https://github.com/domainaware/parsedmarc | ||
|
Sean Whalen
|
aedc7ee48b | fix: upgrade mailsuite requirement to >=2.3.2 for Python 3.14 support (closes #838) | ||
|
Sean WhalenandClaude Fable 5.1
|
b41230dc07 |
Make the SIGHUP configuration reload atomic (#910)
* fix: make the SIGHUP configuration reload atomic The reload built the replacement output clients, then immediately closed the old ones and swapped them in -- and only afterwards ran seven more fallible steps: load_reverse_dns_map() (which reloads the PSL overrides first), load_ip_db(), configure_ipinfo_api(), the setattr loop onto opts, _build_parser_config(), the watch-parameter int() conversions, and the log-level/FileHandler refresh. An exception in any of them was caught and logged as "Config reload failed, continuing with previous config", which was false: the new clients were already live and receiving reports, the old ones were closed and unrecoverable, opts could be half-applied, and both loaders empty their target before reading anything -- so a bad map path emptied REVERSE_DNS_MAP until some later uncached lookup happened to refill it from the previous configuration's paths (both get_service_from_reverse_dns_base_domain() and get_ip_address_info() reload a map they find empty), and a bad PSL overrides path emptied parsedmarc.utils.psl_overrides without touching the map, so nothing triggered that lazy reload and base domains were folded without the overrides until a restart. The reload is now three phases. Phase 1 stages every fallible step off to the side, in this order: _load_config/_parse_config, _init_output_clients, the derived values (_build_parser_config and the three watch-parameter int()s), load_reverse_dns_map into a fresh dict, load_ip_db, configure_ipinfo_api. The derived values run early because they write nothing at all, and the two loaders that write process-wide state run last, so as little as possible can fail behind them -- load_ip_db()'s download in particular rewrites the shared MMDB cache file before it assigns _IP_DB_PATH, and that is the one phase-1 side effect no snapshot can undo (what it leaves is the same refresh startup performs). The parsedmarc.utils globals the loaders assign are snapshotted by a new _utils_globals_snapshot(). Enumerated from the loaders' bodies, those are psl_overrides (cleared and repopulated in place, no `global` statement -- the list object itself is the state), _IP_DB_PATH (the only global load_ip_db assigns) and _IPINFO_API_TOKEN (the only one configure_ipinfo_api assigns, and assigned before the probe that can fail). _LAST_LOGGED_IP_DB_PATH is deliberately not covered: it is assigned by _get_ip_database_path(), which no loader the reload calls reaches, and it only decides whether the database path is logged again. Phase 2 commits with assignments only, REVERSE_DNS_MAP in place -- cli.py imports that name from parsedmarc, so rebinding it here would desync this module from parsedmarc.REVERSE_DNS_MAP and from the config.REVERSE_DNS_MAP that ParserConfig.__setstate__ rebinds to in a worker, and the ParserConfig staged in phase 1 already holds the pre-reload object. Phase 3 closes the replaced clients last and best effort, so a dying connection cannot undo the commit. On failure the replacement clients are closed, then the search backends' default alias is restored -- that order, because a fully built _ElasticsearchHandle releases the alias as it closes (see #906 and #902) -- and then the utils globals are put back. REVERSE_DNS_MAP, opts, clients, parser_config and the three watch parameters are never written before the commit, so there is nothing to undo for them; the per-step end state is traced in a comment at the rollback site, including which restores are load-bearing at which depth and which are defensive. configure_ipinfo_api's sys.exit(1) on an invalid token still bypasses all of this: SystemExit is not an Exception, and the process is leaving. Closing the replaced log file was the one statement in the commit block that could still raise (a full disk, a stale mount); it is now caught and reported after the handler is detached, since by that point the reloaded configuration is live and rolling back to report a failed reload would be a lie. Nothing runs concurrently with the swap: the reload happens in the main thread between watch_inbox() calls, after the mailbox backend has returned at a batch boundary, so no save callback is in flight, and parallel.py creates and tears down its ProcessPoolExecutor inside a single parallel_map() call -- its workers are separate processes with their own copies of REVERSE_DNS_MAP and the utils globals either way. The previous code mutated the same state at the same point, so this is not a change in exposure. #906 made _init_output_clients() itself all-or-nothing; this does the same for the whole reload. The docs already promised that a failed reload leaves "the previous configuration fully active"; that promise is now true, and usage.md spells out what it covers and the one thing it does not. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: open the replacement log file during reload staging, not after commit The logging refresh was the one fallible step left after the commit: phase 2 closed and removed the old FileHandler, then tried to open the new log_file inside a try/except that only warned. On an unwritable path the old handler was already gone, file logging was dead, opts.active_log_file was set to the new path anyway, and the reload was logged as successful. Because the old-versus-new comparison uses active_log_file, a later SIGHUP with the same path -- after the operator fixed permissions -- was a no-op, so only a restart recovered file logging. The replacement FileHandler is now constructed in phase 1, after the watch-parameter conversions and before the loaders (constructing it opens the file immediately, and that is the only thing in the refresh that can fail on a bad path). The rollback closes it if a later step fails; phase 2 only attaches it. An unwritable log_file is therefore a reload failure, with the previous log file still attached and receiving the traceback. Startup keeps its warning-only behaviour: there is no previous configuration to fall back to there. Found by Copilot's review of #910. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: retry a log file that could not be opened at startup on the next reload Startup recorded opts.active_log_file = opts.log_file even when opening the FileHandler failed (a warning there, since there is no previous configuration to keep). The SIGHUP reload only stages a replacement handler when the new config's log_file differs from active_log_file, so a reload naming the same path never retried the open: file logging stayed dead until restart, even after the operator fixed the directory or permissions. active_log_file now starts as None and is set only after the handler is attached, so an unchanged log_file that failed at startup is retried by the next reload, and one that opened at startup is not reopened. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: make the rollback's staged log handler close best-effort; correct the docs on what aborts a reload FileHandler.close() can raise while flushing or closing (a full disk, a stale network mount). In the reload's rollback that raise escaped the except block: the search-alias and utils-globals restores never ran, the original staging failure was replaced, and the watcher exited. The close is now logged and swallowed like the commit path's close of the replaced handler, so the restores that follow it always run. usage.md said "unreachable output destinations" abort a reload. That is true of PostgreSQL, Kafka, and TCP/TLS syslog, whose clients connect in their constructors, but not of Elasticsearch or OpenSearch: set_hosts() builds the client without connecting, and migrate_indexes() catches every cluster call and warns, so a reload pointing at an unreachable search cluster still commits and fails on the first save. The list now names what actually raises during staging and calls out the exception. Both found by Copilot's second review of #910. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Fable 5.1
|
316910e1ea |
fix: never close a caller-supplied file object in parse_report_file/extract_report (#909)
parse_report_file() and extract_report() both closed a file-like object the caller passed in once they finished reading it (on the success path only). A function must not close a handle it did not open: the caller may still want to seek(0) and retry, log tell(), or reuse the handle. - parse_report_file(): the caller-supplied-object branch no longer calls .close(). A path is still opened and closed via `with`; bytes are still wrapped in a function-owned BytesIO that the function closes. - extract_report(): the seekable-stream branch previously aliased the caller's stream into `file_object` and then unconditionally closed it in `finally`, on the success path and on the exception path alike. An `owns_file_object` flag now tracks whether `file_object` is a buffer this function created (str/bytes input, or a copy made from a non-seekable caller stream) versus an alias of the caller's own seekable stream; only the former is closed. - parse_aggregate_report_file() forwards its input straight to extract_report() and does no closing of its own, so it inherits the fix; its docstring now says so and points to extract_report()'s for the exact post-call position. Both extract_report()'s and parse_report_file()'s docstrings now state the contract explicitly. extract_report()'s docstring initially claimed a successful call leaves a seekable caller stream "positioned at 0" and described the seek(0) as seeking "back" to the caller's prior position; neither is true. The unconditional stream.seek(0) after reading the 6-byte header runs regardless of where the caller's stream was positioned, and the subsequent content read (member/gzip/text) advances the stream again, so a successful call typically leaves it at EOF, not 0 (measured: XML 866, gzip 428, zip 901/974). The docstring now matches parse_report_file()'s wording: left open, positioned wherever the function's own reads left it. Added testParseReportFileLeavesCallerHandleOpenOnSuccess, testExtractReportLeavesSeekableCallerStreamOpen (success path), and testExtractReportLeavesSeekableCallerStreamOpenOnError (exception path, completing the "never closed on success or failure" claim, which had gone untested on its exception half) -- asserting real BytesIO `.closed` state and, for the success cases, that the handle is still readable after seek(0). All three fail against the pre-fix code; verified with `parsedmarc.__file__` that the import resolved to this worktree's package, not a stale venv install, before trusting the failure/pass results. Updated testParseReportFileLeavesCallerHandleOpenOnReadError's docstring, which described the old "closed on success" behavior as intentional. Also added testParseAggregateReportFileLeavesCallerStreamOpen: the contract paragraph added to parse_aggregate_report_file()'s docstring was previously untested through that entry point -- its existing stream-based tests only exercise extract_report() and parse_report_file() directly, while parse_aggregate_report_file() itself was only ever called with bytes. The new test passes a caller-owned BytesIO through parse_aggregate_report_file() and asserts it is still open after both a successful parse and an InvalidAggregateReport raised on garbage content. It fails against the pre-fix code (AssertionError: True is not false on the success-path assertion); verified via `parsedmarc.__file__` that the import resolved to this worktree's package before trusting the result. CHANGELOG: added an Unreleased/Changes entry (behavior change for library callers, naming all three affected functions) and corrected the existing Unreleased/Bug fixes entry for parse_report_file's path-close fix, whose closing sentence about caller-supplied objects this PR makes false and whose cross-reference to "the Changes section below" pointed the wrong direction (Changes is above Bug fixes). Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Fable 5.1
|
550244c6d7 |
fix: roll back the search backends' default alias when output init fails (#906)
_init_output_clients() is now all-or-nothing against the Elasticsearch and OpenSearch connection registries: either it returns the fully built client dict, or it closes everything it built and leaves both "default" aliases naming exactly what they named on entry. elastic.set_hosts()/opensearch.set_hosts() end in connections.create_connection(), which registers the new client under the process-wide "default" alias the moment it is constructed (elasticsearch/dsl/connections.py:81-88 and opensearchpy/connection/connections.py:87-95, as installed: 8.19.3 / 3.2.0). Everything that runs after that -- the index migration, and every output configured later -- can still fail. On the SIGHUP reload path _main() builds the replacement clients before closing the old ones and, when the build raises, logs "Config reload failed, continuing with previous config" and keeps the old opts. But the alias had already been handed to the new client, so every save -- elastic.py's Search and Document.save() calls, which resolve the "default" alias through elasticsearch/dsl/_sync/document.py:99-100 -- reached the new cluster while the index prefixes/suffixes and index_prefix_domain_map still came from the old configuration. The half-built client was never closed either, nor was any client built earlier in the same failed call (that half also leaked on every attempt of the startup retry loop). The handler tears down first and restores second -- with the teardown in a try/finally, since a second Ctrl-C landing in it propagates straight through _close_output_clients, which swallows only Exception -- and the two steps are not interchangeable. With an _ElasticsearchHandle already in `clients`, teardown closes its client and releases the alias -- which still names that client -- and the restore then re-registers the previous client into an unset alias. Restoring first would put the previous client back and only then close the handle, which rests the whole rollback on the handle declining to touch an alias that no longer names its own client: true only since #902, and a property of the handle rather than of this function. Tearing down first keeps the guarantee local. Both failure points are traced in the comment on the handler. Closing the discarded client is best-effort, and closing it twice is safe: Elasticsearch.close() -> Transport.close() closes each node's urllib3 pool (elastic_transport/_transport.py:499-504, _node/_http_urllib3.py:224-228, and urllib3 pool close() is a no-op once cleared), and OpenSearch's connection close() guards on `if self.pool` (opensearchpy/connection/http_urllib3.py:323-328). Taking the snapshot cannot itself open a connection: get_connection() lazily builds a client from kwargs left behind by configure() (dsl/connections.py:90-115), and parsedmarc never calls configure(). The guard catches BaseException rather than Exception because migrate_indexes() wraps every cluster call in `except Exception` and logs a warning (elastic.py:824-829, and again in each of the per-index loops that follow), so what escapes the Elasticsearch block after set_hosts() is, in practice, a KeyboardInterrupt landing in one of those calls -- which is what the regression tests inject at the SDK transport boundary. For the same reason, closing the discarded client swallows BaseException: an interrupt there must not cost the alias its hand-back, and the original failure is re-raised afterwards either way. The alias is not the only module-level state set_hosts() writes: elastic.set_hosts() also assigns elastic._SERVERLESS (elastic.py:620-622) -- before it constructs the client, so it can be stale even on a failure that never reached the registry -- and create_indexes() consults it (elastic.py:652-659) to decide whether to strip the shard settings Serverless rejects. It is snapshotted and restored alongside the alias. opensearch.py has no equivalent (no `global` statement in the file). Shape: the existing body keeps its indentation as _build_output_clients(), which fills a `clients` dict the caller passes in and is explicitly not transactional; the rollback lives in a short _init_output_clients() wrapper that owns that dict on both paths. Passing the dict in is what lets the wrapper close what was already built after the build raises. The public name is unchanged, so the tests and _main() call sites are untouched. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5
|
0f3b973be5 |
fix: close the search client a handle owns, not the default alias (#902)
_ElasticsearchHandle.close() and _OpenSearchHandle.close() re-resolved
the client library's "default" connection alias at close time. That is
wrong on the SIGHUP reload path, which is not a shutdown path:
_main's reload block calls _init_output_clients() -- and therefore
set_hosts() -> connections.create_connection() -- before
_close_output_clients() on the old clients, deliberately, so a broken
new config leaves the old clients running. By then the alias names the
*new* client, so the old handle closed the new client and then dropped
the alias entirely. Every later save raised
KeyError("There is no connection with alias 'default'.") until
parsedmarc was restarted, and the original client was leaked.
Verified against the installed SDK sources rather than the docs
(AGENTS.md, "Verify bug claims against authoritative sources", point 3):
elasticsearch/dsl/connections.py (elasticsearch 9.x) and
opensearchpy/connection/connections.py (opensearch-py 3.x) both define
def create_connection(self, alias="default", **kwargs):
conn = self._conns[alias] = <Client>(**kwargs)
-- so create_connection() overwrites whatever the alias held, and
returns the very object it stored (elasticsearch.dsl passes it through
_with_user_agent(), which mutates headers and returns the same object,
so identity is preserved) --
def remove_connection(self, alias):
errors = 0
for d in (self._conns, self._kwargs):
try:
del d[alias]
except KeyError:
errors += 1
if errors == 2:
raise KeyError(f"There is no connection with alias {alias!r}.")
-- i.e. it deletes the alias outright, with no regard for which client
currently holds it, and
def get_connection(self, alias="default"):
if not isinstance(alias, str):
return alias # (opensearch-py; elasticsearch.dsl
# returns _with_user_agent(alias))
try:
return self._conns[alias]
...
raise KeyError(f"There is no connection with alias {alias!r}.")
-- a str only comes *back* out of get_connection() when a str was
passed *in* as the alias, which these call sites never do; the client
stored under "default" is always the object create_connection() built.
The old `isinstance(conn, str)` guard was therefore dead code and is
removed rather than carried forward (AGENTS.md: delete unreachable
branches, don't hide them). add_connection(alias, conn) stores an
arbitrary object as-is, which is what the new tests use.
set_hosts() now returns the client it registered, _init_output_clients()
passes that client to the handle, and close() closes that object and
gives up the alias only when the registry's current "default" *is* that
same object (identity check). The init/close order in the reload block
is unchanged.
tests/test_cli.py gains TestSearchBackendHandles, which drives the real
connection registries (the SDK boundary) with fake client objects, so no
network is touched: per backend, a reload-shaped test asserts the old
client is closed, the new one is not, and the alias still resolves to the
new client; a shutdown-shaped test asserts the alias is released when it
is still ours; and a best-effort test asserts a raising close() neither
propagates nor blocks the alias release. Each test restores whatever the
process-wide registry held beforehand. All six fail against the unfixed
handles.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Sean WhalenandClaude Opus 5
|
77045c2df0 |
chore: address CodeQL code-quality findings (no behavior change) (#900)
- py/empty-except: add a short comment to each of the 10 flagged
`except <Type>: pass` blocks explaining why swallowing the
exception is correct there (best-effort cleanup/close, or
intentional fall-through to the next report format).
- py/unnecessary-lambda: replace `map(lambda x: parse_email_address(x), ...)`
with `map(parse_email_address, ...)` at the 5 flagged sites in
parsedmarc/utils.py; parse_email_address takes exactly one
positional argument, so this is behavior-preserving. Ran
`ruff format` afterward, which collapsed 3 of the now-shorter
calls onto single lines.
- py/imprecise-assert: replace `assertTrue(a > b)` / `assertTrue(a >= b)`
with `assertGreater(a, b)` / `assertGreaterEqual(a, b)` at the 9
flagged test sites; none carried a custom assertion message.
- .vscode/settings.json: drop three cSpell whitelist entries that are
pure misspellings ("passsword", "httpasswd", "unparasable") and
appear nowhere else in the tracked tree, so a recurrence of the typo
they used to hide (see #888) would be caught again. The correctly
spelled "htpasswd" and "unparseable" entries are untouched.
Verified clean: ruff check, ruff format --check, pyright (0
errors/warnings), and pytest tests/ (989 tests collected and passing
before and after, GITHUB_ACTIONS=true).
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Sean WhalenandClaude Opus 5
|
d185bd0526 |
Fix file descriptor leak in parse_report_file (CodeQL py/file-not-closed) (#899)
parse_report_file's path branch opened the file with open(file_path, "rb") and then, on a later line, called file_object.read() followed by file_object.close() with no exception handling between them. An exception from read() (e.g. an OSError from the underlying storage) skipped close() and left the descriptor to be released only when Python's garbage collector eventually finalized the object, rather than closed deterministically (CPython's io.IOBase.__del__ closes an unclosed file on finalization: https://docs.python.org/3/library/io.html). This is the pattern CodeQL's py/file-not-closed query flags, found in a local code-quality scan. The path branch now opens the file with a `with` block, so the handle is closed on both the success and exception paths. The fix is scoped to only that branch: a BytesIO created internally from bytes/bytearray/memoryview input, and a file-like object supplied by the caller, are both closed only after a successful read and left open if read() raises, exactly as before this commit. Widening that to close a caller-supplied handle on the exception path too would be a behavior change outside this fix's scope. Also closes tests/test_init.py's TestGetDmarcReportsFromMailboxMaildir ._deliver, which did open(source, "rb").read() with no close() at all; it now uses a `with` block. Two regression tests cover the narrowed contract: - testParseReportFileClosesHandleOnReadError patches builtins.open (the SDK boundary) with a fake handle implementing the context-manager protocol, and asserts close() is called when read() raises. Verified against origin/master's version of parsedmarc/__init__.py: the test fails with "AssertionError: False is not true" (close was never called), confirming it catches the leak. - testParseReportFileLeavesCallerHandleOpenOnReadError passes a caller-supplied fake handle directly as input_ and asserts close() is NOT called when read() raises. Verified against a naive whole-block try/finally (wrapping close() around all three branches instead of only the path branch): the test fails with "AssertionError: True is not false" (close was called), confirming it catches the scope widening this PR must avoid. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5
|
95ddc8a323 |
Fix missing-file error paths falling through in find_unknown_base_reverse_dns.py (#903)
Two `if not os.path.exists(path): print(f"Error: ...")` checks printed the intended error message but had no sys.exit(1) after it, so execution fell through into the subsequent open() call on the same missing path and raised an unhandled FileNotFoundError instead of the clean error exit the code clearly intended. Sibling duplicate-entry checks in the same function already did print-then-sys.exit(1); these two now match. Sites fixed (both in _main()): - the nested load_list() helper's missing-file check (used for known_unknown_base_reverse_dns.txt and psl_overrides.txt) - the base_reverse_dns_map.csv missing-file check Both sites now have regression tests in a new TestFindUnknownBaseReverseDNS class: - the load_list() site: calling _main() in a temp directory with no known_unknown_base_reverse_dns.txt now raises SystemExit(1) instead of FileNotFoundError. - the base_reverse_dns_map.csv site: _load_as_name_index() does its external work entirely through maxminddb.open_database(), the actual SDK boundary, so that call is mocked to a context manager over an empty iterable rather than loading the real ~23MB bundled MMDB or mocking an internal helper. Calling _main() in a temp directory with no base_reverse_dns_map.csv now raises SystemExit(1) instead of FileNotFoundError. Both tests capture stdout and assert on the specific "Error: ... does not exist" message, pinning the exit to the intended site rather than any sys.exit(1) in the function. Cleanup uses two separate addCleanup calls (rmtree registered before chdir, so LIFO order runs chdir first) instead of one lambda wrapping both, so rmtree still runs even if chdir were to raise. Also fixed an adjacent prose bug a few lines from the second site: the "is in known_unknown... and base_reverse_dns_map..." error message was missing a space after "Error:" and was a backslash-continued f-string that embedded the source's literal indentation in the printed output. It now prints as a single clean line, consistent with the file's other error messages. Logged the user-facing symptom (clean error replaced by a FileNotFoundError traceback) under CHANGELOG.md's Unreleased/Bug fixes section, matching the project's precedent of logging maintainer-tooling fixes (e.g. the sortlists.py entry). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5
|
4298603713 |
Fix CodeQL findings: use sys.exit(), remove dead index_date stores (#898)
py/use-of-exit-or-quit (29 sites): replace bare `exit(...)` calls with `sys.exit(...)` in cli.py (20 sites), sortlists.py (4 sites), and find_unknown_base_reverse_dns.py (5 sites). `exit` is injected into the interactive namespace by the `site` module, so it is not guaranteed to exist under `python -S` or an embedded interpreter — the very error paths that called it would raise NameError instead of exiting cleanly. `sys.exit` is always available and was already used elsewhere in these files. sortlists.py did not import `sys`; the import was added to its existing stdlib import block. find_unknown_base_reverse_dns.py already imported `sys` (used at line 71), so only the call sites changed there. Checked tests/test_cli.py for anything patching `exit` directly — none do; the SystemExit-based assertions exercise the same behavior for either spelling, so no test changes were needed. py/multiple-definition (4 sites): removed the dead `index_date` computation in save_aggregate_report_to_elasticsearch() (elastic.py:944-947) and save_aggregate_report_to_opensearch() (opensearch.py:875-878). Both blocks derived index_date from the report-level begin_date, but the per-record loop immediately below unconditionally recomputes index_date from each record's own begin_date before the only read of the variable (the index name build near the end of the loop body). Verified by reading both functions that no code between the outer assignment and the loop's recompute reads index_date, and that the recompute is unconditional on every path that reaches the read, before deleting. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 
|
e1c2e42d5d |
Dockerfile: bind-mount the wheel so it stops shipping in every published image (#893)
* Dockerfile: bind-mount the wheel instead of COPYing it The runtime stage COPYs the built wheel out of the build stage and the RUN that installs it rm -rf's it again. A RUN cannot remove what an earlier instruction already committed -- it writes a whiteout on top -- so the COPY layer ships in every pull. Measured on ghcr.io/domainaware/parsedmarc:11.0.0: on amd64 layer 4 is 10,713,473 bytes of a 267,653,434-byte image (4.0%); on arm64 10,713,473 of 270,400,579 (3.96%). Layer 5 carries tmp/.wh.dist. The 10.5.0 and 10.0.0 tags carry the same layer. Bind mounts are not committed to layers, so the rm -rf /tmp/dist clause is no longer needed and is removed. A # syntax=docker/dockerfile:1 directive is added so RUN --mount is guaranteed available. * Add --no-cache-dir option to pip install Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Dockerfile: drop the syntax directive; document the size win `RUN --mount=type=bind,from=<stage>` has been stable in BuildKit's built-in Dockerfile frontend since 2020, and the Docker workflow builds through docker/setup-buildx-action + docker/build-push-action, so the mount works without `# syntax=docker/dockerfile:1`. The directive only adds an unpinned Docker Hub round trip before the Dockerfile can be read (`resolve image config for docker-image://docker.io/docker/dockerfile:1`), on a floating tag, and it does not help the legacy non-BuildKit builder, which rejects `RUN --mount` either way. Verified on linux/amd64 with BuildKit (buildx 0.37.0, Docker 29.8.0): building with and without the directive produced five layers totalling 163,757,439 and 163,757,152 compressed bytes respectively -- a 287-byte gzip nondeterminism, otherwise identical. The master baseline built six layers totalling 272,138,724 bytes, whose wheel layer of 10,713,475 bytes reproduces the published 11.0.0 image's 10,713,473 to within two bytes. The comment block is trimmed to state the invariant rather than narrate what the code used to do, and CHANGELOG.md gains an entry: the change is user-visible, and the ~40% reduction is mostly the pip download cache that --no-cache-dir removes, not the wheel layer the PR was opened for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
github-actions[bot]andseanthegeek
|
de88cd893f |
chore: update IPinfo Lite MMDB (#896)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Fable 5.1
|
09c88ca2a3 |
11.0.1 release: harden failure-report sample filenames and cap decompressed report size (#895)
* Harden failure-report sample filenames and cap decompressed report size Fixes the two open security advisories. GHSA-c284-w5m6-jhjm (path traversal, affects 9.0.6 through 11.0.0): save_output() named each failure report's message sample after the sample's Subject header, falling back to the raw subject whenever sanitizing it produced an empty string. A subject of only path separators and dots -- "../../../" or "/" -- sanitizes to "", so the raw value reached os.path.join() and the .eml landed outside the samples directory or at an absolute path. That subject comes from a message that failed authentication, so any sender a monitored mailbox accepts controls it. The name is now sanitized at write time and falls back to "sample", and the caller-supplied filename_safe_subject key is no longer trusted. get_filename_safe_string() also strips NUL (which would otherwise make open() raise ValueError: embedded null byte and hold back the whole mailbox batch), truncates before stripping trailing characters, and strips trailing spaces along with trailing dots, since Windows drops both when creating a file; its docstring now states the guarantees callers depend on. GHSA-43qf-f35w-2x4r (unbounded decompression, affects all versions through 11.0.0): extract_report() inflated gzip with one unbounded zlib.decompress() and read zip members with an unbounded .read(). The attachment's content is chosen by its sender, and deflate reaches about 1000:1 on degenerate input, so a 100 KB attachment inflated to 100 MB with a ~209 MiB peak. Extraction now stops at MAX_DECOMPRESSED_REPORT_SIZE (100 MiB) and raises ParserError. The gzip path moves to a zlib.decompressobj() bounded by max_length, which does not raise on a stream that ends early the way the one-shot call did, so the helper checks decompressor.eof itself; a stream with trailing bytes after the gzip member still extracts, matching the old behavior. Both behaviors were observed against the unmodified code first and are pinned by tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Strip NUL from CSV fields so Python 3.10's csv writer accepts them Python 3.10's csv writer raises _csv.Error: need to escape, but no escapechar set on any field containing NUL (CPython issue 97503, a 3.10 regression fixed in 3.11+). Failure report text fields (subject, user agent, authentication results, addresses, etc.) come from untrusted mail, so a NUL byte in one made parsed_failure_reports_to_csv() -- and therefore save_output() -- raise on 3.10, which the CLI's except (OSError, ValueError) around save_output does not catch. NUL is now stripped from every CSV field on all Python versions via a shared _csv_safe() helper, applied in all three CSV writers, so output is identical regardless of interpreter version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * 11.0.1 release: bump version and finalize changelog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>11.0.1 |
||
|
github-actions[bot]andseanthegeek
|
823b0a1811 |
chore: update IPinfo Lite MMDB (#894)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Fable 5
|
15e7db1811 |
11.0.0 release: bump version and finalize changelog (#892)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>11.0.0 |
||
|
Sean WhalenandClaude Fable 5
|
0d832ff682 |
Make dashboard-dev-bootstrap.sh work with Docker or Podman (#891)
The bootstrap script hardcoded `docker compose`, locking out contributors on Podman-based systems. It now auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (`docker compose`/`docker-compose`, `podman compose`/`podman-compose`), selectable explicitly with --backend docker|podman or the CONTAINER_BACKEND environment variable (the flag wins). Detection requires a live `<engine> info` call, so a leftover docker CLI with no running daemon does not shadow a working podman, and both compose forms are probed with `version` so a broken install fails at startup instead of partway through the run. Adds -h/--help documenting the flag and the script's env knobs. Every container invocation already flowed through the single COMPOSE array, so the rest of the script is unchanged. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |