mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-11 03:11:33 +00:00
e8292bef7034e7ca3310cd7c9ccae09deff44203
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
2ee209005b |
Merge commit from fork
* chore: update IPinfo Lite MMDB (#941) Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Cap report date range and indexes created per report A report whose <date_range> spans millions of days was normalized into one record per day, exhausting memory, and with Elasticsearch or OpenSearch output created one dated index (one shard) per day, exhausting the cluster's shard budget. Reject a range longer than MAX_REPORT_SPAN_DAYS (366) or negative with InvalidAggregateReport. In both save functions, once a report has used MAX_INDEXES_PER_REPORT (31) distinct dated indexes, route further records to the report's first index, keeping their own dates, and log one warning. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Catch output-stage exceptions and validate SMTP TLS report types One report sent to a monitored mailbox could make a configured output raise an exception that process_reports() did not catch. It escaped through the mailbox save callback, _main logged "Mailbox Error" and exited 1, and the batch stayed in the mailbox, so every one-shot run died on the same message and duplicated the legitimate reports in non-deduplicating outputs. Three sender-controlled inputs did it: - A failure report whose sample has an empty From:, To: or Reply-To: (To: undisclosed-recipients:; included): mail-parser gives an empty address list, and the Elasticsearch and OpenSearch saves indexed [0] (IndexError). - An SMTP TLS report with a wrong-typed field: "policy-string": [1] made the CSV writer raise TypeError; "start-datetime": 5 made the Elasticsearch, OpenSearch and Splunk saves raise AttributeError and _ensure_utc_suffix() in the PostgreSQL save raise TypeError. - An aggregate report whose <reason> has no <type> (KeyError in the CSV writer) or a <type> that xmltodict parses to a dict or a list (TypeError in its join). Fix: - Every output destination block in process_reports() now ends with the catch-all the aggregate Elasticsearch block had: the failure and SMTP TLS Elasticsearch/OpenSearch blocks, every PostgreSQL and Splunk HEC block, and the file output (its OSError/ValueError branch is kept). The error goes through log_output_error(), so the batch is reported unsaved and fail_on_output_error still raises ParserError. - parse_smtp_tls_report_json() and its helpers check each field against the type RFC 8460 section 4.4 gives it and raise InvalidSMTPTLSReport otherwise: strings for organization-name, report-id, policy-domain, policy-type, result-type and the optional failure-details fields; contact-info a string or a list of strings; policy-string and a list mx-host-pattern lists of strings; the session counts integers or null; failure-details a list of objects; start-datetime and end-datetime strings of at most 64 characters that the outputs' own conversions accept. dateutil took 12.4 s to fail on a 1,000,000-digit date, hence the length bound. - _parse_report_record() makes a reason's type and comment a string or None (a dict unwrapped with _text(), a list as its str()), and the aggregate CSV writer reads both with .get(). - The Elasticsearch and OpenSearch failure saves flatten From, To and Reply-To with a helper that returns None for an empty list, and leave such a header out of the document and the duplicate query. - The Splunk SMTP TLS save converts begin_date inside its try, so a bad value raises SplunkError; _ensure_utc_suffix() returns a non-string unchanged. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Limit XML nesting depth and node count in aggregate reports parse_aggregate_report_xml() gave the decompressed report (up to the 100 MiB decompression cap) to xmltodict with no limit on depth or element count. xmltodict keeps one stack entry per open element and expat reports missing end tags only at the end of the input, so a few KB of gzipped unclosed <a> tags grew to gigabytes of RSS and the kernel OOM killer ended the process before the message was moved; a wide document of millions of siblings did the same at a smaller scale. A failed first parse also re-parsed the document through lxml recovery, which has no node limit, and a second xmltodict pass. _NoDoctypeExpat.ParserCreate() now returns a proxy around the expat parser. pyexpat parsers cannot be subclassed and xmltodict assigns its handlers after ParserCreate() returns, so the proxy's __setattr__ wraps StartElementHandler to count depth and nodes (elements plus attributes) and EndElementHandler to decrement depth, and forwards every other attribute. Over MAX_XML_DEPTH (100) or MAX_XML_NODES (5,000,000) it raises before xmltodict's handler runs. Both xmltodict.parse() call sites turn that into InvalidAggregateReport naming the limit; the first does so instead of trying lxml recovery. The lxml recovery path now runs a counting pass first, with a parser target that builds no tree and counts elements, attributes and namespace declarations against MAX_XML_NODES; the tree is built only for a document within the limit. Depth there is bounded by libxml2's own limit of 256 (huge_tree stays off) and then by the second xmltodict pass. A 22 MiB wide document behind a malformed XML declaration peaked at +1532 MiB before this pass and +23 MiB after. The largest aggregate sample (909 KB) has 32,019 nodes, so a legitimate report at the 100 MiB cap stays near 3.6 million. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Create Elasticsearch/OpenSearch indexes with mapping guards create_indexes() and create_data_streams() created indexes and the data stream template with settings only, so the cluster typed every report field from the first document that held it, with date_detection on. A sender could be first into a day's index (its name comes from the report) and poison it: a report_id of "2026-11-09" mapped report_id as a date, a failure sample header named "Subject.x" mapped sample.headers.subject as an object, and samples with invented header names used up index.mapping.total_fields.limit. Every later legitimate report for that index was then rejected and ended up in Unsaved; in data stream mode this lasted until a rollover. Both functions now take a mappings body, and the three save functions pass one per report type. Each turns date_detection off and declares the fields that hold dates (@timestamp, date_range, date_begin, date_end, arrival_date, sample.date) as dates, the type date detection gave them before; on the sample reports every other field keeps its old type. The failure mapping maps sample.headers as an object with dynamic: false and declares, as text with a keyword sub-field (the dynamic string default), every header the shipped dashboards reference plus date, message-id and cc; all headers still stay in _source. A "." in a header name is replaced with "_" before the header is stored. The index is now created through the client's indices.create with the settings and mappings body, because opensearch-py's Index.create cannot take a mappings argument; the data stream template merges the mappings with its @timestamp date mapping. Existing indexes are not changed. The other report fields stay dynamic and object-mapped (no nested), which the dashboards need (issue #169). Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read the RFC 8460 mx-host key in SMTP TLS reports RFC 8460 section 4.4 keys a policy's MX host patterns "mx-host"; its schema line is "mx-host": mx-host-pattern, where mx-host-pattern is the value type, described as "a JSON array of strings", and the RFC's Appendix B example gives the value as a plain string. The parser read only "mx-host-pattern", so every RFC-conformant report lost the field (samples/smtp_tls/rfc8460.json, the Appendix B example, among them). _parse_smtp_tls_report_policy() now reads "mx-host" as a string or a list of strings, a string becoming a one-element list, and falls back to "mx-host-pattern" (same rules) when "mx-host" is absent, null or empty. Any other type raises InvalidSMTPTLSReport. The output key is still mx_host_patterns, so the PostgreSQL TEXT[] column, the Elasticsearch/OpenSearch field and the CSV column are unchanged. Refs #942 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject over-limit messages with the mailsuite guard parse_report_email() and parse_email() handed every message to mail-parser and the email package with no bound on blank lines, header lines, header field size, MIME parts or nested multiparts, so a small crafted message took seconds to minutes and gigabytes to parse while the rest of the mailbox waited. Both now call mailsuite.utils.check_message_limits() first, one linear pass. parse_report_email() turns MessageLimitExceeded into InvalidDMARCReport so the mailbox loop files the message under Invalid; parse_failure_report() turns one from the sample into InvalidFailureReport. The mailsuite floor rises to 2.3.5, which adds the guard. Messages nested deeper than 100 multiparts are now rejected by the guard instead of reaching the recursion handler; the tests that used the 400-level message expect the new message. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Cite RFC 7489 §6.3 for the daily report interval The comment on MAX_REPORT_SPAN_DAYS cited RFC 7489 Appendix A for the one-day aggregate report. The daily interval comes from §6.3: the "ri" tag requests the interval between aggregate reports, and its default is 86400 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the 11.2.2 changelog: five advisories and #942 One Security bullet per advisory (GHSA-qcc2, GHSA-q7gm, GHSA-p85v, GHSA-4272, GHSA-qpcg) and a Bug fixes bullet for the RFC 8460 mx-host key. The version in constants.py is bumped by the release pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the date range rejection message and pyright's analysis With the XML limits and the date range limit merged, parse_aggregate_report_xml() became too complex for pyright to analyze ("Code is too complex to analyze"), and the date range rejection, raised inside the function's catch-all, reached the caller as "Unexpected error: Report date range spans ...". The date range check moves to _check_report_span() and the second xmltodict pass to _parse_limited_xml(), which raises the XML limit breach as InvalidAggregateReport; the catch-all now re-raises an InvalidAggregateReport as it is. The tests assert the message is not re-wrapped. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Put the over-limit failure samples in a delivery-status body mailsuite 2.3.5's check_message_limits() follows the MIME structure and counts blank lines only inside message/delivery-status bodies, the shape whose parse cost is per blank line; blank lines elsewhere in a body are not limited. The two failure-sample tests used blank lines in a plain body, which the final check accepts, so they now put them in a delivery-status body. parsedmarc's call with the default limits is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe the final mailsuite check and the index cap in the docs The GHSA-qcc2 changelog bullet now describes check_message_limits() as mailsuite 2.3.5 ships it: limits follow the MIME structure, and body lines outside message/delivery-status bodies are not limited. The changelog section moves to Unreleased, as parsedmarc's AGENTS.md has fix pull requests write it; the release pull request names the version. The Elasticsearch and OpenSearch pages now say that an address header with no address is left out of the failure document, and that a report spanning more than 31 days saves its later records to its first dated index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound records times days in normalization and fix the span message The date range limit bounds how far one record is split, but not how many records are split: 10,000 records over 366 days, each with a count of 1,000,000 (3.3 MB of XML), produced 3,670,000 records in 13.5 s and 1.7 GB, and 30,000 of them (a 44 KB gzip) ran out of memory and reached the caller as "Unexpected error: ". A report whose split would exceed MAX_NORMALIZED_RECORDS (250,000) now keeps its records whole over its own range, with timespan_requires_normalization false and one warning; a large legitimate multi-day report is kept rather than rejected. The record loop moves to _parse_report_records() so pyright can still analyze parse_aggregate_report_xml(). The span message used float division, which raises OverflowError for an <end> of 4,001 digits; it now uses integer division. The index-routing warning escapes the sender's report ID with sanitize_for_log(). The changelog and the Elasticsearch and OpenSearch pages say the index cap is per report, that dated index names still come from report dates, and that data streams are affected by neither. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop XML attributes from aggregate reports before they are used xmltodict turns an element with an attribute, such as <p a="1">none</p>, into {"@a": "1", "#text": "none"}. Such objects passed through the parser into Elasticsearch and OpenSearch documents for version, p, fo, disposition, envelope_to and other scalar fields, so the first one mapped the field as an object and every later report to that index failed, and hundreds of attributes on one element used up total_fields.limit. The parsed report now goes through one linear pass that replaces an element with only attributes and text by its text and drops attributes from the others; the existing _text() calls now receive strings. The changelog and the Elasticsearch and OpenSearch pages also say that an existing data stream's next backing index gets the guarded mappings, since parsedmarc puts the template again in every process, and the indexed-header comment and changelog name cc as the one header added to those the dashboards reference. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject lone surrogates in SMTP TLS reports and guard the stdout print An SMTP TLS string field holding a lone surrogate (JSON "\ud800") passed every type check and survived json.dumps(ensure_ascii=False), and print(output_str) in process_reports(), outside every try, raised UnicodeEncodeError: parsedmarc exited 1, and in mailbox mode the error escaped the save callback. RFC 8460 section 4 reports are I-JSON, whose RFC 7493 section 2.1 forbids surrogate code points, so the string and string-list checks now reject one as InvalidSMTPTLSReport, and the print records a UnicodeEncodeError as a "Standard output" output error. The _smtp_tls_datetime() docstring no longer says a parsed date cannot fail in any output: dateutil accepts partial dates that PostgreSQL may reject, which is contained as PostgreSQLError. The _reason_text() docstring says a dict with child elements gives None. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe mailsuite 2.3.5's address group handling in the changelog The final mailsuite 2.3.5 no longer rejects an address header by its comma count; its parse_email() removes RFC 5322 group syntax from address header values over 8 KiB before parsing them. The boundary limit counts lines that are the boundary of an open multipart. parsedmarc's call with the default limits is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the keep-alive call of the extracted record loop _parse_report_records() holds the record loop moved out of parse_aggregate_report_xml(); its keep-alive branch had no test. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a policy override type with child elements With XML attributes dropped at parse time, a dict reaches _reason_text() only for an element with child elements; the test pins its None result and the CSV's "none". Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check plain-string contact-info and mx-host for lone surrogates The surrogate check covered the string fields and string lists, but a contact-info or mx-host given as one string took a branch that skipped it, and a lone surrogate there still made the stdout print raise. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which sample headers the failure document leaves out The GHSA-qpcg changelog bullet said every sample header stays in _source, but the failure save leaves out a From, To or Reply-To header that holds no address, as the Elasticsearch and OpenSearch pages say. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep sender-derived numbers out of the date range message The rejection message printed the report's span in days, so an <end> of thousands of digits put a number of thousands of digits into the exception text and the log. It now says only that the range is negative or longer than MAX_REPORT_SPAN_DAYS. Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Store every string-typed aggregate field as a string or null Dropping attributes did not cover child elements: <version><x>1</x></version>, <p><x>1</x></p> or <p>none<x/></p> still reached the parsed report, and the Elasticsearch and OpenSearch documents, as a dict, and the first such document mapped the field as an object. _scalar() now makes each field the schema types as a string a string (an element's own text) or None, applied by _scalarize_report() to version, the report_metadata strings and every policy_published child, and by _parse_report_record() to source_ip, disposition, dkim, spf, the identifiers and every auth_results string. A <count> with no text of its own takes the ValueError path of a non-numeric count. _drop_xml_attributes() stays, so a field missed here still loses its attribute keys. The <email> dict unwrap is gone, since the field can no longer be a dict; a text-only report_metadata still raises the AttributeError that reports a missing section. Comments that described attribute dicts now say the dict shape comes only from child elements. The namespace test, which the new code could not fail, is replaced by one that detects RFC 9990 through an attributed np field. Tests walk the to_dict() of a mocked Elasticsearch and OpenSearch save and allow lists and objects only in the known object fields, so a scalar field added later is covered too. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record a closed stdout pipe as an output error and fix two comments The standard-output guard in process_reports() caught only UnicodeEncodeError, so a closed pipe (BrokenPipeError, an OSError) still ended the run; it now records both as a "Standard output" output error. _reason_text()'s docstring said a dict gives None because the element has no text of its own, but mixed content (<type>forwarded<x/></type>) gives the text beside the children; the docstring now says what the code does, and a test pins it. The comment above the reason normalization no longer describes attribute dicts, which no longer reach it. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the normalization limit at its boundary and reword the index cap The suite passed with the limit's <= changed to < and with _days_touched()'s ceiling changed to a floor. New tests: 1,000 records over 250 days (exactly MAX_NORMALIZED_RECORDS) are split; 683 records over 366 days (249,978) are split and 684 (250,344) are not; 682 records over 366 days starting 1 s after midnight touch 367 days (250,294) and are not split. Each mutant was run against these tests and fails: the < mutant fails the exact-limit test, the floor mutant the after-midnight test. The Elasticsearch and OpenSearch pages now say the index cap applies when a report's records would use more than 31 dated indexes, which is what it counts (with monthly_indexes the span limit keeps a report under it). Refs GHSA-p85v-578x-3wcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Match the tests and changelog to mailsuite 2.3.5's final check mailsuite 2.3.5's check_message_limits() now parses the message with the email package's own feed parser under compat32 and counts the parts it creates (each message/delivery-status block among them), their nesting depth and its boundary tests, then limits header field length and count on the parsed tree; the limit names changed. The over-limit failure samples now trip max_parts, and the deeply nested poison message max_depth; the tests assert those texts. The GHSA-qcc2 changelog bullet describes the final check, the epilogue mail-parser parses again, and the changes that affect only mailsuite's own parse_email(). Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a repeated policy_published and note mixed-content behaviour No test failed when _scalarize_report() stopped coercing the fields of a repeated <policy_published> (the list case): with that mutant, a report with two policy_published elements and <p><x>1</x></p> puts a dict into published_policy.p. The parse test and the Elasticsearch and OpenSearch document-walk tests now include that report; each fails on the mutant. The changelog says that an element with text and a stray child element now yields its text (samples/aggregate/invalid_xml.xml gives header_from "bad" where it gave ""). AGENTS.md and the _normalize_result_word() and _scalarize_report() docstrings describe what runs now: attributes are dropped as soon as the XML is parsed, and _scalar() coerces the string fields, among them the RFC 7489 and RFC 9990 policy_published fields. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Coerce policy override reasons with the same rule as other strings _reason_text() turned a repeated <type> or <comment> into the str() of the list, while _scalar() turns every other repeated string field into None. One class of input now has one rule: the reason's type and comment go through _scalar(), so a repeated one is None ("none" in CSV), and _reason_text() is removed. The test and the GHSA-4272 changelog bullet say so. Refs GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test an aggregate <error> with a child element With attributes dropped and reasons coerced by _scalar(), only <error>, which may repeat, still reaches _text() with a dict; its branch had no test. An <error> with text beside a child element keeps the text, and one with no text is dropped. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe mailsuite's six message limits in the changelog mailsuite 2.3.5's check now counts every line of every header block (folded lines and lines the parser keeps only as defects included) and stops the parse early, and limits the total length of the address headers mail-parser parses to 262,144 characters. The GHSA-qcc2 bullet names the six limits. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render a report email's parts only where their text is read parse_report_email() did str(payload_obj[0]) for every part walk() visits, before the multipart/ continue. str() of a composite part renders its whole subtree, and walk() visits every level, so a nesting within mailsuite's limits rendered its leaf once per level: 98 nested multiparts around a 20 MiB one-line leaf took 26.6 s, and a chain of 99 message/rfc822 parts around a 10 MiB leaf 11.3 s. A part is now rendered only in the branches that read it; other message/* bodies are skipped, since walk() visits their parts. Both shapes now take under 0.5 s, and every sample under samples/failure/ parses to the same output. A feedback report's sample is taken from the first sample part only: a message/rfc822 nested inside the sample used to replace it, but a feedback report has one sample (RFC 6591 section 3.1). Multipart nesting alone no longer reaches the recursion limit, so the GHSA-qxwv test now reaches the recursion guard through a deeply nested sample, which is still rendered. The changelog describes this and the address headers mailsuite 2.3.5 counts (From, Sender, Reply-To, To, Cc, Bcc, Delivered-To, in every part; mail-parser parses all but Sender). Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Parse aggregate XML without attributes and lower the node limit _drop_xml_attributes() rebuilt the parsed tree while the original was alive, and the counting wrappers of _LimitedExpatParser referred to the proxy, closing a reference cycle through the expat parser that kept xmltodict's tree alive until the cyclic garbage collector ran. Both xmltodict passes now use xml_attribs=False, so no attribute key is ever built (expat still hands the attributes to the start handler, so they are still counted: a test shows one element with 1,000 attributes is 1,001 nodes), _drop_xml_attributes() is removed, and the counts live in an _XMLCounts object the wrappers share, so a parse leaves no cycle (a test asserts gc.collect() finds nothing after one). Peak extra memory for 2,000,000 distinct sibling elements: +999 MiB before, +386 MiB after (master +391 MiB); for one element with 2,000,000 attributes: +1,274 MiB before, +445 MiB after (master +734 MiB). At the old limit of 5,000,000 nodes a report could still need about 3 GB. Measured after the change, the worst shape is distinct elements with text behind a malformed XML declaration (the lxml recovery path): 559 bytes per node, +533 MiB for 999,000 nodes and +1,023 MiB for 2,000,000; on the expat path about 220 bytes per node. MAX_XML_NODES is now 1,000,000, so a report at the limit needs at most about 550 MiB; the largest sample has 32,019 nodes (909 KB), and the limit is a report of about 28 MB. The constant's comment and the changelog state the bound. AGENTS.md describes the attribute handling as it now is, and says that <error>, which may repeat, goes through _text() and stays a list. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qualify the mixed-content compatibility note to string fields An element with text beside a stray child element yields its text only for the fields the schema types as strings; a <begin> or <end> with mixed content is still rejected, as before. Refs GHSA-qpcg-29vr-jwm3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject over-long XML tags before any parser reads them expat and libxml2 build a start tag's whole attribute list before they report the element, so the node counter only sees the attributes after they cost memory: one element with 4,999,000 attributes (56 MiB) took 5.25 s and +767 MiB before the node limit rejected it, and a single element is bounded only by the 100 MiB decompression cap. parse_aggregate_report_xml() now scans the decoded text once, before the first parse and so before the lxml recovery path too, and rejects a document in which any text between a "<" and the next ">" (or the end of the text) is longer than MAX_XML_TAG_LENGTH (65,536 characters). The scan uses no regex. It splits the text on ">" one 64 KiB chunk at a time, carrying the unfinished piece from its last "<": a single split of the whole text makes one string per tag, which peaked at +1,318 MiB (1.9 s) for 100 MiB of "<a/>"; the chunked scan needs about 2 MiB and takes 0.6 s on that text, and a test bounds its memory and covers a tag across a chunk boundary. The longest tag in the samples is 108 characters. The same shape is now rejected in under 0.1 s, with a tracemalloc peak under 200 MB in the test, behind a well-formed or a malformed declaration. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render a report email's feedback-report part only once The message/* skip exempted message/feedback-report, which was rendered with str() every time walk() reached one: a chain of 99 nested message/feedback-report parts around a 10 MiB leaf passed mailsuite's check in 0.02 s and then took 11.6 s. A feedback report has one message/feedback-report part (RFC 6591 section 3), so, as with the sample, only the first is read and rendered; the chain now takes 0.56 s. The failure samples parse to byte-identical output. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound attribute counts by their = signs and cap reports at 32 MiB XML allows a raw > inside a quoted attribute value, so attributes such as x=">" cut a tag into short pieces that pass the tag scan: 4,999,000 of them cost 5.61 s and +781 MiB, and 8,000,000 (98 MiB) 8.9 s and +1,215 MiB, before the node limit fired. Every attribute needs an =, so a document with more = characters than MAX_XML_NODES is now rejected in one str.count() pass, before the tag scan; the samples hold at most 3. The tag scan stays for long tags, and its docstring says it does not bound attribute counts. The stated memory bound was false: text per node is bounded only by the size of the report, so at the node limit with 80 characters per element (94 MiB) the recovery path reached +827 MiB. MAX_DECOMPRESSED_REPORT_SIZE falls from 100 MiB to 32 MiB (the node limit already means a legitimate report is at most about 28 MB, at the samples' 35 nodes per KB, and the largest sample is 909 KB), and parse_aggregate_report_xml() holds an uncompressed report to the same length, so the bound holds for every input. Measured at the cap, 999,000 distinct elements with 14 characters of text (31 MiB) need +587 MiB on the recovery path and +302 MiB on the expat path; the constant comment and the changelog state it. When the first parse fails, the exception's traceback kept the partial xmltodict tree alive while the recovery parse ran; it is cleared first (+603 to +587 MiB, and +263 to +241 MiB for 330,000 elements of 80 characters). A test shows the recovery path leaves the same 18 lxml helper objects for a document ten times larger, so no tree survives a parse there either. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let the RFC 6591 feedback part win over legacy report text The guard that reads only the first message/feedback-report part tested feedback_report, which the legacy text/plain branch ("A message claiming to be from you has failed ... detected.") also sets, so a legacy text part followed by a real feedback-report part and sample now took the legacy part and never read the ARF, where master read the ARF. The guard now tests is_feedback_report, and reading the feedback-report part discards a sample a legacy part set before it. The same rule closes an older gap: a text/plain part inside the sample, which is the spoofed message's own text, could hold the legacy phrase and replace the report's Source-IP and its sample. Once a feedback-report part is read, the legacy branch is skipped; a legacy report has no such part. Tests cover both orders and the in-sample case; the failure samples parse to byte-identical output. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold every report input to the size cap and free failed parses parse_report_file() kept the aggregate parser's exception, whose traceback holds the parser's frames and the parsed tree, while it tried SMTP TLS and then the email parser: a 31 MiB file at the cap cost +1,046 MiB there against +587 MiB parsing it directly. It now keeps only each failure's text and raises from a fresh exception of the relevant format, so the chain survives without the tree. A file that begins with "<" is not tried as JSON (json.loads() decoded it all) or as an email, one that begins with "{" is not tried as an email (an RFC 822 message begins with a header line), and text over the size cap is never handed to the email parser, which has none: a 200 MiB <x/> file reached +2,801 MiB in 4.6 s and now costs +232 MiB (reading the file) in 0.05 s. extract_report() reads an uncompressed report at most one byte past the cap instead of decoding it whole. The cap counted characters for text and bytes for compressed reports, so a non-ASCII report could be four times the cap in bytes: 999,000 elements of 14 emoji (31.2 Mi characters, 71 MiB of UTF-8) cost +649 MiB. _exceeds_size_cap() measures text as UTF-8 (ASCII by length, other text 1 MiB of characters at a time, without a full copy), and bytes are checked before they are decoded. Clearing e.__traceback__ inside the except block did nothing on Python 3.10, which keeps the handled exception's traceback while the handler runs. The lxml recovery now runs after the except block, in _recover_xml(), so the first parse's frames and partial tree are unreachable when it starts on every supported Python; a test holds a weak reference to the first parse's counter and fails on the old code under Python 3.10.19 and, with the clearing removed, under 3.14. The changelog's examples are re-measured under the 32 MiB cap, and the test docstrings cite the same figures. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip the parts inside a failure report's sample Once the sample part was taken, walk() still visited its descendants, and they reached the SMTP TLS and base64 attachment branches: a sample whose spoofed message carried an SVG or zipped HTML attachment made the email rejected ("Missing field: 'feedback'", "Unable to parse"), and a gzipped aggregate report or a tlsrpt+json attachment inside it was returned as that report instead of the failure report. The sample is the reported message, so after it is taken every part inside it is skipped (by the id() of each part its walk() yields). Tests cover SVG, zipped HTML, JSON, gzipped aggregate XML, tlsrpt+json and PDF attachments; the failure samples parse to byte-identical output. The Bug fixes bullet also says that a legacy text part followed by a feedback-report part with no sample after it is now invalid, with a test. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cap SMTP TLS reports and test the format skips parse_smtp_tls_report_json() decoded and json.loads()-ed a report of any size; only the gzipped path was capped, and 60 MiB of "[]," cost +1,565 MiB. It now rejects bytes or text over MAX_DECOMPRESSED_REPORT_SIZE first, as InvalidSMTPTLSReport, so every report type is held to the cap. parse_report_file() names the actual reason it skips the email parser (the leading "<" or "{", or the size cap) rather than a combined one, and a size message for bytes no longer says "of UTF-8". New tests kill the mutants the review found surviving: removing the "<" skip for JSON, narrowing the email skip to "<", the bytes check >= for >, and removing the SMTP TLS cap. The constant comment and the changelog state the memory at the cap as measured on the current code: +345 MiB on the lxml recovery path and +302 MiB on expat parsing the text directly, +500 MiB and +452 MiB through parse_report_file(). The changelog says only the exception's cause changed for files skipped by format, not the message. Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip every sample candidate and correct the stated measurements sample_parts held only the first sample candidate's parts, so a later message/rfc822 (or one after a text/rfc822-headers sample) was dropped as a sample but its parts were still walked, and a gzipped aggregate report inside it was returned in place of the failure report. Every candidate's parts are now skipped. Tests cover both shapes, and a test that a gzipped aggregate report beside the sample is still returned pins the rule (inside the sample: skipped; beside it: read) and fails when the set is built from the whole message. parse_report_file() no longer builds email reasons for content that begins with "<" or "{", which its message never shows, and the size reason reads "Invalid report email: the report is larger than 32 MiB". The memory figures are re-measured in a fresh process with the input read from disk before the baseline: +399 MiB on the recovery path and +338 to +355 MiB on expat for the 31 MiB shape, +1,505 MiB for 60 MiB of JSON before the SMTP TLS cap; the constant comment, the changelog and the test docstrings state them. RFC 6591 section 3.1 is cited for the sample's place, and application/json is named among the attachments that made a failure report invalid. Refs GHSA-qcc2-4x3v-v4g7 Refs GHSA-q7gm-8cj9-94hr Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Scale the timed tests' bounds to the machine running them The mailsuite fix's first CI run on GitHub's shared runners failed six timed tests: the suite ran four to six times slower there than on a workstation, and checks doing real work took longer than their fixed bounds. Several bounds here have far less margin (0.2 s on a test that takes 0.18 s whole), so they would fail the same way. tests/timing.py adds budget(seconds), which measures a fixed reference parse twice just before the timed call and multiplies the bound by the slowdown against a reference machine; a faster machine keeps the written bound. Every timed assert now compares against budget(<written seconds>) computed before the timer starts. No written bound changed and no assert was removed. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 5.5
|
56e5a09923 |
Add opt-in Elasticsearch/OpenSearch data streams (#937)
* Add opt-in Elasticsearch/OpenSearch data streams (#842) New use_data_streams option in [elasticsearch] and [opensearch] saves each report type to a data stream named like today's index without the date suffix, instead of daily or monthly indexes. parsedmarc puts an exact-name index template (priority 501, @timestamp mapped as date, <stream>@custom component template on Elasticsearch) and creates the stream; retention is left to operator-managed ILM/ISM policies, with documented examples. Dated indexes are untouched and still searched for duplicates and matched by the dashboards. Every document now carries @timestamp and is saved with op_type=create in both modes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix data stream prose found by cross-check review (layer 1) Aggregate @timestamp is each record's begin date, not the report's; the cluster only rolls backing indices over under an attached policy; move the Elasticsearch 8.7 note after the template it refers to; clarify the name-collision paragraph, OpenSearch-side comments, the migrate_indexes docstring, and the test-module headers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tighten names and wiring tests found by cross-check review (layer 2) Split the use_data_streams CLI passthrough test per backend so a call site reading the other section's flag fails; name the dated-mode tests for both changes they assert; compute index_date only in dated mode in every save function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Align data stream docs with Elastic/OpenSearch references (layer 3) Say the stream must exist before reindexing into it (ES reindex API: auto-creation needs a data-stream template); an existing stream's write index needs the ILM policy attached directly since nothing rolls it over otherwise; target the stream name in the _settings example; note OpenSearch's leading-dot rule for stream names and Fleet's template priorities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cover the Loki client-initialization error wrapper _parse_config accepts a [loki] token alongside username/password; only the LokiClient constructor rejects the combination. Exercise that path through _init_output_clients with the real constructor so the RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
07bca1ad28 |
Make the output and mailbox integrations optional extras (#888)
* Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Sean WhalenandClaude Opus 5
|
b304cf8639 |
Cover index_prefix_domain_map and index_suffix in index migrations (#868) (#869)
* Cover index_prefix_domain_map and index_suffix in index migrations (#868) The startup Elasticsearch/OpenSearch index migrations built their target index names from the [elasticsearch]/[opensearch] index_prefix and index_suffix options alone, while the save path also honors general.index_prefix_domain_map. A multi-tenant deployment therefore ran the backfill guard `count` against dmarc_aggregate*/smtp_tls*, patterns matching none of its real <tenant>_dmarc_aggregate-* indexes. The query passes allow_no_indices=True, so a zero-match wildcard returns count 0 -- indistinguishable from "already backfilled" -- and the backfill was skipped silently, with no log line at any level. Resolve migration index names through a new _migration_index_names() helper that widens both configurable axes: one name per tenant prefix in the map plus the unprefixed name (unmapped domains are still saved unprefixed), and, when an index_suffix is set, the unsuffixed name alongside the suffixed one so history predating the suffix is covered. A configured index_prefix still wins outright and suppresses the map fan-out, matching save-time precedence. The key normalization is now shared with get_index_prefix() via _normalize_index_prefix(), so the names parsedmarc migrates cannot drift from the ones it writes. The resolved lists are logged at DEBUG, and the SIGHUP reload path passes the freshly parsed map, so a newly onboarded tenant is covered without a restart. Also repair the legacy published_policy.fo migration, which has been unable to complete since mapping types were removed in Elasticsearch 7 (and never existed in OpenSearch): it read the field mapping in the type-keyed response shape, so the check always fell through, and its put_mapping() call passed a doc_type argument neither current client accepts. It now reads either response shape, uses each client's current signature, and takes its index names in a separate legacy_fo_indexes argument -- exact names, since 5.0.0 introduced date-suffixed index names in the same release that fixed the fo declaration, but prefixed and suffixed where configured, since both options date back to 4.1.0. Tenant prefixes are excluded: index_prefix_domain_map arrived in 8.19.0, and this migration renames the index it rebuilds. The Elasticsearch copy, removed as unreachable during the #806 client migration, is restored now that the cause is understood. Finally, reject an index_prefix_domain_map YAML file that is not a mapping of string tenant names to lists of domains, instead of raising mid-save on a non-string key or silently matching the wrong domains on a scalar value -- `in` on a str is a substring test, so "example.co" matches "example.com" (https://docs.python.org/3/reference/expressions.html#membership-test-operations). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix OpenSearch capitalization and spacing in the usage docs Copilot review of #869: the `index_prefix_domain_map` option line spelled "OpenSearch" as "Opensearch" and had a doubled space before the type. Both predate this branch but sit inside a hunk it rewrites. Also wrapped the line to match the continuation-indent style of every other option in the list, and corrected the same misspelling in the multi-tenant section a few lines above the paragraph this branch added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Require index_prefix_domain_map domain lists to hold strings Copilot review of #869: the shape check verified that each value was a list but not what the list contained, so `tenant_a: [42]` passed and then never compared equal to any domain the save path looks up -- the same silent-misbehavior class the check exists to reject, and a contradiction of the "any other shape is rejected at startup" claim in the docs. Check the list's items too, and reword the error message, comment, CHANGELOG and docs to state the rule the check now enforces: a mapping of tenant names to lists of domain names, all strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the legacy fo migration retry-safe, and nest its mapping body Copilot review of #869, verified against live Elasticsearch 8.19 and OpenSearch 3 containers rather than mocks. Retry safety (a real defect): an attempt interrupted between creating the -v2 index and deleting the original left debris that made create() fail with "resource already exists" on every later startup, inside the same try that swallows the error -- so the index was never migrated and the debris document survived. Reproduced on a live cluster. Discard a leftover target first; that is safe precisely because reaching this point means the original still holds the data, since it is deleted only once the reindex has succeeded. Mapping body: the reviewer's concern that a dotted key under `properties` risks a runtime failure does not hold -- both clusters accept it and produce a byte-identical mapping, with the dot expanded into published_policy -> properties -> fo. Switch to the nested object form anyway, since dot expansion is conditional on the object's `subobjects` setting and this shape never is, and derive the object/leaf names from the dotted constant so the write cannot drift from the field the read looks up. Both fo-migration suites now build per-name Index mocks. A single shared mock cannot express "the original exists but its migration target does not", which is the ordinary case and the one the retry fix turns on; the happy path now also asserts the target index is never deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Fable 5
|
23c5ea9ad6 |
Fix DKIM/SPF and SMTP TLS detail-table cross-products in dashboards (#169) (#839)
* Fix DKIM/SPF alignment detail cross-product in dashboards (#169) Elasticsearch and OpenSearch dynamic-map the dkim_results/spf_results object arrays as `object` (create_indexes never registers the DSL document mappings), so Lucene flattens each array into independent multi-valued fields and stacked terms aggregations on dkim_results.selector/.domain/.result return every combination of values across a report's signatures — each phantom row repeating the full message count. Aggregate documents now also carry dkim_results_combined and spf_results_combined: one "selector / domain / result" ("scope / domain / result") string per auth result, composed in add_dkim_result/add_spf_result. The Kibana/OpenSearch Dashboards and Grafana (Elasticsearch) alignment-detail tables aggregate those instead, and the Splunk detail panels pair the values with mvzip/mvexpand. A documented idempotent _update_by_query backfills documents saved by older versions; the query matches only documents that have auth results and lack the combined fields, because an `exists` query cannot see an empty array. Also corrects the dead _SPFResult.results (plural) declaration to `result` (the save path always wrote the singular key), fixes the result parameter annotations on add_dkim_result/add_spf_result, and removes the Grafana dmarcian.com DKIM-checker data link, which required the separate domain/selector columns. The SMTP TLS visualizations have the same class of defect and are tracked separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review findings on #839 Reword the combined-field regression test docstrings: the DKIM/SPF auth results are dynamic-mapped as plain `object`, not the `nested` mapping type the previous wording implied — the distinction is the crux of the fix. Also drop the inert renameByName entries Copilot flagged on the Grafana Overview and DKIM Alignment Details panels, which referenced fields those panels' queries no longer (or never) produced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add PR #839 review lessons to AGENTS.md Extend the "Review passes cover prose" section with two rules from the #839 Copilot findings: docstrings/comments get the same text-level review pass as docs and dashboard labels (with suspicion for dual-use terms like "nested" near Elasticsearch code), and inert config entries inside hunks a PR already rewrites should be cleaned rather than preserved to minimize the diff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reflow create_indexes comments flagged by Copilot The line wrap placed "#169" directly after the comment marker, so the raw source read "# #169;". Reword so the issue reference stays on one line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Extend the hunk-proofreading rule with rendered-text wraps Fold the PR #839 second-round Copilot lesson into the existing rule: proofread how wrapped lines render (comment markers, punctuation at wrap points), not just the wording itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Rename Overview combined-result column labels (Copilot round 3) The Overview table's "DKIM Auth Result" / "SPF Auth Result" labels were kept when the columns switched to the combined "selector / domain / result" values, leaving the headers misleading. Rename them to match the detail panels' convention and retarget the byName width overrides that matched the old labels, widening them for the longer values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document per-signature row semantics in the alignment tables A message carrying multiple DKIM signatures appears once per signature in the details tables, so summing the messages column across rows can exceed the total message count. State that explicitly rather than leaving readers to infer it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot round-4 findings on dashboards Fix three pre-existing saved-object title typos in the OpenSearch ndjson (leading space on "Aggregate DMARC passed DMARC", trailing space on "Aggregate DMARC reporting organizations", double space in "map of message sources by country"), in both the top-level title and the embedded visState title. Normalize the Splunk DKIM details placeholders: the base search's fillnull renders wholly-missing DKIM fields as the literal string "null", so unsigned mail showed "null / null / null" while the SPF panel shows "none". Rewrite the values to "none" after the signature split, where the fields are single-valued and the mvzip pairing cannot be disturbed. Verified against the dev Splunk that no truncation or mis-pairing occurs either way, since fillnull guarantees the fields are never actually null. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Backfill combined DKIM/SPF fields automatically at startup migrate_indexes() now backfills dkim_results_combined and spf_results_combined on aggregate documents saved by older versions, so ES/OS users get historical data in the reworked alignment tables without running the documented _update_by_query by hand. The backfill is submitted as a non-blocking background task (wait_for_completion=false, conflicts=proceed) guarded by a cheap count query, making repeated startups a fast no-op once an index is backfilled; any cluster error is logged as a warning and retried at the next startup rather than raised. The manual command remains documented for users who upgrade dashboards without pointing the new parsedmarc at the cluster or who want to control write-load timing. The legacy published_policy.fo long-to-text reindex migration in the OpenSearch module is kept ahead of the new backfill, for clusters upgraded from very old data. Verified end-to-end against the live dev environment: a real CLI startup backfilled 9 stripped OpenSearch documents (logged with task ID) while the already-backfilled Elasticsearch side stayed silent, and a second startup was silent on both engines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Match backfill guard on either domain or result subfield End-to-end upgrade testing (real parsedmarc 10.2.4 ingest, then a branch startup) surfaced that an exists query cannot see an empty string: a text field with no tokens is invisible to exists. The parsers we audited never store an auth result with an empty or missing domain — they drop such entries entirely, so the previous domain-only guard was sufficient for their data — but the storage shape of every historical parsedmarc version can't be audited, so the guard (and the documented manual command) now matches either the domain or the result subfield per protocol. Matching either costs nothing and cannot skip a document that has something to backfill. Verified by recomputing expected combined values from _source for all 2,299 documents on both engines: every document with stored auth results has exactly the recomputed pairs, and the legacy fo migration correctly did not fire on typeless indexes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add auth-result filter controls to the Kibana/OSD aggregate dashboard The combined per-signature columns fixed the #169 cross-product but left no way to click-filter by an individual selector, domain, or result. Add an "Aggregate DMARC auth result filters" input_control_vis panel above the SPF/DKIM details tables with six option-list dropdowns (DKIM selector/domain/result, SPF scope/domain/result) that emit ordinary dashboard-wide filter pills. Works on both Kibana 8.19 and OpenSearch Dashboards 3, verified by driving the controls in both UIs against the issue's two-signature repro report. Documented in kibana.md, including the flat-mapping caveat: combining two component filters matches documents where any signature satisfies each condition individually. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scale Grafana source-country map markers with message volume The "Map of Message Source Countries" panel drew fixed 5 px dark-green markers at 50% opacity — nearly invisible on the dark basemap, so the panel read as empty even when data was flowing (verified via the query API). Markers now scale with Sum(message_count) (min 4, max 30 px) at 0.8 opacity in a higher-contrast green. Pre-existing issue; the identically-styled failure-dashboard map panel is intentionally left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Correct the nested-mapping rationale in the create_indexes comments The comments claimed Kibana/OSD/Grafana "cannot terms-aggregate fields inside a nested mapping" — too absolute. Fact-checked empirically and against primary docs: Kibana/OSD visual editors (Lens and classic Visualize) do not support nested fields, but Vega panels can run nested aggregations (they just cannot render tables, per Elastic's docs), and Grafana >= 9.4 has a nested bucket aggregation (grafana/grafana#62301) but no reverse_nested, so parent-level metrics like Sum(message_count) return 0 inside per-signature buckets (reproduced live). Conclusion unchanged: the dynamic object mapping stays load-bearing for the shipped dashboards. PR #839's body was updated with the same correction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Pin ruff exactly, matching the existing pyright pin rationale CI installs the [build] extra fresh on every run, and ruff was the one lint tool left unpinned. ruff 0.16.0 (released this week) began flagging this codebase's str.format() house style, so every PR started failing lint on lines it never touched. Pin to 0.15.21 — the version the codebase is clean under — with the same bump-deliberately comment pyright carries. Upgrading to 0.16 and converting to f-strings can be its own PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot findings: harden migrate_indexes, normalize panel titles Three unresolved review threads, all verified against cli.py's re-raising init handler before fixing: - elastic.py/opensearch.py: connections.get_connection() sat outside migrate_indexes()'s try/except, so a connection-registration failure would abort startup despite the docstring's promise that migration errors are caught and logged. Now caught, logged, and skipped until the next startup. - opensearch.py: the legacy published_policy.fo migration loop did unguarded network I/O (exists/get_field_mapping/reindex/delete), so a transient cluster error aborted startup on the OpenSearch path while the identical situation on the Elasticsearch path was logged and survived. Each index's migration attempt is now wrapped, warns, and moves on. - opensearch_dashboards.ndjson: normalized two pre-existing panel titles in the aggregate dashboard's panelsJSON ("Reporting organizations " trailing space, "Map of message sources by country" double space). Regression tests assert migrate_indexes never propagates connection or per-index cluster errors on either backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Clarify that Nested() on auth-result fields is in-memory shape only Copilot flagged that _AggregateReportDoc declares dkim_results and spf_results with Nested(...) while the create_indexes comment insists the stored mapping must stay dynamic `object`. Both are true: the Nested declaration only shapes the DSL's in-memory document building and is never installed as a mapping, because create_indexes skips Index.document() registration. Say so at both sites, in both backends, so nobody "fixes" the mismatch by registering the mapping — which would install real nested mappings and blank the dashboards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refer to the filter panel by its displayed title in docs and CHANGELOG The dashboard convention is a short panel display title backed by a long-form saved-object name ("SPF details" / "Aggregate DMARC SPF details"), and the new controls panel follows it. The docs and CHANGELOG named the panel by its saved-object title, which is not what a user sees on the dashboard; use the displayed "Auth result filters" instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix misspelled column label in the failure email samples table The "DMARC failure email samples" visualization labeled its authentication_results column "autentication_results". The underlying field reference was already correct; only the user-facing customLabel was misspelled. A sweep of every title and customLabel in the ndjson found no other misspellings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Extend the combined-field fix to SMTP TLS documents SMTP TLS reports have the same cross-product defect as the DKIM/SPF alignment tables (issue #169), one level deeper: policies is an object array and each policy's failure_details is an object array inside it, so stacked terms aggregations on their subfields fabricate rows. Documents now also carry policies_combined ("domain / type" per policy) and failure_details_combined ("domain / type / result / sending mta / receiving ip / mx" per failure detail), composed at save time with the same "none" fallbacks as the aggregate fields. migrate_indexes() gains smtp_tls_indexes and backfills old documents with the same guarded, non-blocking update_by_query pattern; cli.py wires the index name in on both backends, and the manual _update_by_query command is documented. Also fixes two adjacent dead fields: add_failure_details stored additional_information_uri under the wrong constructor kwarg (additional_information), and receiving_mx_hostname had no declaration despite always being stored. Verified live on ES 8.19 and OpenSearch 3: a two-policy repro report yields exactly 2 policy rows and 2 failure-detail rows via the combined fields where the old stacked aggregations return 4 of each; the startup backfill converted the 4 pre-existing sample documents on both engines with zero recompute mismatches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Rework the SMTP TLS dashboards onto the combined fields Kibana/OSD: "SMTP TLS domains" replaces its stacked policy_domain × policy_type terms with one terms agg on policies_combined.keyword; "SMTP TLS failure details" replaces six stacked terms spanning both array levels with one on failure_details_combined.keyword; the smtp_tls* index-pattern field cache gains the new fields. The "reporting organizations" table only buckets on doc-level org_name and needed no change. Splunk: the base search now expands policies at the JSON level (spath + mvexpand) so policy fields are scalars per event, and the failure details panel expands the second level the same way — sums are the detail's own failed_session_count, correctly paired. Verified via the search REST API: a two-policy repro returns exactly one row per real failure detail with per-detail counts. kibana.md documents the per-policy/per-detail row semantics and the honest caveat that session-count sums remain per report document. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Persist additional_info_uri from parsed SMTP TLS failure details Copilot caught that the savers read additional_information_uri from the parsed failure-detail dict, but the parser's key is additional_info_uri (SMTPTLSFailureDetailsOptional in types.py, set in parse_smtp_tls_report_json), so the URI was never persisted — the read-side half of the dead-field bug whose write-side half (wrong constructor kwarg) was fixed earlier. Read the parser's key first, keeping the long-form key as a fallback for dicts built by other callers. Regression test proven to fail on the unfixed savers. Also restructured the expected combined-string test values into named locals so no implicit string concatenation sits inside a list literal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Say "inner doc", not "nested doc", in the singular-key test docstrings Final review sweep: in this codebase "nested" is reserved for the Elasticsearch mapping type, and these InnerDoc-serialization docstrings used it colloquially — the same dual-use-term trap documented in AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Fable 5
|
40509f801b |
Migrate Elasticsearch output to the elasticsearch-py 8.x client (#822)
* Migrate Elasticsearch output to the elasticsearch-py 8.x client (#806) The mandatory elasticsearch<7.14.0 + elasticsearch-dsl==7.4.0 pins transitively forced urllib3<2 (EOL 1.26.x) onto every install. The old <7.14.0 cap only existed to dodge the client product check that broke OpenSearch users (#452, #653) — obsolete now that parsedmarc has a dedicated [opensearch] backend on opensearch-py. - Depend on elasticsearch>=8.18,<9 and drop elasticsearch-dsl entirely (the DSL ships inside the client as elasticsearch.dsl since 8.18.0). The 8.x client's elastic-transport allows urllib3>=1.26.2,<3, so installs can now resolve urllib3 2.x. The 8.x line supports both Elasticsearch 8.x and 9.x servers; ES 7.x servers are no longer supported, and OpenSearch users pointing [elasticsearch] at an OpenSearch cluster must switch to the [opensearch] section. - set_hosts() now builds 8.x connection kwargs (scheme-qualified host URLs, request_timeout, basic_auth) while keeping the function signature and every INI option unchanged. - migrate_indexes() is now a documented no-op kept for API compatibility: its only migration (re-typing published_policy.fo from long to text) applied exclusively to indices carrying the legacy ES 6-era "doc" mapping type, which cannot exist on any server the 8.x client can reach. - The elasticsearch.dsl 8.x stubs use dataclass_transform and don't surface pre-8.x-style bare `name = Text()` fields as constructor parameters; each Document/InnerDoc class now carries a TYPE_CHECKING-only `__init__(*args, **kwargs)` declaration matching the real runtime signature, which also made nine pre-existing pyright ignores unnecessary. Verified with ruff, pyright (0 errors/0 warnings), the full pytest suite (718 passed), and a CLI run over the bundled samples; CI's live elasticsearch:8.19.7 service exercises the new client end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use pass instead of ... in TYPE_CHECKING __init__ stubs CodeQL flags an ellipsis-only body as "Statement has no effect" (12 alerts on PR #822); pass is equivalent at runtime and to the type checker and keeps the alerts from resurfacing on every future scan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Sean WhalenandClaude Opus 4.7
|
411f5a8886 |
chore: tidy cSpell config; fix two doc typos (#779)
- Ignore data/export trees via cSpell.ignorePaths: parsedmarc/resources/** (maps tooling holds thousands of intentional foreign-language classifier keywords + bundled data), plus samples/** and dashboards/** (report samples and dashboard exports). These are data, not whitelist vocabulary, so excluding them keeps the editor quiet without bloating the word list. - Add the remaining genuine false-positives across code, docs, CI workflows, and editor config to cSpell.words (technical terms, library names, SQL/identifier tokens, brand/operator and multilingual examples from AGENTS.md, plus charliermarsh/junitxml/mktemp/pipefail/seanthegeek). - Fix two genuine typos found while triaging rather than whitelisting them: "maidir" -> "maildir" and "connexion" -> "connection". Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
ae1e5adb66 |
Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659)
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @
|
||
|
4e8c28bbc0 |
Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737)
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
826e78c390 |
Fix DMARC dashboard metrics (OSD + Splunk) and add dashboard-dev bootstrap (#736)
* OSD: fix aggregate dashboard metrics to sum(message_count) 13 panels on the DMARC aggregate dashboard were aggregating with `count` (number of OSD docs) when they should have been summing `message_count`. Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from the XML and carries an integer message_count, so doc-counting reports "distinct sources" rather than "messages". Panels with titles like "Message volume by header from", "DMARC passage over time", etc. were producing misleading numbers. Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and DKIM details. (DMARC failure email samples kept count — one OSD doc per RUF sample, so it's correct. SMTP TLS panels untouched — they sum the right session-count fields.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: align dashboards with OSD and fix query bugs Aggregate dashboard: - Add "Message sources by Autonomous System" panel (source_asn / as_name / as_domain), formatted "AS<n>" at render with eval, matching the OSD addition. - DKIM details: add the missing dkim_aligned column. - SPF details: reorder columns to OSD order (spf_aligned at end). - Map / country titles renamed to match OSD ("Map of message sources by country", "Message sources by country"). - Map widget: stats count by Country -> stats sum(message_count) by Country, so the choropleth shades by message volume not record count. - fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain, source_country to mirror OSD's missing-bucket labels. - charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart. Forensic dashboard: - Restructure to match OSD's two-panel layout (markdown + samples table). - Drop the country map / IP table / country-ISO table panels (not in OSD). - Samples table columns aligned to OSD: arrival_date_utc, source.ip_address, from, subject, reply_to, authentication_results. - Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=* required field to exist; LinkedIn RUF sample with null From was filtered out). SMTP TLS dashboard: - Reorder metrics to OSD order (successful before failed). - Domains panel: add policy_type bucket. - Failure details: replace search-time `failed_session_count>0` (which doesn't evaluate against multivalued JSON paths in Splunk) with `result_type=*` for presence + post-stats `where failed_sessions>0`. Drop _time/successful_sessions columns; reorder to match OSD. - Wire the existing policy_type input into all three searches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add dashboard-dev bootstrap script and VSCode task dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml, seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini, and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana, and Splunk. Idempotent: existence checks skip provisioning that's already done; only the dashboard imports re-run unconditionally on every invocation (that's the point of running it after a dashboard edit). Notable provisioning quirks the script handles: - Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships with indexes=[] and index=default; rewrites it to allow the email index. - ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily parsedmarc indexes via _cat/indices and deletes one at a time. - Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1 deletes and recreates the email index (then re-applies the HEC token). - OSD security plugin tenants: imports target global_tenant explicitly via the securitytenant header so they're visible to the shared workspace rather than landing in the API user's private tenant. Override with OSD_TENANT=<name>. - Splunk ships an in-product announcement view (scheduled_export_dashboard) with sharing=global; the script narrows it to sharing=app so it stops showing up in every app's dashboards list. Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump version to 9.10.3 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: warn against the "Create new objects with unique IDs" import mode OSD's import dialog has two modes: the default "Check for existing objects" (which honors saved-object IDs and overwrites in place when "Automatically overwrite conflicts" is on) and "Create new objects with unique IDs" (which imports under fresh UUIDs and leaves the buggy originals untouched). Picking the second one means the dashboards keep rendering the wrong numbers because the originals are never replaced. Spell that out so users don't fall into the trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: label the metric column "messages" instead of "Sum of message_count" OSD's table column header defaults to "Sum of message_count" when the metric agg has no customLabel. "messages" reads better and matches what the panels are actually counting. Applies to all 15 aggregate-DMARC visualizations that use sum(message_count). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable Trim the verbose technical exposition; lead each fix with the user-visible symptom. Move the action-required call out to its own header in upgrade notes so the re-import instructions don't get lost in a wall of text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move per-tool dashboard exports under a single dashboards/ directory Consolidates the four sibling top-level folders (kibana/, opensearch/, grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/. Updates the only path references in tracked files: bootstrap script (5 lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in docs/source/elasticsearch.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: restore the "DKIM alignment" panel title on the aggregate dashboard The DKIM alignment panel had no title override in panelsJSON, so OSD fell back to the visualization's own name ("Aggregate DMARC DKIM alignment"). Every other pie/table on the same dashboard sets a clean title (SPF alignment, Passed DMARC, etc.) — this was a stray regression. Set the panel title to "DKIM alignment" to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: color the message-disposition timechart by severity Reject is red, quarantine is yellow, none is green — same semantic mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart, applied via charting.fieldColors. Matches OSD's existing color overrides on the equivalent viz. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: clarify that "Create new objects with unique IDs" is the default The OSD import dialog defaults to that mode — users have to actively switch away from it, not just avoid picking it. Reword the upgrade note to lead with the switch and explain why the default would silently preserve the bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
 Anael Mobilia
|
3a1360a47a |
Always use systemctl command (#482)
In order to align with the rest of the documentation |
||
 Anael MobiliaandAnael Mobilia
|
3bf0bea710 |
Fix typo on documentation (#443)
* Doc cleanup/improvement * Fix typos --------- Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com> |
||
|
Anael MobiliaandAnael Mobilia
|
732547e622 |
Doc cleanup/improvement (#432)
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com> |
||
|
Anael MobiliaandAnael Mobilia
|
bf69ea8ccc |
Fix typos (#413)
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com> |
||
|
Anael Mobilia
|
062d6ea821 | Use a local network connection to elasticsearch (#407) | ||
 Sean Whalen
|
cd475255c5 | Documentation cleanup | ||
 Ben CompanjenandSean Whalen
|
2b35b785c6 |
Split and Organise documentation files (#404)
* Set global TOC collapse to false * Split documentation I tried to split the index.md file into logical parts, not changing the contents. I did add a space and change one HTTP URL to HTTPS. --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |