diff --git a/CHANGELOG.md b/CHANGELOG.md index 45b4d64a..840d27bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,11 @@ - **`n_procs` parallel parsing now covers messages from mbox files and mailbox connections** ([#147](https://github.com/domainaware/parsedmarc/issues/147)), not just report files passed directly as CLI arguments: IMAP, Microsoft Graph, Gmail API, and Maildir connections, including watch mode. `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all gained an `n_procs` keyword argument. Parallel workers are now a reused process pool for the whole run, with a bounded submission window that keeps at most roughly `2 * n_procs` messages in flight at a time so memory stays bounded even for huge mboxes; the main process sends periodic IMAP keepalives while workers parse. - **Added `ParserConfig`, a single frozen dataclass carrying every parsing/enrichment option** ([#503](https://github.com/domainaware/parsedmarc/issues/503)): offline mode, IP database path, reverse DNS map and PSL overrides paths/URLs, DNS nameservers/timeout/retries, `strip_attachment_payloads`, `normalize_timespan_threshold_hours`, and the three caches the parser and enrichment code share across calls (IP address info, seen aggregate report IDs, and the reverse DNS map). `parse_aggregate_report_xml()`, `parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`, `parse_report_file()`, `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all accept a keyword-only `config=` argument; when it's provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument continues to work unchanged when `config=` is omitted. Every explicitly constructed `ParserConfig` owns fresh, isolated caches; omitting `config=` falls back to the existing process-wide shared default caches, unchanged and identity-preserved (`parsedmarc.IP_ADDRESS_CACHE`, `parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, `parsedmarc.REVERSE_DNS_MAP`). Caches never cross multiprocessing worker boundaries: `ParserConfig.__getstate__` drops the three cache fields when a config is pickled for a worker, and each worker accumulates its own from that point on. `keep_alive` and `n_procs` remain separate keyword arguments — they control process/worker orchestration, not parsing or enrichment behavior, so they're deliberately not `ParserConfig` fields. See the new "Using parsedmarc as a library" section of the usage docs for an example. +- **Added a "Domain policy" dropdown filter to the Splunk aggregate DMARC dashboard** ([#854](https://github.com/domainaware/parsedmarc/issues/854)): filters every panel on the published DMARC policy (`published_policy.p`) or subdomain policy (`published_policy.sp`), making it easy to see which domains with a `none` policy are ready to move to `quarantine` or `reject`. It sits immediately before the "Message disposition" dropdown and offers the same choices (`any`/`none`/`quarantine`/`reject`, defaulting to `any`). ### Bug fixes +- **Fixed the broken "XML files" link on the Splunk docs page**: it pointed at the old `splunk/` repository path instead of `dashboards/splunk/`, where the dashboards have lived since the `dashboards/` directory was introduced. - **A report file whose parsing raised an unexpected non-parser exception no longer hangs the CLI forever.** The old direct-file parallel implementation spawned a fresh child process per file and blocked the parent on a pipe read; a child that crashed with anything other than a `ParserError` never wrote to that pipe, so the parent waited indefinitely. The new implementation returns the error as a value and logs `Failed to parse ` instead. - **Direct-file parallel parsing no longer stalls a whole batch on one slow file, and no longer spawns a fresh interpreter per file.** The old implementation processed files in hard batches of `n_procs`, so a single slow file delayed every other file in its batch; the new pooled-worker implementation streams files through a reused pool instead. - **A one-shot mailbox run (no `--watch`) now honors `[general] dns_timeout` and `dns_retries`** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). The CLI's call to `get_dmarc_reports_from_mailbox()` never forwarded those two options, so every one-shot mailbox run silently used the library's own hardcoded default (`dns_timeout=6.0`, `dns_retries=0`) regardless of what the operator configured; watch-mode runs were unaffected since the `watch_inbox()` call site did pass them. The CLI now builds a single `ParserConfig` per run (via the new `_build_parser_config()` helper) from parsed options and passes it as `config=` to every parsing entry point — the direct-file, mbox, one-shot mailbox, and watch call sites, plus the SIGHUP config-reload path — instead of forwarding option keyword arguments by hand at each call site. diff --git a/dashboards/splunk/dmarc_aggregate_dashboard.xml b/dashboards/splunk/dmarc_aggregate_dashboard.xml index f630bc06..50ca00e9 100644 --- a/dashboards/splunk/dmarc_aggregate_dashboard.xml +++ b/dashboards/splunk/dmarc_aggregate_dashboard.xml @@ -3,7 +3,7 @@ A summary of aggregate DMARC report data - index="email" sourcetype="dmarc:aggregate" spf_aligned=$spf_aligned$ dkim_aligned=$dkim_aligned$ passed_dmarc=$passed_dmarc$ org_name=$org_name$ source_reverse_dns=$source_reverse_dns$ header_from=$header_from$ envelope_from=$envelope_from$ disposition=$disposition$ source_ip_address=$source_ip_address$ source_base_domain=$source_base_domain$ source_country=$source_country$ + index="email" sourcetype="dmarc:aggregate" spf_aligned=$spf_aligned$ dkim_aligned=$dkim_aligned$ passed_dmarc=$passed_dmarc$ org_name=$org_name$ source_reverse_dns=$source_reverse_dns$ header_from=$header_from$ envelope_from=$envelope_from$ (published_policy.p=$domain_policy$ OR published_policy.sp=$domain_policy$) disposition=$disposition$ source_ip_address=$source_ip_address$ source_base_domain=$source_base_domain$ source_country=$source_country$ | rename spf_results{}.domain as envelope_domain spf_results{}.result as spf_result spf_results{}.scope as spf_scope dkim_results{}.selector as dkim_selector dkim_results{}.domain as dkim_domain dkim_results{}.result as dkim_result | fillnull value=null source_reverse_dns source_base_domain dkim_selector dkim_domain dkim_result source_type source_name source_as_name | search dkim_selector=$dkim_selector$ dkim_domain=$dkim_domain$ source_type="$source_type$" source_name="$source_name$" source_as_name="$source_as_name$" @@ -46,6 +46,14 @@ * + + + any + none + quarantine + reject + * + any diff --git a/docs/source/splunk.md b/docs/source/splunk.md index c884ef8f..80ed5805 100644 --- a/docs/source/splunk.md +++ b/docs/source/splunk.md @@ -18,5 +18,5 @@ The Splunk dashboards display the same content and layout as the Kibana dashboards, although the Kibana dashboards have slightly easier and more flexible filtering options. -[xml files]: https://github.com/domainaware/parsedmarc/tree/master/splunk +[xml files]: https://github.com/domainaware/parsedmarc/tree/master/dashboards/splunk [http event collector (hec)]: http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC