Release 11.2.0 (#938)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -1,30 +1,23 @@
 # Changelog
 
-## Unreleased
+## 11.2.0
 
 ### New features
 
 - **Grafana Loki output.** A new `[loki]` section pushes every aggregate, failure, and SMTP TLS report record as one JSON log line to the Loki push API (`/loki/api/v1/push`), using the same flat records as the syslog and GELF outputs plus `report_type` and `domain`. Streams are labeled `job=parsedmarc` and `host=<FQDN>`, overridable and extendable with `labels`, plus the record fields named in `label_fields` (default `report_type, domain`). Entries are timestamped at ingest by default, or at the begin of the reported period with `timestamp = report_begin`. Basic authentication, a bearer token, and a tenant ID (`X-Scope-OrgID`) are supported, and large reports are split into pushes of `batch_size` lines. Redirects are not followed, so a 3xx response counts as a failed push. A failed push raises `LokiError`, so it counts as a failed save and holds the batch's mailbox messages back for a retry. Needs only `httpx`, so it is part of the base install.
 - **Grafana dashboard for the Loki output.** `dashboards/grafana/Grafana-DMARC_Reports-Loki.json` mirrors the PostgreSQL dashboard's layout with LogQL queries against the Loki output's lines, filtered by the `domain` stream label. The dashboard dev stack now runs Loki, seeds it with the sample reports, and imports the dashboard.
+- **Elasticsearch and OpenSearch output can now write to data streams** ([#842](https://github.com/domainaware/parsedmarc/issues/842), relates to [#476](https://github.com/domainaware/parsedmarc/issues/476)). The new `use_data_streams` option in `[elasticsearch]` and `[opensearch]` (default `False`) saves each report type to one data stream named like today's indexes without the date suffix (`dmarc_aggregate`, `dmarc_failure`, `smtp_tls`, with `index_prefix`, `index_suffix`, or the `index_prefix_domain_map` prefix applied as before), instead of creating an index per day or month. That gives each report type one stable name, keeps the shard count down, and lets the cluster roll over and delete old data. parsedmarc puts an index template named after each stream, whose `index_patterns` is the exact stream name so it never applies to the dated indexes, and creates the stream; it does not create or manage retention policies. On Elasticsearch, attach an ILM policy through an optional `<stream>@custom` component template, which the template includes; on OpenSearch, an ISM policy attaches itself through its `ism_template`. `docs/source/elasticsearch.md` and `docs/source/opensearch.md` have example policies. Existing dated indexes are left alone and keep matching the dashboards' `dmarc_aggregate*`, `dmarc_f*`, and `smtp_tls*` patterns next to the stream, and the duplicate check searches both, so a report saved before the switch is not saved again after it. `monthly_indexes` is ignored when `use_data_streams` is on, and a warning says so. If a concrete index or alias already holds a stream's name, as an undated index from a release before 5.0.0 might, saving fails with an error that says so instead of writing into it. Every saved document now also carries an `@timestamp` field, in both modes: each record's begin date (`date_begin`) for aggregate reports, which is the report's begin date unless parsedmarc split a report spanning more than a day into daily records; the arrival date for failure reports; and the report's begin date for SMTP TLS reports. These are the same dates that pick the dated index. Documents are now saved with `op_type=create`, which data streams require; without a document ID this is the same as the default for dated indexes.
 
 ### Security
 
 - **S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report** ([GHSA-7wqr-f6g9-cm29](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-7wqr-f6g9-cm29), affects 11.1.0 and earlier). `S3Client.save_report_to_s3()` put the sender-controlled report ID into the object key as-is, and sent `org_name` and the other metadata values as-is. A report ID such as `../../x` added path segments to the key, an ID could carry `\`, `%`, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit ([AWS, "Naming Amazon S3 objects"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html)). botocore rejects non-ASCII metadata with `ParamValidationError`, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise `HTTPClientError: Invalid header value` before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the `x-amz-meta-*` header; the 2 KB metadata limit ([AWS, "Working with object metadata"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)) was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are `/`, every AWS "character to avoid" (`` \ { } ^ [ ] " < > ~ # | ` `` and `%`), control characters, and non-ASCII characters, so `%` appears in a key only as the escape character. The punctuation real report IDs use, such as `:`, `@`, `.`, and `_`, is kept. Each metadata value is percent-encoded (`%`, control characters, and non-ASCII characters) so `urllib.parse.unquote()` reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in `_<hash>.json`, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
 - **Credentials in URLs and the IPinfo API token are no longer written to the log** ([GHSA-hjr3-rx4c-24cv](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-hjr3-rx4c-24cv), affects 11.1.0 and earlier). The `[elasticsearch]` and `[opensearch]` `hosts` settings and the `[splunk_hec]` `url` setting accept URLs, and a URL can carry credentials (`https://user:secret@host:9200`). With `--debug`, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to `log_file`. With `--verbose` or `--debug`, parsedmarc also forwards the `httpx` library's `HTTP Request:` line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in `[general]` `nameservers` (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the `ipinfo_api_token` was logged on every IPinfo lookup as the `token` query parameter. The password, or the whole userinfo when it has no `:`, is now replaced with `***` in these lines (`https://user:***@host:9200`), and so is the `token` query value. Credentials in a `[general]` `nameservers` entry are also masked in the DNS pre-flight error, and credentials in either that setting or a `--nameservers` entry are masked in the `--debug` reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable `https://` or `tls://` entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded `/`, `?`, or `#` (see below), is now rejected before any query is sent. The download URLs `ipinfo_url`, `reverse_dns_map_url`, and `psl_overrides_url` (`[general]`) can carry credentials too: they were logged as-is at `--debug` in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with `--warnings`, `--verbose`, or `--debug`) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by `***`. A malformed `[postgresql]` `connection_string` could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors (`[loki]` `url`, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded `/`, `?`, or `#` ends the URL's authority early ([RFC 3986, section 3.2](https://www.rfc-editor.org/rfc/rfc3986#section-3.2)), so httpx reads the text after the colon as the port and raises `InvalidURL("Invalid port: 'p4ssw0rd'")`, with a chained `ValueError` that repeats that part of the password. That text reached the log from `[loki]` `url` (at startup and on a SIGHUP reload), from `[splunk_hec]` `url` (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from `ipinfo_url` (in the "Failed to save IP database" warning). A malformed `psl_overrides_url` crashed startup with the traceback, because `InvalidURL` is not an `httpx.HTTPError`. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed `psl_overrides_url` now logs a warning and falls back to the `local_psl_overrides_path` file when that is set, or else to the bundled file, and a malformed `reverse_dns_map_url` or `ipinfo_url` is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no `@` in its authority, so the `--debug` "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last `@` (`https://***@host:3100`). A URL with no credentials but an `@` in its path or query is masked the same way in these lines (`https://host/path@file` is logged as `https://***@file`). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: `user` and `password` (or `api_key`) for `[elasticsearch]` and `[opensearch]`, `user` and `password` for `[postgresql]`, `username` and `password` (or `token`) for `[loki]`, and `token` for `[splunk_hec]`. Reported by helixnebulatech.
-
-### New features
-
-- **Elasticsearch and OpenSearch output can now write to data streams** ([#842](https://github.com/domainaware/parsedmarc/issues/842), relates to [#476](https://github.com/domainaware/parsedmarc/issues/476)). The new `use_data_streams` option in `[elasticsearch]` and `[opensearch]` (default `False`) saves each report type to one data stream named like today's indexes without the date suffix (`dmarc_aggregate`, `dmarc_failure`, `smtp_tls`, with `index_prefix`, `index_suffix`, or the `index_prefix_domain_map` prefix applied as before), instead of creating an index per day or month. That gives each report type one stable name, keeps the shard count down, and lets the cluster roll over and delete old data. parsedmarc puts an index template named after each stream, whose `index_patterns` is the exact stream name so it never applies to the dated indexes, and creates the stream; it does not create or manage retention policies. On Elasticsearch, attach an ILM policy through an optional `<stream>@custom` component template, which the template includes; on OpenSearch, an ISM policy attaches itself through its `ism_template`. `docs/source/elasticsearch.md` and `docs/source/opensearch.md` have example policies. Existing dated indexes are left alone and keep matching the dashboards' `dmarc_aggregate*`, `dmarc_f*`, and `smtp_tls*` patterns next to the stream, and the duplicate check searches both, so a report saved before the switch is not saved again after it. `monthly_indexes` is ignored when `use_data_streams` is on, and a warning says so. If a concrete index or alias already holds a stream's name, as an undated index from a release before 5.0.0 might, saving fails with an error that says so instead of writing into it. Every saved document now also carries an `@timestamp` field, in both modes: each record's begin date (`date_begin`) for aggregate reports, which is the report's begin date unless parsedmarc split a report spanning more than a day into daily records; the arrival date for failure reports; and the report's begin date for SMTP TLS reports. These are the same dates that pick the dated index. Documents are now saved with `op_type=create`, which data streams require; without a document ID this is the same as the default for dated indexes.
-
-### Changes
-
 - **mailsuite 2.3.3 or later is now required.** mailsuite 2.3.3 ([GHSA-9jx3-p8wm-g8vw](https://github.com/seanthegeek/mailsuite/security/advisories/GHSA-9jx3-p8wm-g8vw)) makes the Maildir connection return a message's stored bytes from `fetch_message()` and rename the file in `move_message()`, instead of re-serializing it. A report email that cannot be re-serialized (the [GHSA-qxwv-2vwj-v828](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-qxwv-2vwj-v828) messages) is therefore now moved to the `Invalid` folder on Maildir too, instead of staying in the inbox and being logged on every run.
 
 ### Bug fixes
 
 - **Failure reports are now saved to S3.** `save_report_to_s3()` read `report["report_metadata"]` for every report type except SMTP TLS, but failure reports have no `report_metadata` (see `FailureReport` in `parsedmarc/types.py`), so every failure report raised `KeyError` since S3 output was added in 7.0.0. The error was counted as an S3 output error, so the batch holding the report was held back. A failure report is now saved under `<path>/failure/year=YYYY/month=MM/day=DD/<reported domain>_<hash>.json`, dated by its UTC arrival date, with `reported_domain` and `arrival_date` metadata. Its hash covers the report type, the arrival date, reported domain, original mail from, original recipient, original envelope ID, source IP address, and the message sample.
 - **DNS over HTTPS works again under dnspython 2.9** ([#935](https://github.com/domainaware/parsedmarc/issues/935)). dnspython 2.9.0 moved DNS over HTTPS from httpx to httpx2, and `dns.query.https()` now raises `ValueError: session parameter must be an httpx2.Client` when it is given an `httpx.Client`. parsedmarc always passed an `httpx.Client`, so every `https://` nameserver failed, and the resolver retried it until the lookup timed out. The shared DoH client is now built from the HTTP library dnspython itself imported: httpx2 on dnspython 2.9+, httpx on 2.7 and 2.8. The `nameservers` pre-flight check also reported that failure as `no PTR record for 1.1.1.1`, because `get_reverse_dns()` returns `None` for every DNS error. The check now queries the PTR record directly, so it reports `no PTR record` only when the nameserver answers that the record does not exist, and otherwise shows the resolver's actual error. The `httpx2` and `httpcore2` loggers are now included in `--debug` dependency logging, since dnspython 2.9 DoH traffic is logged under those names. An invalid `dns_test_address` is now reported by name in the pre-flight error. Reported by florianutz.
-
 - **Keep distinct aggregate reports when one reporter reuses a Report-ID for different policy domains.** RFC 9990 §3.5.1 scopes Report-ID uniqueness to the same domain. Direct report files, mbox files, and mailbox ingestion now all include the policy domain in one shared dedup key, using a JSON array to avoid ambiguous field separators. ASCII domain case differences still count as the same domain (RFC 4343 §3), without changing the returned report. Same-domain repeats remain suppressed — including a report passed both as a file and inside an mbox in one CLI run — and mailbox keys are still committed only after a successful save callback.
 
 ## 11.1.0

parsedmarc/constants.py

@@ -1,4 +1,4 @@
-__version__ = "11.1.0"
+__version__ = "11.2.0"
 
 USER_AGENT = f"parsedmarc/{__version__}"