Compare commits

...
3 Commits
Author SHA1 Message Date
48ea47adee Merge commit from fork
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv)

With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init
lines and opensearch.set_hosts() logged configured URLs verbatim, so
userinfo passwords reached the console and log_file (CWE-532).

Triage found more paths of the same class:
- the httpx "HTTP Request:" line, forwarded at --verbose/--debug,
  contains the full URL (str(httpx.URL) keeps userinfo and query), so
  webhook and HEC URL credentials and the IPinfo ?token= were logged
  on every request;
- the PSL overrides, IP database and reverse DNS map downloaders logged
  their URL at DEBUG and str(HTTPStatusError), which embeds the full
  URL, at WARNING;
- libpq echoes the DSN or a password fragment for some malformed
  PostgreSQL connection strings, logged as an output-client error.

Add redact_url_credentials() (string methods only; urlsplit misparses
scheme-less "user:pw@host:port") and use it at each log site; add a
filter on the httpx logger that masks userinfo and the token query
value; build download-failure warnings from the status code and the
redacted request URL instead of the exception text; replace psycopg
ProgrammingError detail from a connection_string with a fixed message.
Docs now recommend separate user/password settings over URL
credentials.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Mask credentials in the Loki URL in the log and LokiError text

The Loki output (#926) landed on master after this fix was written. Its
url keeps userinfo (copy_with(path=...) preserves it), and the --debug
"Initializing Loki client" line and both LokiError messages, which the
CLI logs on a failed push, included it as-is: the same CWE-532 path as
the Splunk HEC line. Mask it with redact_url_credentials() at all three
sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers

Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532):

- httpx.InvalidURL quotes the component it rejected, so a password with
  an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there)
  leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats
  it. Loki, Splunk HEC, the webhook and the three download loaders now
  report a fixed "details withheld" message raised from None.
  InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url
  crashed startup; the loaders now treat it as a download failure.
- redact_url_credentials() masks through the last "@" when the authority
  has no "@" but one follows it, so the --debug lines no longer print
  such passwords; the userinfo lookup is shared with the new nameserver
  masking.
- dnspython 2.9 DoH logs its request line on the httpx2 logger, which was
  forwarded but not filtered; the filter now covers httpx2 and
  httpx2.URL.
- [general] nameservers / --nameservers credentials are masked in the
  DNS pre-flight error, reverse DNS debug errors and per-report errors,
  and unusable https:// / tls:// ports are rejected before any query.
- Docs recommend token/username settings for [loki] and [splunk_hec];
  AGENTS.md gains rules for keeping credentials out of logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
2026-10-31 17:14:00 -04:00
Sean WhalenandClaude Opus 5.5 56e5a09923 Add opt-in Elasticsearch/OpenSearch data streams (#937)
* Add opt-in Elasticsearch/OpenSearch data streams (#842)

New use_data_streams option in [elasticsearch] and [opensearch] saves
each report type to a data stream named like today's index without the
date suffix, instead of daily or monthly indexes. parsedmarc puts an
exact-name index template (priority 501, @timestamp mapped as date,
<stream>@custom component template on Elasticsearch) and creates the
stream; retention is left to operator-managed ILM/ISM policies, with
documented examples. Dated indexes are untouched and still searched for
duplicates and matched by the dashboards. Every document now carries
@timestamp and is saved with op_type=create in both modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix data stream prose found by cross-check review (layer 1)

Aggregate @timestamp is each record's begin date, not the report's; the
cluster only rolls backing indices over under an attached policy; move
the Elasticsearch 8.7 note after the template it refers to; clarify the
name-collision paragraph, OpenSearch-side comments, the migrate_indexes
docstring, and the test-module headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tighten names and wiring tests found by cross-check review (layer 2)

Split the use_data_streams CLI passthrough test per backend so a call
site reading the other section's flag fails; name the dated-mode tests
for both changes they assert; compute index_date only in dated mode in
every save function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Align data stream docs with Elastic/OpenSearch references (layer 3)

Say the stream must exist before reindexing into it (ES reindex API:
auto-creation needs a data-stream template); an existing stream's write
index needs the ILM policy attached directly since nothing rolls it over
otherwise; target the stream name in the _settings example; note
OpenSearch's leading-dot rule for stream names and Fleet's template
priorities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cover the Loki client-initialization error wrapper

_parse_config accepts a [loki] token alongside username/password; only
the LokiClient constructor rejects the combination. Exercise that path
through _init_output_clients with the real constructor so the
RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:37:37 -04:00
Sean WhalenandClaude Opus 5.5 70f96d0734 Fix DNS over HTTPS under dnspython 2.9 and the misleading pre-flight error (#935) (#936)
* Fix DNS over HTTPS under dnspython 2.9 and the pre-flight error (#935)

dnspython 2.9 moved its [doh] extra from httpx to httpx2, and
dns.query.https() rejects a session of any other class before sending
anything (dnspython 2.9.1 dns/query.py:551: `if session and not
isinstance(session, httpx2.Client): raise ValueError("session parameter
must be an httpx2.Client")`; 2.8.0 checks httpx.Client at :566).
_get_doh_session() always built an httpx.Client, so every https://
nameserver failed and the resolver retried it until LifetimeTimeout.

The shared DoH client is now built from the Client class of the module
dnspython itself imported (dns.query.httpx2 on 2.9+, dns.query.httpx on
2.7/2.8), so it matches dnspython's isinstance check even when httpx2 is
installed beside dnspython 2.8. httpx2 keeps httpx's
`allow_env_proxies = trust_env and transport is None` (_client.py:658),
so the #880 proxy behavior carries over.

The nameservers pre-flight check called get_reverse_dns(), which turns
every DNS error into None, so the failure above was reported as "no PTR
record for 1.1.1.1". It now calls query_dns() directly and reports
"no PTR record" only for NXDOMAIN/NoAnswer, and the resolver's actual
error otherwise.

Reported by florianutz.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review of the DNS over HTTPS fix (#935)

- Add the httpx2 and httpcore2 loggers to the dependency loggers so
  --debug shows DoH traffic under dnspython 2.9+.
- Report an invalid dns_test_address by name and value in the
  pre-flight check instead of "Text input is malformed.", without
  attempting a query.
- Rewrap an over-long docstring line in _nameservers_to_resolver_input.
- Soften the test docstring about the no-[doh]-extra fallback.
- Correct the pyproject comment on what the [doh] extra installs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rewrap the dnspython [doh] comment in pyproject.toml (#935)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:16:26 -04:00
22 changed files with 3682 additions and 215 deletions

No files matched your search

+12
View File
@@ -135,6 +135,18 @@ Assume every regex in the package runs on content a remote sender controls (mess
- **Time the pattern on an adversarial input before shipping it**, not just on the samples: a 1 MB string of the repeated opener with no closer, and a 200,000-character whitespace run. A linear pattern finishes in milliseconds; anything that takes seconds is a bug. Add that input as a time-bound regression test with a generous bound (seconds) and cite the measured old timing in the docstring, as the regression tests in `tests/test_init.py` and `tests/test_utils.py` do.
- Python 3.10 is the floor, so possessive quantifiers and atomic groups (3.11+) are not available; fix the pattern's structure instead.
### Credentials must never reach logs or error text
Assume every configured URL, host list, connection string, and token carries a secret, and that every log line and exception message ends up on a console, in `log_file`, or in a pasted bug report (CWE-532). GHSA-hjr3-rx4c-24cv found leaks in `--debug` client-init lines, the forwarded httpx request line, download warnings, libpq's DSN echo, and httpx's `InvalidURL` text. Before writing a log call or an exception message that touches configuration:
- **Never log a configured value as-is.** Pass URLs and host lists through `redact_url_credentials()` (`parsedmarc/utils.py`), which masks userinfo (`https://user:***@host`), handles scheme-less `user:pw@host:port`, and, when the authority has no `@` but one follows it (a `/`, `?`, or `#` in the password, but also a credential-free URL or free text with a later `@`), masks through the last `@` (`https://***@host`). Redaction is log-only: the SDK or transport still gets the real value, so mask at the log call and not in `opts` or a client attribute.
- **Treat third-party exception text as a leak until shown otherwise.** `httpx.HTTPStatusError` embeds the full request URL. `httpx.InvalidURL` quotes the component it rejected, so a password with an unencoded `/`, `?`, or `#` becomes `Invalid port: 'p4ssw0rd'`. libpq echoes the DSN for some malformed connection strings. Don't log `str(e)` from parsing, connecting to, or requesting a configured endpoint. Build the message from structured fields (`_describe_http_error()`: status code plus the redacted request URL), or use a fixed "details withheld because they may include credentials" message. Read the library source in the venv to check what its messages contain; don't assume.
- **Raise the replacement `from None`.** `from e` and the implicit `__context__` both print the original exception when it is logged with `logger.exception` (the SIGHUP reload path does this) or when parsedmarc crashes, and `InvalidURL`'s own context is a `ValueError` that repeats the secret.
- **Catch the right class.** `httpx.InvalidURL` is not an `httpx.HTTPError`, so an `except httpx.HTTPError` falls through to a generic handler that logs the raw text, or doesn't catch it at all and crashes startup.
- **Check dependency loggers.** httpx logs every request URL at INFO. `_configure_dependency_logging()` (`parsedmarc/cli.py`) attaches `_RedactUrlCredentialsFilter` to the `httpx` and `httpx2` loggers (dnspython 2.9+ makes DNS over HTTPS requests through httpx2, and `httpx2.URL` is not an `httpx.URL`). A new dependency logger forwarded at `--verbose`/`--debug` needs the same check, and so does a new library that logs its endpoint.
- **Test both halves, including malformed input.** Assert that the secret is absent from the captured log *and* from `"".join(traceback.format_exception(exc))`, and that the real credential still reaches the mocked transport. Include values the samples never contain: a password with `/`, `?`, or `#`; userinfo with no `:` (a bare token); and a scheme-less `user:pw@host:port`.
- **Document the safe setting.** When an output has separate `user`/`username` and `password`, `token`, or `api_key` settings, its `usage.md` entry should recommend them over credentials in the URL.
## Testing standards
These rules govern *every* test added to `tests/`. They exist because the project has been burned by tests that looked like coverage but caught nothing, and by bug claims that turned out to be wrong about the spec. Both failure modes erode trust faster than missing coverage does.
+6
View File
@@ -10,6 +10,11 @@
### Security
- **S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report** ([GHSA-7wqr-f6g9-cm29](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-7wqr-f6g9-cm29), affects 11.1.0 and earlier). `S3Client.save_report_to_s3()` put the sender-controlled report ID into the object key as-is, and sent `org_name` and the other metadata values as-is. A report ID such as `../../x` added path segments to the key, an ID could carry `\`, `%`, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit ([AWS, "Naming Amazon S3 objects"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html)). botocore rejects non-ASCII metadata with `ParamValidationError`, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise `HTTPClientError: Invalid header value` before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the `x-amz-meta-*` header; the 2 KB metadata limit ([AWS, "Working with object metadata"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)) was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are `/`, every AWS "character to avoid" (`` \ { } ^ [ ] " < > ~ # | ` `` and `%`), control characters, and non-ASCII characters, so `%` appears in a key only as the escape character. The punctuation real report IDs use, such as `:`, `@`, `.`, and `_`, is kept. Each metadata value is percent-encoded (`%`, control characters, and non-ASCII characters) so `urllib.parse.unquote()` reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in `_<hash>.json`, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
- **Credentials in URLs and the IPinfo API token are no longer written to the log** ([GHSA-hjr3-rx4c-24cv](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-hjr3-rx4c-24cv), affects 11.1.0 and earlier). The `[elasticsearch]` and `[opensearch]` `hosts` settings and the `[splunk_hec]` `url` setting accept URLs, and a URL can carry credentials (`https://user:secret@host:9200`). With `--debug`, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to `log_file`. With `--verbose` or `--debug`, parsedmarc also forwards the `httpx` library's `HTTP Request:` line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in `[general]` `nameservers` (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the `ipinfo_api_token` was logged on every IPinfo lookup as the `token` query parameter. The password, or the whole userinfo when it has no `:`, is now replaced with `***` in these lines (`https://user:***@host:9200`), and so is the `token` query value. Credentials in a `[general]` `nameservers` entry are also masked in the DNS pre-flight error, and credentials in either that setting or a `--nameservers` entry are masked in the `--debug` reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable `https://` or `tls://` entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded `/`, `?`, or `#` (see below), is now rejected before any query is sent. The download URLs `ipinfo_url`, `reverse_dns_map_url`, and `psl_overrides_url` (`[general]`) can carry credentials too: they were logged as-is at `--debug` in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with `--warnings`, `--verbose`, or `--debug`) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by `***`. A malformed `[postgresql]` `connection_string` could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors (`[loki]` `url`, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded `/`, `?`, or `#` ends the URL's authority early ([RFC 3986, section 3.2](https://www.rfc-editor.org/rfc/rfc3986#section-3.2)), so httpx reads the text after the colon as the port and raises `InvalidURL("Invalid port: 'p4ssw0rd'")`, with a chained `ValueError` that repeats that part of the password. That text reached the log from `[loki]` `url` (at startup and on a SIGHUP reload), from `[splunk_hec]` `url` (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from `ipinfo_url` (in the "Failed to save IP database" warning). A malformed `psl_overrides_url` crashed startup with the traceback, because `InvalidURL` is not an `httpx.HTTPError`. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed `psl_overrides_url` now logs a warning and falls back to the `local_psl_overrides_path` file when that is set, or else to the bundled file, and a malformed `reverse_dns_map_url` or `ipinfo_url` is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no `@` in its authority, so the `--debug` "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last `@` (`https://***@host:3100`). A URL with no credentials but an `@` in its path or query is masked the same way in these lines (`https://host/path@file` is logged as `https://***@file`). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: `user` and `password` (or `api_key`) for `[elasticsearch]` and `[opensearch]`, `user` and `password` for `[postgresql]`, `username` and `password` (or `token`) for `[loki]`, and `token` for `[splunk_hec]`. Reported by helixnebulatech.
### New features
- **Elasticsearch and OpenSearch output can now write to data streams** ([#842](https://github.com/domainaware/parsedmarc/issues/842), relates to [#476](https://github.com/domainaware/parsedmarc/issues/476)). The new `use_data_streams` option in `[elasticsearch]` and `[opensearch]` (default `False`) saves each report type to one data stream named like today's indexes without the date suffix (`dmarc_aggregate`, `dmarc_failure`, `smtp_tls`, with `index_prefix`, `index_suffix`, or the `index_prefix_domain_map` prefix applied as before), instead of creating an index per day or month. That gives each report type one stable name, keeps the shard count down, and lets the cluster roll over and delete old data. parsedmarc puts an index template named after each stream, whose `index_patterns` is the exact stream name so it never applies to the dated indexes, and creates the stream; it does not create or manage retention policies. On Elasticsearch, attach an ILM policy through an optional `<stream>@custom` component template, which the template includes; on OpenSearch, an ISM policy attaches itself through its `ism_template`. `docs/source/elasticsearch.md` and `docs/source/opensearch.md` have example policies. Existing dated indexes are left alone and keep matching the dashboards' `dmarc_aggregate*`, `dmarc_f*`, and `smtp_tls*` patterns next to the stream, and the duplicate check searches both, so a report saved before the switch is not saved again after it. `monthly_indexes` is ignored when `use_data_streams` is on, and a warning says so. If a concrete index or alias already holds a stream's name, as an undated index from a release before 5.0.0 might, saving fails with an error that says so instead of writing into it. Every saved document now also carries an `@timestamp` field, in both modes: each record's begin date (`date_begin`) for aggregate reports, which is the report's begin date unless parsedmarc split a report spanning more than a day into daily records; the arrival date for failure reports; and the report's begin date for SMTP TLS reports. These are the same dates that pick the dated index. Documents are now saved with `op_type=create`, which data streams require; without a document ID this is the same as the default for dated indexes.
### Changes
@@ -18,6 +23,7 @@
### Bug fixes
- **Failure reports are now saved to S3.** `save_report_to_s3()` read `report["report_metadata"]` for every report type except SMTP TLS, but failure reports have no `report_metadata` (see `FailureReport` in `parsedmarc/types.py`), so every failure report raised `KeyError` since S3 output was added in 7.0.0. The error was counted as an S3 output error, so the batch holding the report was held back. A failure report is now saved under `<path>/failure/year=YYYY/month=MM/day=DD/<reported domain>_<hash>.json`, dated by its UTC arrival date, with `reported_domain` and `arrival_date` metadata. Its hash covers the report type, the arrival date, reported domain, original mail from, original recipient, original envelope ID, source IP address, and the message sample.
- **DNS over HTTPS works again under dnspython 2.9** ([#935](https://github.com/domainaware/parsedmarc/issues/935)). dnspython 2.9.0 moved DNS over HTTPS from httpx to httpx2, and `dns.query.https()` now raises `ValueError: session parameter must be an httpx2.Client` when it is given an `httpx.Client`. parsedmarc always passed an `httpx.Client`, so every `https://` nameserver failed, and the resolver retried it until the lookup timed out. The shared DoH client is now built from the HTTP library dnspython itself imported: httpx2 on dnspython 2.9+, httpx on 2.7 and 2.8. The `nameservers` pre-flight check also reported that failure as `no PTR record for 1.1.1.1`, because `get_reverse_dns()` returns `None` for every DNS error. The check now queries the PTR record directly, so it reports `no PTR record` only when the nameserver answers that the record does not exist, and otherwise shows the resolver's actual error. The `httpx2` and `httpcore2` loggers are now included in `--debug` dependency logging, since dnspython 2.9 DoH traffic is logged under those names. An invalid `dns_test_address` is now reported by name in the pre-flight error. Reported by florianutz.
- **Keep distinct aggregate reports when one reporter reuses a Report-ID for different policy domains.** RFC 9990 §3.5.1 scopes Report-ID uniqueness to the same domain. Direct report files, mbox files, and mailbox ingestion now all include the policy domain in one shared dedup key, using a JSON array to avoid ambiguous field separators. ASCII domain case differences still count as the same domain (RFC 4343 §3), without changing the returned report. Same-domain repeats remain suppressed — including a report passed both as a file and inside an mbox in one CLI run — and mailbox keys are still committed only after a successful save callback.
+160 -3
View File
@@ -405,13 +405,170 @@ curl -X POST "http://localhost:9200/smtp_tls*/_update_by_query?conflicts=proceed
It works identically on OpenSearch; just adjust the URL and credentials, same
as the aggregate command above.
## Records retention
## Data streams and records retention
Starting in version 5.0.0, `parsedmarc` stores data in a separate
index for each day to make it easy to comply with records
retention regulations such as GDPR. For more information,
index for each day (or each month, with `monthly_indexes`), named after
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
easy to comply with records retention regulations such as GDPR: delete
the indexes for the dates you no longer keep. For more information,
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
Set `use_data_streams = True` in the `[elasticsearch]` section to write
to [data streams] instead. Each report type then goes to one data stream
named like the indexes without the date: `dmarc_aggregate`,
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
the `index_prefix_domain_map` prefix applied as usual. The cluster
creates the stream's backing indices (`.ds-dmarc_aggregate-…`), and an
ILM policy you attach rolls them over and deletes old ones, so there is
one stable name per report type and far fewer shards.
Before a parsedmarc process first saves to a stream, it puts an index
template named after the stream and creates the stream if it does not
exist yet. The template:
- matches exactly the stream's name, with no wildcard, so it never applies
to the dated indexes;
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
from the configuration, for each new backing index;
- maps `@timestamp` as a date. Every document parsedmarc saves carries
`@timestamp`: each record's begin date (`date_begin`) for aggregate
reports, which is the report's begin date unless parsedmarc split a
report spanning more than a day into daily records; the arrival date
for failure reports; and the report's begin date for SMTP TLS reports.
The other fields are mapped dynamically, as in the dated indexes;
- includes a component template named `<stream>@custom`, such as
`dmarc_aggregate@custom`, if one exists.
Data streams need Elasticsearch 8.7 or later, because the template uses
`ignore_missing_component_templates` to make the `@custom` component
template optional.
parsedmarc puts the template again in every process, so changes made to
it directly do not last. Put your own settings, such as the ILM policy,
in the `@custom` component template instead.
parsedmarc does not create or change any ILM policy. Here is an example
that rolls each stream over every 30 days, or sooner when a primary
shard reaches 50 GB, and deletes a backing index 180 days after its
rollover:
```text
PUT _ilm/policy/parsedmarc
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "50gb"
}
}
},
"delete": {
"min_age": "180d",
"actions": {
"delete": {}
}
}
}
}
}
```
Attach it to each stream through its `@custom` component template:
```text
PUT _component_template/dmarc_aggregate@custom
{
"template": {
"settings": {
"index.lifecycle.name": "parsedmarc"
}
}
}
```
Repeat that for `dmarc_failure@custom` and `smtp_tls@custom`, and for the
prefixed or suffixed names if you use them. The component template
applies only to backing indices created after it. A stream that already
exists has a write index with no policy, so nothing would ever roll it
over; attach the policy to its existing backing indices as well:
```text
PUT dmarc_aggregate/_settings
{
"index.lifecycle.name": "parsedmarc"
}
```
The delete phase's `min_age` counts from the rollover, not from the
report dates, so a backing index is deleted 180 days after it stopped
receiving documents, whatever dates the reports in it cover. If you need
retention by report date, keep using dated indexes and delete them by
name.
On Elastic Cloud Serverless, which has no ILM, use a
[data stream lifecycle] instead:
```text
PUT _data_stream/dmarc_aggregate/_lifecycle
{
"data_retention": "180d"
}
```
### Switching an existing installation to data streams
Dated indexes that already exist are left alone. They keep working next
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
check for duplicate reports searches both, so a report that is already
in a dated index is not saved again into a stream. You can delete the
dated indexes when they age out of your retention period, or move their
documents into the streams. Documents saved by earlier releases have no
`@timestamp`, which a data stream requires, so add it while reindexing.
Run parsedmarc once with `use_data_streams = True` first, so the stream
and its template exist; a reindex into a name that is not a data stream
yet creates an ordinary index of that name instead:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate-*"},
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
"script": {
"lang": "painless",
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
}
}
```
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
`dmarc_failure-*`. Delete the dated indexes once the reindex has
finished; until then, the dashboards count every document twice.
If saving fails with `cannot create the data stream dmarc_aggregate: an
index or alias with that name already exists`, an index or alias already
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
write into such an index rather than mixing new documents into it. Move
its documents to a dated name, which the
dashboards still match, and delete it:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate"},
"dest": {"index": "dmarc_aggregate-legacy"}
}
DELETE dmarc_aggregate
```
[data stream lifecycle]: https://www.elastic.co/docs/manage-data/lifecycle/data-stream
[data streams]: https://www.elastic.co/docs/manage-data/data-store/data-streams
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
+150 -3
View File
@@ -7,8 +7,155 @@ To set up visual dashboards of DMARC data, install OpenSearch and Grafana.
OpenSearch: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/
Grafana: https://grafana.com/docs/grafana/latest/setup-grafana/installation/
## Records retention
## Data streams and records retention
Starting in version 5.0.0, `parsedmarc` stores data in a separate
index for each day to make it easy to comply with records
retention regulations such as GDPR.
index for each day (or each month, with `monthly_indexes`), named after
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
easy to comply with records retention regulations such as GDPR: delete
the indexes for the dates you no longer keep.
Set `use_data_streams = True` in the `[opensearch]` section to write to
[data streams] instead. Each report type then goes to one data stream
named like the indexes without the date: `dmarc_aggregate`,
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
the `index_prefix_domain_map` prefix applied as usual. The cluster
creates the stream's backing indices (`.ds-dmarc_aggregate-000001`, …),
and an [ISM] policy you attach rolls them over and deletes old data, so
there is one stable name per report type and far fewer shards. Amazon
OpenSearch Serverless (`aws_service = aoss`) has no data streams or
ISM, so leave `use_data_streams` off there.
Before a parsedmarc process first saves to a stream, it puts an index
template named after the stream and creates the stream if it does not
exist yet. The template:
- matches exactly the stream's name, with no wildcard, so it never applies
to the dated indexes;
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
from the configuration, for each new backing index;
- maps `@timestamp` as a date. Every document parsedmarc saves carries
`@timestamp`: each record's begin date (`date_begin`) for aggregate
reports, which is the report's begin date unless parsedmarc split a
report spanning more than a day into daily records; the arrival date
for failure reports; and the report's begin date for SMTP TLS reports.
The other fields are mapped dynamically, as in the dated indexes.
parsedmarc puts the template again in every process, so changes made to
it directly do not last.
parsedmarc does not create or change any ISM policy. Here is an example
that rolls each stream over every 30 days, or sooner when a primary
shard reaches 50 GB, and deletes a backing index 180 days after it was
created. Its `ism_template` attaches it to every backing index the
listed streams create from now on. List the stream names themselves,
including any prefix or suffix: a `.ds-dmarc_aggregate-*` pattern does
not match the backing indices.
```text
PUT _plugins/_ism/policies/parsedmarc
{
"policy": {
"description": "parsedmarc data stream retention",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_index_age": "30d",
"min_primary_shard_size": "50gb"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {"min_index_age": "180d"}
}
]
},
{
"name": "delete",
"actions": [{"delete": {}}],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": ["dmarc_aggregate", "dmarc_failure", "smtp_tls"],
"priority": 100
}
]
}
}
```
The `ism_template` only applies to backing indices created after the
policy. To attach the policy to the backing indices of a stream that
already exists:
```text
POST _plugins/_ism/add/dmarc_aggregate
{
"policy_id": "parsedmarc"
}
```
`min_index_age` counts from when the backing index was created, not
from the report dates, so a backing index is deleted 180 days after it
was created, whatever dates the reports in it cover. If you need
retention by report date, keep using dated indexes and delete them by
name.
### Switching an existing installation to data streams
Dated indexes that already exist are left alone. They keep working next
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
check for duplicate reports searches both, so a report that is already
in a dated index is not saved again into a stream. You can delete the
dated indexes when they age out of your retention period, or move their
documents into the streams. Documents saved by earlier releases have no
`@timestamp`, which a data stream requires, so add it while reindexing.
Run parsedmarc once with `use_data_streams = True` first, so the stream
and its template exist; a reindex into a name that is not a data stream
yet creates an ordinary index of that name instead:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate-*"},
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
"script": {
"lang": "painless",
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
}
}
```
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
`dmarc_failure-*`. Delete the dated indexes once the reindex has
finished; until then, the dashboards count every document twice.
If saving fails with `cannot create the data stream dmarc_aggregate: an
index or alias with that name already exists`, an index or alias already
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
write into such an index rather than mixing new documents into it. Move
its documents to a dated name, which the
dashboards still match, and delete it:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate"},
"dest": {"index": "dmarc_aggregate-legacy"}
}
DELETE dmarc_aggregate
```
[data streams]: https://docs.opensearch.org/latest/im-plugin/data-streams/
[ISM]: https://docs.opensearch.org/latest/im-plugin/ism/index/
+54 -29
View File
@@ -91,7 +91,9 @@ hosts = 127.0.0.1:9200
ssl = False
[opensearch]
hosts = https://admin:admin@127.0.0.1:9200
hosts = https://127.0.0.1:9200
user = admin
password = PasswordGoesHere
ssl = True
[splunk_hec]
@@ -571,14 +573,10 @@ The full set of configuration options is:
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
`https://user:secret@localhost`)
:::{note}
Special characters in the username or password must be
[URL encoded].
:::
- `hosts` - str: A comma-separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
Do not put credentials in the URL; use `user` and `password`
(or `api_key`) instead.
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
@@ -591,28 +589,35 @@ The full set of configuration options is:
- `index_suffix` - str: A suffix to apply to the index names
- `index_prefix` - str: A prefix to apply to the index names
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
(ignored, with a warning, when `use_data_streams` is `True`)
- `number_of_shards` - int: The number of shards to use when
creating the index (Default: `1`)
creating the index, or each backing index of a data stream
(Default: `1`)
- `number_of_replicas` - int: The number of replicas to use when
creating the index (Default: `0`)
creating the index, or each backing index of a data stream
(Default: `0`)
- `use_data_streams` - bool: Save each report type to a data stream
named like the index without the date suffix (e.g. `dmarc_aggregate`)
instead of to daily or monthly indexes. parsedmarc creates the stream
and its index template; retention is left to an ILM policy you attach. Requires Elasticsearch
8.7 or later.
See [Data streams and records retention](elasticsearch.md#data-streams-and-records-retention)
(Default: `False`)
- `serverless` - bool: Set to `True` when targeting an Elastic Cloud
Serverless project. Serverless manages sharding and replication itself
and rejects the `number_of_shards` / `number_of_replicas` index settings
with HTTP 400. With this flag set, parsedmarc strips those keys from the
settings sent at index creation; any other settings (e.g.
`refresh_interval`) are passed through unchanged (Default: `False`)
settings sent at index creation (or in a data stream's index template);
any other settings (e.g. `refresh_interval`) are passed through
unchanged (Default: `False`)
- `opensearch`
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
`https://user:secret@localhost`)
:::{note}
Special characters in the username or password must be
[URL encoded].
:::
- `hosts` - str: A comma-separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
Do not put credentials in the URL; use `user` and `password`
(or `api_key`) instead.
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
@@ -629,12 +634,24 @@ The full set of configuration options is:
- `index_suffix` - str: A suffix to apply to the index names
- `index_prefix` - str: A prefix to apply to the index names
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
(ignored, with a warning, when `use_data_streams` is `True`)
- `number_of_shards` - int: The number of shards to use when
creating the index (Default: `1`)
creating the index, or each backing index of a data stream
(Default: `1`)
- `number_of_replicas` - int: The number of replicas to use when
creating the index (Default: `0`)
creating the index, or each backing index of a data stream
(Default: `0`)
- `use_data_streams` - bool: Save each report type to a data stream
named like the index without the date suffix (e.g. `dmarc_aggregate`)
instead of to daily or monthly indexes. parsedmarc creates the stream
and its index template; retention is left to an ISM policy you attach.
OpenSearch rejects data stream names that start with `.`, so an
`index_prefix` starting with `.` cannot be used with this option.
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
(Default: `False`)
- `splunk_hec`
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
Do not put credentials in the URL; use `token` instead.
- `token` - str: The HEC token
- `index` - str: The Splunk index to use
- `skip_certificate_verification` - bool: Skip certificate
@@ -693,8 +710,13 @@ The full set of configuration options is:
- `password` - str: The database user password (Optional)
- `database` - str: The database name (Optional)
- `connection_string` - str: A full libpq connection string or URI
(e.g. `postgresql://user:pass@host/dbname`). When provided,
all individual parameters above are ignored.
(e.g. `postgresql://user@host/dbname`). When provided,
all individual parameters above are ignored. Prefer the separate
`host`, `port`, `user`, `password`, and `database` settings and do
not put the password in the connection string: a malformed
connection string can make the driver echo it back in an error
message. parsedmarc replaces such errors with a fixed message in
its log, but cannot stop the driver from producing them.
The PostgreSQL backend is an optional extra. Install it with
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
@@ -721,11 +743,13 @@ The full set of configuration options is:
database = parsedmarc
```
Or using a DSN/URI:
Or using a DSN/URI, with the password supplied separately (for
example through the `PGPASSWORD` environment variable or a
`~/.pgpass` file, both of which libpq reads):
```ini
[postgresql]
connection_string = postgresql://parsedmarc:secret@localhost/parsedmarc
connection_string = postgresql://parsedmarc@localhost/parsedmarc
```
Saving parsed data to PostgreSQL is controlled by the `[general]`
@@ -932,6 +956,8 @@ The full set of configuration options is:
It expects the default `job=parsedmarc` label and `label_fields`.
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
URL has no path. A URL with a path is used as is.
Do not put credentials in the URL; use `username` and `password`
(or `token`) instead.
- `labels` - str: Comma-separated `name=value` stream labels, merged over
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
(in a container, the FQDN is the container's hostname)
@@ -1546,5 +1572,4 @@ journalctl -u parsedmarc.service -r
:::
[cloudflare's public resolvers]: https://1.1.1.1/
[url encoded]: https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api
+148 -19
View File
@@ -19,6 +19,10 @@ from glob import escape as glob_escape, glob
from ssl import CERT_NONE, create_default_context
from typing import TYPE_CHECKING, Any
import dns.exception
import dns.query
import dns.resolver
import dns.reversename
import httpx
import yaml
from tqdm import tqdm
@@ -127,14 +131,16 @@ import parsedmarc.utils
from parsedmarc.utils import (
InvalidIPinfoAPIKey,
ReverseDNSMap,
_mask_nameserver_secrets,
configure_ipinfo_api,
get_base_domain,
get_reverse_dns,
human_timestamp_to_datetime,
is_mbox,
load_ip_db,
load_psl_overrides,
load_reverse_dns_map,
query_dns,
redact_url_credentials,
sanitize_for_log,
)
@@ -674,9 +680,66 @@ _DEPENDENCY_LOGGERS = (
"msgraph",
"httpx",
"httpcore",
# dnspython 2.9+ does DNS over HTTPS through httpx2 / httpcore2, which
# log under their own names rather than httpx / httpcore.
"httpx2",
"httpcore2",
)
# URL classes whose instances appear as log-record arguments. dnspython
# 2.9+ does DNS over HTTPS through httpx2, whose URL is not an httpx.URL;
# like _doh_client_class() in utils.py, take the module dnspython itself
# imported (absent under dnspython 2.7/2.8, which use httpx).
_HTTP_URL_CLASSES: tuple[type, ...] = (httpx.URL,) + tuple(
module.URL for module in (getattr(dns.query, "httpx2", None),) if module
)
class _RedactUrlCredentialsFilter(logging.Filter):
"""Masks URL credentials and the IPinfo ``token`` query parameter in
the arguments of httpx and httpx2 log records.
httpx and httpx2 log ``HTTP Request: %s %s "%s %d %s"`` with
``request.url``, whose string form includes the userinfo password and
the query string. httpx2 is what dnspython 2.9+ uses for DNS over HTTPS,
so a ``[general]`` ``nameservers`` URL with credentials reaches this
line on every lookup.
The filter never drops a record; the redaction itself is total (string
methods and URL parameter lookups only), so it cannot raise.
Logger-level filters apply only to records created on the logger the
filter is attached to, not to records from its child loggers. httpx
(0.28) logs only via ``logging.getLogger("httpx")`` itself and httpx2
only via ``logging.getLogger("httpx2")`` (``httpx2/_client.py``), so
attaching the filter to those two loggers covers them; a version that
logged on a child logger (``httpx.something``) would bypass it.
"""
def filter(self, record: logging.LogRecord) -> bool:
if isinstance(record.args, tuple):
record.args = tuple(self._redact(arg) for arg in record.args)
return True
@staticmethod
def _redact(arg: Any) -> Any:
if isinstance(arg, _HTTP_URL_CLASSES):
if arg.params.get("token") is not None:
# Drop the token and append the mask as typed: setting it
# with copy_set_param() would percent-encode the "*"s, and
# rewriting the encoded form afterwards would also touch a
# literal "%2A%2A%2A" elsewhere in the URL.
text = str(arg.copy_remove_param("token"))
text, fragment_sep, fragment = text.partition("#")
separator = "&" if "?" in text else "?"
text = f"{text}{separator}token=***{fragment_sep}{fragment}"
return redact_url_credentials(text)
return redact_url_credentials(str(arg))
if isinstance(arg, str):
return redact_url_credentials(arg)
return arg
def _configure_dependency_logging(level: int) -> None:
"""Apply parsedmarc's logging verbosity to dependency loggers.
@@ -690,6 +753,11 @@ def _configure_dependency_logging(level: int) -> None:
duplicates output nor keeps writing to a closed handler. Propagation
to the root logger is disabled so that a stray ``logging.basicConfig()``
anywhere in the process cannot double-print every dependency record.
The ``httpx`` and ``httpx2`` loggers also get a
:class:`_RedactUrlCredentialsFilter`
(installed once, so reloads do not stack copies) that masks URL
credentials and the IPinfo ``token`` query parameter in its
``HTTP Request:`` records.
CLI-only: library consumers configure logging themselves.
"""
dep_level = min(level, logging.WARNING)
@@ -703,6 +771,12 @@ def _configure_dependency_logging(level: int) -> None:
for wanted in logger.handlers:
if wanted not in dep_logger.handlers:
dep_logger.addHandler(wanted)
for name in ("httpx", "httpx2"):
http_logger = logging.getLogger(name)
if not any(
isinstance(f, _RedactUrlCredentialsFilter) for f in http_logger.filters
):
http_logger.addFilter(_RedactUrlCredentialsFilter())
def _load_config(config_file: str | None = None) -> ConfigParser:
@@ -819,22 +893,36 @@ def _parse_config(config: ConfigParser, opts):
opts.dns_test_address = general_config["dns_test_address"]
if "nameservers" in general_config:
opts.nameservers = _str_to_list(general_config["nameservers"])
# nameservers pre-flight check
dummy_hostname = None
# nameservers pre-flight check. query_dns() is called directly
# rather than get_reverse_dns(), which swallows every DNS error
# and returns None, so a broken transport (e.g. every DoH query
# failing, https://github.com/domainaware/parsedmarc/issues/935)
# was reported as a missing PTR record.
try:
dummy_hostname = get_reverse_dns(
opts.dns_test_address,
reverse_name = str(dns.reversename.from_address(opts.dns_test_address))
except dns.exception.SyntaxError as addr_error:
raise ConfigurationError(
f"DNS pre-flight check failed: dns_test_address "
f"{opts.dns_test_address!r} is not a valid IP address"
) from addr_error
try:
query_dns(
reverse_name,
"PTR",
nameservers=opts.nameservers,
timeout=opts.dns_timeout,
)
except Exception as ns_error:
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as ns_error:
raise ConfigurationError(
f"DNS pre-flight check failed: {ns_error}"
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {redact_url_credentials(opts.nameservers or [])}: {ns_error}"
) from ns_error
if not dummy_hostname:
except Exception as ns_error:
# A DNS over HTTPS error can quote the request URL, which
# carries the configured nameserver's credentials.
raise ConfigurationError(
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {opts.nameservers}"
)
f"DNS pre-flight check failed: {type(ns_error).__name__}: "
f"{_mask_nameserver_secrets(str(ns_error), opts.nameservers)}"
) from None
if "save_aggregate" in general_config:
opts.save_aggregate = bool(general_config.getboolean("save_aggregate"))
if "save_failure" in general_config:
@@ -1175,6 +1263,17 @@ def _parse_config(config: ConfigParser, opts):
opts.elasticsearch_serverless = elasticsearch_config.getboolean(
"serverless"
)
if "use_data_streams" in elasticsearch_config:
opts.elasticsearch_use_data_streams = bool(
elasticsearch_config.getboolean("use_data_streams")
)
if getattr(opts, "elasticsearch_use_data_streams", False) and getattr(
opts, "elasticsearch_monthly_indexes", False
):
logger.warning(
"[elasticsearch] monthly_indexes is ignored because "
"use_data_streams is enabled"
)
if "opensearch" in config:
opensearch_config = config["opensearch"]
@@ -1228,6 +1327,17 @@ def _parse_config(config: ConfigParser, opts):
opts.opensearch_aws_region = opensearch_config["aws_region"].strip()
if "aws_service" in opensearch_config:
opts.opensearch_aws_service = opensearch_config["aws_service"].strip()
if "use_data_streams" in opensearch_config:
opts.opensearch_use_data_streams = bool(
opensearch_config.getboolean("use_data_streams")
)
if getattr(opts, "opensearch_use_data_streams", False) and getattr(
opts, "opensearch_monthly_indexes", False
):
logger.warning(
"[opensearch] monthly_indexes is ignored because "
"use_data_streams is enabled"
)
if "splunk_hec" in config.sections():
hec_config = config["splunk_hec"]
@@ -1795,9 +1905,12 @@ def _search_alias_snapshot() -> list[tuple[Any, Any, Any]]:
# private is the same liberty this function already takes with
# ``connections``; without it, a failed reload that flipped
# ``[elasticsearch] serverless`` would pair the restored old client
# with the new config's flag. opensearch.py declares no module
# globals at all (no ``global`` statement in the file), so there is
# nothing to pair with it.
# with the new config's flag. opensearch.py has no equivalent flag,
# so there is nothing to pair with it. Both set_hosts() also clear
# their module's _READY_DATA_STREAMS memo, which is deliberately not
# snapshotted: an emptied memo only makes the next save re-put its
# data stream's index template, which is idempotent, so it is
# correct for the restored client too.
serverless = elastic._SERVERLESS if module is elastic else None
snapshot.append((module, connection, serverless))
return snapshot
@@ -2027,7 +2140,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
"HEC token and HEC index are required when using HEC URL"
)
try:
logger.debug("Initializing Splunk HEC client: url=%s", opts.hec)
logger.debug(
"Initializing Splunk HEC client: url=%s",
redact_url_credentials(opts.hec),
)
verify = True
if opts.hec_skip_certificate_verification:
verify = False
@@ -2094,7 +2210,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
try:
if opts.loki_url:
logger.debug("Initializing Loki client: url=%s", opts.loki_url)
logger.debug(
"Initializing Loki client: url=%s",
redact_url_credentials(opts.loki_url),
)
clients["loki_client"] = loki.LokiClient(
opts.loki_url,
labels=opts.loki_labels,
@@ -2122,8 +2241,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
# places rather than one: connections.create_connection() registers the
# new client under the ``default`` alias, and elastic.set_hosts() also
# assigns elastic._SERVERLESS. _init_output_clients() rolls both back if
# a later step fails. They are still initialized last, so that a failure
# in any other output happens before either registry has been touched.
# a later step fails. (Both set_hosts() also clear their data stream
# memo, which needs no rollback; see _search_alias_snapshot().) They are
# still initialized last, so that a failure in any other output happens
# before either registry has been touched.
if opts.save_aggregate or opts.save_failure or opts.save_smtp_tls:
# Scoped to the same condition as the constructors below, which is
# also the condition under which process_reports() dereferences
@@ -2137,7 +2258,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
if opts.elasticsearch_hosts:
logger.debug(
"Initializing Elasticsearch client: hosts=%s, ssl=%s",
opts.elasticsearch_hosts,
redact_url_credentials(opts.elasticsearch_hosts),
opts.elasticsearch_ssl,
)
es_aggregate_indexes = _migration_index_names(
@@ -2208,7 +2329,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
if opts.opensearch_hosts:
logger.debug(
"Initializing OpenSearch client: hosts=%s, ssl=%s",
opts.opensearch_hosts,
redact_url_credentials(opts.opensearch_hosts),
opts.opensearch_ssl,
)
os_aggregate_indexes = _migration_index_names(
@@ -2544,6 +2665,7 @@ def _main():
index_prefix=opts.elasticsearch_index_prefix
or get_index_prefix(report),
monthly_indexes=opts.elasticsearch_monthly_indexes,
use_data_streams=opts.elasticsearch_use_data_streams,
number_of_shards=shards,
number_of_replicas=replicas,
)
@@ -2564,6 +2686,7 @@ def _main():
index_prefix=opts.opensearch_index_prefix
or get_index_prefix(report),
monthly_indexes=opts.opensearch_monthly_indexes,
use_data_streams=opts.opensearch_use_data_streams,
number_of_shards=shards,
number_of_replicas=replicas,
)
@@ -2643,6 +2766,7 @@ def _main():
index_prefix=opts.elasticsearch_index_prefix
or get_index_prefix(report),
monthly_indexes=opts.elasticsearch_monthly_indexes,
use_data_streams=opts.elasticsearch_use_data_streams,
number_of_shards=shards,
number_of_replicas=replicas,
)
@@ -2663,6 +2787,7 @@ def _main():
index_prefix=opts.opensearch_index_prefix
or get_index_prefix(report),
monthly_indexes=opts.opensearch_monthly_indexes,
use_data_streams=opts.opensearch_use_data_streams,
number_of_shards=shards,
number_of_replicas=replicas,
)
@@ -2742,6 +2867,7 @@ def _main():
index_prefix=opts.elasticsearch_index_prefix
or get_index_prefix(report),
monthly_indexes=opts.elasticsearch_monthly_indexes,
use_data_streams=opts.elasticsearch_use_data_streams,
number_of_shards=shards,
number_of_replicas=replicas,
)
@@ -2762,6 +2888,7 @@ def _main():
index_prefix=opts.opensearch_index_prefix
or get_index_prefix(report),
monthly_indexes=opts.opensearch_monthly_indexes,
use_data_streams=opts.opensearch_use_data_streams,
number_of_shards=shards,
number_of_replicas=replicas,
)
@@ -3053,6 +3180,7 @@ def _main():
elasticsearch_password=None,
elasticsearch_api_key=None,
elasticsearch_serverless=False,
elasticsearch_use_data_streams=False,
opensearch_hosts=None,
opensearch_timeout=60,
opensearch_number_of_shards=1,
@@ -3069,6 +3197,7 @@ def _main():
opensearch_auth_type="basic",
opensearch_aws_region=None,
opensearch_aws_service="es",
opensearch_use_data_streams=False,
kafka_hosts=None,
kafka_username=None,
kafka_password=None,
+188 -33
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING, Any
from elasticsearch import Elasticsearch
from elasticsearch import Elasticsearch, NotFoundError
from elasticsearch.dsl import (
Boolean,
Date,
@@ -43,6 +43,22 @@ _SERVERLESS = False
# settings (e.g. ``refresh_interval``) are accepted and pass through.
_SERVERLESS_REJECTED_SETTINGS = frozenset({"number_of_shards", "number_of_replicas"})
# Names of the data streams whose index template and stream
# ``create_data_streams`` has already set up in this process, so each one
# costs its API calls once per process rather than once per saved document.
# ``set_hosts`` clears it, so a new connection (including a SIGHUP reload,
# which may point at another cluster or change the index settings)
# re-applies the template. Forgetting an entry is always safe: it only
# repeats idempotent calls.
_READY_DATA_STREAMS: set[str] = set()
# Priority of the index template ``create_data_streams`` installs.
# Elasticsearch's built-in templates have priority 100 and Fleet's
# integration templates up to 200, and its documentation recommends a
# priority higher than 500 for custom templates whose patterns overlap them. The template's ``index_patterns`` is the
# exact stream name, so an overlap is unlikely in the first place.
_DATA_STREAM_TEMPLATE_PRIORITY = 501
# Guard query for the dkim_results_combined/spf_results_combined backfill
# (see ``migrate_indexes``). Matches only documents that have at least one
# DKIM or SPF auth result and are missing the corresponding combined field.
@@ -620,6 +636,7 @@ def set_hosts(
# Module-global; see the _SERVERLESS comment at the top of the module.
global _SERVERLESS
_SERVERLESS = serverless
_READY_DATA_STREAMS.clear()
if not isinstance(hosts, list):
hosts = [hosts]
scheme = "https://" if use_ssl else "http://"
@@ -639,6 +656,27 @@ def set_hosts(
return connections.create_connection(**conn_params)
def _effective_index_settings(settings: dict[str, Any] | None) -> dict[str, Any]:
"""Return the index settings to send, adjusted for Serverless.
Args:
settings (dict | None): The requested index settings, or ``None``
for the defaults (one shard, no replicas).
Returns:
dict: In Serverless mode, the requested settings minus the keys in
``_SERVERLESS_REJECTED_SETTINGS``, and no defaults at all;
otherwise a copy of the requested settings, or the defaults.
"""
if settings is None:
return {} if _SERVERLESS else {"number_of_shards": 1, "number_of_replicas": 0}
if _SERVERLESS:
return {
k: v for k, v in settings.items() if k not in _SERVERLESS_REJECTED_SETTINGS
}
return dict(settings)
def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
"""
Create Elasticsearch indexes
@@ -649,16 +687,7 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
``_SERVERLESS_REJECTED_SETTINGS`` are filtered out and the
remaining keys are passed through; defaults are skipped entirely.
"""
if settings is None:
effective_settings: dict[str, Any] = (
{} if _SERVERLESS else {"number_of_shards": 1, "number_of_replicas": 0}
)
elif _SERVERLESS:
effective_settings = {
k: v for k, v in settings.items() if k not in _SERVERLESS_REJECTED_SETTINGS
}
else:
effective_settings = dict(settings)
effective_settings = _effective_index_settings(settings)
for name in names:
index = Index(name)
@@ -684,6 +713,88 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
def create_data_streams(names: list[str], settings: dict[str, Any] | None = None):
"""
Create Elasticsearch data streams and their index templates
For each name, puts an index template named after the stream whose
``index_patterns`` is exactly that name -- no wildcard, so it never
applies to the date-suffixed indexes parsedmarc writes without data
streams -- and then creates the data stream if it does not exist yet.
The template composes an optional ``<name>@custom`` component template,
which is where an operator attaches an ILM policy or other settings:
parsedmarc puts its own template again in every process that writes to
the stream, so edits made to that template directly do not last. Each
name is set up once per process; ``set_hosts`` resets that.
The template maps only ``@timestamp``. Like ``create_indexes``, it
deliberately installs no mappings for the report fields themselves (see
issue #169), so they keep their dynamic ``object`` mapping.
Args:
names (list): A list of data stream names
settings (dict): Index settings for the stream's backing indices,
adjusted for Serverless the same way ``create_indexes`` does.
Raises:
ElasticsearchError: A concrete index or alias already holds one of
the names, or the cluster returned an error. A name that fails
is not remembered, so the next save retries it.
"""
effective_settings = _effective_index_settings(settings)
for name in names:
if name in _READY_DATA_STREAMS:
continue
custom_template = f"{name}@custom"
try:
client = connections.get_connection()
# A missing name is a 404, but a name held by a concrete index
# answers 200 with an empty list (verified against
# Elasticsearch 8.19), so look for the stream in the response.
try:
response = client.indices.get_data_stream(name=name)
stream_exists = any(
stream.get("name") == name
for stream in response.get("data_streams", [])
)
except NotFoundError:
stream_exists = False
# Checked before anything is written, so a name collision leaves
# no template behind.
collision = not stream_exists and Index(name).exists()
if not collision:
client.indices.put_index_template(
name=name,
index_patterns=[name],
data_stream={},
composed_of=[custom_template],
ignore_missing_component_templates=[custom_template],
priority=_DATA_STREAM_TEMPLATE_PRIORITY,
template={
"settings": effective_settings,
"mappings": {"properties": {"@timestamp": {"type": "date"}}},
},
meta={
"description": (
"Managed by parsedmarc, which overwrites it. Put "
f"customizations in {custom_template}."
)
},
)
if not stream_exists:
logger.debug(f"Creating Elasticsearch data stream: {name}")
client.indices.create_data_stream(name=name)
except Exception as e:
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
if collision:
raise ElasticsearchError(
f"Elasticsearch error: cannot create the data stream {name}: "
"an index or alias with that name already exists"
)
_READY_DATA_STREAMS.add(name)
_LEGACY_FO_FIELD = "published_policy.fo"
# The same field split into the object/leaf names a mapping body nests it
# under. Derived from the dotted name so the mapping written below cannot
@@ -757,6 +868,12 @@ def migrate_indexes(
next startup, and the manual ``_update_by_query`` commands documented in
``docs/source/elasticsearch.md`` remain available in the meantime.
Both work unchanged with ``use_data_streams``: a data stream carries the
undated name itself, so the backfill patterns match it as well as any
dated indexes, and a stream that holds that exact name is skipped by
the ``published_policy.fo`` migration, because the backing index it
inspects maps the field as ``text``, not ``long``.
Args:
aggregate_indexes (list): A list of aggregate index names
failure_indexes (list): A list of failure index names
@@ -922,6 +1039,7 @@ def save_aggregate_report_to_elasticsearch(
monthly_indexes: bool | None = False,
number_of_shards: int = 1,
number_of_replicas: int = 0,
use_data_streams: bool = False,
):
"""
Saves a parsed DMARC aggregate report to Elasticsearch
@@ -931,8 +1049,11 @@ def save_aggregate_report_to_elasticsearch(
index_suffix (str): The suffix of the name of the index to save to
index_prefix (str): The prefix of the name of the index to save to
monthly_indexes (bool): Use monthly indexes instead of daily indexes
(ignored when ``use_data_streams`` is True)
number_of_shards (int): The number of shards to use in the index
number_of_replicas (int): The number of replicas to use in the index
use_data_streams (bool): Save to a data stream named like the index,
without the date suffix, instead of to a dated index
Raises:
AlreadySaved
@@ -959,6 +1080,8 @@ def save_aggregate_report_to_elasticsearch(
search_index = "dmarc_aggregate*"
if index_prefix is not None:
search_index = f"{index_prefix}{search_index}"
# The trailing wildcard matches the data stream as well as the dated
# indexes, so duplicates are caught across a switch between the two.
search = Search(index=search_index)
query = org_name_query & report_id_query & domain_query
query = query & begin_date_query & end_date_query
@@ -1005,10 +1128,6 @@ def save_aggregate_report_to_elasticsearch(
)
normalized_timespan = record["normalized_timespan"]
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
aggregate_report["begin_date"] = begin_date
aggregate_report["end_date"] = end_date
date_range = [aggregate_report["begin_date"], aggregate_report["end_date"]]
@@ -1049,6 +1168,7 @@ def save_aggregate_report_to_elasticsearch(
"discovery_method"
),
generator=metadata.get("generator"),
**{"@timestamp": begin_date},
)
for override in record["policy_evaluated"]["policy_override_reasons"]:
@@ -1078,15 +1198,24 @@ def save_aggregate_report_to_elasticsearch(
if index_prefix:
index = f"{index_prefix}{index}"
index = f"{index}-{index_date}"
index_settings = dict(
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
)
create_indexes([index], index_settings)
if use_data_streams:
create_data_streams([index], index_settings)
else:
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
create_indexes([index], index_settings)
agg_doc.meta.index = index # pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]
try:
agg_doc.save()
# Data streams accept only op_type=create. No _id is set, so
# for a dated index it means the same as the default.
agg_doc.save(op_type="create")
except Exception as e:
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
@@ -1098,6 +1227,7 @@ def save_failure_report_to_elasticsearch(
monthly_indexes: bool | None = False,
number_of_shards: int = 1,
number_of_replicas: int = 0,
use_data_streams: bool = False,
):
"""
Saves a parsed DMARC failure report to Elasticsearch
@@ -1107,10 +1237,14 @@ def save_failure_report_to_elasticsearch(
index_suffix (str): The suffix of the name of the index to save to
index_prefix (str): The prefix of the name of the index to save to
monthly_indexes (bool): Use monthly indexes instead of daily
indexes
indexes (ignored when ``use_data_streams``
is True)
number_of_shards (int): The number of shards to use in the index
number_of_replicas (int): The number of replicas to use in the
index
use_data_streams (bool): Save to a data stream named like the
index, without the date suffix, instead
of to a dated index
Raises:
AlreadySaved
@@ -1145,6 +1279,8 @@ def save_failure_report_to_elasticsearch(
search_index = ",".join(
f"{index_prefix}{part}" for part in search_index.split(",")
)
# The trailing wildcards match the data stream as well as the dated
# indexes, so duplicates are caught across a switch between the two.
search = Search(index=search_index)
q = Q(dict(match=dict(arrival_date=arrival_date_epoch_milliseconds))) # pyright: ignore[reportArgumentType]
@@ -1253,6 +1389,7 @@ def save_failure_report_to_elasticsearch(
dkim_domain=failure_report["dkim_domain"],
original_rcpt_to=failure_report["original_rcpt_to"],
sample=sample,
**{"@timestamp": arrival_date},
)
index = "dmarc_failure"
@@ -1260,18 +1397,23 @@ def save_failure_report_to_elasticsearch(
index = f"{index}_{index_suffix}"
if index_prefix:
index = f"{index_prefix}{index}"
if monthly_indexes:
index_date = arrival_date.strftime("%Y-%m")
else:
index_date = arrival_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
index_settings = dict(
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
)
create_indexes([index], index_settings)
if use_data_streams:
create_data_streams([index], index_settings)
else:
if monthly_indexes:
index_date = arrival_date.strftime("%Y-%m")
else:
index_date = arrival_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
create_indexes([index], index_settings)
failure_doc.meta.index = index # pyright: ignore[reportAttributeAccessIssue, reportOptionalMemberAccess]
try:
failure_doc.save()
# Data streams accept only op_type=create. No _id is set, so
# for a dated index it means the same as the default.
failure_doc.save(op_type="create")
except Exception as e:
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
except KeyError as e:
@@ -1287,6 +1429,7 @@ def save_smtp_tls_report_to_elasticsearch(
monthly_indexes: bool = False,
number_of_shards: int = 1,
number_of_replicas: int = 0,
use_data_streams: bool = False,
):
"""
Saves a parsed SMTP TLS report to Elasticsearch
@@ -1296,8 +1439,11 @@ def save_smtp_tls_report_to_elasticsearch(
index_suffix (str): The suffix of the name of the index to save to
index_prefix (str): The prefix of the name of the index to save to
monthly_indexes (bool): Use monthly indexes instead of daily indexes
(ignored when ``use_data_streams`` is True)
number_of_shards (int): The number of shards to use in the index
number_of_replicas (int): The number of replicas to use in the index
use_data_streams (bool): Save to a data stream named like the index,
without the date suffix, instead of to a dated index
Raises:
AlreadySaved
@@ -1310,10 +1456,6 @@ def save_smtp_tls_report_to_elasticsearch(
end_date = human_timestamp_to_datetime(report["end_date"], to_utc=True)
begin_date_human = begin_date.strftime("%Y-%m-%d %H:%M:%SZ")
end_date_human = end_date.strftime("%Y-%m-%d %H:%M:%SZ")
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
report = report.copy()
report["begin_date"] = begin_date
report["end_date"] = end_date
@@ -1329,6 +1471,8 @@ def save_smtp_tls_report_to_elasticsearch(
search_index = "smtp_tls*"
if index_prefix is not None:
search_index = f"{index_prefix}{search_index}"
# The trailing wildcard matches the data stream as well as the dated
# indexes, so duplicates are caught across a switch between the two.
search = Search(index=search_index)
query = org_name_query & report_id_query
query = query & begin_date_query & end_date_query
@@ -1355,7 +1499,12 @@ def save_smtp_tls_report_to_elasticsearch(
index = f"{index}_{index_suffix}"
if index_prefix:
index = f"{index_prefix}{index}"
index = f"{index}-{index_date}"
if not use_data_streams:
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
index_settings = dict(
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
)
@@ -1367,6 +1516,7 @@ def save_smtp_tls_report_to_elasticsearch(
date_end=report["end_date"],
contact_info=report["contact_info"],
report_id=report["report_id"],
**{"@timestamp": report["begin_date"]},
)
for policy in report["policies"]:
@@ -1448,11 +1598,16 @@ def save_smtp_tls_report_to_elasticsearch(
)
smtp_tls_doc.policies.append(policy_doc)
create_indexes([index], index_settings)
if use_data_streams:
create_data_streams([index], index_settings)
else:
create_indexes([index], index_settings)
smtp_tls_doc.meta.index = index # pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]
try:
smtp_tls_doc.save()
# Data streams accept only op_type=create. No _id is set, so for a
# dated index it means the same as the default.
smtp_tls_doc.save(op_type="create")
except Exception as e:
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
+21 -8
View File
@@ -19,7 +19,7 @@ from parsedmarc import (
from parsedmarc.constants import USER_AGENT
from parsedmarc.log import logger
from parsedmarc.types import AggregateReport, FailureReport, SMTPTLSReport
from parsedmarc.utils import human_timestamp_to_unix_timestamp
from parsedmarc.utils import human_timestamp_to_unix_timestamp, redact_url_credentials
PUSH_PATH = "/loki/api/v1/push"
TIMESTAMP_MODES = ("ingest", "report_begin")
@@ -102,11 +102,22 @@ class LokiClient(object):
if not _LABEL_NAME.match(name):
raise ValueError(f"Invalid Loki label name '{name}'")
parsed = httpx.URL(url)
# copy_with(path=...) keeps any query string, userinfo and port intact
self.url = (
str(parsed.copy_with(path=PUSH_PATH)) if parsed.path in ("", "/") else url
)
try:
parsed = httpx.URL(url)
# copy_with(path=...) keeps any query string, userinfo and port intact
self.url = (
str(parsed.copy_with(path=PUSH_PATH))
if parsed.path in ("", "/")
else url
)
except httpx.InvalidURL:
# httpx quotes part of the URL (part of a password containing
# "/", "?" or "#" ends up as the "port"), and chains a ValueError
# that repeats it, so neither is kept.
raise ValueError(
"Invalid Loki URL (details withheld because they may include "
"credentials)"
) from None
self.timestamp = timestamp
self.timeout = timeout
self.batch_size = max(1, int(batch_size))
@@ -196,14 +207,16 @@ class LokiClient(object):
self.url, content=json.dumps(body), timeout=self.timeout
)
except httpx.HTTPError as error_:
raise LokiError(f"Loki push to {self.url} failed: {error_}") from error_
raise LokiError(
f"Loki push to {redact_url_credentials(self.url)} failed: {error_}"
) from error_
# Loki answers 204 when it stored the push; 200 is also accepted.
# Other 2xx codes are not success: with block_ingestion_until set it
# returns a custom 260 and discards the logs
# (block_ingestion_status_code).
if response.status_code not in (200, 204):
raise LokiError(
f"Loki push to {self.url} failed with HTTP "
f"Loki push to {redact_url_credentials(self.url)} failed with HTTP "
f"{response.status_code}: {response.text[:500]}"
)
+183 -24
View File
@@ -16,6 +16,7 @@ from opensearchpy import (
Ip,
Keyword,
Nested,
NotFoundError,
Object,
OpenSearch,
Q,
@@ -28,13 +29,30 @@ from opensearchpy.helpers import reindex
from parsedmarc import InvalidFailureReport
from parsedmarc.log import logger
from parsedmarc.utils import human_timestamp_to_datetime
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
class OpenSearchError(Exception):
"""Raised when an OpenSearch error occurs"""
# Names of the data streams whose index template and stream
# ``create_data_streams`` has already set up in this process, so each one
# costs its API calls once per process rather than once per saved document.
# ``set_hosts`` clears it, so a new connection (including a SIGHUP reload,
# which may point at another cluster or change the index settings)
# re-applies the template. Forgetting an entry is always safe: it only
# repeats idempotent calls.
_READY_DATA_STREAMS: set[str] = set()
# Priority of the index template ``create_data_streams`` installs. OpenSearch
# itself installs no index templates, but plugins may; 501 stays above the
# usual priority-100 convention and matches the Elasticsearch module. The
# template's ``index_patterns`` is the exact stream name, so an overlap with
# another template is unlikely in the first place.
_DATA_STREAM_TEMPLATE_PRIORITY = 501
# Guard query for the dkim_results_combined/spf_results_combined backfill
# (see ``migrate_indexes``). Matches only documents that have at least one
# DKIM or SPF auth result and are missing the corresponding combined field.
@@ -531,9 +549,14 @@ def set_hosts(
alias. Callers that need to close this exact client later (rather than
whatever holds the alias at that point) should hold on to it.
"""
_READY_DATA_STREAMS.clear()
if not isinstance(hosts, list):
hosts = [hosts]
logger.debug("Connecting to OpenSearch: hosts=%s, use_ssl=%s", hosts, use_ssl)
logger.debug(
"Connecting to OpenSearch: hosts=%s, use_ssl=%s",
redact_url_credentials(hosts),
use_ssl,
)
conn_params = {"hosts": hosts, "timeout": timeout}
if use_ssl:
conn_params["use_ssl"] = True
@@ -605,6 +628,98 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
def create_data_streams(names: list[str], settings: dict[str, Any] | None = None):
"""
Create OpenSearch data streams and their index templates
For each name, puts an index template named after the stream whose
``index_patterns`` is exactly that name -- no wildcard, so it never
applies to the date-suffixed indexes parsedmarc writes without data
streams -- and then creates the data stream if it does not exist yet.
parsedmarc puts its template again in every process that writes to the
stream, so edits made to it directly do not last; an ISM policy attaches
itself to the stream's backing indices through its own
``ism_template`` instead. Each name is set up once per process;
``set_hosts`` resets that.
The template maps only ``@timestamp``. Like ``create_indexes``, it
deliberately installs no mappings for the report fields themselves (see
issue #169), so they keep their dynamic ``object`` mapping.
Args:
names (list): A list of data stream names
settings (dict): Index settings for the stream's backing indices;
one shard and no replicas when ``None``, as in
``create_indexes``.
Raises:
OpenSearchError: A concrete index or alias already holds one of the
names, or the cluster returned an error. A name that fails is
not remembered, so the next save retries it.
"""
if settings is None:
effective_settings: dict[str, Any] = {
"number_of_shards": 1,
"number_of_replicas": 0,
}
else:
effective_settings = dict(settings)
for name in names:
if name in _READY_DATA_STREAMS:
continue
try:
client = connections.get_connection()
# OpenSearch 3 answers both a missing name and a name held by a
# concrete index with a 404. Elasticsearch answers the latter
# with 200 and an empty list, so, as in the Elasticsearch module,
# look for the stream in the response rather than trust the
# status alone.
try:
response = client.indices.get_data_stream(name=name)
stream_exists = any(
stream.get("name") == name
for stream in response.get("data_streams", [])
)
except NotFoundError:
stream_exists = False
# Checked before anything is written, so a name collision leaves
# no template behind.
collision = not stream_exists and Index(name).exists()
if not collision:
client.indices.put_index_template(
name=name,
body={
"index_patterns": [name],
"data_stream": {},
"priority": _DATA_STREAM_TEMPLATE_PRIORITY,
"template": {
"settings": effective_settings,
"mappings": {
"properties": {"@timestamp": {"type": "date"}}
},
},
"_meta": {
"description": (
"Managed by parsedmarc, which overwrites it. "
"Attach ISM policies with an ism_template."
)
},
},
)
if not stream_exists:
logger.debug(f"Creating OpenSearch data stream: {name}")
client.indices.create_data_stream(name=name)
except Exception as e:
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
if collision:
raise OpenSearchError(
f"OpenSearch error: cannot create the data stream {name}: "
"an index or alias with that name already exists"
)
_READY_DATA_STREAMS.add(name)
_LEGACY_FO_FIELD = "published_policy.fo"
# The same field split into the object/leaf names a mapping body nests it
# under. Derived from the dotted name so the mapping written below cannot
@@ -678,6 +793,12 @@ def migrate_indexes(
``failure_details_combined`` fields (same issue #169) on SMTP TLS
report documents, for each name in ``smtp_tls_indexes``.
All three work unchanged with ``use_data_streams``: a data stream
carries the undated name itself, so the backfill patterns match it as
well as any dated indexes, and a stream that holds that exact name is
skipped by the ``published_policy.fo`` migration, because the backing
index it inspects maps the field as ``text``, not ``long``.
Args:
aggregate_indexes (list): A list of aggregate index names
failure_indexes (list): A list of failure index names
@@ -853,6 +974,7 @@ def save_aggregate_report_to_opensearch(
monthly_indexes: bool = False,
number_of_shards: int = 1,
number_of_replicas: int = 0,
use_data_streams: bool = False,
):
"""
Saves a parsed DMARC aggregate report to OpenSearch
@@ -862,8 +984,11 @@ def save_aggregate_report_to_opensearch(
index_suffix (str): The suffix of the name of the index to save to
index_prefix (str): The prefix of the name of the index to save to
monthly_indexes (bool): Use monthly indexes instead of daily indexes
(ignored when ``use_data_streams`` is True)
number_of_shards (int): The number of shards to use in the index
number_of_replicas (int): The number of replicas to use in the index
use_data_streams (bool): Save to a data stream named like the index,
without the date suffix, instead of to a dated index
Raises:
AlreadySaved
@@ -890,6 +1015,8 @@ def save_aggregate_report_to_opensearch(
search_index = "dmarc_aggregate*"
if index_prefix is not None:
search_index = f"{index_prefix}{search_index}"
# The trailing wildcard matches the data stream as well as the dated
# indexes, so duplicates are caught across a switch between the two.
search = Search(index=search_index)
query = org_name_query & report_id_query & domain_query
query = query & begin_date_query & end_date_query
@@ -933,10 +1060,6 @@ def save_aggregate_report_to_opensearch(
)
normalized_timespan = record["normalized_timespan"]
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
aggregate_report["begin_date"] = begin_date
aggregate_report["end_date"] = end_date
date_range = [aggregate_report["begin_date"], aggregate_report["end_date"]]
@@ -977,6 +1100,7 @@ def save_aggregate_report_to_opensearch(
"discovery_method"
),
generator=metadata.get("generator"),
**{"@timestamp": begin_date},
)
for override in record["policy_evaluated"]["policy_override_reasons"]:
@@ -1006,15 +1130,24 @@ def save_aggregate_report_to_opensearch(
if index_prefix:
index = f"{index_prefix}{index}"
index = f"{index}-{index_date}"
index_settings = dict(
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
)
create_indexes([index], index_settings)
if use_data_streams:
create_data_streams([index], index_settings)
else:
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
create_indexes([index], index_settings)
agg_doc.meta.index = index
try:
agg_doc.save()
# Data streams accept only op_type=create. No _id is set, so
# for a dated index it means the same as the default.
agg_doc.save(op_type="create")
except Exception as e:
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
@@ -1026,6 +1159,7 @@ def save_failure_report_to_opensearch(
monthly_indexes: bool = False,
number_of_shards: int = 1,
number_of_replicas: int = 0,
use_data_streams: bool = False,
):
"""
Saves a parsed DMARC failure report to OpenSearch
@@ -1035,10 +1169,14 @@ def save_failure_report_to_opensearch(
index_suffix (str): The suffix of the name of the index to save to
index_prefix (str): The prefix of the name of the index to save to
monthly_indexes (bool): Use monthly indexes instead of daily
indexes
indexes (ignored when ``use_data_streams``
is True)
number_of_shards (int): The number of shards to use in the index
number_of_replicas (int): The number of replicas to use in the
index
use_data_streams (bool): Save to a data stream named like the
index, without the date suffix, instead
of to a dated index
Raises:
AlreadySaved
@@ -1073,6 +1211,8 @@ def save_failure_report_to_opensearch(
search_index = ",".join(
f"{index_prefix}{part}" for part in search_index.split(",")
)
# The trailing wildcards match the data stream as well as the dated
# indexes, so duplicates are caught across a switch between the two.
search = Search(index=search_index)
q = Q(dict(match=dict(arrival_date=arrival_date_epoch_milliseconds)))
@@ -1179,6 +1319,7 @@ def save_failure_report_to_opensearch(
dkim_domain=failure_report["dkim_domain"],
original_rcpt_to=failure_report["original_rcpt_to"],
sample=sample,
**{"@timestamp": arrival_date},
)
index = "dmarc_failure"
@@ -1186,18 +1327,23 @@ def save_failure_report_to_opensearch(
index = f"{index}_{index_suffix}"
if index_prefix:
index = f"{index_prefix}{index}"
if monthly_indexes:
index_date = arrival_date.strftime("%Y-%m")
else:
index_date = arrival_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
index_settings = dict(
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
)
create_indexes([index], index_settings)
if use_data_streams:
create_data_streams([index], index_settings)
else:
if monthly_indexes:
index_date = arrival_date.strftime("%Y-%m")
else:
index_date = arrival_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
create_indexes([index], index_settings)
failure_doc.meta.index = index
try:
failure_doc.save()
# Data streams accept only op_type=create. No _id is set, so
# for a dated index it means the same as the default.
failure_doc.save(op_type="create")
except Exception as e:
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
except KeyError as e:
@@ -1213,6 +1359,7 @@ def save_smtp_tls_report_to_opensearch(
monthly_indexes: bool = False,
number_of_shards: int = 1,
number_of_replicas: int = 0,
use_data_streams: bool = False,
):
"""
Saves a parsed SMTP TLS report to OpenSearch
@@ -1222,8 +1369,11 @@ def save_smtp_tls_report_to_opensearch(
index_suffix (str): The suffix of the name of the index to save to
index_prefix (str): The prefix of the name of the index to save to
monthly_indexes (bool): Use monthly indexes instead of daily indexes
(ignored when ``use_data_streams`` is True)
number_of_shards (int): The number of shards to use in the index
number_of_replicas (int): The number of replicas to use in the index
use_data_streams (bool): Save to a data stream named like the index,
without the date suffix, instead of to a dated index
Raises:
AlreadySaved
@@ -1236,10 +1386,6 @@ def save_smtp_tls_report_to_opensearch(
end_date = human_timestamp_to_datetime(report["end_date"], to_utc=True)
begin_date_human = begin_date.strftime("%Y-%m-%d %H:%M:%SZ")
end_date_human = end_date.strftime("%Y-%m-%d %H:%M:%SZ")
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
report = report.copy()
report["begin_date"] = begin_date
report["end_date"] = end_date
@@ -1255,6 +1401,8 @@ def save_smtp_tls_report_to_opensearch(
search_index = "smtp_tls*"
if index_prefix is not None:
search_index = f"{index_prefix}{search_index}"
# The trailing wildcard matches the data stream as well as the dated
# indexes, so duplicates are caught across a switch between the two.
search = Search(index=search_index)
query = org_name_query & report_id_query
query = query & begin_date_query & end_date_query
@@ -1281,7 +1429,12 @@ def save_smtp_tls_report_to_opensearch(
index = f"{index}_{index_suffix}"
if index_prefix:
index = f"{index_prefix}{index}"
index = f"{index}-{index_date}"
if not use_data_streams:
if monthly_indexes:
index_date = begin_date.strftime("%Y-%m")
else:
index_date = begin_date.strftime("%Y-%m-%d")
index = f"{index}-{index_date}"
index_settings = dict(
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
)
@@ -1293,6 +1446,7 @@ def save_smtp_tls_report_to_opensearch(
date_end=report["end_date"],
contact_info=report["contact_info"],
report_id=report["report_id"],
**{"@timestamp": report["begin_date"]},
)
for policy in report["policies"]:
@@ -1374,11 +1528,16 @@ def save_smtp_tls_report_to_opensearch(
)
smtp_tls_doc.policies.append(policy_doc)
create_indexes([index], index_settings)
if use_data_streams:
create_data_streams([index], index_settings)
else:
create_indexes([index], index_settings)
smtp_tls_doc.meta.index = index
try:
smtp_tls_doc.save()
# Data streams accept only op_type=create. No _id is set, so for a
# dated index it means the same as the default.
smtp_tls_doc.save(op_type="create")
except Exception as e:
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
+11 -2
View File
@@ -20,7 +20,7 @@ else:
psycopg_json = None
from parsedmarc.log import logger
from parsedmarc.utils import human_timestamp_to_datetime
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
# psycopg is an optional dependency (the PostgreSQL backend is opt-in). The
# pure helper functions below work without it; only PostgreSQLClient needs a
@@ -183,7 +183,16 @@ class PostgreSQLClient:
)
conn.autocommit = False
except psycopg.Error as exc:
raise PostgreSQLError(str(exc)) from exc
# libpq reports a malformed connection string by echoing the
# whole string, or the fragment it could not parse, and either
# can contain the password. psycopg raises ProgrammingError for
# those parse failures, so withhold their detail entirely.
if self._connection_string and isinstance(exc, psycopg.ProgrammingError):
raise PostgreSQLError(
"Invalid PostgreSQL connection_string (details withheld "
"because they may include the password)"
) from None
raise PostgreSQLError(redact_url_credentials(str(exc))) from exc
self._conn = conn
return conn
+13
View File
@@ -13,6 +13,13 @@ from parsedmarc.constants import USER_AGENT
from parsedmarc.log import logger
from parsedmarc.utils import human_timestamp_to_unix_timestamp
# httpx.InvalidURL quotes part of the URL, which can be part of a password
# (one containing "/", "?" or "#" ends up as the "port"), so its text is
# replaced by this fixed message.
_INVALID_URL_MESSAGE = (
"Invalid Splunk HEC URL (details withheld because they may include credentials)"
)
class SplunkError(RuntimeError):
"""Raised when a Splunk API error occurs"""
@@ -136,6 +143,8 @@ class HECClient(object):
self.url, content="".join(json_lines), timeout=self.timeout
)
response = response.json()
except httpx.InvalidURL:
raise SplunkError(_INVALID_URL_MESSAGE) from None
except Exception as e:
raise SplunkError(e.__str__())
if response["code"] != 0:
@@ -179,6 +188,8 @@ class HECClient(object):
self.url, content="".join(json_lines), timeout=self.timeout
)
response = response.json()
except httpx.InvalidURL:
raise SplunkError(_INVALID_URL_MESSAGE) from None
except Exception as e:
raise SplunkError(e.__str__())
if response["code"] != 0:
@@ -218,6 +229,8 @@ class HECClient(object):
self.url, content="".join(json_lines), timeout=self.timeout
)
response = response.json()
except httpx.InvalidURL:
raise SplunkError(_INVALID_URL_MESSAGE) from None
except Exception as e:
raise SplunkError(e.__str__())
if response["code"] != 0:
+351 -52
View File
@@ -16,8 +16,8 @@ import shutil
import subprocess
import tempfile
from datetime import datetime, timedelta, timezone
from typing import TypedDict, cast
from urllib.parse import urlsplit
from typing import TYPE_CHECKING, TypedDict, cast
from urllib.parse import SplitResult, urlsplit
import mailparser
from expiringdict import ExpiringDict
@@ -46,6 +46,11 @@ from parsedmarc.constants import (
)
from parsedmarc.log import logger
if TYPE_CHECKING:
# dnspython 2.9+ installs httpx2 through its [doh] extra; 2.7 and 2.8
# install httpx, so httpx2 may be absent
import httpx2 # pyright: ignore[reportMissingImports]
# Errors considered transient and retryable by query_dns. LifetimeTimeout is
# dnspython's deadline expiry; NoNameservers typically wraps a SERVFAIL from
# upstream; OSError covers socket-level failures during TCP fallback.
@@ -55,9 +60,9 @@ _RETRYABLE_DNS_ERRORS = (
OSError,
)
# The process-wide httpx client used for DNS over HTTPS queries, and the PID
# The process-wide HTTP client used for DNS over HTTPS queries, and the PID
# it was created under. See _get_doh_session().
_DOH_SESSION: httpx.Client | None = None
_DOH_SESSION: httpx.Client | httpx2.Client | None = None
_DOH_SESSION_PID: int | None = None
null_file = subprocess.DEVNULL
@@ -108,15 +113,17 @@ def load_psl_overrides(
if not (offline or always_use_local_file):
try:
logger.debug(f"Trying to fetch PSL overrides from {url}...")
logger.debug(
f"Trying to fetch PSL overrides from {redact_url_credentials(url)}..."
)
headers = {"User-Agent": USER_AGENT}
response = httpx.get(
url, headers=headers, timeout=60, follow_redirects=True
)
response.raise_for_status()
_load_text(response.text)
except httpx.HTTPError as e:
logger.warning(f"Failed to fetch PSL overrides: {e}")
except (httpx.HTTPError, httpx.InvalidURL) as e:
logger.warning(f"Failed to fetch PSL overrides: {_describe_http_error(e)}")
if len(psl_overrides) == 0:
path = local_file_path or str(
@@ -204,9 +211,32 @@ def get_base_domain(domain: str) -> str | None:
return publicsuffix
def _get_doh_session() -> httpx.Client:
def _doh_client_class() -> type[httpx.Client] | type[httpx2.Client]:
"""
Returns the shared ``httpx.Client`` used for DNS over HTTPS queries.
Returns the HTTP client class ``dns.query.https()`` accepts as a
``session``.
dnspython 2.9 moved DNS over HTTPS from httpx to httpx2, and
``dns.query.https()`` rejects a session of the other library's class
(``ValueError: session parameter must be an httpx2.Client``), so an
``httpx.Client`` broke every DoH query under dnspython 2.9
(https://github.com/domainaware/parsedmarc/issues/935). The class is
taken from the module dnspython itself imported, so it matches its
``isinstance`` check even when httpx2 is installed beside dnspython 2.8.
Returns:
type: ``httpx2.Client`` under dnspython 2.9+, ``httpx.Client`` before
"""
return (
getattr(dns.query, "httpx2", None) or getattr(dns.query, "httpx", httpx)
).Client
def _get_doh_session() -> httpx.Client | httpx2.Client:
"""
Returns the shared HTTP client used for DNS over HTTPS queries: an
``httpx2.Client`` under dnspython 2.9+, an ``httpx.Client`` before (see
``_doh_client_class()``).
The client is created on first use and reused afterwards, so DoH queries
share TLS connections instead of renegotiating one per lookup. It is
@@ -217,41 +247,43 @@ def _get_doh_session() -> httpx.Client:
created under, because a ``fork()``-based worker pool (``n_procs``) would
otherwise inherit — and concurrently use — the parent's sockets.
``httpx.Client`` defaults are what make this work behind a corporate
proxy: ``trust_env=True`` honors ``HTTP_PROXY``/``HTTPS_PROXY``/
``NO_PROXY`` and ``SSL_CERT_FILE``/``SSL_CERT_DIR``, and ``verify=True``
keeps certificate verification on. Neither is overridden here.
The client defaults (identical in httpx and httpx2) are what make this
work behind a corporate proxy: ``trust_env=True`` honors ``HTTP_PROXY``/
``HTTPS_PROXY``/``NO_PROXY`` and ``SSL_CERT_FILE``/``SSL_CERT_DIR``, and
``verify=True`` keeps certificate verification on. Neither is overridden
here.
Returns:
httpx.Client: The shared DoH client for this process
httpx.Client | httpx2.Client: The shared DoH client for this process
"""
global _DOH_SESSION, _DOH_SESSION_PID
pid = os.getpid()
if _DOH_SESSION is None or _DOH_SESSION_PID != pid:
_DOH_SESSION = httpx.Client(http1=True, http2=True)
_DOH_SESSION = _doh_client_class()(http1=True, http2=True)
_DOH_SESSION_PID = pid
return _DOH_SESSION
class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
"""
A DNS over HTTPS nameserver that queries through a shared ``httpx``
client.
A DNS over HTTPS nameserver that queries through a shared HTTP client
(httpx2 on dnspython 2.9+, httpx before).
dnspython's stock ``DoHNameserver`` calls ``dns.query.https()`` without a
``session``, which makes that function build an ``httpx.Client`` with its
own custom transport — and httpx only reads proxy environment variables
when no transport is supplied (``allow_env_proxies = trust_env and
transport is None``). Stock DoH therefore ignores ``HTTPS_PROXY``
entirely — and a proxy is the only way out of the networks this exists
for (https://github.com/domainaware/parsedmarc/issues/880).
``session``, which makes that function build a client with its own
custom transport — and httpx and httpx2 only read proxy environment
variables when no transport is supplied (``allow_env_proxies =
trust_env and transport is None``). Stock DoH therefore ignores
``HTTPS_PROXY`` entirely — and a proxy is the only way out of the
networks this exists for
(https://github.com/domainaware/parsedmarc/issues/880).
Passing our own session instead gets environment proxies, environment CA
configuration (``SSL_CERT_FILE``), and connection reuse across queries.
``bootstrap_address`` is deliberately not passed: with a session, the DoH
server's hostname is resolved by httpx — locally through the OS resolver,
or by the proxy itself via ``CONNECT`` when one is configured — so no
UDP/53 access is required.
server's hostname is resolved by the HTTP client — locally through the
OS resolver, or by the proxy itself via ``CONNECT`` when one is
configured — so no UDP/53 access is required.
"""
def query(
@@ -279,6 +311,44 @@ class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
)
def _nameserver_port(entry: str, parts: SplitResult, kind: str) -> int | None:
"""
Returns the port of a parsed DNS over HTTPS or TLS nameserver URL.
``urlsplit()`` only validates the port when it is read, and its
``ValueError`` quotes the text it rejected. A password containing an
unencoded ``/``, ``?`` or ``#`` ends the authority early (RFC 3986
section 3.2), which makes the start of the password that text
(``https://doh:p4ss/w0rd@host`` gives ``Port could not be cast to
integer value as 'p4ss'``). dnspython 2.9.1 reads the same port in
``DoHNameserver.answer_port()`` and lets that error escape
``Resolver.resolve()``. So the port is read here, and a bad one is
reported with the entry's credentials masked and the original error
suppressed.
Args:
entry (str): The configured nameserver
parts (SplitResult): ``urlsplit(entry)``
kind (str): The transport name for the message, e.g.
``"DNS over TLS"``
Returns:
int | None: The port, or ``None`` when the URL has none
Raises:
ValueError: The port is not an integer from 0 to 65535
"""
try:
return parts.port
except ValueError:
pass
raise ValueError(
f"Invalid {kind} nameserver {_redact_one_url_credentials(entry)}: the "
"port is not an integer from 0 to 65535 (percent-encode any /, ? or # "
"in a password)"
)
def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
"""
Parses a ``tls://ip[:port][#hostname]`` nameserver entry.
@@ -297,26 +367,25 @@ def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
is not a literal IP address, or extra URL components
"""
parts = urlsplit(entry)
try:
port = parts.port
except ValueError as e:
# urlsplit only validates the port when it is accessed
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: {e}") from e
port = _nameserver_port(entry, parts, "DNS over TLS")
# The entry may carry userinfo, which is rejected below, so the
# messages quote it with its credentials masked
shown = _redact_one_url_credentials(entry)
if parts.username or parts.path or parts.query:
# Catch tls://9.9.9.9/dns.quad9.net — a plausible slash-for-#
# typo that would otherwise "work" with no certificate identity
# and fail only at query time with an opaque TLS error
raise ValueError(
f"Invalid DNS over TLS nameserver {entry}: only "
f"Invalid DNS over TLS nameserver {shown}: only "
"tls://ip[:port][#hostname] is supported — the TLS certificate "
"identity is given after #, not /"
)
address = parts.hostname
if not address:
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: missing IP address")
raise ValueError(f"Invalid DNS over TLS nameserver {shown}: missing IP address")
if not dns.inet.is_address(address):
raise ValueError(
f"Invalid DNS over TLS nameserver {entry}: {address} is not an IP "
f"Invalid DNS over TLS nameserver {shown}: {address} is not an IP "
"address. Use tls://ip[:port][#hostname], where the optional "
"#hostname is the TLS certificate identity of the server"
)
@@ -334,11 +403,11 @@ def _nameservers_to_resolver_input(
``dns.resolver.Resolver.nameservers`` accepts.
``https://`` entries become DNS over HTTPS nameservers that share this
process's ``httpx`` client (so proxy and CA environment variables apply),
and ``tls://ip[:port][#hostname]`` entries become DNS over TLS
nameservers. Everything else — plain IPv4/IPv6 addresses — is passed
through untouched, leaving dnspython to enrich and validate it exactly as
before.
process's HTTP client (httpx2 on dnspython 2.9+, httpx before), so proxy
and CA environment variables apply, and ``tls://ip[:port][#hostname]``
entries become DNS over TLS nameservers. Everything else — plain
IPv4/IPv6 addresses — is passed through untouched, leaving dnspython to
enrich and validate it exactly as before.
Args:
nameservers (list[str]): The configured nameservers
@@ -348,20 +417,25 @@ def _nameservers_to_resolver_input(
in the configured order
Raises:
ValueError: A ``tls://`` entry is malformed
ValueError: A ``tls://`` entry is malformed, or an ``https://``
entry has an unusable port
"""
resolver_input: list[str | dns.nameserver.Nameserver] = []
for entry in nameservers:
try:
# urlsplit lowercases the scheme, so HTTPS:// and TLS:// work too
scheme = urlsplit(entry).scheme
parts = urlsplit(entry)
except ValueError:
# e.g. an unbalanced IPv6 bracket; let dnspython reject it with
# its own message about what a nameserver may be
scheme = ""
if scheme == "https":
resolver_input.append(entry)
continue
if parts.scheme == "https":
# Validated now because dnspython reads the port only after a
# query fails, and its error then quotes the rejected text
_nameserver_port(entry, parts, "DNS over HTTPS")
resolver_input.append(_SessionDoHNameserver(entry))
elif scheme == "tls":
elif parts.scheme == "tls":
resolver_input.append(_parse_dot_nameserver(entry))
else:
resolver_input.append(entry)
@@ -409,6 +483,10 @@ def query_dns(
Returns:
list: A list of answers
Raises:
ValueError: A nameserver is malformed. The message masks the
nameserver's URL credentials.
"""
domain = str(domain).lower()
record_type = record_type.upper()
@@ -427,7 +505,17 @@ def query_dns(
"2606:4700:4700::1111",
"2606:4700:4700::1001",
]
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
rejection = None
try:
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
except ValueError as e:
# dnspython's own rejection quotes the entry as configured
# ("nameserver https://user:pw@[x is not ..."), credentials and all
rejection = _mask_nameserver_secrets(str(e), nameservers)
if rejection is not None:
# Raised outside the except block so that the unmasked error is not
# kept as __context__
raise ValueError(rejection)
# Cap per-query UDP timeout at 1s so dnspython retries within the
# lifetime window on transient packet loss — otherwise with a single
# nameserver and timeout == lifetime, one dropped UDP datagram consumes
@@ -502,7 +590,9 @@ def get_reverse_dns(
)[0]
except dns.exception.DNSException as e:
logger.debug(f"get_reverse_dns({ip_address}) exception: {e}")
# LifetimeTimeout and NoNameservers quote each nameserver URL
message = _mask_nameserver_secrets(str(e), nameservers)
logger.debug(f"get_reverse_dns({ip_address}) exception: {message}")
return hostname
@@ -654,7 +744,9 @@ def load_ip_db(
if not (offline or always_use_local_file):
try:
logger.debug(f"Trying to fetch IP database from {url}...")
logger.debug(
f"Trying to fetch IP database from {redact_url_credentials(url)}..."
)
headers = {"User-Agent": USER_AGENT}
response = httpx.get(
url, headers=headers, timeout=60, follow_redirects=True
@@ -668,8 +760,8 @@ def load_ip_db(
_IP_DB_PATH = cached_path
logger.info("IP database updated successfully")
return
except httpx.HTTPError as e:
logger.warning(f"Failed to fetch IP database: {e}")
except (httpx.HTTPError, httpx.InvalidURL) as e:
logger.warning(f"Failed to fetch IP database: {_describe_http_error(e)}")
except Exception as e:
logger.warning(f"Failed to save IP database: {e}")
@@ -1050,7 +1142,9 @@ def load_reverse_dns_map(
if not (offline or always_use_local_file):
try:
logger.debug(f"Trying to fetch reverse DNS map from {url}...")
logger.debug(
f"Trying to fetch reverse DNS map from {redact_url_credentials(url)}..."
)
headers = {"User-Agent": USER_AGENT}
response = httpx.get(
url, headers=headers, timeout=60, follow_redirects=True
@@ -1059,8 +1153,10 @@ def load_reverse_dns_map(
csv_file.write(response.text)
csv_file.seek(0)
load_csv(csv_file)
except httpx.HTTPError as e:
logger.warning(f"Failed to fetch reverse DNS map: {e}")
except (httpx.HTTPError, httpx.InvalidURL) as e:
logger.warning(
f"Failed to fetch reverse DNS map: {_describe_http_error(e)}"
)
except Exception:
logger.warning("Not a valid CSV file")
csv_file.seek(0)
@@ -1327,6 +1423,209 @@ def sanitize_for_log(value: str) -> str:
)
def redact_url_credentials(value: str | list[str]) -> str | list[str]:
"""Masks the credentials in a URL or host string so it is safe to log.
A configured URL such as ``https://user:secret@host:9200`` carries a
password; writing it to the log (CWE-532) leaks it to the console and
to ``log_file``. When the userinfo contains a ``:``, only the part
after the first ``:`` is replaced (``https://user:***@host:9200``);
when it has no ``:`` the whole userinfo is replaced
(``https://***@host``), since a bare token is itself the secret.
The authority is located with string methods only, so scheme-less
values such as ``user:secret@localhost:9200`` (which ``urlsplit()``
would misparse as scheme ``user``) are handled too.
A password containing an unencoded ``/``, ``?``, or ``#`` is not
allowed by RFC 3986 (the authority ends at that character, so clients
misparse such a URL too), and it leaves the authority with no ``@``.
When the authority has no ``@`` but one appears later in the value,
everything from the start of the authority through the *last* ``@``
is replaced (``https://admin:p4ss/w0rd@host:3100`` becomes
``https://***@host:3100``). The last ``@`` is used because the
password may itself contain ``@``. This over-masks a URL that has no
userinfo but has an ``@`` in its path or query:
``https://host/path@file`` becomes ``https://***@file``, and
``https://host/?redirect=https://u:p@evil`` becomes
``https://***@evil``. That is acceptable because the result is only
for the log. When the authority does contain an ``@``, only its
userinfo is masked, so credentials in a later query-string value
(``https://a:b@host/?redirect=https://u:p@evil``) are not detected,
and neither is the rest of a password containing both ``@`` and one
of ``/?#`` (``https://user:p@ss/word@host`` keeps ``ss/word``).
The value must be a single URL (or host) per string: split a
comma-separated list first, or pass a ``list``. The authority is the
first one found, so later entries in a comma-joined string are not
masked.
Never raises; input that does not contain userinfo, and input that is
not a ``str`` or ``list``, is returned unchanged.
Args:
value (str | list[str]): A URL or host string, or a list of them
Returns:
str | list[str]: ``value`` with credentials masked. A ``list`` is
returned as a new list; the input list is not modified.
"""
if isinstance(value, list):
return [
_redact_one_url_credentials(item) if isinstance(item, str) else item
for item in value
]
if not isinstance(value, str):
return value
return _redact_one_url_credentials(value)
def _find_url_userinfo(value: str) -> tuple[int, int, bool] | None:
"""Locates the userinfo of one URL or host string.
This is the lookup :func:`redact_url_credentials` masks and
:func:`_url_secrets` extracts the secret from, shared so that the two
cannot disagree about where the credentials are. It uses string
methods only, so it is linear in the length of ``value`` and never
raises.
Args:
value (str): A URL or host string
Returns:
tuple[int, int, bool] | None: ``None`` when no ``@`` follows the
start of the authority. Otherwise ``(start, end, in_authority)``,
where ``value[start:end]`` is the userinfo and ``value[end]`` is the
``@`` after it. ``in_authority`` is ``True`` when that ``@`` is the
last one inside the authority, and ``False`` when the authority has
no ``@`` (a ``/``, ``?`` or ``#`` in the password ends it early, as
in ``https://admin:p4ss/w0rd@host``) and the ``@`` is the last one
in the whole value; the password may itself contain ``@``.
"""
# A "://" counts as the scheme separator only when nothing before it
# could end an authority: in "user:secret@host/x?r=http://y" the "://"
# is inside the query, and the authority starts at offset 0.
scheme_end = value.find("://")
if scheme_end != -1 and not any(c in value[:scheme_end] for c in "/?#@"):
start = scheme_end + 3
elif value.startswith("//"):
start = 2
else:
start = 0
end = len(value)
for delimiter in "/?#":
position = value.find(delimiter, start)
if position != -1:
end = min(end, position)
at = value.rfind("@", start, end)
if at != -1:
return start, at, True
at = value.rfind("@", start)
if at != -1:
return start, at, False
return None
def _redact_one_url_credentials(value: str) -> str:
"""Masks the userinfo of one URL or host string; see
:func:`redact_url_credentials`."""
found = _find_url_userinfo(value)
if found is None:
return value
start, at, in_authority = found
if not in_authority:
# Which "/", "?" or "#" ended the authority is unknown, so mask
# everything through the "@" rather than guess where the user
# name stops.
return f"{value[:start]}***{value[at:]}"
user, colon, _ = value[start:at].partition(":")
masked = f"{user}:***" if colon else "***"
return f"{value[:start]}{masked}{value[at:]}"
def _url_secrets(value: str) -> list[str]:
"""Returns the secret text in one URL or host string's userinfo.
The secret is the userinfo after its first ``:``, or the whole userinfo
when it has no ``:`` (a bare token is itself the secret), located by
:func:`_find_url_userinfo`. When the secret contains an unencoded
``/``, ``?`` or ``#``, its text before the first of them is returned
too: ``urlsplit()`` (and so dnspython) and httpx read that part as the
port and quote it in their errors (``Port could not be cast to integer
value as 'p4ss'``). Empty strings are never returned, so the result is
safe to use as ``str.replace()`` targets. Never raises.
Args:
value (str): A URL or host string
Returns:
list[str]: The secret and, if it differs and is not empty, its
prefix before the first ``/``, ``?`` or ``#``, longest first; an
empty list when ``value`` has no userinfo or an empty secret.
"""
found = _find_url_userinfo(value)
if found is None:
return []
start, at, _ = found
user, colon, password = value[start:at].partition(":")
secret = password if colon else user
prefix_end = len(secret)
for delimiter in "/?#":
position = secret.find(delimiter)
if position != -1:
prefix_end = min(prefix_end, position)
return [text for text in dict.fromkeys((secret, secret[:prefix_end])) if text]
def _mask_nameserver_secrets(text: str, nameservers: list[str] | None) -> str:
"""Masks the credentials of the configured nameserver URLs in free text.
A DNS query error (dnspython's ``LifetimeTimeout``, a DNS over HTTPS
status error) quotes each failing nameserver URL, sometimes several
times in different forms, and ``redact_url_credentials()`` masks only
one URL per string. So each configured entry is replaced by its masked
form, and each secret :func:`_url_secrets` finds in it (the password,
and its start when a ``/``, ``?`` or ``#`` in it gets read as the port)
is replaced by ``***`` wherever it still appears. A short secret masks
every occurrence of that text, including ones that are not the
password; the result is only for error messages and logs. Never
raises.
Args:
text (str): The text to mask, typically ``str()`` of an exception
nameservers (list[str] | None): The configured nameservers
Returns:
str: ``text`` with the nameservers' credentials masked
"""
for entry in nameservers or []:
text = text.replace(entry, _redact_one_url_credentials(entry))
for secret in _url_secrets(entry):
text = text.replace(secret, "***")
return text
def _describe_http_error(e: httpx.HTTPError | httpx.InvalidURL) -> str:
"""Describes an httpx error for the log without leaking URL credentials.
``str(HTTPStatusError)`` embeds the server's reason phrase before the
URL, so a phrase containing ``/``, ``?``, ``#`` or ``@`` defeats
string-based redaction of the message. A status error is therefore
rebuilt from its structured parts. Transport errors (httpx 0.28) do not
embed the request URL, so their message is used as is. ``InvalidURL``
(not an ``HTTPError``) is raised while httpx parses the URL, and its
message quotes the component it rejected -- part of a password
containing ``/``, ``?`` or ``#`` ends up as the "port" -- so it is
replaced by a fixed message.
"""
if isinstance(e, httpx.InvalidURL):
return "invalid URL (details withheld because they may include credentials)"
if isinstance(e, httpx.HTTPStatusError):
url = redact_url_credentials(str(e.request.url))
return f"HTTP {e.response.status_code} for url '{url}'"
return f"{type(e).__name__}: {e}"
def get_filename_safe_string(string: str | None) -> str:
"""
Converts a string to a string that is safe to use as a filename
+6
View File
@@ -62,6 +62,12 @@ class WebhookClient(object):
self.session.post(webhook_url, data=payload, timeout=self.timeout)
else:
self.session.post(webhook_url, content=payload, timeout=self.timeout)
except httpx.InvalidURL:
# httpx quotes part of the URL, which can be part of a password
logger.error(
"Webhook Error: invalid webhook URL (details withheld because "
"they may include credentials)"
)
except Exception as error_:
logger.error(f"Webhook Error: {error_.__str__()}")
+6 -3
View File
@@ -43,9 +43,12 @@ requires-python = ">=3.10"
# the Elasticsearch, OpenSearch, Kafka, AWS, Azure, Gmail, and Microsoft
# Graph SDKs (see issue #883).
dependencies = [
# The [doh] extra supplies the httpx/h2 floors DNS over HTTPS needs;
# 2.7.0 is the floor verified against the dns.nameserver and
# dns.query.https(session=...) APIs utils.py builds on.
# The [doh] extra supplies the HTTP client DNS over HTTPS needs (httpx,
# httpcore and h2 on dnspython 2.7 and 2.8; httpx2, httpcore2 and h2 on
# 2.9+; utils.py builds its session from whichever client library
# dnspython imported). 2.7.0 is the floor verified against the
# dns.nameserver and dns.query.https(session=...) APIs utils.py builds
# on.
"dnspython[doh]>=2.7.0",
"expiringdict>=1.1.4",
# The runtime HTTP library (utils.py fetches, webhook and Splunk HEC
+741 -7
View File
@@ -6,6 +6,7 @@ import importlib.abc
import importlib.machinery
import gzip
import io
import importlib.util
import json
import logging
import mailbox
@@ -15,6 +16,7 @@ import signal
import stat
import sys
import tempfile
import traceback
import unittest
import zipfile
from collections.abc import Iterator, Sequence
@@ -27,6 +29,11 @@ from types import ModuleType, SimpleNamespace
from typing import cast
from unittest.mock import MagicMock, patch
import dns.message
import dns.rcode
import dns.rdata
import dns.rdataclass
import dns.rdatatype
import httpx
import opensearchpy
from azure.core.exceptions import ClientAuthenticationError
@@ -2923,6 +2930,11 @@ client_assertion = s3cret-signed-jwt-assertion
self._run_main(cfg_path, "--debug")
parsedmarc_logger = logging.getLogger("parsedmarc.log")
# dnspython 2.9+ logs DNS over HTTPS traffic under httpx2 and
# httpcore2 (httpx and httpcore on 2.7 and 2.8), so all four must
# follow --debug.
for wanted_name in ("httpx", "httpcore", "httpx2", "httpcore2"):
self.assertIn(wanted_name, parsedmarc.cli._DEPENDENCY_LOGGERS)
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
dep = logging.getLogger(name)
self.assertEqual(dep.level, logging.DEBUG, name)
@@ -5442,6 +5454,323 @@ def _es_request_without_network(client, method, path, **kwargs):
return {"count": 0}
def _os_request_without_network(transport, method, url, *args, **kwargs):
"""Stand-in for ``opensearchpy.Transport.perform_request``: answers
"no such index" / zero documents like ``_es_request_without_network``."""
if method == "HEAD":
return {}
return {"count": 0}
class TestInitOutputClientsDoesNotLogCredentials(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: with --debug, the "Initializing ... client"
lines must mask credentials embedded in configured URLs."""
SECRET = "s3cretPW"
def _init(self, opts):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
clients = parsedmarc.cli._init_output_clients(opts)
self.addCleanup(parsedmarc.cli._close_output_clients, clients)
return "\n".join(logs.output)
def test_hec_url_password_is_masked(self):
opts = _output_client_opts(
hec=f"https://hecuser:{self.SECRET}@hec.example:8088",
hec_token="tok",
hec_index="email",
)
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
self.assertIn("url=https://hecuser:***@hec.example:8088", output)
def test_hec_malformed_url_password_is_masked(self):
"""A "/", "?" or "#" in the password ends the authority before the
"@", which left the --debug line unmasked; it is now masked through
the last "@"."""
for password in ("p4ssw0rd/extra", "p4ss?w0rd", "p4ss#w0rd"):
with self.subTest(password=password):
opts = _output_client_opts(
hec=f"https://hecuser:{password}@hec.example:8088",
hec_token="tok",
hec_index="email",
)
output = self._init(opts)
self.assertNotIn("p4ss", output)
self.assertIn("url=https://***@hec.example:8088", output)
def test_loki_url_password_is_masked(self):
opts = _output_client_opts(
loki_url=f"https://lokiuser:{self.SECRET}@loki.example:3100",
loki_timestamp="ingest",
loki_timeout=1.0,
loki_batch_size=100,
)
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
self.assertIn("url=https://lokiuser:***@loki.example:3100", output)
def test_loki_malformed_url_error_has_no_password(self):
"""A password with an unencoded "/" ends the URL's authority early
(RFC 3986 section 3.2), so httpx raises ``InvalidURL("Invalid port:
'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py), whose text and chained
ValueError both repeat it. Neither the startup error nor the
``--debug`` "Initializing Loki client" line may show any part of
the password."""
malformed = (
("https://admin:p4ssw0rd/extra@loki.example:3100", "p4ssw0rd"),
("https://admin:p4ss?w0rd@loki.example:3100", "p4ss"),
("https://admin:p4ss#w0rd@loki.example:3100", "p4ss"),
)
for url, secret in malformed:
with self.subTest(url=url):
opts = _output_client_opts(
loki_url=url,
loki_timestamp="ingest",
loki_timeout=1.0,
loki_batch_size=100,
)
with (
self.assertRaises(RuntimeError) as ctx,
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
):
parsedmarc.cli._init_output_clients(opts)
self.assertEqual(
str(ctx.exception),
"Loki: Invalid Loki URL (details withheld because they may "
"include credentials)",
)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
output = "\n".join(logs.output)
self.assertNotIn(secret, output)
self.assertIn("url=https://***@loki.example:3100", output)
def test_elasticsearch_hosts_password_is_masked(self):
connections = parsedmarc.elastic.connections
_isolate_default_alias(self, connections)
opts = _output_client_opts(
save_aggregate=True,
elasticsearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
elasticsearch_timeout=1.0,
)
with (
patch.object(Elasticsearch, "perform_request", _es_request_without_network),
patch(
"parsedmarc.elastic.connections.create_connection",
wraps=connections.create_connection,
) as mock_conn,
):
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
self.assertIn("Initializing Elasticsearch client", output)
# Redaction is log-only: the client still gets the real credentials
self.assertEqual(
mock_conn.call_args.kwargs["hosts"],
[f"https://u:{self.SECRET}@127.0.0.1:9299"],
)
self.assertIn("https://u:***@127.0.0.1:9299", output)
def test_opensearch_hosts_password_is_masked_in_cli_and_set_hosts(self):
connections = opensearch_module.connections
_isolate_default_alias(self, connections)
opts = _output_client_opts(
save_aggregate=True,
opensearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
opensearch_timeout=1.0,
)
with patch.object(
opensearchpy.Transport, "perform_request", _os_request_without_network
):
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
lines = output.splitlines()
for marker in ("Initializing OpenSearch client", "Connecting to OpenSearch"):
with self.subTest(marker=marker):
matching = [line for line in lines if marker in line]
self.assertEqual(len(matching), 1)
self.assertIn("https://u:***@127.0.0.1:9299", matching[0])
class TestHttpxLogRedaction(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: httpx logs the full request URL at INFO."""
def setUp(self):
saved = {}
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
dep = logging.getLogger(name)
saved[name] = (
dep.level,
list(dep.handlers),
dep.propagate,
list(dep.filters),
)
def restore():
for name, (level, handlers, propagate, filters) in saved.items():
dep = logging.getLogger(name)
dep.setLevel(level)
dep.handlers = handlers
dep.propagate = propagate
dep.filters = filters
self.addCleanup(restore)
parsedmarc.cli._configure_dependency_logging(logging.INFO)
def _client(self, seen):
def handler(request):
seen.append(request)
return httpx.Response(200)
return httpx.Client(transport=httpx.MockTransport(handler))
def test_userinfo_password_masked_but_auth_still_sent(self):
seen = []
with self._client(seen) as client:
with self.assertLogs("httpx", level="INFO") as logs:
client.post("https://user:secret@hook.example/x", content=b"{}")
output = "\n".join(logs.output)
self.assertNotIn("secret", output)
self.assertIn("https://user:***@hook.example/x", output)
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
def test_ipinfo_token_query_parameter_masked(self):
seen = []
with self._client(seen) as client:
with self.assertLogs("httpx", level="INFO") as logs:
client.get(
"https://api.ipinfo.io/lite/8.8.8.8", params={"token": "TOKSECRET"}
)
output = "\n".join(logs.output)
self.assertNotIn("TOKSECRET", output)
self.assertIn("api.ipinfo.io/lite/8.8.8.8", output)
self.assertIn("token=***", output)
self.assertEqual(seen[0].url.params["token"], "TOKSECRET")
def test_token_mask_leaves_literal_encoded_stars_in_path_alone(self):
"""A path containing a literal ``%2A%2A%2A`` is not rewritten; only
the token value is masked, with and without other query params and
a fragment."""
for url, expected in (
(
"https://h.example/a%2A%2A%2Ab?token=TOKSECRET",
"https://h.example/a%2A%2A%2Ab?token=***",
),
(
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET",
"https://h.example/a%2A%2A%2Ab?x=1&token=***",
),
(
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET#f",
"https://h.example/a%2A%2A%2Ab?x=1&token=***#f",
),
):
with self.subTest(url=url):
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(
httpx.URL(url)
)
self.assertNotIn("TOKSECRET", result)
self.assertEqual(result, expected)
def test_reconfiguring_does_not_stack_filters(self):
parsedmarc.cli._configure_dependency_logging(logging.INFO)
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
installed = [
f
for f in logging.getLogger("httpx").filters
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
]
self.assertEqual(len(installed), 1)
def test_non_tuple_args_and_other_arg_types_pass_through(self):
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "x %d", (5,), None)
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
self.assertEqual(record.args, (5,))
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "plain", None, None)
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
self.assertIsNone(record.args)
@unittest.skipUnless(
importlib.util.find_spec("httpx2") is not None,
"httpx2 (dnspython 2.9+ DNS over HTTPS) is not installed",
)
class TestHttpx2LogRedaction(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: dnspython 2.9+ does DNS over HTTPS through
httpx2, which logs the request URL under the ``httpx2`` logger with an
``httpx2.URL``, a different class from ``httpx.URL``."""
def setUp(self):
import httpx2 # pyright: ignore[reportMissingImports]
self.httpx2 = httpx2
saved = {}
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
dep = logging.getLogger(name)
saved[name] = (
dep.level,
list(dep.handlers),
dep.propagate,
list(dep.filters),
)
def restore():
for name, (level, handlers, propagate, filters) in saved.items():
dep = logging.getLogger(name)
dep.setLevel(level)
dep.handlers = handlers
dep.propagate = propagate
dep.filters = filters
self.addCleanup(restore)
parsedmarc.cli._configure_dependency_logging(logging.INFO)
def test_httpx2_client_request_line_masks_userinfo(self):
"""httpx2 itself emits the line; the credential still reaches the
transport."""
seen = []
def handler(request):
seen.append(request)
return self.httpx2.Response(200)
with self.httpx2.Client(transport=self.httpx2.MockTransport(handler)) as client:
with self.assertLogs("httpx2", level="INFO") as logs:
client.post("https://doh:S3CRET@doh.example/dns-query", content=b"{}")
output = "\n".join(logs.output)
self.assertNotIn("S3CRET", output)
self.assertIn("https://doh:***@doh.example/dns-query", output)
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
def test_httpx2_url_token_query_parameter_masked(self):
url = self.httpx2.URL("https://h.example/a?x=1&token=TOKSECRET#f")
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(url)
self.assertEqual(result, "https://h.example/a?x=1&token=***#f")
def test_httpx2_logger_record_args_masked_and_ints_untouched(self):
record = logging.LogRecord(
"httpx2",
logging.INFO,
"f",
1,
"%s %d",
(self.httpx2.URL("https://u:S3CRET@h.example/x"), 200),
None,
)
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
self.assertEqual(record.args, ("https://u:***@h.example/x", 200))
def test_reconfiguring_attaches_one_filter_to_httpx2(self):
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
parsedmarc.cli._configure_dependency_logging(logging.INFO)
installed = [
f
for f in logging.getLogger("httpx2").filters
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
]
self.assertEqual(len(installed), 1)
class TestInitOutputClientsRollback(unittest.TestCase):
"""_init_output_clients() is all-or-nothing against the search backends'
process-wide ``default`` connection alias.
@@ -7519,7 +7848,8 @@ class TestParseConfigGeneral(unittest.TestCase):
"""A nameservers list may mix DNS over HTTPS URLs and DNS over TLS
entries with plain IP addresses (issue #880); _parse_config splits
and strips them like any other list value, and hands them to the DNS
pre-flight check (mocked here so no network is needed)."""
pre-flight check (dns.query.https is mocked so no network is
needed)."""
from parsedmarc.cli import _parse_config
cp = _config_with(
@@ -7533,18 +7863,223 @@ class TestParseConfigGeneral(unittest.TestCase):
},
)
opts = _opts()
with patch(
"parsedmarc.cli.get_reverse_dns", return_value="one.one.one.one"
) as mock_reverse_dns:
queried: list[tuple[str, str]] = []
def responder(request, where, *args, **kwargs):
queried.append((where, request.question[0].to_text()))
response = dns.message.make_response(request)
rrset = response.find_rrset(
response.answer,
request.question[0].name,
dns.rdataclass.IN,
dns.rdatatype.PTR,
create=True,
)
rrset.add(dns.rdata.from_text("IN", "PTR", "one.one.one.one."), 300)
return response
with patch("dns.query.https", side_effect=responder):
_parse_config(cp, opts)
self.assertEqual(
opts.nameservers,
["https://cloudflare-dns.com/dns-query", "tls://9.9.9.9#dns.quad9.net"],
)
# The first (DoH) nameserver answered the pre-flight PTR query for
# dns_test_address, so the DoT one was never needed
self.assertEqual(
mock_reverse_dns.call_args.kwargs["nameservers"], opts.nameservers
queried,
[
(
"https://cloudflare-dns.com/dns-query",
"1.1.1.1.in-addr.arpa. IN PTR",
)
],
)
def test_general_nameservers_pre_flight_reports_the_resolver_error(self):
"""A nameserver that fails every query is reported with the
resolver's own error, not as a missing PTR record. Issue #935: under
dnspython 2.9 every DoH query raised "session parameter must be an
httpx2.Client", and the pre-flight check said "no PTR record for
1.1.1.1", because get_reverse_dns() turns every DNS error into
None."""
from parsedmarc.cli import ConfigurationError, _parse_config
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "0.2",
"nameservers": "https://dns.example/dns-query",
},
)
error = ValueError("session parameter must be an httpx2.Client")
with patch("dns.query.https", side_effect=error):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
message = str(ctx.exception)
self.assertIn("DNS pre-flight check failed", message)
self.assertIn("session parameter must be an httpx2.Client", message)
self.assertNotIn("no PTR record", message)
def test_general_nameservers_pre_flight_reports_a_missing_ptr_record(self):
"""When the nameserver answers but has no PTR record for
dns_test_address — NXDOMAIN, or NOERROR with an empty answer
section — the pre-flight error says so and names the nameservers."""
from parsedmarc.cli import ConfigurationError, _parse_config
def nxdomain(request, *args, **kwargs):
response = dns.message.make_response(request)
response.set_rcode(dns.rcode.NXDOMAIN)
return response
def no_answer(request, *args, **kwargs):
return dns.message.make_response(request)
for name, responder in (("NXDOMAIN", nxdomain), ("NoAnswer", no_answer)):
with self.subTest(name):
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "2.0",
"nameservers": "https://dns.example/dns-query",
},
)
with patch("dns.query.https", side_effect=responder):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
self.assertIn(
"DNS pre-flight check failed: no PTR record for 1.1.1.1 "
"from ['https://dns.example/dns-query']",
str(ctx.exception),
)
def test_general_nameservers_pre_flight_errors_mask_url_credentials(self):
"""GHSA-hjr3-rx4c-24cv: the pre-flight error is logged at the default
level, so neither the "no PTR record ... from <nameservers>" list nor
a transport error that quotes the DoH request URL may carry the
nameserver URL's password, and the chained exception (printed on a
crash) must not repeat it."""
from parsedmarc.cli import ConfigurationError, _parse_config
url = "https://doh:S3CRETPW@dns.example/dns-query"
def nxdomain(request, *args, **kwargs):
response = dns.message.make_response(request)
response.set_rcode(dns.rcode.NXDOMAIN)
return response
def transport_error(request, *args, **kwargs):
raise RuntimeError(f"Server error '502' for url '{url}'")
for name, responder, expected in (
("NXDOMAIN", nxdomain, "https://doh:***@dns.example/dns-query"),
("transport", transport_error, "https://doh:***@dns.example/dns-query"),
):
with self.subTest(name):
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "2.0",
"nameservers": url,
},
)
with patch("dns.query.https", side_effect=responder):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn("S3CRETPW", str(ctx.exception))
self.assertNotIn("S3CRETPW", trace)
self.assertIn(expected, str(ctx.exception))
def test_general_nameservers_pre_flight_masks_malformed_url_passwords(self):
"""GHSA-hjr3-rx4c-24cv: a password containing an unencoded ``/``,
``?`` or ``#`` ends the URL authority early (RFC 3986 section 3.2),
so ``urlsplit()`` reports no password and reads its start as the
port. dnspython 2.9.1 then raised ``ValueError("Port could not be
cast to integer value as 'p4ss'")`` from
``DoHNameserver.answer_port()`` (``dns/nameserver.py``), and
``_parse_dot_nameserver()`` quoted the same urlsplit message for a
``tls://`` entry, so the pre-flight error printed the start of the
password. Neither the password nor that prefix may reach the
message or the printed traceback. No query is sent: every entry is
rejected before dnspython contacts a server, which the patched
transports would turn into a different error."""
from parsedmarc.cli import ConfigurationError, _parse_config
for url, expected in (
(
"https://doh:p4ss/w0rd@dns.example/dns-query",
"https://***@dns.example/dns-query",
),
(
"https://doh:p4ss?w0rd@dns.example/dns-query",
"https://***@dns.example/dns-query",
),
(
"https://doh:p4ss#w0rd@dns.example/dns-query",
"https://***@dns.example/dns-query",
),
("tls://dot:p4ss/w0rd@1.1.1.1", "tls://***@1.1.1.1"),
):
with self.subTest(url=url):
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "0.2",
"nameservers": url,
},
)
unreachable = OSError("no query may be sent")
with (
patch("dns.query.https", side_effect=unreachable),
patch("dns.query.tls", side_effect=unreachable),
):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
message = str(ctx.exception)
trace = "".join(traceback.format_exception(ctx.exception))
for secret in ("p4ss", "w0rd"):
self.assertNotIn(secret, message)
self.assertNotIn(secret, trace)
self.assertIn(expected, message)
self.assertIn("DNS pre-flight check failed: ValueError: ", message)
self.assertNotIn("no query may be sent", message)
def test_general_nameservers_pre_flight_rejects_invalid_test_address(self):
"""An unparseable dns_test_address is reported by setting name and
value (dns.reversename.from_address raises dns.exception.SyntaxError,
"Text input is malformed.", for non-IP text), and no DNS query is
attempted."""
from parsedmarc.cli import ConfigurationError, _parse_config
for address in ("not-an-ip", "1.1.1"):
with self.subTest(address):
cp = _config_with(
"general",
{
"dns_test_address": address,
"dns_timeout": "2.0",
"nameservers": "https://dns.example/dns-query",
},
)
with (
patch("dns.query.https") as https,
patch("dns.query.udp") as udp,
):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
self.assertEqual(
f"DNS pre-flight check failed: dns_test_address {address!r} "
"is not a valid IP address",
str(ctx.exception),
)
https.assert_not_called()
udp.assert_not_called()
def test_general_nameservers_malformed_dot_entry_fails_pre_flight(self):
"""A malformed DNS over TLS nameservers entry raises a
ConfigurationError at startup, before any mailbox work begins. No
@@ -7837,6 +8372,175 @@ class TestParseConfigOpenSearch(unittest.TestCase):
_parse_config(cp, _opts())
class TestParseConfigUseDataStreams(unittest.TestCase):
"""``use_data_streams`` in [elasticsearch] and [opensearch] (issue
#842), and the warning that ``monthly_indexes`` is ignored with it."""
SECTIONS = ("elasticsearch", "opensearch")
def test_option_parsed_for_each_section(self):
from parsedmarc.cli import _parse_config
for section in self.SECTIONS:
for raw, expected in (("true", True), ("false", False)):
with self.subTest(section=section, value=raw):
cp = _config_with(
section, {"hosts": "h:9200", "use_data_streams": raw}
)
opts = _opts()
_parse_config(cp, opts)
self.assertIs(
getattr(opts, f"{section}_use_data_streams"), expected
)
def test_option_absent_leaves_attribute_unset(self):
"""The default comes from the CLI's Namespace defaults, which the
real CLI builds before _parse_config runs."""
from parsedmarc.cli import _parse_config
for section in self.SECTIONS:
with self.subTest(section=section):
cp = _config_with(section, {"hosts": "h:9200"})
opts = _opts()
_parse_config(cp, opts)
self.assertFalse(hasattr(opts, f"{section}_use_data_streams"))
def test_monthly_indexes_with_data_streams_warns(self):
from parsedmarc.cli import _parse_config
for section in self.SECTIONS:
with self.subTest(section=section):
cp = _config_with(
section,
{
"hosts": "h:9200",
"monthly_indexes": "true",
"use_data_streams": "true",
},
)
opts = _opts()
with self.assertLogs("parsedmarc.log", level="WARNING") as cm:
_parse_config(cp, opts)
self.assertEqual(len(cm.records), 1)
self.assertIn(f"[{section}] monthly_indexes", cm.output[0])
self.assertIn("use_data_streams", cm.output[0])
# Still parsed; the save functions ignore it in stream mode.
self.assertTrue(getattr(opts, f"{section}_monthly_indexes"))
def test_no_warning_unless_both_are_enabled(self):
from parsedmarc.cli import _parse_config
combinations = (
{"monthly_indexes": "true"},
{"use_data_streams": "true"},
{"monthly_indexes": "true", "use_data_streams": "false"},
{"monthly_indexes": "false", "use_data_streams": "true"},
)
for section in self.SECTIONS:
for settings in combinations:
with self.subTest(section=section, settings=settings):
cp = _config_with(section, {"hosts": "h:9200", **settings})
with self.assertNoLogs("parsedmarc.log", level="WARNING"):
_parse_config(cp, _opts())
class TestUseDataStreamsPassthrough(unittest.TestCase):
"""Each backend's parsed option reaches that backend's three save
calls, and only those: enabling it in one section leaves the other
backend's saves False, and it is False when absent from both."""
def _run(self, es_extra: str = "", os_extra: str = "") -> dict[str, MagicMock]:
names = (
"elastic.save_aggregate_report_to_elasticsearch",
"elastic.save_failure_report_to_elasticsearch",
"elastic.save_smtp_tls_report_to_elasticsearch",
"opensearch.save_aggregate_report_to_opensearch",
"opensearch.save_failure_report_to_opensearch",
"opensearch.save_smtp_tls_report_to_opensearch",
)
config = f"""[general]
save_aggregate = true
save_failure = true
save_smtp_tls = true
silent = true
[imap]
host = imap.example.com
user = test-user
password = test-password
[elasticsearch]
hosts = localhost
{es_extra}
[opensearch]
hosts = localhost
{os_extra}
"""
with tempfile.NamedTemporaryFile(
"w", suffix=".ini", delete=False
) as config_file:
config_file.write(config)
config_path = config_file.name
self.addCleanup(lambda: os.path.exists(config_path) and os.remove(config_path))
saves = {name: MagicMock() for name in names}
patches = [patch(f"parsedmarc.cli.{name}", saves[name]) for name in names]
patches += [
patch("parsedmarc.cli.elastic.set_hosts"),
patch("parsedmarc.cli.elastic.migrate_indexes"),
patch("parsedmarc.cli.opensearch.set_hosts"),
patch("parsedmarc.cli.opensearch.migrate_indexes"),
patch("parsedmarc.cli.IMAPConnection", return_value=object()),
patch(
"parsedmarc.cli.get_dmarc_reports_from_mailbox",
side_effect=_fetch_invoking_save_callback(
{
"aggregate_reports": [
{"policy_published": {"domain": "example.com"}}
],
"failure_reports": [{"reported_domain": "example.com"}],
"smtp_tls_reports": [
{"policies": [{"policy_domain": "example.com"}]}
],
}
),
),
patch.object(sys, "argv", ["parsedmarc", "-c", config_path]),
]
for p in patches:
p.start()
self.addCleanup(p.stop)
parsedmarc.cli._main()
return saves
def _assert_flags(
self, saves: dict[str, MagicMock], elastic: bool, opensearch: bool
) -> None:
for name, save in saves.items():
expected = elastic if name.startswith("elastic.") else opensearch
with self.subTest(save=name):
save.assert_called_once()
self.assertIs(save.call_args.kwargs["use_data_streams"], expected)
def test_enabled_in_both_sections_reaches_every_save_call(self):
enabled = "use_data_streams = true"
saves = self._run(es_extra=enabled, os_extra=enabled)
self._assert_flags(saves, elastic=True, opensearch=True)
def test_elasticsearch_section_only_reaches_elasticsearch_saves(self):
saves = self._run(es_extra="use_data_streams = true")
self._assert_flags(saves, elastic=True, opensearch=False)
def test_opensearch_section_only_reaches_opensearch_saves(self):
saves = self._run(os_extra="use_data_streams = true")
self._assert_flags(saves, elastic=False, opensearch=True)
def test_absent_defaults_to_false(self):
saves = self._run()
self._assert_flags(saves, elastic=False, opensearch=False)
class TestParseConfigSplunkHec(unittest.TestCase):
def test_splunk_hec_complete(self):
from parsedmarc.cli import _parse_config
@@ -9365,8 +10069,9 @@ class TestParseConfigLoki(unittest.TestCase):
class TestLokiCliWiring(unittest.TestCase):
"""End-to-end: a [loki] config reaches LokiClient, every report type
reaches its save method, and a failed push counts as an output error
(so mailbox messages are held back instead of archived)."""
reaches its save method, a failed push counts as an output error
(so mailbox messages are held back instead of archived), and a config
LokiClient rejects fails client initialization under the Loki name."""
def _run_main(self, reports, extra_general="", save_side_effect=None):
config = f"""[general]
@@ -9459,6 +10164,35 @@ tenant_id = fosdem
)
self.assertEqual(ctx.exception.code, 1)
def test_rejected_client_config_names_the_loki_section(self):
"""A [loki] config that parses but that LokiClient rejects is
reported as a ``Loki:`` RuntimeError.
_parse_config accepts ``token`` alongside ``username``/``password``;
only the real LokiClient constructor enforces that they are
exclusive, so this is the path a misconfigured section takes at
startup and on a SIGHUP reload.
"""
config = _config_with(
"loki",
{"url": "http://loki.example.com:3100", "token": "t", "username": "u"},
)
# The non-None Namespace defaults _main() gives the loki_* options.
opts = _output_client_opts(
loki_timestamp="ingest",
loki_skip_certificate_verification=False,
loki_timeout=60,
loki_batch_size=1000,
)
parsedmarc.cli._parse_config(config, opts)
with self.assertRaises(RuntimeError) as context:
parsedmarc.cli._init_output_clients(opts)
self.assertEqual(
str(context.exception),
"Loki: Loki token and username/password are exclusive",
)
self.assertIsInstance(context.exception.__cause__, ValueError)
if __name__ == "__main__":
unittest.main(verbosity=2)
+423 -3
View File
@@ -1,21 +1,29 @@
"""Tests for parsedmarc.elastic
Mocks at the elasticsearch.dsl SDK boundary (connections.create_connection,
Index, Search, Document.save) so the tests verify the parsedmarc-side
transformation logic — document construction, index naming, deduplication
queries, error wrapping — without needing a running Elasticsearch cluster.
Index, Search, Document.save, or a mock client registered as the
``default`` connection so the real Document.save runs) so the tests
verify the parsedmarc-side transformation logic — document
construction, index naming, deduplication queries, error wrapping —
without needing a running Elasticsearch cluster.
"""
import time
import unittest
from collections.abc import Iterator
from contextlib import contextmanager
from datetime import datetime, timezone
from typing import cast
from unittest.mock import MagicMock, call, patch
from elasticsearch import NotFoundError
import parsedmarc.elastic as elastic_module
from parsedmarc import InvalidFailureReport, parse_report_file
from parsedmarc.elastic import (
AlreadySaved,
ElasticsearchError,
create_data_streams,
create_indexes,
migrate_indexes,
save_aggregate_report_to_elasticsearch,
@@ -1528,6 +1536,418 @@ class TestSaveSmtpTlsReport(unittest.TestCase):
self.assertEqual(failure_detail.failure_reason_code, "no-policy-served")
# ---------------------------------------------------------------------------
# Data streams (issue #842)
# ---------------------------------------------------------------------------
def _not_found() -> NotFoundError:
"""The exception elasticsearch-py raises for an HTTP 404."""
return NotFoundError(
"index_not_found_exception", meta=MagicMock(status=404), body={}
)
def _stream_listing(*names: str) -> dict:
"""A GET _data_stream/<name> response body listing *names*."""
return {"data_streams": [{"name": name} for name in names]}
@contextmanager
def _registered_client() -> Iterator[MagicMock]:
"""Register a mock client as the ``default`` connection.
Both ``create_data_streams`` and ``Document.save`` resolve the client
through the connection registry, so registering the mock there (rather
than patching either function) lets the real save path run and the test
read what reached the transport. The previous registration is put back
afterwards.
"""
client = MagicMock()
client.index.return_value = {"_index": "x", "_id": "1", "result": "created"}
try:
previous = elastic_module.connections.get_connection("default")
except KeyError:
previous = None
elastic_module.connections.add_connection("default", client)
try:
yield client
finally:
if previous is None:
elastic_module.connections.remove_connection("default")
else:
elastic_module.connections.add_connection("default", previous)
class _DataStreamTestCase(unittest.TestCase):
"""Isolates each test from the module's data stream memo and the
Serverless flag."""
def setUp(self):
self._serverless = elastic_module._SERVERLESS
elastic_module._SERVERLESS = False
elastic_module._READY_DATA_STREAMS.clear()
def tearDown(self):
elastic_module._SERVERLESS = self._serverless
elastic_module._READY_DATA_STREAMS.clear()
class TestCreateDataStreams(_DataStreamTestCase):
def test_creates_template_and_stream_when_missing(self):
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(
["dmarc_aggregate"],
{"number_of_shards": 2, "number_of_replicas": 1},
)
template = client.indices.put_index_template.call_args.kwargs
self.assertEqual(template["name"], "dmarc_aggregate")
# Exact name, no wildcard: the template must never claim the
# date-suffixed indexes written without data streams.
self.assertEqual(template["index_patterns"], ["dmarc_aggregate"])
self.assertEqual(template["data_stream"], {})
self.assertEqual(template["priority"], 501)
self.assertEqual(template["composed_of"], ["dmarc_aggregate@custom"])
self.assertEqual(
template["ignore_missing_component_templates"],
["dmarc_aggregate@custom"],
)
self.assertEqual(
template["template"],
{
"settings": {"number_of_shards": 2, "number_of_replicas": 1},
"mappings": {"properties": {"@timestamp": {"type": "date"}}},
},
)
self.assertIn("dmarc_aggregate@custom", template["meta"]["description"])
client.indices.create_data_stream.assert_called_once_with(
name="dmarc_aggregate"
)
# No concrete index is created for a stream.
mock_index_cls.return_value.create.assert_not_called()
self.assertIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
def test_default_settings_are_one_shard_no_replicas(self):
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(["smtp_tls"])
settings = client.indices.put_index_template.call_args.kwargs["template"][
"settings"
]
self.assertEqual(settings, {"number_of_shards": 1, "number_of_replicas": 0})
def test_existing_stream_gets_template_but_is_not_recreated(self):
"""The template is put again so that settings changes reach the
next backing index; the stream itself already exists."""
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.return_value = _stream_listing(
"dmarc_failure"
)
create_data_streams(["dmarc_failure"])
client.indices.put_index_template.assert_called_once()
client.indices.create_data_stream.assert_not_called()
mock_index_cls.return_value.exists.assert_not_called()
self.assertIn("dmarc_failure", elastic_module._READY_DATA_STREAMS)
def test_concrete_index_holding_the_name_raises_without_writing(self):
"""Elasticsearch answers GET _data_stream/<name> for a name held by
a concrete index with 200 and an empty list, not a 404 (verified
against 8.19). Treating that 200 as "stream exists" would send
every document into the old undated index, such as one left by a
pre-5.0.0 release."""
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.return_value = _stream_listing()
mock_index_cls.return_value.exists.return_value = True
with self.assertRaises(ElasticsearchError) as ctx:
create_data_streams(["dmarc_aggregate"])
mock_index_cls.assert_called_with("dmarc_aggregate")
self.assertIn("already exists", str(ctx.exception))
self.assertIn("dmarc_aggregate", str(ctx.exception))
client.indices.put_index_template.assert_not_called()
client.indices.create_data_stream.assert_not_called()
self.assertNotIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
def test_alias_holding_the_name_raises(self):
"""A missing stream is a 404; an alias with the name still exists
as far as Index.exists() is concerned."""
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = True
with self.assertRaises(ElasticsearchError):
create_data_streams(["smtp_tls"])
client.indices.put_index_template.assert_not_called()
client.indices.create_data_stream.assert_not_called()
def test_ready_stream_is_not_set_up_again(self):
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(["dmarc_aggregate"])
create_data_streams(["dmarc_aggregate"])
self.assertEqual(client.indices.get_data_stream.call_count, 1)
self.assertEqual(client.indices.put_index_template.call_count, 1)
self.assertEqual(client.indices.create_data_stream.call_count, 1)
def test_set_hosts_forgets_ready_streams(self):
"""A new connection may be a different cluster or carry new index
settings, so the template has to be put again."""
elastic_module._READY_DATA_STREAMS.add("dmarc_aggregate")
with patch("parsedmarc.elastic.connections.create_connection"):
set_hosts("es:9200")
self.assertEqual(elastic_module._READY_DATA_STREAMS, set())
def test_sdk_error_is_wrapped_and_retried_next_time(self):
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
client.indices.put_index_template.side_effect = RuntimeError("cluster down")
with self.assertRaises(ElasticsearchError) as ctx:
create_data_streams(["dmarc_aggregate"])
self.assertIn("cluster down", str(ctx.exception))
self.assertNotIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
client.indices.create_data_stream.assert_not_called()
client.indices.put_index_template.side_effect = None
create_data_streams(["dmarc_aggregate"])
client.indices.create_data_stream.assert_called_once_with(
name="dmarc_aggregate"
)
self.assertIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
def test_serverless_strips_rejected_settings_from_template(self):
elastic_module._SERVERLESS = True
with (
_registered_client() as client,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(
["dmarc_aggregate"],
{
"number_of_shards": 1,
"number_of_replicas": 0,
"refresh_interval": "5s",
},
)
settings = client.indices.put_index_template.call_args.kwargs["template"][
"settings"
]
self.assertEqual(settings, {"refresh_interval": "5s"})
class TestSaveToDataStreams(_DataStreamTestCase):
"""Each save function, run through the real Document.save() into a
mock client: in stream mode the document goes to the undated name
with op_type=create and an @timestamp; in the default mode it goes to
the dated index as before, with the same op_type and @timestamp."""
def _save(self, save_function, report, **kwargs):
with (
_registered_client() as client,
patch("parsedmarc.elastic.Search") as mock_search_cls,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
mock_search_cls.return_value.execute.return_value = []
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
save_function(report, **kwargs)
return client, mock_search_cls, mock_index_cls
def test_aggregate_report_goes_to_the_stream(self):
client, mock_search_cls, mock_index_cls = self._save(
save_aggregate_report_to_elasticsearch,
_aggregate_report(),
index_prefix="cust_",
index_suffix="tenant_a",
# Ignored in stream mode.
monthly_indexes=True,
use_data_streams=True,
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "cust_dmarc_aggregate_tenant_a")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 15, tzinfo=timezone.utc),
)
self.assertEqual(index_call["body"]["report_id"], "agg-1")
self.assertEqual(
client.indices.put_index_template.call_args.kwargs["index_patterns"],
["cust_dmarc_aggregate_tenant_a"],
)
client.indices.create_data_stream.assert_called_once_with(
name="cust_dmarc_aggregate_tenant_a"
)
mock_index_cls.return_value.create.assert_not_called()
# The dedup search still uses the wildcard, which matches the
# stream as well as dated indexes written before the switch.
self.assertEqual(
mock_search_cls.call_args.kwargs["index"],
"cust_dmarc_aggregate_tenant_a*",
)
def test_aggregate_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
self,
):
client, _, mock_index_cls = self._save(
save_aggregate_report_to_elasticsearch, _aggregate_report()
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "dmarc_aggregate-2024-01-15")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 15, tzinfo=timezone.utc),
)
mock_index_cls.return_value.create.assert_called_once()
client.indices.put_index_template.assert_not_called()
client.indices.create_data_stream.assert_not_called()
def test_failure_report_goes_to_the_stream(self):
client, mock_search_cls, mock_index_cls = self._save(
save_failure_report_to_elasticsearch,
_failure_report(),
monthly_indexes=True,
use_data_streams=True,
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "dmarc_failure")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 1, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_called_once_with(name="dmarc_failure")
mock_index_cls.return_value.create.assert_not_called()
self.assertEqual(
mock_search_cls.call_args.kwargs["index"],
"dmarc_failure*,dmarc_forensic*",
)
def test_failure_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(self):
client, _, _ = self._save(
save_failure_report_to_elasticsearch, _failure_report()
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "dmarc_failure-2024-01-01")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 1, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_not_called()
def test_smtp_tls_report_goes_to_the_stream(self):
client, mock_search_cls, mock_index_cls = self._save(
save_smtp_tls_report_to_elasticsearch,
_smtp_tls_report(),
index_prefix="cust_",
use_data_streams=True,
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "cust_smtp_tls")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 2, 3, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_called_once_with(name="cust_smtp_tls")
mock_index_cls.return_value.create.assert_not_called()
self.assertEqual(mock_search_cls.call_args.kwargs["index"], "cust_smtp_tls*")
def test_smtp_tls_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
self,
):
client, _, _ = self._save(
save_smtp_tls_report_to_elasticsearch, _smtp_tls_report()
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "smtp_tls-2024-02-03")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 2, 3, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_not_called()
def test_stream_set_up_once_for_a_multi_record_report(self):
report = _aggregate_report()
report["records"].append(report["records"][0].copy())
client, _, _ = self._save(
save_aggregate_report_to_elasticsearch, report, use_data_streams=True
)
self.assertEqual(client.index.call_count, 2)
self.assertEqual(client.indices.put_index_template.call_count, 1)
def test_name_collision_saves_nothing(self):
with (
_registered_client() as client,
patch("parsedmarc.elastic.Search") as mock_search_cls,
patch("parsedmarc.elastic.Index") as mock_index_cls,
):
mock_search_cls.return_value.execute.return_value = []
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = True
with self.assertRaises(ElasticsearchError):
save_smtp_tls_report_to_elasticsearch(
_smtp_tls_report(), use_data_streams=True
)
client.index.assert_not_called()
class TestMigrateIndexesWithDataStream(unittest.TestCase):
def test_legacy_fo_migration_skips_a_stream_with_the_name(self):
"""With use_data_streams the undated legacy name is the stream
itself. Index.exists() is true for it, and the field mapping comes
back keyed by a backing index; its fo is text, so nothing is
migrated or deleted."""
with (
patch("parsedmarc.elastic.Index") as mock_index_cls,
patch("parsedmarc.elastic.reindex") as mock_reindex,
):
index = mock_index_cls.return_value
index.exists.return_value = True
index.get_field_mapping.return_value = {
".ds-dmarc_aggregate-2026.10.31-000001": {
"mappings": {
"published_policy.fo": {
"full_name": "published_policy.fo",
"mapping": {"fo": {"type": "text"}},
}
}
}
}
migrate_indexes(legacy_fo_indexes=["dmarc_aggregate"])
mock_reindex.assert_not_called()
index.create.assert_not_called()
index.delete.assert_not_called()
class TestBackwardCompatAlias(unittest.TestCase):
def test_save_forensic_alias_points_to_save_failure(self):
self.assertIs(
+61
View File
@@ -3,6 +3,7 @@
import json
import os
import time
import traceback
import unittest
from typing import cast
from unittest.mock import MagicMock, patch
@@ -164,6 +165,44 @@ class TestLokiClientInit(unittest.TestCase):
self.assertIsInstance(client.session.auth, httpx.BasicAuth)
class TestLokiInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py:
``raise InvalidURL(f"Invalid port: {port!r}")``). Its implicit
``__context__`` is a ValueError that repeats that part of the password
too."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@loki.example.org:3100", "p4ssw0rd"),
("https://admin:p4ss?w0rd@loki.example.org:3100", "p4ss"),
("https://admin:p4ss#w0rd@loki.example.org:3100", "p4ss"),
)
def test_invalid_url_raises_fixed_message_without_password(self):
for url, secret in self.MALFORMED:
with self.subTest(url=url):
with self.assertRaises(ValueError) as ctx:
LokiClient(url)
self.assertEqual(
str(ctx.exception),
"Invalid Loki URL (details withheld because they may "
"include credentials)",
)
self.assertIsNone(ctx.exception.__cause__)
self.assertTrue(ctx.exception.__suppress_context__)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
def test_valid_url_keeps_userinfo_for_requests(self):
client = LokiClient("https://admin:p4ssw0rd@loki.example.org:3100")
self.assertEqual(
client.url,
"https://admin:p4ssw0rd@loki.example.org:3100/loki/api/v1/push",
)
@patch(
"parsedmarc.loki.parsed_aggregate_reports_to_csv_rows", return_value=AGGREGATE_ROWS
)
@@ -370,6 +409,28 @@ class TestLokiErrors(unittest.TestCase):
client.save_aggregate_report_to_loki(AGGREGATE)
self.assertIn("refused", str(ctx.exception))
def test_errors_mask_url_password(self, _rows):
"""GHSA-hjr3-rx4c-24cv: the URL keeps its userinfo, and the CLI
logs LokiError text, so the password must be masked in it."""
secret = "s3cretPW"
client = LokiClient(f"http://u:{secret}@loki.example.org:3100")
client.session = MagicMock()
_session(client).post.return_value = _ok_response(500)
with self.assertRaises(LokiError) as status_ctx:
client.save_aggregate_report_to_loki(AGGREGATE)
_session(client).post.side_effect = httpx.ConnectError("refused")
with self.assertRaises(LokiError) as network_ctx:
client.save_aggregate_report_to_loki(AGGREGATE)
for ctx in (status_ctx, network_ctx):
with self.subTest(error=str(ctx.exception)):
self.assertNotIn(secret, str(ctx.exception))
self.assertIn("http://u:***@loki.example.org:3100", str(ctx.exception))
# Redaction is message-only: the request still goes to the real URL
self.assertEqual(
_session(client).post.call_args.args[0],
f"http://u:{secret}@loki.example.org:3100/loki/api/v1/push",
)
def test_blocked_ingestion_status_raises(self, _rows):
"""Loki returns 260 (block_ingestion_status_code default) and drops
the logs when ingestion is blocked; that must not read as success."""
+418 -3
View File
@@ -1,21 +1,29 @@
"""Tests for parsedmarc.opensearch
Mocks at the opensearch-dsl SDK boundary (connections.create_connection,
Index, Search, Document.save) so the tests verify the parsedmarc-side
transformation logic — document construction, index naming, deduplication
queries, error wrapping — without needing a running OpenSearch cluster.
Index, Search, Document.save, or a mock client registered as the
``default`` connection so the real Document.save runs) so the tests
verify the parsedmarc-side transformation logic — document
construction, index naming, deduplication queries, error wrapping —
without needing a running OpenSearch cluster.
"""
import time
import unittest
from collections.abc import Iterator
from contextlib import contextmanager
from datetime import datetime, timezone
from typing import cast
from unittest.mock import MagicMock, call, patch
from opensearchpy import NotFoundError
import parsedmarc.opensearch as opensearch_module
from parsedmarc import InvalidFailureReport, parse_report_file
from parsedmarc.opensearch import (
AlreadySaved,
OpenSearchError,
create_data_streams,
create_indexes,
migrate_indexes,
save_aggregate_report_to_opensearch,
@@ -233,6 +241,30 @@ class TestSetHosts(unittest.TestCase):
kwargs = mock_conn.call_args.kwargs
self.assertEqual(kwargs["hosts"], ["es1:9200", "es2:9200"])
def test_debug_log_masks_credentials_but_connects_with_them(self):
"""GHSA-hjr3-rx4c-24cv: the "Connecting to OpenSearch" debug line
must not carry the URL password, while the client still gets the
real URL."""
url = "https://admin:s3cretPW@es.example:9200"
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
set_hosts(url)
output = "\n".join(logs.output)
self.assertNotIn("s3cretPW", output)
self.assertIn("https://admin:***@es.example:9200", output)
self.assertEqual(mock_conn.call_args.kwargs["hosts"], [url])
def test_debug_log_masks_schemeless_host_list(self):
hosts = ["admin:s3cretPW@es1:9200", "es2:9200"]
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
set_hosts(hosts)
output = "\n".join(logs.output)
self.assertNotIn("s3cretPW", output)
self.assertIn("admin:***@es1:9200", output)
self.assertIn("es2:9200", output)
self.assertEqual(mock_conn.call_args.kwargs["hosts"], hosts)
def test_timeout_default_60s(self):
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
set_hosts("es:9200")
@@ -1519,6 +1551,389 @@ class TestSaveSmtpTlsReport(unittest.TestCase):
self.assertEqual(failure_detail.failure_reason_code, "no-policy-served")
# ---------------------------------------------------------------------------
# Data streams (issue #842)
# ---------------------------------------------------------------------------
def _not_found() -> NotFoundError:
"""The exception opensearch-py raises for an HTTP 404."""
return NotFoundError(404, "index_not_found_exception", {})
def _stream_listing(*names: str) -> dict:
"""A GET _data_stream/<name> response body listing *names*."""
return {"data_streams": [{"name": name} for name in names]}
@contextmanager
def _registered_client() -> Iterator[MagicMock]:
"""Register a mock client as the ``default`` connection.
Both ``create_data_streams`` and ``Document.save`` resolve the client
through the connection registry, so registering the mock there (rather
than patching either function) lets the real save path run and the test
read what reached the transport. The previous registration is put back
afterwards.
"""
client = MagicMock()
client.index.return_value = {"_index": "x", "_id": "1", "result": "created"}
try:
previous = opensearch_module.connections.get_connection("default")
except KeyError:
previous = None
opensearch_module.connections.add_connection("default", client)
try:
yield client
finally:
if previous is None:
opensearch_module.connections.remove_connection("default")
else:
opensearch_module.connections.add_connection("default", previous)
class _DataStreamTestCase(unittest.TestCase):
"""Isolates each test from the module's data stream memo."""
def setUp(self):
opensearch_module._READY_DATA_STREAMS.clear()
def tearDown(self):
opensearch_module._READY_DATA_STREAMS.clear()
class TestCreateDataStreams(_DataStreamTestCase):
def test_creates_template_and_stream_when_missing(self):
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(
["dmarc_aggregate"],
{"number_of_shards": 2, "number_of_replicas": 1},
)
template_call = client.indices.put_index_template.call_args.kwargs
self.assertEqual(template_call["name"], "dmarc_aggregate")
body = template_call["body"]
# Exact name, no wildcard: the template must never claim the
# date-suffixed indexes written without data streams.
self.assertEqual(body["index_patterns"], ["dmarc_aggregate"])
self.assertEqual(body["data_stream"], {})
self.assertEqual(body["priority"], 501)
self.assertNotIn("composed_of", body)
self.assertEqual(
body["template"],
{
"settings": {"number_of_shards": 2, "number_of_replicas": 1},
"mappings": {"properties": {"@timestamp": {"type": "date"}}},
},
)
self.assertIn("ism_template", body["_meta"]["description"])
client.indices.create_data_stream.assert_called_once_with(
name="dmarc_aggregate"
)
# No concrete index is created for a stream.
mock_index_cls.return_value.create.assert_not_called()
self.assertIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
def test_default_settings_are_one_shard_no_replicas(self):
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(["smtp_tls"])
settings = client.indices.put_index_template.call_args.kwargs["body"][
"template"
]["settings"]
self.assertEqual(settings, {"number_of_shards": 1, "number_of_replicas": 0})
def test_existing_stream_gets_template_but_is_not_recreated(self):
"""The template is put again so that settings changes reach the
next backing index; the stream itself already exists."""
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.return_value = _stream_listing(
"dmarc_failure"
)
create_data_streams(["dmarc_failure"])
client.indices.put_index_template.assert_called_once()
client.indices.create_data_stream.assert_not_called()
mock_index_cls.return_value.exists.assert_not_called()
self.assertIn("dmarc_failure", opensearch_module._READY_DATA_STREAMS)
def test_concrete_index_holding_the_name_raises_without_writing(self):
"""OpenSearch 3 answers GET _data_stream/<name> for a name held by
a concrete index with a 404, so the Index.exists() check is what
catches it, before anything is written. Writing to that name would
otherwise go into the old undated index, such as one left by a
pre-5.0.0 release."""
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = True
with self.assertRaises(OpenSearchError) as ctx:
create_data_streams(["dmarc_aggregate"])
mock_index_cls.assert_called_with("dmarc_aggregate")
self.assertIn("already exists", str(ctx.exception))
self.assertIn("dmarc_aggregate", str(ctx.exception))
client.indices.put_index_template.assert_not_called()
client.indices.create_data_stream.assert_not_called()
self.assertNotIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
def test_empty_listing_is_not_taken_as_an_existing_stream(self):
"""Elasticsearch answers a name held by a concrete index with 200
and an empty list; the shared logic must not read that as "the
stream exists" if an OpenSearch release ever does the same."""
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.return_value = _stream_listing()
mock_index_cls.return_value.exists.return_value = True
with self.assertRaises(OpenSearchError):
create_data_streams(["smtp_tls"])
client.indices.put_index_template.assert_not_called()
client.indices.create_data_stream.assert_not_called()
def test_ready_stream_is_not_set_up_again(self):
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
create_data_streams(["dmarc_aggregate"])
create_data_streams(["dmarc_aggregate"])
self.assertEqual(client.indices.get_data_stream.call_count, 1)
self.assertEqual(client.indices.put_index_template.call_count, 1)
self.assertEqual(client.indices.create_data_stream.call_count, 1)
def test_set_hosts_forgets_ready_streams(self):
"""A new connection may be a different cluster or carry new index
settings, so the template has to be put again."""
opensearch_module._READY_DATA_STREAMS.add("dmarc_aggregate")
with patch("parsedmarc.opensearch.connections.create_connection"):
set_hosts("os:9200")
self.assertEqual(opensearch_module._READY_DATA_STREAMS, set())
def test_sdk_error_is_wrapped_and_retried_next_time(self):
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
client.indices.put_index_template.side_effect = RuntimeError("cluster down")
with self.assertRaises(OpenSearchError) as ctx:
create_data_streams(["dmarc_aggregate"])
self.assertIn("cluster down", str(ctx.exception))
self.assertNotIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
client.indices.create_data_stream.assert_not_called()
client.indices.put_index_template.side_effect = None
create_data_streams(["dmarc_aggregate"])
client.indices.create_data_stream.assert_called_once_with(
name="dmarc_aggregate"
)
self.assertIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
class TestSaveToDataStreams(_DataStreamTestCase):
"""Each save function, run through the real Document.save() into a
mock client: in stream mode the document goes to the undated name
with op_type=create and an @timestamp; in the default mode it goes to
the dated index as before, with the same op_type and @timestamp."""
def _save(self, save_function, report, **kwargs):
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Search") as mock_search_cls,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
mock_search_cls.return_value.execute.return_value = []
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = False
save_function(report, **kwargs)
return client, mock_search_cls, mock_index_cls
def test_aggregate_report_goes_to_the_stream(self):
client, mock_search_cls, mock_index_cls = self._save(
save_aggregate_report_to_opensearch,
_aggregate_report(),
index_prefix="cust_",
index_suffix="tenant_a",
# Ignored in stream mode.
monthly_indexes=True,
use_data_streams=True,
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "cust_dmarc_aggregate_tenant_a")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 15, tzinfo=timezone.utc),
)
self.assertEqual(index_call["body"]["report_id"], "agg-1")
self.assertEqual(
client.indices.put_index_template.call_args.kwargs["body"][
"index_patterns"
],
["cust_dmarc_aggregate_tenant_a"],
)
client.indices.create_data_stream.assert_called_once_with(
name="cust_dmarc_aggregate_tenant_a"
)
mock_index_cls.return_value.create.assert_not_called()
# The dedup search still uses the wildcard, which matches the
# stream as well as dated indexes written before the switch.
self.assertEqual(
mock_search_cls.call_args.kwargs["index"],
"cust_dmarc_aggregate_tenant_a*",
)
def test_aggregate_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
self,
):
client, _, mock_index_cls = self._save(
save_aggregate_report_to_opensearch, _aggregate_report()
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "dmarc_aggregate-2024-01-15")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 15, tzinfo=timezone.utc),
)
mock_index_cls.return_value.create.assert_called_once()
client.indices.put_index_template.assert_not_called()
client.indices.create_data_stream.assert_not_called()
def test_failure_report_goes_to_the_stream(self):
client, mock_search_cls, mock_index_cls = self._save(
save_failure_report_to_opensearch,
_failure_report(),
monthly_indexes=True,
use_data_streams=True,
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "dmarc_failure")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 1, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_called_once_with(name="dmarc_failure")
mock_index_cls.return_value.create.assert_not_called()
self.assertEqual(
mock_search_cls.call_args.kwargs["index"],
"dmarc_failure*,dmarc_forensic*",
)
def test_failure_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(self):
client, _, _ = self._save(save_failure_report_to_opensearch, _failure_report())
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "dmarc_failure-2024-01-01")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 1, 1, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_not_called()
def test_smtp_tls_report_goes_to_the_stream(self):
client, mock_search_cls, mock_index_cls = self._save(
save_smtp_tls_report_to_opensearch,
_smtp_tls_report(),
index_prefix="cust_",
use_data_streams=True,
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "cust_smtp_tls")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 2, 3, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_called_once_with(name="cust_smtp_tls")
mock_index_cls.return_value.create.assert_not_called()
self.assertEqual(mock_search_cls.call_args.kwargs["index"], "cust_smtp_tls*")
def test_smtp_tls_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
self,
):
client, _, _ = self._save(
save_smtp_tls_report_to_opensearch, _smtp_tls_report()
)
index_call = client.index.call_args.kwargs
self.assertEqual(index_call["index"], "smtp_tls-2024-02-03")
self.assertEqual(index_call["op_type"], "create")
self.assertEqual(
index_call["body"]["@timestamp"],
datetime(2024, 2, 3, tzinfo=timezone.utc),
)
client.indices.create_data_stream.assert_not_called()
def test_stream_set_up_once_for_a_multi_record_report(self):
report = _aggregate_report()
report["records"].append(report["records"][0].copy())
client, _, _ = self._save(
save_aggregate_report_to_opensearch, report, use_data_streams=True
)
self.assertEqual(client.index.call_count, 2)
self.assertEqual(client.indices.put_index_template.call_count, 1)
def test_name_collision_saves_nothing(self):
with (
_registered_client() as client,
patch("parsedmarc.opensearch.Search") as mock_search_cls,
patch("parsedmarc.opensearch.Index") as mock_index_cls,
):
mock_search_cls.return_value.execute.return_value = []
client.indices.get_data_stream.side_effect = _not_found()
mock_index_cls.return_value.exists.return_value = True
with self.assertRaises(OpenSearchError):
save_smtp_tls_report_to_opensearch(
_smtp_tls_report(), use_data_streams=True
)
client.index.assert_not_called()
class TestMigrateIndexesWithDataStream(unittest.TestCase):
def test_legacy_fo_migration_skips_a_stream_with_the_name(self):
"""With use_data_streams the undated legacy name is the stream
itself. Index.exists() is true for it, and the field mapping comes
back keyed by a backing index; its fo is text, so nothing is
migrated or deleted."""
with (
patch("parsedmarc.opensearch.Index") as mock_index_cls,
patch("parsedmarc.opensearch.reindex") as mock_reindex,
):
index = mock_index_cls.return_value
index.exists.return_value = True
index.get_field_mapping.return_value = {
".ds-dmarc_aggregate-000001": {
"mappings": {
"published_policy.fo": {
"full_name": "published_policy.fo",
"mapping": {"fo": {"type": "text"}},
}
}
}
}
migrate_indexes(legacy_fo_indexes=["dmarc_aggregate"])
mock_reindex.assert_not_called()
index.create.assert_not_called()
index.delete.assert_not_called()
class TestBackwardCompatAlias(unittest.TestCase):
def test_save_forensic_alias_points_to_save_failure(self):
self.assertIs(
+63
View File
@@ -311,11 +311,74 @@ class TestPostgreSQLConstruction(unittest.TestCase):
"""A driver-level connection error is wrapped in PostgreSQLError."""
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
mock_psycopg.Error = Exception
mock_psycopg.ProgrammingError = type("ProgrammingError", (Exception,), {})
mock_psycopg.connect.side_effect = mock_psycopg.Error("refused")
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(host="localhost")
self.assertIn("refused", str(ctx.exception))
def test_connect_failure_detail_is_url_redacted(self):
"""A non-parse driver error keeps its detail, with URL credentials
masked."""
programming_error = type("ProgrammingError", (Exception,), {})
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
mock_psycopg.Error = Exception
mock_psycopg.ProgrammingError = programming_error
mock_psycopg.connect.side_effect = mock_psycopg.Error(
'connection to "postgresql://u:PGSECRET@db.example/x" failed'
)
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(connection_string="postgresql://u:PGSECRET@h/db")
message = str(ctx.exception)
self.assertNotIn("PGSECRET", message)
self.assertIn("u:***@db.example/x", message)
def test_malformed_connection_string_detail_withheld(self):
"""A ProgrammingError from parsing the connection string (which libpq
words with the whole string or a password fragment) is replaced with
a fixed message that still identifies the problem."""
programming_error = type("ProgrammingError", (Exception,), {})
for dsn, driver_message in (
(
"postgresql://u:PGSECRET@[::1/db",
'end of string reached when looking for matching "]" in IPv6 '
'host address in URI: "postgresql://u:PGSECRET@[::1/db"',
),
(
"postgresql://u:PG%ZZSECRET@h/db",
'invalid percent-encoded token: "PG%ZZSECRET"',
),
):
with self.subTest(dsn=dsn):
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
mock_psycopg.Error = Exception
mock_psycopg.ProgrammingError = programming_error
mock_psycopg.connect.side_effect = programming_error(driver_message)
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(connection_string=dsn)
message = str(ctx.exception)
self.assertNotIn("PGSECRET", message)
self.assertNotIn("SECRET", message)
self.assertIn("Invalid PostgreSQL connection_string", message)
@unittest.skipIf(
importlib.util.find_spec("psycopg") is None, "psycopg is not installed"
)
def test_real_psycopg_malformed_connection_string_withheld(self):
"""With the real driver, both malformed strings fail in parsing (no
network I/O) and the raised message carries no part of the password."""
for dsn in (
"postgresql://u:PGSECRET@[::1/db",
"postgresql://u:PG%ZZSECRET@h/db",
):
with self.subTest(dsn=dsn):
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(connection_string=dsn)
self.assertNotIn("SECRET", str(ctx.exception))
self.assertIn(
"Invalid PostgreSQL connection_string", str(ctx.exception)
)
def test_create_tables_executes_all_ddl(self):
"""create_tables issues CREATE TABLE for every table and the indexes."""
client, mock_conn = _make_client()
+53
View File
@@ -2,6 +2,7 @@
import json
import time
import traceback
import unittest
from unittest.mock import MagicMock, patch
@@ -469,6 +470,58 @@ class TestSaveSmtpTlsReportsToSplunk(unittest.TestCase):
)
class TestInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
any network I/O. The error reaches the log on every batch."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@splunk.example.com:8088", "p4ssw0rd"),
("https://admin:p4ss?w0rd@splunk.example.com:8088", "p4ss"),
("https://admin:p4ss#w0rd@splunk.example.com:8088", "p4ss"),
)
def test_invalid_url_raises_fixed_message_without_password(self):
saves = (
("aggregate", "save_aggregate_reports_to_splunk", _aggregate_report),
("failure", "save_failure_reports_to_splunk", _failure_report),
("smtp_tls", "save_smtp_tls_reports_to_splunk", _smtp_tls_report),
)
for url, secret in self.MALFORMED:
client = HECClient(url=url, access_token="tok", index="dmarc")
for name, method, make_report in saves:
with self.subTest(url=url, report=name):
with self.assertRaises(SplunkError) as ctx:
getattr(client, method)(make_report())
self.assertEqual(
str(ctx.exception),
"Invalid Splunk HEC URL (details withheld because they "
"may include credentials)",
)
self.assertIsNone(ctx.exception.__cause__)
self.assertTrue(ctx.exception.__suppress_context__)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
def test_valid_url_is_the_one_posted(self):
client = HECClient(
url="https://admin:p4ssw0rd@splunk.example.com:8088",
access_token="tok",
index="dmarc",
)
client.session = MagicMock()
client.session.post.return_value = _ok_response()
client.save_aggregate_reports_to_splunk(_aggregate_report())
self.assertEqual(
client.session.post.call_args.args[0],
"https://admin:p4ssw0rd@splunk.example.com:8088"
"/services/collector/event/1.0",
)
class TestHECClientClose(unittest.TestCase):
def test_close_closes_session(self):
client = _client()
+570 -26
View File
@@ -4,16 +4,19 @@ import os
import shutil
import tempfile
import time
import traceback
import unittest
from datetime import datetime, timezone
from importlib.resources import files
from tempfile import NamedTemporaryFile
from types import SimpleNamespace
from typing import cast
from unittest.mock import MagicMock, patch
import dns.exception
import dns.message
import dns.nameserver
import dns.query
import dns.rdata
import dns.rdataclass
import dns.rdatatype
@@ -514,6 +517,122 @@ class Test(unittest.TestCase):
self.assertTrue(parsedmarc.utils.is_outlook_msg(magic))
class TestRedactUrlCredentials(unittest.TestCase):
"""Tests for parsedmarc.utils.redact_url_credentials()"""
def testMasksUserinfoAndKeepsHost(self):
cases = [
("https://user:secret@host", "https://user:***@host"),
("https://user:secret@host:9200/", "https://user:***@host:9200/"),
("localhost:9200", "localhost:9200"),
("127.0.0.1:9299", "127.0.0.1:9299"),
("user:secret@localhost:9200", "user:***@localhost:9200"),
# A "://" in the path or query of a scheme-less value is not
# the scheme separator; the userinfo at offset 0 is still masked
(
"user:secret@host:9200/x?r=http://y",
"user:***@host:9200/x?r=http://y",
),
("user:secret@host?r=a://b", "user:***@host?r=a://b"),
("https://user:p@ss@host/x", "https://user:***@host/x"),
("https://user:p:w@[::1]:9200/", "https://user:***@[::1]:9200/"),
("https://[::1]:9200", "https://[::1]:9200"),
("//user:pw@host", "//user:***@host"),
# Documented over-masking: with no "@" in the authority, an "@"
# later in the value is masked through, since it may end a
# password that contains "/", "?" or "#"
("http://host/path?e=a@b.com", "http://***@b.com"),
("https://host/path@file", "https://***@file"),
("https://host/?redirect=https://u:p@evil", "https://***@evil"),
# Documented non-goal: with userinfo in the authority, credentials
# inside a later query value are not masked
(
"https://a:b@host/?redirect=https://u:p@evil",
"https://a:***@host/?redirect=https://u:p@evil",
),
("https://tok@host", "https://***@host"),
("user@host:9200", "***@host:9200"),
("https://user:@host", "https://user:***@host"),
("https://u%40x:p%3Aw@h", "https://u%40x:***@h"),
("not a url @ all", "***@ all"),
("", ""),
("::::", "::::"),
]
for value, expected in cases:
with self.subTest(value=value):
self.assertEqual(
parsedmarc.utils.redact_url_credentials(value), expected
)
def testMasksPasswordContainingAuthorityDelimiter(self):
"""A password with an unencoded "/", "?" or "#" ends the authority
before the "@" (RFC 3986 section 3.2), which used to leave the value
unmasked; everything through the last "@" is masked instead."""
cases = [
("https://admin:p4ssw0rd/extra@host:3100", "https://***@host:3100"),
("https://admin:p4ss?w0rd@host", "https://***@host"),
("https://admin:p4ss#w0rd@host", "https://***@host"),
("admin:p4ss/w0rd@host:9200", "***@host:9200"),
("//admin:p4ss/w0rd@host", "//***@host"),
# The password may itself contain "@"; the last one is used
("https://admin:p4ss/w@rd@host:3100/x", "https://***@host:3100/x"),
# A "://" after the "/" is not the scheme separator
(
"admin:p4ss/w0rd@host/x?r=http://y",
"***@host/x?r=http://y",
),
]
for value, expected in cases:
with self.subTest(value=value):
result = parsedmarc.utils.redact_url_credentials(value)
self.assertEqual(result, expected)
self.assertNotIn("p4ss", result)
def testListMasksPasswordContainingAuthorityDelimiter(self):
self.assertEqual(
parsedmarc.utils.redact_url_credentials(
["https://admin:p4ss/w0rd@h1:9200", "https://h2:9200"]
),
["https://***@h1:9200", "https://h2:9200"],
)
def testListIsRedactedElementWiseIntoANewList(self):
original = ["a:b@h1", "https://h2"]
result = parsedmarc.utils.redact_url_credentials(original)
self.assertEqual(result, ["a:***@h1", "https://h2"])
self.assertIsNot(result, original)
self.assertEqual(original, ["a:b@h1", "https://h2"])
def testListKeepsNonStringItems(self):
mixed = cast(list[str], ["a:b@h1", None])
self.assertEqual(
parsedmarc.utils.redact_url_credentials(mixed), ["a:***@h1", None]
)
def testNonStringInputIsReturnedUnchanged(self):
for value in (None, 42):
with self.subTest(value=value):
self.assertEqual(
parsedmarc.utils.redact_url_credentials(cast(str, value)), value
)
def testLongInputsAreFast(self):
"""No regex is used, so adversarial input is linear. A 1 MB string
of repeated openers and a 200,000-character run of '@' both finish
in milliseconds."""
for value in (
"://" * 350_000,
"@" * 200_000,
"a:b@" * 250_000,
"a:b/" * 250_000 + "@",
"/@" * 500_000,
):
with self.subTest(prefix=value[:8]):
started = time.monotonic()
parsedmarc.utils.redact_url_credentials(value)
self.assertLess(time.monotonic() - started, 5)
class TestSanitizeForLog(unittest.TestCase):
"""Tests for parsedmarc.utils.sanitize_for_log()"""
@@ -539,6 +658,197 @@ class TestSanitizeForLog(unittest.TestCase):
self.assertNotIn("\n", result)
class TestDownloadUrlCredentialsNotLogged(unittest.TestCase):
"""A download URL with userinfo must not leak its password through the
"Trying to fetch" debug line or the warning that embeds the httpx
``HTTPStatusError`` message (which quotes the full URL).
The loaders run for real; only the network is replaced, with an
``httpx.MockTransport`` returning 401 so httpx itself builds the
exception text.
"""
URL = "https://dluser:DLSECRET@downloads.example.test/map.dat"
@staticmethod
def _transport_returning(handler):
def get(url, **kwargs):
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
return client.get(url, **kwargs)
return get
@classmethod
def _status_getter(cls, reason_phrase=None):
extensions = {}
if reason_phrase is not None:
extensions["reason_phrase"] = reason_phrase
return cls._transport_returning(
lambda request: httpx.Response(401, extensions=extensions)
)
@staticmethod
def _connect_error_getter(url, **kwargs):
def handler(request):
raise httpx.ConnectError("connection refused", request=request)
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
return client.get(url, **kwargs)
# Reason phrases containing URL delimiters used to defeat redaction of
# the HTTPStatusError text, which puts the phrase before the URL.
REASON_PHRASES = (
None,
b"Unauthorized/Expired",
b"Unauthorized?",
b"Gone#",
b"Not @ home",
)
def _assert_redacted(self, logs):
output = "\n".join(logs.output)
self.assertNotIn("DLSECRET", output)
self.assertTrue(
any(
line.startswith("DEBUG") and "downloads.example.test" in line
for line in logs.output
),
output,
)
warnings = [line for line in logs.output if line.startswith("WARNING")]
self.assertTrue(
any(
"HTTP 401" in line
and "'https://dluser:***@downloads.example.test/map.dat'" in line
for line in warnings
),
output,
)
def _run_psl(self, url=None):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
parsedmarc.utils.load_psl_overrides(url=url or self.URL)
return logs
def _run_ip_db(self, url=None):
tmp_dir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmp_dir, ignore_errors=True)
old_path = parsedmarc.utils._IP_DB_PATH
self.addCleanup(setattr, parsedmarc.utils, "_IP_DB_PATH", old_path)
parsedmarc.utils._IP_DB_PATH = None
with (
patch("parsedmarc.utils.tempfile.gettempdir", return_value=tmp_dir),
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
):
parsedmarc.utils.load_ip_db(url=url or self.URL)
return logs
def _run_reverse_dns_map(self, url=None):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
parsedmarc.utils.load_reverse_dns_map({}, url=url or self.URL)
return logs
def _check_loader(self, run):
for phrase in self.REASON_PHRASES:
with self.subTest(reason_phrase=phrase):
with patch("parsedmarc.utils.httpx.get", self._status_getter(phrase)):
self._assert_redacted(run())
def _check_transport_error(self, run):
with patch("parsedmarc.utils.httpx.get", self._connect_error_getter):
logs = run()
output = "\n".join(logs.output)
self.assertNotIn("DLSECRET", output)
self.assertTrue(
any(
line.startswith("WARNING")
and "ConnectError: connection refused" in line
for line in logs.output
),
output,
)
# The password holds an unencoded "/", "?" or "#", which RFC 3986 section
# 3.2 ends the authority at, so the text after the colon is read as the
# port. Real httpx raises InvalidURL("Invalid port: 'p4ssw0rd'") while
# parsing, before any network I/O (httpx 0.28.1 _urlparse.py:
# ``raise InvalidURL(f"Invalid port: {port!r}")``). InvalidURL is not an
# httpx.HTTPError, and its message quotes part of the password.
MALFORMED_URLS = (
("https://admin:p4ssw0rd/extra@downloads.example.test/map.dat", "p4ssw0rd"),
("https://admin:p4ss?w0rd@downloads.example.test/map.dat", "p4ss"),
("https://admin:p4ss#w0rd@downloads.example.test/map.dat", "p4ss"),
)
INVALID_URL_MESSAGE = (
"invalid URL (details withheld because they may include credentials)"
)
def _check_invalid_url(self, run, source):
for url, secret in self.MALFORMED_URLS:
with self.subTest(url=url):
requested = []
real_get = httpx.get
def recording_get(requested_url, **kwargs):
requested.append(requested_url)
if requested_url != url:
# e.g. the PSL overrides fetch that
# load_reverse_dns_map triggers; stay offline
raise httpx.ConnectError("offline")
return real_get(requested_url, **kwargs)
with patch("parsedmarc.utils.httpx.get", recording_get):
logs = run(url) # must return, not crash on InvalidURL
# The loader still requested the real URL, not a redacted one
self.assertIn(url, requested)
output = "\n".join(logs.output)
# Neither the warning nor the "Trying to fetch" debug line
# may show any part of the password
self.assertNotIn(secret, output)
self.assertIn(
f"DEBUG:parsedmarc.log:Trying to fetch {source} from "
"https://***@downloads.example.test/map.dat...",
logs.output,
)
self.assertIn(
f"WARNING:parsedmarc.log:Failed to fetch {source}: "
f"{self.INVALID_URL_MESSAGE}",
logs.output,
)
self.assertNotIn("Not a valid CSV file", output)
self.assertNotIn("Failed to save IP database", output)
def test_psl_overrides_loader_invalid_url(self):
"""A malformed ``psl_overrides_url`` used to crash startup with the
InvalidURL traceback; it now warns and falls back to the local file."""
self._check_invalid_url(self._run_psl, "PSL overrides")
self.assertIn(".linode.com", parsedmarc.utils.psl_overrides)
def test_ip_db_loader_invalid_url(self):
self._check_invalid_url(self._run_ip_db, "IP database")
def test_reverse_dns_map_loader_invalid_url(self):
self._check_invalid_url(self._run_reverse_dns_map, "reverse DNS map")
def test_psl_overrides_loader(self):
self._check_loader(self._run_psl)
def test_ip_db_loader(self):
self._check_loader(self._run_ip_db)
def test_reverse_dns_map_loader(self):
self._check_loader(self._run_reverse_dns_map)
def test_psl_overrides_loader_transport_error(self):
self._check_transport_error(self._run_psl)
def test_ip_db_loader_transport_error(self):
self._check_transport_error(self._run_ip_db)
def test_reverse_dns_map_loader_transport_error(self):
self._check_transport_error(self._run_reverse_dns_map)
class TestLoadPSLOverrides(unittest.TestCase):
"""Covers `parsedmarc.utils.load_psl_overrides`."""
@@ -1397,14 +1707,24 @@ class TestQueryDnsRetries(unittest.TestCase):
self.assertEqual(mock_resolve.call_count, 3)
def _dnspython_http_module():
"""Returns the HTTP library dnspython's DoH code imported: httpx2 on
dnspython 2.9+, httpx on 2.7 and 2.8. Derived from dnspython rather than
from parsedmarc's helper, so the tests below check the helper against
what ``dns.query.https()`` actually accepts."""
return getattr(dns.query, "httpx2", None) or httpx
class TestEncryptedDnsNameservers(unittest.TestCase):
"""Tests for the DNS over HTTPS and DNS over TLS transports selected by
the form of each ``nameservers`` entry
(https://github.com/domainaware/parsedmarc/issues/880). The end-to-end
(https://github.com/domainaware/parsedmarc/issues/880). Most end-to-end
tests mock at the dnspython SDK boundary (``dns.query.https`` /
``dns.query.tls``) and assert on the answer query_dns parses back out of
a real DNS response message, so the whole chain — entry mapping,
resolver, nameserver object, transport call — is exercised."""
resolver, nameserver object, transport call — is exercised. The DoH
session regression test (issue #935) runs the real ``dns.query.https``
and fakes only the HTTP transport underneath it."""
def setUp(self):
# The DoH session is process-wide state; keep test ordering from
@@ -1424,25 +1744,31 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
self.addCleanup(restore)
@staticmethod
def _ptr_responder(captured: list, answer: str = "dns.example."):
def _ptr_response(request: dns.message.Message, answer: str = "dns.example."):
"""Builds a response to ``request`` answering its PTR question with
``answer``."""
response = dns.message.make_response(request)
# find_rrset(create=True) rather than appending to response.answer,
# so the message's rrset index is updated too — dnspython looks the
# answer up through that index.
rrset = response.find_rrset(
response.answer,
request.question[0].name,
dns.rdataclass.IN,
dns.rdatatype.PTR,
create=True,
)
rrset.add(dns.rdata.from_text("IN", "PTR", answer), 300)
return response
@classmethod
def _ptr_responder(cls, captured: list, answer: str = "dns.example."):
"""Builds a dns.query stand-in that answers any PTR query with
``answer``, recording the keyword arguments it was called with."""
def responder(request, *args, **kwargs):
captured.append(kwargs)
response = dns.message.make_response(request)
# find_rrset(create=True) rather than appending to
# response.answer, so the message's rrset index is updated too —
# dnspython looks the answer up through that index.
rrset = response.find_rrset(
response.answer,
request.question[0].name,
dns.rdataclass.IN,
dns.rdatatype.PTR,
create=True,
)
rrset.add(dns.rdata.from_text("IN", "PTR", answer), 300)
return response
return cls._ptr_response(request, answer)
return responder
@@ -1541,16 +1867,20 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
must fail at configuration time naming the entry, not parse
"successfully" with no TLS certificate identity and then fail at
query time with an opaque certificate error. Userinfo and query
components are rejected the same way."""
for entry in (
"tls://9.9.9.9/dns.quad9.net",
"tls://user@9.9.9.9",
"tls://9.9.9.9?hostname=dns.quad9.net",
components are rejected the same way, with the userinfo masked in
the message (GHSA-hjr3-rx4c-24cv)."""
for entry, shown in (
("tls://9.9.9.9/dns.quad9.net", "tls://9.9.9.9/dns.quad9.net"),
("tls://user@9.9.9.9", "tls://***@9.9.9.9"),
(
"tls://9.9.9.9?hostname=dns.quad9.net",
"tls://9.9.9.9?hostname=dns.quad9.net",
),
):
with self.subTest(entry=entry):
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils._nameservers_to_resolver_input([entry])
self.assertIn(entry, str(ctx.exception))
self.assertIn(shown, str(ctx.exception))
def test_unsplittable_entry_is_left_for_dnspython_to_reject(self):
"""urlsplit itself raises on some malformed input (an unbalanced
@@ -1563,10 +1893,12 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
with self.assertRaises(ValueError):
dns.resolver.Resolver(configure=False).nameservers = [entry]
def test_doh_query_returns_answers_through_a_shared_httpx_client(self):
def test_doh_query_returns_answers_through_a_shared_http_client(self):
"""An https:// nameserver resolves through dns.query.https, and the
session it is given is the module's shared client, an httpx.Client
with trust_env left on — httpx's documented switch for honoring
session it is given is the module's shared client, an instance of the
Client class of the HTTP library dnspython imported (httpx2 on
dnspython 2.9+, httpx before) with trust_env left on — the
documented switch, in both libraries, for honoring
HTTP_PROXY/HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE, which is what a
proxy-only network needs (issue #880). dnspython's stock DoH
nameserver passes no session at all, and the client dns.query.https
@@ -1586,7 +1918,7 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
self.assertEqual(records, ["dns.example"])
self.assertEqual(len(captured), 1)
session = captured[0]["session"]
self.assertIsInstance(session, httpx.Client)
self.assertIsInstance(session, _dnspython_http_module().Client)
self.assertIs(session.trust_env, True)
self.assertIs(session, parsedmarc.utils._DOH_SESSION)
@@ -1608,6 +1940,72 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
self.assertEqual(captured[0]["server_hostname"], "dns.quad9.net")
self.assertEqual(captured[0]["port"], 853)
def test_doh_session_is_accepted_by_the_real_dns_query_https(self):
"""Regression test for issue #935: dnspython 2.9 moved DoH to
httpx2, and ``dns.query.https()`` rejects a session of any other
class before sending anything (dnspython 2.9.1 ``dns/query.py``
line 551: ``if session and not isinstance(session, httpx2.Client):
raise ValueError("session parameter must be an httpx2.Client")``;
2.8.0 checks ``httpx.Client`` the same way at line 566). The old
``httpx.Client`` session therefore failed every DoH query under 2.9,
and the resolver retried it until the lifetime ran out. Here the real
``dns.query.https`` runs, and only the network is faked, by patching
the ``HTTPTransport`` of the library dnspython imported; a session
of the wrong class raises LifetimeTimeout instead of returning the
answer. The transport also observes the POSTed DNS query itself."""
http_module = _dnspython_http_module()
questions: list = []
def handle_request(transport, request):
query = dns.message.from_wire(request.read())
questions.append(query.question[0].to_text())
return http_module.Response(
200,
headers={"content-type": "application/dns-message"},
# An iterator rather than bytes: a bytes body counts as read
# already, so the client never closes the stream, and
# ``response.elapsed``, which dns.query.https reads, is set
# only on close
content=iter([self._ptr_response(query).to_wire()]),
request=request,
)
with patch.object(http_module.HTTPTransport, "handle_request", handle_request):
records = parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa",
"PTR",
nameservers=["https://dns.example/dns-query"],
timeout=0.5,
)
self.assertEqual(records, ["dns.example"])
self.assertEqual(questions, ["1.0.0.1.in-addr.arpa. IN PTR"])
def test_doh_client_class_prefers_httpx2_when_dnspython_imported_it(self):
"""dnspython 2.9+ binds ``dns.query.httpx2``; its Client is the one
``dns.query.https()`` accepts, even with httpx importable too."""
fake_httpx2 = SimpleNamespace(Client=type("Client", (), {}))
with patch.dict(dns.query.__dict__, {"httpx2": fake_httpx2, "httpx": httpx}):
self.assertIs(parsedmarc.utils._doh_client_class(), fake_httpx2.Client)
def test_doh_client_class_uses_httpx_when_dnspython_imported_it(self):
"""dnspython 2.7 and 2.8 bind ``dns.query.httpx`` and no httpx2, so
the class is ``httpx.Client`` — even if an unrelated httpx2 install
exists, since dnspython 2.8 never checks against it."""
fake_httpx = SimpleNamespace(Client=type("Client", (), {}))
with patch.dict(dns.query.__dict__, {"httpx": fake_httpx}):
dns.query.__dict__.pop("httpx2", None)
self.assertIs(parsedmarc.utils._doh_client_class(), fake_httpx.Client)
def test_doh_client_class_falls_back_to_httpx(self):
"""With neither library bound in dns.query (dnspython without its
[doh] extra), the class falls back to ``httpx.Client``; dnspython
(or httpx, if h2 is also missing) then raises its own error when a
DoH query is made."""
with patch.dict(dns.query.__dict__):
dns.query.__dict__.pop("httpx2", None)
dns.query.__dict__.pop("httpx", None)
self.assertIs(parsedmarc.utils._doh_client_class(), httpx.Client)
def test_doh_session_is_reused_within_a_process(self):
"""Consecutive calls hand back the same client, so DoH lookups
reuse the connection instead of renegotiating TLS per query."""
@@ -1626,6 +2024,152 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
self.assertEqual(parsedmarc.utils._DOH_SESSION_PID, os.getpid())
class TestNameserverCredentialsNotLeaked(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: nameserver URL credentials must not reach an
exception message, its printed traceback, or the log."""
def test_url_secrets_finds_the_password_and_its_port_like_prefix(self):
"""The secret is the userinfo after its first ":" (the whole
userinfo when there is none). A "/", "?" or "#" in it ends the URL
authority (RFC 3986 section 3.2), so urlsplit() reads the text
before it as the port and quotes it in its error; that prefix is a
secret too."""
cases = [
("https://doh:p4ss@dns.example/q", ["p4ss"]),
("https://doh:p4ss/w0rd@dns.example/q", ["p4ss/w0rd", "p4ss"]),
("https://doh:p4ss?w0rd@dns.example/q", ["p4ss?w0rd", "p4ss"]),
("https://doh:p4ss#w0rd@dns.example/q", ["p4ss#w0rd", "p4ss"]),
# The prefix ends at the first of the three delimiters
("https://doh:ab#cd/ef@dns.example/q", ["ab#cd/ef", "ab"]),
("https://doh:p:w@dns.example/q", ["p:w"]),
("https://doh:p@ss@dns.example/q", ["p@ss"]),
("https://S3CRETTOKEN@dns.example/q", ["S3CRETTOKEN"]),
("tls://dot:p4ss/w0rd@1.1.1.1", ["p4ss/w0rd", "p4ss"]),
# Empty secrets are never returned: str.replace("", ...) would
# insert the mask between every character
("https://doh:@dns.example/q", []),
("https://doh:/x@dns.example/q", ["/x"]),
("https://dns.example/dns-query", []),
("8.8.8.8", []),
("", []),
]
for value, expected in cases:
with self.subTest(value=value):
self.assertEqual(parsedmarc.utils._url_secrets(value), expected)
def test_mask_nameserver_secrets_masks_entries_passwords_and_prefixes(self):
"""Each configured entry is replaced by its redacted form, and each
secret in it wherever it still appears, including the bare port-like
prefix that urlsplit() and httpx quote."""
mask = parsedmarc.utils._mask_nameserver_secrets
url = "https://doh:p4ss/w0rd@dns.example/dns-query"
text = (
f"Server {url} answered x; "
"Port could not be cast to integer value as 'p4ss'; "
"Invalid port: 'p4ss/w0rd'"
)
self.assertEqual(
mask(text, [url, "8.8.8.8"]),
"Server https://***@dns.example/dns-query answered x; "
"Port could not be cast to integer value as '***'; "
"Invalid port: '***'",
)
self.assertEqual(
mask(
"Server https://u:PW123@h.example/q failed (PW123)",
["https://u:PW123@h.example/q"],
),
"Server https://u:***@h.example/q failed (***)",
)
self.assertEqual(mask("abc", None), "abc")
self.assertEqual(mask("abc", ["https://u:@h"]), "abc")
def test_mask_nameserver_secrets_handles_unsplittable_entries(self):
"""urlsplit() raises ValueError for an entry such as ``https://[x``;
the masking uses string methods only, so it still returns text."""
text = "Server https://[x failed; Server https://u:PW@[x failed"
self.assertEqual(
parsedmarc.utils._mask_nameserver_secrets(
text, ["https://[x", "https://u:PW@[x"]
),
"Server https://[x failed; Server https://u:***@[x failed",
)
def test_malformed_https_and_tls_ports_are_reported_without_secrets(self):
"""A password containing "/", "?" or "#" makes urlsplit() read its
start as the port, and its ValueError quotes that text; dnspython
2.9.1 raised it from ``DoHNameserver.answer_port()`` only after a
failed query (``dns/nameserver.py``). Both schemes are now rejected
before any query, naming the entry with its credentials masked and
with no chained exception repeating the urlsplit message."""
for entry, shown in (
(
"https://doh:p4ss/w0rd@dns.example/dns-query",
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
),
(
"https://doh:p4ss?w0rd@dns.example/dns-query",
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
),
(
"tls://dot:p4ss#w0rd@1.1.1.1",
"Invalid DNS over TLS nameserver tls://***@1.1.1.1",
),
(
"https://dns.example:notaport/dns-query",
"Invalid DNS over HTTPS nameserver https://dns.example:notaport/",
),
):
with self.subTest(entry=entry):
with patch("dns.query.https", side_effect=AssertionError):
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa",
"PTR",
nameservers=[entry],
timeout=0.2,
)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertIn(shown, str(ctx.exception))
self.assertIsNone(ctx.exception.__cause__)
self.assertIsNone(ctx.exception.__context__)
for secret in ("p4ss", "w0rd"):
self.assertNotIn(secret, trace)
def test_dnspython_rejection_of_an_entry_is_masked(self):
"""dnspython rejects an entry urlsplit() cannot split with a message
that quotes it as configured ("nameserver https://u:pw@[x is not a
dns.nameserver.Nameserver instance ..."); query_dns() masks it."""
entry = "https://doh:S3CRETPW@[dns.example/dns-query"
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa", "PTR", nameservers=[entry], timeout=0.2
)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertIn("https://doh:***@[dns.example/dns-query", str(ctx.exception))
self.assertNotIn("S3CRETPW", trace)
def test_get_reverse_dns_debug_log_masks_nameserver_credentials(self):
"""get_reverse_dns() logs the DNS error it swallows at debug level,
and LifetimeTimeout/NoNameservers quote each failing nameserver
URL, so the configured credentials are masked there too."""
url = "https://doh:S3CRETPW@dns.example/dns-query"
def fail(*args, **kwargs):
raise OSError(f"connection to {url} refused (S3CRETPW)")
with patch("dns.query.https", side_effect=fail):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
result = parsedmarc.utils.get_reverse_dns(
"1.0.0.1", nameservers=[url], timeout=0.2, retries=0
)
self.assertIsNone(result)
output = "\n".join(logs.output)
self.assertIn("get_reverse_dns(1.0.0.1) exception:", output)
self.assertIn("https://doh:***@dns.example/dns-query", output)
self.assertNotIn("S3CRETPW", output)
class TestLoadIpDb(unittest.TestCase):
"""Tests for the load_ip_db() download/cache/bundled fallback chain,
mocking at the httpx SDK boundary."""
+44
View File
@@ -114,6 +114,50 @@ class TestWebhookErrorHandling(unittest.TestCase):
client.save_smtp_tls_report_to_webhook('{"t": 1}')
class TestWebhookInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
any network I/O. The error is logged on every report."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@hooks.example.com/x", "p4ssw0rd"),
("https://admin:p4ss?w0rd@hooks.example.com/x", "p4ss"),
("https://admin:p4ss#w0rd@hooks.example.com/x", "p4ss"),
)
def test_invalid_url_is_logged_without_password(self):
saves = (
"save_aggregate_report_to_webhook",
"save_failure_report_to_webhook",
"save_smtp_tls_report_to_webhook",
)
for url, secret in self.MALFORMED:
client = WebhookClient(url, url, url)
for method in saves:
with self.subTest(url=url, method=method):
with self.assertLogs("parsedmarc.log", level="ERROR") as cm:
getattr(client, method)('{"a": 1}') # must not raise
self.assertEqual(
cm.output,
[
"ERROR:parsedmarc.log:Webhook Error: invalid webhook "
"URL (details withheld because they may include "
"credentials)"
],
)
self.assertNotIn(secret, "\n".join(cm.output))
def test_valid_url_is_the_one_posted(self):
url = "https://admin:p4ssw0rd@hooks.example.com/x"
client = WebhookClient(url, url, url)
client.session = MagicMock()
client.save_aggregate_report_to_webhook('{"a": 1}')
client.session.post.assert_called_once_with(url, content='{"a": 1}', timeout=60)
class TestWebhookClientClose(unittest.TestCase):
def test_close_closes_session(self):
client = _client()