mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-21 06:32:40 +00:00
Compare commits
3
Commits
768bf75480
...
48ea47adee
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
48ea47adee | ||
|
|
56e5a09923 | ||
|
|
70f96d0734 |
No files matched your search
@@ -135,6 +135,18 @@ Assume every regex in the package runs on content a remote sender controls (mess
|
||||
- **Time the pattern on an adversarial input before shipping it**, not just on the samples: a 1 MB string of the repeated opener with no closer, and a 200,000-character whitespace run. A linear pattern finishes in milliseconds; anything that takes seconds is a bug. Add that input as a time-bound regression test with a generous bound (seconds) and cite the measured old timing in the docstring, as the regression tests in `tests/test_init.py` and `tests/test_utils.py` do.
|
||||
- Python 3.10 is the floor, so possessive quantifiers and atomic groups (3.11+) are not available; fix the pattern's structure instead.
|
||||
|
||||
### Credentials must never reach logs or error text
|
||||
|
||||
Assume every configured URL, host list, connection string, and token carries a secret, and that every log line and exception message ends up on a console, in `log_file`, or in a pasted bug report (CWE-532). GHSA-hjr3-rx4c-24cv found leaks in `--debug` client-init lines, the forwarded httpx request line, download warnings, libpq's DSN echo, and httpx's `InvalidURL` text. Before writing a log call or an exception message that touches configuration:
|
||||
|
||||
- **Never log a configured value as-is.** Pass URLs and host lists through `redact_url_credentials()` (`parsedmarc/utils.py`), which masks userinfo (`https://user:***@host`), handles scheme-less `user:pw@host:port`, and, when the authority has no `@` but one follows it (a `/`, `?`, or `#` in the password, but also a credential-free URL or free text with a later `@`), masks through the last `@` (`https://***@host`). Redaction is log-only: the SDK or transport still gets the real value, so mask at the log call and not in `opts` or a client attribute.
|
||||
- **Treat third-party exception text as a leak until shown otherwise.** `httpx.HTTPStatusError` embeds the full request URL. `httpx.InvalidURL` quotes the component it rejected, so a password with an unencoded `/`, `?`, or `#` becomes `Invalid port: 'p4ssw0rd'`. libpq echoes the DSN for some malformed connection strings. Don't log `str(e)` from parsing, connecting to, or requesting a configured endpoint. Build the message from structured fields (`_describe_http_error()`: status code plus the redacted request URL), or use a fixed "details withheld because they may include credentials" message. Read the library source in the venv to check what its messages contain; don't assume.
|
||||
- **Raise the replacement `from None`.** `from e` and the implicit `__context__` both print the original exception when it is logged with `logger.exception` (the SIGHUP reload path does this) or when parsedmarc crashes, and `InvalidURL`'s own context is a `ValueError` that repeats the secret.
|
||||
- **Catch the right class.** `httpx.InvalidURL` is not an `httpx.HTTPError`, so an `except httpx.HTTPError` falls through to a generic handler that logs the raw text, or doesn't catch it at all and crashes startup.
|
||||
- **Check dependency loggers.** httpx logs every request URL at INFO. `_configure_dependency_logging()` (`parsedmarc/cli.py`) attaches `_RedactUrlCredentialsFilter` to the `httpx` and `httpx2` loggers (dnspython 2.9+ makes DNS over HTTPS requests through httpx2, and `httpx2.URL` is not an `httpx.URL`). A new dependency logger forwarded at `--verbose`/`--debug` needs the same check, and so does a new library that logs its endpoint.
|
||||
- **Test both halves, including malformed input.** Assert that the secret is absent from the captured log *and* from `"".join(traceback.format_exception(exc))`, and that the real credential still reaches the mocked transport. Include values the samples never contain: a password with `/`, `?`, or `#`; userinfo with no `:` (a bare token); and a scheme-less `user:pw@host:port`.
|
||||
- **Document the safe setting.** When an output has separate `user`/`username` and `password`, `token`, or `api_key` settings, its `usage.md` entry should recommend them over credentials in the URL.
|
||||
|
||||
## Testing standards
|
||||
|
||||
These rules govern *every* test added to `tests/`. They exist because the project has been burned by tests that looked like coverage but caught nothing, and by bug claims that turned out to be wrong about the spec. Both failure modes erode trust faster than missing coverage does.
|
||||
|
||||
@@ -10,6 +10,11 @@
|
||||
### Security
|
||||
|
||||
- **S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report** ([GHSA-7wqr-f6g9-cm29](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-7wqr-f6g9-cm29), affects 11.1.0 and earlier). `S3Client.save_report_to_s3()` put the sender-controlled report ID into the object key as-is, and sent `org_name` and the other metadata values as-is. A report ID such as `../../x` added path segments to the key, an ID could carry `\`, `%`, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit ([AWS, "Naming Amazon S3 objects"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html)). botocore rejects non-ASCII metadata with `ParamValidationError`, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise `HTTPClientError: Invalid header value` before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the `x-amz-meta-*` header; the 2 KB metadata limit ([AWS, "Working with object metadata"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)) was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are `/`, every AWS "character to avoid" (`` \ { } ^ [ ] " < > ~ # | ` `` and `%`), control characters, and non-ASCII characters, so `%` appears in a key only as the escape character. The punctuation real report IDs use, such as `:`, `@`, `.`, and `_`, is kept. Each metadata value is percent-encoded (`%`, control characters, and non-ASCII characters) so `urllib.parse.unquote()` reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in `_<hash>.json`, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
|
||||
- **Credentials in URLs and the IPinfo API token are no longer written to the log** ([GHSA-hjr3-rx4c-24cv](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-hjr3-rx4c-24cv), affects 11.1.0 and earlier). The `[elasticsearch]` and `[opensearch]` `hosts` settings and the `[splunk_hec]` `url` setting accept URLs, and a URL can carry credentials (`https://user:secret@host:9200`). With `--debug`, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to `log_file`. With `--verbose` or `--debug`, parsedmarc also forwards the `httpx` library's `HTTP Request:` line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in `[general]` `nameservers` (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the `ipinfo_api_token` was logged on every IPinfo lookup as the `token` query parameter. The password, or the whole userinfo when it has no `:`, is now replaced with `***` in these lines (`https://user:***@host:9200`), and so is the `token` query value. Credentials in a `[general]` `nameservers` entry are also masked in the DNS pre-flight error, and credentials in either that setting or a `--nameservers` entry are masked in the `--debug` reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable `https://` or `tls://` entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded `/`, `?`, or `#` (see below), is now rejected before any query is sent. The download URLs `ipinfo_url`, `reverse_dns_map_url`, and `psl_overrides_url` (`[general]`) can carry credentials too: they were logged as-is at `--debug` in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with `--warnings`, `--verbose`, or `--debug`) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by `***`. A malformed `[postgresql]` `connection_string` could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors (`[loki]` `url`, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded `/`, `?`, or `#` ends the URL's authority early ([RFC 3986, section 3.2](https://www.rfc-editor.org/rfc/rfc3986#section-3.2)), so httpx reads the text after the colon as the port and raises `InvalidURL("Invalid port: 'p4ssw0rd'")`, with a chained `ValueError` that repeats that part of the password. That text reached the log from `[loki]` `url` (at startup and on a SIGHUP reload), from `[splunk_hec]` `url` (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from `ipinfo_url` (in the "Failed to save IP database" warning). A malformed `psl_overrides_url` crashed startup with the traceback, because `InvalidURL` is not an `httpx.HTTPError`. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed `psl_overrides_url` now logs a warning and falls back to the `local_psl_overrides_path` file when that is set, or else to the bundled file, and a malformed `reverse_dns_map_url` or `ipinfo_url` is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no `@` in its authority, so the `--debug` "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last `@` (`https://***@host:3100`). A URL with no credentials but an `@` in its path or query is masked the same way in these lines (`https://host/path@file` is logged as `https://***@file`). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: `user` and `password` (or `api_key`) for `[elasticsearch]` and `[opensearch]`, `user` and `password` for `[postgresql]`, `username` and `password` (or `token`) for `[loki]`, and `token` for `[splunk_hec]`. Reported by helixnebulatech.
|
||||
|
||||
### New features
|
||||
|
||||
- **Elasticsearch and OpenSearch output can now write to data streams** ([#842](https://github.com/domainaware/parsedmarc/issues/842), relates to [#476](https://github.com/domainaware/parsedmarc/issues/476)). The new `use_data_streams` option in `[elasticsearch]` and `[opensearch]` (default `False`) saves each report type to one data stream named like today's indexes without the date suffix (`dmarc_aggregate`, `dmarc_failure`, `smtp_tls`, with `index_prefix`, `index_suffix`, or the `index_prefix_domain_map` prefix applied as before), instead of creating an index per day or month. That gives each report type one stable name, keeps the shard count down, and lets the cluster roll over and delete old data. parsedmarc puts an index template named after each stream, whose `index_patterns` is the exact stream name so it never applies to the dated indexes, and creates the stream; it does not create or manage retention policies. On Elasticsearch, attach an ILM policy through an optional `<stream>@custom` component template, which the template includes; on OpenSearch, an ISM policy attaches itself through its `ism_template`. `docs/source/elasticsearch.md` and `docs/source/opensearch.md` have example policies. Existing dated indexes are left alone and keep matching the dashboards' `dmarc_aggregate*`, `dmarc_f*`, and `smtp_tls*` patterns next to the stream, and the duplicate check searches both, so a report saved before the switch is not saved again after it. `monthly_indexes` is ignored when `use_data_streams` is on, and a warning says so. If a concrete index or alias already holds a stream's name, as an undated index from a release before 5.0.0 might, saving fails with an error that says so instead of writing into it. Every saved document now also carries an `@timestamp` field, in both modes: each record's begin date (`date_begin`) for aggregate reports, which is the report's begin date unless parsedmarc split a report spanning more than a day into daily records; the arrival date for failure reports; and the report's begin date for SMTP TLS reports. These are the same dates that pick the dated index. Documents are now saved with `op_type=create`, which data streams require; without a document ID this is the same as the default for dated indexes.
|
||||
|
||||
### Changes
|
||||
|
||||
@@ -18,6 +23,7 @@
|
||||
### Bug fixes
|
||||
|
||||
- **Failure reports are now saved to S3.** `save_report_to_s3()` read `report["report_metadata"]` for every report type except SMTP TLS, but failure reports have no `report_metadata` (see `FailureReport` in `parsedmarc/types.py`), so every failure report raised `KeyError` since S3 output was added in 7.0.0. The error was counted as an S3 output error, so the batch holding the report was held back. A failure report is now saved under `<path>/failure/year=YYYY/month=MM/day=DD/<reported domain>_<hash>.json`, dated by its UTC arrival date, with `reported_domain` and `arrival_date` metadata. Its hash covers the report type, the arrival date, reported domain, original mail from, original recipient, original envelope ID, source IP address, and the message sample.
|
||||
- **DNS over HTTPS works again under dnspython 2.9** ([#935](https://github.com/domainaware/parsedmarc/issues/935)). dnspython 2.9.0 moved DNS over HTTPS from httpx to httpx2, and `dns.query.https()` now raises `ValueError: session parameter must be an httpx2.Client` when it is given an `httpx.Client`. parsedmarc always passed an `httpx.Client`, so every `https://` nameserver failed, and the resolver retried it until the lookup timed out. The shared DoH client is now built from the HTTP library dnspython itself imported: httpx2 on dnspython 2.9+, httpx on 2.7 and 2.8. The `nameservers` pre-flight check also reported that failure as `no PTR record for 1.1.1.1`, because `get_reverse_dns()` returns `None` for every DNS error. The check now queries the PTR record directly, so it reports `no PTR record` only when the nameserver answers that the record does not exist, and otherwise shows the resolver's actual error. The `httpx2` and `httpcore2` loggers are now included in `--debug` dependency logging, since dnspython 2.9 DoH traffic is logged under those names. An invalid `dns_test_address` is now reported by name in the pre-flight error. Reported by florianutz.
|
||||
|
||||
- **Keep distinct aggregate reports when one reporter reuses a Report-ID for different policy domains.** RFC 9990 §3.5.1 scopes Report-ID uniqueness to the same domain. Direct report files, mbox files, and mailbox ingestion now all include the policy domain in one shared dedup key, using a JSON array to avoid ambiguous field separators. ASCII domain case differences still count as the same domain (RFC 4343 §3), without changing the returned report. Same-domain repeats remain suppressed — including a report passed both as a file and inside an mbox in one CLI run — and mailbox keys are still committed only after a successful save callback.
|
||||
|
||||
|
||||
@@ -405,13 +405,170 @@ curl -X POST "http://localhost:9200/smtp_tls*/_update_by_query?conflicts=proceed
|
||||
It works identically on OpenSearch; just adjust the URL and credentials, same
|
||||
as the aggregate command above.
|
||||
|
||||
## Records retention
|
||||
## Data streams and records retention
|
||||
|
||||
Starting in version 5.0.0, `parsedmarc` stores data in a separate
|
||||
index for each day to make it easy to comply with records
|
||||
retention regulations such as GDPR. For more information,
|
||||
index for each day (or each month, with `monthly_indexes`), named after
|
||||
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
|
||||
easy to comply with records retention regulations such as GDPR: delete
|
||||
the indexes for the dates you no longer keep. For more information,
|
||||
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
|
||||
|
||||
Set `use_data_streams = True` in the `[elasticsearch]` section to write
|
||||
to [data streams] instead. Each report type then goes to one data stream
|
||||
named like the indexes without the date: `dmarc_aggregate`,
|
||||
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
|
||||
the `index_prefix_domain_map` prefix applied as usual. The cluster
|
||||
creates the stream's backing indices (`.ds-dmarc_aggregate-…`), and an
|
||||
ILM policy you attach rolls them over and deletes old ones, so there is
|
||||
one stable name per report type and far fewer shards.
|
||||
|
||||
Before a parsedmarc process first saves to a stream, it puts an index
|
||||
template named after the stream and creates the stream if it does not
|
||||
exist yet. The template:
|
||||
|
||||
- matches exactly the stream's name, with no wildcard, so it never applies
|
||||
to the dated indexes;
|
||||
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
|
||||
from the configuration, for each new backing index;
|
||||
- maps `@timestamp` as a date. Every document parsedmarc saves carries
|
||||
`@timestamp`: each record's begin date (`date_begin`) for aggregate
|
||||
reports, which is the report's begin date unless parsedmarc split a
|
||||
report spanning more than a day into daily records; the arrival date
|
||||
for failure reports; and the report's begin date for SMTP TLS reports.
|
||||
The other fields are mapped dynamically, as in the dated indexes;
|
||||
- includes a component template named `<stream>@custom`, such as
|
||||
`dmarc_aggregate@custom`, if one exists.
|
||||
|
||||
Data streams need Elasticsearch 8.7 or later, because the template uses
|
||||
`ignore_missing_component_templates` to make the `@custom` component
|
||||
template optional.
|
||||
|
||||
parsedmarc puts the template again in every process, so changes made to
|
||||
it directly do not last. Put your own settings, such as the ILM policy,
|
||||
in the `@custom` component template instead.
|
||||
|
||||
parsedmarc does not create or change any ILM policy. Here is an example
|
||||
that rolls each stream over every 30 days, or sooner when a primary
|
||||
shard reaches 50 GB, and deletes a backing index 180 days after its
|
||||
rollover:
|
||||
|
||||
```text
|
||||
PUT _ilm/policy/parsedmarc
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover": {
|
||||
"max_age": "30d",
|
||||
"max_primary_shard_size": "50gb"
|
||||
}
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "180d",
|
||||
"actions": {
|
||||
"delete": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Attach it to each stream through its `@custom` component template:
|
||||
|
||||
```text
|
||||
PUT _component_template/dmarc_aggregate@custom
|
||||
{
|
||||
"template": {
|
||||
"settings": {
|
||||
"index.lifecycle.name": "parsedmarc"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Repeat that for `dmarc_failure@custom` and `smtp_tls@custom`, and for the
|
||||
prefixed or suffixed names if you use them. The component template
|
||||
applies only to backing indices created after it. A stream that already
|
||||
exists has a write index with no policy, so nothing would ever roll it
|
||||
over; attach the policy to its existing backing indices as well:
|
||||
|
||||
```text
|
||||
PUT dmarc_aggregate/_settings
|
||||
{
|
||||
"index.lifecycle.name": "parsedmarc"
|
||||
}
|
||||
```
|
||||
|
||||
The delete phase's `min_age` counts from the rollover, not from the
|
||||
report dates, so a backing index is deleted 180 days after it stopped
|
||||
receiving documents, whatever dates the reports in it cover. If you need
|
||||
retention by report date, keep using dated indexes and delete them by
|
||||
name.
|
||||
|
||||
On Elastic Cloud Serverless, which has no ILM, use a
|
||||
[data stream lifecycle] instead:
|
||||
|
||||
```text
|
||||
PUT _data_stream/dmarc_aggregate/_lifecycle
|
||||
{
|
||||
"data_retention": "180d"
|
||||
}
|
||||
```
|
||||
|
||||
### Switching an existing installation to data streams
|
||||
|
||||
Dated indexes that already exist are left alone. They keep working next
|
||||
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
|
||||
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
|
||||
check for duplicate reports searches both, so a report that is already
|
||||
in a dated index is not saved again into a stream. You can delete the
|
||||
dated indexes when they age out of your retention period, or move their
|
||||
documents into the streams. Documents saved by earlier releases have no
|
||||
`@timestamp`, which a data stream requires, so add it while reindexing.
|
||||
Run parsedmarc once with `use_data_streams = True` first, so the stream
|
||||
and its template exist; a reindex into a name that is not a data stream
|
||||
yet creates an ordinary index of that name instead:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate-*"},
|
||||
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
|
||||
`dmarc_failure-*`. Delete the dated indexes once the reindex has
|
||||
finished; until then, the dashboards count every document twice.
|
||||
|
||||
If saving fails with `cannot create the data stream dmarc_aggregate: an
|
||||
index or alias with that name already exists`, an index or alias already
|
||||
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
|
||||
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
|
||||
write into such an index rather than mixing new documents into it. Move
|
||||
its documents to a dated name, which the
|
||||
dashboards still match, and delete it:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate"},
|
||||
"dest": {"index": "dmarc_aggregate-legacy"}
|
||||
}
|
||||
|
||||
DELETE dmarc_aggregate
|
||||
```
|
||||
|
||||
[data stream lifecycle]: https://www.elastic.co/docs/manage-data/lifecycle/data-stream
|
||||
[data streams]: https://www.elastic.co/docs/manage-data/data-store/data-streams
|
||||
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
|
||||
[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
|
||||
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
|
||||
|
||||
+150
-3
@@ -7,8 +7,155 @@ To set up visual dashboards of DMARC data, install OpenSearch and Grafana.
|
||||
OpenSearch: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/
|
||||
Grafana: https://grafana.com/docs/grafana/latest/setup-grafana/installation/
|
||||
|
||||
## Records retention
|
||||
## Data streams and records retention
|
||||
|
||||
Starting in version 5.0.0, `parsedmarc` stores data in a separate
|
||||
index for each day to make it easy to comply with records
|
||||
retention regulations such as GDPR.
|
||||
index for each day (or each month, with `monthly_indexes`), named after
|
||||
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
|
||||
easy to comply with records retention regulations such as GDPR: delete
|
||||
the indexes for the dates you no longer keep.
|
||||
|
||||
Set `use_data_streams = True` in the `[opensearch]` section to write to
|
||||
[data streams] instead. Each report type then goes to one data stream
|
||||
named like the indexes without the date: `dmarc_aggregate`,
|
||||
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
|
||||
the `index_prefix_domain_map` prefix applied as usual. The cluster
|
||||
creates the stream's backing indices (`.ds-dmarc_aggregate-000001`, …),
|
||||
and an [ISM] policy you attach rolls them over and deletes old data, so
|
||||
there is one stable name per report type and far fewer shards. Amazon
|
||||
OpenSearch Serverless (`aws_service = aoss`) has no data streams or
|
||||
ISM, so leave `use_data_streams` off there.
|
||||
|
||||
Before a parsedmarc process first saves to a stream, it puts an index
|
||||
template named after the stream and creates the stream if it does not
|
||||
exist yet. The template:
|
||||
|
||||
- matches exactly the stream's name, with no wildcard, so it never applies
|
||||
to the dated indexes;
|
||||
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
|
||||
from the configuration, for each new backing index;
|
||||
- maps `@timestamp` as a date. Every document parsedmarc saves carries
|
||||
`@timestamp`: each record's begin date (`date_begin`) for aggregate
|
||||
reports, which is the report's begin date unless parsedmarc split a
|
||||
report spanning more than a day into daily records; the arrival date
|
||||
for failure reports; and the report's begin date for SMTP TLS reports.
|
||||
The other fields are mapped dynamically, as in the dated indexes.
|
||||
|
||||
parsedmarc puts the template again in every process, so changes made to
|
||||
it directly do not last.
|
||||
|
||||
parsedmarc does not create or change any ISM policy. Here is an example
|
||||
that rolls each stream over every 30 days, or sooner when a primary
|
||||
shard reaches 50 GB, and deletes a backing index 180 days after it was
|
||||
created. Its `ism_template` attaches it to every backing index the
|
||||
listed streams create from now on. List the stream names themselves,
|
||||
including any prefix or suffix: a `.ds-dmarc_aggregate-*` pattern does
|
||||
not match the backing indices.
|
||||
|
||||
```text
|
||||
PUT _plugins/_ism/policies/parsedmarc
|
||||
{
|
||||
"policy": {
|
||||
"description": "parsedmarc data stream retention",
|
||||
"default_state": "hot",
|
||||
"states": [
|
||||
{
|
||||
"name": "hot",
|
||||
"actions": [
|
||||
{
|
||||
"rollover": {
|
||||
"min_index_age": "30d",
|
||||
"min_primary_shard_size": "50gb"
|
||||
}
|
||||
}
|
||||
],
|
||||
"transitions": [
|
||||
{
|
||||
"state_name": "delete",
|
||||
"conditions": {"min_index_age": "180d"}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "delete",
|
||||
"actions": [{"delete": {}}],
|
||||
"transitions": []
|
||||
}
|
||||
],
|
||||
"ism_template": [
|
||||
{
|
||||
"index_patterns": ["dmarc_aggregate", "dmarc_failure", "smtp_tls"],
|
||||
"priority": 100
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `ism_template` only applies to backing indices created after the
|
||||
policy. To attach the policy to the backing indices of a stream that
|
||||
already exists:
|
||||
|
||||
```text
|
||||
POST _plugins/_ism/add/dmarc_aggregate
|
||||
{
|
||||
"policy_id": "parsedmarc"
|
||||
}
|
||||
```
|
||||
|
||||
`min_index_age` counts from when the backing index was created, not
|
||||
from the report dates, so a backing index is deleted 180 days after it
|
||||
was created, whatever dates the reports in it cover. If you need
|
||||
retention by report date, keep using dated indexes and delete them by
|
||||
name.
|
||||
|
||||
### Switching an existing installation to data streams
|
||||
|
||||
Dated indexes that already exist are left alone. They keep working next
|
||||
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
|
||||
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
|
||||
check for duplicate reports searches both, so a report that is already
|
||||
in a dated index is not saved again into a stream. You can delete the
|
||||
dated indexes when they age out of your retention period, or move their
|
||||
documents into the streams. Documents saved by earlier releases have no
|
||||
`@timestamp`, which a data stream requires, so add it while reindexing.
|
||||
Run parsedmarc once with `use_data_streams = True` first, so the stream
|
||||
and its template exist; a reindex into a name that is not a data stream
|
||||
yet creates an ordinary index of that name instead:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate-*"},
|
||||
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
|
||||
`dmarc_failure-*`. Delete the dated indexes once the reindex has
|
||||
finished; until then, the dashboards count every document twice.
|
||||
|
||||
If saving fails with `cannot create the data stream dmarc_aggregate: an
|
||||
index or alias with that name already exists`, an index or alias already
|
||||
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
|
||||
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
|
||||
write into such an index rather than mixing new documents into it. Move
|
||||
its documents to a dated name, which the
|
||||
dashboards still match, and delete it:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate"},
|
||||
"dest": {"index": "dmarc_aggregate-legacy"}
|
||||
}
|
||||
|
||||
DELETE dmarc_aggregate
|
||||
```
|
||||
|
||||
[data streams]: https://docs.opensearch.org/latest/im-plugin/data-streams/
|
||||
[ISM]: https://docs.opensearch.org/latest/im-plugin/ism/index/
|
||||
+54
-29
@@ -91,7 +91,9 @@ hosts = 127.0.0.1:9200
|
||||
ssl = False
|
||||
|
||||
[opensearch]
|
||||
hosts = https://admin:admin@127.0.0.1:9200
|
||||
hosts = https://127.0.0.1:9200
|
||||
user = admin
|
||||
password = PasswordGoesHere
|
||||
ssl = True
|
||||
|
||||
[splunk_hec]
|
||||
@@ -571,14 +573,10 @@ The full set of configuration options is:
|
||||
|
||||
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
|
||||
:::{note}
|
||||
Special characters in the username or password must be
|
||||
[URL encoded].
|
||||
:::
|
||||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||||
Do not put credentials in the URL; use `user` and `password`
|
||||
(or `api_key`) instead.
|
||||
- `user` - str: Basic auth username
|
||||
- `password` - str: Basic auth password
|
||||
- `api_key` - str: API key
|
||||
@@ -591,28 +589,35 @@ The full set of configuration options is:
|
||||
- `index_suffix` - str: A suffix to apply to the index names
|
||||
- `index_prefix` - str: A prefix to apply to the index names
|
||||
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
|
||||
(ignored, with a warning, when `use_data_streams` is `True`)
|
||||
- `number_of_shards` - int: The number of shards to use when
|
||||
creating the index (Default: `1`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `1`)
|
||||
- `number_of_replicas` - int: The number of replicas to use when
|
||||
creating the index (Default: `0`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `0`)
|
||||
- `use_data_streams` - bool: Save each report type to a data stream
|
||||
named like the index without the date suffix (e.g. `dmarc_aggregate`)
|
||||
instead of to daily or monthly indexes. parsedmarc creates the stream
|
||||
and its index template; retention is left to an ILM policy you attach. Requires Elasticsearch
|
||||
8.7 or later.
|
||||
See [Data streams and records retention](elasticsearch.md#data-streams-and-records-retention)
|
||||
(Default: `False`)
|
||||
- `serverless` - bool: Set to `True` when targeting an Elastic Cloud
|
||||
Serverless project. Serverless manages sharding and replication itself
|
||||
and rejects the `number_of_shards` / `number_of_replicas` index settings
|
||||
with HTTP 400. With this flag set, parsedmarc strips those keys from the
|
||||
settings sent at index creation; any other settings (e.g.
|
||||
`refresh_interval`) are passed through unchanged (Default: `False`)
|
||||
settings sent at index creation (or in a data stream's index template);
|
||||
any other settings (e.g. `refresh_interval`) are passed through
|
||||
unchanged (Default: `False`)
|
||||
- `opensearch`
|
||||
|
||||
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
|
||||
:::{note}
|
||||
Special characters in the username or password must be
|
||||
[URL encoded].
|
||||
:::
|
||||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||||
Do not put credentials in the URL; use `user` and `password`
|
||||
(or `api_key`) instead.
|
||||
- `user` - str: Basic auth username
|
||||
- `password` - str: Basic auth password
|
||||
- `api_key` - str: API key
|
||||
@@ -629,12 +634,24 @@ The full set of configuration options is:
|
||||
- `index_suffix` - str: A suffix to apply to the index names
|
||||
- `index_prefix` - str: A prefix to apply to the index names
|
||||
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
|
||||
(ignored, with a warning, when `use_data_streams` is `True`)
|
||||
- `number_of_shards` - int: The number of shards to use when
|
||||
creating the index (Default: `1`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `1`)
|
||||
- `number_of_replicas` - int: The number of replicas to use when
|
||||
creating the index (Default: `0`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `0`)
|
||||
- `use_data_streams` - bool: Save each report type to a data stream
|
||||
named like the index without the date suffix (e.g. `dmarc_aggregate`)
|
||||
instead of to daily or monthly indexes. parsedmarc creates the stream
|
||||
and its index template; retention is left to an ISM policy you attach.
|
||||
OpenSearch rejects data stream names that start with `.`, so an
|
||||
`index_prefix` starting with `.` cannot be used with this option.
|
||||
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
|
||||
(Default: `False`)
|
||||
- `splunk_hec`
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
|
||||
Do not put credentials in the URL; use `token` instead.
|
||||
- `token` - str: The HEC token
|
||||
- `index` - str: The Splunk index to use
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
@@ -693,8 +710,13 @@ The full set of configuration options is:
|
||||
- `password` - str: The database user password (Optional)
|
||||
- `database` - str: The database name (Optional)
|
||||
- `connection_string` - str: A full libpq connection string or URI
|
||||
(e.g. `postgresql://user:pass@host/dbname`). When provided,
|
||||
all individual parameters above are ignored.
|
||||
(e.g. `postgresql://user@host/dbname`). When provided,
|
||||
all individual parameters above are ignored. Prefer the separate
|
||||
`host`, `port`, `user`, `password`, and `database` settings and do
|
||||
not put the password in the connection string: a malformed
|
||||
connection string can make the driver echo it back in an error
|
||||
message. parsedmarc replaces such errors with a fixed message in
|
||||
its log, but cannot stop the driver from producing them.
|
||||
|
||||
The PostgreSQL backend is an optional extra. Install it with
|
||||
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
|
||||
@@ -721,11 +743,13 @@ The full set of configuration options is:
|
||||
database = parsedmarc
|
||||
```
|
||||
|
||||
Or using a DSN/URI:
|
||||
Or using a DSN/URI, with the password supplied separately (for
|
||||
example through the `PGPASSWORD` environment variable or a
|
||||
`~/.pgpass` file, both of which libpq reads):
|
||||
|
||||
```ini
|
||||
[postgresql]
|
||||
connection_string = postgresql://parsedmarc:secret@localhost/parsedmarc
|
||||
connection_string = postgresql://parsedmarc@localhost/parsedmarc
|
||||
```
|
||||
|
||||
Saving parsed data to PostgreSQL is controlled by the `[general]`
|
||||
@@ -932,6 +956,8 @@ The full set of configuration options is:
|
||||
It expects the default `job=parsedmarc` label and `label_fields`.
|
||||
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
|
||||
URL has no path. A URL with a path is used as is.
|
||||
Do not put credentials in the URL; use `username` and `password`
|
||||
(or `token`) instead.
|
||||
- `labels` - str: Comma-separated `name=value` stream labels, merged over
|
||||
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
|
||||
(in a container, the FQDN is the container's hostname)
|
||||
@@ -1546,5 +1572,4 @@ journalctl -u parsedmarc.service -r
|
||||
:::
|
||||
|
||||
[cloudflare's public resolvers]: https://1.1.1.1/
|
||||
[url encoded]: https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
|
||||
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api
|
||||
+148
-19
@@ -19,6 +19,10 @@ from glob import escape as glob_escape, glob
|
||||
from ssl import CERT_NONE, create_default_context
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import dns.exception
|
||||
import dns.query
|
||||
import dns.resolver
|
||||
import dns.reversename
|
||||
import httpx
|
||||
import yaml
|
||||
from tqdm import tqdm
|
||||
@@ -127,14 +131,16 @@ import parsedmarc.utils
|
||||
from parsedmarc.utils import (
|
||||
InvalidIPinfoAPIKey,
|
||||
ReverseDNSMap,
|
||||
_mask_nameserver_secrets,
|
||||
configure_ipinfo_api,
|
||||
get_base_domain,
|
||||
get_reverse_dns,
|
||||
human_timestamp_to_datetime,
|
||||
is_mbox,
|
||||
load_ip_db,
|
||||
load_psl_overrides,
|
||||
load_reverse_dns_map,
|
||||
query_dns,
|
||||
redact_url_credentials,
|
||||
sanitize_for_log,
|
||||
)
|
||||
|
||||
@@ -674,9 +680,66 @@ _DEPENDENCY_LOGGERS = (
|
||||
"msgraph",
|
||||
"httpx",
|
||||
"httpcore",
|
||||
# dnspython 2.9+ does DNS over HTTPS through httpx2 / httpcore2, which
|
||||
# log under their own names rather than httpx / httpcore.
|
||||
"httpx2",
|
||||
"httpcore2",
|
||||
)
|
||||
|
||||
|
||||
# URL classes whose instances appear as log-record arguments. dnspython
|
||||
# 2.9+ does DNS over HTTPS through httpx2, whose URL is not an httpx.URL;
|
||||
# like _doh_client_class() in utils.py, take the module dnspython itself
|
||||
# imported (absent under dnspython 2.7/2.8, which use httpx).
|
||||
_HTTP_URL_CLASSES: tuple[type, ...] = (httpx.URL,) + tuple(
|
||||
module.URL for module in (getattr(dns.query, "httpx2", None),) if module
|
||||
)
|
||||
|
||||
|
||||
class _RedactUrlCredentialsFilter(logging.Filter):
|
||||
"""Masks URL credentials and the IPinfo ``token`` query parameter in
|
||||
the arguments of httpx and httpx2 log records.
|
||||
|
||||
httpx and httpx2 log ``HTTP Request: %s %s "%s %d %s"`` with
|
||||
``request.url``, whose string form includes the userinfo password and
|
||||
the query string. httpx2 is what dnspython 2.9+ uses for DNS over HTTPS,
|
||||
so a ``[general]`` ``nameservers`` URL with credentials reaches this
|
||||
line on every lookup.
|
||||
The filter never drops a record; the redaction itself is total (string
|
||||
methods and URL parameter lookups only), so it cannot raise.
|
||||
|
||||
Logger-level filters apply only to records created on the logger the
|
||||
filter is attached to, not to records from its child loggers. httpx
|
||||
(0.28) logs only via ``logging.getLogger("httpx")`` itself and httpx2
|
||||
only via ``logging.getLogger("httpx2")`` (``httpx2/_client.py``), so
|
||||
attaching the filter to those two loggers covers them; a version that
|
||||
logged on a child logger (``httpx.something``) would bypass it.
|
||||
"""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
if isinstance(record.args, tuple):
|
||||
record.args = tuple(self._redact(arg) for arg in record.args)
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _redact(arg: Any) -> Any:
|
||||
if isinstance(arg, _HTTP_URL_CLASSES):
|
||||
if arg.params.get("token") is not None:
|
||||
# Drop the token and append the mask as typed: setting it
|
||||
# with copy_set_param() would percent-encode the "*"s, and
|
||||
# rewriting the encoded form afterwards would also touch a
|
||||
# literal "%2A%2A%2A" elsewhere in the URL.
|
||||
text = str(arg.copy_remove_param("token"))
|
||||
text, fragment_sep, fragment = text.partition("#")
|
||||
separator = "&" if "?" in text else "?"
|
||||
text = f"{text}{separator}token=***{fragment_sep}{fragment}"
|
||||
return redact_url_credentials(text)
|
||||
return redact_url_credentials(str(arg))
|
||||
if isinstance(arg, str):
|
||||
return redact_url_credentials(arg)
|
||||
return arg
|
||||
|
||||
|
||||
def _configure_dependency_logging(level: int) -> None:
|
||||
"""Apply parsedmarc's logging verbosity to dependency loggers.
|
||||
|
||||
@@ -690,6 +753,11 @@ def _configure_dependency_logging(level: int) -> None:
|
||||
duplicates output nor keeps writing to a closed handler. Propagation
|
||||
to the root logger is disabled so that a stray ``logging.basicConfig()``
|
||||
anywhere in the process cannot double-print every dependency record.
|
||||
The ``httpx`` and ``httpx2`` loggers also get a
|
||||
:class:`_RedactUrlCredentialsFilter`
|
||||
(installed once, so reloads do not stack copies) that masks URL
|
||||
credentials and the IPinfo ``token`` query parameter in its
|
||||
``HTTP Request:`` records.
|
||||
CLI-only: library consumers configure logging themselves.
|
||||
"""
|
||||
dep_level = min(level, logging.WARNING)
|
||||
@@ -703,6 +771,12 @@ def _configure_dependency_logging(level: int) -> None:
|
||||
for wanted in logger.handlers:
|
||||
if wanted not in dep_logger.handlers:
|
||||
dep_logger.addHandler(wanted)
|
||||
for name in ("httpx", "httpx2"):
|
||||
http_logger = logging.getLogger(name)
|
||||
if not any(
|
||||
isinstance(f, _RedactUrlCredentialsFilter) for f in http_logger.filters
|
||||
):
|
||||
http_logger.addFilter(_RedactUrlCredentialsFilter())
|
||||
|
||||
|
||||
def _load_config(config_file: str | None = None) -> ConfigParser:
|
||||
@@ -819,22 +893,36 @@ def _parse_config(config: ConfigParser, opts):
|
||||
opts.dns_test_address = general_config["dns_test_address"]
|
||||
if "nameservers" in general_config:
|
||||
opts.nameservers = _str_to_list(general_config["nameservers"])
|
||||
# nameservers pre-flight check
|
||||
dummy_hostname = None
|
||||
# nameservers pre-flight check. query_dns() is called directly
|
||||
# rather than get_reverse_dns(), which swallows every DNS error
|
||||
# and returns None, so a broken transport (e.g. every DoH query
|
||||
# failing, https://github.com/domainaware/parsedmarc/issues/935)
|
||||
# was reported as a missing PTR record.
|
||||
try:
|
||||
dummy_hostname = get_reverse_dns(
|
||||
opts.dns_test_address,
|
||||
reverse_name = str(dns.reversename.from_address(opts.dns_test_address))
|
||||
except dns.exception.SyntaxError as addr_error:
|
||||
raise ConfigurationError(
|
||||
f"DNS pre-flight check failed: dns_test_address "
|
||||
f"{opts.dns_test_address!r} is not a valid IP address"
|
||||
) from addr_error
|
||||
try:
|
||||
query_dns(
|
||||
reverse_name,
|
||||
"PTR",
|
||||
nameservers=opts.nameservers,
|
||||
timeout=opts.dns_timeout,
|
||||
)
|
||||
except Exception as ns_error:
|
||||
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as ns_error:
|
||||
raise ConfigurationError(
|
||||
f"DNS pre-flight check failed: {ns_error}"
|
||||
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {redact_url_credentials(opts.nameservers or [])}: {ns_error}"
|
||||
) from ns_error
|
||||
if not dummy_hostname:
|
||||
except Exception as ns_error:
|
||||
# A DNS over HTTPS error can quote the request URL, which
|
||||
# carries the configured nameserver's credentials.
|
||||
raise ConfigurationError(
|
||||
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {opts.nameservers}"
|
||||
)
|
||||
f"DNS pre-flight check failed: {type(ns_error).__name__}: "
|
||||
f"{_mask_nameserver_secrets(str(ns_error), opts.nameservers)}"
|
||||
) from None
|
||||
if "save_aggregate" in general_config:
|
||||
opts.save_aggregate = bool(general_config.getboolean("save_aggregate"))
|
||||
if "save_failure" in general_config:
|
||||
@@ -1175,6 +1263,17 @@ def _parse_config(config: ConfigParser, opts):
|
||||
opts.elasticsearch_serverless = elasticsearch_config.getboolean(
|
||||
"serverless"
|
||||
)
|
||||
if "use_data_streams" in elasticsearch_config:
|
||||
opts.elasticsearch_use_data_streams = bool(
|
||||
elasticsearch_config.getboolean("use_data_streams")
|
||||
)
|
||||
if getattr(opts, "elasticsearch_use_data_streams", False) and getattr(
|
||||
opts, "elasticsearch_monthly_indexes", False
|
||||
):
|
||||
logger.warning(
|
||||
"[elasticsearch] monthly_indexes is ignored because "
|
||||
"use_data_streams is enabled"
|
||||
)
|
||||
|
||||
if "opensearch" in config:
|
||||
opensearch_config = config["opensearch"]
|
||||
@@ -1228,6 +1327,17 @@ def _parse_config(config: ConfigParser, opts):
|
||||
opts.opensearch_aws_region = opensearch_config["aws_region"].strip()
|
||||
if "aws_service" in opensearch_config:
|
||||
opts.opensearch_aws_service = opensearch_config["aws_service"].strip()
|
||||
if "use_data_streams" in opensearch_config:
|
||||
opts.opensearch_use_data_streams = bool(
|
||||
opensearch_config.getboolean("use_data_streams")
|
||||
)
|
||||
if getattr(opts, "opensearch_use_data_streams", False) and getattr(
|
||||
opts, "opensearch_monthly_indexes", False
|
||||
):
|
||||
logger.warning(
|
||||
"[opensearch] monthly_indexes is ignored because "
|
||||
"use_data_streams is enabled"
|
||||
)
|
||||
|
||||
if "splunk_hec" in config.sections():
|
||||
hec_config = config["splunk_hec"]
|
||||
@@ -1795,9 +1905,12 @@ def _search_alias_snapshot() -> list[tuple[Any, Any, Any]]:
|
||||
# private is the same liberty this function already takes with
|
||||
# ``connections``; without it, a failed reload that flipped
|
||||
# ``[elasticsearch] serverless`` would pair the restored old client
|
||||
# with the new config's flag. opensearch.py declares no module
|
||||
# globals at all (no ``global`` statement in the file), so there is
|
||||
# nothing to pair with it.
|
||||
# with the new config's flag. opensearch.py has no equivalent flag,
|
||||
# so there is nothing to pair with it. Both set_hosts() also clear
|
||||
# their module's _READY_DATA_STREAMS memo, which is deliberately not
|
||||
# snapshotted: an emptied memo only makes the next save re-put its
|
||||
# data stream's index template, which is idempotent, so it is
|
||||
# correct for the restored client too.
|
||||
serverless = elastic._SERVERLESS if module is elastic else None
|
||||
snapshot.append((module, connection, serverless))
|
||||
return snapshot
|
||||
@@ -2027,7 +2140,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
"HEC token and HEC index are required when using HEC URL"
|
||||
)
|
||||
try:
|
||||
logger.debug("Initializing Splunk HEC client: url=%s", opts.hec)
|
||||
logger.debug(
|
||||
"Initializing Splunk HEC client: url=%s",
|
||||
redact_url_credentials(opts.hec),
|
||||
)
|
||||
verify = True
|
||||
if opts.hec_skip_certificate_verification:
|
||||
verify = False
|
||||
@@ -2094,7 +2210,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
|
||||
try:
|
||||
if opts.loki_url:
|
||||
logger.debug("Initializing Loki client: url=%s", opts.loki_url)
|
||||
logger.debug(
|
||||
"Initializing Loki client: url=%s",
|
||||
redact_url_credentials(opts.loki_url),
|
||||
)
|
||||
clients["loki_client"] = loki.LokiClient(
|
||||
opts.loki_url,
|
||||
labels=opts.loki_labels,
|
||||
@@ -2122,8 +2241,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
# places rather than one: connections.create_connection() registers the
|
||||
# new client under the ``default`` alias, and elastic.set_hosts() also
|
||||
# assigns elastic._SERVERLESS. _init_output_clients() rolls both back if
|
||||
# a later step fails. They are still initialized last, so that a failure
|
||||
# in any other output happens before either registry has been touched.
|
||||
# a later step fails. (Both set_hosts() also clear their data stream
|
||||
# memo, which needs no rollback; see _search_alias_snapshot().) They are
|
||||
# still initialized last, so that a failure in any other output happens
|
||||
# before either registry has been touched.
|
||||
if opts.save_aggregate or opts.save_failure or opts.save_smtp_tls:
|
||||
# Scoped to the same condition as the constructors below, which is
|
||||
# also the condition under which process_reports() dereferences
|
||||
@@ -2137,7 +2258,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
if opts.elasticsearch_hosts:
|
||||
logger.debug(
|
||||
"Initializing Elasticsearch client: hosts=%s, ssl=%s",
|
||||
opts.elasticsearch_hosts,
|
||||
redact_url_credentials(opts.elasticsearch_hosts),
|
||||
opts.elasticsearch_ssl,
|
||||
)
|
||||
es_aggregate_indexes = _migration_index_names(
|
||||
@@ -2208,7 +2329,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
if opts.opensearch_hosts:
|
||||
logger.debug(
|
||||
"Initializing OpenSearch client: hosts=%s, ssl=%s",
|
||||
opts.opensearch_hosts,
|
||||
redact_url_credentials(opts.opensearch_hosts),
|
||||
opts.opensearch_ssl,
|
||||
)
|
||||
os_aggregate_indexes = _migration_index_names(
|
||||
@@ -2544,6 +2665,7 @@ def _main():
|
||||
index_prefix=opts.elasticsearch_index_prefix
|
||||
or get_index_prefix(report),
|
||||
monthly_indexes=opts.elasticsearch_monthly_indexes,
|
||||
use_data_streams=opts.elasticsearch_use_data_streams,
|
||||
number_of_shards=shards,
|
||||
number_of_replicas=replicas,
|
||||
)
|
||||
@@ -2564,6 +2686,7 @@ def _main():
|
||||
index_prefix=opts.opensearch_index_prefix
|
||||
or get_index_prefix(report),
|
||||
monthly_indexes=opts.opensearch_monthly_indexes,
|
||||
use_data_streams=opts.opensearch_use_data_streams,
|
||||
number_of_shards=shards,
|
||||
number_of_replicas=replicas,
|
||||
)
|
||||
@@ -2643,6 +2766,7 @@ def _main():
|
||||
index_prefix=opts.elasticsearch_index_prefix
|
||||
or get_index_prefix(report),
|
||||
monthly_indexes=opts.elasticsearch_monthly_indexes,
|
||||
use_data_streams=opts.elasticsearch_use_data_streams,
|
||||
number_of_shards=shards,
|
||||
number_of_replicas=replicas,
|
||||
)
|
||||
@@ -2663,6 +2787,7 @@ def _main():
|
||||
index_prefix=opts.opensearch_index_prefix
|
||||
or get_index_prefix(report),
|
||||
monthly_indexes=opts.opensearch_monthly_indexes,
|
||||
use_data_streams=opts.opensearch_use_data_streams,
|
||||
number_of_shards=shards,
|
||||
number_of_replicas=replicas,
|
||||
)
|
||||
@@ -2742,6 +2867,7 @@ def _main():
|
||||
index_prefix=opts.elasticsearch_index_prefix
|
||||
or get_index_prefix(report),
|
||||
monthly_indexes=opts.elasticsearch_monthly_indexes,
|
||||
use_data_streams=opts.elasticsearch_use_data_streams,
|
||||
number_of_shards=shards,
|
||||
number_of_replicas=replicas,
|
||||
)
|
||||
@@ -2762,6 +2888,7 @@ def _main():
|
||||
index_prefix=opts.opensearch_index_prefix
|
||||
or get_index_prefix(report),
|
||||
monthly_indexes=opts.opensearch_monthly_indexes,
|
||||
use_data_streams=opts.opensearch_use_data_streams,
|
||||
number_of_shards=shards,
|
||||
number_of_replicas=replicas,
|
||||
)
|
||||
@@ -3053,6 +3180,7 @@ def _main():
|
||||
elasticsearch_password=None,
|
||||
elasticsearch_api_key=None,
|
||||
elasticsearch_serverless=False,
|
||||
elasticsearch_use_data_streams=False,
|
||||
opensearch_hosts=None,
|
||||
opensearch_timeout=60,
|
||||
opensearch_number_of_shards=1,
|
||||
@@ -3069,6 +3197,7 @@ def _main():
|
||||
opensearch_auth_type="basic",
|
||||
opensearch_aws_region=None,
|
||||
opensearch_aws_service="es",
|
||||
opensearch_use_data_streams=False,
|
||||
kafka_hosts=None,
|
||||
kafka_username=None,
|
||||
kafka_password=None,
|
||||
|
||||
+188
-33
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from elasticsearch import Elasticsearch
|
||||
from elasticsearch import Elasticsearch, NotFoundError
|
||||
from elasticsearch.dsl import (
|
||||
Boolean,
|
||||
Date,
|
||||
@@ -43,6 +43,22 @@ _SERVERLESS = False
|
||||
# settings (e.g. ``refresh_interval``) are accepted and pass through.
|
||||
_SERVERLESS_REJECTED_SETTINGS = frozenset({"number_of_shards", "number_of_replicas"})
|
||||
|
||||
# Names of the data streams whose index template and stream
|
||||
# ``create_data_streams`` has already set up in this process, so each one
|
||||
# costs its API calls once per process rather than once per saved document.
|
||||
# ``set_hosts`` clears it, so a new connection (including a SIGHUP reload,
|
||||
# which may point at another cluster or change the index settings)
|
||||
# re-applies the template. Forgetting an entry is always safe: it only
|
||||
# repeats idempotent calls.
|
||||
_READY_DATA_STREAMS: set[str] = set()
|
||||
|
||||
# Priority of the index template ``create_data_streams`` installs.
|
||||
# Elasticsearch's built-in templates have priority 100 and Fleet's
|
||||
# integration templates up to 200, and its documentation recommends a
|
||||
# priority higher than 500 for custom templates whose patterns overlap them. The template's ``index_patterns`` is the
|
||||
# exact stream name, so an overlap is unlikely in the first place.
|
||||
_DATA_STREAM_TEMPLATE_PRIORITY = 501
|
||||
|
||||
# Guard query for the dkim_results_combined/spf_results_combined backfill
|
||||
# (see ``migrate_indexes``). Matches only documents that have at least one
|
||||
# DKIM or SPF auth result and are missing the corresponding combined field.
|
||||
@@ -620,6 +636,7 @@ def set_hosts(
|
||||
# Module-global; see the _SERVERLESS comment at the top of the module.
|
||||
global _SERVERLESS
|
||||
_SERVERLESS = serverless
|
||||
_READY_DATA_STREAMS.clear()
|
||||
if not isinstance(hosts, list):
|
||||
hosts = [hosts]
|
||||
scheme = "https://" if use_ssl else "http://"
|
||||
@@ -639,6 +656,27 @@ def set_hosts(
|
||||
return connections.create_connection(**conn_params)
|
||||
|
||||
|
||||
def _effective_index_settings(settings: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""Return the index settings to send, adjusted for Serverless.
|
||||
|
||||
Args:
|
||||
settings (dict | None): The requested index settings, or ``None``
|
||||
for the defaults (one shard, no replicas).
|
||||
|
||||
Returns:
|
||||
dict: In Serverless mode, the requested settings minus the keys in
|
||||
``_SERVERLESS_REJECTED_SETTINGS``, and no defaults at all;
|
||||
otherwise a copy of the requested settings, or the defaults.
|
||||
"""
|
||||
if settings is None:
|
||||
return {} if _SERVERLESS else {"number_of_shards": 1, "number_of_replicas": 0}
|
||||
if _SERVERLESS:
|
||||
return {
|
||||
k: v for k, v in settings.items() if k not in _SERVERLESS_REJECTED_SETTINGS
|
||||
}
|
||||
return dict(settings)
|
||||
|
||||
|
||||
def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
|
||||
"""
|
||||
Create Elasticsearch indexes
|
||||
@@ -649,16 +687,7 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
|
||||
``_SERVERLESS_REJECTED_SETTINGS`` are filtered out and the
|
||||
remaining keys are passed through; defaults are skipped entirely.
|
||||
"""
|
||||
if settings is None:
|
||||
effective_settings: dict[str, Any] = (
|
||||
{} if _SERVERLESS else {"number_of_shards": 1, "number_of_replicas": 0}
|
||||
)
|
||||
elif _SERVERLESS:
|
||||
effective_settings = {
|
||||
k: v for k, v in settings.items() if k not in _SERVERLESS_REJECTED_SETTINGS
|
||||
}
|
||||
else:
|
||||
effective_settings = dict(settings)
|
||||
effective_settings = _effective_index_settings(settings)
|
||||
|
||||
for name in names:
|
||||
index = Index(name)
|
||||
@@ -684,6 +713,88 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
|
||||
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
|
||||
|
||||
|
||||
def create_data_streams(names: list[str], settings: dict[str, Any] | None = None):
|
||||
"""
|
||||
Create Elasticsearch data streams and their index templates
|
||||
|
||||
For each name, puts an index template named after the stream whose
|
||||
``index_patterns`` is exactly that name -- no wildcard, so it never
|
||||
applies to the date-suffixed indexes parsedmarc writes without data
|
||||
streams -- and then creates the data stream if it does not exist yet.
|
||||
The template composes an optional ``<name>@custom`` component template,
|
||||
which is where an operator attaches an ILM policy or other settings:
|
||||
parsedmarc puts its own template again in every process that writes to
|
||||
the stream, so edits made to that template directly do not last. Each
|
||||
name is set up once per process; ``set_hosts`` resets that.
|
||||
|
||||
The template maps only ``@timestamp``. Like ``create_indexes``, it
|
||||
deliberately installs no mappings for the report fields themselves (see
|
||||
issue #169), so they keep their dynamic ``object`` mapping.
|
||||
|
||||
Args:
|
||||
names (list): A list of data stream names
|
||||
settings (dict): Index settings for the stream's backing indices,
|
||||
adjusted for Serverless the same way ``create_indexes`` does.
|
||||
|
||||
Raises:
|
||||
ElasticsearchError: A concrete index or alias already holds one of
|
||||
the names, or the cluster returned an error. A name that fails
|
||||
is not remembered, so the next save retries it.
|
||||
"""
|
||||
effective_settings = _effective_index_settings(settings)
|
||||
|
||||
for name in names:
|
||||
if name in _READY_DATA_STREAMS:
|
||||
continue
|
||||
custom_template = f"{name}@custom"
|
||||
try:
|
||||
client = connections.get_connection()
|
||||
# A missing name is a 404, but a name held by a concrete index
|
||||
# answers 200 with an empty list (verified against
|
||||
# Elasticsearch 8.19), so look for the stream in the response.
|
||||
try:
|
||||
response = client.indices.get_data_stream(name=name)
|
||||
stream_exists = any(
|
||||
stream.get("name") == name
|
||||
for stream in response.get("data_streams", [])
|
||||
)
|
||||
except NotFoundError:
|
||||
stream_exists = False
|
||||
# Checked before anything is written, so a name collision leaves
|
||||
# no template behind.
|
||||
collision = not stream_exists and Index(name).exists()
|
||||
if not collision:
|
||||
client.indices.put_index_template(
|
||||
name=name,
|
||||
index_patterns=[name],
|
||||
data_stream={},
|
||||
composed_of=[custom_template],
|
||||
ignore_missing_component_templates=[custom_template],
|
||||
priority=_DATA_STREAM_TEMPLATE_PRIORITY,
|
||||
template={
|
||||
"settings": effective_settings,
|
||||
"mappings": {"properties": {"@timestamp": {"type": "date"}}},
|
||||
},
|
||||
meta={
|
||||
"description": (
|
||||
"Managed by parsedmarc, which overwrites it. Put "
|
||||
f"customizations in {custom_template}."
|
||||
)
|
||||
},
|
||||
)
|
||||
if not stream_exists:
|
||||
logger.debug(f"Creating Elasticsearch data stream: {name}")
|
||||
client.indices.create_data_stream(name=name)
|
||||
except Exception as e:
|
||||
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
|
||||
if collision:
|
||||
raise ElasticsearchError(
|
||||
f"Elasticsearch error: cannot create the data stream {name}: "
|
||||
"an index or alias with that name already exists"
|
||||
)
|
||||
_READY_DATA_STREAMS.add(name)
|
||||
|
||||
|
||||
_LEGACY_FO_FIELD = "published_policy.fo"
|
||||
# The same field split into the object/leaf names a mapping body nests it
|
||||
# under. Derived from the dotted name so the mapping written below cannot
|
||||
@@ -757,6 +868,12 @@ def migrate_indexes(
|
||||
next startup, and the manual ``_update_by_query`` commands documented in
|
||||
``docs/source/elasticsearch.md`` remain available in the meantime.
|
||||
|
||||
Both work unchanged with ``use_data_streams``: a data stream carries the
|
||||
undated name itself, so the backfill patterns match it as well as any
|
||||
dated indexes, and a stream that holds that exact name is skipped by
|
||||
the ``published_policy.fo`` migration, because the backing index it
|
||||
inspects maps the field as ``text``, not ``long``.
|
||||
|
||||
Args:
|
||||
aggregate_indexes (list): A list of aggregate index names
|
||||
failure_indexes (list): A list of failure index names
|
||||
@@ -922,6 +1039,7 @@ def save_aggregate_report_to_elasticsearch(
|
||||
monthly_indexes: bool | None = False,
|
||||
number_of_shards: int = 1,
|
||||
number_of_replicas: int = 0,
|
||||
use_data_streams: bool = False,
|
||||
):
|
||||
"""
|
||||
Saves a parsed DMARC aggregate report to Elasticsearch
|
||||
@@ -931,8 +1049,11 @@ def save_aggregate_report_to_elasticsearch(
|
||||
index_suffix (str): The suffix of the name of the index to save to
|
||||
index_prefix (str): The prefix of the name of the index to save to
|
||||
monthly_indexes (bool): Use monthly indexes instead of daily indexes
|
||||
(ignored when ``use_data_streams`` is True)
|
||||
number_of_shards (int): The number of shards to use in the index
|
||||
number_of_replicas (int): The number of replicas to use in the index
|
||||
use_data_streams (bool): Save to a data stream named like the index,
|
||||
without the date suffix, instead of to a dated index
|
||||
|
||||
Raises:
|
||||
AlreadySaved
|
||||
@@ -959,6 +1080,8 @@ def save_aggregate_report_to_elasticsearch(
|
||||
search_index = "dmarc_aggregate*"
|
||||
if index_prefix is not None:
|
||||
search_index = f"{index_prefix}{search_index}"
|
||||
# The trailing wildcard matches the data stream as well as the dated
|
||||
# indexes, so duplicates are caught across a switch between the two.
|
||||
search = Search(index=search_index)
|
||||
query = org_name_query & report_id_query & domain_query
|
||||
query = query & begin_date_query & end_date_query
|
||||
@@ -1005,10 +1128,6 @@ def save_aggregate_report_to_elasticsearch(
|
||||
)
|
||||
normalized_timespan = record["normalized_timespan"]
|
||||
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
aggregate_report["begin_date"] = begin_date
|
||||
aggregate_report["end_date"] = end_date
|
||||
date_range = [aggregate_report["begin_date"], aggregate_report["end_date"]]
|
||||
@@ -1049,6 +1168,7 @@ def save_aggregate_report_to_elasticsearch(
|
||||
"discovery_method"
|
||||
),
|
||||
generator=metadata.get("generator"),
|
||||
**{"@timestamp": begin_date},
|
||||
)
|
||||
|
||||
for override in record["policy_evaluated"]["policy_override_reasons"]:
|
||||
@@ -1078,15 +1198,24 @@ def save_aggregate_report_to_elasticsearch(
|
||||
if index_prefix:
|
||||
index = f"{index_prefix}{index}"
|
||||
|
||||
index = f"{index}-{index_date}"
|
||||
index_settings = dict(
|
||||
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
|
||||
)
|
||||
create_indexes([index], index_settings)
|
||||
if use_data_streams:
|
||||
create_data_streams([index], index_settings)
|
||||
else:
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
create_indexes([index], index_settings)
|
||||
agg_doc.meta.index = index # pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]
|
||||
|
||||
try:
|
||||
agg_doc.save()
|
||||
# Data streams accept only op_type=create. No _id is set, so
|
||||
# for a dated index it means the same as the default.
|
||||
agg_doc.save(op_type="create")
|
||||
except Exception as e:
|
||||
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
|
||||
|
||||
@@ -1098,6 +1227,7 @@ def save_failure_report_to_elasticsearch(
|
||||
monthly_indexes: bool | None = False,
|
||||
number_of_shards: int = 1,
|
||||
number_of_replicas: int = 0,
|
||||
use_data_streams: bool = False,
|
||||
):
|
||||
"""
|
||||
Saves a parsed DMARC failure report to Elasticsearch
|
||||
@@ -1107,10 +1237,14 @@ def save_failure_report_to_elasticsearch(
|
||||
index_suffix (str): The suffix of the name of the index to save to
|
||||
index_prefix (str): The prefix of the name of the index to save to
|
||||
monthly_indexes (bool): Use monthly indexes instead of daily
|
||||
indexes
|
||||
indexes (ignored when ``use_data_streams``
|
||||
is True)
|
||||
number_of_shards (int): The number of shards to use in the index
|
||||
number_of_replicas (int): The number of replicas to use in the
|
||||
index
|
||||
use_data_streams (bool): Save to a data stream named like the
|
||||
index, without the date suffix, instead
|
||||
of to a dated index
|
||||
|
||||
Raises:
|
||||
AlreadySaved
|
||||
@@ -1145,6 +1279,8 @@ def save_failure_report_to_elasticsearch(
|
||||
search_index = ",".join(
|
||||
f"{index_prefix}{part}" for part in search_index.split(",")
|
||||
)
|
||||
# The trailing wildcards match the data stream as well as the dated
|
||||
# indexes, so duplicates are caught across a switch between the two.
|
||||
search = Search(index=search_index)
|
||||
q = Q(dict(match=dict(arrival_date=arrival_date_epoch_milliseconds))) # pyright: ignore[reportArgumentType]
|
||||
|
||||
@@ -1253,6 +1389,7 @@ def save_failure_report_to_elasticsearch(
|
||||
dkim_domain=failure_report["dkim_domain"],
|
||||
original_rcpt_to=failure_report["original_rcpt_to"],
|
||||
sample=sample,
|
||||
**{"@timestamp": arrival_date},
|
||||
)
|
||||
|
||||
index = "dmarc_failure"
|
||||
@@ -1260,18 +1397,23 @@ def save_failure_report_to_elasticsearch(
|
||||
index = f"{index}_{index_suffix}"
|
||||
if index_prefix:
|
||||
index = f"{index_prefix}{index}"
|
||||
if monthly_indexes:
|
||||
index_date = arrival_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = arrival_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
index_settings = dict(
|
||||
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
|
||||
)
|
||||
create_indexes([index], index_settings)
|
||||
if use_data_streams:
|
||||
create_data_streams([index], index_settings)
|
||||
else:
|
||||
if monthly_indexes:
|
||||
index_date = arrival_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = arrival_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
create_indexes([index], index_settings)
|
||||
failure_doc.meta.index = index # pyright: ignore[reportAttributeAccessIssue, reportOptionalMemberAccess]
|
||||
try:
|
||||
failure_doc.save()
|
||||
# Data streams accept only op_type=create. No _id is set, so
|
||||
# for a dated index it means the same as the default.
|
||||
failure_doc.save(op_type="create")
|
||||
except Exception as e:
|
||||
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
|
||||
except KeyError as e:
|
||||
@@ -1287,6 +1429,7 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
monthly_indexes: bool = False,
|
||||
number_of_shards: int = 1,
|
||||
number_of_replicas: int = 0,
|
||||
use_data_streams: bool = False,
|
||||
):
|
||||
"""
|
||||
Saves a parsed SMTP TLS report to Elasticsearch
|
||||
@@ -1296,8 +1439,11 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
index_suffix (str): The suffix of the name of the index to save to
|
||||
index_prefix (str): The prefix of the name of the index to save to
|
||||
monthly_indexes (bool): Use monthly indexes instead of daily indexes
|
||||
(ignored when ``use_data_streams`` is True)
|
||||
number_of_shards (int): The number of shards to use in the index
|
||||
number_of_replicas (int): The number of replicas to use in the index
|
||||
use_data_streams (bool): Save to a data stream named like the index,
|
||||
without the date suffix, instead of to a dated index
|
||||
|
||||
Raises:
|
||||
AlreadySaved
|
||||
@@ -1310,10 +1456,6 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
end_date = human_timestamp_to_datetime(report["end_date"], to_utc=True)
|
||||
begin_date_human = begin_date.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
end_date_human = end_date.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
report = report.copy()
|
||||
report["begin_date"] = begin_date
|
||||
report["end_date"] = end_date
|
||||
@@ -1329,6 +1471,8 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
search_index = "smtp_tls*"
|
||||
if index_prefix is not None:
|
||||
search_index = f"{index_prefix}{search_index}"
|
||||
# The trailing wildcard matches the data stream as well as the dated
|
||||
# indexes, so duplicates are caught across a switch between the two.
|
||||
search = Search(index=search_index)
|
||||
query = org_name_query & report_id_query
|
||||
query = query & begin_date_query & end_date_query
|
||||
@@ -1355,7 +1499,12 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
index = f"{index}_{index_suffix}"
|
||||
if index_prefix:
|
||||
index = f"{index_prefix}{index}"
|
||||
index = f"{index}-{index_date}"
|
||||
if not use_data_streams:
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
index_settings = dict(
|
||||
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
|
||||
)
|
||||
@@ -1367,6 +1516,7 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
date_end=report["end_date"],
|
||||
contact_info=report["contact_info"],
|
||||
report_id=report["report_id"],
|
||||
**{"@timestamp": report["begin_date"]},
|
||||
)
|
||||
|
||||
for policy in report["policies"]:
|
||||
@@ -1448,11 +1598,16 @@ def save_smtp_tls_report_to_elasticsearch(
|
||||
)
|
||||
smtp_tls_doc.policies.append(policy_doc)
|
||||
|
||||
create_indexes([index], index_settings)
|
||||
if use_data_streams:
|
||||
create_data_streams([index], index_settings)
|
||||
else:
|
||||
create_indexes([index], index_settings)
|
||||
smtp_tls_doc.meta.index = index # pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]
|
||||
|
||||
try:
|
||||
smtp_tls_doc.save()
|
||||
# Data streams accept only op_type=create. No _id is set, so for a
|
||||
# dated index it means the same as the default.
|
||||
smtp_tls_doc.save(op_type="create")
|
||||
except Exception as e:
|
||||
raise ElasticsearchError(f"Elasticsearch error: {e.__str__()}")
|
||||
|
||||
|
||||
+21
-8
@@ -19,7 +19,7 @@ from parsedmarc import (
|
||||
from parsedmarc.constants import USER_AGENT
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.types import AggregateReport, FailureReport, SMTPTLSReport
|
||||
from parsedmarc.utils import human_timestamp_to_unix_timestamp
|
||||
from parsedmarc.utils import human_timestamp_to_unix_timestamp, redact_url_credentials
|
||||
|
||||
PUSH_PATH = "/loki/api/v1/push"
|
||||
TIMESTAMP_MODES = ("ingest", "report_begin")
|
||||
@@ -102,11 +102,22 @@ class LokiClient(object):
|
||||
if not _LABEL_NAME.match(name):
|
||||
raise ValueError(f"Invalid Loki label name '{name}'")
|
||||
|
||||
parsed = httpx.URL(url)
|
||||
# copy_with(path=...) keeps any query string, userinfo and port intact
|
||||
self.url = (
|
||||
str(parsed.copy_with(path=PUSH_PATH)) if parsed.path in ("", "/") else url
|
||||
)
|
||||
try:
|
||||
parsed = httpx.URL(url)
|
||||
# copy_with(path=...) keeps any query string, userinfo and port intact
|
||||
self.url = (
|
||||
str(parsed.copy_with(path=PUSH_PATH))
|
||||
if parsed.path in ("", "/")
|
||||
else url
|
||||
)
|
||||
except httpx.InvalidURL:
|
||||
# httpx quotes part of the URL (part of a password containing
|
||||
# "/", "?" or "#" ends up as the "port"), and chains a ValueError
|
||||
# that repeats it, so neither is kept.
|
||||
raise ValueError(
|
||||
"Invalid Loki URL (details withheld because they may include "
|
||||
"credentials)"
|
||||
) from None
|
||||
self.timestamp = timestamp
|
||||
self.timeout = timeout
|
||||
self.batch_size = max(1, int(batch_size))
|
||||
@@ -196,14 +207,16 @@ class LokiClient(object):
|
||||
self.url, content=json.dumps(body), timeout=self.timeout
|
||||
)
|
||||
except httpx.HTTPError as error_:
|
||||
raise LokiError(f"Loki push to {self.url} failed: {error_}") from error_
|
||||
raise LokiError(
|
||||
f"Loki push to {redact_url_credentials(self.url)} failed: {error_}"
|
||||
) from error_
|
||||
# Loki answers 204 when it stored the push; 200 is also accepted.
|
||||
# Other 2xx codes are not success: with block_ingestion_until set it
|
||||
# returns a custom 260 and discards the logs
|
||||
# (block_ingestion_status_code).
|
||||
if response.status_code not in (200, 204):
|
||||
raise LokiError(
|
||||
f"Loki push to {self.url} failed with HTTP "
|
||||
f"Loki push to {redact_url_credentials(self.url)} failed with HTTP "
|
||||
f"{response.status_code}: {response.text[:500]}"
|
||||
)
|
||||
|
||||
|
||||
+183
-24
@@ -16,6 +16,7 @@ from opensearchpy import (
|
||||
Ip,
|
||||
Keyword,
|
||||
Nested,
|
||||
NotFoundError,
|
||||
Object,
|
||||
OpenSearch,
|
||||
Q,
|
||||
@@ -28,13 +29,30 @@ from opensearchpy.helpers import reindex
|
||||
|
||||
from parsedmarc import InvalidFailureReport
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.utils import human_timestamp_to_datetime
|
||||
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
|
||||
|
||||
|
||||
class OpenSearchError(Exception):
|
||||
"""Raised when an OpenSearch error occurs"""
|
||||
|
||||
|
||||
# Names of the data streams whose index template and stream
|
||||
# ``create_data_streams`` has already set up in this process, so each one
|
||||
# costs its API calls once per process rather than once per saved document.
|
||||
# ``set_hosts`` clears it, so a new connection (including a SIGHUP reload,
|
||||
# which may point at another cluster or change the index settings)
|
||||
# re-applies the template. Forgetting an entry is always safe: it only
|
||||
# repeats idempotent calls.
|
||||
_READY_DATA_STREAMS: set[str] = set()
|
||||
|
||||
# Priority of the index template ``create_data_streams`` installs. OpenSearch
|
||||
# itself installs no index templates, but plugins may; 501 stays above the
|
||||
# usual priority-100 convention and matches the Elasticsearch module. The
|
||||
# template's ``index_patterns`` is the exact stream name, so an overlap with
|
||||
# another template is unlikely in the first place.
|
||||
_DATA_STREAM_TEMPLATE_PRIORITY = 501
|
||||
|
||||
|
||||
# Guard query for the dkim_results_combined/spf_results_combined backfill
|
||||
# (see ``migrate_indexes``). Matches only documents that have at least one
|
||||
# DKIM or SPF auth result and are missing the corresponding combined field.
|
||||
@@ -531,9 +549,14 @@ def set_hosts(
|
||||
alias. Callers that need to close this exact client later (rather than
|
||||
whatever holds the alias at that point) should hold on to it.
|
||||
"""
|
||||
_READY_DATA_STREAMS.clear()
|
||||
if not isinstance(hosts, list):
|
||||
hosts = [hosts]
|
||||
logger.debug("Connecting to OpenSearch: hosts=%s, use_ssl=%s", hosts, use_ssl)
|
||||
logger.debug(
|
||||
"Connecting to OpenSearch: hosts=%s, use_ssl=%s",
|
||||
redact_url_credentials(hosts),
|
||||
use_ssl,
|
||||
)
|
||||
conn_params = {"hosts": hosts, "timeout": timeout}
|
||||
if use_ssl:
|
||||
conn_params["use_ssl"] = True
|
||||
@@ -605,6 +628,98 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
|
||||
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
|
||||
|
||||
|
||||
def create_data_streams(names: list[str], settings: dict[str, Any] | None = None):
|
||||
"""
|
||||
Create OpenSearch data streams and their index templates
|
||||
|
||||
For each name, puts an index template named after the stream whose
|
||||
``index_patterns`` is exactly that name -- no wildcard, so it never
|
||||
applies to the date-suffixed indexes parsedmarc writes without data
|
||||
streams -- and then creates the data stream if it does not exist yet.
|
||||
parsedmarc puts its template again in every process that writes to the
|
||||
stream, so edits made to it directly do not last; an ISM policy attaches
|
||||
itself to the stream's backing indices through its own
|
||||
``ism_template`` instead. Each name is set up once per process;
|
||||
``set_hosts`` resets that.
|
||||
|
||||
The template maps only ``@timestamp``. Like ``create_indexes``, it
|
||||
deliberately installs no mappings for the report fields themselves (see
|
||||
issue #169), so they keep their dynamic ``object`` mapping.
|
||||
|
||||
Args:
|
||||
names (list): A list of data stream names
|
||||
settings (dict): Index settings for the stream's backing indices;
|
||||
one shard and no replicas when ``None``, as in
|
||||
``create_indexes``.
|
||||
|
||||
Raises:
|
||||
OpenSearchError: A concrete index or alias already holds one of the
|
||||
names, or the cluster returned an error. A name that fails is
|
||||
not remembered, so the next save retries it.
|
||||
"""
|
||||
if settings is None:
|
||||
effective_settings: dict[str, Any] = {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
}
|
||||
else:
|
||||
effective_settings = dict(settings)
|
||||
|
||||
for name in names:
|
||||
if name in _READY_DATA_STREAMS:
|
||||
continue
|
||||
try:
|
||||
client = connections.get_connection()
|
||||
# OpenSearch 3 answers both a missing name and a name held by a
|
||||
# concrete index with a 404. Elasticsearch answers the latter
|
||||
# with 200 and an empty list, so, as in the Elasticsearch module,
|
||||
# look for the stream in the response rather than trust the
|
||||
# status alone.
|
||||
try:
|
||||
response = client.indices.get_data_stream(name=name)
|
||||
stream_exists = any(
|
||||
stream.get("name") == name
|
||||
for stream in response.get("data_streams", [])
|
||||
)
|
||||
except NotFoundError:
|
||||
stream_exists = False
|
||||
# Checked before anything is written, so a name collision leaves
|
||||
# no template behind.
|
||||
collision = not stream_exists and Index(name).exists()
|
||||
if not collision:
|
||||
client.indices.put_index_template(
|
||||
name=name,
|
||||
body={
|
||||
"index_patterns": [name],
|
||||
"data_stream": {},
|
||||
"priority": _DATA_STREAM_TEMPLATE_PRIORITY,
|
||||
"template": {
|
||||
"settings": effective_settings,
|
||||
"mappings": {
|
||||
"properties": {"@timestamp": {"type": "date"}}
|
||||
},
|
||||
},
|
||||
"_meta": {
|
||||
"description": (
|
||||
"Managed by parsedmarc, which overwrites it. "
|
||||
"Attach ISM policies with an ism_template."
|
||||
)
|
||||
},
|
||||
},
|
||||
)
|
||||
if not stream_exists:
|
||||
logger.debug(f"Creating OpenSearch data stream: {name}")
|
||||
client.indices.create_data_stream(name=name)
|
||||
except Exception as e:
|
||||
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
|
||||
if collision:
|
||||
raise OpenSearchError(
|
||||
f"OpenSearch error: cannot create the data stream {name}: "
|
||||
"an index or alias with that name already exists"
|
||||
)
|
||||
_READY_DATA_STREAMS.add(name)
|
||||
|
||||
|
||||
_LEGACY_FO_FIELD = "published_policy.fo"
|
||||
# The same field split into the object/leaf names a mapping body nests it
|
||||
# under. Derived from the dotted name so the mapping written below cannot
|
||||
@@ -678,6 +793,12 @@ def migrate_indexes(
|
||||
``failure_details_combined`` fields (same issue #169) on SMTP TLS
|
||||
report documents, for each name in ``smtp_tls_indexes``.
|
||||
|
||||
All three work unchanged with ``use_data_streams``: a data stream
|
||||
carries the undated name itself, so the backfill patterns match it as
|
||||
well as any dated indexes, and a stream that holds that exact name is
|
||||
skipped by the ``published_policy.fo`` migration, because the backing
|
||||
index it inspects maps the field as ``text``, not ``long``.
|
||||
|
||||
Args:
|
||||
aggregate_indexes (list): A list of aggregate index names
|
||||
failure_indexes (list): A list of failure index names
|
||||
@@ -853,6 +974,7 @@ def save_aggregate_report_to_opensearch(
|
||||
monthly_indexes: bool = False,
|
||||
number_of_shards: int = 1,
|
||||
number_of_replicas: int = 0,
|
||||
use_data_streams: bool = False,
|
||||
):
|
||||
"""
|
||||
Saves a parsed DMARC aggregate report to OpenSearch
|
||||
@@ -862,8 +984,11 @@ def save_aggregate_report_to_opensearch(
|
||||
index_suffix (str): The suffix of the name of the index to save to
|
||||
index_prefix (str): The prefix of the name of the index to save to
|
||||
monthly_indexes (bool): Use monthly indexes instead of daily indexes
|
||||
(ignored when ``use_data_streams`` is True)
|
||||
number_of_shards (int): The number of shards to use in the index
|
||||
number_of_replicas (int): The number of replicas to use in the index
|
||||
use_data_streams (bool): Save to a data stream named like the index,
|
||||
without the date suffix, instead of to a dated index
|
||||
|
||||
Raises:
|
||||
AlreadySaved
|
||||
@@ -890,6 +1015,8 @@ def save_aggregate_report_to_opensearch(
|
||||
search_index = "dmarc_aggregate*"
|
||||
if index_prefix is not None:
|
||||
search_index = f"{index_prefix}{search_index}"
|
||||
# The trailing wildcard matches the data stream as well as the dated
|
||||
# indexes, so duplicates are caught across a switch between the two.
|
||||
search = Search(index=search_index)
|
||||
query = org_name_query & report_id_query & domain_query
|
||||
query = query & begin_date_query & end_date_query
|
||||
@@ -933,10 +1060,6 @@ def save_aggregate_report_to_opensearch(
|
||||
)
|
||||
normalized_timespan = record["normalized_timespan"]
|
||||
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
aggregate_report["begin_date"] = begin_date
|
||||
aggregate_report["end_date"] = end_date
|
||||
date_range = [aggregate_report["begin_date"], aggregate_report["end_date"]]
|
||||
@@ -977,6 +1100,7 @@ def save_aggregate_report_to_opensearch(
|
||||
"discovery_method"
|
||||
),
|
||||
generator=metadata.get("generator"),
|
||||
**{"@timestamp": begin_date},
|
||||
)
|
||||
|
||||
for override in record["policy_evaluated"]["policy_override_reasons"]:
|
||||
@@ -1006,15 +1130,24 @@ def save_aggregate_report_to_opensearch(
|
||||
if index_prefix:
|
||||
index = f"{index_prefix}{index}"
|
||||
|
||||
index = f"{index}-{index_date}"
|
||||
index_settings = dict(
|
||||
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
|
||||
)
|
||||
create_indexes([index], index_settings)
|
||||
if use_data_streams:
|
||||
create_data_streams([index], index_settings)
|
||||
else:
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
create_indexes([index], index_settings)
|
||||
agg_doc.meta.index = index
|
||||
|
||||
try:
|
||||
agg_doc.save()
|
||||
# Data streams accept only op_type=create. No _id is set, so
|
||||
# for a dated index it means the same as the default.
|
||||
agg_doc.save(op_type="create")
|
||||
except Exception as e:
|
||||
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
|
||||
|
||||
@@ -1026,6 +1159,7 @@ def save_failure_report_to_opensearch(
|
||||
monthly_indexes: bool = False,
|
||||
number_of_shards: int = 1,
|
||||
number_of_replicas: int = 0,
|
||||
use_data_streams: bool = False,
|
||||
):
|
||||
"""
|
||||
Saves a parsed DMARC failure report to OpenSearch
|
||||
@@ -1035,10 +1169,14 @@ def save_failure_report_to_opensearch(
|
||||
index_suffix (str): The suffix of the name of the index to save to
|
||||
index_prefix (str): The prefix of the name of the index to save to
|
||||
monthly_indexes (bool): Use monthly indexes instead of daily
|
||||
indexes
|
||||
indexes (ignored when ``use_data_streams``
|
||||
is True)
|
||||
number_of_shards (int): The number of shards to use in the index
|
||||
number_of_replicas (int): The number of replicas to use in the
|
||||
index
|
||||
use_data_streams (bool): Save to a data stream named like the
|
||||
index, without the date suffix, instead
|
||||
of to a dated index
|
||||
|
||||
Raises:
|
||||
AlreadySaved
|
||||
@@ -1073,6 +1211,8 @@ def save_failure_report_to_opensearch(
|
||||
search_index = ",".join(
|
||||
f"{index_prefix}{part}" for part in search_index.split(",")
|
||||
)
|
||||
# The trailing wildcards match the data stream as well as the dated
|
||||
# indexes, so duplicates are caught across a switch between the two.
|
||||
search = Search(index=search_index)
|
||||
q = Q(dict(match=dict(arrival_date=arrival_date_epoch_milliseconds)))
|
||||
|
||||
@@ -1179,6 +1319,7 @@ def save_failure_report_to_opensearch(
|
||||
dkim_domain=failure_report["dkim_domain"],
|
||||
original_rcpt_to=failure_report["original_rcpt_to"],
|
||||
sample=sample,
|
||||
**{"@timestamp": arrival_date},
|
||||
)
|
||||
|
||||
index = "dmarc_failure"
|
||||
@@ -1186,18 +1327,23 @@ def save_failure_report_to_opensearch(
|
||||
index = f"{index}_{index_suffix}"
|
||||
if index_prefix:
|
||||
index = f"{index_prefix}{index}"
|
||||
if monthly_indexes:
|
||||
index_date = arrival_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = arrival_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
index_settings = dict(
|
||||
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
|
||||
)
|
||||
create_indexes([index], index_settings)
|
||||
if use_data_streams:
|
||||
create_data_streams([index], index_settings)
|
||||
else:
|
||||
if monthly_indexes:
|
||||
index_date = arrival_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = arrival_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
create_indexes([index], index_settings)
|
||||
failure_doc.meta.index = index
|
||||
try:
|
||||
failure_doc.save()
|
||||
# Data streams accept only op_type=create. No _id is set, so
|
||||
# for a dated index it means the same as the default.
|
||||
failure_doc.save(op_type="create")
|
||||
except Exception as e:
|
||||
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
|
||||
except KeyError as e:
|
||||
@@ -1213,6 +1359,7 @@ def save_smtp_tls_report_to_opensearch(
|
||||
monthly_indexes: bool = False,
|
||||
number_of_shards: int = 1,
|
||||
number_of_replicas: int = 0,
|
||||
use_data_streams: bool = False,
|
||||
):
|
||||
"""
|
||||
Saves a parsed SMTP TLS report to OpenSearch
|
||||
@@ -1222,8 +1369,11 @@ def save_smtp_tls_report_to_opensearch(
|
||||
index_suffix (str): The suffix of the name of the index to save to
|
||||
index_prefix (str): The prefix of the name of the index to save to
|
||||
monthly_indexes (bool): Use monthly indexes instead of daily indexes
|
||||
(ignored when ``use_data_streams`` is True)
|
||||
number_of_shards (int): The number of shards to use in the index
|
||||
number_of_replicas (int): The number of replicas to use in the index
|
||||
use_data_streams (bool): Save to a data stream named like the index,
|
||||
without the date suffix, instead of to a dated index
|
||||
|
||||
Raises:
|
||||
AlreadySaved
|
||||
@@ -1236,10 +1386,6 @@ def save_smtp_tls_report_to_opensearch(
|
||||
end_date = human_timestamp_to_datetime(report["end_date"], to_utc=True)
|
||||
begin_date_human = begin_date.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
end_date_human = end_date.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
report = report.copy()
|
||||
report["begin_date"] = begin_date
|
||||
report["end_date"] = end_date
|
||||
@@ -1255,6 +1401,8 @@ def save_smtp_tls_report_to_opensearch(
|
||||
search_index = "smtp_tls*"
|
||||
if index_prefix is not None:
|
||||
search_index = f"{index_prefix}{search_index}"
|
||||
# The trailing wildcard matches the data stream as well as the dated
|
||||
# indexes, so duplicates are caught across a switch between the two.
|
||||
search = Search(index=search_index)
|
||||
query = org_name_query & report_id_query
|
||||
query = query & begin_date_query & end_date_query
|
||||
@@ -1281,7 +1429,12 @@ def save_smtp_tls_report_to_opensearch(
|
||||
index = f"{index}_{index_suffix}"
|
||||
if index_prefix:
|
||||
index = f"{index_prefix}{index}"
|
||||
index = f"{index}-{index_date}"
|
||||
if not use_data_streams:
|
||||
if monthly_indexes:
|
||||
index_date = begin_date.strftime("%Y-%m")
|
||||
else:
|
||||
index_date = begin_date.strftime("%Y-%m-%d")
|
||||
index = f"{index}-{index_date}"
|
||||
index_settings = dict(
|
||||
number_of_shards=number_of_shards, number_of_replicas=number_of_replicas
|
||||
)
|
||||
@@ -1293,6 +1446,7 @@ def save_smtp_tls_report_to_opensearch(
|
||||
date_end=report["end_date"],
|
||||
contact_info=report["contact_info"],
|
||||
report_id=report["report_id"],
|
||||
**{"@timestamp": report["begin_date"]},
|
||||
)
|
||||
|
||||
for policy in report["policies"]:
|
||||
@@ -1374,11 +1528,16 @@ def save_smtp_tls_report_to_opensearch(
|
||||
)
|
||||
smtp_tls_doc.policies.append(policy_doc)
|
||||
|
||||
create_indexes([index], index_settings)
|
||||
if use_data_streams:
|
||||
create_data_streams([index], index_settings)
|
||||
else:
|
||||
create_indexes([index], index_settings)
|
||||
smtp_tls_doc.meta.index = index
|
||||
|
||||
try:
|
||||
smtp_tls_doc.save()
|
||||
# Data streams accept only op_type=create. No _id is set, so for a
|
||||
# dated index it means the same as the default.
|
||||
smtp_tls_doc.save(op_type="create")
|
||||
except Exception as e:
|
||||
raise OpenSearchError(f"OpenSearch error: {e.__str__()}")
|
||||
|
||||
|
||||
+11
-2
@@ -20,7 +20,7 @@ else:
|
||||
psycopg_json = None
|
||||
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.utils import human_timestamp_to_datetime
|
||||
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
|
||||
|
||||
# psycopg is an optional dependency (the PostgreSQL backend is opt-in). The
|
||||
# pure helper functions below work without it; only PostgreSQLClient needs a
|
||||
@@ -183,7 +183,16 @@ class PostgreSQLClient:
|
||||
)
|
||||
conn.autocommit = False
|
||||
except psycopg.Error as exc:
|
||||
raise PostgreSQLError(str(exc)) from exc
|
||||
# libpq reports a malformed connection string by echoing the
|
||||
# whole string, or the fragment it could not parse, and either
|
||||
# can contain the password. psycopg raises ProgrammingError for
|
||||
# those parse failures, so withhold their detail entirely.
|
||||
if self._connection_string and isinstance(exc, psycopg.ProgrammingError):
|
||||
raise PostgreSQLError(
|
||||
"Invalid PostgreSQL connection_string (details withheld "
|
||||
"because they may include the password)"
|
||||
) from None
|
||||
raise PostgreSQLError(redact_url_credentials(str(exc))) from exc
|
||||
self._conn = conn
|
||||
return conn
|
||||
|
||||
|
||||
@@ -13,6 +13,13 @@ from parsedmarc.constants import USER_AGENT
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.utils import human_timestamp_to_unix_timestamp
|
||||
|
||||
# httpx.InvalidURL quotes part of the URL, which can be part of a password
|
||||
# (one containing "/", "?" or "#" ends up as the "port"), so its text is
|
||||
# replaced by this fixed message.
|
||||
_INVALID_URL_MESSAGE = (
|
||||
"Invalid Splunk HEC URL (details withheld because they may include credentials)"
|
||||
)
|
||||
|
||||
|
||||
class SplunkError(RuntimeError):
|
||||
"""Raised when a Splunk API error occurs"""
|
||||
@@ -136,6 +143,8 @@ class HECClient(object):
|
||||
self.url, content="".join(json_lines), timeout=self.timeout
|
||||
)
|
||||
response = response.json()
|
||||
except httpx.InvalidURL:
|
||||
raise SplunkError(_INVALID_URL_MESSAGE) from None
|
||||
except Exception as e:
|
||||
raise SplunkError(e.__str__())
|
||||
if response["code"] != 0:
|
||||
@@ -179,6 +188,8 @@ class HECClient(object):
|
||||
self.url, content="".join(json_lines), timeout=self.timeout
|
||||
)
|
||||
response = response.json()
|
||||
except httpx.InvalidURL:
|
||||
raise SplunkError(_INVALID_URL_MESSAGE) from None
|
||||
except Exception as e:
|
||||
raise SplunkError(e.__str__())
|
||||
if response["code"] != 0:
|
||||
@@ -218,6 +229,8 @@ class HECClient(object):
|
||||
self.url, content="".join(json_lines), timeout=self.timeout
|
||||
)
|
||||
response = response.json()
|
||||
except httpx.InvalidURL:
|
||||
raise SplunkError(_INVALID_URL_MESSAGE) from None
|
||||
except Exception as e:
|
||||
raise SplunkError(e.__str__())
|
||||
if response["code"] != 0:
|
||||
|
||||
+351
-52
@@ -16,8 +16,8 @@ import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import TypedDict, cast
|
||||
from urllib.parse import urlsplit
|
||||
from typing import TYPE_CHECKING, TypedDict, cast
|
||||
from urllib.parse import SplitResult, urlsplit
|
||||
|
||||
import mailparser
|
||||
from expiringdict import ExpiringDict
|
||||
@@ -46,6 +46,11 @@ from parsedmarc.constants import (
|
||||
)
|
||||
from parsedmarc.log import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# dnspython 2.9+ installs httpx2 through its [doh] extra; 2.7 and 2.8
|
||||
# install httpx, so httpx2 may be absent
|
||||
import httpx2 # pyright: ignore[reportMissingImports]
|
||||
|
||||
# Errors considered transient and retryable by query_dns. LifetimeTimeout is
|
||||
# dnspython's deadline expiry; NoNameservers typically wraps a SERVFAIL from
|
||||
# upstream; OSError covers socket-level failures during TCP fallback.
|
||||
@@ -55,9 +60,9 @@ _RETRYABLE_DNS_ERRORS = (
|
||||
OSError,
|
||||
)
|
||||
|
||||
# The process-wide httpx client used for DNS over HTTPS queries, and the PID
|
||||
# The process-wide HTTP client used for DNS over HTTPS queries, and the PID
|
||||
# it was created under. See _get_doh_session().
|
||||
_DOH_SESSION: httpx.Client | None = None
|
||||
_DOH_SESSION: httpx.Client | httpx2.Client | None = None
|
||||
_DOH_SESSION_PID: int | None = None
|
||||
|
||||
null_file = subprocess.DEVNULL
|
||||
@@ -108,15 +113,17 @@ def load_psl_overrides(
|
||||
|
||||
if not (offline or always_use_local_file):
|
||||
try:
|
||||
logger.debug(f"Trying to fetch PSL overrides from {url}...")
|
||||
logger.debug(
|
||||
f"Trying to fetch PSL overrides from {redact_url_credentials(url)}..."
|
||||
)
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
response = httpx.get(
|
||||
url, headers=headers, timeout=60, follow_redirects=True
|
||||
)
|
||||
response.raise_for_status()
|
||||
_load_text(response.text)
|
||||
except httpx.HTTPError as e:
|
||||
logger.warning(f"Failed to fetch PSL overrides: {e}")
|
||||
except (httpx.HTTPError, httpx.InvalidURL) as e:
|
||||
logger.warning(f"Failed to fetch PSL overrides: {_describe_http_error(e)}")
|
||||
|
||||
if len(psl_overrides) == 0:
|
||||
path = local_file_path or str(
|
||||
@@ -204,9 +211,32 @@ def get_base_domain(domain: str) -> str | None:
|
||||
return publicsuffix
|
||||
|
||||
|
||||
def _get_doh_session() -> httpx.Client:
|
||||
def _doh_client_class() -> type[httpx.Client] | type[httpx2.Client]:
|
||||
"""
|
||||
Returns the shared ``httpx.Client`` used for DNS over HTTPS queries.
|
||||
Returns the HTTP client class ``dns.query.https()`` accepts as a
|
||||
``session``.
|
||||
|
||||
dnspython 2.9 moved DNS over HTTPS from httpx to httpx2, and
|
||||
``dns.query.https()`` rejects a session of the other library's class
|
||||
(``ValueError: session parameter must be an httpx2.Client``), so an
|
||||
``httpx.Client`` broke every DoH query under dnspython 2.9
|
||||
(https://github.com/domainaware/parsedmarc/issues/935). The class is
|
||||
taken from the module dnspython itself imported, so it matches its
|
||||
``isinstance`` check even when httpx2 is installed beside dnspython 2.8.
|
||||
|
||||
Returns:
|
||||
type: ``httpx2.Client`` under dnspython 2.9+, ``httpx.Client`` before
|
||||
"""
|
||||
return (
|
||||
getattr(dns.query, "httpx2", None) or getattr(dns.query, "httpx", httpx)
|
||||
).Client
|
||||
|
||||
|
||||
def _get_doh_session() -> httpx.Client | httpx2.Client:
|
||||
"""
|
||||
Returns the shared HTTP client used for DNS over HTTPS queries: an
|
||||
``httpx2.Client`` under dnspython 2.9+, an ``httpx.Client`` before (see
|
||||
``_doh_client_class()``).
|
||||
|
||||
The client is created on first use and reused afterwards, so DoH queries
|
||||
share TLS connections instead of renegotiating one per lookup. It is
|
||||
@@ -217,41 +247,43 @@ def _get_doh_session() -> httpx.Client:
|
||||
created under, because a ``fork()``-based worker pool (``n_procs``) would
|
||||
otherwise inherit — and concurrently use — the parent's sockets.
|
||||
|
||||
``httpx.Client`` defaults are what make this work behind a corporate
|
||||
proxy: ``trust_env=True`` honors ``HTTP_PROXY``/``HTTPS_PROXY``/
|
||||
``NO_PROXY`` and ``SSL_CERT_FILE``/``SSL_CERT_DIR``, and ``verify=True``
|
||||
keeps certificate verification on. Neither is overridden here.
|
||||
The client defaults (identical in httpx and httpx2) are what make this
|
||||
work behind a corporate proxy: ``trust_env=True`` honors ``HTTP_PROXY``/
|
||||
``HTTPS_PROXY``/``NO_PROXY`` and ``SSL_CERT_FILE``/``SSL_CERT_DIR``, and
|
||||
``verify=True`` keeps certificate verification on. Neither is overridden
|
||||
here.
|
||||
|
||||
Returns:
|
||||
httpx.Client: The shared DoH client for this process
|
||||
httpx.Client | httpx2.Client: The shared DoH client for this process
|
||||
"""
|
||||
global _DOH_SESSION, _DOH_SESSION_PID
|
||||
pid = os.getpid()
|
||||
if _DOH_SESSION is None or _DOH_SESSION_PID != pid:
|
||||
_DOH_SESSION = httpx.Client(http1=True, http2=True)
|
||||
_DOH_SESSION = _doh_client_class()(http1=True, http2=True)
|
||||
_DOH_SESSION_PID = pid
|
||||
return _DOH_SESSION
|
||||
|
||||
|
||||
class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
|
||||
"""
|
||||
A DNS over HTTPS nameserver that queries through a shared ``httpx``
|
||||
client.
|
||||
A DNS over HTTPS nameserver that queries through a shared HTTP client
|
||||
(httpx2 on dnspython 2.9+, httpx before).
|
||||
|
||||
dnspython's stock ``DoHNameserver`` calls ``dns.query.https()`` without a
|
||||
``session``, which makes that function build an ``httpx.Client`` with its
|
||||
own custom transport — and httpx only reads proxy environment variables
|
||||
when no transport is supplied (``allow_env_proxies = trust_env and
|
||||
transport is None``). Stock DoH therefore ignores ``HTTPS_PROXY``
|
||||
entirely — and a proxy is the only way out of the networks this exists
|
||||
for (https://github.com/domainaware/parsedmarc/issues/880).
|
||||
``session``, which makes that function build a client with its own
|
||||
custom transport — and httpx and httpx2 only read proxy environment
|
||||
variables when no transport is supplied (``allow_env_proxies =
|
||||
trust_env and transport is None``). Stock DoH therefore ignores
|
||||
``HTTPS_PROXY`` entirely — and a proxy is the only way out of the
|
||||
networks this exists for
|
||||
(https://github.com/domainaware/parsedmarc/issues/880).
|
||||
|
||||
Passing our own session instead gets environment proxies, environment CA
|
||||
configuration (``SSL_CERT_FILE``), and connection reuse across queries.
|
||||
``bootstrap_address`` is deliberately not passed: with a session, the DoH
|
||||
server's hostname is resolved by httpx — locally through the OS resolver,
|
||||
or by the proxy itself via ``CONNECT`` when one is configured — so no
|
||||
UDP/53 access is required.
|
||||
server's hostname is resolved by the HTTP client — locally through the
|
||||
OS resolver, or by the proxy itself via ``CONNECT`` when one is
|
||||
configured — so no UDP/53 access is required.
|
||||
"""
|
||||
|
||||
def query(
|
||||
@@ -279,6 +311,44 @@ class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
|
||||
)
|
||||
|
||||
|
||||
def _nameserver_port(entry: str, parts: SplitResult, kind: str) -> int | None:
|
||||
"""
|
||||
Returns the port of a parsed DNS over HTTPS or TLS nameserver URL.
|
||||
|
||||
``urlsplit()`` only validates the port when it is read, and its
|
||||
``ValueError`` quotes the text it rejected. A password containing an
|
||||
unencoded ``/``, ``?`` or ``#`` ends the authority early (RFC 3986
|
||||
section 3.2), which makes the start of the password that text
|
||||
(``https://doh:p4ss/w0rd@host`` gives ``Port could not be cast to
|
||||
integer value as 'p4ss'``). dnspython 2.9.1 reads the same port in
|
||||
``DoHNameserver.answer_port()`` and lets that error escape
|
||||
``Resolver.resolve()``. So the port is read here, and a bad one is
|
||||
reported with the entry's credentials masked and the original error
|
||||
suppressed.
|
||||
|
||||
Args:
|
||||
entry (str): The configured nameserver
|
||||
parts (SplitResult): ``urlsplit(entry)``
|
||||
kind (str): The transport name for the message, e.g.
|
||||
``"DNS over TLS"``
|
||||
|
||||
Returns:
|
||||
int | None: The port, or ``None`` when the URL has none
|
||||
|
||||
Raises:
|
||||
ValueError: The port is not an integer from 0 to 65535
|
||||
"""
|
||||
try:
|
||||
return parts.port
|
||||
except ValueError:
|
||||
pass
|
||||
raise ValueError(
|
||||
f"Invalid {kind} nameserver {_redact_one_url_credentials(entry)}: the "
|
||||
"port is not an integer from 0 to 65535 (percent-encode any /, ? or # "
|
||||
"in a password)"
|
||||
)
|
||||
|
||||
|
||||
def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
|
||||
"""
|
||||
Parses a ``tls://ip[:port][#hostname]`` nameserver entry.
|
||||
@@ -297,26 +367,25 @@ def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
|
||||
is not a literal IP address, or extra URL components
|
||||
"""
|
||||
parts = urlsplit(entry)
|
||||
try:
|
||||
port = parts.port
|
||||
except ValueError as e:
|
||||
# urlsplit only validates the port when it is accessed
|
||||
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: {e}") from e
|
||||
port = _nameserver_port(entry, parts, "DNS over TLS")
|
||||
# The entry may carry userinfo, which is rejected below, so the
|
||||
# messages quote it with its credentials masked
|
||||
shown = _redact_one_url_credentials(entry)
|
||||
if parts.username or parts.path or parts.query:
|
||||
# Catch tls://9.9.9.9/dns.quad9.net — a plausible slash-for-#
|
||||
# typo that would otherwise "work" with no certificate identity
|
||||
# and fail only at query time with an opaque TLS error
|
||||
raise ValueError(
|
||||
f"Invalid DNS over TLS nameserver {entry}: only "
|
||||
f"Invalid DNS over TLS nameserver {shown}: only "
|
||||
"tls://ip[:port][#hostname] is supported — the TLS certificate "
|
||||
"identity is given after #, not /"
|
||||
)
|
||||
address = parts.hostname
|
||||
if not address:
|
||||
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: missing IP address")
|
||||
raise ValueError(f"Invalid DNS over TLS nameserver {shown}: missing IP address")
|
||||
if not dns.inet.is_address(address):
|
||||
raise ValueError(
|
||||
f"Invalid DNS over TLS nameserver {entry}: {address} is not an IP "
|
||||
f"Invalid DNS over TLS nameserver {shown}: {address} is not an IP "
|
||||
"address. Use tls://ip[:port][#hostname], where the optional "
|
||||
"#hostname is the TLS certificate identity of the server"
|
||||
)
|
||||
@@ -334,11 +403,11 @@ def _nameservers_to_resolver_input(
|
||||
``dns.resolver.Resolver.nameservers`` accepts.
|
||||
|
||||
``https://`` entries become DNS over HTTPS nameservers that share this
|
||||
process's ``httpx`` client (so proxy and CA environment variables apply),
|
||||
and ``tls://ip[:port][#hostname]`` entries become DNS over TLS
|
||||
nameservers. Everything else — plain IPv4/IPv6 addresses — is passed
|
||||
through untouched, leaving dnspython to enrich and validate it exactly as
|
||||
before.
|
||||
process's HTTP client (httpx2 on dnspython 2.9+, httpx before), so proxy
|
||||
and CA environment variables apply, and ``tls://ip[:port][#hostname]``
|
||||
entries become DNS over TLS nameservers. Everything else — plain
|
||||
IPv4/IPv6 addresses — is passed through untouched, leaving dnspython to
|
||||
enrich and validate it exactly as before.
|
||||
|
||||
Args:
|
||||
nameservers (list[str]): The configured nameservers
|
||||
@@ -348,20 +417,25 @@ def _nameservers_to_resolver_input(
|
||||
in the configured order
|
||||
|
||||
Raises:
|
||||
ValueError: A ``tls://`` entry is malformed
|
||||
ValueError: A ``tls://`` entry is malformed, or an ``https://``
|
||||
entry has an unusable port
|
||||
"""
|
||||
resolver_input: list[str | dns.nameserver.Nameserver] = []
|
||||
for entry in nameservers:
|
||||
try:
|
||||
# urlsplit lowercases the scheme, so HTTPS:// and TLS:// work too
|
||||
scheme = urlsplit(entry).scheme
|
||||
parts = urlsplit(entry)
|
||||
except ValueError:
|
||||
# e.g. an unbalanced IPv6 bracket; let dnspython reject it with
|
||||
# its own message about what a nameserver may be
|
||||
scheme = ""
|
||||
if scheme == "https":
|
||||
resolver_input.append(entry)
|
||||
continue
|
||||
if parts.scheme == "https":
|
||||
# Validated now because dnspython reads the port only after a
|
||||
# query fails, and its error then quotes the rejected text
|
||||
_nameserver_port(entry, parts, "DNS over HTTPS")
|
||||
resolver_input.append(_SessionDoHNameserver(entry))
|
||||
elif scheme == "tls":
|
||||
elif parts.scheme == "tls":
|
||||
resolver_input.append(_parse_dot_nameserver(entry))
|
||||
else:
|
||||
resolver_input.append(entry)
|
||||
@@ -409,6 +483,10 @@ def query_dns(
|
||||
|
||||
Returns:
|
||||
list: A list of answers
|
||||
|
||||
Raises:
|
||||
ValueError: A nameserver is malformed. The message masks the
|
||||
nameserver's URL credentials.
|
||||
"""
|
||||
domain = str(domain).lower()
|
||||
record_type = record_type.upper()
|
||||
@@ -427,7 +505,17 @@ def query_dns(
|
||||
"2606:4700:4700::1111",
|
||||
"2606:4700:4700::1001",
|
||||
]
|
||||
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
|
||||
rejection = None
|
||||
try:
|
||||
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
|
||||
except ValueError as e:
|
||||
# dnspython's own rejection quotes the entry as configured
|
||||
# ("nameserver https://user:pw@[x is not ..."), credentials and all
|
||||
rejection = _mask_nameserver_secrets(str(e), nameservers)
|
||||
if rejection is not None:
|
||||
# Raised outside the except block so that the unmasked error is not
|
||||
# kept as __context__
|
||||
raise ValueError(rejection)
|
||||
# Cap per-query UDP timeout at 1s so dnspython retries within the
|
||||
# lifetime window on transient packet loss — otherwise with a single
|
||||
# nameserver and timeout == lifetime, one dropped UDP datagram consumes
|
||||
@@ -502,7 +590,9 @@ def get_reverse_dns(
|
||||
)[0]
|
||||
|
||||
except dns.exception.DNSException as e:
|
||||
logger.debug(f"get_reverse_dns({ip_address}) exception: {e}")
|
||||
# LifetimeTimeout and NoNameservers quote each nameserver URL
|
||||
message = _mask_nameserver_secrets(str(e), nameservers)
|
||||
logger.debug(f"get_reverse_dns({ip_address}) exception: {message}")
|
||||
|
||||
return hostname
|
||||
|
||||
@@ -654,7 +744,9 @@ def load_ip_db(
|
||||
|
||||
if not (offline or always_use_local_file):
|
||||
try:
|
||||
logger.debug(f"Trying to fetch IP database from {url}...")
|
||||
logger.debug(
|
||||
f"Trying to fetch IP database from {redact_url_credentials(url)}..."
|
||||
)
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
response = httpx.get(
|
||||
url, headers=headers, timeout=60, follow_redirects=True
|
||||
@@ -668,8 +760,8 @@ def load_ip_db(
|
||||
_IP_DB_PATH = cached_path
|
||||
logger.info("IP database updated successfully")
|
||||
return
|
||||
except httpx.HTTPError as e:
|
||||
logger.warning(f"Failed to fetch IP database: {e}")
|
||||
except (httpx.HTTPError, httpx.InvalidURL) as e:
|
||||
logger.warning(f"Failed to fetch IP database: {_describe_http_error(e)}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to save IP database: {e}")
|
||||
|
||||
@@ -1050,7 +1142,9 @@ def load_reverse_dns_map(
|
||||
|
||||
if not (offline or always_use_local_file):
|
||||
try:
|
||||
logger.debug(f"Trying to fetch reverse DNS map from {url}...")
|
||||
logger.debug(
|
||||
f"Trying to fetch reverse DNS map from {redact_url_credentials(url)}..."
|
||||
)
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
response = httpx.get(
|
||||
url, headers=headers, timeout=60, follow_redirects=True
|
||||
@@ -1059,8 +1153,10 @@ def load_reverse_dns_map(
|
||||
csv_file.write(response.text)
|
||||
csv_file.seek(0)
|
||||
load_csv(csv_file)
|
||||
except httpx.HTTPError as e:
|
||||
logger.warning(f"Failed to fetch reverse DNS map: {e}")
|
||||
except (httpx.HTTPError, httpx.InvalidURL) as e:
|
||||
logger.warning(
|
||||
f"Failed to fetch reverse DNS map: {_describe_http_error(e)}"
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Not a valid CSV file")
|
||||
csv_file.seek(0)
|
||||
@@ -1327,6 +1423,209 @@ def sanitize_for_log(value: str) -> str:
|
||||
)
|
||||
|
||||
|
||||
def redact_url_credentials(value: str | list[str]) -> str | list[str]:
|
||||
"""Masks the credentials in a URL or host string so it is safe to log.
|
||||
|
||||
A configured URL such as ``https://user:secret@host:9200`` carries a
|
||||
password; writing it to the log (CWE-532) leaks it to the console and
|
||||
to ``log_file``. When the userinfo contains a ``:``, only the part
|
||||
after the first ``:`` is replaced (``https://user:***@host:9200``);
|
||||
when it has no ``:`` the whole userinfo is replaced
|
||||
(``https://***@host``), since a bare token is itself the secret.
|
||||
|
||||
The authority is located with string methods only, so scheme-less
|
||||
values such as ``user:secret@localhost:9200`` (which ``urlsplit()``
|
||||
would misparse as scheme ``user``) are handled too.
|
||||
|
||||
A password containing an unencoded ``/``, ``?``, or ``#`` is not
|
||||
allowed by RFC 3986 (the authority ends at that character, so clients
|
||||
misparse such a URL too), and it leaves the authority with no ``@``.
|
||||
When the authority has no ``@`` but one appears later in the value,
|
||||
everything from the start of the authority through the *last* ``@``
|
||||
is replaced (``https://admin:p4ss/w0rd@host:3100`` becomes
|
||||
``https://***@host:3100``). The last ``@`` is used because the
|
||||
password may itself contain ``@``. This over-masks a URL that has no
|
||||
userinfo but has an ``@`` in its path or query:
|
||||
``https://host/path@file`` becomes ``https://***@file``, and
|
||||
``https://host/?redirect=https://u:p@evil`` becomes
|
||||
``https://***@evil``. That is acceptable because the result is only
|
||||
for the log. When the authority does contain an ``@``, only its
|
||||
userinfo is masked, so credentials in a later query-string value
|
||||
(``https://a:b@host/?redirect=https://u:p@evil``) are not detected,
|
||||
and neither is the rest of a password containing both ``@`` and one
|
||||
of ``/?#`` (``https://user:p@ss/word@host`` keeps ``ss/word``).
|
||||
|
||||
The value must be a single URL (or host) per string: split a
|
||||
comma-separated list first, or pass a ``list``. The authority is the
|
||||
first one found, so later entries in a comma-joined string are not
|
||||
masked.
|
||||
|
||||
Never raises; input that does not contain userinfo, and input that is
|
||||
not a ``str`` or ``list``, is returned unchanged.
|
||||
|
||||
Args:
|
||||
value (str | list[str]): A URL or host string, or a list of them
|
||||
|
||||
Returns:
|
||||
str | list[str]: ``value`` with credentials masked. A ``list`` is
|
||||
returned as a new list; the input list is not modified.
|
||||
"""
|
||||
if isinstance(value, list):
|
||||
return [
|
||||
_redact_one_url_credentials(item) if isinstance(item, str) else item
|
||||
for item in value
|
||||
]
|
||||
if not isinstance(value, str):
|
||||
return value
|
||||
return _redact_one_url_credentials(value)
|
||||
|
||||
|
||||
def _find_url_userinfo(value: str) -> tuple[int, int, bool] | None:
|
||||
"""Locates the userinfo of one URL or host string.
|
||||
|
||||
This is the lookup :func:`redact_url_credentials` masks and
|
||||
:func:`_url_secrets` extracts the secret from, shared so that the two
|
||||
cannot disagree about where the credentials are. It uses string
|
||||
methods only, so it is linear in the length of ``value`` and never
|
||||
raises.
|
||||
|
||||
Args:
|
||||
value (str): A URL or host string
|
||||
|
||||
Returns:
|
||||
tuple[int, int, bool] | None: ``None`` when no ``@`` follows the
|
||||
start of the authority. Otherwise ``(start, end, in_authority)``,
|
||||
where ``value[start:end]`` is the userinfo and ``value[end]`` is the
|
||||
``@`` after it. ``in_authority`` is ``True`` when that ``@`` is the
|
||||
last one inside the authority, and ``False`` when the authority has
|
||||
no ``@`` (a ``/``, ``?`` or ``#`` in the password ends it early, as
|
||||
in ``https://admin:p4ss/w0rd@host``) and the ``@`` is the last one
|
||||
in the whole value; the password may itself contain ``@``.
|
||||
"""
|
||||
# A "://" counts as the scheme separator only when nothing before it
|
||||
# could end an authority: in "user:secret@host/x?r=http://y" the "://"
|
||||
# is inside the query, and the authority starts at offset 0.
|
||||
scheme_end = value.find("://")
|
||||
if scheme_end != -1 and not any(c in value[:scheme_end] for c in "/?#@"):
|
||||
start = scheme_end + 3
|
||||
elif value.startswith("//"):
|
||||
start = 2
|
||||
else:
|
||||
start = 0
|
||||
end = len(value)
|
||||
for delimiter in "/?#":
|
||||
position = value.find(delimiter, start)
|
||||
if position != -1:
|
||||
end = min(end, position)
|
||||
at = value.rfind("@", start, end)
|
||||
if at != -1:
|
||||
return start, at, True
|
||||
at = value.rfind("@", start)
|
||||
if at != -1:
|
||||
return start, at, False
|
||||
return None
|
||||
|
||||
|
||||
def _redact_one_url_credentials(value: str) -> str:
|
||||
"""Masks the userinfo of one URL or host string; see
|
||||
:func:`redact_url_credentials`."""
|
||||
found = _find_url_userinfo(value)
|
||||
if found is None:
|
||||
return value
|
||||
start, at, in_authority = found
|
||||
if not in_authority:
|
||||
# Which "/", "?" or "#" ended the authority is unknown, so mask
|
||||
# everything through the "@" rather than guess where the user
|
||||
# name stops.
|
||||
return f"{value[:start]}***{value[at:]}"
|
||||
user, colon, _ = value[start:at].partition(":")
|
||||
masked = f"{user}:***" if colon else "***"
|
||||
return f"{value[:start]}{masked}{value[at:]}"
|
||||
|
||||
|
||||
def _url_secrets(value: str) -> list[str]:
|
||||
"""Returns the secret text in one URL or host string's userinfo.
|
||||
|
||||
The secret is the userinfo after its first ``:``, or the whole userinfo
|
||||
when it has no ``:`` (a bare token is itself the secret), located by
|
||||
:func:`_find_url_userinfo`. When the secret contains an unencoded
|
||||
``/``, ``?`` or ``#``, its text before the first of them is returned
|
||||
too: ``urlsplit()`` (and so dnspython) and httpx read that part as the
|
||||
port and quote it in their errors (``Port could not be cast to integer
|
||||
value as 'p4ss'``). Empty strings are never returned, so the result is
|
||||
safe to use as ``str.replace()`` targets. Never raises.
|
||||
|
||||
Args:
|
||||
value (str): A URL or host string
|
||||
|
||||
Returns:
|
||||
list[str]: The secret and, if it differs and is not empty, its
|
||||
prefix before the first ``/``, ``?`` or ``#``, longest first; an
|
||||
empty list when ``value`` has no userinfo or an empty secret.
|
||||
"""
|
||||
found = _find_url_userinfo(value)
|
||||
if found is None:
|
||||
return []
|
||||
start, at, _ = found
|
||||
user, colon, password = value[start:at].partition(":")
|
||||
secret = password if colon else user
|
||||
prefix_end = len(secret)
|
||||
for delimiter in "/?#":
|
||||
position = secret.find(delimiter)
|
||||
if position != -1:
|
||||
prefix_end = min(prefix_end, position)
|
||||
return [text for text in dict.fromkeys((secret, secret[:prefix_end])) if text]
|
||||
|
||||
|
||||
def _mask_nameserver_secrets(text: str, nameservers: list[str] | None) -> str:
|
||||
"""Masks the credentials of the configured nameserver URLs in free text.
|
||||
|
||||
A DNS query error (dnspython's ``LifetimeTimeout``, a DNS over HTTPS
|
||||
status error) quotes each failing nameserver URL, sometimes several
|
||||
times in different forms, and ``redact_url_credentials()`` masks only
|
||||
one URL per string. So each configured entry is replaced by its masked
|
||||
form, and each secret :func:`_url_secrets` finds in it (the password,
|
||||
and its start when a ``/``, ``?`` or ``#`` in it gets read as the port)
|
||||
is replaced by ``***`` wherever it still appears. A short secret masks
|
||||
every occurrence of that text, including ones that are not the
|
||||
password; the result is only for error messages and logs. Never
|
||||
raises.
|
||||
|
||||
Args:
|
||||
text (str): The text to mask, typically ``str()`` of an exception
|
||||
nameservers (list[str] | None): The configured nameservers
|
||||
|
||||
Returns:
|
||||
str: ``text`` with the nameservers' credentials masked
|
||||
"""
|
||||
for entry in nameservers or []:
|
||||
text = text.replace(entry, _redact_one_url_credentials(entry))
|
||||
for secret in _url_secrets(entry):
|
||||
text = text.replace(secret, "***")
|
||||
return text
|
||||
|
||||
|
||||
def _describe_http_error(e: httpx.HTTPError | httpx.InvalidURL) -> str:
|
||||
"""Describes an httpx error for the log without leaking URL credentials.
|
||||
|
||||
``str(HTTPStatusError)`` embeds the server's reason phrase before the
|
||||
URL, so a phrase containing ``/``, ``?``, ``#`` or ``@`` defeats
|
||||
string-based redaction of the message. A status error is therefore
|
||||
rebuilt from its structured parts. Transport errors (httpx 0.28) do not
|
||||
embed the request URL, so their message is used as is. ``InvalidURL``
|
||||
(not an ``HTTPError``) is raised while httpx parses the URL, and its
|
||||
message quotes the component it rejected -- part of a password
|
||||
containing ``/``, ``?`` or ``#`` ends up as the "port" -- so it is
|
||||
replaced by a fixed message.
|
||||
"""
|
||||
if isinstance(e, httpx.InvalidURL):
|
||||
return "invalid URL (details withheld because they may include credentials)"
|
||||
if isinstance(e, httpx.HTTPStatusError):
|
||||
url = redact_url_credentials(str(e.request.url))
|
||||
return f"HTTP {e.response.status_code} for url '{url}'"
|
||||
return f"{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
def get_filename_safe_string(string: str | None) -> str:
|
||||
"""
|
||||
Converts a string to a string that is safe to use as a filename
|
||||
|
||||
@@ -62,6 +62,12 @@ class WebhookClient(object):
|
||||
self.session.post(webhook_url, data=payload, timeout=self.timeout)
|
||||
else:
|
||||
self.session.post(webhook_url, content=payload, timeout=self.timeout)
|
||||
except httpx.InvalidURL:
|
||||
# httpx quotes part of the URL, which can be part of a password
|
||||
logger.error(
|
||||
"Webhook Error: invalid webhook URL (details withheld because "
|
||||
"they may include credentials)"
|
||||
)
|
||||
except Exception as error_:
|
||||
logger.error(f"Webhook Error: {error_.__str__()}")
|
||||
|
||||
|
||||
+6
-3
@@ -43,9 +43,12 @@ requires-python = ">=3.10"
|
||||
# the Elasticsearch, OpenSearch, Kafka, AWS, Azure, Gmail, and Microsoft
|
||||
# Graph SDKs (see issue #883).
|
||||
dependencies = [
|
||||
# The [doh] extra supplies the httpx/h2 floors DNS over HTTPS needs;
|
||||
# 2.7.0 is the floor verified against the dns.nameserver and
|
||||
# dns.query.https(session=...) APIs utils.py builds on.
|
||||
# The [doh] extra supplies the HTTP client DNS over HTTPS needs (httpx,
|
||||
# httpcore and h2 on dnspython 2.7 and 2.8; httpx2, httpcore2 and h2 on
|
||||
# 2.9+; utils.py builds its session from whichever client library
|
||||
# dnspython imported). 2.7.0 is the floor verified against the
|
||||
# dns.nameserver and dns.query.https(session=...) APIs utils.py builds
|
||||
# on.
|
||||
"dnspython[doh]>=2.7.0",
|
||||
"expiringdict>=1.1.4",
|
||||
# The runtime HTTP library (utils.py fetches, webhook and Splunk HEC
|
||||
|
||||
+741
-7
@@ -6,6 +6,7 @@ import importlib.abc
|
||||
import importlib.machinery
|
||||
import gzip
|
||||
import io
|
||||
import importlib.util
|
||||
import json
|
||||
import logging
|
||||
import mailbox
|
||||
@@ -15,6 +16,7 @@ import signal
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import traceback
|
||||
import unittest
|
||||
import zipfile
|
||||
from collections.abc import Iterator, Sequence
|
||||
@@ -27,6 +29,11 @@ from types import ModuleType, SimpleNamespace
|
||||
from typing import cast
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import dns.message
|
||||
import dns.rcode
|
||||
import dns.rdata
|
||||
import dns.rdataclass
|
||||
import dns.rdatatype
|
||||
import httpx
|
||||
import opensearchpy
|
||||
from azure.core.exceptions import ClientAuthenticationError
|
||||
@@ -2923,6 +2930,11 @@ client_assertion = s3cret-signed-jwt-assertion
|
||||
self._run_main(cfg_path, "--debug")
|
||||
|
||||
parsedmarc_logger = logging.getLogger("parsedmarc.log")
|
||||
# dnspython 2.9+ logs DNS over HTTPS traffic under httpx2 and
|
||||
# httpcore2 (httpx and httpcore on 2.7 and 2.8), so all four must
|
||||
# follow --debug.
|
||||
for wanted_name in ("httpx", "httpcore", "httpx2", "httpcore2"):
|
||||
self.assertIn(wanted_name, parsedmarc.cli._DEPENDENCY_LOGGERS)
|
||||
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
|
||||
dep = logging.getLogger(name)
|
||||
self.assertEqual(dep.level, logging.DEBUG, name)
|
||||
@@ -5442,6 +5454,323 @@ def _es_request_without_network(client, method, path, **kwargs):
|
||||
return {"count": 0}
|
||||
|
||||
|
||||
def _os_request_without_network(transport, method, url, *args, **kwargs):
|
||||
"""Stand-in for ``opensearchpy.Transport.perform_request``: answers
|
||||
"no such index" / zero documents like ``_es_request_without_network``."""
|
||||
if method == "HEAD":
|
||||
return {}
|
||||
return {"count": 0}
|
||||
|
||||
|
||||
class TestInitOutputClientsDoesNotLogCredentials(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: with --debug, the "Initializing ... client"
|
||||
lines must mask credentials embedded in configured URLs."""
|
||||
|
||||
SECRET = "s3cretPW"
|
||||
|
||||
def _init(self, opts):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
clients = parsedmarc.cli._init_output_clients(opts)
|
||||
self.addCleanup(parsedmarc.cli._close_output_clients, clients)
|
||||
return "\n".join(logs.output)
|
||||
|
||||
def test_hec_url_password_is_masked(self):
|
||||
opts = _output_client_opts(
|
||||
hec=f"https://hecuser:{self.SECRET}@hec.example:8088",
|
||||
hec_token="tok",
|
||||
hec_index="email",
|
||||
)
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
self.assertIn("url=https://hecuser:***@hec.example:8088", output)
|
||||
|
||||
def test_hec_malformed_url_password_is_masked(self):
|
||||
"""A "/", "?" or "#" in the password ends the authority before the
|
||||
"@", which left the --debug line unmasked; it is now masked through
|
||||
the last "@"."""
|
||||
for password in ("p4ssw0rd/extra", "p4ss?w0rd", "p4ss#w0rd"):
|
||||
with self.subTest(password=password):
|
||||
opts = _output_client_opts(
|
||||
hec=f"https://hecuser:{password}@hec.example:8088",
|
||||
hec_token="tok",
|
||||
hec_index="email",
|
||||
)
|
||||
output = self._init(opts)
|
||||
self.assertNotIn("p4ss", output)
|
||||
self.assertIn("url=https://***@hec.example:8088", output)
|
||||
|
||||
def test_loki_url_password_is_masked(self):
|
||||
opts = _output_client_opts(
|
||||
loki_url=f"https://lokiuser:{self.SECRET}@loki.example:3100",
|
||||
loki_timestamp="ingest",
|
||||
loki_timeout=1.0,
|
||||
loki_batch_size=100,
|
||||
)
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
self.assertIn("url=https://lokiuser:***@loki.example:3100", output)
|
||||
|
||||
def test_loki_malformed_url_error_has_no_password(self):
|
||||
"""A password with an unencoded "/" ends the URL's authority early
|
||||
(RFC 3986 section 3.2), so httpx raises ``InvalidURL("Invalid port:
|
||||
'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py), whose text and chained
|
||||
ValueError both repeat it. Neither the startup error nor the
|
||||
``--debug`` "Initializing Loki client" line may show any part of
|
||||
the password."""
|
||||
malformed = (
|
||||
("https://admin:p4ssw0rd/extra@loki.example:3100", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@loki.example:3100", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@loki.example:3100", "p4ss"),
|
||||
)
|
||||
for url, secret in malformed:
|
||||
with self.subTest(url=url):
|
||||
opts = _output_client_opts(
|
||||
loki_url=url,
|
||||
loki_timestamp="ingest",
|
||||
loki_timeout=1.0,
|
||||
loki_batch_size=100,
|
||||
)
|
||||
with (
|
||||
self.assertRaises(RuntimeError) as ctx,
|
||||
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
|
||||
):
|
||||
parsedmarc.cli._init_output_clients(opts)
|
||||
self.assertEqual(
|
||||
str(ctx.exception),
|
||||
"Loki: Invalid Loki URL (details withheld because they may "
|
||||
"include credentials)",
|
||||
)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn(secret, trace)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn(secret, output)
|
||||
self.assertIn("url=https://***@loki.example:3100", output)
|
||||
|
||||
def test_elasticsearch_hosts_password_is_masked(self):
|
||||
connections = parsedmarc.elastic.connections
|
||||
_isolate_default_alias(self, connections)
|
||||
opts = _output_client_opts(
|
||||
save_aggregate=True,
|
||||
elasticsearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
|
||||
elasticsearch_timeout=1.0,
|
||||
)
|
||||
with (
|
||||
patch.object(Elasticsearch, "perform_request", _es_request_without_network),
|
||||
patch(
|
||||
"parsedmarc.elastic.connections.create_connection",
|
||||
wraps=connections.create_connection,
|
||||
) as mock_conn,
|
||||
):
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
self.assertIn("Initializing Elasticsearch client", output)
|
||||
# Redaction is log-only: the client still gets the real credentials
|
||||
self.assertEqual(
|
||||
mock_conn.call_args.kwargs["hosts"],
|
||||
[f"https://u:{self.SECRET}@127.0.0.1:9299"],
|
||||
)
|
||||
self.assertIn("https://u:***@127.0.0.1:9299", output)
|
||||
|
||||
def test_opensearch_hosts_password_is_masked_in_cli_and_set_hosts(self):
|
||||
connections = opensearch_module.connections
|
||||
_isolate_default_alias(self, connections)
|
||||
opts = _output_client_opts(
|
||||
save_aggregate=True,
|
||||
opensearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
|
||||
opensearch_timeout=1.0,
|
||||
)
|
||||
with patch.object(
|
||||
opensearchpy.Transport, "perform_request", _os_request_without_network
|
||||
):
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
lines = output.splitlines()
|
||||
for marker in ("Initializing OpenSearch client", "Connecting to OpenSearch"):
|
||||
with self.subTest(marker=marker):
|
||||
matching = [line for line in lines if marker in line]
|
||||
self.assertEqual(len(matching), 1)
|
||||
self.assertIn("https://u:***@127.0.0.1:9299", matching[0])
|
||||
|
||||
|
||||
class TestHttpxLogRedaction(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: httpx logs the full request URL at INFO."""
|
||||
|
||||
def setUp(self):
|
||||
saved = {}
|
||||
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
|
||||
dep = logging.getLogger(name)
|
||||
saved[name] = (
|
||||
dep.level,
|
||||
list(dep.handlers),
|
||||
dep.propagate,
|
||||
list(dep.filters),
|
||||
)
|
||||
|
||||
def restore():
|
||||
for name, (level, handlers, propagate, filters) in saved.items():
|
||||
dep = logging.getLogger(name)
|
||||
dep.setLevel(level)
|
||||
dep.handlers = handlers
|
||||
dep.propagate = propagate
|
||||
dep.filters = filters
|
||||
|
||||
self.addCleanup(restore)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
|
||||
def _client(self, seen):
|
||||
def handler(request):
|
||||
seen.append(request)
|
||||
return httpx.Response(200)
|
||||
|
||||
return httpx.Client(transport=httpx.MockTransport(handler))
|
||||
|
||||
def test_userinfo_password_masked_but_auth_still_sent(self):
|
||||
seen = []
|
||||
with self._client(seen) as client:
|
||||
with self.assertLogs("httpx", level="INFO") as logs:
|
||||
client.post("https://user:secret@hook.example/x", content=b"{}")
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("secret", output)
|
||||
self.assertIn("https://user:***@hook.example/x", output)
|
||||
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
|
||||
|
||||
def test_ipinfo_token_query_parameter_masked(self):
|
||||
seen = []
|
||||
with self._client(seen) as client:
|
||||
with self.assertLogs("httpx", level="INFO") as logs:
|
||||
client.get(
|
||||
"https://api.ipinfo.io/lite/8.8.8.8", params={"token": "TOKSECRET"}
|
||||
)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("TOKSECRET", output)
|
||||
self.assertIn("api.ipinfo.io/lite/8.8.8.8", output)
|
||||
self.assertIn("token=***", output)
|
||||
self.assertEqual(seen[0].url.params["token"], "TOKSECRET")
|
||||
|
||||
def test_token_mask_leaves_literal_encoded_stars_in_path_alone(self):
|
||||
"""A path containing a literal ``%2A%2A%2A`` is not rewritten; only
|
||||
the token value is masked, with and without other query params and
|
||||
a fragment."""
|
||||
for url, expected in (
|
||||
(
|
||||
"https://h.example/a%2A%2A%2Ab?token=TOKSECRET",
|
||||
"https://h.example/a%2A%2A%2Ab?token=***",
|
||||
),
|
||||
(
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET",
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=***",
|
||||
),
|
||||
(
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET#f",
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=***#f",
|
||||
),
|
||||
):
|
||||
with self.subTest(url=url):
|
||||
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(
|
||||
httpx.URL(url)
|
||||
)
|
||||
self.assertNotIn("TOKSECRET", result)
|
||||
self.assertEqual(result, expected)
|
||||
|
||||
def test_reconfiguring_does_not_stack_filters(self):
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
|
||||
installed = [
|
||||
f
|
||||
for f in logging.getLogger("httpx").filters
|
||||
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
|
||||
]
|
||||
self.assertEqual(len(installed), 1)
|
||||
|
||||
def test_non_tuple_args_and_other_arg_types_pass_through(self):
|
||||
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "x %d", (5,), None)
|
||||
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
|
||||
self.assertEqual(record.args, (5,))
|
||||
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "plain", None, None)
|
||||
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
|
||||
self.assertIsNone(record.args)
|
||||
|
||||
|
||||
@unittest.skipUnless(
|
||||
importlib.util.find_spec("httpx2") is not None,
|
||||
"httpx2 (dnspython 2.9+ DNS over HTTPS) is not installed",
|
||||
)
|
||||
class TestHttpx2LogRedaction(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: dnspython 2.9+ does DNS over HTTPS through
|
||||
httpx2, which logs the request URL under the ``httpx2`` logger with an
|
||||
``httpx2.URL``, a different class from ``httpx.URL``."""
|
||||
|
||||
def setUp(self):
|
||||
import httpx2 # pyright: ignore[reportMissingImports]
|
||||
|
||||
self.httpx2 = httpx2
|
||||
saved = {}
|
||||
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
|
||||
dep = logging.getLogger(name)
|
||||
saved[name] = (
|
||||
dep.level,
|
||||
list(dep.handlers),
|
||||
dep.propagate,
|
||||
list(dep.filters),
|
||||
)
|
||||
|
||||
def restore():
|
||||
for name, (level, handlers, propagate, filters) in saved.items():
|
||||
dep = logging.getLogger(name)
|
||||
dep.setLevel(level)
|
||||
dep.handlers = handlers
|
||||
dep.propagate = propagate
|
||||
dep.filters = filters
|
||||
|
||||
self.addCleanup(restore)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
|
||||
def test_httpx2_client_request_line_masks_userinfo(self):
|
||||
"""httpx2 itself emits the line; the credential still reaches the
|
||||
transport."""
|
||||
seen = []
|
||||
|
||||
def handler(request):
|
||||
seen.append(request)
|
||||
return self.httpx2.Response(200)
|
||||
|
||||
with self.httpx2.Client(transport=self.httpx2.MockTransport(handler)) as client:
|
||||
with self.assertLogs("httpx2", level="INFO") as logs:
|
||||
client.post("https://doh:S3CRET@doh.example/dns-query", content=b"{}")
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("S3CRET", output)
|
||||
self.assertIn("https://doh:***@doh.example/dns-query", output)
|
||||
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
|
||||
|
||||
def test_httpx2_url_token_query_parameter_masked(self):
|
||||
url = self.httpx2.URL("https://h.example/a?x=1&token=TOKSECRET#f")
|
||||
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(url)
|
||||
self.assertEqual(result, "https://h.example/a?x=1&token=***#f")
|
||||
|
||||
def test_httpx2_logger_record_args_masked_and_ints_untouched(self):
|
||||
record = logging.LogRecord(
|
||||
"httpx2",
|
||||
logging.INFO,
|
||||
"f",
|
||||
1,
|
||||
"%s %d",
|
||||
(self.httpx2.URL("https://u:S3CRET@h.example/x"), 200),
|
||||
None,
|
||||
)
|
||||
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
|
||||
self.assertEqual(record.args, ("https://u:***@h.example/x", 200))
|
||||
|
||||
def test_reconfiguring_attaches_one_filter_to_httpx2(self):
|
||||
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
installed = [
|
||||
f
|
||||
for f in logging.getLogger("httpx2").filters
|
||||
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
|
||||
]
|
||||
self.assertEqual(len(installed), 1)
|
||||
|
||||
|
||||
class TestInitOutputClientsRollback(unittest.TestCase):
|
||||
"""_init_output_clients() is all-or-nothing against the search backends'
|
||||
process-wide ``default`` connection alias.
|
||||
@@ -7519,7 +7848,8 @@ class TestParseConfigGeneral(unittest.TestCase):
|
||||
"""A nameservers list may mix DNS over HTTPS URLs and DNS over TLS
|
||||
entries with plain IP addresses (issue #880); _parse_config splits
|
||||
and strips them like any other list value, and hands them to the DNS
|
||||
pre-flight check (mocked here so no network is needed)."""
|
||||
pre-flight check (dns.query.https is mocked so no network is
|
||||
needed)."""
|
||||
from parsedmarc.cli import _parse_config
|
||||
|
||||
cp = _config_with(
|
||||
@@ -7533,18 +7863,223 @@ class TestParseConfigGeneral(unittest.TestCase):
|
||||
},
|
||||
)
|
||||
opts = _opts()
|
||||
with patch(
|
||||
"parsedmarc.cli.get_reverse_dns", return_value="one.one.one.one"
|
||||
) as mock_reverse_dns:
|
||||
queried: list[tuple[str, str]] = []
|
||||
|
||||
def responder(request, where, *args, **kwargs):
|
||||
queried.append((where, request.question[0].to_text()))
|
||||
response = dns.message.make_response(request)
|
||||
rrset = response.find_rrset(
|
||||
response.answer,
|
||||
request.question[0].name,
|
||||
dns.rdataclass.IN,
|
||||
dns.rdatatype.PTR,
|
||||
create=True,
|
||||
)
|
||||
rrset.add(dns.rdata.from_text("IN", "PTR", "one.one.one.one."), 300)
|
||||
return response
|
||||
|
||||
with patch("dns.query.https", side_effect=responder):
|
||||
_parse_config(cp, opts)
|
||||
self.assertEqual(
|
||||
opts.nameservers,
|
||||
["https://cloudflare-dns.com/dns-query", "tls://9.9.9.9#dns.quad9.net"],
|
||||
)
|
||||
# The first (DoH) nameserver answered the pre-flight PTR query for
|
||||
# dns_test_address, so the DoT one was never needed
|
||||
self.assertEqual(
|
||||
mock_reverse_dns.call_args.kwargs["nameservers"], opts.nameservers
|
||||
queried,
|
||||
[
|
||||
(
|
||||
"https://cloudflare-dns.com/dns-query",
|
||||
"1.1.1.1.in-addr.arpa. IN PTR",
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
def test_general_nameservers_pre_flight_reports_the_resolver_error(self):
|
||||
"""A nameserver that fails every query is reported with the
|
||||
resolver's own error, not as a missing PTR record. Issue #935: under
|
||||
dnspython 2.9 every DoH query raised "session parameter must be an
|
||||
httpx2.Client", and the pre-flight check said "no PTR record for
|
||||
1.1.1.1", because get_reverse_dns() turns every DNS error into
|
||||
None."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": "1.1.1.1",
|
||||
"dns_timeout": "0.2",
|
||||
"nameservers": "https://dns.example/dns-query",
|
||||
},
|
||||
)
|
||||
error = ValueError("session parameter must be an httpx2.Client")
|
||||
with patch("dns.query.https", side_effect=error):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
message = str(ctx.exception)
|
||||
self.assertIn("DNS pre-flight check failed", message)
|
||||
self.assertIn("session parameter must be an httpx2.Client", message)
|
||||
self.assertNotIn("no PTR record", message)
|
||||
|
||||
def test_general_nameservers_pre_flight_reports_a_missing_ptr_record(self):
|
||||
"""When the nameserver answers but has no PTR record for
|
||||
dns_test_address — NXDOMAIN, or NOERROR with an empty answer
|
||||
section — the pre-flight error says so and names the nameservers."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
def nxdomain(request, *args, **kwargs):
|
||||
response = dns.message.make_response(request)
|
||||
response.set_rcode(dns.rcode.NXDOMAIN)
|
||||
return response
|
||||
|
||||
def no_answer(request, *args, **kwargs):
|
||||
return dns.message.make_response(request)
|
||||
|
||||
for name, responder in (("NXDOMAIN", nxdomain), ("NoAnswer", no_answer)):
|
||||
with self.subTest(name):
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": "1.1.1.1",
|
||||
"dns_timeout": "2.0",
|
||||
"nameservers": "https://dns.example/dns-query",
|
||||
},
|
||||
)
|
||||
with patch("dns.query.https", side_effect=responder):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
self.assertIn(
|
||||
"DNS pre-flight check failed: no PTR record for 1.1.1.1 "
|
||||
"from ['https://dns.example/dns-query']",
|
||||
str(ctx.exception),
|
||||
)
|
||||
|
||||
def test_general_nameservers_pre_flight_errors_mask_url_credentials(self):
|
||||
"""GHSA-hjr3-rx4c-24cv: the pre-flight error is logged at the default
|
||||
level, so neither the "no PTR record ... from <nameservers>" list nor
|
||||
a transport error that quotes the DoH request URL may carry the
|
||||
nameserver URL's password, and the chained exception (printed on a
|
||||
crash) must not repeat it."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
url = "https://doh:S3CRETPW@dns.example/dns-query"
|
||||
|
||||
def nxdomain(request, *args, **kwargs):
|
||||
response = dns.message.make_response(request)
|
||||
response.set_rcode(dns.rcode.NXDOMAIN)
|
||||
return response
|
||||
|
||||
def transport_error(request, *args, **kwargs):
|
||||
raise RuntimeError(f"Server error '502' for url '{url}'")
|
||||
|
||||
for name, responder, expected in (
|
||||
("NXDOMAIN", nxdomain, "https://doh:***@dns.example/dns-query"),
|
||||
("transport", transport_error, "https://doh:***@dns.example/dns-query"),
|
||||
):
|
||||
with self.subTest(name):
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": "1.1.1.1",
|
||||
"dns_timeout": "2.0",
|
||||
"nameservers": url,
|
||||
},
|
||||
)
|
||||
with patch("dns.query.https", side_effect=responder):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn("S3CRETPW", str(ctx.exception))
|
||||
self.assertNotIn("S3CRETPW", trace)
|
||||
self.assertIn(expected, str(ctx.exception))
|
||||
|
||||
def test_general_nameservers_pre_flight_masks_malformed_url_passwords(self):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password containing an unencoded ``/``,
|
||||
``?`` or ``#`` ends the URL authority early (RFC 3986 section 3.2),
|
||||
so ``urlsplit()`` reports no password and reads its start as the
|
||||
port. dnspython 2.9.1 then raised ``ValueError("Port could not be
|
||||
cast to integer value as 'p4ss'")`` from
|
||||
``DoHNameserver.answer_port()`` (``dns/nameserver.py``), and
|
||||
``_parse_dot_nameserver()`` quoted the same urlsplit message for a
|
||||
``tls://`` entry, so the pre-flight error printed the start of the
|
||||
password. Neither the password nor that prefix may reach the
|
||||
message or the printed traceback. No query is sent: every entry is
|
||||
rejected before dnspython contacts a server, which the patched
|
||||
transports would turn into a different error."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
for url, expected in (
|
||||
(
|
||||
"https://doh:p4ss/w0rd@dns.example/dns-query",
|
||||
"https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"https://doh:p4ss?w0rd@dns.example/dns-query",
|
||||
"https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"https://doh:p4ss#w0rd@dns.example/dns-query",
|
||||
"https://***@dns.example/dns-query",
|
||||
),
|
||||
("tls://dot:p4ss/w0rd@1.1.1.1", "tls://***@1.1.1.1"),
|
||||
):
|
||||
with self.subTest(url=url):
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": "1.1.1.1",
|
||||
"dns_timeout": "0.2",
|
||||
"nameservers": url,
|
||||
},
|
||||
)
|
||||
unreachable = OSError("no query may be sent")
|
||||
with (
|
||||
patch("dns.query.https", side_effect=unreachable),
|
||||
patch("dns.query.tls", side_effect=unreachable),
|
||||
):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
message = str(ctx.exception)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
for secret in ("p4ss", "w0rd"):
|
||||
self.assertNotIn(secret, message)
|
||||
self.assertNotIn(secret, trace)
|
||||
self.assertIn(expected, message)
|
||||
self.assertIn("DNS pre-flight check failed: ValueError: ", message)
|
||||
self.assertNotIn("no query may be sent", message)
|
||||
|
||||
def test_general_nameservers_pre_flight_rejects_invalid_test_address(self):
|
||||
"""An unparseable dns_test_address is reported by setting name and
|
||||
value (dns.reversename.from_address raises dns.exception.SyntaxError,
|
||||
"Text input is malformed.", for non-IP text), and no DNS query is
|
||||
attempted."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
for address in ("not-an-ip", "1.1.1"):
|
||||
with self.subTest(address):
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": address,
|
||||
"dns_timeout": "2.0",
|
||||
"nameservers": "https://dns.example/dns-query",
|
||||
},
|
||||
)
|
||||
with (
|
||||
patch("dns.query.https") as https,
|
||||
patch("dns.query.udp") as udp,
|
||||
):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
self.assertEqual(
|
||||
f"DNS pre-flight check failed: dns_test_address {address!r} "
|
||||
"is not a valid IP address",
|
||||
str(ctx.exception),
|
||||
)
|
||||
https.assert_not_called()
|
||||
udp.assert_not_called()
|
||||
|
||||
def test_general_nameservers_malformed_dot_entry_fails_pre_flight(self):
|
||||
"""A malformed DNS over TLS nameservers entry raises a
|
||||
ConfigurationError at startup, before any mailbox work begins. No
|
||||
@@ -7837,6 +8372,175 @@ class TestParseConfigOpenSearch(unittest.TestCase):
|
||||
_parse_config(cp, _opts())
|
||||
|
||||
|
||||
class TestParseConfigUseDataStreams(unittest.TestCase):
|
||||
"""``use_data_streams`` in [elasticsearch] and [opensearch] (issue
|
||||
#842), and the warning that ``monthly_indexes`` is ignored with it."""
|
||||
|
||||
SECTIONS = ("elasticsearch", "opensearch")
|
||||
|
||||
def test_option_parsed_for_each_section(self):
|
||||
from parsedmarc.cli import _parse_config
|
||||
|
||||
for section in self.SECTIONS:
|
||||
for raw, expected in (("true", True), ("false", False)):
|
||||
with self.subTest(section=section, value=raw):
|
||||
cp = _config_with(
|
||||
section, {"hosts": "h:9200", "use_data_streams": raw}
|
||||
)
|
||||
opts = _opts()
|
||||
_parse_config(cp, opts)
|
||||
self.assertIs(
|
||||
getattr(opts, f"{section}_use_data_streams"), expected
|
||||
)
|
||||
|
||||
def test_option_absent_leaves_attribute_unset(self):
|
||||
"""The default comes from the CLI's Namespace defaults, which the
|
||||
real CLI builds before _parse_config runs."""
|
||||
from parsedmarc.cli import _parse_config
|
||||
|
||||
for section in self.SECTIONS:
|
||||
with self.subTest(section=section):
|
||||
cp = _config_with(section, {"hosts": "h:9200"})
|
||||
opts = _opts()
|
||||
_parse_config(cp, opts)
|
||||
self.assertFalse(hasattr(opts, f"{section}_use_data_streams"))
|
||||
|
||||
def test_monthly_indexes_with_data_streams_warns(self):
|
||||
from parsedmarc.cli import _parse_config
|
||||
|
||||
for section in self.SECTIONS:
|
||||
with self.subTest(section=section):
|
||||
cp = _config_with(
|
||||
section,
|
||||
{
|
||||
"hosts": "h:9200",
|
||||
"monthly_indexes": "true",
|
||||
"use_data_streams": "true",
|
||||
},
|
||||
)
|
||||
opts = _opts()
|
||||
with self.assertLogs("parsedmarc.log", level="WARNING") as cm:
|
||||
_parse_config(cp, opts)
|
||||
self.assertEqual(len(cm.records), 1)
|
||||
self.assertIn(f"[{section}] monthly_indexes", cm.output[0])
|
||||
self.assertIn("use_data_streams", cm.output[0])
|
||||
# Still parsed; the save functions ignore it in stream mode.
|
||||
self.assertTrue(getattr(opts, f"{section}_monthly_indexes"))
|
||||
|
||||
def test_no_warning_unless_both_are_enabled(self):
|
||||
from parsedmarc.cli import _parse_config
|
||||
|
||||
combinations = (
|
||||
{"monthly_indexes": "true"},
|
||||
{"use_data_streams": "true"},
|
||||
{"monthly_indexes": "true", "use_data_streams": "false"},
|
||||
{"monthly_indexes": "false", "use_data_streams": "true"},
|
||||
)
|
||||
for section in self.SECTIONS:
|
||||
for settings in combinations:
|
||||
with self.subTest(section=section, settings=settings):
|
||||
cp = _config_with(section, {"hosts": "h:9200", **settings})
|
||||
with self.assertNoLogs("parsedmarc.log", level="WARNING"):
|
||||
_parse_config(cp, _opts())
|
||||
|
||||
|
||||
class TestUseDataStreamsPassthrough(unittest.TestCase):
|
||||
"""Each backend's parsed option reaches that backend's three save
|
||||
calls, and only those: enabling it in one section leaves the other
|
||||
backend's saves False, and it is False when absent from both."""
|
||||
|
||||
def _run(self, es_extra: str = "", os_extra: str = "") -> dict[str, MagicMock]:
|
||||
names = (
|
||||
"elastic.save_aggregate_report_to_elasticsearch",
|
||||
"elastic.save_failure_report_to_elasticsearch",
|
||||
"elastic.save_smtp_tls_report_to_elasticsearch",
|
||||
"opensearch.save_aggregate_report_to_opensearch",
|
||||
"opensearch.save_failure_report_to_opensearch",
|
||||
"opensearch.save_smtp_tls_report_to_opensearch",
|
||||
)
|
||||
config = f"""[general]
|
||||
save_aggregate = true
|
||||
save_failure = true
|
||||
save_smtp_tls = true
|
||||
silent = true
|
||||
|
||||
[imap]
|
||||
host = imap.example.com
|
||||
user = test-user
|
||||
password = test-password
|
||||
|
||||
[elasticsearch]
|
||||
hosts = localhost
|
||||
{es_extra}
|
||||
|
||||
[opensearch]
|
||||
hosts = localhost
|
||||
{os_extra}
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w", suffix=".ini", delete=False
|
||||
) as config_file:
|
||||
config_file.write(config)
|
||||
config_path = config_file.name
|
||||
self.addCleanup(lambda: os.path.exists(config_path) and os.remove(config_path))
|
||||
|
||||
saves = {name: MagicMock() for name in names}
|
||||
patches = [patch(f"parsedmarc.cli.{name}", saves[name]) for name in names]
|
||||
patches += [
|
||||
patch("parsedmarc.cli.elastic.set_hosts"),
|
||||
patch("parsedmarc.cli.elastic.migrate_indexes"),
|
||||
patch("parsedmarc.cli.opensearch.set_hosts"),
|
||||
patch("parsedmarc.cli.opensearch.migrate_indexes"),
|
||||
patch("parsedmarc.cli.IMAPConnection", return_value=object()),
|
||||
patch(
|
||||
"parsedmarc.cli.get_dmarc_reports_from_mailbox",
|
||||
side_effect=_fetch_invoking_save_callback(
|
||||
{
|
||||
"aggregate_reports": [
|
||||
{"policy_published": {"domain": "example.com"}}
|
||||
],
|
||||
"failure_reports": [{"reported_domain": "example.com"}],
|
||||
"smtp_tls_reports": [
|
||||
{"policies": [{"policy_domain": "example.com"}]}
|
||||
],
|
||||
}
|
||||
),
|
||||
),
|
||||
patch.object(sys, "argv", ["parsedmarc", "-c", config_path]),
|
||||
]
|
||||
for p in patches:
|
||||
p.start()
|
||||
self.addCleanup(p.stop)
|
||||
parsedmarc.cli._main()
|
||||
return saves
|
||||
|
||||
def _assert_flags(
|
||||
self, saves: dict[str, MagicMock], elastic: bool, opensearch: bool
|
||||
) -> None:
|
||||
for name, save in saves.items():
|
||||
expected = elastic if name.startswith("elastic.") else opensearch
|
||||
with self.subTest(save=name):
|
||||
save.assert_called_once()
|
||||
self.assertIs(save.call_args.kwargs["use_data_streams"], expected)
|
||||
|
||||
def test_enabled_in_both_sections_reaches_every_save_call(self):
|
||||
enabled = "use_data_streams = true"
|
||||
saves = self._run(es_extra=enabled, os_extra=enabled)
|
||||
self._assert_flags(saves, elastic=True, opensearch=True)
|
||||
|
||||
def test_elasticsearch_section_only_reaches_elasticsearch_saves(self):
|
||||
saves = self._run(es_extra="use_data_streams = true")
|
||||
self._assert_flags(saves, elastic=True, opensearch=False)
|
||||
|
||||
def test_opensearch_section_only_reaches_opensearch_saves(self):
|
||||
saves = self._run(os_extra="use_data_streams = true")
|
||||
self._assert_flags(saves, elastic=False, opensearch=True)
|
||||
|
||||
def test_absent_defaults_to_false(self):
|
||||
saves = self._run()
|
||||
self._assert_flags(saves, elastic=False, opensearch=False)
|
||||
|
||||
|
||||
class TestParseConfigSplunkHec(unittest.TestCase):
|
||||
def test_splunk_hec_complete(self):
|
||||
from parsedmarc.cli import _parse_config
|
||||
@@ -9365,8 +10069,9 @@ class TestParseConfigLoki(unittest.TestCase):
|
||||
|
||||
class TestLokiCliWiring(unittest.TestCase):
|
||||
"""End-to-end: a [loki] config reaches LokiClient, every report type
|
||||
reaches its save method, and a failed push counts as an output error
|
||||
(so mailbox messages are held back instead of archived)."""
|
||||
reaches its save method, a failed push counts as an output error
|
||||
(so mailbox messages are held back instead of archived), and a config
|
||||
LokiClient rejects fails client initialization under the Loki name."""
|
||||
|
||||
def _run_main(self, reports, extra_general="", save_side_effect=None):
|
||||
config = f"""[general]
|
||||
@@ -9459,6 +10164,35 @@ tenant_id = fosdem
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, 1)
|
||||
|
||||
def test_rejected_client_config_names_the_loki_section(self):
|
||||
"""A [loki] config that parses but that LokiClient rejects is
|
||||
reported as a ``Loki:`` RuntimeError.
|
||||
|
||||
_parse_config accepts ``token`` alongside ``username``/``password``;
|
||||
only the real LokiClient constructor enforces that they are
|
||||
exclusive, so this is the path a misconfigured section takes at
|
||||
startup and on a SIGHUP reload.
|
||||
"""
|
||||
config = _config_with(
|
||||
"loki",
|
||||
{"url": "http://loki.example.com:3100", "token": "t", "username": "u"},
|
||||
)
|
||||
# The non-None Namespace defaults _main() gives the loki_* options.
|
||||
opts = _output_client_opts(
|
||||
loki_timestamp="ingest",
|
||||
loki_skip_certificate_verification=False,
|
||||
loki_timeout=60,
|
||||
loki_batch_size=1000,
|
||||
)
|
||||
parsedmarc.cli._parse_config(config, opts)
|
||||
with self.assertRaises(RuntimeError) as context:
|
||||
parsedmarc.cli._init_output_clients(opts)
|
||||
self.assertEqual(
|
||||
str(context.exception),
|
||||
"Loki: Loki token and username/password are exclusive",
|
||||
)
|
||||
self.assertIsInstance(context.exception.__cause__, ValueError)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
+423
-3
@@ -1,21 +1,29 @@
|
||||
"""Tests for parsedmarc.elastic
|
||||
|
||||
Mocks at the elasticsearch.dsl SDK boundary (connections.create_connection,
|
||||
Index, Search, Document.save) so the tests verify the parsedmarc-side
|
||||
transformation logic — document construction, index naming, deduplication
|
||||
queries, error wrapping — without needing a running Elasticsearch cluster.
|
||||
Index, Search, Document.save, or a mock client registered as the
|
||||
``default`` connection so the real Document.save runs) so the tests
|
||||
verify the parsedmarc-side transformation logic — document
|
||||
construction, index naming, deduplication queries, error wrapping —
|
||||
without needing a running Elasticsearch cluster.
|
||||
"""
|
||||
|
||||
import time
|
||||
import unittest
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from typing import cast
|
||||
from unittest.mock import MagicMock, call, patch
|
||||
|
||||
from elasticsearch import NotFoundError
|
||||
|
||||
import parsedmarc.elastic as elastic_module
|
||||
from parsedmarc import InvalidFailureReport, parse_report_file
|
||||
from parsedmarc.elastic import (
|
||||
AlreadySaved,
|
||||
ElasticsearchError,
|
||||
create_data_streams,
|
||||
create_indexes,
|
||||
migrate_indexes,
|
||||
save_aggregate_report_to_elasticsearch,
|
||||
@@ -1528,6 +1536,418 @@ class TestSaveSmtpTlsReport(unittest.TestCase):
|
||||
self.assertEqual(failure_detail.failure_reason_code, "no-policy-served")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data streams (issue #842)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _not_found() -> NotFoundError:
|
||||
"""The exception elasticsearch-py raises for an HTTP 404."""
|
||||
return NotFoundError(
|
||||
"index_not_found_exception", meta=MagicMock(status=404), body={}
|
||||
)
|
||||
|
||||
|
||||
def _stream_listing(*names: str) -> dict:
|
||||
"""A GET _data_stream/<name> response body listing *names*."""
|
||||
return {"data_streams": [{"name": name} for name in names]}
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _registered_client() -> Iterator[MagicMock]:
|
||||
"""Register a mock client as the ``default`` connection.
|
||||
|
||||
Both ``create_data_streams`` and ``Document.save`` resolve the client
|
||||
through the connection registry, so registering the mock there (rather
|
||||
than patching either function) lets the real save path run and the test
|
||||
read what reached the transport. The previous registration is put back
|
||||
afterwards.
|
||||
"""
|
||||
client = MagicMock()
|
||||
client.index.return_value = {"_index": "x", "_id": "1", "result": "created"}
|
||||
try:
|
||||
previous = elastic_module.connections.get_connection("default")
|
||||
except KeyError:
|
||||
previous = None
|
||||
elastic_module.connections.add_connection("default", client)
|
||||
try:
|
||||
yield client
|
||||
finally:
|
||||
if previous is None:
|
||||
elastic_module.connections.remove_connection("default")
|
||||
else:
|
||||
elastic_module.connections.add_connection("default", previous)
|
||||
|
||||
|
||||
class _DataStreamTestCase(unittest.TestCase):
|
||||
"""Isolates each test from the module's data stream memo and the
|
||||
Serverless flag."""
|
||||
|
||||
def setUp(self):
|
||||
self._serverless = elastic_module._SERVERLESS
|
||||
elastic_module._SERVERLESS = False
|
||||
elastic_module._READY_DATA_STREAMS.clear()
|
||||
|
||||
def tearDown(self):
|
||||
elastic_module._SERVERLESS = self._serverless
|
||||
elastic_module._READY_DATA_STREAMS.clear()
|
||||
|
||||
|
||||
class TestCreateDataStreams(_DataStreamTestCase):
|
||||
def test_creates_template_and_stream_when_missing(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(
|
||||
["dmarc_aggregate"],
|
||||
{"number_of_shards": 2, "number_of_replicas": 1},
|
||||
)
|
||||
template = client.indices.put_index_template.call_args.kwargs
|
||||
self.assertEqual(template["name"], "dmarc_aggregate")
|
||||
# Exact name, no wildcard: the template must never claim the
|
||||
# date-suffixed indexes written without data streams.
|
||||
self.assertEqual(template["index_patterns"], ["dmarc_aggregate"])
|
||||
self.assertEqual(template["data_stream"], {})
|
||||
self.assertEqual(template["priority"], 501)
|
||||
self.assertEqual(template["composed_of"], ["dmarc_aggregate@custom"])
|
||||
self.assertEqual(
|
||||
template["ignore_missing_component_templates"],
|
||||
["dmarc_aggregate@custom"],
|
||||
)
|
||||
self.assertEqual(
|
||||
template["template"],
|
||||
{
|
||||
"settings": {"number_of_shards": 2, "number_of_replicas": 1},
|
||||
"mappings": {"properties": {"@timestamp": {"type": "date"}}},
|
||||
},
|
||||
)
|
||||
self.assertIn("dmarc_aggregate@custom", template["meta"]["description"])
|
||||
client.indices.create_data_stream.assert_called_once_with(
|
||||
name="dmarc_aggregate"
|
||||
)
|
||||
# No concrete index is created for a stream.
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
self.assertIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_default_settings_are_one_shard_no_replicas(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(["smtp_tls"])
|
||||
settings = client.indices.put_index_template.call_args.kwargs["template"][
|
||||
"settings"
|
||||
]
|
||||
self.assertEqual(settings, {"number_of_shards": 1, "number_of_replicas": 0})
|
||||
|
||||
def test_existing_stream_gets_template_but_is_not_recreated(self):
|
||||
"""The template is put again so that settings changes reach the
|
||||
next backing index; the stream itself already exists."""
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.return_value = _stream_listing(
|
||||
"dmarc_failure"
|
||||
)
|
||||
create_data_streams(["dmarc_failure"])
|
||||
client.indices.put_index_template.assert_called_once()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
mock_index_cls.return_value.exists.assert_not_called()
|
||||
self.assertIn("dmarc_failure", elastic_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_concrete_index_holding_the_name_raises_without_writing(self):
|
||||
"""Elasticsearch answers GET _data_stream/<name> for a name held by
|
||||
a concrete index with 200 and an empty list, not a 404 (verified
|
||||
against 8.19). Treating that 200 as "stream exists" would send
|
||||
every document into the old undated index, such as one left by a
|
||||
pre-5.0.0 release."""
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.return_value = _stream_listing()
|
||||
mock_index_cls.return_value.exists.return_value = True
|
||||
with self.assertRaises(ElasticsearchError) as ctx:
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
mock_index_cls.assert_called_with("dmarc_aggregate")
|
||||
self.assertIn("already exists", str(ctx.exception))
|
||||
self.assertIn("dmarc_aggregate", str(ctx.exception))
|
||||
client.indices.put_index_template.assert_not_called()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
self.assertNotIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_alias_holding_the_name_raises(self):
|
||||
"""A missing stream is a 404; an alias with the name still exists
|
||||
as far as Index.exists() is concerned."""
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = True
|
||||
with self.assertRaises(ElasticsearchError):
|
||||
create_data_streams(["smtp_tls"])
|
||||
client.indices.put_index_template.assert_not_called()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_ready_stream_is_not_set_up_again(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
self.assertEqual(client.indices.get_data_stream.call_count, 1)
|
||||
self.assertEqual(client.indices.put_index_template.call_count, 1)
|
||||
self.assertEqual(client.indices.create_data_stream.call_count, 1)
|
||||
|
||||
def test_set_hosts_forgets_ready_streams(self):
|
||||
"""A new connection may be a different cluster or carry new index
|
||||
settings, so the template has to be put again."""
|
||||
elastic_module._READY_DATA_STREAMS.add("dmarc_aggregate")
|
||||
with patch("parsedmarc.elastic.connections.create_connection"):
|
||||
set_hosts("es:9200")
|
||||
self.assertEqual(elastic_module._READY_DATA_STREAMS, set())
|
||||
|
||||
def test_sdk_error_is_wrapped_and_retried_next_time(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
client.indices.put_index_template.side_effect = RuntimeError("cluster down")
|
||||
with self.assertRaises(ElasticsearchError) as ctx:
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
self.assertIn("cluster down", str(ctx.exception))
|
||||
self.assertNotIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
client.indices.put_index_template.side_effect = None
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
client.indices.create_data_stream.assert_called_once_with(
|
||||
name="dmarc_aggregate"
|
||||
)
|
||||
self.assertIn("dmarc_aggregate", elastic_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_serverless_strips_rejected_settings_from_template(self):
|
||||
elastic_module._SERVERLESS = True
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(
|
||||
["dmarc_aggregate"],
|
||||
{
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
"refresh_interval": "5s",
|
||||
},
|
||||
)
|
||||
settings = client.indices.put_index_template.call_args.kwargs["template"][
|
||||
"settings"
|
||||
]
|
||||
self.assertEqual(settings, {"refresh_interval": "5s"})
|
||||
|
||||
|
||||
class TestSaveToDataStreams(_DataStreamTestCase):
|
||||
"""Each save function, run through the real Document.save() into a
|
||||
mock client: in stream mode the document goes to the undated name
|
||||
with op_type=create and an @timestamp; in the default mode it goes to
|
||||
the dated index as before, with the same op_type and @timestamp."""
|
||||
|
||||
def _save(self, save_function, report, **kwargs):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Search") as mock_search_cls,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
mock_search_cls.return_value.execute.return_value = []
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
save_function(report, **kwargs)
|
||||
return client, mock_search_cls, mock_index_cls
|
||||
|
||||
def test_aggregate_report_goes_to_the_stream(self):
|
||||
client, mock_search_cls, mock_index_cls = self._save(
|
||||
save_aggregate_report_to_elasticsearch,
|
||||
_aggregate_report(),
|
||||
index_prefix="cust_",
|
||||
index_suffix="tenant_a",
|
||||
# Ignored in stream mode.
|
||||
monthly_indexes=True,
|
||||
use_data_streams=True,
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "cust_dmarc_aggregate_tenant_a")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
self.assertEqual(index_call["body"]["report_id"], "agg-1")
|
||||
self.assertEqual(
|
||||
client.indices.put_index_template.call_args.kwargs["index_patterns"],
|
||||
["cust_dmarc_aggregate_tenant_a"],
|
||||
)
|
||||
client.indices.create_data_stream.assert_called_once_with(
|
||||
name="cust_dmarc_aggregate_tenant_a"
|
||||
)
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
# The dedup search still uses the wildcard, which matches the
|
||||
# stream as well as dated indexes written before the switch.
|
||||
self.assertEqual(
|
||||
mock_search_cls.call_args.kwargs["index"],
|
||||
"cust_dmarc_aggregate_tenant_a*",
|
||||
)
|
||||
|
||||
def test_aggregate_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
|
||||
self,
|
||||
):
|
||||
client, _, mock_index_cls = self._save(
|
||||
save_aggregate_report_to_elasticsearch, _aggregate_report()
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "dmarc_aggregate-2024-01-15")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
mock_index_cls.return_value.create.assert_called_once()
|
||||
client.indices.put_index_template.assert_not_called()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_failure_report_goes_to_the_stream(self):
|
||||
client, mock_search_cls, mock_index_cls = self._save(
|
||||
save_failure_report_to_elasticsearch,
|
||||
_failure_report(),
|
||||
monthly_indexes=True,
|
||||
use_data_streams=True,
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "dmarc_failure")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_called_once_with(name="dmarc_failure")
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
self.assertEqual(
|
||||
mock_search_cls.call_args.kwargs["index"],
|
||||
"dmarc_failure*,dmarc_forensic*",
|
||||
)
|
||||
|
||||
def test_failure_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(self):
|
||||
client, _, _ = self._save(
|
||||
save_failure_report_to_elasticsearch, _failure_report()
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "dmarc_failure-2024-01-01")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_smtp_tls_report_goes_to_the_stream(self):
|
||||
client, mock_search_cls, mock_index_cls = self._save(
|
||||
save_smtp_tls_report_to_elasticsearch,
|
||||
_smtp_tls_report(),
|
||||
index_prefix="cust_",
|
||||
use_data_streams=True,
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "cust_smtp_tls")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 2, 3, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_called_once_with(name="cust_smtp_tls")
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
self.assertEqual(mock_search_cls.call_args.kwargs["index"], "cust_smtp_tls*")
|
||||
|
||||
def test_smtp_tls_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
|
||||
self,
|
||||
):
|
||||
client, _, _ = self._save(
|
||||
save_smtp_tls_report_to_elasticsearch, _smtp_tls_report()
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "smtp_tls-2024-02-03")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 2, 3, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_stream_set_up_once_for_a_multi_record_report(self):
|
||||
report = _aggregate_report()
|
||||
report["records"].append(report["records"][0].copy())
|
||||
client, _, _ = self._save(
|
||||
save_aggregate_report_to_elasticsearch, report, use_data_streams=True
|
||||
)
|
||||
self.assertEqual(client.index.call_count, 2)
|
||||
self.assertEqual(client.indices.put_index_template.call_count, 1)
|
||||
|
||||
def test_name_collision_saves_nothing(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.elastic.Search") as mock_search_cls,
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
):
|
||||
mock_search_cls.return_value.execute.return_value = []
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = True
|
||||
with self.assertRaises(ElasticsearchError):
|
||||
save_smtp_tls_report_to_elasticsearch(
|
||||
_smtp_tls_report(), use_data_streams=True
|
||||
)
|
||||
client.index.assert_not_called()
|
||||
|
||||
|
||||
class TestMigrateIndexesWithDataStream(unittest.TestCase):
|
||||
def test_legacy_fo_migration_skips_a_stream_with_the_name(self):
|
||||
"""With use_data_streams the undated legacy name is the stream
|
||||
itself. Index.exists() is true for it, and the field mapping comes
|
||||
back keyed by a backing index; its fo is text, so nothing is
|
||||
migrated or deleted."""
|
||||
with (
|
||||
patch("parsedmarc.elastic.Index") as mock_index_cls,
|
||||
patch("parsedmarc.elastic.reindex") as mock_reindex,
|
||||
):
|
||||
index = mock_index_cls.return_value
|
||||
index.exists.return_value = True
|
||||
index.get_field_mapping.return_value = {
|
||||
".ds-dmarc_aggregate-2026.10.31-000001": {
|
||||
"mappings": {
|
||||
"published_policy.fo": {
|
||||
"full_name": "published_policy.fo",
|
||||
"mapping": {"fo": {"type": "text"}},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
migrate_indexes(legacy_fo_indexes=["dmarc_aggregate"])
|
||||
mock_reindex.assert_not_called()
|
||||
index.create.assert_not_called()
|
||||
index.delete.assert_not_called()
|
||||
|
||||
|
||||
class TestBackwardCompatAlias(unittest.TestCase):
|
||||
def test_save_forensic_alias_points_to_save_failure(self):
|
||||
self.assertIs(
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import traceback
|
||||
import unittest
|
||||
from typing import cast
|
||||
from unittest.mock import MagicMock, patch
|
||||
@@ -164,6 +165,44 @@ class TestLokiClientInit(unittest.TestCase):
|
||||
self.assertIsInstance(client.session.auth, httpx.BasicAuth)
|
||||
|
||||
|
||||
class TestLokiInvalidUrlDoesNotLeakPassword(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
|
||||
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
|
||||
text after the colon as the port and raises
|
||||
``InvalidURL("Invalid port: 'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py:
|
||||
``raise InvalidURL(f"Invalid port: {port!r}")``). Its implicit
|
||||
``__context__`` is a ValueError that repeats that part of the password
|
||||
too."""
|
||||
|
||||
MALFORMED = (
|
||||
("https://admin:p4ssw0rd/extra@loki.example.org:3100", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@loki.example.org:3100", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@loki.example.org:3100", "p4ss"),
|
||||
)
|
||||
|
||||
def test_invalid_url_raises_fixed_message_without_password(self):
|
||||
for url, secret in self.MALFORMED:
|
||||
with self.subTest(url=url):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
LokiClient(url)
|
||||
self.assertEqual(
|
||||
str(ctx.exception),
|
||||
"Invalid Loki URL (details withheld because they may "
|
||||
"include credentials)",
|
||||
)
|
||||
self.assertIsNone(ctx.exception.__cause__)
|
||||
self.assertTrue(ctx.exception.__suppress_context__)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn(secret, trace)
|
||||
|
||||
def test_valid_url_keeps_userinfo_for_requests(self):
|
||||
client = LokiClient("https://admin:p4ssw0rd@loki.example.org:3100")
|
||||
self.assertEqual(
|
||||
client.url,
|
||||
"https://admin:p4ssw0rd@loki.example.org:3100/loki/api/v1/push",
|
||||
)
|
||||
|
||||
|
||||
@patch(
|
||||
"parsedmarc.loki.parsed_aggregate_reports_to_csv_rows", return_value=AGGREGATE_ROWS
|
||||
)
|
||||
@@ -370,6 +409,28 @@ class TestLokiErrors(unittest.TestCase):
|
||||
client.save_aggregate_report_to_loki(AGGREGATE)
|
||||
self.assertIn("refused", str(ctx.exception))
|
||||
|
||||
def test_errors_mask_url_password(self, _rows):
|
||||
"""GHSA-hjr3-rx4c-24cv: the URL keeps its userinfo, and the CLI
|
||||
logs LokiError text, so the password must be masked in it."""
|
||||
secret = "s3cretPW"
|
||||
client = LokiClient(f"http://u:{secret}@loki.example.org:3100")
|
||||
client.session = MagicMock()
|
||||
_session(client).post.return_value = _ok_response(500)
|
||||
with self.assertRaises(LokiError) as status_ctx:
|
||||
client.save_aggregate_report_to_loki(AGGREGATE)
|
||||
_session(client).post.side_effect = httpx.ConnectError("refused")
|
||||
with self.assertRaises(LokiError) as network_ctx:
|
||||
client.save_aggregate_report_to_loki(AGGREGATE)
|
||||
for ctx in (status_ctx, network_ctx):
|
||||
with self.subTest(error=str(ctx.exception)):
|
||||
self.assertNotIn(secret, str(ctx.exception))
|
||||
self.assertIn("http://u:***@loki.example.org:3100", str(ctx.exception))
|
||||
# Redaction is message-only: the request still goes to the real URL
|
||||
self.assertEqual(
|
||||
_session(client).post.call_args.args[0],
|
||||
f"http://u:{secret}@loki.example.org:3100/loki/api/v1/push",
|
||||
)
|
||||
|
||||
def test_blocked_ingestion_status_raises(self, _rows):
|
||||
"""Loki returns 260 (block_ingestion_status_code default) and drops
|
||||
the logs when ingestion is blocked; that must not read as success."""
|
||||
|
||||
+418
-3
@@ -1,21 +1,29 @@
|
||||
"""Tests for parsedmarc.opensearch
|
||||
|
||||
Mocks at the opensearch-dsl SDK boundary (connections.create_connection,
|
||||
Index, Search, Document.save) so the tests verify the parsedmarc-side
|
||||
transformation logic — document construction, index naming, deduplication
|
||||
queries, error wrapping — without needing a running OpenSearch cluster.
|
||||
Index, Search, Document.save, or a mock client registered as the
|
||||
``default`` connection so the real Document.save runs) so the tests
|
||||
verify the parsedmarc-side transformation logic — document
|
||||
construction, index naming, deduplication queries, error wrapping —
|
||||
without needing a running OpenSearch cluster.
|
||||
"""
|
||||
|
||||
import time
|
||||
import unittest
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from typing import cast
|
||||
from unittest.mock import MagicMock, call, patch
|
||||
|
||||
from opensearchpy import NotFoundError
|
||||
|
||||
import parsedmarc.opensearch as opensearch_module
|
||||
from parsedmarc import InvalidFailureReport, parse_report_file
|
||||
from parsedmarc.opensearch import (
|
||||
AlreadySaved,
|
||||
OpenSearchError,
|
||||
create_data_streams,
|
||||
create_indexes,
|
||||
migrate_indexes,
|
||||
save_aggregate_report_to_opensearch,
|
||||
@@ -233,6 +241,30 @@ class TestSetHosts(unittest.TestCase):
|
||||
kwargs = mock_conn.call_args.kwargs
|
||||
self.assertEqual(kwargs["hosts"], ["es1:9200", "es2:9200"])
|
||||
|
||||
def test_debug_log_masks_credentials_but_connects_with_them(self):
|
||||
"""GHSA-hjr3-rx4c-24cv: the "Connecting to OpenSearch" debug line
|
||||
must not carry the URL password, while the client still gets the
|
||||
real URL."""
|
||||
url = "https://admin:s3cretPW@es.example:9200"
|
||||
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
set_hosts(url)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("s3cretPW", output)
|
||||
self.assertIn("https://admin:***@es.example:9200", output)
|
||||
self.assertEqual(mock_conn.call_args.kwargs["hosts"], [url])
|
||||
|
||||
def test_debug_log_masks_schemeless_host_list(self):
|
||||
hosts = ["admin:s3cretPW@es1:9200", "es2:9200"]
|
||||
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
set_hosts(hosts)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("s3cretPW", output)
|
||||
self.assertIn("admin:***@es1:9200", output)
|
||||
self.assertIn("es2:9200", output)
|
||||
self.assertEqual(mock_conn.call_args.kwargs["hosts"], hosts)
|
||||
|
||||
def test_timeout_default_60s(self):
|
||||
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
|
||||
set_hosts("es:9200")
|
||||
@@ -1519,6 +1551,389 @@ class TestSaveSmtpTlsReport(unittest.TestCase):
|
||||
self.assertEqual(failure_detail.failure_reason_code, "no-policy-served")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data streams (issue #842)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _not_found() -> NotFoundError:
|
||||
"""The exception opensearch-py raises for an HTTP 404."""
|
||||
return NotFoundError(404, "index_not_found_exception", {})
|
||||
|
||||
|
||||
def _stream_listing(*names: str) -> dict:
|
||||
"""A GET _data_stream/<name> response body listing *names*."""
|
||||
return {"data_streams": [{"name": name} for name in names]}
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _registered_client() -> Iterator[MagicMock]:
|
||||
"""Register a mock client as the ``default`` connection.
|
||||
|
||||
Both ``create_data_streams`` and ``Document.save`` resolve the client
|
||||
through the connection registry, so registering the mock there (rather
|
||||
than patching either function) lets the real save path run and the test
|
||||
read what reached the transport. The previous registration is put back
|
||||
afterwards.
|
||||
"""
|
||||
client = MagicMock()
|
||||
client.index.return_value = {"_index": "x", "_id": "1", "result": "created"}
|
||||
try:
|
||||
previous = opensearch_module.connections.get_connection("default")
|
||||
except KeyError:
|
||||
previous = None
|
||||
opensearch_module.connections.add_connection("default", client)
|
||||
try:
|
||||
yield client
|
||||
finally:
|
||||
if previous is None:
|
||||
opensearch_module.connections.remove_connection("default")
|
||||
else:
|
||||
opensearch_module.connections.add_connection("default", previous)
|
||||
|
||||
|
||||
class _DataStreamTestCase(unittest.TestCase):
|
||||
"""Isolates each test from the module's data stream memo."""
|
||||
|
||||
def setUp(self):
|
||||
opensearch_module._READY_DATA_STREAMS.clear()
|
||||
|
||||
def tearDown(self):
|
||||
opensearch_module._READY_DATA_STREAMS.clear()
|
||||
|
||||
|
||||
class TestCreateDataStreams(_DataStreamTestCase):
|
||||
def test_creates_template_and_stream_when_missing(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(
|
||||
["dmarc_aggregate"],
|
||||
{"number_of_shards": 2, "number_of_replicas": 1},
|
||||
)
|
||||
template_call = client.indices.put_index_template.call_args.kwargs
|
||||
self.assertEqual(template_call["name"], "dmarc_aggregate")
|
||||
body = template_call["body"]
|
||||
# Exact name, no wildcard: the template must never claim the
|
||||
# date-suffixed indexes written without data streams.
|
||||
self.assertEqual(body["index_patterns"], ["dmarc_aggregate"])
|
||||
self.assertEqual(body["data_stream"], {})
|
||||
self.assertEqual(body["priority"], 501)
|
||||
self.assertNotIn("composed_of", body)
|
||||
self.assertEqual(
|
||||
body["template"],
|
||||
{
|
||||
"settings": {"number_of_shards": 2, "number_of_replicas": 1},
|
||||
"mappings": {"properties": {"@timestamp": {"type": "date"}}},
|
||||
},
|
||||
)
|
||||
self.assertIn("ism_template", body["_meta"]["description"])
|
||||
client.indices.create_data_stream.assert_called_once_with(
|
||||
name="dmarc_aggregate"
|
||||
)
|
||||
# No concrete index is created for a stream.
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
self.assertIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_default_settings_are_one_shard_no_replicas(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(["smtp_tls"])
|
||||
settings = client.indices.put_index_template.call_args.kwargs["body"][
|
||||
"template"
|
||||
]["settings"]
|
||||
self.assertEqual(settings, {"number_of_shards": 1, "number_of_replicas": 0})
|
||||
|
||||
def test_existing_stream_gets_template_but_is_not_recreated(self):
|
||||
"""The template is put again so that settings changes reach the
|
||||
next backing index; the stream itself already exists."""
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.return_value = _stream_listing(
|
||||
"dmarc_failure"
|
||||
)
|
||||
create_data_streams(["dmarc_failure"])
|
||||
client.indices.put_index_template.assert_called_once()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
mock_index_cls.return_value.exists.assert_not_called()
|
||||
self.assertIn("dmarc_failure", opensearch_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_concrete_index_holding_the_name_raises_without_writing(self):
|
||||
"""OpenSearch 3 answers GET _data_stream/<name> for a name held by
|
||||
a concrete index with a 404, so the Index.exists() check is what
|
||||
catches it, before anything is written. Writing to that name would
|
||||
otherwise go into the old undated index, such as one left by a
|
||||
pre-5.0.0 release."""
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = True
|
||||
with self.assertRaises(OpenSearchError) as ctx:
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
mock_index_cls.assert_called_with("dmarc_aggregate")
|
||||
self.assertIn("already exists", str(ctx.exception))
|
||||
self.assertIn("dmarc_aggregate", str(ctx.exception))
|
||||
client.indices.put_index_template.assert_not_called()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
self.assertNotIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
|
||||
|
||||
def test_empty_listing_is_not_taken_as_an_existing_stream(self):
|
||||
"""Elasticsearch answers a name held by a concrete index with 200
|
||||
and an empty list; the shared logic must not read that as "the
|
||||
stream exists" if an OpenSearch release ever does the same."""
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.return_value = _stream_listing()
|
||||
mock_index_cls.return_value.exists.return_value = True
|
||||
with self.assertRaises(OpenSearchError):
|
||||
create_data_streams(["smtp_tls"])
|
||||
client.indices.put_index_template.assert_not_called()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_ready_stream_is_not_set_up_again(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
self.assertEqual(client.indices.get_data_stream.call_count, 1)
|
||||
self.assertEqual(client.indices.put_index_template.call_count, 1)
|
||||
self.assertEqual(client.indices.create_data_stream.call_count, 1)
|
||||
|
||||
def test_set_hosts_forgets_ready_streams(self):
|
||||
"""A new connection may be a different cluster or carry new index
|
||||
settings, so the template has to be put again."""
|
||||
opensearch_module._READY_DATA_STREAMS.add("dmarc_aggregate")
|
||||
with patch("parsedmarc.opensearch.connections.create_connection"):
|
||||
set_hosts("os:9200")
|
||||
self.assertEqual(opensearch_module._READY_DATA_STREAMS, set())
|
||||
|
||||
def test_sdk_error_is_wrapped_and_retried_next_time(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
client.indices.put_index_template.side_effect = RuntimeError("cluster down")
|
||||
with self.assertRaises(OpenSearchError) as ctx:
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
self.assertIn("cluster down", str(ctx.exception))
|
||||
self.assertNotIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
client.indices.put_index_template.side_effect = None
|
||||
create_data_streams(["dmarc_aggregate"])
|
||||
client.indices.create_data_stream.assert_called_once_with(
|
||||
name="dmarc_aggregate"
|
||||
)
|
||||
self.assertIn("dmarc_aggregate", opensearch_module._READY_DATA_STREAMS)
|
||||
|
||||
|
||||
class TestSaveToDataStreams(_DataStreamTestCase):
|
||||
"""Each save function, run through the real Document.save() into a
|
||||
mock client: in stream mode the document goes to the undated name
|
||||
with op_type=create and an @timestamp; in the default mode it goes to
|
||||
the dated index as before, with the same op_type and @timestamp."""
|
||||
|
||||
def _save(self, save_function, report, **kwargs):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Search") as mock_search_cls,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
mock_search_cls.return_value.execute.return_value = []
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = False
|
||||
save_function(report, **kwargs)
|
||||
return client, mock_search_cls, mock_index_cls
|
||||
|
||||
def test_aggregate_report_goes_to_the_stream(self):
|
||||
client, mock_search_cls, mock_index_cls = self._save(
|
||||
save_aggregate_report_to_opensearch,
|
||||
_aggregate_report(),
|
||||
index_prefix="cust_",
|
||||
index_suffix="tenant_a",
|
||||
# Ignored in stream mode.
|
||||
monthly_indexes=True,
|
||||
use_data_streams=True,
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "cust_dmarc_aggregate_tenant_a")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
self.assertEqual(index_call["body"]["report_id"], "agg-1")
|
||||
self.assertEqual(
|
||||
client.indices.put_index_template.call_args.kwargs["body"][
|
||||
"index_patterns"
|
||||
],
|
||||
["cust_dmarc_aggregate_tenant_a"],
|
||||
)
|
||||
client.indices.create_data_stream.assert_called_once_with(
|
||||
name="cust_dmarc_aggregate_tenant_a"
|
||||
)
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
# The dedup search still uses the wildcard, which matches the
|
||||
# stream as well as dated indexes written before the switch.
|
||||
self.assertEqual(
|
||||
mock_search_cls.call_args.kwargs["index"],
|
||||
"cust_dmarc_aggregate_tenant_a*",
|
||||
)
|
||||
|
||||
def test_aggregate_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
|
||||
self,
|
||||
):
|
||||
client, _, mock_index_cls = self._save(
|
||||
save_aggregate_report_to_opensearch, _aggregate_report()
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "dmarc_aggregate-2024-01-15")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
mock_index_cls.return_value.create.assert_called_once()
|
||||
client.indices.put_index_template.assert_not_called()
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_failure_report_goes_to_the_stream(self):
|
||||
client, mock_search_cls, mock_index_cls = self._save(
|
||||
save_failure_report_to_opensearch,
|
||||
_failure_report(),
|
||||
monthly_indexes=True,
|
||||
use_data_streams=True,
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "dmarc_failure")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_called_once_with(name="dmarc_failure")
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
self.assertEqual(
|
||||
mock_search_cls.call_args.kwargs["index"],
|
||||
"dmarc_failure*,dmarc_forensic*",
|
||||
)
|
||||
|
||||
def test_failure_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(self):
|
||||
client, _, _ = self._save(save_failure_report_to_opensearch, _failure_report())
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "dmarc_failure-2024-01-01")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_smtp_tls_report_goes_to_the_stream(self):
|
||||
client, mock_search_cls, mock_index_cls = self._save(
|
||||
save_smtp_tls_report_to_opensearch,
|
||||
_smtp_tls_report(),
|
||||
index_prefix="cust_",
|
||||
use_data_streams=True,
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "cust_smtp_tls")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 2, 3, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_called_once_with(name="cust_smtp_tls")
|
||||
mock_index_cls.return_value.create.assert_not_called()
|
||||
self.assertEqual(mock_search_cls.call_args.kwargs["index"], "cust_smtp_tls*")
|
||||
|
||||
def test_smtp_tls_report_dated_mode_unchanged_apart_from_timestamp_and_op_type(
|
||||
self,
|
||||
):
|
||||
client, _, _ = self._save(
|
||||
save_smtp_tls_report_to_opensearch, _smtp_tls_report()
|
||||
)
|
||||
index_call = client.index.call_args.kwargs
|
||||
self.assertEqual(index_call["index"], "smtp_tls-2024-02-03")
|
||||
self.assertEqual(index_call["op_type"], "create")
|
||||
self.assertEqual(
|
||||
index_call["body"]["@timestamp"],
|
||||
datetime(2024, 2, 3, tzinfo=timezone.utc),
|
||||
)
|
||||
client.indices.create_data_stream.assert_not_called()
|
||||
|
||||
def test_stream_set_up_once_for_a_multi_record_report(self):
|
||||
report = _aggregate_report()
|
||||
report["records"].append(report["records"][0].copy())
|
||||
client, _, _ = self._save(
|
||||
save_aggregate_report_to_opensearch, report, use_data_streams=True
|
||||
)
|
||||
self.assertEqual(client.index.call_count, 2)
|
||||
self.assertEqual(client.indices.put_index_template.call_count, 1)
|
||||
|
||||
def test_name_collision_saves_nothing(self):
|
||||
with (
|
||||
_registered_client() as client,
|
||||
patch("parsedmarc.opensearch.Search") as mock_search_cls,
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
):
|
||||
mock_search_cls.return_value.execute.return_value = []
|
||||
client.indices.get_data_stream.side_effect = _not_found()
|
||||
mock_index_cls.return_value.exists.return_value = True
|
||||
with self.assertRaises(OpenSearchError):
|
||||
save_smtp_tls_report_to_opensearch(
|
||||
_smtp_tls_report(), use_data_streams=True
|
||||
)
|
||||
client.index.assert_not_called()
|
||||
|
||||
|
||||
class TestMigrateIndexesWithDataStream(unittest.TestCase):
|
||||
def test_legacy_fo_migration_skips_a_stream_with_the_name(self):
|
||||
"""With use_data_streams the undated legacy name is the stream
|
||||
itself. Index.exists() is true for it, and the field mapping comes
|
||||
back keyed by a backing index; its fo is text, so nothing is
|
||||
migrated or deleted."""
|
||||
with (
|
||||
patch("parsedmarc.opensearch.Index") as mock_index_cls,
|
||||
patch("parsedmarc.opensearch.reindex") as mock_reindex,
|
||||
):
|
||||
index = mock_index_cls.return_value
|
||||
index.exists.return_value = True
|
||||
index.get_field_mapping.return_value = {
|
||||
".ds-dmarc_aggregate-000001": {
|
||||
"mappings": {
|
||||
"published_policy.fo": {
|
||||
"full_name": "published_policy.fo",
|
||||
"mapping": {"fo": {"type": "text"}},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
migrate_indexes(legacy_fo_indexes=["dmarc_aggregate"])
|
||||
mock_reindex.assert_not_called()
|
||||
index.create.assert_not_called()
|
||||
index.delete.assert_not_called()
|
||||
|
||||
|
||||
class TestBackwardCompatAlias(unittest.TestCase):
|
||||
def test_save_forensic_alias_points_to_save_failure(self):
|
||||
self.assertIs(
|
||||
|
||||
@@ -311,11 +311,74 @@ class TestPostgreSQLConstruction(unittest.TestCase):
|
||||
"""A driver-level connection error is wrapped in PostgreSQLError."""
|
||||
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
|
||||
mock_psycopg.Error = Exception
|
||||
mock_psycopg.ProgrammingError = type("ProgrammingError", (Exception,), {})
|
||||
mock_psycopg.connect.side_effect = mock_psycopg.Error("refused")
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(host="localhost")
|
||||
self.assertIn("refused", str(ctx.exception))
|
||||
|
||||
def test_connect_failure_detail_is_url_redacted(self):
|
||||
"""A non-parse driver error keeps its detail, with URL credentials
|
||||
masked."""
|
||||
programming_error = type("ProgrammingError", (Exception,), {})
|
||||
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
|
||||
mock_psycopg.Error = Exception
|
||||
mock_psycopg.ProgrammingError = programming_error
|
||||
mock_psycopg.connect.side_effect = mock_psycopg.Error(
|
||||
'connection to "postgresql://u:PGSECRET@db.example/x" failed'
|
||||
)
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(connection_string="postgresql://u:PGSECRET@h/db")
|
||||
message = str(ctx.exception)
|
||||
self.assertNotIn("PGSECRET", message)
|
||||
self.assertIn("u:***@db.example/x", message)
|
||||
|
||||
def test_malformed_connection_string_detail_withheld(self):
|
||||
"""A ProgrammingError from parsing the connection string (which libpq
|
||||
words with the whole string or a password fragment) is replaced with
|
||||
a fixed message that still identifies the problem."""
|
||||
programming_error = type("ProgrammingError", (Exception,), {})
|
||||
for dsn, driver_message in (
|
||||
(
|
||||
"postgresql://u:PGSECRET@[::1/db",
|
||||
'end of string reached when looking for matching "]" in IPv6 '
|
||||
'host address in URI: "postgresql://u:PGSECRET@[::1/db"',
|
||||
),
|
||||
(
|
||||
"postgresql://u:PG%ZZSECRET@h/db",
|
||||
'invalid percent-encoded token: "PG%ZZSECRET"',
|
||||
),
|
||||
):
|
||||
with self.subTest(dsn=dsn):
|
||||
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
|
||||
mock_psycopg.Error = Exception
|
||||
mock_psycopg.ProgrammingError = programming_error
|
||||
mock_psycopg.connect.side_effect = programming_error(driver_message)
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(connection_string=dsn)
|
||||
message = str(ctx.exception)
|
||||
self.assertNotIn("PGSECRET", message)
|
||||
self.assertNotIn("SECRET", message)
|
||||
self.assertIn("Invalid PostgreSQL connection_string", message)
|
||||
|
||||
@unittest.skipIf(
|
||||
importlib.util.find_spec("psycopg") is None, "psycopg is not installed"
|
||||
)
|
||||
def test_real_psycopg_malformed_connection_string_withheld(self):
|
||||
"""With the real driver, both malformed strings fail in parsing (no
|
||||
network I/O) and the raised message carries no part of the password."""
|
||||
for dsn in (
|
||||
"postgresql://u:PGSECRET@[::1/db",
|
||||
"postgresql://u:PG%ZZSECRET@h/db",
|
||||
):
|
||||
with self.subTest(dsn=dsn):
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(connection_string=dsn)
|
||||
self.assertNotIn("SECRET", str(ctx.exception))
|
||||
self.assertIn(
|
||||
"Invalid PostgreSQL connection_string", str(ctx.exception)
|
||||
)
|
||||
|
||||
def test_create_tables_executes_all_ddl(self):
|
||||
"""create_tables issues CREATE TABLE for every table and the indexes."""
|
||||
client, mock_conn = _make_client()
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import json
|
||||
import time
|
||||
import traceback
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@@ -469,6 +470,58 @@ class TestSaveSmtpTlsReportsToSplunk(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestInvalidUrlDoesNotLeakPassword(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
|
||||
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
|
||||
text after the colon as the port and raises
|
||||
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
|
||||
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
|
||||
any network I/O. The error reaches the log on every batch."""
|
||||
|
||||
MALFORMED = (
|
||||
("https://admin:p4ssw0rd/extra@splunk.example.com:8088", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@splunk.example.com:8088", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@splunk.example.com:8088", "p4ss"),
|
||||
)
|
||||
|
||||
def test_invalid_url_raises_fixed_message_without_password(self):
|
||||
saves = (
|
||||
("aggregate", "save_aggregate_reports_to_splunk", _aggregate_report),
|
||||
("failure", "save_failure_reports_to_splunk", _failure_report),
|
||||
("smtp_tls", "save_smtp_tls_reports_to_splunk", _smtp_tls_report),
|
||||
)
|
||||
for url, secret in self.MALFORMED:
|
||||
client = HECClient(url=url, access_token="tok", index="dmarc")
|
||||
for name, method, make_report in saves:
|
||||
with self.subTest(url=url, report=name):
|
||||
with self.assertRaises(SplunkError) as ctx:
|
||||
getattr(client, method)(make_report())
|
||||
self.assertEqual(
|
||||
str(ctx.exception),
|
||||
"Invalid Splunk HEC URL (details withheld because they "
|
||||
"may include credentials)",
|
||||
)
|
||||
self.assertIsNone(ctx.exception.__cause__)
|
||||
self.assertTrue(ctx.exception.__suppress_context__)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn(secret, trace)
|
||||
|
||||
def test_valid_url_is_the_one_posted(self):
|
||||
client = HECClient(
|
||||
url="https://admin:p4ssw0rd@splunk.example.com:8088",
|
||||
access_token="tok",
|
||||
index="dmarc",
|
||||
)
|
||||
client.session = MagicMock()
|
||||
client.session.post.return_value = _ok_response()
|
||||
client.save_aggregate_reports_to_splunk(_aggregate_report())
|
||||
self.assertEqual(
|
||||
client.session.post.call_args.args[0],
|
||||
"https://admin:p4ssw0rd@splunk.example.com:8088"
|
||||
"/services/collector/event/1.0",
|
||||
)
|
||||
|
||||
|
||||
class TestHECClientClose(unittest.TestCase):
|
||||
def test_close_closes_session(self):
|
||||
client = _client()
|
||||
|
||||
+570
-26
@@ -4,16 +4,19 @@ import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import traceback
|
||||
import unittest
|
||||
from datetime import datetime, timezone
|
||||
from importlib.resources import files
|
||||
from tempfile import NamedTemporaryFile
|
||||
from types import SimpleNamespace
|
||||
from typing import cast
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import dns.exception
|
||||
import dns.message
|
||||
import dns.nameserver
|
||||
import dns.query
|
||||
import dns.rdata
|
||||
import dns.rdataclass
|
||||
import dns.rdatatype
|
||||
@@ -514,6 +517,122 @@ class Test(unittest.TestCase):
|
||||
self.assertTrue(parsedmarc.utils.is_outlook_msg(magic))
|
||||
|
||||
|
||||
class TestRedactUrlCredentials(unittest.TestCase):
|
||||
"""Tests for parsedmarc.utils.redact_url_credentials()"""
|
||||
|
||||
def testMasksUserinfoAndKeepsHost(self):
|
||||
cases = [
|
||||
("https://user:secret@host", "https://user:***@host"),
|
||||
("https://user:secret@host:9200/", "https://user:***@host:9200/"),
|
||||
("localhost:9200", "localhost:9200"),
|
||||
("127.0.0.1:9299", "127.0.0.1:9299"),
|
||||
("user:secret@localhost:9200", "user:***@localhost:9200"),
|
||||
# A "://" in the path or query of a scheme-less value is not
|
||||
# the scheme separator; the userinfo at offset 0 is still masked
|
||||
(
|
||||
"user:secret@host:9200/x?r=http://y",
|
||||
"user:***@host:9200/x?r=http://y",
|
||||
),
|
||||
("user:secret@host?r=a://b", "user:***@host?r=a://b"),
|
||||
("https://user:p@ss@host/x", "https://user:***@host/x"),
|
||||
("https://user:p:w@[::1]:9200/", "https://user:***@[::1]:9200/"),
|
||||
("https://[::1]:9200", "https://[::1]:9200"),
|
||||
("//user:pw@host", "//user:***@host"),
|
||||
# Documented over-masking: with no "@" in the authority, an "@"
|
||||
# later in the value is masked through, since it may end a
|
||||
# password that contains "/", "?" or "#"
|
||||
("http://host/path?e=a@b.com", "http://***@b.com"),
|
||||
("https://host/path@file", "https://***@file"),
|
||||
("https://host/?redirect=https://u:p@evil", "https://***@evil"),
|
||||
# Documented non-goal: with userinfo in the authority, credentials
|
||||
# inside a later query value are not masked
|
||||
(
|
||||
"https://a:b@host/?redirect=https://u:p@evil",
|
||||
"https://a:***@host/?redirect=https://u:p@evil",
|
||||
),
|
||||
("https://tok@host", "https://***@host"),
|
||||
("user@host:9200", "***@host:9200"),
|
||||
("https://user:@host", "https://user:***@host"),
|
||||
("https://u%40x:p%3Aw@h", "https://u%40x:***@h"),
|
||||
("not a url @ all", "***@ all"),
|
||||
("", ""),
|
||||
("::::", "::::"),
|
||||
]
|
||||
for value, expected in cases:
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(value), expected
|
||||
)
|
||||
|
||||
def testMasksPasswordContainingAuthorityDelimiter(self):
|
||||
"""A password with an unencoded "/", "?" or "#" ends the authority
|
||||
before the "@" (RFC 3986 section 3.2), which used to leave the value
|
||||
unmasked; everything through the last "@" is masked instead."""
|
||||
cases = [
|
||||
("https://admin:p4ssw0rd/extra@host:3100", "https://***@host:3100"),
|
||||
("https://admin:p4ss?w0rd@host", "https://***@host"),
|
||||
("https://admin:p4ss#w0rd@host", "https://***@host"),
|
||||
("admin:p4ss/w0rd@host:9200", "***@host:9200"),
|
||||
("//admin:p4ss/w0rd@host", "//***@host"),
|
||||
# The password may itself contain "@"; the last one is used
|
||||
("https://admin:p4ss/w@rd@host:3100/x", "https://***@host:3100/x"),
|
||||
# A "://" after the "/" is not the scheme separator
|
||||
(
|
||||
"admin:p4ss/w0rd@host/x?r=http://y",
|
||||
"***@host/x?r=http://y",
|
||||
),
|
||||
]
|
||||
for value, expected in cases:
|
||||
with self.subTest(value=value):
|
||||
result = parsedmarc.utils.redact_url_credentials(value)
|
||||
self.assertEqual(result, expected)
|
||||
self.assertNotIn("p4ss", result)
|
||||
|
||||
def testListMasksPasswordContainingAuthorityDelimiter(self):
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(
|
||||
["https://admin:p4ss/w0rd@h1:9200", "https://h2:9200"]
|
||||
),
|
||||
["https://***@h1:9200", "https://h2:9200"],
|
||||
)
|
||||
|
||||
def testListIsRedactedElementWiseIntoANewList(self):
|
||||
original = ["a:b@h1", "https://h2"]
|
||||
result = parsedmarc.utils.redact_url_credentials(original)
|
||||
self.assertEqual(result, ["a:***@h1", "https://h2"])
|
||||
self.assertIsNot(result, original)
|
||||
self.assertEqual(original, ["a:b@h1", "https://h2"])
|
||||
|
||||
def testListKeepsNonStringItems(self):
|
||||
mixed = cast(list[str], ["a:b@h1", None])
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(mixed), ["a:***@h1", None]
|
||||
)
|
||||
|
||||
def testNonStringInputIsReturnedUnchanged(self):
|
||||
for value in (None, 42):
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(cast(str, value)), value
|
||||
)
|
||||
|
||||
def testLongInputsAreFast(self):
|
||||
"""No regex is used, so adversarial input is linear. A 1 MB string
|
||||
of repeated openers and a 200,000-character run of '@' both finish
|
||||
in milliseconds."""
|
||||
for value in (
|
||||
"://" * 350_000,
|
||||
"@" * 200_000,
|
||||
"a:b@" * 250_000,
|
||||
"a:b/" * 250_000 + "@",
|
||||
"/@" * 500_000,
|
||||
):
|
||||
with self.subTest(prefix=value[:8]):
|
||||
started = time.monotonic()
|
||||
parsedmarc.utils.redact_url_credentials(value)
|
||||
self.assertLess(time.monotonic() - started, 5)
|
||||
|
||||
|
||||
class TestSanitizeForLog(unittest.TestCase):
|
||||
"""Tests for parsedmarc.utils.sanitize_for_log()"""
|
||||
|
||||
@@ -539,6 +658,197 @@ class TestSanitizeForLog(unittest.TestCase):
|
||||
self.assertNotIn("\n", result)
|
||||
|
||||
|
||||
class TestDownloadUrlCredentialsNotLogged(unittest.TestCase):
|
||||
"""A download URL with userinfo must not leak its password through the
|
||||
"Trying to fetch" debug line or the warning that embeds the httpx
|
||||
``HTTPStatusError`` message (which quotes the full URL).
|
||||
|
||||
The loaders run for real; only the network is replaced, with an
|
||||
``httpx.MockTransport`` returning 401 so httpx itself builds the
|
||||
exception text.
|
||||
"""
|
||||
|
||||
URL = "https://dluser:DLSECRET@downloads.example.test/map.dat"
|
||||
|
||||
@staticmethod
|
||||
def _transport_returning(handler):
|
||||
def get(url, **kwargs):
|
||||
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
|
||||
return client.get(url, **kwargs)
|
||||
|
||||
return get
|
||||
|
||||
@classmethod
|
||||
def _status_getter(cls, reason_phrase=None):
|
||||
extensions = {}
|
||||
if reason_phrase is not None:
|
||||
extensions["reason_phrase"] = reason_phrase
|
||||
return cls._transport_returning(
|
||||
lambda request: httpx.Response(401, extensions=extensions)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _connect_error_getter(url, **kwargs):
|
||||
def handler(request):
|
||||
raise httpx.ConnectError("connection refused", request=request)
|
||||
|
||||
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
|
||||
return client.get(url, **kwargs)
|
||||
|
||||
# Reason phrases containing URL delimiters used to defeat redaction of
|
||||
# the HTTPStatusError text, which puts the phrase before the URL.
|
||||
REASON_PHRASES = (
|
||||
None,
|
||||
b"Unauthorized/Expired",
|
||||
b"Unauthorized?",
|
||||
b"Gone#",
|
||||
b"Not @ home",
|
||||
)
|
||||
|
||||
def _assert_redacted(self, logs):
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("DLSECRET", output)
|
||||
self.assertTrue(
|
||||
any(
|
||||
line.startswith("DEBUG") and "downloads.example.test" in line
|
||||
for line in logs.output
|
||||
),
|
||||
output,
|
||||
)
|
||||
warnings = [line for line in logs.output if line.startswith("WARNING")]
|
||||
self.assertTrue(
|
||||
any(
|
||||
"HTTP 401" in line
|
||||
and "'https://dluser:***@downloads.example.test/map.dat'" in line
|
||||
for line in warnings
|
||||
),
|
||||
output,
|
||||
)
|
||||
|
||||
def _run_psl(self, url=None):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
parsedmarc.utils.load_psl_overrides(url=url or self.URL)
|
||||
return logs
|
||||
|
||||
def _run_ip_db(self, url=None):
|
||||
tmp_dir = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp_dir, ignore_errors=True)
|
||||
old_path = parsedmarc.utils._IP_DB_PATH
|
||||
self.addCleanup(setattr, parsedmarc.utils, "_IP_DB_PATH", old_path)
|
||||
parsedmarc.utils._IP_DB_PATH = None
|
||||
with (
|
||||
patch("parsedmarc.utils.tempfile.gettempdir", return_value=tmp_dir),
|
||||
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
|
||||
):
|
||||
parsedmarc.utils.load_ip_db(url=url or self.URL)
|
||||
return logs
|
||||
|
||||
def _run_reverse_dns_map(self, url=None):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
parsedmarc.utils.load_reverse_dns_map({}, url=url or self.URL)
|
||||
return logs
|
||||
|
||||
def _check_loader(self, run):
|
||||
for phrase in self.REASON_PHRASES:
|
||||
with self.subTest(reason_phrase=phrase):
|
||||
with patch("parsedmarc.utils.httpx.get", self._status_getter(phrase)):
|
||||
self._assert_redacted(run())
|
||||
|
||||
def _check_transport_error(self, run):
|
||||
with patch("parsedmarc.utils.httpx.get", self._connect_error_getter):
|
||||
logs = run()
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("DLSECRET", output)
|
||||
self.assertTrue(
|
||||
any(
|
||||
line.startswith("WARNING")
|
||||
and "ConnectError: connection refused" in line
|
||||
for line in logs.output
|
||||
),
|
||||
output,
|
||||
)
|
||||
|
||||
# The password holds an unencoded "/", "?" or "#", which RFC 3986 section
|
||||
# 3.2 ends the authority at, so the text after the colon is read as the
|
||||
# port. Real httpx raises InvalidURL("Invalid port: 'p4ssw0rd'") while
|
||||
# parsing, before any network I/O (httpx 0.28.1 _urlparse.py:
|
||||
# ``raise InvalidURL(f"Invalid port: {port!r}")``). InvalidURL is not an
|
||||
# httpx.HTTPError, and its message quotes part of the password.
|
||||
MALFORMED_URLS = (
|
||||
("https://admin:p4ssw0rd/extra@downloads.example.test/map.dat", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@downloads.example.test/map.dat", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@downloads.example.test/map.dat", "p4ss"),
|
||||
)
|
||||
INVALID_URL_MESSAGE = (
|
||||
"invalid URL (details withheld because they may include credentials)"
|
||||
)
|
||||
|
||||
def _check_invalid_url(self, run, source):
|
||||
for url, secret in self.MALFORMED_URLS:
|
||||
with self.subTest(url=url):
|
||||
requested = []
|
||||
real_get = httpx.get
|
||||
|
||||
def recording_get(requested_url, **kwargs):
|
||||
requested.append(requested_url)
|
||||
if requested_url != url:
|
||||
# e.g. the PSL overrides fetch that
|
||||
# load_reverse_dns_map triggers; stay offline
|
||||
raise httpx.ConnectError("offline")
|
||||
return real_get(requested_url, **kwargs)
|
||||
|
||||
with patch("parsedmarc.utils.httpx.get", recording_get):
|
||||
logs = run(url) # must return, not crash on InvalidURL
|
||||
# The loader still requested the real URL, not a redacted one
|
||||
self.assertIn(url, requested)
|
||||
output = "\n".join(logs.output)
|
||||
# Neither the warning nor the "Trying to fetch" debug line
|
||||
# may show any part of the password
|
||||
self.assertNotIn(secret, output)
|
||||
self.assertIn(
|
||||
f"DEBUG:parsedmarc.log:Trying to fetch {source} from "
|
||||
"https://***@downloads.example.test/map.dat...",
|
||||
logs.output,
|
||||
)
|
||||
self.assertIn(
|
||||
f"WARNING:parsedmarc.log:Failed to fetch {source}: "
|
||||
f"{self.INVALID_URL_MESSAGE}",
|
||||
logs.output,
|
||||
)
|
||||
self.assertNotIn("Not a valid CSV file", output)
|
||||
self.assertNotIn("Failed to save IP database", output)
|
||||
|
||||
def test_psl_overrides_loader_invalid_url(self):
|
||||
"""A malformed ``psl_overrides_url`` used to crash startup with the
|
||||
InvalidURL traceback; it now warns and falls back to the local file."""
|
||||
self._check_invalid_url(self._run_psl, "PSL overrides")
|
||||
self.assertIn(".linode.com", parsedmarc.utils.psl_overrides)
|
||||
|
||||
def test_ip_db_loader_invalid_url(self):
|
||||
self._check_invalid_url(self._run_ip_db, "IP database")
|
||||
|
||||
def test_reverse_dns_map_loader_invalid_url(self):
|
||||
self._check_invalid_url(self._run_reverse_dns_map, "reverse DNS map")
|
||||
|
||||
def test_psl_overrides_loader(self):
|
||||
self._check_loader(self._run_psl)
|
||||
|
||||
def test_ip_db_loader(self):
|
||||
self._check_loader(self._run_ip_db)
|
||||
|
||||
def test_reverse_dns_map_loader(self):
|
||||
self._check_loader(self._run_reverse_dns_map)
|
||||
|
||||
def test_psl_overrides_loader_transport_error(self):
|
||||
self._check_transport_error(self._run_psl)
|
||||
|
||||
def test_ip_db_loader_transport_error(self):
|
||||
self._check_transport_error(self._run_ip_db)
|
||||
|
||||
def test_reverse_dns_map_loader_transport_error(self):
|
||||
self._check_transport_error(self._run_reverse_dns_map)
|
||||
|
||||
|
||||
class TestLoadPSLOverrides(unittest.TestCase):
|
||||
"""Covers `parsedmarc.utils.load_psl_overrides`."""
|
||||
|
||||
@@ -1397,14 +1707,24 @@ class TestQueryDnsRetries(unittest.TestCase):
|
||||
self.assertEqual(mock_resolve.call_count, 3)
|
||||
|
||||
|
||||
def _dnspython_http_module():
|
||||
"""Returns the HTTP library dnspython's DoH code imported: httpx2 on
|
||||
dnspython 2.9+, httpx on 2.7 and 2.8. Derived from dnspython rather than
|
||||
from parsedmarc's helper, so the tests below check the helper against
|
||||
what ``dns.query.https()`` actually accepts."""
|
||||
return getattr(dns.query, "httpx2", None) or httpx
|
||||
|
||||
|
||||
class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
"""Tests for the DNS over HTTPS and DNS over TLS transports selected by
|
||||
the form of each ``nameservers`` entry
|
||||
(https://github.com/domainaware/parsedmarc/issues/880). The end-to-end
|
||||
(https://github.com/domainaware/parsedmarc/issues/880). Most end-to-end
|
||||
tests mock at the dnspython SDK boundary (``dns.query.https`` /
|
||||
``dns.query.tls``) and assert on the answer query_dns parses back out of
|
||||
a real DNS response message, so the whole chain — entry mapping,
|
||||
resolver, nameserver object, transport call — is exercised."""
|
||||
resolver, nameserver object, transport call — is exercised. The DoH
|
||||
session regression test (issue #935) runs the real ``dns.query.https``
|
||||
and fakes only the HTTP transport underneath it."""
|
||||
|
||||
def setUp(self):
|
||||
# The DoH session is process-wide state; keep test ordering from
|
||||
@@ -1424,25 +1744,31 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
self.addCleanup(restore)
|
||||
|
||||
@staticmethod
|
||||
def _ptr_responder(captured: list, answer: str = "dns.example."):
|
||||
def _ptr_response(request: dns.message.Message, answer: str = "dns.example."):
|
||||
"""Builds a response to ``request`` answering its PTR question with
|
||||
``answer``."""
|
||||
response = dns.message.make_response(request)
|
||||
# find_rrset(create=True) rather than appending to response.answer,
|
||||
# so the message's rrset index is updated too — dnspython looks the
|
||||
# answer up through that index.
|
||||
rrset = response.find_rrset(
|
||||
response.answer,
|
||||
request.question[0].name,
|
||||
dns.rdataclass.IN,
|
||||
dns.rdatatype.PTR,
|
||||
create=True,
|
||||
)
|
||||
rrset.add(dns.rdata.from_text("IN", "PTR", answer), 300)
|
||||
return response
|
||||
|
||||
@classmethod
|
||||
def _ptr_responder(cls, captured: list, answer: str = "dns.example."):
|
||||
"""Builds a dns.query stand-in that answers any PTR query with
|
||||
``answer``, recording the keyword arguments it was called with."""
|
||||
|
||||
def responder(request, *args, **kwargs):
|
||||
captured.append(kwargs)
|
||||
response = dns.message.make_response(request)
|
||||
# find_rrset(create=True) rather than appending to
|
||||
# response.answer, so the message's rrset index is updated too —
|
||||
# dnspython looks the answer up through that index.
|
||||
rrset = response.find_rrset(
|
||||
response.answer,
|
||||
request.question[0].name,
|
||||
dns.rdataclass.IN,
|
||||
dns.rdatatype.PTR,
|
||||
create=True,
|
||||
)
|
||||
rrset.add(dns.rdata.from_text("IN", "PTR", answer), 300)
|
||||
return response
|
||||
return cls._ptr_response(request, answer)
|
||||
|
||||
return responder
|
||||
|
||||
@@ -1541,16 +1867,20 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
must fail at configuration time naming the entry, not parse
|
||||
"successfully" with no TLS certificate identity and then fail at
|
||||
query time with an opaque certificate error. Userinfo and query
|
||||
components are rejected the same way."""
|
||||
for entry in (
|
||||
"tls://9.9.9.9/dns.quad9.net",
|
||||
"tls://user@9.9.9.9",
|
||||
"tls://9.9.9.9?hostname=dns.quad9.net",
|
||||
components are rejected the same way, with the userinfo masked in
|
||||
the message (GHSA-hjr3-rx4c-24cv)."""
|
||||
for entry, shown in (
|
||||
("tls://9.9.9.9/dns.quad9.net", "tls://9.9.9.9/dns.quad9.net"),
|
||||
("tls://user@9.9.9.9", "tls://***@9.9.9.9"),
|
||||
(
|
||||
"tls://9.9.9.9?hostname=dns.quad9.net",
|
||||
"tls://9.9.9.9?hostname=dns.quad9.net",
|
||||
),
|
||||
):
|
||||
with self.subTest(entry=entry):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
parsedmarc.utils._nameservers_to_resolver_input([entry])
|
||||
self.assertIn(entry, str(ctx.exception))
|
||||
self.assertIn(shown, str(ctx.exception))
|
||||
|
||||
def test_unsplittable_entry_is_left_for_dnspython_to_reject(self):
|
||||
"""urlsplit itself raises on some malformed input (an unbalanced
|
||||
@@ -1563,10 +1893,12 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
with self.assertRaises(ValueError):
|
||||
dns.resolver.Resolver(configure=False).nameservers = [entry]
|
||||
|
||||
def test_doh_query_returns_answers_through_a_shared_httpx_client(self):
|
||||
def test_doh_query_returns_answers_through_a_shared_http_client(self):
|
||||
"""An https:// nameserver resolves through dns.query.https, and the
|
||||
session it is given is the module's shared client, an httpx.Client
|
||||
with trust_env left on — httpx's documented switch for honoring
|
||||
session it is given is the module's shared client, an instance of the
|
||||
Client class of the HTTP library dnspython imported (httpx2 on
|
||||
dnspython 2.9+, httpx before) with trust_env left on — the
|
||||
documented switch, in both libraries, for honoring
|
||||
HTTP_PROXY/HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE, which is what a
|
||||
proxy-only network needs (issue #880). dnspython's stock DoH
|
||||
nameserver passes no session at all, and the client dns.query.https
|
||||
@@ -1586,7 +1918,7 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
self.assertEqual(records, ["dns.example"])
|
||||
self.assertEqual(len(captured), 1)
|
||||
session = captured[0]["session"]
|
||||
self.assertIsInstance(session, httpx.Client)
|
||||
self.assertIsInstance(session, _dnspython_http_module().Client)
|
||||
self.assertIs(session.trust_env, True)
|
||||
self.assertIs(session, parsedmarc.utils._DOH_SESSION)
|
||||
|
||||
@@ -1608,6 +1940,72 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
self.assertEqual(captured[0]["server_hostname"], "dns.quad9.net")
|
||||
self.assertEqual(captured[0]["port"], 853)
|
||||
|
||||
def test_doh_session_is_accepted_by_the_real_dns_query_https(self):
|
||||
"""Regression test for issue #935: dnspython 2.9 moved DoH to
|
||||
httpx2, and ``dns.query.https()`` rejects a session of any other
|
||||
class before sending anything (dnspython 2.9.1 ``dns/query.py``
|
||||
line 551: ``if session and not isinstance(session, httpx2.Client):
|
||||
raise ValueError("session parameter must be an httpx2.Client")``;
|
||||
2.8.0 checks ``httpx.Client`` the same way at line 566). The old
|
||||
``httpx.Client`` session therefore failed every DoH query under 2.9,
|
||||
and the resolver retried it until the lifetime ran out. Here the real
|
||||
``dns.query.https`` runs, and only the network is faked, by patching
|
||||
the ``HTTPTransport`` of the library dnspython imported; a session
|
||||
of the wrong class raises LifetimeTimeout instead of returning the
|
||||
answer. The transport also observes the POSTed DNS query itself."""
|
||||
http_module = _dnspython_http_module()
|
||||
questions: list = []
|
||||
|
||||
def handle_request(transport, request):
|
||||
query = dns.message.from_wire(request.read())
|
||||
questions.append(query.question[0].to_text())
|
||||
return http_module.Response(
|
||||
200,
|
||||
headers={"content-type": "application/dns-message"},
|
||||
# An iterator rather than bytes: a bytes body counts as read
|
||||
# already, so the client never closes the stream, and
|
||||
# ``response.elapsed``, which dns.query.https reads, is set
|
||||
# only on close
|
||||
content=iter([self._ptr_response(query).to_wire()]),
|
||||
request=request,
|
||||
)
|
||||
|
||||
with patch.object(http_module.HTTPTransport, "handle_request", handle_request):
|
||||
records = parsedmarc.utils.query_dns(
|
||||
"1.0.0.1.in-addr.arpa",
|
||||
"PTR",
|
||||
nameservers=["https://dns.example/dns-query"],
|
||||
timeout=0.5,
|
||||
)
|
||||
self.assertEqual(records, ["dns.example"])
|
||||
self.assertEqual(questions, ["1.0.0.1.in-addr.arpa. IN PTR"])
|
||||
|
||||
def test_doh_client_class_prefers_httpx2_when_dnspython_imported_it(self):
|
||||
"""dnspython 2.9+ binds ``dns.query.httpx2``; its Client is the one
|
||||
``dns.query.https()`` accepts, even with httpx importable too."""
|
||||
fake_httpx2 = SimpleNamespace(Client=type("Client", (), {}))
|
||||
with patch.dict(dns.query.__dict__, {"httpx2": fake_httpx2, "httpx": httpx}):
|
||||
self.assertIs(parsedmarc.utils._doh_client_class(), fake_httpx2.Client)
|
||||
|
||||
def test_doh_client_class_uses_httpx_when_dnspython_imported_it(self):
|
||||
"""dnspython 2.7 and 2.8 bind ``dns.query.httpx`` and no httpx2, so
|
||||
the class is ``httpx.Client`` — even if an unrelated httpx2 install
|
||||
exists, since dnspython 2.8 never checks against it."""
|
||||
fake_httpx = SimpleNamespace(Client=type("Client", (), {}))
|
||||
with patch.dict(dns.query.__dict__, {"httpx": fake_httpx}):
|
||||
dns.query.__dict__.pop("httpx2", None)
|
||||
self.assertIs(parsedmarc.utils._doh_client_class(), fake_httpx.Client)
|
||||
|
||||
def test_doh_client_class_falls_back_to_httpx(self):
|
||||
"""With neither library bound in dns.query (dnspython without its
|
||||
[doh] extra), the class falls back to ``httpx.Client``; dnspython
|
||||
(or httpx, if h2 is also missing) then raises its own error when a
|
||||
DoH query is made."""
|
||||
with patch.dict(dns.query.__dict__):
|
||||
dns.query.__dict__.pop("httpx2", None)
|
||||
dns.query.__dict__.pop("httpx", None)
|
||||
self.assertIs(parsedmarc.utils._doh_client_class(), httpx.Client)
|
||||
|
||||
def test_doh_session_is_reused_within_a_process(self):
|
||||
"""Consecutive calls hand back the same client, so DoH lookups
|
||||
reuse the connection instead of renegotiating TLS per query."""
|
||||
@@ -1626,6 +2024,152 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
self.assertEqual(parsedmarc.utils._DOH_SESSION_PID, os.getpid())
|
||||
|
||||
|
||||
class TestNameserverCredentialsNotLeaked(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: nameserver URL credentials must not reach an
|
||||
exception message, its printed traceback, or the log."""
|
||||
|
||||
def test_url_secrets_finds_the_password_and_its_port_like_prefix(self):
|
||||
"""The secret is the userinfo after its first ":" (the whole
|
||||
userinfo when there is none). A "/", "?" or "#" in it ends the URL
|
||||
authority (RFC 3986 section 3.2), so urlsplit() reads the text
|
||||
before it as the port and quotes it in its error; that prefix is a
|
||||
secret too."""
|
||||
cases = [
|
||||
("https://doh:p4ss@dns.example/q", ["p4ss"]),
|
||||
("https://doh:p4ss/w0rd@dns.example/q", ["p4ss/w0rd", "p4ss"]),
|
||||
("https://doh:p4ss?w0rd@dns.example/q", ["p4ss?w0rd", "p4ss"]),
|
||||
("https://doh:p4ss#w0rd@dns.example/q", ["p4ss#w0rd", "p4ss"]),
|
||||
# The prefix ends at the first of the three delimiters
|
||||
("https://doh:ab#cd/ef@dns.example/q", ["ab#cd/ef", "ab"]),
|
||||
("https://doh:p:w@dns.example/q", ["p:w"]),
|
||||
("https://doh:p@ss@dns.example/q", ["p@ss"]),
|
||||
("https://S3CRETTOKEN@dns.example/q", ["S3CRETTOKEN"]),
|
||||
("tls://dot:p4ss/w0rd@1.1.1.1", ["p4ss/w0rd", "p4ss"]),
|
||||
# Empty secrets are never returned: str.replace("", ...) would
|
||||
# insert the mask between every character
|
||||
("https://doh:@dns.example/q", []),
|
||||
("https://doh:/x@dns.example/q", ["/x"]),
|
||||
("https://dns.example/dns-query", []),
|
||||
("8.8.8.8", []),
|
||||
("", []),
|
||||
]
|
||||
for value, expected in cases:
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(parsedmarc.utils._url_secrets(value), expected)
|
||||
|
||||
def test_mask_nameserver_secrets_masks_entries_passwords_and_prefixes(self):
|
||||
"""Each configured entry is replaced by its redacted form, and each
|
||||
secret in it wherever it still appears, including the bare port-like
|
||||
prefix that urlsplit() and httpx quote."""
|
||||
mask = parsedmarc.utils._mask_nameserver_secrets
|
||||
url = "https://doh:p4ss/w0rd@dns.example/dns-query"
|
||||
text = (
|
||||
f"Server {url} answered x; "
|
||||
"Port could not be cast to integer value as 'p4ss'; "
|
||||
"Invalid port: 'p4ss/w0rd'"
|
||||
)
|
||||
self.assertEqual(
|
||||
mask(text, [url, "8.8.8.8"]),
|
||||
"Server https://***@dns.example/dns-query answered x; "
|
||||
"Port could not be cast to integer value as '***'; "
|
||||
"Invalid port: '***'",
|
||||
)
|
||||
self.assertEqual(
|
||||
mask(
|
||||
"Server https://u:PW123@h.example/q failed (PW123)",
|
||||
["https://u:PW123@h.example/q"],
|
||||
),
|
||||
"Server https://u:***@h.example/q failed (***)",
|
||||
)
|
||||
self.assertEqual(mask("abc", None), "abc")
|
||||
self.assertEqual(mask("abc", ["https://u:@h"]), "abc")
|
||||
|
||||
def test_mask_nameserver_secrets_handles_unsplittable_entries(self):
|
||||
"""urlsplit() raises ValueError for an entry such as ``https://[x``;
|
||||
the masking uses string methods only, so it still returns text."""
|
||||
text = "Server https://[x failed; Server https://u:PW@[x failed"
|
||||
self.assertEqual(
|
||||
parsedmarc.utils._mask_nameserver_secrets(
|
||||
text, ["https://[x", "https://u:PW@[x"]
|
||||
),
|
||||
"Server https://[x failed; Server https://u:***@[x failed",
|
||||
)
|
||||
|
||||
def test_malformed_https_and_tls_ports_are_reported_without_secrets(self):
|
||||
"""A password containing "/", "?" or "#" makes urlsplit() read its
|
||||
start as the port, and its ValueError quotes that text; dnspython
|
||||
2.9.1 raised it from ``DoHNameserver.answer_port()`` only after a
|
||||
failed query (``dns/nameserver.py``). Both schemes are now rejected
|
||||
before any query, naming the entry with its credentials masked and
|
||||
with no chained exception repeating the urlsplit message."""
|
||||
for entry, shown in (
|
||||
(
|
||||
"https://doh:p4ss/w0rd@dns.example/dns-query",
|
||||
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"https://doh:p4ss?w0rd@dns.example/dns-query",
|
||||
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"tls://dot:p4ss#w0rd@1.1.1.1",
|
||||
"Invalid DNS over TLS nameserver tls://***@1.1.1.1",
|
||||
),
|
||||
(
|
||||
"https://dns.example:notaport/dns-query",
|
||||
"Invalid DNS over HTTPS nameserver https://dns.example:notaport/",
|
||||
),
|
||||
):
|
||||
with self.subTest(entry=entry):
|
||||
with patch("dns.query.https", side_effect=AssertionError):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
parsedmarc.utils.query_dns(
|
||||
"1.0.0.1.in-addr.arpa",
|
||||
"PTR",
|
||||
nameservers=[entry],
|
||||
timeout=0.2,
|
||||
)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertIn(shown, str(ctx.exception))
|
||||
self.assertIsNone(ctx.exception.__cause__)
|
||||
self.assertIsNone(ctx.exception.__context__)
|
||||
for secret in ("p4ss", "w0rd"):
|
||||
self.assertNotIn(secret, trace)
|
||||
|
||||
def test_dnspython_rejection_of_an_entry_is_masked(self):
|
||||
"""dnspython rejects an entry urlsplit() cannot split with a message
|
||||
that quotes it as configured ("nameserver https://u:pw@[x is not a
|
||||
dns.nameserver.Nameserver instance ..."); query_dns() masks it."""
|
||||
entry = "https://doh:S3CRETPW@[dns.example/dns-query"
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
parsedmarc.utils.query_dns(
|
||||
"1.0.0.1.in-addr.arpa", "PTR", nameservers=[entry], timeout=0.2
|
||||
)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertIn("https://doh:***@[dns.example/dns-query", str(ctx.exception))
|
||||
self.assertNotIn("S3CRETPW", trace)
|
||||
|
||||
def test_get_reverse_dns_debug_log_masks_nameserver_credentials(self):
|
||||
"""get_reverse_dns() logs the DNS error it swallows at debug level,
|
||||
and LifetimeTimeout/NoNameservers quote each failing nameserver
|
||||
URL, so the configured credentials are masked there too."""
|
||||
url = "https://doh:S3CRETPW@dns.example/dns-query"
|
||||
|
||||
def fail(*args, **kwargs):
|
||||
raise OSError(f"connection to {url} refused (S3CRETPW)")
|
||||
|
||||
with patch("dns.query.https", side_effect=fail):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
result = parsedmarc.utils.get_reverse_dns(
|
||||
"1.0.0.1", nameservers=[url], timeout=0.2, retries=0
|
||||
)
|
||||
self.assertIsNone(result)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertIn("get_reverse_dns(1.0.0.1) exception:", output)
|
||||
self.assertIn("https://doh:***@dns.example/dns-query", output)
|
||||
self.assertNotIn("S3CRETPW", output)
|
||||
|
||||
|
||||
class TestLoadIpDb(unittest.TestCase):
|
||||
"""Tests for the load_ip_db() download/cache/bundled fallback chain,
|
||||
mocking at the httpx SDK boundary."""
|
||||
|
||||
@@ -114,6 +114,50 @@ class TestWebhookErrorHandling(unittest.TestCase):
|
||||
client.save_smtp_tls_report_to_webhook('{"t": 1}')
|
||||
|
||||
|
||||
class TestWebhookInvalidUrlDoesNotLeakPassword(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
|
||||
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
|
||||
text after the colon as the port and raises
|
||||
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
|
||||
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
|
||||
any network I/O. The error is logged on every report."""
|
||||
|
||||
MALFORMED = (
|
||||
("https://admin:p4ssw0rd/extra@hooks.example.com/x", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@hooks.example.com/x", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@hooks.example.com/x", "p4ss"),
|
||||
)
|
||||
|
||||
def test_invalid_url_is_logged_without_password(self):
|
||||
saves = (
|
||||
"save_aggregate_report_to_webhook",
|
||||
"save_failure_report_to_webhook",
|
||||
"save_smtp_tls_report_to_webhook",
|
||||
)
|
||||
for url, secret in self.MALFORMED:
|
||||
client = WebhookClient(url, url, url)
|
||||
for method in saves:
|
||||
with self.subTest(url=url, method=method):
|
||||
with self.assertLogs("parsedmarc.log", level="ERROR") as cm:
|
||||
getattr(client, method)('{"a": 1}') # must not raise
|
||||
self.assertEqual(
|
||||
cm.output,
|
||||
[
|
||||
"ERROR:parsedmarc.log:Webhook Error: invalid webhook "
|
||||
"URL (details withheld because they may include "
|
||||
"credentials)"
|
||||
],
|
||||
)
|
||||
self.assertNotIn(secret, "\n".join(cm.output))
|
||||
|
||||
def test_valid_url_is_the_one_posted(self):
|
||||
url = "https://admin:p4ssw0rd@hooks.example.com/x"
|
||||
client = WebhookClient(url, url, url)
|
||||
client.session = MagicMock()
|
||||
client.save_aggregate_report_to_webhook('{"a": 1}')
|
||||
client.session.post.assert_called_once_with(url, content='{"a": 1}', timeout=60)
|
||||
|
||||
|
||||
class TestWebhookClientClose(unittest.TestCase):
|
||||
def test_close_closes_session(self):
|
||||
client = _client()
|
||||
|
||||
Reference in new issue
Block a user