mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-06 16:59:23 +00:00
Merge commit from fork
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv) With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init lines and opensearch.set_hosts() logged configured URLs verbatim, so userinfo passwords reached the console and log_file (CWE-532). Triage found more paths of the same class: - the httpx "HTTP Request:" line, forwarded at --verbose/--debug, contains the full URL (str(httpx.URL) keeps userinfo and query), so webhook and HEC URL credentials and the IPinfo ?token= were logged on every request; - the PSL overrides, IP database and reverse DNS map downloaders logged their URL at DEBUG and str(HTTPStatusError), which embeds the full URL, at WARNING; - libpq echoes the DSN or a password fragment for some malformed PostgreSQL connection strings, logged as an output-client error. Add redact_url_credentials() (string methods only; urlsplit misparses scheme-less "user:pw@host:port") and use it at each log site; add a filter on the httpx logger that masks userinfo and the token query value; build download-failure warnings from the status code and the redacted request URL instead of the exception text; replace psycopg ProgrammingError detail from a connection_string with a fixed message. Docs now recommend separate user/password settings over URL credentials. Reported by helixnebulatech. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Mask credentials in the Loki URL in the log and LokiError text The Loki output (#926) landed on master after this fix was written. Its url keeps userinfo (copy_with(path=...) preserves it), and the --debug "Initializing Loki client" line and both LokiError messages, which the CLI logs on a failed push, included it as-is: the same CWE-532 path as the Splunk HEC line. Mask it with redact_url_credentials() at all three sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532): - httpx.InvalidURL quotes the component it rejected, so a password with an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there) leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats it. Loki, Splunk HEC, the webhook and the three download loaders now report a fixed "details withheld" message raised from None. InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url crashed startup; the loaders now treat it as a download failure. - redact_url_credentials() masks through the last "@" when the authority has no "@" but one follows it, so the --debug lines no longer print such passwords; the userinfo lookup is shared with the new nameserver masking. - dnspython 2.9 DoH logs its request line on the httpx2 logger, which was forwarded but not filtered; the filter now covers httpx2 and httpx2.URL. - [general] nameservers / --nameservers credentials are masked in the DNS pre-flight error, reverse DNS debug errors and per-report errors, and unusable https:// / tls:// ports are rejected before any query. - Docs recommend token/username settings for [loki] and [splunk_hec]; AGENTS.md gains rules for keeping credentials out of logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
17 files changed
+1595
-73
No files matched your search
@@ -135,6 +135,18 @@ Assume every regex in the package runs on content a remote sender controls (mess
|
||||
- **Time the pattern on an adversarial input before shipping it**, not just on the samples: a 1 MB string of the repeated opener with no closer, and a 200,000-character whitespace run. A linear pattern finishes in milliseconds; anything that takes seconds is a bug. Add that input as a time-bound regression test with a generous bound (seconds) and cite the measured old timing in the docstring, as the regression tests in `tests/test_init.py` and `tests/test_utils.py` do.
|
||||
- Python 3.10 is the floor, so possessive quantifiers and atomic groups (3.11+) are not available; fix the pattern's structure instead.
|
||||
|
||||
### Credentials must never reach logs or error text
|
||||
|
||||
Assume every configured URL, host list, connection string, and token carries a secret, and that every log line and exception message ends up on a console, in `log_file`, or in a pasted bug report (CWE-532). GHSA-hjr3-rx4c-24cv found leaks in `--debug` client-init lines, the forwarded httpx request line, download warnings, libpq's DSN echo, and httpx's `InvalidURL` text. Before writing a log call or an exception message that touches configuration:
|
||||
|
||||
- **Never log a configured value as-is.** Pass URLs and host lists through `redact_url_credentials()` (`parsedmarc/utils.py`), which masks userinfo (`https://user:***@host`), handles scheme-less `user:pw@host:port`, and, when the authority has no `@` but one follows it (a `/`, `?`, or `#` in the password, but also a credential-free URL or free text with a later `@`), masks through the last `@` (`https://***@host`). Redaction is log-only: the SDK or transport still gets the real value, so mask at the log call and not in `opts` or a client attribute.
|
||||
- **Treat third-party exception text as a leak until shown otherwise.** `httpx.HTTPStatusError` embeds the full request URL. `httpx.InvalidURL` quotes the component it rejected, so a password with an unencoded `/`, `?`, or `#` becomes `Invalid port: 'p4ssw0rd'`. libpq echoes the DSN for some malformed connection strings. Don't log `str(e)` from parsing, connecting to, or requesting a configured endpoint. Build the message from structured fields (`_describe_http_error()`: status code plus the redacted request URL), or use a fixed "details withheld because they may include credentials" message. Read the library source in the venv to check what its messages contain; don't assume.
|
||||
- **Raise the replacement `from None`.** `from e` and the implicit `__context__` both print the original exception when it is logged with `logger.exception` (the SIGHUP reload path does this) or when parsedmarc crashes, and `InvalidURL`'s own context is a `ValueError` that repeats the secret.
|
||||
- **Catch the right class.** `httpx.InvalidURL` is not an `httpx.HTTPError`, so an `except httpx.HTTPError` falls through to a generic handler that logs the raw text, or doesn't catch it at all and crashes startup.
|
||||
- **Check dependency loggers.** httpx logs every request URL at INFO. `_configure_dependency_logging()` (`parsedmarc/cli.py`) attaches `_RedactUrlCredentialsFilter` to the `httpx` and `httpx2` loggers (dnspython 2.9+ makes DNS over HTTPS requests through httpx2, and `httpx2.URL` is not an `httpx.URL`). A new dependency logger forwarded at `--verbose`/`--debug` needs the same check, and so does a new library that logs its endpoint.
|
||||
- **Test both halves, including malformed input.** Assert that the secret is absent from the captured log *and* from `"".join(traceback.format_exception(exc))`, and that the real credential still reaches the mocked transport. Include values the samples never contain: a password with `/`, `?`, or `#`; userinfo with no `:` (a bare token); and a scheme-less `user:pw@host:port`.
|
||||
- **Document the safe setting.** When an output has separate `user`/`username` and `password`, `token`, or `api_key` settings, its `usage.md` entry should recommend them over credentials in the URL.
|
||||
|
||||
## Testing standards
|
||||
|
||||
These rules govern *every* test added to `tests/`. They exist because the project has been burned by tests that looked like coverage but caught nothing, and by bug claims that turned out to be wrong about the spec. Both failure modes erode trust faster than missing coverage does.
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
### Security
|
||||
|
||||
- **S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report** ([GHSA-7wqr-f6g9-cm29](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-7wqr-f6g9-cm29), affects 11.1.0 and earlier). `S3Client.save_report_to_s3()` put the sender-controlled report ID into the object key as-is, and sent `org_name` and the other metadata values as-is. A report ID such as `../../x` added path segments to the key, an ID could carry `\`, `%`, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit ([AWS, "Naming Amazon S3 objects"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html)). botocore rejects non-ASCII metadata with `ParamValidationError`, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise `HTTPClientError: Invalid header value` before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the `x-amz-meta-*` header; the 2 KB metadata limit ([AWS, "Working with object metadata"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)) was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are `/`, every AWS "character to avoid" (`` \ { } ^ [ ] " < > ~ # | ` `` and `%`), control characters, and non-ASCII characters, so `%` appears in a key only as the escape character. The punctuation real report IDs use, such as `:`, `@`, `.`, and `_`, is kept. Each metadata value is percent-encoded (`%`, control characters, and non-ASCII characters) so `urllib.parse.unquote()` reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in `_<hash>.json`, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
|
||||
- **Credentials in URLs and the IPinfo API token are no longer written to the log** ([GHSA-hjr3-rx4c-24cv](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-hjr3-rx4c-24cv), affects 11.1.0 and earlier). The `[elasticsearch]` and `[opensearch]` `hosts` settings and the `[splunk_hec]` `url` setting accept URLs, and a URL can carry credentials (`https://user:secret@host:9200`). With `--debug`, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to `log_file`. With `--verbose` or `--debug`, parsedmarc also forwards the `httpx` library's `HTTP Request:` line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in `[general]` `nameservers` (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the `ipinfo_api_token` was logged on every IPinfo lookup as the `token` query parameter. The password, or the whole userinfo when it has no `:`, is now replaced with `***` in these lines (`https://user:***@host:9200`), and so is the `token` query value. Credentials in a `[general]` `nameservers` entry are also masked in the DNS pre-flight error, and credentials in either that setting or a `--nameservers` entry are masked in the `--debug` reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable `https://` or `tls://` entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded `/`, `?`, or `#` (see below), is now rejected before any query is sent. The download URLs `ipinfo_url`, `reverse_dns_map_url`, and `psl_overrides_url` (`[general]`) can carry credentials too: they were logged as-is at `--debug` in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with `--warnings`, `--verbose`, or `--debug`) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by `***`. A malformed `[postgresql]` `connection_string` could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors (`[loki]` `url`, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded `/`, `?`, or `#` ends the URL's authority early ([RFC 3986, section 3.2](https://www.rfc-editor.org/rfc/rfc3986#section-3.2)), so httpx reads the text after the colon as the port and raises `InvalidURL("Invalid port: 'p4ssw0rd'")`, with a chained `ValueError` that repeats that part of the password. That text reached the log from `[loki]` `url` (at startup and on a SIGHUP reload), from `[splunk_hec]` `url` (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from `ipinfo_url` (in the "Failed to save IP database" warning). A malformed `psl_overrides_url` crashed startup with the traceback, because `InvalidURL` is not an `httpx.HTTPError`. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed `psl_overrides_url` now logs a warning and falls back to the `local_psl_overrides_path` file when that is set, or else to the bundled file, and a malformed `reverse_dns_map_url` or `ipinfo_url` is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no `@` in its authority, so the `--debug` "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last `@` (`https://***@host:3100`). A URL with no credentials but an `@` in its path or query is masked the same way in these lines (`https://host/path@file` is logged as `https://***@file`). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: `user` and `password` (or `api_key`) for `[elasticsearch]` and `[opensearch]`, `user` and `password` for `[postgresql]`, `username` and `password` (or `token`) for `[loki]`, and `token` for `[splunk_hec]`. Reported by helixnebulatech.
|
||||
|
||||
### New features
|
||||
|
||||
|
||||
+26
-23
@@ -91,7 +91,9 @@ hosts = 127.0.0.1:9200
|
||||
ssl = False
|
||||
|
||||
[opensearch]
|
||||
hosts = https://admin:admin@127.0.0.1:9200
|
||||
hosts = https://127.0.0.1:9200
|
||||
user = admin
|
||||
password = PasswordGoesHere
|
||||
ssl = True
|
||||
|
||||
[splunk_hec]
|
||||
@@ -571,14 +573,10 @@ The full set of configuration options is:
|
||||
|
||||
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
|
||||
:::{note}
|
||||
Special characters in the username or password must be
|
||||
[URL encoded].
|
||||
:::
|
||||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||||
Do not put credentials in the URL; use `user` and `password`
|
||||
(or `api_key`) instead.
|
||||
- `user` - str: Basic auth username
|
||||
- `password` - str: Basic auth password
|
||||
- `api_key` - str: API key
|
||||
@@ -616,14 +614,10 @@ The full set of configuration options is:
|
||||
|
||||
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
|
||||
:::{note}
|
||||
Special characters in the username or password must be
|
||||
[URL encoded].
|
||||
:::
|
||||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||||
Do not put credentials in the URL; use `user` and `password`
|
||||
(or `api_key`) instead.
|
||||
- `user` - str: Basic auth username
|
||||
- `password` - str: Basic auth password
|
||||
- `api_key` - str: API key
|
||||
@@ -656,7 +650,8 @@ The full set of configuration options is:
|
||||
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
|
||||
(Default: `False`)
|
||||
- `splunk_hec`
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
|
||||
Do not put credentials in the URL; use `token` instead.
|
||||
- `token` - str: The HEC token
|
||||
- `index` - str: The Splunk index to use
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
@@ -715,8 +710,13 @@ The full set of configuration options is:
|
||||
- `password` - str: The database user password (Optional)
|
||||
- `database` - str: The database name (Optional)
|
||||
- `connection_string` - str: A full libpq connection string or URI
|
||||
(e.g. `postgresql://user:pass@host/dbname`). When provided,
|
||||
all individual parameters above are ignored.
|
||||
(e.g. `postgresql://user@host/dbname`). When provided,
|
||||
all individual parameters above are ignored. Prefer the separate
|
||||
`host`, `port`, `user`, `password`, and `database` settings and do
|
||||
not put the password in the connection string: a malformed
|
||||
connection string can make the driver echo it back in an error
|
||||
message. parsedmarc replaces such errors with a fixed message in
|
||||
its log, but cannot stop the driver from producing them.
|
||||
|
||||
The PostgreSQL backend is an optional extra. Install it with
|
||||
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
|
||||
@@ -743,11 +743,13 @@ The full set of configuration options is:
|
||||
database = parsedmarc
|
||||
```
|
||||
|
||||
Or using a DSN/URI:
|
||||
Or using a DSN/URI, with the password supplied separately (for
|
||||
example through the `PGPASSWORD` environment variable or a
|
||||
`~/.pgpass` file, both of which libpq reads):
|
||||
|
||||
```ini
|
||||
[postgresql]
|
||||
connection_string = postgresql://parsedmarc:secret@localhost/parsedmarc
|
||||
connection_string = postgresql://parsedmarc@localhost/parsedmarc
|
||||
```
|
||||
|
||||
Saving parsed data to PostgreSQL is controlled by the `[general]`
|
||||
@@ -954,6 +956,8 @@ The full set of configuration options is:
|
||||
It expects the default `job=parsedmarc` label and `label_fields`.
|
||||
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
|
||||
URL has no path. A URL with a path is used as is.
|
||||
Do not put credentials in the URL; use `username` and `password`
|
||||
(or `token`) instead.
|
||||
- `labels` - str: Comma-separated `name=value` stream labels, merged over
|
||||
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
|
||||
(in a container, the FQDN is the container's hostname)
|
||||
@@ -1568,5 +1572,4 @@ journalctl -u parsedmarc.service -r
|
||||
:::
|
||||
|
||||
[cloudflare's public resolvers]: https://1.1.1.1/
|
||||
[url encoded]: https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
|
||||
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api
|
||||
+83
-7
@@ -20,6 +20,7 @@ from ssl import CERT_NONE, create_default_context
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import dns.exception
|
||||
import dns.query
|
||||
import dns.resolver
|
||||
import dns.reversename
|
||||
import httpx
|
||||
@@ -130,6 +131,7 @@ import parsedmarc.utils
|
||||
from parsedmarc.utils import (
|
||||
InvalidIPinfoAPIKey,
|
||||
ReverseDNSMap,
|
||||
_mask_nameserver_secrets,
|
||||
configure_ipinfo_api,
|
||||
get_base_domain,
|
||||
human_timestamp_to_datetime,
|
||||
@@ -138,6 +140,7 @@ from parsedmarc.utils import (
|
||||
load_psl_overrides,
|
||||
load_reverse_dns_map,
|
||||
query_dns,
|
||||
redact_url_credentials,
|
||||
sanitize_for_log,
|
||||
)
|
||||
|
||||
@@ -684,6 +687,59 @@ _DEPENDENCY_LOGGERS = (
|
||||
)
|
||||
|
||||
|
||||
# URL classes whose instances appear as log-record arguments. dnspython
|
||||
# 2.9+ does DNS over HTTPS through httpx2, whose URL is not an httpx.URL;
|
||||
# like _doh_client_class() in utils.py, take the module dnspython itself
|
||||
# imported (absent under dnspython 2.7/2.8, which use httpx).
|
||||
_HTTP_URL_CLASSES: tuple[type, ...] = (httpx.URL,) + tuple(
|
||||
module.URL for module in (getattr(dns.query, "httpx2", None),) if module
|
||||
)
|
||||
|
||||
|
||||
class _RedactUrlCredentialsFilter(logging.Filter):
|
||||
"""Masks URL credentials and the IPinfo ``token`` query parameter in
|
||||
the arguments of httpx and httpx2 log records.
|
||||
|
||||
httpx and httpx2 log ``HTTP Request: %s %s "%s %d %s"`` with
|
||||
``request.url``, whose string form includes the userinfo password and
|
||||
the query string. httpx2 is what dnspython 2.9+ uses for DNS over HTTPS,
|
||||
so a ``[general]`` ``nameservers`` URL with credentials reaches this
|
||||
line on every lookup.
|
||||
The filter never drops a record; the redaction itself is total (string
|
||||
methods and URL parameter lookups only), so it cannot raise.
|
||||
|
||||
Logger-level filters apply only to records created on the logger the
|
||||
filter is attached to, not to records from its child loggers. httpx
|
||||
(0.28) logs only via ``logging.getLogger("httpx")`` itself and httpx2
|
||||
only via ``logging.getLogger("httpx2")`` (``httpx2/_client.py``), so
|
||||
attaching the filter to those two loggers covers them; a version that
|
||||
logged on a child logger (``httpx.something``) would bypass it.
|
||||
"""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
if isinstance(record.args, tuple):
|
||||
record.args = tuple(self._redact(arg) for arg in record.args)
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _redact(arg: Any) -> Any:
|
||||
if isinstance(arg, _HTTP_URL_CLASSES):
|
||||
if arg.params.get("token") is not None:
|
||||
# Drop the token and append the mask as typed: setting it
|
||||
# with copy_set_param() would percent-encode the "*"s, and
|
||||
# rewriting the encoded form afterwards would also touch a
|
||||
# literal "%2A%2A%2A" elsewhere in the URL.
|
||||
text = str(arg.copy_remove_param("token"))
|
||||
text, fragment_sep, fragment = text.partition("#")
|
||||
separator = "&" if "?" in text else "?"
|
||||
text = f"{text}{separator}token=***{fragment_sep}{fragment}"
|
||||
return redact_url_credentials(text)
|
||||
return redact_url_credentials(str(arg))
|
||||
if isinstance(arg, str):
|
||||
return redact_url_credentials(arg)
|
||||
return arg
|
||||
|
||||
|
||||
def _configure_dependency_logging(level: int) -> None:
|
||||
"""Apply parsedmarc's logging verbosity to dependency loggers.
|
||||
|
||||
@@ -697,6 +753,11 @@ def _configure_dependency_logging(level: int) -> None:
|
||||
duplicates output nor keeps writing to a closed handler. Propagation
|
||||
to the root logger is disabled so that a stray ``logging.basicConfig()``
|
||||
anywhere in the process cannot double-print every dependency record.
|
||||
The ``httpx`` and ``httpx2`` loggers also get a
|
||||
:class:`_RedactUrlCredentialsFilter`
|
||||
(installed once, so reloads do not stack copies) that masks URL
|
||||
credentials and the IPinfo ``token`` query parameter in its
|
||||
``HTTP Request:`` records.
|
||||
CLI-only: library consumers configure logging themselves.
|
||||
"""
|
||||
dep_level = min(level, logging.WARNING)
|
||||
@@ -710,6 +771,12 @@ def _configure_dependency_logging(level: int) -> None:
|
||||
for wanted in logger.handlers:
|
||||
if wanted not in dep_logger.handlers:
|
||||
dep_logger.addHandler(wanted)
|
||||
for name in ("httpx", "httpx2"):
|
||||
http_logger = logging.getLogger(name)
|
||||
if not any(
|
||||
isinstance(f, _RedactUrlCredentialsFilter) for f in http_logger.filters
|
||||
):
|
||||
http_logger.addFilter(_RedactUrlCredentialsFilter())
|
||||
|
||||
|
||||
def _load_config(config_file: str | None = None) -> ConfigParser:
|
||||
@@ -847,12 +914,15 @@ def _parse_config(config: ConfigParser, opts):
|
||||
)
|
||||
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as ns_error:
|
||||
raise ConfigurationError(
|
||||
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {opts.nameservers}: {ns_error}"
|
||||
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {redact_url_credentials(opts.nameservers or [])}: {ns_error}"
|
||||
) from ns_error
|
||||
except Exception as ns_error:
|
||||
# A DNS over HTTPS error can quote the request URL, which
|
||||
# carries the configured nameserver's credentials.
|
||||
raise ConfigurationError(
|
||||
f"DNS pre-flight check failed: {ns_error}"
|
||||
) from ns_error
|
||||
f"DNS pre-flight check failed: {type(ns_error).__name__}: "
|
||||
f"{_mask_nameserver_secrets(str(ns_error), opts.nameservers)}"
|
||||
) from None
|
||||
if "save_aggregate" in general_config:
|
||||
opts.save_aggregate = bool(general_config.getboolean("save_aggregate"))
|
||||
if "save_failure" in general_config:
|
||||
@@ -2070,7 +2140,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
"HEC token and HEC index are required when using HEC URL"
|
||||
)
|
||||
try:
|
||||
logger.debug("Initializing Splunk HEC client: url=%s", opts.hec)
|
||||
logger.debug(
|
||||
"Initializing Splunk HEC client: url=%s",
|
||||
redact_url_credentials(opts.hec),
|
||||
)
|
||||
verify = True
|
||||
if opts.hec_skip_certificate_verification:
|
||||
verify = False
|
||||
@@ -2137,7 +2210,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
|
||||
try:
|
||||
if opts.loki_url:
|
||||
logger.debug("Initializing Loki client: url=%s", opts.loki_url)
|
||||
logger.debug(
|
||||
"Initializing Loki client: url=%s",
|
||||
redact_url_credentials(opts.loki_url),
|
||||
)
|
||||
clients["loki_client"] = loki.LokiClient(
|
||||
opts.loki_url,
|
||||
labels=opts.loki_labels,
|
||||
@@ -2182,7 +2258,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
if opts.elasticsearch_hosts:
|
||||
logger.debug(
|
||||
"Initializing Elasticsearch client: hosts=%s, ssl=%s",
|
||||
opts.elasticsearch_hosts,
|
||||
redact_url_credentials(opts.elasticsearch_hosts),
|
||||
opts.elasticsearch_ssl,
|
||||
)
|
||||
es_aggregate_indexes = _migration_index_names(
|
||||
@@ -2253,7 +2329,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
|
||||
if opts.opensearch_hosts:
|
||||
logger.debug(
|
||||
"Initializing OpenSearch client: hosts=%s, ssl=%s",
|
||||
opts.opensearch_hosts,
|
||||
redact_url_credentials(opts.opensearch_hosts),
|
||||
opts.opensearch_ssl,
|
||||
)
|
||||
os_aggregate_indexes = _migration_index_names(
|
||||
|
||||
+21
-8
@@ -19,7 +19,7 @@ from parsedmarc import (
|
||||
from parsedmarc.constants import USER_AGENT
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.types import AggregateReport, FailureReport, SMTPTLSReport
|
||||
from parsedmarc.utils import human_timestamp_to_unix_timestamp
|
||||
from parsedmarc.utils import human_timestamp_to_unix_timestamp, redact_url_credentials
|
||||
|
||||
PUSH_PATH = "/loki/api/v1/push"
|
||||
TIMESTAMP_MODES = ("ingest", "report_begin")
|
||||
@@ -102,11 +102,22 @@ class LokiClient(object):
|
||||
if not _LABEL_NAME.match(name):
|
||||
raise ValueError(f"Invalid Loki label name '{name}'")
|
||||
|
||||
parsed = httpx.URL(url)
|
||||
# copy_with(path=...) keeps any query string, userinfo and port intact
|
||||
self.url = (
|
||||
str(parsed.copy_with(path=PUSH_PATH)) if parsed.path in ("", "/") else url
|
||||
)
|
||||
try:
|
||||
parsed = httpx.URL(url)
|
||||
# copy_with(path=...) keeps any query string, userinfo and port intact
|
||||
self.url = (
|
||||
str(parsed.copy_with(path=PUSH_PATH))
|
||||
if parsed.path in ("", "/")
|
||||
else url
|
||||
)
|
||||
except httpx.InvalidURL:
|
||||
# httpx quotes part of the URL (part of a password containing
|
||||
# "/", "?" or "#" ends up as the "port"), and chains a ValueError
|
||||
# that repeats it, so neither is kept.
|
||||
raise ValueError(
|
||||
"Invalid Loki URL (details withheld because they may include "
|
||||
"credentials)"
|
||||
) from None
|
||||
self.timestamp = timestamp
|
||||
self.timeout = timeout
|
||||
self.batch_size = max(1, int(batch_size))
|
||||
@@ -196,14 +207,16 @@ class LokiClient(object):
|
||||
self.url, content=json.dumps(body), timeout=self.timeout
|
||||
)
|
||||
except httpx.HTTPError as error_:
|
||||
raise LokiError(f"Loki push to {self.url} failed: {error_}") from error_
|
||||
raise LokiError(
|
||||
f"Loki push to {redact_url_credentials(self.url)} failed: {error_}"
|
||||
) from error_
|
||||
# Loki answers 204 when it stored the push; 200 is also accepted.
|
||||
# Other 2xx codes are not success: with block_ingestion_until set it
|
||||
# returns a custom 260 and discards the logs
|
||||
# (block_ingestion_status_code).
|
||||
if response.status_code not in (200, 204):
|
||||
raise LokiError(
|
||||
f"Loki push to {self.url} failed with HTTP "
|
||||
f"Loki push to {redact_url_credentials(self.url)} failed with HTTP "
|
||||
f"{response.status_code}: {response.text[:500]}"
|
||||
)
|
||||
|
||||
|
||||
@@ -29,7 +29,7 @@ from opensearchpy.helpers import reindex
|
||||
|
||||
from parsedmarc import InvalidFailureReport
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.utils import human_timestamp_to_datetime
|
||||
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
|
||||
|
||||
|
||||
class OpenSearchError(Exception):
|
||||
@@ -552,7 +552,11 @@ def set_hosts(
|
||||
_READY_DATA_STREAMS.clear()
|
||||
if not isinstance(hosts, list):
|
||||
hosts = [hosts]
|
||||
logger.debug("Connecting to OpenSearch: hosts=%s, use_ssl=%s", hosts, use_ssl)
|
||||
logger.debug(
|
||||
"Connecting to OpenSearch: hosts=%s, use_ssl=%s",
|
||||
redact_url_credentials(hosts),
|
||||
use_ssl,
|
||||
)
|
||||
conn_params = {"hosts": hosts, "timeout": timeout}
|
||||
if use_ssl:
|
||||
conn_params["use_ssl"] = True
|
||||
|
||||
+11
-2
@@ -20,7 +20,7 @@ else:
|
||||
psycopg_json = None
|
||||
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.utils import human_timestamp_to_datetime
|
||||
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
|
||||
|
||||
# psycopg is an optional dependency (the PostgreSQL backend is opt-in). The
|
||||
# pure helper functions below work without it; only PostgreSQLClient needs a
|
||||
@@ -183,7 +183,16 @@ class PostgreSQLClient:
|
||||
)
|
||||
conn.autocommit = False
|
||||
except psycopg.Error as exc:
|
||||
raise PostgreSQLError(str(exc)) from exc
|
||||
# libpq reports a malformed connection string by echoing the
|
||||
# whole string, or the fragment it could not parse, and either
|
||||
# can contain the password. psycopg raises ProgrammingError for
|
||||
# those parse failures, so withhold their detail entirely.
|
||||
if self._connection_string and isinstance(exc, psycopg.ProgrammingError):
|
||||
raise PostgreSQLError(
|
||||
"Invalid PostgreSQL connection_string (details withheld "
|
||||
"because they may include the password)"
|
||||
) from None
|
||||
raise PostgreSQLError(redact_url_credentials(str(exc))) from exc
|
||||
self._conn = conn
|
||||
return conn
|
||||
|
||||
|
||||
@@ -13,6 +13,13 @@ from parsedmarc.constants import USER_AGENT
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.utils import human_timestamp_to_unix_timestamp
|
||||
|
||||
# httpx.InvalidURL quotes part of the URL, which can be part of a password
|
||||
# (one containing "/", "?" or "#" ends up as the "port"), so its text is
|
||||
# replaced by this fixed message.
|
||||
_INVALID_URL_MESSAGE = (
|
||||
"Invalid Splunk HEC URL (details withheld because they may include credentials)"
|
||||
)
|
||||
|
||||
|
||||
class SplunkError(RuntimeError):
|
||||
"""Raised when a Splunk API error occurs"""
|
||||
@@ -136,6 +143,8 @@ class HECClient(object):
|
||||
self.url, content="".join(json_lines), timeout=self.timeout
|
||||
)
|
||||
response = response.json()
|
||||
except httpx.InvalidURL:
|
||||
raise SplunkError(_INVALID_URL_MESSAGE) from None
|
||||
except Exception as e:
|
||||
raise SplunkError(e.__str__())
|
||||
if response["code"] != 0:
|
||||
@@ -179,6 +188,8 @@ class HECClient(object):
|
||||
self.url, content="".join(json_lines), timeout=self.timeout
|
||||
)
|
||||
response = response.json()
|
||||
except httpx.InvalidURL:
|
||||
raise SplunkError(_INVALID_URL_MESSAGE) from None
|
||||
except Exception as e:
|
||||
raise SplunkError(e.__str__())
|
||||
if response["code"] != 0:
|
||||
@@ -218,6 +229,8 @@ class HECClient(object):
|
||||
self.url, content="".join(json_lines), timeout=self.timeout
|
||||
)
|
||||
response = response.json()
|
||||
except httpx.InvalidURL:
|
||||
raise SplunkError(_INVALID_URL_MESSAGE) from None
|
||||
except Exception as e:
|
||||
raise SplunkError(e.__str__())
|
||||
if response["code"] != 0:
|
||||
|
||||
+294
-25
@@ -17,7 +17,7 @@ import subprocess
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import TYPE_CHECKING, TypedDict, cast
|
||||
from urllib.parse import urlsplit
|
||||
from urllib.parse import SplitResult, urlsplit
|
||||
|
||||
import mailparser
|
||||
from expiringdict import ExpiringDict
|
||||
@@ -113,15 +113,17 @@ def load_psl_overrides(
|
||||
|
||||
if not (offline or always_use_local_file):
|
||||
try:
|
||||
logger.debug(f"Trying to fetch PSL overrides from {url}...")
|
||||
logger.debug(
|
||||
f"Trying to fetch PSL overrides from {redact_url_credentials(url)}..."
|
||||
)
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
response = httpx.get(
|
||||
url, headers=headers, timeout=60, follow_redirects=True
|
||||
)
|
||||
response.raise_for_status()
|
||||
_load_text(response.text)
|
||||
except httpx.HTTPError as e:
|
||||
logger.warning(f"Failed to fetch PSL overrides: {e}")
|
||||
except (httpx.HTTPError, httpx.InvalidURL) as e:
|
||||
logger.warning(f"Failed to fetch PSL overrides: {_describe_http_error(e)}")
|
||||
|
||||
if len(psl_overrides) == 0:
|
||||
path = local_file_path or str(
|
||||
@@ -309,6 +311,44 @@ class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
|
||||
)
|
||||
|
||||
|
||||
def _nameserver_port(entry: str, parts: SplitResult, kind: str) -> int | None:
|
||||
"""
|
||||
Returns the port of a parsed DNS over HTTPS or TLS nameserver URL.
|
||||
|
||||
``urlsplit()`` only validates the port when it is read, and its
|
||||
``ValueError`` quotes the text it rejected. A password containing an
|
||||
unencoded ``/``, ``?`` or ``#`` ends the authority early (RFC 3986
|
||||
section 3.2), which makes the start of the password that text
|
||||
(``https://doh:p4ss/w0rd@host`` gives ``Port could not be cast to
|
||||
integer value as 'p4ss'``). dnspython 2.9.1 reads the same port in
|
||||
``DoHNameserver.answer_port()`` and lets that error escape
|
||||
``Resolver.resolve()``. So the port is read here, and a bad one is
|
||||
reported with the entry's credentials masked and the original error
|
||||
suppressed.
|
||||
|
||||
Args:
|
||||
entry (str): The configured nameserver
|
||||
parts (SplitResult): ``urlsplit(entry)``
|
||||
kind (str): The transport name for the message, e.g.
|
||||
``"DNS over TLS"``
|
||||
|
||||
Returns:
|
||||
int | None: The port, or ``None`` when the URL has none
|
||||
|
||||
Raises:
|
||||
ValueError: The port is not an integer from 0 to 65535
|
||||
"""
|
||||
try:
|
||||
return parts.port
|
||||
except ValueError:
|
||||
pass
|
||||
raise ValueError(
|
||||
f"Invalid {kind} nameserver {_redact_one_url_credentials(entry)}: the "
|
||||
"port is not an integer from 0 to 65535 (percent-encode any /, ? or # "
|
||||
"in a password)"
|
||||
)
|
||||
|
||||
|
||||
def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
|
||||
"""
|
||||
Parses a ``tls://ip[:port][#hostname]`` nameserver entry.
|
||||
@@ -327,26 +367,25 @@ def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
|
||||
is not a literal IP address, or extra URL components
|
||||
"""
|
||||
parts = urlsplit(entry)
|
||||
try:
|
||||
port = parts.port
|
||||
except ValueError as e:
|
||||
# urlsplit only validates the port when it is accessed
|
||||
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: {e}") from e
|
||||
port = _nameserver_port(entry, parts, "DNS over TLS")
|
||||
# The entry may carry userinfo, which is rejected below, so the
|
||||
# messages quote it with its credentials masked
|
||||
shown = _redact_one_url_credentials(entry)
|
||||
if parts.username or parts.path or parts.query:
|
||||
# Catch tls://9.9.9.9/dns.quad9.net — a plausible slash-for-#
|
||||
# typo that would otherwise "work" with no certificate identity
|
||||
# and fail only at query time with an opaque TLS error
|
||||
raise ValueError(
|
||||
f"Invalid DNS over TLS nameserver {entry}: only "
|
||||
f"Invalid DNS over TLS nameserver {shown}: only "
|
||||
"tls://ip[:port][#hostname] is supported — the TLS certificate "
|
||||
"identity is given after #, not /"
|
||||
)
|
||||
address = parts.hostname
|
||||
if not address:
|
||||
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: missing IP address")
|
||||
raise ValueError(f"Invalid DNS over TLS nameserver {shown}: missing IP address")
|
||||
if not dns.inet.is_address(address):
|
||||
raise ValueError(
|
||||
f"Invalid DNS over TLS nameserver {entry}: {address} is not an IP "
|
||||
f"Invalid DNS over TLS nameserver {shown}: {address} is not an IP "
|
||||
"address. Use tls://ip[:port][#hostname], where the optional "
|
||||
"#hostname is the TLS certificate identity of the server"
|
||||
)
|
||||
@@ -378,20 +417,25 @@ def _nameservers_to_resolver_input(
|
||||
in the configured order
|
||||
|
||||
Raises:
|
||||
ValueError: A ``tls://`` entry is malformed
|
||||
ValueError: A ``tls://`` entry is malformed, or an ``https://``
|
||||
entry has an unusable port
|
||||
"""
|
||||
resolver_input: list[str | dns.nameserver.Nameserver] = []
|
||||
for entry in nameservers:
|
||||
try:
|
||||
# urlsplit lowercases the scheme, so HTTPS:// and TLS:// work too
|
||||
scheme = urlsplit(entry).scheme
|
||||
parts = urlsplit(entry)
|
||||
except ValueError:
|
||||
# e.g. an unbalanced IPv6 bracket; let dnspython reject it with
|
||||
# its own message about what a nameserver may be
|
||||
scheme = ""
|
||||
if scheme == "https":
|
||||
resolver_input.append(entry)
|
||||
continue
|
||||
if parts.scheme == "https":
|
||||
# Validated now because dnspython reads the port only after a
|
||||
# query fails, and its error then quotes the rejected text
|
||||
_nameserver_port(entry, parts, "DNS over HTTPS")
|
||||
resolver_input.append(_SessionDoHNameserver(entry))
|
||||
elif scheme == "tls":
|
||||
elif parts.scheme == "tls":
|
||||
resolver_input.append(_parse_dot_nameserver(entry))
|
||||
else:
|
||||
resolver_input.append(entry)
|
||||
@@ -439,6 +483,10 @@ def query_dns(
|
||||
|
||||
Returns:
|
||||
list: A list of answers
|
||||
|
||||
Raises:
|
||||
ValueError: A nameserver is malformed. The message masks the
|
||||
nameserver's URL credentials.
|
||||
"""
|
||||
domain = str(domain).lower()
|
||||
record_type = record_type.upper()
|
||||
@@ -457,7 +505,17 @@ def query_dns(
|
||||
"2606:4700:4700::1111",
|
||||
"2606:4700:4700::1001",
|
||||
]
|
||||
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
|
||||
rejection = None
|
||||
try:
|
||||
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
|
||||
except ValueError as e:
|
||||
# dnspython's own rejection quotes the entry as configured
|
||||
# ("nameserver https://user:pw@[x is not ..."), credentials and all
|
||||
rejection = _mask_nameserver_secrets(str(e), nameservers)
|
||||
if rejection is not None:
|
||||
# Raised outside the except block so that the unmasked error is not
|
||||
# kept as __context__
|
||||
raise ValueError(rejection)
|
||||
# Cap per-query UDP timeout at 1s so dnspython retries within the
|
||||
# lifetime window on transient packet loss — otherwise with a single
|
||||
# nameserver and timeout == lifetime, one dropped UDP datagram consumes
|
||||
@@ -532,7 +590,9 @@ def get_reverse_dns(
|
||||
)[0]
|
||||
|
||||
except dns.exception.DNSException as e:
|
||||
logger.debug(f"get_reverse_dns({ip_address}) exception: {e}")
|
||||
# LifetimeTimeout and NoNameservers quote each nameserver URL
|
||||
message = _mask_nameserver_secrets(str(e), nameservers)
|
||||
logger.debug(f"get_reverse_dns({ip_address}) exception: {message}")
|
||||
|
||||
return hostname
|
||||
|
||||
@@ -684,7 +744,9 @@ def load_ip_db(
|
||||
|
||||
if not (offline or always_use_local_file):
|
||||
try:
|
||||
logger.debug(f"Trying to fetch IP database from {url}...")
|
||||
logger.debug(
|
||||
f"Trying to fetch IP database from {redact_url_credentials(url)}..."
|
||||
)
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
response = httpx.get(
|
||||
url, headers=headers, timeout=60, follow_redirects=True
|
||||
@@ -698,8 +760,8 @@ def load_ip_db(
|
||||
_IP_DB_PATH = cached_path
|
||||
logger.info("IP database updated successfully")
|
||||
return
|
||||
except httpx.HTTPError as e:
|
||||
logger.warning(f"Failed to fetch IP database: {e}")
|
||||
except (httpx.HTTPError, httpx.InvalidURL) as e:
|
||||
logger.warning(f"Failed to fetch IP database: {_describe_http_error(e)}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to save IP database: {e}")
|
||||
|
||||
@@ -1080,7 +1142,9 @@ def load_reverse_dns_map(
|
||||
|
||||
if not (offline or always_use_local_file):
|
||||
try:
|
||||
logger.debug(f"Trying to fetch reverse DNS map from {url}...")
|
||||
logger.debug(
|
||||
f"Trying to fetch reverse DNS map from {redact_url_credentials(url)}..."
|
||||
)
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
response = httpx.get(
|
||||
url, headers=headers, timeout=60, follow_redirects=True
|
||||
@@ -1089,8 +1153,10 @@ def load_reverse_dns_map(
|
||||
csv_file.write(response.text)
|
||||
csv_file.seek(0)
|
||||
load_csv(csv_file)
|
||||
except httpx.HTTPError as e:
|
||||
logger.warning(f"Failed to fetch reverse DNS map: {e}")
|
||||
except (httpx.HTTPError, httpx.InvalidURL) as e:
|
||||
logger.warning(
|
||||
f"Failed to fetch reverse DNS map: {_describe_http_error(e)}"
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Not a valid CSV file")
|
||||
csv_file.seek(0)
|
||||
@@ -1357,6 +1423,209 @@ def sanitize_for_log(value: str) -> str:
|
||||
)
|
||||
|
||||
|
||||
def redact_url_credentials(value: str | list[str]) -> str | list[str]:
|
||||
"""Masks the credentials in a URL or host string so it is safe to log.
|
||||
|
||||
A configured URL such as ``https://user:secret@host:9200`` carries a
|
||||
password; writing it to the log (CWE-532) leaks it to the console and
|
||||
to ``log_file``. When the userinfo contains a ``:``, only the part
|
||||
after the first ``:`` is replaced (``https://user:***@host:9200``);
|
||||
when it has no ``:`` the whole userinfo is replaced
|
||||
(``https://***@host``), since a bare token is itself the secret.
|
||||
|
||||
The authority is located with string methods only, so scheme-less
|
||||
values such as ``user:secret@localhost:9200`` (which ``urlsplit()``
|
||||
would misparse as scheme ``user``) are handled too.
|
||||
|
||||
A password containing an unencoded ``/``, ``?``, or ``#`` is not
|
||||
allowed by RFC 3986 (the authority ends at that character, so clients
|
||||
misparse such a URL too), and it leaves the authority with no ``@``.
|
||||
When the authority has no ``@`` but one appears later in the value,
|
||||
everything from the start of the authority through the *last* ``@``
|
||||
is replaced (``https://admin:p4ss/w0rd@host:3100`` becomes
|
||||
``https://***@host:3100``). The last ``@`` is used because the
|
||||
password may itself contain ``@``. This over-masks a URL that has no
|
||||
userinfo but has an ``@`` in its path or query:
|
||||
``https://host/path@file`` becomes ``https://***@file``, and
|
||||
``https://host/?redirect=https://u:p@evil`` becomes
|
||||
``https://***@evil``. That is acceptable because the result is only
|
||||
for the log. When the authority does contain an ``@``, only its
|
||||
userinfo is masked, so credentials in a later query-string value
|
||||
(``https://a:b@host/?redirect=https://u:p@evil``) are not detected,
|
||||
and neither is the rest of a password containing both ``@`` and one
|
||||
of ``/?#`` (``https://user:p@ss/word@host`` keeps ``ss/word``).
|
||||
|
||||
The value must be a single URL (or host) per string: split a
|
||||
comma-separated list first, or pass a ``list``. The authority is the
|
||||
first one found, so later entries in a comma-joined string are not
|
||||
masked.
|
||||
|
||||
Never raises; input that does not contain userinfo, and input that is
|
||||
not a ``str`` or ``list``, is returned unchanged.
|
||||
|
||||
Args:
|
||||
value (str | list[str]): A URL or host string, or a list of them
|
||||
|
||||
Returns:
|
||||
str | list[str]: ``value`` with credentials masked. A ``list`` is
|
||||
returned as a new list; the input list is not modified.
|
||||
"""
|
||||
if isinstance(value, list):
|
||||
return [
|
||||
_redact_one_url_credentials(item) if isinstance(item, str) else item
|
||||
for item in value
|
||||
]
|
||||
if not isinstance(value, str):
|
||||
return value
|
||||
return _redact_one_url_credentials(value)
|
||||
|
||||
|
||||
def _find_url_userinfo(value: str) -> tuple[int, int, bool] | None:
|
||||
"""Locates the userinfo of one URL or host string.
|
||||
|
||||
This is the lookup :func:`redact_url_credentials` masks and
|
||||
:func:`_url_secrets` extracts the secret from, shared so that the two
|
||||
cannot disagree about where the credentials are. It uses string
|
||||
methods only, so it is linear in the length of ``value`` and never
|
||||
raises.
|
||||
|
||||
Args:
|
||||
value (str): A URL or host string
|
||||
|
||||
Returns:
|
||||
tuple[int, int, bool] | None: ``None`` when no ``@`` follows the
|
||||
start of the authority. Otherwise ``(start, end, in_authority)``,
|
||||
where ``value[start:end]`` is the userinfo and ``value[end]`` is the
|
||||
``@`` after it. ``in_authority`` is ``True`` when that ``@`` is the
|
||||
last one inside the authority, and ``False`` when the authority has
|
||||
no ``@`` (a ``/``, ``?`` or ``#`` in the password ends it early, as
|
||||
in ``https://admin:p4ss/w0rd@host``) and the ``@`` is the last one
|
||||
in the whole value; the password may itself contain ``@``.
|
||||
"""
|
||||
# A "://" counts as the scheme separator only when nothing before it
|
||||
# could end an authority: in "user:secret@host/x?r=http://y" the "://"
|
||||
# is inside the query, and the authority starts at offset 0.
|
||||
scheme_end = value.find("://")
|
||||
if scheme_end != -1 and not any(c in value[:scheme_end] for c in "/?#@"):
|
||||
start = scheme_end + 3
|
||||
elif value.startswith("//"):
|
||||
start = 2
|
||||
else:
|
||||
start = 0
|
||||
end = len(value)
|
||||
for delimiter in "/?#":
|
||||
position = value.find(delimiter, start)
|
||||
if position != -1:
|
||||
end = min(end, position)
|
||||
at = value.rfind("@", start, end)
|
||||
if at != -1:
|
||||
return start, at, True
|
||||
at = value.rfind("@", start)
|
||||
if at != -1:
|
||||
return start, at, False
|
||||
return None
|
||||
|
||||
|
||||
def _redact_one_url_credentials(value: str) -> str:
|
||||
"""Masks the userinfo of one URL or host string; see
|
||||
:func:`redact_url_credentials`."""
|
||||
found = _find_url_userinfo(value)
|
||||
if found is None:
|
||||
return value
|
||||
start, at, in_authority = found
|
||||
if not in_authority:
|
||||
# Which "/", "?" or "#" ended the authority is unknown, so mask
|
||||
# everything through the "@" rather than guess where the user
|
||||
# name stops.
|
||||
return f"{value[:start]}***{value[at:]}"
|
||||
user, colon, _ = value[start:at].partition(":")
|
||||
masked = f"{user}:***" if colon else "***"
|
||||
return f"{value[:start]}{masked}{value[at:]}"
|
||||
|
||||
|
||||
def _url_secrets(value: str) -> list[str]:
|
||||
"""Returns the secret text in one URL or host string's userinfo.
|
||||
|
||||
The secret is the userinfo after its first ``:``, or the whole userinfo
|
||||
when it has no ``:`` (a bare token is itself the secret), located by
|
||||
:func:`_find_url_userinfo`. When the secret contains an unencoded
|
||||
``/``, ``?`` or ``#``, its text before the first of them is returned
|
||||
too: ``urlsplit()`` (and so dnspython) and httpx read that part as the
|
||||
port and quote it in their errors (``Port could not be cast to integer
|
||||
value as 'p4ss'``). Empty strings are never returned, so the result is
|
||||
safe to use as ``str.replace()`` targets. Never raises.
|
||||
|
||||
Args:
|
||||
value (str): A URL or host string
|
||||
|
||||
Returns:
|
||||
list[str]: The secret and, if it differs and is not empty, its
|
||||
prefix before the first ``/``, ``?`` or ``#``, longest first; an
|
||||
empty list when ``value`` has no userinfo or an empty secret.
|
||||
"""
|
||||
found = _find_url_userinfo(value)
|
||||
if found is None:
|
||||
return []
|
||||
start, at, _ = found
|
||||
user, colon, password = value[start:at].partition(":")
|
||||
secret = password if colon else user
|
||||
prefix_end = len(secret)
|
||||
for delimiter in "/?#":
|
||||
position = secret.find(delimiter)
|
||||
if position != -1:
|
||||
prefix_end = min(prefix_end, position)
|
||||
return [text for text in dict.fromkeys((secret, secret[:prefix_end])) if text]
|
||||
|
||||
|
||||
def _mask_nameserver_secrets(text: str, nameservers: list[str] | None) -> str:
|
||||
"""Masks the credentials of the configured nameserver URLs in free text.
|
||||
|
||||
A DNS query error (dnspython's ``LifetimeTimeout``, a DNS over HTTPS
|
||||
status error) quotes each failing nameserver URL, sometimes several
|
||||
times in different forms, and ``redact_url_credentials()`` masks only
|
||||
one URL per string. So each configured entry is replaced by its masked
|
||||
form, and each secret :func:`_url_secrets` finds in it (the password,
|
||||
and its start when a ``/``, ``?`` or ``#`` in it gets read as the port)
|
||||
is replaced by ``***`` wherever it still appears. A short secret masks
|
||||
every occurrence of that text, including ones that are not the
|
||||
password; the result is only for error messages and logs. Never
|
||||
raises.
|
||||
|
||||
Args:
|
||||
text (str): The text to mask, typically ``str()`` of an exception
|
||||
nameservers (list[str] | None): The configured nameservers
|
||||
|
||||
Returns:
|
||||
str: ``text`` with the nameservers' credentials masked
|
||||
"""
|
||||
for entry in nameservers or []:
|
||||
text = text.replace(entry, _redact_one_url_credentials(entry))
|
||||
for secret in _url_secrets(entry):
|
||||
text = text.replace(secret, "***")
|
||||
return text
|
||||
|
||||
|
||||
def _describe_http_error(e: httpx.HTTPError | httpx.InvalidURL) -> str:
|
||||
"""Describes an httpx error for the log without leaking URL credentials.
|
||||
|
||||
``str(HTTPStatusError)`` embeds the server's reason phrase before the
|
||||
URL, so a phrase containing ``/``, ``?``, ``#`` or ``@`` defeats
|
||||
string-based redaction of the message. A status error is therefore
|
||||
rebuilt from its structured parts. Transport errors (httpx 0.28) do not
|
||||
embed the request URL, so their message is used as is. ``InvalidURL``
|
||||
(not an ``HTTPError``) is raised while httpx parses the URL, and its
|
||||
message quotes the component it rejected -- part of a password
|
||||
containing ``/``, ``?`` or ``#`` ends up as the "port" -- so it is
|
||||
replaced by a fixed message.
|
||||
"""
|
||||
if isinstance(e, httpx.InvalidURL):
|
||||
return "invalid URL (details withheld because they may include credentials)"
|
||||
if isinstance(e, httpx.HTTPStatusError):
|
||||
url = redact_url_credentials(str(e.request.url))
|
||||
return f"HTTP {e.response.status_code} for url '{url}'"
|
||||
return f"{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
def get_filename_safe_string(string: str | None) -> str:
|
||||
"""
|
||||
Converts a string to a string that is safe to use as a filename
|
||||
|
||||
@@ -62,6 +62,12 @@ class WebhookClient(object):
|
||||
self.session.post(webhook_url, data=payload, timeout=self.timeout)
|
||||
else:
|
||||
self.session.post(webhook_url, content=payload, timeout=self.timeout)
|
||||
except httpx.InvalidURL:
|
||||
# httpx quotes part of the URL, which can be part of a password
|
||||
logger.error(
|
||||
"Webhook Error: invalid webhook URL (details withheld because "
|
||||
"they may include credentials)"
|
||||
)
|
||||
except Exception as error_:
|
||||
logger.error(f"Webhook Error: {error_.__str__()}")
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import importlib.abc
|
||||
import importlib.machinery
|
||||
import gzip
|
||||
import io
|
||||
import importlib.util
|
||||
import json
|
||||
import logging
|
||||
import mailbox
|
||||
@@ -15,6 +16,7 @@ import signal
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import traceback
|
||||
import unittest
|
||||
import zipfile
|
||||
from collections.abc import Iterator, Sequence
|
||||
@@ -5452,6 +5454,323 @@ def _es_request_without_network(client, method, path, **kwargs):
|
||||
return {"count": 0}
|
||||
|
||||
|
||||
def _os_request_without_network(transport, method, url, *args, **kwargs):
|
||||
"""Stand-in for ``opensearchpy.Transport.perform_request``: answers
|
||||
"no such index" / zero documents like ``_es_request_without_network``."""
|
||||
if method == "HEAD":
|
||||
return {}
|
||||
return {"count": 0}
|
||||
|
||||
|
||||
class TestInitOutputClientsDoesNotLogCredentials(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: with --debug, the "Initializing ... client"
|
||||
lines must mask credentials embedded in configured URLs."""
|
||||
|
||||
SECRET = "s3cretPW"
|
||||
|
||||
def _init(self, opts):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
clients = parsedmarc.cli._init_output_clients(opts)
|
||||
self.addCleanup(parsedmarc.cli._close_output_clients, clients)
|
||||
return "\n".join(logs.output)
|
||||
|
||||
def test_hec_url_password_is_masked(self):
|
||||
opts = _output_client_opts(
|
||||
hec=f"https://hecuser:{self.SECRET}@hec.example:8088",
|
||||
hec_token="tok",
|
||||
hec_index="email",
|
||||
)
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
self.assertIn("url=https://hecuser:***@hec.example:8088", output)
|
||||
|
||||
def test_hec_malformed_url_password_is_masked(self):
|
||||
"""A "/", "?" or "#" in the password ends the authority before the
|
||||
"@", which left the --debug line unmasked; it is now masked through
|
||||
the last "@"."""
|
||||
for password in ("p4ssw0rd/extra", "p4ss?w0rd", "p4ss#w0rd"):
|
||||
with self.subTest(password=password):
|
||||
opts = _output_client_opts(
|
||||
hec=f"https://hecuser:{password}@hec.example:8088",
|
||||
hec_token="tok",
|
||||
hec_index="email",
|
||||
)
|
||||
output = self._init(opts)
|
||||
self.assertNotIn("p4ss", output)
|
||||
self.assertIn("url=https://***@hec.example:8088", output)
|
||||
|
||||
def test_loki_url_password_is_masked(self):
|
||||
opts = _output_client_opts(
|
||||
loki_url=f"https://lokiuser:{self.SECRET}@loki.example:3100",
|
||||
loki_timestamp="ingest",
|
||||
loki_timeout=1.0,
|
||||
loki_batch_size=100,
|
||||
)
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
self.assertIn("url=https://lokiuser:***@loki.example:3100", output)
|
||||
|
||||
def test_loki_malformed_url_error_has_no_password(self):
|
||||
"""A password with an unencoded "/" ends the URL's authority early
|
||||
(RFC 3986 section 3.2), so httpx raises ``InvalidURL("Invalid port:
|
||||
'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py), whose text and chained
|
||||
ValueError both repeat it. Neither the startup error nor the
|
||||
``--debug`` "Initializing Loki client" line may show any part of
|
||||
the password."""
|
||||
malformed = (
|
||||
("https://admin:p4ssw0rd/extra@loki.example:3100", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@loki.example:3100", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@loki.example:3100", "p4ss"),
|
||||
)
|
||||
for url, secret in malformed:
|
||||
with self.subTest(url=url):
|
||||
opts = _output_client_opts(
|
||||
loki_url=url,
|
||||
loki_timestamp="ingest",
|
||||
loki_timeout=1.0,
|
||||
loki_batch_size=100,
|
||||
)
|
||||
with (
|
||||
self.assertRaises(RuntimeError) as ctx,
|
||||
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
|
||||
):
|
||||
parsedmarc.cli._init_output_clients(opts)
|
||||
self.assertEqual(
|
||||
str(ctx.exception),
|
||||
"Loki: Invalid Loki URL (details withheld because they may "
|
||||
"include credentials)",
|
||||
)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn(secret, trace)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn(secret, output)
|
||||
self.assertIn("url=https://***@loki.example:3100", output)
|
||||
|
||||
def test_elasticsearch_hosts_password_is_masked(self):
|
||||
connections = parsedmarc.elastic.connections
|
||||
_isolate_default_alias(self, connections)
|
||||
opts = _output_client_opts(
|
||||
save_aggregate=True,
|
||||
elasticsearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
|
||||
elasticsearch_timeout=1.0,
|
||||
)
|
||||
with (
|
||||
patch.object(Elasticsearch, "perform_request", _es_request_without_network),
|
||||
patch(
|
||||
"parsedmarc.elastic.connections.create_connection",
|
||||
wraps=connections.create_connection,
|
||||
) as mock_conn,
|
||||
):
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
self.assertIn("Initializing Elasticsearch client", output)
|
||||
# Redaction is log-only: the client still gets the real credentials
|
||||
self.assertEqual(
|
||||
mock_conn.call_args.kwargs["hosts"],
|
||||
[f"https://u:{self.SECRET}@127.0.0.1:9299"],
|
||||
)
|
||||
self.assertIn("https://u:***@127.0.0.1:9299", output)
|
||||
|
||||
def test_opensearch_hosts_password_is_masked_in_cli_and_set_hosts(self):
|
||||
connections = opensearch_module.connections
|
||||
_isolate_default_alias(self, connections)
|
||||
opts = _output_client_opts(
|
||||
save_aggregate=True,
|
||||
opensearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
|
||||
opensearch_timeout=1.0,
|
||||
)
|
||||
with patch.object(
|
||||
opensearchpy.Transport, "perform_request", _os_request_without_network
|
||||
):
|
||||
output = self._init(opts)
|
||||
self.assertNotIn(self.SECRET, output)
|
||||
lines = output.splitlines()
|
||||
for marker in ("Initializing OpenSearch client", "Connecting to OpenSearch"):
|
||||
with self.subTest(marker=marker):
|
||||
matching = [line for line in lines if marker in line]
|
||||
self.assertEqual(len(matching), 1)
|
||||
self.assertIn("https://u:***@127.0.0.1:9299", matching[0])
|
||||
|
||||
|
||||
class TestHttpxLogRedaction(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: httpx logs the full request URL at INFO."""
|
||||
|
||||
def setUp(self):
|
||||
saved = {}
|
||||
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
|
||||
dep = logging.getLogger(name)
|
||||
saved[name] = (
|
||||
dep.level,
|
||||
list(dep.handlers),
|
||||
dep.propagate,
|
||||
list(dep.filters),
|
||||
)
|
||||
|
||||
def restore():
|
||||
for name, (level, handlers, propagate, filters) in saved.items():
|
||||
dep = logging.getLogger(name)
|
||||
dep.setLevel(level)
|
||||
dep.handlers = handlers
|
||||
dep.propagate = propagate
|
||||
dep.filters = filters
|
||||
|
||||
self.addCleanup(restore)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
|
||||
def _client(self, seen):
|
||||
def handler(request):
|
||||
seen.append(request)
|
||||
return httpx.Response(200)
|
||||
|
||||
return httpx.Client(transport=httpx.MockTransport(handler))
|
||||
|
||||
def test_userinfo_password_masked_but_auth_still_sent(self):
|
||||
seen = []
|
||||
with self._client(seen) as client:
|
||||
with self.assertLogs("httpx", level="INFO") as logs:
|
||||
client.post("https://user:secret@hook.example/x", content=b"{}")
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("secret", output)
|
||||
self.assertIn("https://user:***@hook.example/x", output)
|
||||
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
|
||||
|
||||
def test_ipinfo_token_query_parameter_masked(self):
|
||||
seen = []
|
||||
with self._client(seen) as client:
|
||||
with self.assertLogs("httpx", level="INFO") as logs:
|
||||
client.get(
|
||||
"https://api.ipinfo.io/lite/8.8.8.8", params={"token": "TOKSECRET"}
|
||||
)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("TOKSECRET", output)
|
||||
self.assertIn("api.ipinfo.io/lite/8.8.8.8", output)
|
||||
self.assertIn("token=***", output)
|
||||
self.assertEqual(seen[0].url.params["token"], "TOKSECRET")
|
||||
|
||||
def test_token_mask_leaves_literal_encoded_stars_in_path_alone(self):
|
||||
"""A path containing a literal ``%2A%2A%2A`` is not rewritten; only
|
||||
the token value is masked, with and without other query params and
|
||||
a fragment."""
|
||||
for url, expected in (
|
||||
(
|
||||
"https://h.example/a%2A%2A%2Ab?token=TOKSECRET",
|
||||
"https://h.example/a%2A%2A%2Ab?token=***",
|
||||
),
|
||||
(
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET",
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=***",
|
||||
),
|
||||
(
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET#f",
|
||||
"https://h.example/a%2A%2A%2Ab?x=1&token=***#f",
|
||||
),
|
||||
):
|
||||
with self.subTest(url=url):
|
||||
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(
|
||||
httpx.URL(url)
|
||||
)
|
||||
self.assertNotIn("TOKSECRET", result)
|
||||
self.assertEqual(result, expected)
|
||||
|
||||
def test_reconfiguring_does_not_stack_filters(self):
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
|
||||
installed = [
|
||||
f
|
||||
for f in logging.getLogger("httpx").filters
|
||||
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
|
||||
]
|
||||
self.assertEqual(len(installed), 1)
|
||||
|
||||
def test_non_tuple_args_and_other_arg_types_pass_through(self):
|
||||
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "x %d", (5,), None)
|
||||
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
|
||||
self.assertEqual(record.args, (5,))
|
||||
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "plain", None, None)
|
||||
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
|
||||
self.assertIsNone(record.args)
|
||||
|
||||
|
||||
@unittest.skipUnless(
|
||||
importlib.util.find_spec("httpx2") is not None,
|
||||
"httpx2 (dnspython 2.9+ DNS over HTTPS) is not installed",
|
||||
)
|
||||
class TestHttpx2LogRedaction(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: dnspython 2.9+ does DNS over HTTPS through
|
||||
httpx2, which logs the request URL under the ``httpx2`` logger with an
|
||||
``httpx2.URL``, a different class from ``httpx.URL``."""
|
||||
|
||||
def setUp(self):
|
||||
import httpx2 # pyright: ignore[reportMissingImports]
|
||||
|
||||
self.httpx2 = httpx2
|
||||
saved = {}
|
||||
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
|
||||
dep = logging.getLogger(name)
|
||||
saved[name] = (
|
||||
dep.level,
|
||||
list(dep.handlers),
|
||||
dep.propagate,
|
||||
list(dep.filters),
|
||||
)
|
||||
|
||||
def restore():
|
||||
for name, (level, handlers, propagate, filters) in saved.items():
|
||||
dep = logging.getLogger(name)
|
||||
dep.setLevel(level)
|
||||
dep.handlers = handlers
|
||||
dep.propagate = propagate
|
||||
dep.filters = filters
|
||||
|
||||
self.addCleanup(restore)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
|
||||
def test_httpx2_client_request_line_masks_userinfo(self):
|
||||
"""httpx2 itself emits the line; the credential still reaches the
|
||||
transport."""
|
||||
seen = []
|
||||
|
||||
def handler(request):
|
||||
seen.append(request)
|
||||
return self.httpx2.Response(200)
|
||||
|
||||
with self.httpx2.Client(transport=self.httpx2.MockTransport(handler)) as client:
|
||||
with self.assertLogs("httpx2", level="INFO") as logs:
|
||||
client.post("https://doh:S3CRET@doh.example/dns-query", content=b"{}")
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("S3CRET", output)
|
||||
self.assertIn("https://doh:***@doh.example/dns-query", output)
|
||||
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
|
||||
|
||||
def test_httpx2_url_token_query_parameter_masked(self):
|
||||
url = self.httpx2.URL("https://h.example/a?x=1&token=TOKSECRET#f")
|
||||
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(url)
|
||||
self.assertEqual(result, "https://h.example/a?x=1&token=***#f")
|
||||
|
||||
def test_httpx2_logger_record_args_masked_and_ints_untouched(self):
|
||||
record = logging.LogRecord(
|
||||
"httpx2",
|
||||
logging.INFO,
|
||||
"f",
|
||||
1,
|
||||
"%s %d",
|
||||
(self.httpx2.URL("https://u:S3CRET@h.example/x"), 200),
|
||||
None,
|
||||
)
|
||||
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
|
||||
self.assertEqual(record.args, ("https://u:***@h.example/x", 200))
|
||||
|
||||
def test_reconfiguring_attaches_one_filter_to_httpx2(self):
|
||||
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
|
||||
parsedmarc.cli._configure_dependency_logging(logging.INFO)
|
||||
installed = [
|
||||
f
|
||||
for f in logging.getLogger("httpx2").filters
|
||||
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
|
||||
]
|
||||
self.assertEqual(len(installed), 1)
|
||||
|
||||
|
||||
class TestInitOutputClientsRollback(unittest.TestCase):
|
||||
"""_init_output_clients() is all-or-nothing against the search backends'
|
||||
process-wide ``default`` connection alias.
|
||||
@@ -7636,6 +7955,100 @@ class TestParseConfigGeneral(unittest.TestCase):
|
||||
str(ctx.exception),
|
||||
)
|
||||
|
||||
def test_general_nameservers_pre_flight_errors_mask_url_credentials(self):
|
||||
"""GHSA-hjr3-rx4c-24cv: the pre-flight error is logged at the default
|
||||
level, so neither the "no PTR record ... from <nameservers>" list nor
|
||||
a transport error that quotes the DoH request URL may carry the
|
||||
nameserver URL's password, and the chained exception (printed on a
|
||||
crash) must not repeat it."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
url = "https://doh:S3CRETPW@dns.example/dns-query"
|
||||
|
||||
def nxdomain(request, *args, **kwargs):
|
||||
response = dns.message.make_response(request)
|
||||
response.set_rcode(dns.rcode.NXDOMAIN)
|
||||
return response
|
||||
|
||||
def transport_error(request, *args, **kwargs):
|
||||
raise RuntimeError(f"Server error '502' for url '{url}'")
|
||||
|
||||
for name, responder, expected in (
|
||||
("NXDOMAIN", nxdomain, "https://doh:***@dns.example/dns-query"),
|
||||
("transport", transport_error, "https://doh:***@dns.example/dns-query"),
|
||||
):
|
||||
with self.subTest(name):
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": "1.1.1.1",
|
||||
"dns_timeout": "2.0",
|
||||
"nameservers": url,
|
||||
},
|
||||
)
|
||||
with patch("dns.query.https", side_effect=responder):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn("S3CRETPW", str(ctx.exception))
|
||||
self.assertNotIn("S3CRETPW", trace)
|
||||
self.assertIn(expected, str(ctx.exception))
|
||||
|
||||
def test_general_nameservers_pre_flight_masks_malformed_url_passwords(self):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password containing an unencoded ``/``,
|
||||
``?`` or ``#`` ends the URL authority early (RFC 3986 section 3.2),
|
||||
so ``urlsplit()`` reports no password and reads its start as the
|
||||
port. dnspython 2.9.1 then raised ``ValueError("Port could not be
|
||||
cast to integer value as 'p4ss'")`` from
|
||||
``DoHNameserver.answer_port()`` (``dns/nameserver.py``), and
|
||||
``_parse_dot_nameserver()`` quoted the same urlsplit message for a
|
||||
``tls://`` entry, so the pre-flight error printed the start of the
|
||||
password. Neither the password nor that prefix may reach the
|
||||
message or the printed traceback. No query is sent: every entry is
|
||||
rejected before dnspython contacts a server, which the patched
|
||||
transports would turn into a different error."""
|
||||
from parsedmarc.cli import ConfigurationError, _parse_config
|
||||
|
||||
for url, expected in (
|
||||
(
|
||||
"https://doh:p4ss/w0rd@dns.example/dns-query",
|
||||
"https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"https://doh:p4ss?w0rd@dns.example/dns-query",
|
||||
"https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"https://doh:p4ss#w0rd@dns.example/dns-query",
|
||||
"https://***@dns.example/dns-query",
|
||||
),
|
||||
("tls://dot:p4ss/w0rd@1.1.1.1", "tls://***@1.1.1.1"),
|
||||
):
|
||||
with self.subTest(url=url):
|
||||
cp = _config_with(
|
||||
"general",
|
||||
{
|
||||
"dns_test_address": "1.1.1.1",
|
||||
"dns_timeout": "0.2",
|
||||
"nameservers": url,
|
||||
},
|
||||
)
|
||||
unreachable = OSError("no query may be sent")
|
||||
with (
|
||||
patch("dns.query.https", side_effect=unreachable),
|
||||
patch("dns.query.tls", side_effect=unreachable),
|
||||
):
|
||||
with self.assertRaises(ConfigurationError) as ctx:
|
||||
_parse_config(cp, _opts())
|
||||
message = str(ctx.exception)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
for secret in ("p4ss", "w0rd"):
|
||||
self.assertNotIn(secret, message)
|
||||
self.assertNotIn(secret, trace)
|
||||
self.assertIn(expected, message)
|
||||
self.assertIn("DNS pre-flight check failed: ValueError: ", message)
|
||||
self.assertNotIn("no query may be sent", message)
|
||||
|
||||
def test_general_nameservers_pre_flight_rejects_invalid_test_address(self):
|
||||
"""An unparseable dns_test_address is reported by setting name and
|
||||
value (dns.reversename.from_address raises dns.exception.SyntaxError,
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import traceback
|
||||
import unittest
|
||||
from typing import cast
|
||||
from unittest.mock import MagicMock, patch
|
||||
@@ -164,6 +165,44 @@ class TestLokiClientInit(unittest.TestCase):
|
||||
self.assertIsInstance(client.session.auth, httpx.BasicAuth)
|
||||
|
||||
|
||||
class TestLokiInvalidUrlDoesNotLeakPassword(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
|
||||
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
|
||||
text after the colon as the port and raises
|
||||
``InvalidURL("Invalid port: 'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py:
|
||||
``raise InvalidURL(f"Invalid port: {port!r}")``). Its implicit
|
||||
``__context__`` is a ValueError that repeats that part of the password
|
||||
too."""
|
||||
|
||||
MALFORMED = (
|
||||
("https://admin:p4ssw0rd/extra@loki.example.org:3100", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@loki.example.org:3100", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@loki.example.org:3100", "p4ss"),
|
||||
)
|
||||
|
||||
def test_invalid_url_raises_fixed_message_without_password(self):
|
||||
for url, secret in self.MALFORMED:
|
||||
with self.subTest(url=url):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
LokiClient(url)
|
||||
self.assertEqual(
|
||||
str(ctx.exception),
|
||||
"Invalid Loki URL (details withheld because they may "
|
||||
"include credentials)",
|
||||
)
|
||||
self.assertIsNone(ctx.exception.__cause__)
|
||||
self.assertTrue(ctx.exception.__suppress_context__)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn(secret, trace)
|
||||
|
||||
def test_valid_url_keeps_userinfo_for_requests(self):
|
||||
client = LokiClient("https://admin:p4ssw0rd@loki.example.org:3100")
|
||||
self.assertEqual(
|
||||
client.url,
|
||||
"https://admin:p4ssw0rd@loki.example.org:3100/loki/api/v1/push",
|
||||
)
|
||||
|
||||
|
||||
@patch(
|
||||
"parsedmarc.loki.parsed_aggregate_reports_to_csv_rows", return_value=AGGREGATE_ROWS
|
||||
)
|
||||
@@ -370,6 +409,28 @@ class TestLokiErrors(unittest.TestCase):
|
||||
client.save_aggregate_report_to_loki(AGGREGATE)
|
||||
self.assertIn("refused", str(ctx.exception))
|
||||
|
||||
def test_errors_mask_url_password(self, _rows):
|
||||
"""GHSA-hjr3-rx4c-24cv: the URL keeps its userinfo, and the CLI
|
||||
logs LokiError text, so the password must be masked in it."""
|
||||
secret = "s3cretPW"
|
||||
client = LokiClient(f"http://u:{secret}@loki.example.org:3100")
|
||||
client.session = MagicMock()
|
||||
_session(client).post.return_value = _ok_response(500)
|
||||
with self.assertRaises(LokiError) as status_ctx:
|
||||
client.save_aggregate_report_to_loki(AGGREGATE)
|
||||
_session(client).post.side_effect = httpx.ConnectError("refused")
|
||||
with self.assertRaises(LokiError) as network_ctx:
|
||||
client.save_aggregate_report_to_loki(AGGREGATE)
|
||||
for ctx in (status_ctx, network_ctx):
|
||||
with self.subTest(error=str(ctx.exception)):
|
||||
self.assertNotIn(secret, str(ctx.exception))
|
||||
self.assertIn("http://u:***@loki.example.org:3100", str(ctx.exception))
|
||||
# Redaction is message-only: the request still goes to the real URL
|
||||
self.assertEqual(
|
||||
_session(client).post.call_args.args[0],
|
||||
f"http://u:{secret}@loki.example.org:3100/loki/api/v1/push",
|
||||
)
|
||||
|
||||
def test_blocked_ingestion_status_raises(self, _rows):
|
||||
"""Loki returns 260 (block_ingestion_status_code default) and drops
|
||||
the logs when ingestion is blocked; that must not read as success."""
|
||||
|
||||
@@ -241,6 +241,30 @@ class TestSetHosts(unittest.TestCase):
|
||||
kwargs = mock_conn.call_args.kwargs
|
||||
self.assertEqual(kwargs["hosts"], ["es1:9200", "es2:9200"])
|
||||
|
||||
def test_debug_log_masks_credentials_but_connects_with_them(self):
|
||||
"""GHSA-hjr3-rx4c-24cv: the "Connecting to OpenSearch" debug line
|
||||
must not carry the URL password, while the client still gets the
|
||||
real URL."""
|
||||
url = "https://admin:s3cretPW@es.example:9200"
|
||||
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
set_hosts(url)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("s3cretPW", output)
|
||||
self.assertIn("https://admin:***@es.example:9200", output)
|
||||
self.assertEqual(mock_conn.call_args.kwargs["hosts"], [url])
|
||||
|
||||
def test_debug_log_masks_schemeless_host_list(self):
|
||||
hosts = ["admin:s3cretPW@es1:9200", "es2:9200"]
|
||||
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
set_hosts(hosts)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("s3cretPW", output)
|
||||
self.assertIn("admin:***@es1:9200", output)
|
||||
self.assertIn("es2:9200", output)
|
||||
self.assertEqual(mock_conn.call_args.kwargs["hosts"], hosts)
|
||||
|
||||
def test_timeout_default_60s(self):
|
||||
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
|
||||
set_hosts("es:9200")
|
||||
|
||||
@@ -311,11 +311,74 @@ class TestPostgreSQLConstruction(unittest.TestCase):
|
||||
"""A driver-level connection error is wrapped in PostgreSQLError."""
|
||||
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
|
||||
mock_psycopg.Error = Exception
|
||||
mock_psycopg.ProgrammingError = type("ProgrammingError", (Exception,), {})
|
||||
mock_psycopg.connect.side_effect = mock_psycopg.Error("refused")
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(host="localhost")
|
||||
self.assertIn("refused", str(ctx.exception))
|
||||
|
||||
def test_connect_failure_detail_is_url_redacted(self):
|
||||
"""A non-parse driver error keeps its detail, with URL credentials
|
||||
masked."""
|
||||
programming_error = type("ProgrammingError", (Exception,), {})
|
||||
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
|
||||
mock_psycopg.Error = Exception
|
||||
mock_psycopg.ProgrammingError = programming_error
|
||||
mock_psycopg.connect.side_effect = mock_psycopg.Error(
|
||||
'connection to "postgresql://u:PGSECRET@db.example/x" failed'
|
||||
)
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(connection_string="postgresql://u:PGSECRET@h/db")
|
||||
message = str(ctx.exception)
|
||||
self.assertNotIn("PGSECRET", message)
|
||||
self.assertIn("u:***@db.example/x", message)
|
||||
|
||||
def test_malformed_connection_string_detail_withheld(self):
|
||||
"""A ProgrammingError from parsing the connection string (which libpq
|
||||
words with the whole string or a password fragment) is replaced with
|
||||
a fixed message that still identifies the problem."""
|
||||
programming_error = type("ProgrammingError", (Exception,), {})
|
||||
for dsn, driver_message in (
|
||||
(
|
||||
"postgresql://u:PGSECRET@[::1/db",
|
||||
'end of string reached when looking for matching "]" in IPv6 '
|
||||
'host address in URI: "postgresql://u:PGSECRET@[::1/db"',
|
||||
),
|
||||
(
|
||||
"postgresql://u:PG%ZZSECRET@h/db",
|
||||
'invalid percent-encoded token: "PG%ZZSECRET"',
|
||||
),
|
||||
):
|
||||
with self.subTest(dsn=dsn):
|
||||
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
|
||||
mock_psycopg.Error = Exception
|
||||
mock_psycopg.ProgrammingError = programming_error
|
||||
mock_psycopg.connect.side_effect = programming_error(driver_message)
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(connection_string=dsn)
|
||||
message = str(ctx.exception)
|
||||
self.assertNotIn("PGSECRET", message)
|
||||
self.assertNotIn("SECRET", message)
|
||||
self.assertIn("Invalid PostgreSQL connection_string", message)
|
||||
|
||||
@unittest.skipIf(
|
||||
importlib.util.find_spec("psycopg") is None, "psycopg is not installed"
|
||||
)
|
||||
def test_real_psycopg_malformed_connection_string_withheld(self):
|
||||
"""With the real driver, both malformed strings fail in parsing (no
|
||||
network I/O) and the raised message carries no part of the password."""
|
||||
for dsn in (
|
||||
"postgresql://u:PGSECRET@[::1/db",
|
||||
"postgresql://u:PG%ZZSECRET@h/db",
|
||||
):
|
||||
with self.subTest(dsn=dsn):
|
||||
with self.assertRaises(PostgreSQLError) as ctx:
|
||||
PostgreSQLClient(connection_string=dsn)
|
||||
self.assertNotIn("SECRET", str(ctx.exception))
|
||||
self.assertIn(
|
||||
"Invalid PostgreSQL connection_string", str(ctx.exception)
|
||||
)
|
||||
|
||||
def test_create_tables_executes_all_ddl(self):
|
||||
"""create_tables issues CREATE TABLE for every table and the indexes."""
|
||||
client, mock_conn = _make_client()
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import json
|
||||
import time
|
||||
import traceback
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@@ -469,6 +470,58 @@ class TestSaveSmtpTlsReportsToSplunk(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestInvalidUrlDoesNotLeakPassword(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
|
||||
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
|
||||
text after the colon as the port and raises
|
||||
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
|
||||
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
|
||||
any network I/O. The error reaches the log on every batch."""
|
||||
|
||||
MALFORMED = (
|
||||
("https://admin:p4ssw0rd/extra@splunk.example.com:8088", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@splunk.example.com:8088", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@splunk.example.com:8088", "p4ss"),
|
||||
)
|
||||
|
||||
def test_invalid_url_raises_fixed_message_without_password(self):
|
||||
saves = (
|
||||
("aggregate", "save_aggregate_reports_to_splunk", _aggregate_report),
|
||||
("failure", "save_failure_reports_to_splunk", _failure_report),
|
||||
("smtp_tls", "save_smtp_tls_reports_to_splunk", _smtp_tls_report),
|
||||
)
|
||||
for url, secret in self.MALFORMED:
|
||||
client = HECClient(url=url, access_token="tok", index="dmarc")
|
||||
for name, method, make_report in saves:
|
||||
with self.subTest(url=url, report=name):
|
||||
with self.assertRaises(SplunkError) as ctx:
|
||||
getattr(client, method)(make_report())
|
||||
self.assertEqual(
|
||||
str(ctx.exception),
|
||||
"Invalid Splunk HEC URL (details withheld because they "
|
||||
"may include credentials)",
|
||||
)
|
||||
self.assertIsNone(ctx.exception.__cause__)
|
||||
self.assertTrue(ctx.exception.__suppress_context__)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertNotIn(secret, trace)
|
||||
|
||||
def test_valid_url_is_the_one_posted(self):
|
||||
client = HECClient(
|
||||
url="https://admin:p4ssw0rd@splunk.example.com:8088",
|
||||
access_token="tok",
|
||||
index="dmarc",
|
||||
)
|
||||
client.session = MagicMock()
|
||||
client.session.post.return_value = _ok_response()
|
||||
client.save_aggregate_reports_to_splunk(_aggregate_report())
|
||||
self.assertEqual(
|
||||
client.session.post.call_args.args[0],
|
||||
"https://admin:p4ssw0rd@splunk.example.com:8088"
|
||||
"/services/collector/event/1.0",
|
||||
)
|
||||
|
||||
|
||||
class TestHECClientClose(unittest.TestCase):
|
||||
def test_close_closes_session(self):
|
||||
client = _client()
|
||||
|
||||
+464
-6
@@ -4,6 +4,7 @@ import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import traceback
|
||||
import unittest
|
||||
from datetime import datetime, timezone
|
||||
from importlib.resources import files
|
||||
@@ -516,6 +517,122 @@ class Test(unittest.TestCase):
|
||||
self.assertTrue(parsedmarc.utils.is_outlook_msg(magic))
|
||||
|
||||
|
||||
class TestRedactUrlCredentials(unittest.TestCase):
|
||||
"""Tests for parsedmarc.utils.redact_url_credentials()"""
|
||||
|
||||
def testMasksUserinfoAndKeepsHost(self):
|
||||
cases = [
|
||||
("https://user:secret@host", "https://user:***@host"),
|
||||
("https://user:secret@host:9200/", "https://user:***@host:9200/"),
|
||||
("localhost:9200", "localhost:9200"),
|
||||
("127.0.0.1:9299", "127.0.0.1:9299"),
|
||||
("user:secret@localhost:9200", "user:***@localhost:9200"),
|
||||
# A "://" in the path or query of a scheme-less value is not
|
||||
# the scheme separator; the userinfo at offset 0 is still masked
|
||||
(
|
||||
"user:secret@host:9200/x?r=http://y",
|
||||
"user:***@host:9200/x?r=http://y",
|
||||
),
|
||||
("user:secret@host?r=a://b", "user:***@host?r=a://b"),
|
||||
("https://user:p@ss@host/x", "https://user:***@host/x"),
|
||||
("https://user:p:w@[::1]:9200/", "https://user:***@[::1]:9200/"),
|
||||
("https://[::1]:9200", "https://[::1]:9200"),
|
||||
("//user:pw@host", "//user:***@host"),
|
||||
# Documented over-masking: with no "@" in the authority, an "@"
|
||||
# later in the value is masked through, since it may end a
|
||||
# password that contains "/", "?" or "#"
|
||||
("http://host/path?e=a@b.com", "http://***@b.com"),
|
||||
("https://host/path@file", "https://***@file"),
|
||||
("https://host/?redirect=https://u:p@evil", "https://***@evil"),
|
||||
# Documented non-goal: with userinfo in the authority, credentials
|
||||
# inside a later query value are not masked
|
||||
(
|
||||
"https://a:b@host/?redirect=https://u:p@evil",
|
||||
"https://a:***@host/?redirect=https://u:p@evil",
|
||||
),
|
||||
("https://tok@host", "https://***@host"),
|
||||
("user@host:9200", "***@host:9200"),
|
||||
("https://user:@host", "https://user:***@host"),
|
||||
("https://u%40x:p%3Aw@h", "https://u%40x:***@h"),
|
||||
("not a url @ all", "***@ all"),
|
||||
("", ""),
|
||||
("::::", "::::"),
|
||||
]
|
||||
for value, expected in cases:
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(value), expected
|
||||
)
|
||||
|
||||
def testMasksPasswordContainingAuthorityDelimiter(self):
|
||||
"""A password with an unencoded "/", "?" or "#" ends the authority
|
||||
before the "@" (RFC 3986 section 3.2), which used to leave the value
|
||||
unmasked; everything through the last "@" is masked instead."""
|
||||
cases = [
|
||||
("https://admin:p4ssw0rd/extra@host:3100", "https://***@host:3100"),
|
||||
("https://admin:p4ss?w0rd@host", "https://***@host"),
|
||||
("https://admin:p4ss#w0rd@host", "https://***@host"),
|
||||
("admin:p4ss/w0rd@host:9200", "***@host:9200"),
|
||||
("//admin:p4ss/w0rd@host", "//***@host"),
|
||||
# The password may itself contain "@"; the last one is used
|
||||
("https://admin:p4ss/w@rd@host:3100/x", "https://***@host:3100/x"),
|
||||
# A "://" after the "/" is not the scheme separator
|
||||
(
|
||||
"admin:p4ss/w0rd@host/x?r=http://y",
|
||||
"***@host/x?r=http://y",
|
||||
),
|
||||
]
|
||||
for value, expected in cases:
|
||||
with self.subTest(value=value):
|
||||
result = parsedmarc.utils.redact_url_credentials(value)
|
||||
self.assertEqual(result, expected)
|
||||
self.assertNotIn("p4ss", result)
|
||||
|
||||
def testListMasksPasswordContainingAuthorityDelimiter(self):
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(
|
||||
["https://admin:p4ss/w0rd@h1:9200", "https://h2:9200"]
|
||||
),
|
||||
["https://***@h1:9200", "https://h2:9200"],
|
||||
)
|
||||
|
||||
def testListIsRedactedElementWiseIntoANewList(self):
|
||||
original = ["a:b@h1", "https://h2"]
|
||||
result = parsedmarc.utils.redact_url_credentials(original)
|
||||
self.assertEqual(result, ["a:***@h1", "https://h2"])
|
||||
self.assertIsNot(result, original)
|
||||
self.assertEqual(original, ["a:b@h1", "https://h2"])
|
||||
|
||||
def testListKeepsNonStringItems(self):
|
||||
mixed = cast(list[str], ["a:b@h1", None])
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(mixed), ["a:***@h1", None]
|
||||
)
|
||||
|
||||
def testNonStringInputIsReturnedUnchanged(self):
|
||||
for value in (None, 42):
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.redact_url_credentials(cast(str, value)), value
|
||||
)
|
||||
|
||||
def testLongInputsAreFast(self):
|
||||
"""No regex is used, so adversarial input is linear. A 1 MB string
|
||||
of repeated openers and a 200,000-character run of '@' both finish
|
||||
in milliseconds."""
|
||||
for value in (
|
||||
"://" * 350_000,
|
||||
"@" * 200_000,
|
||||
"a:b@" * 250_000,
|
||||
"a:b/" * 250_000 + "@",
|
||||
"/@" * 500_000,
|
||||
):
|
||||
with self.subTest(prefix=value[:8]):
|
||||
started = time.monotonic()
|
||||
parsedmarc.utils.redact_url_credentials(value)
|
||||
self.assertLess(time.monotonic() - started, 5)
|
||||
|
||||
|
||||
class TestSanitizeForLog(unittest.TestCase):
|
||||
"""Tests for parsedmarc.utils.sanitize_for_log()"""
|
||||
|
||||
@@ -541,6 +658,197 @@ class TestSanitizeForLog(unittest.TestCase):
|
||||
self.assertNotIn("\n", result)
|
||||
|
||||
|
||||
class TestDownloadUrlCredentialsNotLogged(unittest.TestCase):
|
||||
"""A download URL with userinfo must not leak its password through the
|
||||
"Trying to fetch" debug line or the warning that embeds the httpx
|
||||
``HTTPStatusError`` message (which quotes the full URL).
|
||||
|
||||
The loaders run for real; only the network is replaced, with an
|
||||
``httpx.MockTransport`` returning 401 so httpx itself builds the
|
||||
exception text.
|
||||
"""
|
||||
|
||||
URL = "https://dluser:DLSECRET@downloads.example.test/map.dat"
|
||||
|
||||
@staticmethod
|
||||
def _transport_returning(handler):
|
||||
def get(url, **kwargs):
|
||||
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
|
||||
return client.get(url, **kwargs)
|
||||
|
||||
return get
|
||||
|
||||
@classmethod
|
||||
def _status_getter(cls, reason_phrase=None):
|
||||
extensions = {}
|
||||
if reason_phrase is not None:
|
||||
extensions["reason_phrase"] = reason_phrase
|
||||
return cls._transport_returning(
|
||||
lambda request: httpx.Response(401, extensions=extensions)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _connect_error_getter(url, **kwargs):
|
||||
def handler(request):
|
||||
raise httpx.ConnectError("connection refused", request=request)
|
||||
|
||||
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
|
||||
return client.get(url, **kwargs)
|
||||
|
||||
# Reason phrases containing URL delimiters used to defeat redaction of
|
||||
# the HTTPStatusError text, which puts the phrase before the URL.
|
||||
REASON_PHRASES = (
|
||||
None,
|
||||
b"Unauthorized/Expired",
|
||||
b"Unauthorized?",
|
||||
b"Gone#",
|
||||
b"Not @ home",
|
||||
)
|
||||
|
||||
def _assert_redacted(self, logs):
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("DLSECRET", output)
|
||||
self.assertTrue(
|
||||
any(
|
||||
line.startswith("DEBUG") and "downloads.example.test" in line
|
||||
for line in logs.output
|
||||
),
|
||||
output,
|
||||
)
|
||||
warnings = [line for line in logs.output if line.startswith("WARNING")]
|
||||
self.assertTrue(
|
||||
any(
|
||||
"HTTP 401" in line
|
||||
and "'https://dluser:***@downloads.example.test/map.dat'" in line
|
||||
for line in warnings
|
||||
),
|
||||
output,
|
||||
)
|
||||
|
||||
def _run_psl(self, url=None):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
parsedmarc.utils.load_psl_overrides(url=url or self.URL)
|
||||
return logs
|
||||
|
||||
def _run_ip_db(self, url=None):
|
||||
tmp_dir = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp_dir, ignore_errors=True)
|
||||
old_path = parsedmarc.utils._IP_DB_PATH
|
||||
self.addCleanup(setattr, parsedmarc.utils, "_IP_DB_PATH", old_path)
|
||||
parsedmarc.utils._IP_DB_PATH = None
|
||||
with (
|
||||
patch("parsedmarc.utils.tempfile.gettempdir", return_value=tmp_dir),
|
||||
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
|
||||
):
|
||||
parsedmarc.utils.load_ip_db(url=url or self.URL)
|
||||
return logs
|
||||
|
||||
def _run_reverse_dns_map(self, url=None):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
parsedmarc.utils.load_reverse_dns_map({}, url=url or self.URL)
|
||||
return logs
|
||||
|
||||
def _check_loader(self, run):
|
||||
for phrase in self.REASON_PHRASES:
|
||||
with self.subTest(reason_phrase=phrase):
|
||||
with patch("parsedmarc.utils.httpx.get", self._status_getter(phrase)):
|
||||
self._assert_redacted(run())
|
||||
|
||||
def _check_transport_error(self, run):
|
||||
with patch("parsedmarc.utils.httpx.get", self._connect_error_getter):
|
||||
logs = run()
|
||||
output = "\n".join(logs.output)
|
||||
self.assertNotIn("DLSECRET", output)
|
||||
self.assertTrue(
|
||||
any(
|
||||
line.startswith("WARNING")
|
||||
and "ConnectError: connection refused" in line
|
||||
for line in logs.output
|
||||
),
|
||||
output,
|
||||
)
|
||||
|
||||
# The password holds an unencoded "/", "?" or "#", which RFC 3986 section
|
||||
# 3.2 ends the authority at, so the text after the colon is read as the
|
||||
# port. Real httpx raises InvalidURL("Invalid port: 'p4ssw0rd'") while
|
||||
# parsing, before any network I/O (httpx 0.28.1 _urlparse.py:
|
||||
# ``raise InvalidURL(f"Invalid port: {port!r}")``). InvalidURL is not an
|
||||
# httpx.HTTPError, and its message quotes part of the password.
|
||||
MALFORMED_URLS = (
|
||||
("https://admin:p4ssw0rd/extra@downloads.example.test/map.dat", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@downloads.example.test/map.dat", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@downloads.example.test/map.dat", "p4ss"),
|
||||
)
|
||||
INVALID_URL_MESSAGE = (
|
||||
"invalid URL (details withheld because they may include credentials)"
|
||||
)
|
||||
|
||||
def _check_invalid_url(self, run, source):
|
||||
for url, secret in self.MALFORMED_URLS:
|
||||
with self.subTest(url=url):
|
||||
requested = []
|
||||
real_get = httpx.get
|
||||
|
||||
def recording_get(requested_url, **kwargs):
|
||||
requested.append(requested_url)
|
||||
if requested_url != url:
|
||||
# e.g. the PSL overrides fetch that
|
||||
# load_reverse_dns_map triggers; stay offline
|
||||
raise httpx.ConnectError("offline")
|
||||
return real_get(requested_url, **kwargs)
|
||||
|
||||
with patch("parsedmarc.utils.httpx.get", recording_get):
|
||||
logs = run(url) # must return, not crash on InvalidURL
|
||||
# The loader still requested the real URL, not a redacted one
|
||||
self.assertIn(url, requested)
|
||||
output = "\n".join(logs.output)
|
||||
# Neither the warning nor the "Trying to fetch" debug line
|
||||
# may show any part of the password
|
||||
self.assertNotIn(secret, output)
|
||||
self.assertIn(
|
||||
f"DEBUG:parsedmarc.log:Trying to fetch {source} from "
|
||||
"https://***@downloads.example.test/map.dat...",
|
||||
logs.output,
|
||||
)
|
||||
self.assertIn(
|
||||
f"WARNING:parsedmarc.log:Failed to fetch {source}: "
|
||||
f"{self.INVALID_URL_MESSAGE}",
|
||||
logs.output,
|
||||
)
|
||||
self.assertNotIn("Not a valid CSV file", output)
|
||||
self.assertNotIn("Failed to save IP database", output)
|
||||
|
||||
def test_psl_overrides_loader_invalid_url(self):
|
||||
"""A malformed ``psl_overrides_url`` used to crash startup with the
|
||||
InvalidURL traceback; it now warns and falls back to the local file."""
|
||||
self._check_invalid_url(self._run_psl, "PSL overrides")
|
||||
self.assertIn(".linode.com", parsedmarc.utils.psl_overrides)
|
||||
|
||||
def test_ip_db_loader_invalid_url(self):
|
||||
self._check_invalid_url(self._run_ip_db, "IP database")
|
||||
|
||||
def test_reverse_dns_map_loader_invalid_url(self):
|
||||
self._check_invalid_url(self._run_reverse_dns_map, "reverse DNS map")
|
||||
|
||||
def test_psl_overrides_loader(self):
|
||||
self._check_loader(self._run_psl)
|
||||
|
||||
def test_ip_db_loader(self):
|
||||
self._check_loader(self._run_ip_db)
|
||||
|
||||
def test_reverse_dns_map_loader(self):
|
||||
self._check_loader(self._run_reverse_dns_map)
|
||||
|
||||
def test_psl_overrides_loader_transport_error(self):
|
||||
self._check_transport_error(self._run_psl)
|
||||
|
||||
def test_ip_db_loader_transport_error(self):
|
||||
self._check_transport_error(self._run_ip_db)
|
||||
|
||||
def test_reverse_dns_map_loader_transport_error(self):
|
||||
self._check_transport_error(self._run_reverse_dns_map)
|
||||
|
||||
|
||||
class TestLoadPSLOverrides(unittest.TestCase):
|
||||
"""Covers `parsedmarc.utils.load_psl_overrides`."""
|
||||
|
||||
@@ -1559,16 +1867,20 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
must fail at configuration time naming the entry, not parse
|
||||
"successfully" with no TLS certificate identity and then fail at
|
||||
query time with an opaque certificate error. Userinfo and query
|
||||
components are rejected the same way."""
|
||||
for entry in (
|
||||
"tls://9.9.9.9/dns.quad9.net",
|
||||
"tls://user@9.9.9.9",
|
||||
"tls://9.9.9.9?hostname=dns.quad9.net",
|
||||
components are rejected the same way, with the userinfo masked in
|
||||
the message (GHSA-hjr3-rx4c-24cv)."""
|
||||
for entry, shown in (
|
||||
("tls://9.9.9.9/dns.quad9.net", "tls://9.9.9.9/dns.quad9.net"),
|
||||
("tls://user@9.9.9.9", "tls://***@9.9.9.9"),
|
||||
(
|
||||
"tls://9.9.9.9?hostname=dns.quad9.net",
|
||||
"tls://9.9.9.9?hostname=dns.quad9.net",
|
||||
),
|
||||
):
|
||||
with self.subTest(entry=entry):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
parsedmarc.utils._nameservers_to_resolver_input([entry])
|
||||
self.assertIn(entry, str(ctx.exception))
|
||||
self.assertIn(shown, str(ctx.exception))
|
||||
|
||||
def test_unsplittable_entry_is_left_for_dnspython_to_reject(self):
|
||||
"""urlsplit itself raises on some malformed input (an unbalanced
|
||||
@@ -1712,6 +2024,152 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
|
||||
self.assertEqual(parsedmarc.utils._DOH_SESSION_PID, os.getpid())
|
||||
|
||||
|
||||
class TestNameserverCredentialsNotLeaked(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: nameserver URL credentials must not reach an
|
||||
exception message, its printed traceback, or the log."""
|
||||
|
||||
def test_url_secrets_finds_the_password_and_its_port_like_prefix(self):
|
||||
"""The secret is the userinfo after its first ":" (the whole
|
||||
userinfo when there is none). A "/", "?" or "#" in it ends the URL
|
||||
authority (RFC 3986 section 3.2), so urlsplit() reads the text
|
||||
before it as the port and quotes it in its error; that prefix is a
|
||||
secret too."""
|
||||
cases = [
|
||||
("https://doh:p4ss@dns.example/q", ["p4ss"]),
|
||||
("https://doh:p4ss/w0rd@dns.example/q", ["p4ss/w0rd", "p4ss"]),
|
||||
("https://doh:p4ss?w0rd@dns.example/q", ["p4ss?w0rd", "p4ss"]),
|
||||
("https://doh:p4ss#w0rd@dns.example/q", ["p4ss#w0rd", "p4ss"]),
|
||||
# The prefix ends at the first of the three delimiters
|
||||
("https://doh:ab#cd/ef@dns.example/q", ["ab#cd/ef", "ab"]),
|
||||
("https://doh:p:w@dns.example/q", ["p:w"]),
|
||||
("https://doh:p@ss@dns.example/q", ["p@ss"]),
|
||||
("https://S3CRETTOKEN@dns.example/q", ["S3CRETTOKEN"]),
|
||||
("tls://dot:p4ss/w0rd@1.1.1.1", ["p4ss/w0rd", "p4ss"]),
|
||||
# Empty secrets are never returned: str.replace("", ...) would
|
||||
# insert the mask between every character
|
||||
("https://doh:@dns.example/q", []),
|
||||
("https://doh:/x@dns.example/q", ["/x"]),
|
||||
("https://dns.example/dns-query", []),
|
||||
("8.8.8.8", []),
|
||||
("", []),
|
||||
]
|
||||
for value, expected in cases:
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(parsedmarc.utils._url_secrets(value), expected)
|
||||
|
||||
def test_mask_nameserver_secrets_masks_entries_passwords_and_prefixes(self):
|
||||
"""Each configured entry is replaced by its redacted form, and each
|
||||
secret in it wherever it still appears, including the bare port-like
|
||||
prefix that urlsplit() and httpx quote."""
|
||||
mask = parsedmarc.utils._mask_nameserver_secrets
|
||||
url = "https://doh:p4ss/w0rd@dns.example/dns-query"
|
||||
text = (
|
||||
f"Server {url} answered x; "
|
||||
"Port could not be cast to integer value as 'p4ss'; "
|
||||
"Invalid port: 'p4ss/w0rd'"
|
||||
)
|
||||
self.assertEqual(
|
||||
mask(text, [url, "8.8.8.8"]),
|
||||
"Server https://***@dns.example/dns-query answered x; "
|
||||
"Port could not be cast to integer value as '***'; "
|
||||
"Invalid port: '***'",
|
||||
)
|
||||
self.assertEqual(
|
||||
mask(
|
||||
"Server https://u:PW123@h.example/q failed (PW123)",
|
||||
["https://u:PW123@h.example/q"],
|
||||
),
|
||||
"Server https://u:***@h.example/q failed (***)",
|
||||
)
|
||||
self.assertEqual(mask("abc", None), "abc")
|
||||
self.assertEqual(mask("abc", ["https://u:@h"]), "abc")
|
||||
|
||||
def test_mask_nameserver_secrets_handles_unsplittable_entries(self):
|
||||
"""urlsplit() raises ValueError for an entry such as ``https://[x``;
|
||||
the masking uses string methods only, so it still returns text."""
|
||||
text = "Server https://[x failed; Server https://u:PW@[x failed"
|
||||
self.assertEqual(
|
||||
parsedmarc.utils._mask_nameserver_secrets(
|
||||
text, ["https://[x", "https://u:PW@[x"]
|
||||
),
|
||||
"Server https://[x failed; Server https://u:***@[x failed",
|
||||
)
|
||||
|
||||
def test_malformed_https_and_tls_ports_are_reported_without_secrets(self):
|
||||
"""A password containing "/", "?" or "#" makes urlsplit() read its
|
||||
start as the port, and its ValueError quotes that text; dnspython
|
||||
2.9.1 raised it from ``DoHNameserver.answer_port()`` only after a
|
||||
failed query (``dns/nameserver.py``). Both schemes are now rejected
|
||||
before any query, naming the entry with its credentials masked and
|
||||
with no chained exception repeating the urlsplit message."""
|
||||
for entry, shown in (
|
||||
(
|
||||
"https://doh:p4ss/w0rd@dns.example/dns-query",
|
||||
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"https://doh:p4ss?w0rd@dns.example/dns-query",
|
||||
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
|
||||
),
|
||||
(
|
||||
"tls://dot:p4ss#w0rd@1.1.1.1",
|
||||
"Invalid DNS over TLS nameserver tls://***@1.1.1.1",
|
||||
),
|
||||
(
|
||||
"https://dns.example:notaport/dns-query",
|
||||
"Invalid DNS over HTTPS nameserver https://dns.example:notaport/",
|
||||
),
|
||||
):
|
||||
with self.subTest(entry=entry):
|
||||
with patch("dns.query.https", side_effect=AssertionError):
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
parsedmarc.utils.query_dns(
|
||||
"1.0.0.1.in-addr.arpa",
|
||||
"PTR",
|
||||
nameservers=[entry],
|
||||
timeout=0.2,
|
||||
)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertIn(shown, str(ctx.exception))
|
||||
self.assertIsNone(ctx.exception.__cause__)
|
||||
self.assertIsNone(ctx.exception.__context__)
|
||||
for secret in ("p4ss", "w0rd"):
|
||||
self.assertNotIn(secret, trace)
|
||||
|
||||
def test_dnspython_rejection_of_an_entry_is_masked(self):
|
||||
"""dnspython rejects an entry urlsplit() cannot split with a message
|
||||
that quotes it as configured ("nameserver https://u:pw@[x is not a
|
||||
dns.nameserver.Nameserver instance ..."); query_dns() masks it."""
|
||||
entry = "https://doh:S3CRETPW@[dns.example/dns-query"
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
parsedmarc.utils.query_dns(
|
||||
"1.0.0.1.in-addr.arpa", "PTR", nameservers=[entry], timeout=0.2
|
||||
)
|
||||
trace = "".join(traceback.format_exception(ctx.exception))
|
||||
self.assertIn("https://doh:***@[dns.example/dns-query", str(ctx.exception))
|
||||
self.assertNotIn("S3CRETPW", trace)
|
||||
|
||||
def test_get_reverse_dns_debug_log_masks_nameserver_credentials(self):
|
||||
"""get_reverse_dns() logs the DNS error it swallows at debug level,
|
||||
and LifetimeTimeout/NoNameservers quote each failing nameserver
|
||||
URL, so the configured credentials are masked there too."""
|
||||
url = "https://doh:S3CRETPW@dns.example/dns-query"
|
||||
|
||||
def fail(*args, **kwargs):
|
||||
raise OSError(f"connection to {url} refused (S3CRETPW)")
|
||||
|
||||
with patch("dns.query.https", side_effect=fail):
|
||||
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
|
||||
result = parsedmarc.utils.get_reverse_dns(
|
||||
"1.0.0.1", nameservers=[url], timeout=0.2, retries=0
|
||||
)
|
||||
self.assertIsNone(result)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertIn("get_reverse_dns(1.0.0.1) exception:", output)
|
||||
self.assertIn("https://doh:***@dns.example/dns-query", output)
|
||||
self.assertNotIn("S3CRETPW", output)
|
||||
|
||||
|
||||
class TestLoadIpDb(unittest.TestCase):
|
||||
"""Tests for the load_ip_db() download/cache/bundled fallback chain,
|
||||
mocking at the httpx SDK boundary."""
|
||||
|
||||
@@ -114,6 +114,50 @@ class TestWebhookErrorHandling(unittest.TestCase):
|
||||
client.save_smtp_tls_report_to_webhook('{"t": 1}')
|
||||
|
||||
|
||||
class TestWebhookInvalidUrlDoesNotLeakPassword(unittest.TestCase):
|
||||
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
|
||||
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
|
||||
text after the colon as the port and raises
|
||||
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
|
||||
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
|
||||
any network I/O. The error is logged on every report."""
|
||||
|
||||
MALFORMED = (
|
||||
("https://admin:p4ssw0rd/extra@hooks.example.com/x", "p4ssw0rd"),
|
||||
("https://admin:p4ss?w0rd@hooks.example.com/x", "p4ss"),
|
||||
("https://admin:p4ss#w0rd@hooks.example.com/x", "p4ss"),
|
||||
)
|
||||
|
||||
def test_invalid_url_is_logged_without_password(self):
|
||||
saves = (
|
||||
"save_aggregate_report_to_webhook",
|
||||
"save_failure_report_to_webhook",
|
||||
"save_smtp_tls_report_to_webhook",
|
||||
)
|
||||
for url, secret in self.MALFORMED:
|
||||
client = WebhookClient(url, url, url)
|
||||
for method in saves:
|
||||
with self.subTest(url=url, method=method):
|
||||
with self.assertLogs("parsedmarc.log", level="ERROR") as cm:
|
||||
getattr(client, method)('{"a": 1}') # must not raise
|
||||
self.assertEqual(
|
||||
cm.output,
|
||||
[
|
||||
"ERROR:parsedmarc.log:Webhook Error: invalid webhook "
|
||||
"URL (details withheld because they may include "
|
||||
"credentials)"
|
||||
],
|
||||
)
|
||||
self.assertNotIn(secret, "\n".join(cm.output))
|
||||
|
||||
def test_valid_url_is_the_one_posted(self):
|
||||
url = "https://admin:p4ssw0rd@hooks.example.com/x"
|
||||
client = WebhookClient(url, url, url)
|
||||
client.session = MagicMock()
|
||||
client.save_aggregate_report_to_webhook('{"a": 1}')
|
||||
client.session.post.assert_called_once_with(url, content='{"a": 1}', timeout=60)
|
||||
|
||||
|
||||
class TestWebhookClientClose(unittest.TestCase):
|
||||
def test_close_closes_session(self):
|
||||
client = _client()
|
||||
|
||||
Reference in new issue
Block a user