Merge commit from fork

* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv)

With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init
lines and opensearch.set_hosts() logged configured URLs verbatim, so
userinfo passwords reached the console and log_file (CWE-532).

Triage found more paths of the same class:
- the httpx "HTTP Request:" line, forwarded at --verbose/--debug,
  contains the full URL (str(httpx.URL) keeps userinfo and query), so
  webhook and HEC URL credentials and the IPinfo ?token= were logged
  on every request;
- the PSL overrides, IP database and reverse DNS map downloaders logged
  their URL at DEBUG and str(HTTPStatusError), which embeds the full
  URL, at WARNING;
- libpq echoes the DSN or a password fragment for some malformed
  PostgreSQL connection strings, logged as an output-client error.

Add redact_url_credentials() (string methods only; urlsplit misparses
scheme-less "user:pw@host:port") and use it at each log site; add a
filter on the httpx logger that masks userinfo and the token query
value; build download-failure warnings from the status code and the
redacted request URL instead of the exception text; replace psycopg
ProgrammingError detail from a connection_string with a fixed message.
Docs now recommend separate user/password settings over URL
credentials.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Mask credentials in the Loki URL in the log and LokiError text

The Loki output (#926) landed on master after this fix was written. Its
url keeps userinfo (copy_with(path=...) preserves it), and the --debug
"Initializing Loki client" line and both LokiError messages, which the
CLI logs on a failed push, included it as-is: the same CWE-532 path as
the Splunk HEC line. Mask it with redact_url_credentials() at all three
sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers

Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532):

- httpx.InvalidURL quotes the component it rejected, so a password with
  an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there)
  leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats
  it. Loki, Splunk HEC, the webhook and the three download loaders now
  report a fixed "details withheld" message raised from None.
  InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url
  crashed startup; the loaders now treat it as a download failure.
- redact_url_credentials() masks through the last "@" when the authority
  has no "@" but one follows it, so the --debug lines no longer print
  such passwords; the userinfo lookup is shared with the new nameserver
  masking.
- dnspython 2.9 DoH logs its request line on the httpx2 logger, which was
  forwarded but not filtered; the filter now covers httpx2 and
  httpx2.URL.
- [general] nameservers / --nameservers credentials are masked in the
  DNS pre-flight error, reverse DNS debug errors and per-report errors,
  and unusable https:// / tls:// ports are rejected before any query.
- Docs recommend token/username settings for [loki] and [splunk_hec];
  AGENTS.md gains rules for keeping credentials out of logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-10-31 17:14:00 -04:00
1 parent 56e5a09923
commit 48ea47adee
17 files changed
+1595 -73

No files matched your search

+12
View File
@@ -135,6 +135,18 @@ Assume every regex in the package runs on content a remote sender controls (mess
- **Time the pattern on an adversarial input before shipping it**, not just on the samples: a 1 MB string of the repeated opener with no closer, and a 200,000-character whitespace run. A linear pattern finishes in milliseconds; anything that takes seconds is a bug. Add that input as a time-bound regression test with a generous bound (seconds) and cite the measured old timing in the docstring, as the regression tests in `tests/test_init.py` and `tests/test_utils.py` do.
- Python 3.10 is the floor, so possessive quantifiers and atomic groups (3.11+) are not available; fix the pattern's structure instead.
### Credentials must never reach logs or error text
Assume every configured URL, host list, connection string, and token carries a secret, and that every log line and exception message ends up on a console, in `log_file`, or in a pasted bug report (CWE-532). GHSA-hjr3-rx4c-24cv found leaks in `--debug` client-init lines, the forwarded httpx request line, download warnings, libpq's DSN echo, and httpx's `InvalidURL` text. Before writing a log call or an exception message that touches configuration:
- **Never log a configured value as-is.** Pass URLs and host lists through `redact_url_credentials()` (`parsedmarc/utils.py`), which masks userinfo (`https://user:***@host`), handles scheme-less `user:pw@host:port`, and, when the authority has no `@` but one follows it (a `/`, `?`, or `#` in the password, but also a credential-free URL or free text with a later `@`), masks through the last `@` (`https://***@host`). Redaction is log-only: the SDK or transport still gets the real value, so mask at the log call and not in `opts` or a client attribute.
- **Treat third-party exception text as a leak until shown otherwise.** `httpx.HTTPStatusError` embeds the full request URL. `httpx.InvalidURL` quotes the component it rejected, so a password with an unencoded `/`, `?`, or `#` becomes `Invalid port: 'p4ssw0rd'`. libpq echoes the DSN for some malformed connection strings. Don't log `str(e)` from parsing, connecting to, or requesting a configured endpoint. Build the message from structured fields (`_describe_http_error()`: status code plus the redacted request URL), or use a fixed "details withheld because they may include credentials" message. Read the library source in the venv to check what its messages contain; don't assume.
- **Raise the replacement `from None`.** `from e` and the implicit `__context__` both print the original exception when it is logged with `logger.exception` (the SIGHUP reload path does this) or when parsedmarc crashes, and `InvalidURL`'s own context is a `ValueError` that repeats the secret.
- **Catch the right class.** `httpx.InvalidURL` is not an `httpx.HTTPError`, so an `except httpx.HTTPError` falls through to a generic handler that logs the raw text, or doesn't catch it at all and crashes startup.
- **Check dependency loggers.** httpx logs every request URL at INFO. `_configure_dependency_logging()` (`parsedmarc/cli.py`) attaches `_RedactUrlCredentialsFilter` to the `httpx` and `httpx2` loggers (dnspython 2.9+ makes DNS over HTTPS requests through httpx2, and `httpx2.URL` is not an `httpx.URL`). A new dependency logger forwarded at `--verbose`/`--debug` needs the same check, and so does a new library that logs its endpoint.
- **Test both halves, including malformed input.** Assert that the secret is absent from the captured log *and* from `"".join(traceback.format_exception(exc))`, and that the real credential still reaches the mocked transport. Include values the samples never contain: a password with `/`, `?`, or `#`; userinfo with no `:` (a bare token); and a scheme-less `user:pw@host:port`.
- **Document the safe setting.** When an output has separate `user`/`username` and `password`, `token`, or `api_key` settings, its `usage.md` entry should recommend them over credentials in the URL.
## Testing standards
These rules govern *every* test added to `tests/`. They exist because the project has been burned by tests that looked like coverage but caught nothing, and by bug claims that turned out to be wrong about the spec. Both failure modes erode trust faster than missing coverage does.
+1
View File
@@ -10,6 +10,7 @@
### Security
- **S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report** ([GHSA-7wqr-f6g9-cm29](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-7wqr-f6g9-cm29), affects 11.1.0 and earlier). `S3Client.save_report_to_s3()` put the sender-controlled report ID into the object key as-is, and sent `org_name` and the other metadata values as-is. A report ID such as `../../x` added path segments to the key, an ID could carry `\`, `%`, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit ([AWS, "Naming Amazon S3 objects"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html)). botocore rejects non-ASCII metadata with `ParamValidationError`, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise `HTTPClientError: Invalid header value` before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the `x-amz-meta-*` header; the 2 KB metadata limit ([AWS, "Working with object metadata"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)) was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are `/`, every AWS "character to avoid" (`` \ { } ^ [ ] " < > ~ # | ` `` and `%`), control characters, and non-ASCII characters, so `%` appears in a key only as the escape character. The punctuation real report IDs use, such as `:`, `@`, `.`, and `_`, is kept. Each metadata value is percent-encoded (`%`, control characters, and non-ASCII characters) so `urllib.parse.unquote()` reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in `_<hash>.json`, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
- **Credentials in URLs and the IPinfo API token are no longer written to the log** ([GHSA-hjr3-rx4c-24cv](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-hjr3-rx4c-24cv), affects 11.1.0 and earlier). The `[elasticsearch]` and `[opensearch]` `hosts` settings and the `[splunk_hec]` `url` setting accept URLs, and a URL can carry credentials (`https://user:secret@host:9200`). With `--debug`, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to `log_file`. With `--verbose` or `--debug`, parsedmarc also forwards the `httpx` library's `HTTP Request:` line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in `[general]` `nameservers` (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the `ipinfo_api_token` was logged on every IPinfo lookup as the `token` query parameter. The password, or the whole userinfo when it has no `:`, is now replaced with `***` in these lines (`https://user:***@host:9200`), and so is the `token` query value. Credentials in a `[general]` `nameservers` entry are also masked in the DNS pre-flight error, and credentials in either that setting or a `--nameservers` entry are masked in the `--debug` reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable `https://` or `tls://` entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded `/`, `?`, or `#` (see below), is now rejected before any query is sent. The download URLs `ipinfo_url`, `reverse_dns_map_url`, and `psl_overrides_url` (`[general]`) can carry credentials too: they were logged as-is at `--debug` in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with `--warnings`, `--verbose`, or `--debug`) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by `***`. A malformed `[postgresql]` `connection_string` could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors (`[loki]` `url`, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded `/`, `?`, or `#` ends the URL's authority early ([RFC 3986, section 3.2](https://www.rfc-editor.org/rfc/rfc3986#section-3.2)), so httpx reads the text after the colon as the port and raises `InvalidURL("Invalid port: 'p4ssw0rd'")`, with a chained `ValueError` that repeats that part of the password. That text reached the log from `[loki]` `url` (at startup and on a SIGHUP reload), from `[splunk_hec]` `url` (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from `ipinfo_url` (in the "Failed to save IP database" warning). A malformed `psl_overrides_url` crashed startup with the traceback, because `InvalidURL` is not an `httpx.HTTPError`. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed `psl_overrides_url` now logs a warning and falls back to the `local_psl_overrides_path` file when that is set, or else to the bundled file, and a malformed `reverse_dns_map_url` or `ipinfo_url` is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no `@` in its authority, so the `--debug` "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last `@` (`https://***@host:3100`). A URL with no credentials but an `@` in its path or query is masked the same way in these lines (`https://host/path@file` is logged as `https://***@file`). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: `user` and `password` (or `api_key`) for `[elasticsearch]` and `[opensearch]`, `user` and `password` for `[postgresql]`, `username` and `password` (or `token`) for `[loki]`, and `token` for `[splunk_hec]`. Reported by helixnebulatech.
### New features
+26 -23
View File
@@ -91,7 +91,9 @@ hosts = 127.0.0.1:9200
ssl = False
[opensearch]
hosts = https://admin:admin@127.0.0.1:9200
hosts = https://127.0.0.1:9200
user = admin
password = PasswordGoesHere
ssl = True
[splunk_hec]
@@ -571,14 +573,10 @@ The full set of configuration options is:
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
`https://user:secret@localhost`)
:::{note}
Special characters in the username or password must be
[URL encoded].
:::
- `hosts` - str: A comma-separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
Do not put credentials in the URL; use `user` and `password`
(or `api_key`) instead.
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
@@ -616,14 +614,10 @@ The full set of configuration options is:
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
`https://user:secret@localhost`)
:::{note}
Special characters in the username or password must be
[URL encoded].
:::
- `hosts` - str: A comma-separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
Do not put credentials in the URL; use `user` and `password`
(or `api_key`) instead.
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
@@ -656,7 +650,8 @@ The full set of configuration options is:
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
(Default: `False`)
- `splunk_hec`
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
Do not put credentials in the URL; use `token` instead.
- `token` - str: The HEC token
- `index` - str: The Splunk index to use
- `skip_certificate_verification` - bool: Skip certificate
@@ -715,8 +710,13 @@ The full set of configuration options is:
- `password` - str: The database user password (Optional)
- `database` - str: The database name (Optional)
- `connection_string` - str: A full libpq connection string or URI
(e.g. `postgresql://user:pass@host/dbname`). When provided,
all individual parameters above are ignored.
(e.g. `postgresql://user@host/dbname`). When provided,
all individual parameters above are ignored. Prefer the separate
`host`, `port`, `user`, `password`, and `database` settings and do
not put the password in the connection string: a malformed
connection string can make the driver echo it back in an error
message. parsedmarc replaces such errors with a fixed message in
its log, but cannot stop the driver from producing them.
The PostgreSQL backend is an optional extra. Install it with
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
@@ -743,11 +743,13 @@ The full set of configuration options is:
database = parsedmarc
```
Or using a DSN/URI:
Or using a DSN/URI, with the password supplied separately (for
example through the `PGPASSWORD` environment variable or a
`~/.pgpass` file, both of which libpq reads):
```ini
[postgresql]
connection_string = postgresql://parsedmarc:secret@localhost/parsedmarc
connection_string = postgresql://parsedmarc@localhost/parsedmarc
```
Saving parsed data to PostgreSQL is controlled by the `[general]`
@@ -954,6 +956,8 @@ The full set of configuration options is:
It expects the default `job=parsedmarc` label and `label_fields`.
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
URL has no path. A URL with a path is used as is.
Do not put credentials in the URL; use `username` and `password`
(or `token`) instead.
- `labels` - str: Comma-separated `name=value` stream labels, merged over
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
(in a container, the FQDN is the container's hostname)
@@ -1568,5 +1572,4 @@ journalctl -u parsedmarc.service -r
:::
[cloudflare's public resolvers]: https://1.1.1.1/
[url encoded]: https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api
+83 -7
View File
@@ -20,6 +20,7 @@ from ssl import CERT_NONE, create_default_context
from typing import TYPE_CHECKING, Any
import dns.exception
import dns.query
import dns.resolver
import dns.reversename
import httpx
@@ -130,6 +131,7 @@ import parsedmarc.utils
from parsedmarc.utils import (
InvalidIPinfoAPIKey,
ReverseDNSMap,
_mask_nameserver_secrets,
configure_ipinfo_api,
get_base_domain,
human_timestamp_to_datetime,
@@ -138,6 +140,7 @@ from parsedmarc.utils import (
load_psl_overrides,
load_reverse_dns_map,
query_dns,
redact_url_credentials,
sanitize_for_log,
)
@@ -684,6 +687,59 @@ _DEPENDENCY_LOGGERS = (
)
# URL classes whose instances appear as log-record arguments. dnspython
# 2.9+ does DNS over HTTPS through httpx2, whose URL is not an httpx.URL;
# like _doh_client_class() in utils.py, take the module dnspython itself
# imported (absent under dnspython 2.7/2.8, which use httpx).
_HTTP_URL_CLASSES: tuple[type, ...] = (httpx.URL,) + tuple(
module.URL for module in (getattr(dns.query, "httpx2", None),) if module
)
class _RedactUrlCredentialsFilter(logging.Filter):
"""Masks URL credentials and the IPinfo ``token`` query parameter in
the arguments of httpx and httpx2 log records.
httpx and httpx2 log ``HTTP Request: %s %s "%s %d %s"`` with
``request.url``, whose string form includes the userinfo password and
the query string. httpx2 is what dnspython 2.9+ uses for DNS over HTTPS,
so a ``[general]`` ``nameservers`` URL with credentials reaches this
line on every lookup.
The filter never drops a record; the redaction itself is total (string
methods and URL parameter lookups only), so it cannot raise.
Logger-level filters apply only to records created on the logger the
filter is attached to, not to records from its child loggers. httpx
(0.28) logs only via ``logging.getLogger("httpx")`` itself and httpx2
only via ``logging.getLogger("httpx2")`` (``httpx2/_client.py``), so
attaching the filter to those two loggers covers them; a version that
logged on a child logger (``httpx.something``) would bypass it.
"""
def filter(self, record: logging.LogRecord) -> bool:
if isinstance(record.args, tuple):
record.args = tuple(self._redact(arg) for arg in record.args)
return True
@staticmethod
def _redact(arg: Any) -> Any:
if isinstance(arg, _HTTP_URL_CLASSES):
if arg.params.get("token") is not None:
# Drop the token and append the mask as typed: setting it
# with copy_set_param() would percent-encode the "*"s, and
# rewriting the encoded form afterwards would also touch a
# literal "%2A%2A%2A" elsewhere in the URL.
text = str(arg.copy_remove_param("token"))
text, fragment_sep, fragment = text.partition("#")
separator = "&" if "?" in text else "?"
text = f"{text}{separator}token=***{fragment_sep}{fragment}"
return redact_url_credentials(text)
return redact_url_credentials(str(arg))
if isinstance(arg, str):
return redact_url_credentials(arg)
return arg
def _configure_dependency_logging(level: int) -> None:
"""Apply parsedmarc's logging verbosity to dependency loggers.
@@ -697,6 +753,11 @@ def _configure_dependency_logging(level: int) -> None:
duplicates output nor keeps writing to a closed handler. Propagation
to the root logger is disabled so that a stray ``logging.basicConfig()``
anywhere in the process cannot double-print every dependency record.
The ``httpx`` and ``httpx2`` loggers also get a
:class:`_RedactUrlCredentialsFilter`
(installed once, so reloads do not stack copies) that masks URL
credentials and the IPinfo ``token`` query parameter in its
``HTTP Request:`` records.
CLI-only: library consumers configure logging themselves.
"""
dep_level = min(level, logging.WARNING)
@@ -710,6 +771,12 @@ def _configure_dependency_logging(level: int) -> None:
for wanted in logger.handlers:
if wanted not in dep_logger.handlers:
dep_logger.addHandler(wanted)
for name in ("httpx", "httpx2"):
http_logger = logging.getLogger(name)
if not any(
isinstance(f, _RedactUrlCredentialsFilter) for f in http_logger.filters
):
http_logger.addFilter(_RedactUrlCredentialsFilter())
def _load_config(config_file: str | None = None) -> ConfigParser:
@@ -847,12 +914,15 @@ def _parse_config(config: ConfigParser, opts):
)
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as ns_error:
raise ConfigurationError(
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {opts.nameservers}: {ns_error}"
f"DNS pre-flight check failed: no PTR record for {opts.dns_test_address} from {redact_url_credentials(opts.nameservers or [])}: {ns_error}"
) from ns_error
except Exception as ns_error:
# A DNS over HTTPS error can quote the request URL, which
# carries the configured nameserver's credentials.
raise ConfigurationError(
f"DNS pre-flight check failed: {ns_error}"
) from ns_error
f"DNS pre-flight check failed: {type(ns_error).__name__}: "
f"{_mask_nameserver_secrets(str(ns_error), opts.nameservers)}"
) from None
if "save_aggregate" in general_config:
opts.save_aggregate = bool(general_config.getboolean("save_aggregate"))
if "save_failure" in general_config:
@@ -2070,7 +2140,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
"HEC token and HEC index are required when using HEC URL"
)
try:
logger.debug("Initializing Splunk HEC client: url=%s", opts.hec)
logger.debug(
"Initializing Splunk HEC client: url=%s",
redact_url_credentials(opts.hec),
)
verify = True
if opts.hec_skip_certificate_verification:
verify = False
@@ -2137,7 +2210,10 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
try:
if opts.loki_url:
logger.debug("Initializing Loki client: url=%s", opts.loki_url)
logger.debug(
"Initializing Loki client: url=%s",
redact_url_credentials(opts.loki_url),
)
clients["loki_client"] = loki.LokiClient(
opts.loki_url,
labels=opts.loki_labels,
@@ -2182,7 +2258,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
if opts.elasticsearch_hosts:
logger.debug(
"Initializing Elasticsearch client: hosts=%s, ssl=%s",
opts.elasticsearch_hosts,
redact_url_credentials(opts.elasticsearch_hosts),
opts.elasticsearch_ssl,
)
es_aggregate_indexes = _migration_index_names(
@@ -2253,7 +2329,7 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None):
if opts.opensearch_hosts:
logger.debug(
"Initializing OpenSearch client: hosts=%s, ssl=%s",
opts.opensearch_hosts,
redact_url_credentials(opts.opensearch_hosts),
opts.opensearch_ssl,
)
os_aggregate_indexes = _migration_index_names(
+21 -8
View File
@@ -19,7 +19,7 @@ from parsedmarc import (
from parsedmarc.constants import USER_AGENT
from parsedmarc.log import logger
from parsedmarc.types import AggregateReport, FailureReport, SMTPTLSReport
from parsedmarc.utils import human_timestamp_to_unix_timestamp
from parsedmarc.utils import human_timestamp_to_unix_timestamp, redact_url_credentials
PUSH_PATH = "/loki/api/v1/push"
TIMESTAMP_MODES = ("ingest", "report_begin")
@@ -102,11 +102,22 @@ class LokiClient(object):
if not _LABEL_NAME.match(name):
raise ValueError(f"Invalid Loki label name '{name}'")
parsed = httpx.URL(url)
# copy_with(path=...) keeps any query string, userinfo and port intact
self.url = (
str(parsed.copy_with(path=PUSH_PATH)) if parsed.path in ("", "/") else url
)
try:
parsed = httpx.URL(url)
# copy_with(path=...) keeps any query string, userinfo and port intact
self.url = (
str(parsed.copy_with(path=PUSH_PATH))
if parsed.path in ("", "/")
else url
)
except httpx.InvalidURL:
# httpx quotes part of the URL (part of a password containing
# "/", "?" or "#" ends up as the "port"), and chains a ValueError
# that repeats it, so neither is kept.
raise ValueError(
"Invalid Loki URL (details withheld because they may include "
"credentials)"
) from None
self.timestamp = timestamp
self.timeout = timeout
self.batch_size = max(1, int(batch_size))
@@ -196,14 +207,16 @@ class LokiClient(object):
self.url, content=json.dumps(body), timeout=self.timeout
)
except httpx.HTTPError as error_:
raise LokiError(f"Loki push to {self.url} failed: {error_}") from error_
raise LokiError(
f"Loki push to {redact_url_credentials(self.url)} failed: {error_}"
) from error_
# Loki answers 204 when it stored the push; 200 is also accepted.
# Other 2xx codes are not success: with block_ingestion_until set it
# returns a custom 260 and discards the logs
# (block_ingestion_status_code).
if response.status_code not in (200, 204):
raise LokiError(
f"Loki push to {self.url} failed with HTTP "
f"Loki push to {redact_url_credentials(self.url)} failed with HTTP "
f"{response.status_code}: {response.text[:500]}"
)
+6 -2
View File
@@ -29,7 +29,7 @@ from opensearchpy.helpers import reindex
from parsedmarc import InvalidFailureReport
from parsedmarc.log import logger
from parsedmarc.utils import human_timestamp_to_datetime
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
class OpenSearchError(Exception):
@@ -552,7 +552,11 @@ def set_hosts(
_READY_DATA_STREAMS.clear()
if not isinstance(hosts, list):
hosts = [hosts]
logger.debug("Connecting to OpenSearch: hosts=%s, use_ssl=%s", hosts, use_ssl)
logger.debug(
"Connecting to OpenSearch: hosts=%s, use_ssl=%s",
redact_url_credentials(hosts),
use_ssl,
)
conn_params = {"hosts": hosts, "timeout": timeout}
if use_ssl:
conn_params["use_ssl"] = True
+11 -2
View File
@@ -20,7 +20,7 @@ else:
psycopg_json = None
from parsedmarc.log import logger
from parsedmarc.utils import human_timestamp_to_datetime
from parsedmarc.utils import human_timestamp_to_datetime, redact_url_credentials
# psycopg is an optional dependency (the PostgreSQL backend is opt-in). The
# pure helper functions below work without it; only PostgreSQLClient needs a
@@ -183,7 +183,16 @@ class PostgreSQLClient:
)
conn.autocommit = False
except psycopg.Error as exc:
raise PostgreSQLError(str(exc)) from exc
# libpq reports a malformed connection string by echoing the
# whole string, or the fragment it could not parse, and either
# can contain the password. psycopg raises ProgrammingError for
# those parse failures, so withhold their detail entirely.
if self._connection_string and isinstance(exc, psycopg.ProgrammingError):
raise PostgreSQLError(
"Invalid PostgreSQL connection_string (details withheld "
"because they may include the password)"
) from None
raise PostgreSQLError(redact_url_credentials(str(exc))) from exc
self._conn = conn
return conn
+13
View File
@@ -13,6 +13,13 @@ from parsedmarc.constants import USER_AGENT
from parsedmarc.log import logger
from parsedmarc.utils import human_timestamp_to_unix_timestamp
# httpx.InvalidURL quotes part of the URL, which can be part of a password
# (one containing "/", "?" or "#" ends up as the "port"), so its text is
# replaced by this fixed message.
_INVALID_URL_MESSAGE = (
"Invalid Splunk HEC URL (details withheld because they may include credentials)"
)
class SplunkError(RuntimeError):
"""Raised when a Splunk API error occurs"""
@@ -136,6 +143,8 @@ class HECClient(object):
self.url, content="".join(json_lines), timeout=self.timeout
)
response = response.json()
except httpx.InvalidURL:
raise SplunkError(_INVALID_URL_MESSAGE) from None
except Exception as e:
raise SplunkError(e.__str__())
if response["code"] != 0:
@@ -179,6 +188,8 @@ class HECClient(object):
self.url, content="".join(json_lines), timeout=self.timeout
)
response = response.json()
except httpx.InvalidURL:
raise SplunkError(_INVALID_URL_MESSAGE) from None
except Exception as e:
raise SplunkError(e.__str__())
if response["code"] != 0:
@@ -218,6 +229,8 @@ class HECClient(object):
self.url, content="".join(json_lines), timeout=self.timeout
)
response = response.json()
except httpx.InvalidURL:
raise SplunkError(_INVALID_URL_MESSAGE) from None
except Exception as e:
raise SplunkError(e.__str__())
if response["code"] != 0:
+294 -25
View File
@@ -17,7 +17,7 @@ import subprocess
import tempfile
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, TypedDict, cast
from urllib.parse import urlsplit
from urllib.parse import SplitResult, urlsplit
import mailparser
from expiringdict import ExpiringDict
@@ -113,15 +113,17 @@ def load_psl_overrides(
if not (offline or always_use_local_file):
try:
logger.debug(f"Trying to fetch PSL overrides from {url}...")
logger.debug(
f"Trying to fetch PSL overrides from {redact_url_credentials(url)}..."
)
headers = {"User-Agent": USER_AGENT}
response = httpx.get(
url, headers=headers, timeout=60, follow_redirects=True
)
response.raise_for_status()
_load_text(response.text)
except httpx.HTTPError as e:
logger.warning(f"Failed to fetch PSL overrides: {e}")
except (httpx.HTTPError, httpx.InvalidURL) as e:
logger.warning(f"Failed to fetch PSL overrides: {_describe_http_error(e)}")
if len(psl_overrides) == 0:
path = local_file_path or str(
@@ -309,6 +311,44 @@ class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
)
def _nameserver_port(entry: str, parts: SplitResult, kind: str) -> int | None:
"""
Returns the port of a parsed DNS over HTTPS or TLS nameserver URL.
``urlsplit()`` only validates the port when it is read, and its
``ValueError`` quotes the text it rejected. A password containing an
unencoded ``/``, ``?`` or ``#`` ends the authority early (RFC 3986
section 3.2), which makes the start of the password that text
(``https://doh:p4ss/w0rd@host`` gives ``Port could not be cast to
integer value as 'p4ss'``). dnspython 2.9.1 reads the same port in
``DoHNameserver.answer_port()`` and lets that error escape
``Resolver.resolve()``. So the port is read here, and a bad one is
reported with the entry's credentials masked and the original error
suppressed.
Args:
entry (str): The configured nameserver
parts (SplitResult): ``urlsplit(entry)``
kind (str): The transport name for the message, e.g.
``"DNS over TLS"``
Returns:
int | None: The port, or ``None`` when the URL has none
Raises:
ValueError: The port is not an integer from 0 to 65535
"""
try:
return parts.port
except ValueError:
pass
raise ValueError(
f"Invalid {kind} nameserver {_redact_one_url_credentials(entry)}: the "
"port is not an integer from 0 to 65535 (percent-encode any /, ? or # "
"in a password)"
)
def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
"""
Parses a ``tls://ip[:port][#hostname]`` nameserver entry.
@@ -327,26 +367,25 @@ def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
is not a literal IP address, or extra URL components
"""
parts = urlsplit(entry)
try:
port = parts.port
except ValueError as e:
# urlsplit only validates the port when it is accessed
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: {e}") from e
port = _nameserver_port(entry, parts, "DNS over TLS")
# The entry may carry userinfo, which is rejected below, so the
# messages quote it with its credentials masked
shown = _redact_one_url_credentials(entry)
if parts.username or parts.path or parts.query:
# Catch tls://9.9.9.9/dns.quad9.net — a plausible slash-for-#
# typo that would otherwise "work" with no certificate identity
# and fail only at query time with an opaque TLS error
raise ValueError(
f"Invalid DNS over TLS nameserver {entry}: only "
f"Invalid DNS over TLS nameserver {shown}: only "
"tls://ip[:port][#hostname] is supported — the TLS certificate "
"identity is given after #, not /"
)
address = parts.hostname
if not address:
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: missing IP address")
raise ValueError(f"Invalid DNS over TLS nameserver {shown}: missing IP address")
if not dns.inet.is_address(address):
raise ValueError(
f"Invalid DNS over TLS nameserver {entry}: {address} is not an IP "
f"Invalid DNS over TLS nameserver {shown}: {address} is not an IP "
"address. Use tls://ip[:port][#hostname], where the optional "
"#hostname is the TLS certificate identity of the server"
)
@@ -378,20 +417,25 @@ def _nameservers_to_resolver_input(
in the configured order
Raises:
ValueError: A ``tls://`` entry is malformed
ValueError: A ``tls://`` entry is malformed, or an ``https://``
entry has an unusable port
"""
resolver_input: list[str | dns.nameserver.Nameserver] = []
for entry in nameservers:
try:
# urlsplit lowercases the scheme, so HTTPS:// and TLS:// work too
scheme = urlsplit(entry).scheme
parts = urlsplit(entry)
except ValueError:
# e.g. an unbalanced IPv6 bracket; let dnspython reject it with
# its own message about what a nameserver may be
scheme = ""
if scheme == "https":
resolver_input.append(entry)
continue
if parts.scheme == "https":
# Validated now because dnspython reads the port only after a
# query fails, and its error then quotes the rejected text
_nameserver_port(entry, parts, "DNS over HTTPS")
resolver_input.append(_SessionDoHNameserver(entry))
elif scheme == "tls":
elif parts.scheme == "tls":
resolver_input.append(_parse_dot_nameserver(entry))
else:
resolver_input.append(entry)
@@ -439,6 +483,10 @@ def query_dns(
Returns:
list: A list of answers
Raises:
ValueError: A nameserver is malformed. The message masks the
nameserver's URL credentials.
"""
domain = str(domain).lower()
record_type = record_type.upper()
@@ -457,7 +505,17 @@ def query_dns(
"2606:4700:4700::1111",
"2606:4700:4700::1001",
]
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
rejection = None
try:
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
except ValueError as e:
# dnspython's own rejection quotes the entry as configured
# ("nameserver https://user:pw@[x is not ..."), credentials and all
rejection = _mask_nameserver_secrets(str(e), nameservers)
if rejection is not None:
# Raised outside the except block so that the unmasked error is not
# kept as __context__
raise ValueError(rejection)
# Cap per-query UDP timeout at 1s so dnspython retries within the
# lifetime window on transient packet loss — otherwise with a single
# nameserver and timeout == lifetime, one dropped UDP datagram consumes
@@ -532,7 +590,9 @@ def get_reverse_dns(
)[0]
except dns.exception.DNSException as e:
logger.debug(f"get_reverse_dns({ip_address}) exception: {e}")
# LifetimeTimeout and NoNameservers quote each nameserver URL
message = _mask_nameserver_secrets(str(e), nameservers)
logger.debug(f"get_reverse_dns({ip_address}) exception: {message}")
return hostname
@@ -684,7 +744,9 @@ def load_ip_db(
if not (offline or always_use_local_file):
try:
logger.debug(f"Trying to fetch IP database from {url}...")
logger.debug(
f"Trying to fetch IP database from {redact_url_credentials(url)}..."
)
headers = {"User-Agent": USER_AGENT}
response = httpx.get(
url, headers=headers, timeout=60, follow_redirects=True
@@ -698,8 +760,8 @@ def load_ip_db(
_IP_DB_PATH = cached_path
logger.info("IP database updated successfully")
return
except httpx.HTTPError as e:
logger.warning(f"Failed to fetch IP database: {e}")
except (httpx.HTTPError, httpx.InvalidURL) as e:
logger.warning(f"Failed to fetch IP database: {_describe_http_error(e)}")
except Exception as e:
logger.warning(f"Failed to save IP database: {e}")
@@ -1080,7 +1142,9 @@ def load_reverse_dns_map(
if not (offline or always_use_local_file):
try:
logger.debug(f"Trying to fetch reverse DNS map from {url}...")
logger.debug(
f"Trying to fetch reverse DNS map from {redact_url_credentials(url)}..."
)
headers = {"User-Agent": USER_AGENT}
response = httpx.get(
url, headers=headers, timeout=60, follow_redirects=True
@@ -1089,8 +1153,10 @@ def load_reverse_dns_map(
csv_file.write(response.text)
csv_file.seek(0)
load_csv(csv_file)
except httpx.HTTPError as e:
logger.warning(f"Failed to fetch reverse DNS map: {e}")
except (httpx.HTTPError, httpx.InvalidURL) as e:
logger.warning(
f"Failed to fetch reverse DNS map: {_describe_http_error(e)}"
)
except Exception:
logger.warning("Not a valid CSV file")
csv_file.seek(0)
@@ -1357,6 +1423,209 @@ def sanitize_for_log(value: str) -> str:
)
def redact_url_credentials(value: str | list[str]) -> str | list[str]:
"""Masks the credentials in a URL or host string so it is safe to log.
A configured URL such as ``https://user:secret@host:9200`` carries a
password; writing it to the log (CWE-532) leaks it to the console and
to ``log_file``. When the userinfo contains a ``:``, only the part
after the first ``:`` is replaced (``https://user:***@host:9200``);
when it has no ``:`` the whole userinfo is replaced
(``https://***@host``), since a bare token is itself the secret.
The authority is located with string methods only, so scheme-less
values such as ``user:secret@localhost:9200`` (which ``urlsplit()``
would misparse as scheme ``user``) are handled too.
A password containing an unencoded ``/``, ``?``, or ``#`` is not
allowed by RFC 3986 (the authority ends at that character, so clients
misparse such a URL too), and it leaves the authority with no ``@``.
When the authority has no ``@`` but one appears later in the value,
everything from the start of the authority through the *last* ``@``
is replaced (``https://admin:p4ss/w0rd@host:3100`` becomes
``https://***@host:3100``). The last ``@`` is used because the
password may itself contain ``@``. This over-masks a URL that has no
userinfo but has an ``@`` in its path or query:
``https://host/path@file`` becomes ``https://***@file``, and
``https://host/?redirect=https://u:p@evil`` becomes
``https://***@evil``. That is acceptable because the result is only
for the log. When the authority does contain an ``@``, only its
userinfo is masked, so credentials in a later query-string value
(``https://a:b@host/?redirect=https://u:p@evil``) are not detected,
and neither is the rest of a password containing both ``@`` and one
of ``/?#`` (``https://user:p@ss/word@host`` keeps ``ss/word``).
The value must be a single URL (or host) per string: split a
comma-separated list first, or pass a ``list``. The authority is the
first one found, so later entries in a comma-joined string are not
masked.
Never raises; input that does not contain userinfo, and input that is
not a ``str`` or ``list``, is returned unchanged.
Args:
value (str | list[str]): A URL or host string, or a list of them
Returns:
str | list[str]: ``value`` with credentials masked. A ``list`` is
returned as a new list; the input list is not modified.
"""
if isinstance(value, list):
return [
_redact_one_url_credentials(item) if isinstance(item, str) else item
for item in value
]
if not isinstance(value, str):
return value
return _redact_one_url_credentials(value)
def _find_url_userinfo(value: str) -> tuple[int, int, bool] | None:
"""Locates the userinfo of one URL or host string.
This is the lookup :func:`redact_url_credentials` masks and
:func:`_url_secrets` extracts the secret from, shared so that the two
cannot disagree about where the credentials are. It uses string
methods only, so it is linear in the length of ``value`` and never
raises.
Args:
value (str): A URL or host string
Returns:
tuple[int, int, bool] | None: ``None`` when no ``@`` follows the
start of the authority. Otherwise ``(start, end, in_authority)``,
where ``value[start:end]`` is the userinfo and ``value[end]`` is the
``@`` after it. ``in_authority`` is ``True`` when that ``@`` is the
last one inside the authority, and ``False`` when the authority has
no ``@`` (a ``/``, ``?`` or ``#`` in the password ends it early, as
in ``https://admin:p4ss/w0rd@host``) and the ``@`` is the last one
in the whole value; the password may itself contain ``@``.
"""
# A "://" counts as the scheme separator only when nothing before it
# could end an authority: in "user:secret@host/x?r=http://y" the "://"
# is inside the query, and the authority starts at offset 0.
scheme_end = value.find("://")
if scheme_end != -1 and not any(c in value[:scheme_end] for c in "/?#@"):
start = scheme_end + 3
elif value.startswith("//"):
start = 2
else:
start = 0
end = len(value)
for delimiter in "/?#":
position = value.find(delimiter, start)
if position != -1:
end = min(end, position)
at = value.rfind("@", start, end)
if at != -1:
return start, at, True
at = value.rfind("@", start)
if at != -1:
return start, at, False
return None
def _redact_one_url_credentials(value: str) -> str:
"""Masks the userinfo of one URL or host string; see
:func:`redact_url_credentials`."""
found = _find_url_userinfo(value)
if found is None:
return value
start, at, in_authority = found
if not in_authority:
# Which "/", "?" or "#" ended the authority is unknown, so mask
# everything through the "@" rather than guess where the user
# name stops.
return f"{value[:start]}***{value[at:]}"
user, colon, _ = value[start:at].partition(":")
masked = f"{user}:***" if colon else "***"
return f"{value[:start]}{masked}{value[at:]}"
def _url_secrets(value: str) -> list[str]:
"""Returns the secret text in one URL or host string's userinfo.
The secret is the userinfo after its first ``:``, or the whole userinfo
when it has no ``:`` (a bare token is itself the secret), located by
:func:`_find_url_userinfo`. When the secret contains an unencoded
``/``, ``?`` or ``#``, its text before the first of them is returned
too: ``urlsplit()`` (and so dnspython) and httpx read that part as the
port and quote it in their errors (``Port could not be cast to integer
value as 'p4ss'``). Empty strings are never returned, so the result is
safe to use as ``str.replace()`` targets. Never raises.
Args:
value (str): A URL or host string
Returns:
list[str]: The secret and, if it differs and is not empty, its
prefix before the first ``/``, ``?`` or ``#``, longest first; an
empty list when ``value`` has no userinfo or an empty secret.
"""
found = _find_url_userinfo(value)
if found is None:
return []
start, at, _ = found
user, colon, password = value[start:at].partition(":")
secret = password if colon else user
prefix_end = len(secret)
for delimiter in "/?#":
position = secret.find(delimiter)
if position != -1:
prefix_end = min(prefix_end, position)
return [text for text in dict.fromkeys((secret, secret[:prefix_end])) if text]
def _mask_nameserver_secrets(text: str, nameservers: list[str] | None) -> str:
"""Masks the credentials of the configured nameserver URLs in free text.
A DNS query error (dnspython's ``LifetimeTimeout``, a DNS over HTTPS
status error) quotes each failing nameserver URL, sometimes several
times in different forms, and ``redact_url_credentials()`` masks only
one URL per string. So each configured entry is replaced by its masked
form, and each secret :func:`_url_secrets` finds in it (the password,
and its start when a ``/``, ``?`` or ``#`` in it gets read as the port)
is replaced by ``***`` wherever it still appears. A short secret masks
every occurrence of that text, including ones that are not the
password; the result is only for error messages and logs. Never
raises.
Args:
text (str): The text to mask, typically ``str()`` of an exception
nameservers (list[str] | None): The configured nameservers
Returns:
str: ``text`` with the nameservers' credentials masked
"""
for entry in nameservers or []:
text = text.replace(entry, _redact_one_url_credentials(entry))
for secret in _url_secrets(entry):
text = text.replace(secret, "***")
return text
def _describe_http_error(e: httpx.HTTPError | httpx.InvalidURL) -> str:
"""Describes an httpx error for the log without leaking URL credentials.
``str(HTTPStatusError)`` embeds the server's reason phrase before the
URL, so a phrase containing ``/``, ``?``, ``#`` or ``@`` defeats
string-based redaction of the message. A status error is therefore
rebuilt from its structured parts. Transport errors (httpx 0.28) do not
embed the request URL, so their message is used as is. ``InvalidURL``
(not an ``HTTPError``) is raised while httpx parses the URL, and its
message quotes the component it rejected -- part of a password
containing ``/``, ``?`` or ``#`` ends up as the "port" -- so it is
replaced by a fixed message.
"""
if isinstance(e, httpx.InvalidURL):
return "invalid URL (details withheld because they may include credentials)"
if isinstance(e, httpx.HTTPStatusError):
url = redact_url_credentials(str(e.request.url))
return f"HTTP {e.response.status_code} for url '{url}'"
return f"{type(e).__name__}: {e}"
def get_filename_safe_string(string: str | None) -> str:
"""
Converts a string to a string that is safe to use as a filename
+6
View File
@@ -62,6 +62,12 @@ class WebhookClient(object):
self.session.post(webhook_url, data=payload, timeout=self.timeout)
else:
self.session.post(webhook_url, content=payload, timeout=self.timeout)
except httpx.InvalidURL:
# httpx quotes part of the URL, which can be part of a password
logger.error(
"Webhook Error: invalid webhook URL (details withheld because "
"they may include credentials)"
)
except Exception as error_:
logger.error(f"Webhook Error: {error_.__str__()}")
+413
View File
@@ -6,6 +6,7 @@ import importlib.abc
import importlib.machinery
import gzip
import io
import importlib.util
import json
import logging
import mailbox
@@ -15,6 +16,7 @@ import signal
import stat
import sys
import tempfile
import traceback
import unittest
import zipfile
from collections.abc import Iterator, Sequence
@@ -5452,6 +5454,323 @@ def _es_request_without_network(client, method, path, **kwargs):
return {"count": 0}
def _os_request_without_network(transport, method, url, *args, **kwargs):
"""Stand-in for ``opensearchpy.Transport.perform_request``: answers
"no such index" / zero documents like ``_es_request_without_network``."""
if method == "HEAD":
return {}
return {"count": 0}
class TestInitOutputClientsDoesNotLogCredentials(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: with --debug, the "Initializing ... client"
lines must mask credentials embedded in configured URLs."""
SECRET = "s3cretPW"
def _init(self, opts):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
clients = parsedmarc.cli._init_output_clients(opts)
self.addCleanup(parsedmarc.cli._close_output_clients, clients)
return "\n".join(logs.output)
def test_hec_url_password_is_masked(self):
opts = _output_client_opts(
hec=f"https://hecuser:{self.SECRET}@hec.example:8088",
hec_token="tok",
hec_index="email",
)
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
self.assertIn("url=https://hecuser:***@hec.example:8088", output)
def test_hec_malformed_url_password_is_masked(self):
"""A "/", "?" or "#" in the password ends the authority before the
"@", which left the --debug line unmasked; it is now masked through
the last "@"."""
for password in ("p4ssw0rd/extra", "p4ss?w0rd", "p4ss#w0rd"):
with self.subTest(password=password):
opts = _output_client_opts(
hec=f"https://hecuser:{password}@hec.example:8088",
hec_token="tok",
hec_index="email",
)
output = self._init(opts)
self.assertNotIn("p4ss", output)
self.assertIn("url=https://***@hec.example:8088", output)
def test_loki_url_password_is_masked(self):
opts = _output_client_opts(
loki_url=f"https://lokiuser:{self.SECRET}@loki.example:3100",
loki_timestamp="ingest",
loki_timeout=1.0,
loki_batch_size=100,
)
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
self.assertIn("url=https://lokiuser:***@loki.example:3100", output)
def test_loki_malformed_url_error_has_no_password(self):
"""A password with an unencoded "/" ends the URL's authority early
(RFC 3986 section 3.2), so httpx raises ``InvalidURL("Invalid port:
'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py), whose text and chained
ValueError both repeat it. Neither the startup error nor the
``--debug`` "Initializing Loki client" line may show any part of
the password."""
malformed = (
("https://admin:p4ssw0rd/extra@loki.example:3100", "p4ssw0rd"),
("https://admin:p4ss?w0rd@loki.example:3100", "p4ss"),
("https://admin:p4ss#w0rd@loki.example:3100", "p4ss"),
)
for url, secret in malformed:
with self.subTest(url=url):
opts = _output_client_opts(
loki_url=url,
loki_timestamp="ingest",
loki_timeout=1.0,
loki_batch_size=100,
)
with (
self.assertRaises(RuntimeError) as ctx,
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
):
parsedmarc.cli._init_output_clients(opts)
self.assertEqual(
str(ctx.exception),
"Loki: Invalid Loki URL (details withheld because they may "
"include credentials)",
)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
output = "\n".join(logs.output)
self.assertNotIn(secret, output)
self.assertIn("url=https://***@loki.example:3100", output)
def test_elasticsearch_hosts_password_is_masked(self):
connections = parsedmarc.elastic.connections
_isolate_default_alias(self, connections)
opts = _output_client_opts(
save_aggregate=True,
elasticsearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
elasticsearch_timeout=1.0,
)
with (
patch.object(Elasticsearch, "perform_request", _es_request_without_network),
patch(
"parsedmarc.elastic.connections.create_connection",
wraps=connections.create_connection,
) as mock_conn,
):
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
self.assertIn("Initializing Elasticsearch client", output)
# Redaction is log-only: the client still gets the real credentials
self.assertEqual(
mock_conn.call_args.kwargs["hosts"],
[f"https://u:{self.SECRET}@127.0.0.1:9299"],
)
self.assertIn("https://u:***@127.0.0.1:9299", output)
def test_opensearch_hosts_password_is_masked_in_cli_and_set_hosts(self):
connections = opensearch_module.connections
_isolate_default_alias(self, connections)
opts = _output_client_opts(
save_aggregate=True,
opensearch_hosts=[f"https://u:{self.SECRET}@127.0.0.1:9299"],
opensearch_timeout=1.0,
)
with patch.object(
opensearchpy.Transport, "perform_request", _os_request_without_network
):
output = self._init(opts)
self.assertNotIn(self.SECRET, output)
lines = output.splitlines()
for marker in ("Initializing OpenSearch client", "Connecting to OpenSearch"):
with self.subTest(marker=marker):
matching = [line for line in lines if marker in line]
self.assertEqual(len(matching), 1)
self.assertIn("https://u:***@127.0.0.1:9299", matching[0])
class TestHttpxLogRedaction(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: httpx logs the full request URL at INFO."""
def setUp(self):
saved = {}
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
dep = logging.getLogger(name)
saved[name] = (
dep.level,
list(dep.handlers),
dep.propagate,
list(dep.filters),
)
def restore():
for name, (level, handlers, propagate, filters) in saved.items():
dep = logging.getLogger(name)
dep.setLevel(level)
dep.handlers = handlers
dep.propagate = propagate
dep.filters = filters
self.addCleanup(restore)
parsedmarc.cli._configure_dependency_logging(logging.INFO)
def _client(self, seen):
def handler(request):
seen.append(request)
return httpx.Response(200)
return httpx.Client(transport=httpx.MockTransport(handler))
def test_userinfo_password_masked_but_auth_still_sent(self):
seen = []
with self._client(seen) as client:
with self.assertLogs("httpx", level="INFO") as logs:
client.post("https://user:secret@hook.example/x", content=b"{}")
output = "\n".join(logs.output)
self.assertNotIn("secret", output)
self.assertIn("https://user:***@hook.example/x", output)
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
def test_ipinfo_token_query_parameter_masked(self):
seen = []
with self._client(seen) as client:
with self.assertLogs("httpx", level="INFO") as logs:
client.get(
"https://api.ipinfo.io/lite/8.8.8.8", params={"token": "TOKSECRET"}
)
output = "\n".join(logs.output)
self.assertNotIn("TOKSECRET", output)
self.assertIn("api.ipinfo.io/lite/8.8.8.8", output)
self.assertIn("token=***", output)
self.assertEqual(seen[0].url.params["token"], "TOKSECRET")
def test_token_mask_leaves_literal_encoded_stars_in_path_alone(self):
"""A path containing a literal ``%2A%2A%2A`` is not rewritten; only
the token value is masked, with and without other query params and
a fragment."""
for url, expected in (
(
"https://h.example/a%2A%2A%2Ab?token=TOKSECRET",
"https://h.example/a%2A%2A%2Ab?token=***",
),
(
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET",
"https://h.example/a%2A%2A%2Ab?x=1&token=***",
),
(
"https://h.example/a%2A%2A%2Ab?x=1&token=TOKSECRET#f",
"https://h.example/a%2A%2A%2Ab?x=1&token=***#f",
),
):
with self.subTest(url=url):
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(
httpx.URL(url)
)
self.assertNotIn("TOKSECRET", result)
self.assertEqual(result, expected)
def test_reconfiguring_does_not_stack_filters(self):
parsedmarc.cli._configure_dependency_logging(logging.INFO)
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
installed = [
f
for f in logging.getLogger("httpx").filters
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
]
self.assertEqual(len(installed), 1)
def test_non_tuple_args_and_other_arg_types_pass_through(self):
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "x %d", (5,), None)
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
self.assertEqual(record.args, (5,))
record = logging.LogRecord("httpx", logging.INFO, "f", 1, "plain", None, None)
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
self.assertIsNone(record.args)
@unittest.skipUnless(
importlib.util.find_spec("httpx2") is not None,
"httpx2 (dnspython 2.9+ DNS over HTTPS) is not installed",
)
class TestHttpx2LogRedaction(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: dnspython 2.9+ does DNS over HTTPS through
httpx2, which logs the request URL under the ``httpx2`` logger with an
``httpx2.URL``, a different class from ``httpx.URL``."""
def setUp(self):
import httpx2 # pyright: ignore[reportMissingImports]
self.httpx2 = httpx2
saved = {}
for name in parsedmarc.cli._DEPENDENCY_LOGGERS:
dep = logging.getLogger(name)
saved[name] = (
dep.level,
list(dep.handlers),
dep.propagate,
list(dep.filters),
)
def restore():
for name, (level, handlers, propagate, filters) in saved.items():
dep = logging.getLogger(name)
dep.setLevel(level)
dep.handlers = handlers
dep.propagate = propagate
dep.filters = filters
self.addCleanup(restore)
parsedmarc.cli._configure_dependency_logging(logging.INFO)
def test_httpx2_client_request_line_masks_userinfo(self):
"""httpx2 itself emits the line; the credential still reaches the
transport."""
seen = []
def handler(request):
seen.append(request)
return self.httpx2.Response(200)
with self.httpx2.Client(transport=self.httpx2.MockTransport(handler)) as client:
with self.assertLogs("httpx2", level="INFO") as logs:
client.post("https://doh:S3CRET@doh.example/dns-query", content=b"{}")
output = "\n".join(logs.output)
self.assertNotIn("S3CRET", output)
self.assertIn("https://doh:***@doh.example/dns-query", output)
self.assertTrue(seen[0].headers["Authorization"].startswith("Basic "))
def test_httpx2_url_token_query_parameter_masked(self):
url = self.httpx2.URL("https://h.example/a?x=1&token=TOKSECRET#f")
result = parsedmarc.cli._RedactUrlCredentialsFilter._redact(url)
self.assertEqual(result, "https://h.example/a?x=1&token=***#f")
def test_httpx2_logger_record_args_masked_and_ints_untouched(self):
record = logging.LogRecord(
"httpx2",
logging.INFO,
"f",
1,
"%s %d",
(self.httpx2.URL("https://u:S3CRET@h.example/x"), 200),
None,
)
self.assertTrue(parsedmarc.cli._RedactUrlCredentialsFilter().filter(record))
self.assertEqual(record.args, ("https://u:***@h.example/x", 200))
def test_reconfiguring_attaches_one_filter_to_httpx2(self):
parsedmarc.cli._configure_dependency_logging(logging.DEBUG)
parsedmarc.cli._configure_dependency_logging(logging.INFO)
installed = [
f
for f in logging.getLogger("httpx2").filters
if isinstance(f, parsedmarc.cli._RedactUrlCredentialsFilter)
]
self.assertEqual(len(installed), 1)
class TestInitOutputClientsRollback(unittest.TestCase):
"""_init_output_clients() is all-or-nothing against the search backends'
process-wide ``default`` connection alias.
@@ -7636,6 +7955,100 @@ class TestParseConfigGeneral(unittest.TestCase):
str(ctx.exception),
)
def test_general_nameservers_pre_flight_errors_mask_url_credentials(self):
"""GHSA-hjr3-rx4c-24cv: the pre-flight error is logged at the default
level, so neither the "no PTR record ... from <nameservers>" list nor
a transport error that quotes the DoH request URL may carry the
nameserver URL's password, and the chained exception (printed on a
crash) must not repeat it."""
from parsedmarc.cli import ConfigurationError, _parse_config
url = "https://doh:S3CRETPW@dns.example/dns-query"
def nxdomain(request, *args, **kwargs):
response = dns.message.make_response(request)
response.set_rcode(dns.rcode.NXDOMAIN)
return response
def transport_error(request, *args, **kwargs):
raise RuntimeError(f"Server error '502' for url '{url}'")
for name, responder, expected in (
("NXDOMAIN", nxdomain, "https://doh:***@dns.example/dns-query"),
("transport", transport_error, "https://doh:***@dns.example/dns-query"),
):
with self.subTest(name):
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "2.0",
"nameservers": url,
},
)
with patch("dns.query.https", side_effect=responder):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn("S3CRETPW", str(ctx.exception))
self.assertNotIn("S3CRETPW", trace)
self.assertIn(expected, str(ctx.exception))
def test_general_nameservers_pre_flight_masks_malformed_url_passwords(self):
"""GHSA-hjr3-rx4c-24cv: a password containing an unencoded ``/``,
``?`` or ``#`` ends the URL authority early (RFC 3986 section 3.2),
so ``urlsplit()`` reports no password and reads its start as the
port. dnspython 2.9.1 then raised ``ValueError("Port could not be
cast to integer value as 'p4ss'")`` from
``DoHNameserver.answer_port()`` (``dns/nameserver.py``), and
``_parse_dot_nameserver()`` quoted the same urlsplit message for a
``tls://`` entry, so the pre-flight error printed the start of the
password. Neither the password nor that prefix may reach the
message or the printed traceback. No query is sent: every entry is
rejected before dnspython contacts a server, which the patched
transports would turn into a different error."""
from parsedmarc.cli import ConfigurationError, _parse_config
for url, expected in (
(
"https://doh:p4ss/w0rd@dns.example/dns-query",
"https://***@dns.example/dns-query",
),
(
"https://doh:p4ss?w0rd@dns.example/dns-query",
"https://***@dns.example/dns-query",
),
(
"https://doh:p4ss#w0rd@dns.example/dns-query",
"https://***@dns.example/dns-query",
),
("tls://dot:p4ss/w0rd@1.1.1.1", "tls://***@1.1.1.1"),
):
with self.subTest(url=url):
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "0.2",
"nameservers": url,
},
)
unreachable = OSError("no query may be sent")
with (
patch("dns.query.https", side_effect=unreachable),
patch("dns.query.tls", side_effect=unreachable),
):
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, _opts())
message = str(ctx.exception)
trace = "".join(traceback.format_exception(ctx.exception))
for secret in ("p4ss", "w0rd"):
self.assertNotIn(secret, message)
self.assertNotIn(secret, trace)
self.assertIn(expected, message)
self.assertIn("DNS pre-flight check failed: ValueError: ", message)
self.assertNotIn("no query may be sent", message)
def test_general_nameservers_pre_flight_rejects_invalid_test_address(self):
"""An unparseable dns_test_address is reported by setting name and
value (dns.reversename.from_address raises dns.exception.SyntaxError,
+61
View File
@@ -3,6 +3,7 @@
import json
import os
import time
import traceback
import unittest
from typing import cast
from unittest.mock import MagicMock, patch
@@ -164,6 +165,44 @@ class TestLokiClientInit(unittest.TestCase):
self.assertIsInstance(client.session.auth, httpx.BasicAuth)
class TestLokiInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` (httpx 0.28.1 _urlparse.py:
``raise InvalidURL(f"Invalid port: {port!r}")``). Its implicit
``__context__`` is a ValueError that repeats that part of the password
too."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@loki.example.org:3100", "p4ssw0rd"),
("https://admin:p4ss?w0rd@loki.example.org:3100", "p4ss"),
("https://admin:p4ss#w0rd@loki.example.org:3100", "p4ss"),
)
def test_invalid_url_raises_fixed_message_without_password(self):
for url, secret in self.MALFORMED:
with self.subTest(url=url):
with self.assertRaises(ValueError) as ctx:
LokiClient(url)
self.assertEqual(
str(ctx.exception),
"Invalid Loki URL (details withheld because they may "
"include credentials)",
)
self.assertIsNone(ctx.exception.__cause__)
self.assertTrue(ctx.exception.__suppress_context__)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
def test_valid_url_keeps_userinfo_for_requests(self):
client = LokiClient("https://admin:p4ssw0rd@loki.example.org:3100")
self.assertEqual(
client.url,
"https://admin:p4ssw0rd@loki.example.org:3100/loki/api/v1/push",
)
@patch(
"parsedmarc.loki.parsed_aggregate_reports_to_csv_rows", return_value=AGGREGATE_ROWS
)
@@ -370,6 +409,28 @@ class TestLokiErrors(unittest.TestCase):
client.save_aggregate_report_to_loki(AGGREGATE)
self.assertIn("refused", str(ctx.exception))
def test_errors_mask_url_password(self, _rows):
"""GHSA-hjr3-rx4c-24cv: the URL keeps its userinfo, and the CLI
logs LokiError text, so the password must be masked in it."""
secret = "s3cretPW"
client = LokiClient(f"http://u:{secret}@loki.example.org:3100")
client.session = MagicMock()
_session(client).post.return_value = _ok_response(500)
with self.assertRaises(LokiError) as status_ctx:
client.save_aggregate_report_to_loki(AGGREGATE)
_session(client).post.side_effect = httpx.ConnectError("refused")
with self.assertRaises(LokiError) as network_ctx:
client.save_aggregate_report_to_loki(AGGREGATE)
for ctx in (status_ctx, network_ctx):
with self.subTest(error=str(ctx.exception)):
self.assertNotIn(secret, str(ctx.exception))
self.assertIn("http://u:***@loki.example.org:3100", str(ctx.exception))
# Redaction is message-only: the request still goes to the real URL
self.assertEqual(
_session(client).post.call_args.args[0],
f"http://u:{secret}@loki.example.org:3100/loki/api/v1/push",
)
def test_blocked_ingestion_status_raises(self, _rows):
"""Loki returns 260 (block_ingestion_status_code default) and drops
the logs when ingestion is blocked; that must not read as success."""
+24
View File
@@ -241,6 +241,30 @@ class TestSetHosts(unittest.TestCase):
kwargs = mock_conn.call_args.kwargs
self.assertEqual(kwargs["hosts"], ["es1:9200", "es2:9200"])
def test_debug_log_masks_credentials_but_connects_with_them(self):
"""GHSA-hjr3-rx4c-24cv: the "Connecting to OpenSearch" debug line
must not carry the URL password, while the client still gets the
real URL."""
url = "https://admin:s3cretPW@es.example:9200"
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
set_hosts(url)
output = "\n".join(logs.output)
self.assertNotIn("s3cretPW", output)
self.assertIn("https://admin:***@es.example:9200", output)
self.assertEqual(mock_conn.call_args.kwargs["hosts"], [url])
def test_debug_log_masks_schemeless_host_list(self):
hosts = ["admin:s3cretPW@es1:9200", "es2:9200"]
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
set_hosts(hosts)
output = "\n".join(logs.output)
self.assertNotIn("s3cretPW", output)
self.assertIn("admin:***@es1:9200", output)
self.assertIn("es2:9200", output)
self.assertEqual(mock_conn.call_args.kwargs["hosts"], hosts)
def test_timeout_default_60s(self):
with patch("parsedmarc.opensearch.connections.create_connection") as mock_conn:
set_hosts("es:9200")
+63
View File
@@ -311,11 +311,74 @@ class TestPostgreSQLConstruction(unittest.TestCase):
"""A driver-level connection error is wrapped in PostgreSQLError."""
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
mock_psycopg.Error = Exception
mock_psycopg.ProgrammingError = type("ProgrammingError", (Exception,), {})
mock_psycopg.connect.side_effect = mock_psycopg.Error("refused")
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(host="localhost")
self.assertIn("refused", str(ctx.exception))
def test_connect_failure_detail_is_url_redacted(self):
"""A non-parse driver error keeps its detail, with URL credentials
masked."""
programming_error = type("ProgrammingError", (Exception,), {})
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
mock_psycopg.Error = Exception
mock_psycopg.ProgrammingError = programming_error
mock_psycopg.connect.side_effect = mock_psycopg.Error(
'connection to "postgresql://u:PGSECRET@db.example/x" failed'
)
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(connection_string="postgresql://u:PGSECRET@h/db")
message = str(ctx.exception)
self.assertNotIn("PGSECRET", message)
self.assertIn("u:***@db.example/x", message)
def test_malformed_connection_string_detail_withheld(self):
"""A ProgrammingError from parsing the connection string (which libpq
words with the whole string or a password fragment) is replaced with
a fixed message that still identifies the problem."""
programming_error = type("ProgrammingError", (Exception,), {})
for dsn, driver_message in (
(
"postgresql://u:PGSECRET@[::1/db",
'end of string reached when looking for matching "]" in IPv6 '
'host address in URI: "postgresql://u:PGSECRET@[::1/db"',
),
(
"postgresql://u:PG%ZZSECRET@h/db",
'invalid percent-encoded token: "PG%ZZSECRET"',
),
):
with self.subTest(dsn=dsn):
with patch("parsedmarc.postgres.psycopg") as mock_psycopg:
mock_psycopg.Error = Exception
mock_psycopg.ProgrammingError = programming_error
mock_psycopg.connect.side_effect = programming_error(driver_message)
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(connection_string=dsn)
message = str(ctx.exception)
self.assertNotIn("PGSECRET", message)
self.assertNotIn("SECRET", message)
self.assertIn("Invalid PostgreSQL connection_string", message)
@unittest.skipIf(
importlib.util.find_spec("psycopg") is None, "psycopg is not installed"
)
def test_real_psycopg_malformed_connection_string_withheld(self):
"""With the real driver, both malformed strings fail in parsing (no
network I/O) and the raised message carries no part of the password."""
for dsn in (
"postgresql://u:PGSECRET@[::1/db",
"postgresql://u:PG%ZZSECRET@h/db",
):
with self.subTest(dsn=dsn):
with self.assertRaises(PostgreSQLError) as ctx:
PostgreSQLClient(connection_string=dsn)
self.assertNotIn("SECRET", str(ctx.exception))
self.assertIn(
"Invalid PostgreSQL connection_string", str(ctx.exception)
)
def test_create_tables_executes_all_ddl(self):
"""create_tables issues CREATE TABLE for every table and the indexes."""
client, mock_conn = _make_client()
+53
View File
@@ -2,6 +2,7 @@
import json
import time
import traceback
import unittest
from unittest.mock import MagicMock, patch
@@ -469,6 +470,58 @@ class TestSaveSmtpTlsReportsToSplunk(unittest.TestCase):
)
class TestInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
any network I/O. The error reaches the log on every batch."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@splunk.example.com:8088", "p4ssw0rd"),
("https://admin:p4ss?w0rd@splunk.example.com:8088", "p4ss"),
("https://admin:p4ss#w0rd@splunk.example.com:8088", "p4ss"),
)
def test_invalid_url_raises_fixed_message_without_password(self):
saves = (
("aggregate", "save_aggregate_reports_to_splunk", _aggregate_report),
("failure", "save_failure_reports_to_splunk", _failure_report),
("smtp_tls", "save_smtp_tls_reports_to_splunk", _smtp_tls_report),
)
for url, secret in self.MALFORMED:
client = HECClient(url=url, access_token="tok", index="dmarc")
for name, method, make_report in saves:
with self.subTest(url=url, report=name):
with self.assertRaises(SplunkError) as ctx:
getattr(client, method)(make_report())
self.assertEqual(
str(ctx.exception),
"Invalid Splunk HEC URL (details withheld because they "
"may include credentials)",
)
self.assertIsNone(ctx.exception.__cause__)
self.assertTrue(ctx.exception.__suppress_context__)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
def test_valid_url_is_the_one_posted(self):
client = HECClient(
url="https://admin:p4ssw0rd@splunk.example.com:8088",
access_token="tok",
index="dmarc",
)
client.session = MagicMock()
client.session.post.return_value = _ok_response()
client.save_aggregate_reports_to_splunk(_aggregate_report())
self.assertEqual(
client.session.post.call_args.args[0],
"https://admin:p4ssw0rd@splunk.example.com:8088"
"/services/collector/event/1.0",
)
class TestHECClientClose(unittest.TestCase):
def test_close_closes_session(self):
client = _client()
+464 -6
View File
@@ -4,6 +4,7 @@ import os
import shutil
import tempfile
import time
import traceback
import unittest
from datetime import datetime, timezone
from importlib.resources import files
@@ -516,6 +517,122 @@ class Test(unittest.TestCase):
self.assertTrue(parsedmarc.utils.is_outlook_msg(magic))
class TestRedactUrlCredentials(unittest.TestCase):
"""Tests for parsedmarc.utils.redact_url_credentials()"""
def testMasksUserinfoAndKeepsHost(self):
cases = [
("https://user:secret@host", "https://user:***@host"),
("https://user:secret@host:9200/", "https://user:***@host:9200/"),
("localhost:9200", "localhost:9200"),
("127.0.0.1:9299", "127.0.0.1:9299"),
("user:secret@localhost:9200", "user:***@localhost:9200"),
# A "://" in the path or query of a scheme-less value is not
# the scheme separator; the userinfo at offset 0 is still masked
(
"user:secret@host:9200/x?r=http://y",
"user:***@host:9200/x?r=http://y",
),
("user:secret@host?r=a://b", "user:***@host?r=a://b"),
("https://user:p@ss@host/x", "https://user:***@host/x"),
("https://user:p:w@[::1]:9200/", "https://user:***@[::1]:9200/"),
("https://[::1]:9200", "https://[::1]:9200"),
("//user:pw@host", "//user:***@host"),
# Documented over-masking: with no "@" in the authority, an "@"
# later in the value is masked through, since it may end a
# password that contains "/", "?" or "#"
("http://host/path?e=a@b.com", "http://***@b.com"),
("https://host/path@file", "https://***@file"),
("https://host/?redirect=https://u:p@evil", "https://***@evil"),
# Documented non-goal: with userinfo in the authority, credentials
# inside a later query value are not masked
(
"https://a:b@host/?redirect=https://u:p@evil",
"https://a:***@host/?redirect=https://u:p@evil",
),
("https://tok@host", "https://***@host"),
("user@host:9200", "***@host:9200"),
("https://user:@host", "https://user:***@host"),
("https://u%40x:p%3Aw@h", "https://u%40x:***@h"),
("not a url @ all", "***@ all"),
("", ""),
("::::", "::::"),
]
for value, expected in cases:
with self.subTest(value=value):
self.assertEqual(
parsedmarc.utils.redact_url_credentials(value), expected
)
def testMasksPasswordContainingAuthorityDelimiter(self):
"""A password with an unencoded "/", "?" or "#" ends the authority
before the "@" (RFC 3986 section 3.2), which used to leave the value
unmasked; everything through the last "@" is masked instead."""
cases = [
("https://admin:p4ssw0rd/extra@host:3100", "https://***@host:3100"),
("https://admin:p4ss?w0rd@host", "https://***@host"),
("https://admin:p4ss#w0rd@host", "https://***@host"),
("admin:p4ss/w0rd@host:9200", "***@host:9200"),
("//admin:p4ss/w0rd@host", "//***@host"),
# The password may itself contain "@"; the last one is used
("https://admin:p4ss/w@rd@host:3100/x", "https://***@host:3100/x"),
# A "://" after the "/" is not the scheme separator
(
"admin:p4ss/w0rd@host/x?r=http://y",
"***@host/x?r=http://y",
),
]
for value, expected in cases:
with self.subTest(value=value):
result = parsedmarc.utils.redact_url_credentials(value)
self.assertEqual(result, expected)
self.assertNotIn("p4ss", result)
def testListMasksPasswordContainingAuthorityDelimiter(self):
self.assertEqual(
parsedmarc.utils.redact_url_credentials(
["https://admin:p4ss/w0rd@h1:9200", "https://h2:9200"]
),
["https://***@h1:9200", "https://h2:9200"],
)
def testListIsRedactedElementWiseIntoANewList(self):
original = ["a:b@h1", "https://h2"]
result = parsedmarc.utils.redact_url_credentials(original)
self.assertEqual(result, ["a:***@h1", "https://h2"])
self.assertIsNot(result, original)
self.assertEqual(original, ["a:b@h1", "https://h2"])
def testListKeepsNonStringItems(self):
mixed = cast(list[str], ["a:b@h1", None])
self.assertEqual(
parsedmarc.utils.redact_url_credentials(mixed), ["a:***@h1", None]
)
def testNonStringInputIsReturnedUnchanged(self):
for value in (None, 42):
with self.subTest(value=value):
self.assertEqual(
parsedmarc.utils.redact_url_credentials(cast(str, value)), value
)
def testLongInputsAreFast(self):
"""No regex is used, so adversarial input is linear. A 1 MB string
of repeated openers and a 200,000-character run of '@' both finish
in milliseconds."""
for value in (
"://" * 350_000,
"@" * 200_000,
"a:b@" * 250_000,
"a:b/" * 250_000 + "@",
"/@" * 500_000,
):
with self.subTest(prefix=value[:8]):
started = time.monotonic()
parsedmarc.utils.redact_url_credentials(value)
self.assertLess(time.monotonic() - started, 5)
class TestSanitizeForLog(unittest.TestCase):
"""Tests for parsedmarc.utils.sanitize_for_log()"""
@@ -541,6 +658,197 @@ class TestSanitizeForLog(unittest.TestCase):
self.assertNotIn("\n", result)
class TestDownloadUrlCredentialsNotLogged(unittest.TestCase):
"""A download URL with userinfo must not leak its password through the
"Trying to fetch" debug line or the warning that embeds the httpx
``HTTPStatusError`` message (which quotes the full URL).
The loaders run for real; only the network is replaced, with an
``httpx.MockTransport`` returning 401 so httpx itself builds the
exception text.
"""
URL = "https://dluser:DLSECRET@downloads.example.test/map.dat"
@staticmethod
def _transport_returning(handler):
def get(url, **kwargs):
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
return client.get(url, **kwargs)
return get
@classmethod
def _status_getter(cls, reason_phrase=None):
extensions = {}
if reason_phrase is not None:
extensions["reason_phrase"] = reason_phrase
return cls._transport_returning(
lambda request: httpx.Response(401, extensions=extensions)
)
@staticmethod
def _connect_error_getter(url, **kwargs):
def handler(request):
raise httpx.ConnectError("connection refused", request=request)
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
return client.get(url, **kwargs)
# Reason phrases containing URL delimiters used to defeat redaction of
# the HTTPStatusError text, which puts the phrase before the URL.
REASON_PHRASES = (
None,
b"Unauthorized/Expired",
b"Unauthorized?",
b"Gone#",
b"Not @ home",
)
def _assert_redacted(self, logs):
output = "\n".join(logs.output)
self.assertNotIn("DLSECRET", output)
self.assertTrue(
any(
line.startswith("DEBUG") and "downloads.example.test" in line
for line in logs.output
),
output,
)
warnings = [line for line in logs.output if line.startswith("WARNING")]
self.assertTrue(
any(
"HTTP 401" in line
and "'https://dluser:***@downloads.example.test/map.dat'" in line
for line in warnings
),
output,
)
def _run_psl(self, url=None):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
parsedmarc.utils.load_psl_overrides(url=url or self.URL)
return logs
def _run_ip_db(self, url=None):
tmp_dir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmp_dir, ignore_errors=True)
old_path = parsedmarc.utils._IP_DB_PATH
self.addCleanup(setattr, parsedmarc.utils, "_IP_DB_PATH", old_path)
parsedmarc.utils._IP_DB_PATH = None
with (
patch("parsedmarc.utils.tempfile.gettempdir", return_value=tmp_dir),
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
):
parsedmarc.utils.load_ip_db(url=url or self.URL)
return logs
def _run_reverse_dns_map(self, url=None):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
parsedmarc.utils.load_reverse_dns_map({}, url=url or self.URL)
return logs
def _check_loader(self, run):
for phrase in self.REASON_PHRASES:
with self.subTest(reason_phrase=phrase):
with patch("parsedmarc.utils.httpx.get", self._status_getter(phrase)):
self._assert_redacted(run())
def _check_transport_error(self, run):
with patch("parsedmarc.utils.httpx.get", self._connect_error_getter):
logs = run()
output = "\n".join(logs.output)
self.assertNotIn("DLSECRET", output)
self.assertTrue(
any(
line.startswith("WARNING")
and "ConnectError: connection refused" in line
for line in logs.output
),
output,
)
# The password holds an unencoded "/", "?" or "#", which RFC 3986 section
# 3.2 ends the authority at, so the text after the colon is read as the
# port. Real httpx raises InvalidURL("Invalid port: 'p4ssw0rd'") while
# parsing, before any network I/O (httpx 0.28.1 _urlparse.py:
# ``raise InvalidURL(f"Invalid port: {port!r}")``). InvalidURL is not an
# httpx.HTTPError, and its message quotes part of the password.
MALFORMED_URLS = (
("https://admin:p4ssw0rd/extra@downloads.example.test/map.dat", "p4ssw0rd"),
("https://admin:p4ss?w0rd@downloads.example.test/map.dat", "p4ss"),
("https://admin:p4ss#w0rd@downloads.example.test/map.dat", "p4ss"),
)
INVALID_URL_MESSAGE = (
"invalid URL (details withheld because they may include credentials)"
)
def _check_invalid_url(self, run, source):
for url, secret in self.MALFORMED_URLS:
with self.subTest(url=url):
requested = []
real_get = httpx.get
def recording_get(requested_url, **kwargs):
requested.append(requested_url)
if requested_url != url:
# e.g. the PSL overrides fetch that
# load_reverse_dns_map triggers; stay offline
raise httpx.ConnectError("offline")
return real_get(requested_url, **kwargs)
with patch("parsedmarc.utils.httpx.get", recording_get):
logs = run(url) # must return, not crash on InvalidURL
# The loader still requested the real URL, not a redacted one
self.assertIn(url, requested)
output = "\n".join(logs.output)
# Neither the warning nor the "Trying to fetch" debug line
# may show any part of the password
self.assertNotIn(secret, output)
self.assertIn(
f"DEBUG:parsedmarc.log:Trying to fetch {source} from "
"https://***@downloads.example.test/map.dat...",
logs.output,
)
self.assertIn(
f"WARNING:parsedmarc.log:Failed to fetch {source}: "
f"{self.INVALID_URL_MESSAGE}",
logs.output,
)
self.assertNotIn("Not a valid CSV file", output)
self.assertNotIn("Failed to save IP database", output)
def test_psl_overrides_loader_invalid_url(self):
"""A malformed ``psl_overrides_url`` used to crash startup with the
InvalidURL traceback; it now warns and falls back to the local file."""
self._check_invalid_url(self._run_psl, "PSL overrides")
self.assertIn(".linode.com", parsedmarc.utils.psl_overrides)
def test_ip_db_loader_invalid_url(self):
self._check_invalid_url(self._run_ip_db, "IP database")
def test_reverse_dns_map_loader_invalid_url(self):
self._check_invalid_url(self._run_reverse_dns_map, "reverse DNS map")
def test_psl_overrides_loader(self):
self._check_loader(self._run_psl)
def test_ip_db_loader(self):
self._check_loader(self._run_ip_db)
def test_reverse_dns_map_loader(self):
self._check_loader(self._run_reverse_dns_map)
def test_psl_overrides_loader_transport_error(self):
self._check_transport_error(self._run_psl)
def test_ip_db_loader_transport_error(self):
self._check_transport_error(self._run_ip_db)
def test_reverse_dns_map_loader_transport_error(self):
self._check_transport_error(self._run_reverse_dns_map)
class TestLoadPSLOverrides(unittest.TestCase):
"""Covers `parsedmarc.utils.load_psl_overrides`."""
@@ -1559,16 +1867,20 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
must fail at configuration time naming the entry, not parse
"successfully" with no TLS certificate identity and then fail at
query time with an opaque certificate error. Userinfo and query
components are rejected the same way."""
for entry in (
"tls://9.9.9.9/dns.quad9.net",
"tls://user@9.9.9.9",
"tls://9.9.9.9?hostname=dns.quad9.net",
components are rejected the same way, with the userinfo masked in
the message (GHSA-hjr3-rx4c-24cv)."""
for entry, shown in (
("tls://9.9.9.9/dns.quad9.net", "tls://9.9.9.9/dns.quad9.net"),
("tls://user@9.9.9.9", "tls://***@9.9.9.9"),
(
"tls://9.9.9.9?hostname=dns.quad9.net",
"tls://9.9.9.9?hostname=dns.quad9.net",
),
):
with self.subTest(entry=entry):
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils._nameservers_to_resolver_input([entry])
self.assertIn(entry, str(ctx.exception))
self.assertIn(shown, str(ctx.exception))
def test_unsplittable_entry_is_left_for_dnspython_to_reject(self):
"""urlsplit itself raises on some malformed input (an unbalanced
@@ -1712,6 +2024,152 @@ class TestEncryptedDnsNameservers(unittest.TestCase):
self.assertEqual(parsedmarc.utils._DOH_SESSION_PID, os.getpid())
class TestNameserverCredentialsNotLeaked(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: nameserver URL credentials must not reach an
exception message, its printed traceback, or the log."""
def test_url_secrets_finds_the_password_and_its_port_like_prefix(self):
"""The secret is the userinfo after its first ":" (the whole
userinfo when there is none). A "/", "?" or "#" in it ends the URL
authority (RFC 3986 section 3.2), so urlsplit() reads the text
before it as the port and quotes it in its error; that prefix is a
secret too."""
cases = [
("https://doh:p4ss@dns.example/q", ["p4ss"]),
("https://doh:p4ss/w0rd@dns.example/q", ["p4ss/w0rd", "p4ss"]),
("https://doh:p4ss?w0rd@dns.example/q", ["p4ss?w0rd", "p4ss"]),
("https://doh:p4ss#w0rd@dns.example/q", ["p4ss#w0rd", "p4ss"]),
# The prefix ends at the first of the three delimiters
("https://doh:ab#cd/ef@dns.example/q", ["ab#cd/ef", "ab"]),
("https://doh:p:w@dns.example/q", ["p:w"]),
("https://doh:p@ss@dns.example/q", ["p@ss"]),
("https://S3CRETTOKEN@dns.example/q", ["S3CRETTOKEN"]),
("tls://dot:p4ss/w0rd@1.1.1.1", ["p4ss/w0rd", "p4ss"]),
# Empty secrets are never returned: str.replace("", ...) would
# insert the mask between every character
("https://doh:@dns.example/q", []),
("https://doh:/x@dns.example/q", ["/x"]),
("https://dns.example/dns-query", []),
("8.8.8.8", []),
("", []),
]
for value, expected in cases:
with self.subTest(value=value):
self.assertEqual(parsedmarc.utils._url_secrets(value), expected)
def test_mask_nameserver_secrets_masks_entries_passwords_and_prefixes(self):
"""Each configured entry is replaced by its redacted form, and each
secret in it wherever it still appears, including the bare port-like
prefix that urlsplit() and httpx quote."""
mask = parsedmarc.utils._mask_nameserver_secrets
url = "https://doh:p4ss/w0rd@dns.example/dns-query"
text = (
f"Server {url} answered x; "
"Port could not be cast to integer value as 'p4ss'; "
"Invalid port: 'p4ss/w0rd'"
)
self.assertEqual(
mask(text, [url, "8.8.8.8"]),
"Server https://***@dns.example/dns-query answered x; "
"Port could not be cast to integer value as '***'; "
"Invalid port: '***'",
)
self.assertEqual(
mask(
"Server https://u:PW123@h.example/q failed (PW123)",
["https://u:PW123@h.example/q"],
),
"Server https://u:***@h.example/q failed (***)",
)
self.assertEqual(mask("abc", None), "abc")
self.assertEqual(mask("abc", ["https://u:@h"]), "abc")
def test_mask_nameserver_secrets_handles_unsplittable_entries(self):
"""urlsplit() raises ValueError for an entry such as ``https://[x``;
the masking uses string methods only, so it still returns text."""
text = "Server https://[x failed; Server https://u:PW@[x failed"
self.assertEqual(
parsedmarc.utils._mask_nameserver_secrets(
text, ["https://[x", "https://u:PW@[x"]
),
"Server https://[x failed; Server https://u:***@[x failed",
)
def test_malformed_https_and_tls_ports_are_reported_without_secrets(self):
"""A password containing "/", "?" or "#" makes urlsplit() read its
start as the port, and its ValueError quotes that text; dnspython
2.9.1 raised it from ``DoHNameserver.answer_port()`` only after a
failed query (``dns/nameserver.py``). Both schemes are now rejected
before any query, naming the entry with its credentials masked and
with no chained exception repeating the urlsplit message."""
for entry, shown in (
(
"https://doh:p4ss/w0rd@dns.example/dns-query",
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
),
(
"https://doh:p4ss?w0rd@dns.example/dns-query",
"Invalid DNS over HTTPS nameserver https://***@dns.example/dns-query",
),
(
"tls://dot:p4ss#w0rd@1.1.1.1",
"Invalid DNS over TLS nameserver tls://***@1.1.1.1",
),
(
"https://dns.example:notaport/dns-query",
"Invalid DNS over HTTPS nameserver https://dns.example:notaport/",
),
):
with self.subTest(entry=entry):
with patch("dns.query.https", side_effect=AssertionError):
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa",
"PTR",
nameservers=[entry],
timeout=0.2,
)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertIn(shown, str(ctx.exception))
self.assertIsNone(ctx.exception.__cause__)
self.assertIsNone(ctx.exception.__context__)
for secret in ("p4ss", "w0rd"):
self.assertNotIn(secret, trace)
def test_dnspython_rejection_of_an_entry_is_masked(self):
"""dnspython rejects an entry urlsplit() cannot split with a message
that quotes it as configured ("nameserver https://u:pw@[x is not a
dns.nameserver.Nameserver instance ..."); query_dns() masks it."""
entry = "https://doh:S3CRETPW@[dns.example/dns-query"
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa", "PTR", nameservers=[entry], timeout=0.2
)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertIn("https://doh:***@[dns.example/dns-query", str(ctx.exception))
self.assertNotIn("S3CRETPW", trace)
def test_get_reverse_dns_debug_log_masks_nameserver_credentials(self):
"""get_reverse_dns() logs the DNS error it swallows at debug level,
and LifetimeTimeout/NoNameservers quote each failing nameserver
URL, so the configured credentials are masked there too."""
url = "https://doh:S3CRETPW@dns.example/dns-query"
def fail(*args, **kwargs):
raise OSError(f"connection to {url} refused (S3CRETPW)")
with patch("dns.query.https", side_effect=fail):
with self.assertLogs("parsedmarc.log", level="DEBUG") as logs:
result = parsedmarc.utils.get_reverse_dns(
"1.0.0.1", nameservers=[url], timeout=0.2, retries=0
)
self.assertIsNone(result)
output = "\n".join(logs.output)
self.assertIn("get_reverse_dns(1.0.0.1) exception:", output)
self.assertIn("https://doh:***@dns.example/dns-query", output)
self.assertNotIn("S3CRETPW", output)
class TestLoadIpDb(unittest.TestCase):
"""Tests for the load_ip_db() download/cache/bundled fallback chain,
mocking at the httpx SDK boundary."""
+44
View File
@@ -114,6 +114,50 @@ class TestWebhookErrorHandling(unittest.TestCase):
client.save_smtp_tls_report_to_webhook('{"t": 1}')
class TestWebhookInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
any network I/O. The error is logged on every report."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@hooks.example.com/x", "p4ssw0rd"),
("https://admin:p4ss?w0rd@hooks.example.com/x", "p4ss"),
("https://admin:p4ss#w0rd@hooks.example.com/x", "p4ss"),
)
def test_invalid_url_is_logged_without_password(self):
saves = (
"save_aggregate_report_to_webhook",
"save_failure_report_to_webhook",
"save_smtp_tls_report_to_webhook",
)
for url, secret in self.MALFORMED:
client = WebhookClient(url, url, url)
for method in saves:
with self.subTest(url=url, method=method):
with self.assertLogs("parsedmarc.log", level="ERROR") as cm:
getattr(client, method)('{"a": 1}') # must not raise
self.assertEqual(
cm.output,
[
"ERROR:parsedmarc.log:Webhook Error: invalid webhook "
"URL (details withheld because they may include "
"credentials)"
],
)
self.assertNotIn(secret, "\n".join(cm.output))
def test_valid_url_is_the_one_posted(self):
url = "https://admin:p4ssw0rd@hooks.example.com/x"
client = WebhookClient(url, url, url)
client.session = MagicMock()
client.save_aggregate_report_to_webhook('{"a": 1}')
client.session.post.assert_called_once_with(url, content='{"a": 1}', timeout=60)
class TestWebhookClientClose(unittest.TestCase):
def test_close_closes_session(self):
client = _client()