Merge commit from fork

* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv)

With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init
lines and opensearch.set_hosts() logged configured URLs verbatim, so
userinfo passwords reached the console and log_file (CWE-532).

Triage found more paths of the same class:
- the httpx "HTTP Request:" line, forwarded at --verbose/--debug,
  contains the full URL (str(httpx.URL) keeps userinfo and query), so
  webhook and HEC URL credentials and the IPinfo ?token= were logged
  on every request;
- the PSL overrides, IP database and reverse DNS map downloaders logged
  their URL at DEBUG and str(HTTPStatusError), which embeds the full
  URL, at WARNING;
- libpq echoes the DSN or a password fragment for some malformed
  PostgreSQL connection strings, logged as an output-client error.

Add redact_url_credentials() (string methods only; urlsplit misparses
scheme-less "user:pw@host:port") and use it at each log site; add a
filter on the httpx logger that masks userinfo and the token query
value; build download-failure warnings from the status code and the
redacted request URL instead of the exception text; replace psycopg
ProgrammingError detail from a connection_string with a fixed message.
Docs now recommend separate user/password settings over URL
credentials.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Mask credentials in the Loki URL in the log and LokiError text

The Loki output (#926) landed on master after this fix was written. Its
url keeps userinfo (copy_with(path=...) preserves it), and the --debug
"Initializing Loki client" line and both LokiError messages, which the
CLI logs on a failed push, included it as-is: the same CWE-532 path as
the Splunk HEC line. Mask it with redact_url_credentials() at all three
sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers

Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532):

- httpx.InvalidURL quotes the component it rejected, so a password with
  an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there)
  leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats
  it. Loki, Splunk HEC, the webhook and the three download loaders now
  report a fixed "details withheld" message raised from None.
  InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url
  crashed startup; the loaders now treat it as a download failure.
- redact_url_credentials() masks through the last "@" when the authority
  has no "@" but one follows it, so the --debug lines no longer print
  such passwords; the userinfo lookup is shared with the new nameserver
  masking.
- dnspython 2.9 DoH logs its request line on the httpx2 logger, which was
  forwarded but not filtered; the filter now covers httpx2 and
  httpx2.URL.
- [general] nameservers / --nameservers credentials are masked in the
  DNS pre-flight error, reverse DNS debug errors and per-report errors,
  and unusable https:// / tls:// ports are rejected before any query.
- Docs recommend token/username settings for [loki] and [splunk_hec];
  AGENTS.md gains rules for keeping credentials out of logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-10-31 17:14:00 -04:00
1 parent 56e5a09923
commit 48ea47adee
17 files changed
+1595 -73

No files matched your search

+12
View File
@@ -135,6 +135,18 @@ Assume every regex in the package runs on content a remote sender controls (mess
- **Time the pattern on an adversarial input before shipping it**, not just on the samples: a 1 MB string of the repeated opener with no closer, and a 200,000-character whitespace run. A linear pattern finishes in milliseconds; anything that takes seconds is a bug. Add that input as a time-bound regression test with a generous bound (seconds) and cite the measured old timing in the docstring, as the regression tests in `tests/test_init.py` and `tests/test_utils.py` do.
- Python 3.10 is the floor, so possessive quantifiers and atomic groups (3.11+) are not available; fix the pattern's structure instead.
### Credentials must never reach logs or error text
Assume every configured URL, host list, connection string, and token carries a secret, and that every log line and exception message ends up on a console, in `log_file`, or in a pasted bug report (CWE-532). GHSA-hjr3-rx4c-24cv found leaks in `--debug` client-init lines, the forwarded httpx request line, download warnings, libpq's DSN echo, and httpx's `InvalidURL` text. Before writing a log call or an exception message that touches configuration:
- **Never log a configured value as-is.** Pass URLs and host lists through `redact_url_credentials()` (`parsedmarc/utils.py`), which masks userinfo (`https://user:***@host`), handles scheme-less `user:pw@host:port`, and, when the authority has no `@` but one follows it (a `/`, `?`, or `#` in the password, but also a credential-free URL or free text with a later `@`), masks through the last `@` (`https://***@host`). Redaction is log-only: the SDK or transport still gets the real value, so mask at the log call and not in `opts` or a client attribute.
- **Treat third-party exception text as a leak until shown otherwise.** `httpx.HTTPStatusError` embeds the full request URL. `httpx.InvalidURL` quotes the component it rejected, so a password with an unencoded `/`, `?`, or `#` becomes `Invalid port: 'p4ssw0rd'`. libpq echoes the DSN for some malformed connection strings. Don't log `str(e)` from parsing, connecting to, or requesting a configured endpoint. Build the message from structured fields (`_describe_http_error()`: status code plus the redacted request URL), or use a fixed "details withheld because they may include credentials" message. Read the library source in the venv to check what its messages contain; don't assume.
- **Raise the replacement `from None`.** `from e` and the implicit `__context__` both print the original exception when it is logged with `logger.exception` (the SIGHUP reload path does this) or when parsedmarc crashes, and `InvalidURL`'s own context is a `ValueError` that repeats the secret.
- **Catch the right class.** `httpx.InvalidURL` is not an `httpx.HTTPError`, so an `except httpx.HTTPError` falls through to a generic handler that logs the raw text, or doesn't catch it at all and crashes startup.
- **Check dependency loggers.** httpx logs every request URL at INFO. `_configure_dependency_logging()` (`parsedmarc/cli.py`) attaches `_RedactUrlCredentialsFilter` to the `httpx` and `httpx2` loggers (dnspython 2.9+ makes DNS over HTTPS requests through httpx2, and `httpx2.URL` is not an `httpx.URL`). A new dependency logger forwarded at `--verbose`/`--debug` needs the same check, and so does a new library that logs its endpoint.
- **Test both halves, including malformed input.** Assert that the secret is absent from the captured log *and* from `"".join(traceback.format_exception(exc))`, and that the real credential still reaches the mocked transport. Include values the samples never contain: a password with `/`, `?`, or `#`; userinfo with no `:` (a bare token); and a scheme-less `user:pw@host:port`.
- **Document the safe setting.** When an output has separate `user`/`username` and `password`, `token`, or `api_key` settings, its `usage.md` entry should recommend them over credentials in the URL.
## Testing standards
These rules govern *every* test added to `tests/`. They exist because the project has been burned by tests that looked like coverage but caught nothing, and by bug claims that turned out to be wrong about the spec. Both failure modes erode trust faster than missing coverage does.