mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-20 22:26:53 +00:00
Merge commit from fork
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv) With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init lines and opensearch.set_hosts() logged configured URLs verbatim, so userinfo passwords reached the console and log_file (CWE-532). Triage found more paths of the same class: - the httpx "HTTP Request:" line, forwarded at --verbose/--debug, contains the full URL (str(httpx.URL) keeps userinfo and query), so webhook and HEC URL credentials and the IPinfo ?token= were logged on every request; - the PSL overrides, IP database and reverse DNS map downloaders logged their URL at DEBUG and str(HTTPStatusError), which embeds the full URL, at WARNING; - libpq echoes the DSN or a password fragment for some malformed PostgreSQL connection strings, logged as an output-client error. Add redact_url_credentials() (string methods only; urlsplit misparses scheme-less "user:pw@host:port") and use it at each log site; add a filter on the httpx logger that masks userinfo and the token query value; build download-failure warnings from the status code and the redacted request URL instead of the exception text; replace psycopg ProgrammingError detail from a connection_string with a fixed message. Docs now recommend separate user/password settings over URL credentials. Reported by helixnebulatech. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Mask credentials in the Loki URL in the log and LokiError text The Loki output (#926) landed on master after this fix was written. Its url keeps userinfo (copy_with(path=...) preserves it), and the --debug "Initializing Loki client" line and both LokiError messages, which the CLI logs on a failed push, included it as-is: the same CWE-532 path as the Splunk HEC line. Mask it with redact_url_credentials() at all three sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532): - httpx.InvalidURL quotes the component it rejected, so a password with an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there) leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats it. Loki, Splunk HEC, the webhook and the three download loaders now report a fixed "details withheld" message raised from None. InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url crashed startup; the loaders now treat it as a download failure. - redact_url_credentials() masks through the last "@" when the authority has no "@" but one follows it, so the --debug lines no longer print such passwords; the userinfo lookup is shared with the new nameserver masking. - dnspython 2.9 DoH logs its request line on the httpx2 logger, which was forwarded but not filtered; the filter now covers httpx2 and httpx2.URL. - [general] nameservers / --nameservers credentials are masked in the DNS pre-flight error, reverse DNS debug errors and per-report errors, and unusable https:// / tls:// ports are rejected before any query. - Docs recommend token/username settings for [loki] and [splunk_hec]; AGENTS.md gains rules for keeping credentials out of logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
17 files changed
+1595
-73
No files matched your search
@@ -135,6 +135,18 @@ Assume every regex in the package runs on content a remote sender controls (mess
|
||||
- **Time the pattern on an adversarial input before shipping it**, not just on the samples: a 1 MB string of the repeated opener with no closer, and a 200,000-character whitespace run. A linear pattern finishes in milliseconds; anything that takes seconds is a bug. Add that input as a time-bound regression test with a generous bound (seconds) and cite the measured old timing in the docstring, as the regression tests in `tests/test_init.py` and `tests/test_utils.py` do.
|
||||
- Python 3.10 is the floor, so possessive quantifiers and atomic groups (3.11+) are not available; fix the pattern's structure instead.
|
||||
|
||||
### Credentials must never reach logs or error text
|
||||
|
||||
Assume every configured URL, host list, connection string, and token carries a secret, and that every log line and exception message ends up on a console, in `log_file`, or in a pasted bug report (CWE-532). GHSA-hjr3-rx4c-24cv found leaks in `--debug` client-init lines, the forwarded httpx request line, download warnings, libpq's DSN echo, and httpx's `InvalidURL` text. Before writing a log call or an exception message that touches configuration:
|
||||
|
||||
- **Never log a configured value as-is.** Pass URLs and host lists through `redact_url_credentials()` (`parsedmarc/utils.py`), which masks userinfo (`https://user:***@host`), handles scheme-less `user:pw@host:port`, and, when the authority has no `@` but one follows it (a `/`, `?`, or `#` in the password, but also a credential-free URL or free text with a later `@`), masks through the last `@` (`https://***@host`). Redaction is log-only: the SDK or transport still gets the real value, so mask at the log call and not in `opts` or a client attribute.
|
||||
- **Treat third-party exception text as a leak until shown otherwise.** `httpx.HTTPStatusError` embeds the full request URL. `httpx.InvalidURL` quotes the component it rejected, so a password with an unencoded `/`, `?`, or `#` becomes `Invalid port: 'p4ssw0rd'`. libpq echoes the DSN for some malformed connection strings. Don't log `str(e)` from parsing, connecting to, or requesting a configured endpoint. Build the message from structured fields (`_describe_http_error()`: status code plus the redacted request URL), or use a fixed "details withheld because they may include credentials" message. Read the library source in the venv to check what its messages contain; don't assume.
|
||||
- **Raise the replacement `from None`.** `from e` and the implicit `__context__` both print the original exception when it is logged with `logger.exception` (the SIGHUP reload path does this) or when parsedmarc crashes, and `InvalidURL`'s own context is a `ValueError` that repeats the secret.
|
||||
- **Catch the right class.** `httpx.InvalidURL` is not an `httpx.HTTPError`, so an `except httpx.HTTPError` falls through to a generic handler that logs the raw text, or doesn't catch it at all and crashes startup.
|
||||
- **Check dependency loggers.** httpx logs every request URL at INFO. `_configure_dependency_logging()` (`parsedmarc/cli.py`) attaches `_RedactUrlCredentialsFilter` to the `httpx` and `httpx2` loggers (dnspython 2.9+ makes DNS over HTTPS requests through httpx2, and `httpx2.URL` is not an `httpx.URL`). A new dependency logger forwarded at `--verbose`/`--debug` needs the same check, and so does a new library that logs its endpoint.
|
||||
- **Test both halves, including malformed input.** Assert that the secret is absent from the captured log *and* from `"".join(traceback.format_exception(exc))`, and that the real credential still reaches the mocked transport. Include values the samples never contain: a password with `/`, `?`, or `#`; userinfo with no `:` (a bare token); and a scheme-less `user:pw@host:port`.
|
||||
- **Document the safe setting.** When an output has separate `user`/`username` and `password`, `token`, or `api_key` settings, its `usage.md` entry should recommend them over credentials in the URL.
|
||||
|
||||
## Testing standards
|
||||
|
||||
These rules govern *every* test added to `tests/`. They exist because the project has been burned by tests that looked like coverage but caught nothing, and by bug claims that turned out to be wrong about the spec. Both failure modes erode trust faster than missing coverage does.
|
||||
|
||||
Reference in new issue
Block a user