Merge commit from fork

* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv)

With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init
lines and opensearch.set_hosts() logged configured URLs verbatim, so
userinfo passwords reached the console and log_file (CWE-532).

Triage found more paths of the same class:
- the httpx "HTTP Request:" line, forwarded at --verbose/--debug,
  contains the full URL (str(httpx.URL) keeps userinfo and query), so
  webhook and HEC URL credentials and the IPinfo ?token= were logged
  on every request;
- the PSL overrides, IP database and reverse DNS map downloaders logged
  their URL at DEBUG and str(HTTPStatusError), which embeds the full
  URL, at WARNING;
- libpq echoes the DSN or a password fragment for some malformed
  PostgreSQL connection strings, logged as an output-client error.

Add redact_url_credentials() (string methods only; urlsplit misparses
scheme-less "user:pw@host:port") and use it at each log site; add a
filter on the httpx logger that masks userinfo and the token query
value; build download-failure warnings from the status code and the
redacted request URL instead of the exception text; replace psycopg
ProgrammingError detail from a connection_string with a fixed message.
Docs now recommend separate user/password settings over URL
credentials.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Mask credentials in the Loki URL in the log and LokiError text

The Loki output (#926) landed on master after this fix was written. Its
url keeps userinfo (copy_with(path=...) preserves it), and the --debug
"Initializing Loki client" line and both LokiError messages, which the
CLI logs on a failed push, included it as-is: the same CWE-532 path as
the Splunk HEC line. Mask it with redact_url_credentials() at all three
sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers

Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532):

- httpx.InvalidURL quotes the component it rejected, so a password with
  an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there)
  leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats
  it. Loki, Splunk HEC, the webhook and the three download loaders now
  report a fixed "details withheld" message raised from None.
  InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url
  crashed startup; the loaders now treat it as a download failure.
- redact_url_credentials() masks through the last "@" when the authority
  has no "@" but one follows it, so the --debug lines no longer print
  such passwords; the userinfo lookup is shared with the new nameserver
  masking.
- dnspython 2.9 DoH logs its request line on the httpx2 logger, which was
  forwarded but not filtered; the filter now covers httpx2 and
  httpx2.URL.
- [general] nameservers / --nameservers credentials are masked in the
  DNS pre-flight error, reverse DNS debug errors and per-report errors,
  and unusable https:// / tls:// ports are rejected before any query.
- Docs recommend token/username settings for [loki] and [splunk_hec];
  AGENTS.md gains rules for keeping credentials out of logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-10-31 17:14:00 -04:00
1 parent 56e5a09923
commit 48ea47adee
17 files changed
+1595 -73

No files matched your search

+26 -23
View File
@@ -91,7 +91,9 @@ hosts = 127.0.0.1:9200
ssl = False
[opensearch]
hosts = https://admin:admin@127.0.0.1:9200
hosts = https://127.0.0.1:9200
user = admin
password = PasswordGoesHere
ssl = True
[splunk_hec]
@@ -571,14 +573,10 @@ The full set of configuration options is:
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
`https://user:secret@localhost`)
:::{note}
Special characters in the username or password must be
[URL encoded].
:::
- `hosts` - str: A comma-separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
Do not put credentials in the URL; use `user` and `password`
(or `api_key`) instead.
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
@@ -616,14 +614,10 @@ The full set of configuration options is:
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
`https://user:secret@localhost`)
:::{note}
Special characters in the username or password must be
[URL encoded].
:::
- `hosts` - str: A comma-separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
Do not put credentials in the URL; use `user` and `password`
(or `api_key`) instead.
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
@@ -656,7 +650,8 @@ The full set of configuration options is:
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
(Default: `False`)
- `splunk_hec`
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
Do not put credentials in the URL; use `token` instead.
- `token` - str: The HEC token
- `index` - str: The Splunk index to use
- `skip_certificate_verification` - bool: Skip certificate
@@ -715,8 +710,13 @@ The full set of configuration options is:
- `password` - str: The database user password (Optional)
- `database` - str: The database name (Optional)
- `connection_string` - str: A full libpq connection string or URI
(e.g. `postgresql://user:pass@host/dbname`). When provided,
all individual parameters above are ignored.
(e.g. `postgresql://user@host/dbname`). When provided,
all individual parameters above are ignored. Prefer the separate
`host`, `port`, `user`, `password`, and `database` settings and do
not put the password in the connection string: a malformed
connection string can make the driver echo it back in an error
message. parsedmarc replaces such errors with a fixed message in
its log, but cannot stop the driver from producing them.
The PostgreSQL backend is an optional extra. Install it with
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
@@ -743,11 +743,13 @@ The full set of configuration options is:
database = parsedmarc
```
Or using a DSN/URI:
Or using a DSN/URI, with the password supplied separately (for
example through the `PGPASSWORD` environment variable or a
`~/.pgpass` file, both of which libpq reads):
```ini
[postgresql]
connection_string = postgresql://parsedmarc:secret@localhost/parsedmarc
connection_string = postgresql://parsedmarc@localhost/parsedmarc
```
Saving parsed data to PostgreSQL is controlled by the `[general]`
@@ -954,6 +956,8 @@ The full set of configuration options is:
It expects the default `job=parsedmarc` label and `label_fields`.
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
URL has no path. A URL with a path is used as is.
Do not put credentials in the URL; use `username` and `password`
(or `token`) instead.
- `labels` - str: Comma-separated `name=value` stream labels, merged over
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
(in a container, the FQDN is the container's hostname)
@@ -1568,5 +1572,4 @@ journalctl -u parsedmarc.service -r
:::
[cloudflare's public resolvers]: https://1.1.1.1/
[url encoded]: https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api