mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-06 16:59:23 +00:00
Merge commit from fork
* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv) With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init lines and opensearch.set_hosts() logged configured URLs verbatim, so userinfo passwords reached the console and log_file (CWE-532). Triage found more paths of the same class: - the httpx "HTTP Request:" line, forwarded at --verbose/--debug, contains the full URL (str(httpx.URL) keeps userinfo and query), so webhook and HEC URL credentials and the IPinfo ?token= were logged on every request; - the PSL overrides, IP database and reverse DNS map downloaders logged their URL at DEBUG and str(HTTPStatusError), which embeds the full URL, at WARNING; - libpq echoes the DSN or a password fragment for some malformed PostgreSQL connection strings, logged as an output-client error. Add redact_url_credentials() (string methods only; urlsplit misparses scheme-less "user:pw@host:port") and use it at each log site; add a filter on the httpx logger that masks userinfo and the token query value; build download-failure warnings from the status code and the redacted request URL instead of the exception text; replace psycopg ProgrammingError detail from a connection_string with a fixed message. Docs now recommend separate user/password settings over URL credentials. Reported by helixnebulatech. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Mask credentials in the Loki URL in the log and LokiError text The Loki output (#926) landed on master after this fix was written. Its url keeps userinfo (copy_with(path=...) preserves it), and the --debug "Initializing Loki client" line and both LokiError messages, which the CLI logs on a failed push, included it as-is: the same CWE-532 path as the Splunk HEC line. Mask it with redact_url_credentials() at all three sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532): - httpx.InvalidURL quotes the component it rejected, so a password with an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there) leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats it. Loki, Splunk HEC, the webhook and the three download loaders now report a fixed "details withheld" message raised from None. InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url crashed startup; the loaders now treat it as a download failure. - redact_url_credentials() masks through the last "@" when the authority has no "@" but one follows it, so the --debug lines no longer print such passwords; the userinfo lookup is shared with the new nameserver masking. - dnspython 2.9 DoH logs its request line on the httpx2 logger, which was forwarded but not filtered; the filter now covers httpx2 and httpx2.URL. - [general] nameservers / --nameservers credentials are masked in the DNS pre-flight error, reverse DNS debug errors and per-report errors, and unusable https:// / tls:// ports are rejected before any query. - Docs recommend token/username settings for [loki] and [splunk_hec]; AGENTS.md gains rules for keeping credentials out of logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
17 files changed
+1595
-73
No files matched your search
+26
-23
@@ -91,7 +91,9 @@ hosts = 127.0.0.1:9200
|
||||
ssl = False
|
||||
|
||||
[opensearch]
|
||||
hosts = https://admin:admin@127.0.0.1:9200
|
||||
hosts = https://127.0.0.1:9200
|
||||
user = admin
|
||||
password = PasswordGoesHere
|
||||
ssl = True
|
||||
|
||||
[splunk_hec]
|
||||
@@ -571,14 +573,10 @@ The full set of configuration options is:
|
||||
|
||||
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
|
||||
:::{note}
|
||||
Special characters in the username or password must be
|
||||
[URL encoded].
|
||||
:::
|
||||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||||
Do not put credentials in the URL; use `user` and `password`
|
||||
(or `api_key`) instead.
|
||||
- `user` - str: Basic auth username
|
||||
- `password` - str: Basic auth password
|
||||
- `api_key` - str: API key
|
||||
@@ -616,14 +614,10 @@ The full set of configuration options is:
|
||||
|
||||
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
|
||||
:::{note}
|
||||
Special characters in the username or password must be
|
||||
[URL encoded].
|
||||
:::
|
||||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||||
Do not put credentials in the URL; use `user` and `password`
|
||||
(or `api_key`) instead.
|
||||
- `user` - str: Basic auth username
|
||||
- `password` - str: Basic auth password
|
||||
- `api_key` - str: API key
|
||||
@@ -656,7 +650,8 @@ The full set of configuration options is:
|
||||
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
|
||||
(Default: `False`)
|
||||
- `splunk_hec`
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
|
||||
Do not put credentials in the URL; use `token` instead.
|
||||
- `token` - str: The HEC token
|
||||
- `index` - str: The Splunk index to use
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
@@ -715,8 +710,13 @@ The full set of configuration options is:
|
||||
- `password` - str: The database user password (Optional)
|
||||
- `database` - str: The database name (Optional)
|
||||
- `connection_string` - str: A full libpq connection string or URI
|
||||
(e.g. `postgresql://user:pass@host/dbname`). When provided,
|
||||
all individual parameters above are ignored.
|
||||
(e.g. `postgresql://user@host/dbname`). When provided,
|
||||
all individual parameters above are ignored. Prefer the separate
|
||||
`host`, `port`, `user`, `password`, and `database` settings and do
|
||||
not put the password in the connection string: a malformed
|
||||
connection string can make the driver echo it back in an error
|
||||
message. parsedmarc replaces such errors with a fixed message in
|
||||
its log, but cannot stop the driver from producing them.
|
||||
|
||||
The PostgreSQL backend is an optional extra. Install it with
|
||||
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
|
||||
@@ -743,11 +743,13 @@ The full set of configuration options is:
|
||||
database = parsedmarc
|
||||
```
|
||||
|
||||
Or using a DSN/URI:
|
||||
Or using a DSN/URI, with the password supplied separately (for
|
||||
example through the `PGPASSWORD` environment variable or a
|
||||
`~/.pgpass` file, both of which libpq reads):
|
||||
|
||||
```ini
|
||||
[postgresql]
|
||||
connection_string = postgresql://parsedmarc:secret@localhost/parsedmarc
|
||||
connection_string = postgresql://parsedmarc@localhost/parsedmarc
|
||||
```
|
||||
|
||||
Saving parsed data to PostgreSQL is controlled by the `[general]`
|
||||
@@ -954,6 +956,8 @@ The full set of configuration options is:
|
||||
It expects the default `job=parsedmarc` label and `label_fields`.
|
||||
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
|
||||
URL has no path. A URL with a path is used as is.
|
||||
Do not put credentials in the URL; use `username` and `password`
|
||||
(or `token`) instead.
|
||||
- `labels` - str: Comma-separated `name=value` stream labels, merged over
|
||||
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
|
||||
(in a container, the FQDN is the container's hostname)
|
||||
@@ -1568,5 +1572,4 @@ journalctl -u parsedmarc.service -r
|
||||
:::
|
||||
|
||||
[cloudflare's public resolvers]: https://1.1.1.1/
|
||||
[url encoded]: https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
|
||||
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api
|
||||
Reference in new issue
Block a user