Merge commit from fork

* Stop logging credentials from configured URLs and the IPinfo token (GHSA-hjr3-rx4c-24cv)

With --debug, the Splunk HEC, Elasticsearch and OpenSearch client-init
lines and opensearch.set_hosts() logged configured URLs verbatim, so
userinfo passwords reached the console and log_file (CWE-532).

Triage found more paths of the same class:
- the httpx "HTTP Request:" line, forwarded at --verbose/--debug,
  contains the full URL (str(httpx.URL) keeps userinfo and query), so
  webhook and HEC URL credentials and the IPinfo ?token= were logged
  on every request;
- the PSL overrides, IP database and reverse DNS map downloaders logged
  their URL at DEBUG and str(HTTPStatusError), which embeds the full
  URL, at WARNING;
- libpq echoes the DSN or a password fragment for some malformed
  PostgreSQL connection strings, logged as an output-client error.

Add redact_url_credentials() (string methods only; urlsplit misparses
scheme-less "user:pw@host:port") and use it at each log site; add a
filter on the httpx logger that masks userinfo and the token query
value; build download-failure warnings from the status code and the
redacted request URL instead of the exception text; replace psycopg
ProgrammingError detail from a connection_string with a fixed message.
Docs now recommend separate user/password settings over URL
credentials.

Reported by helixnebulatech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Mask credentials in the Loki URL in the log and LokiError text

The Loki output (#926) landed on master after this fix was written. Its
url keeps userinfo (copy_with(path=...) preserves it), and the --debug
"Initializing Loki client" line and both LokiError messages, which the
CLI logs on a failed push, included it as-is: the same CWE-532 path as
the Splunk HEC line. Mask it with redact_url_credentials() at all three
sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Withhold InvalidURL details, mask malformed URLs, and cover httpx2 and DoH nameservers

Follow-up for GHSA-hjr3-rx4c-24cv (CWE-532):

- httpx.InvalidURL quotes the component it rejected, so a password with
  an unencoded "/", "?" or "#" (RFC 3986 3.2: the authority ends there)
  leaked as "Invalid port: 'p4ssw0rd'", and its chained ValueError repeats
  it. Loki, Splunk HEC, the webhook and the three download loaders now
  report a fixed "details withheld" message raised from None.
  InvalidURL is not an httpx.HTTPError, so a malformed psl_overrides_url
  crashed startup; the loaders now treat it as a download failure.
- redact_url_credentials() masks through the last "@" when the authority
  has no "@" but one follows it, so the --debug lines no longer print
  such passwords; the userinfo lookup is shared with the new nameserver
  masking.
- dnspython 2.9 DoH logs its request line on the httpx2 logger, which was
  forwarded but not filtered; the filter now covers httpx2 and
  httpx2.URL.
- [general] nameservers / --nameservers credentials are masked in the
  DNS pre-flight error, reverse DNS debug errors and per-report errors,
  and unusable https:// / tls:// ports are rejected before any query.
- Docs recommend token/username settings for [loki] and [splunk_hec];
  AGENTS.md gains rules for keeping credentials out of logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-10-31 17:14:00 -04:00
1 parent 56e5a09923
commit 48ea47adee
17 files changed
+1595 -73

No files matched your search

+53
View File
@@ -2,6 +2,7 @@
import json
import time
import traceback
import unittest
from unittest.mock import MagicMock, patch
@@ -469,6 +470,58 @@ class TestSaveSmtpTlsReportsToSplunk(unittest.TestCase):
)
class TestInvalidUrlDoesNotLeakPassword(unittest.TestCase):
"""GHSA-hjr3-rx4c-24cv: a password with an unencoded "/", "?" or "#"
ends the URL's authority early (RFC 3986 section 3.2), so httpx reads the
text after the colon as the port and raises
``InvalidURL("Invalid port: 'p4ssw0rd'")`` when posting (httpx 0.28.1
_urlparse.py: ``raise InvalidURL(f"Invalid port: {port!r}")``), before
any network I/O. The error reaches the log on every batch."""
MALFORMED = (
("https://admin:p4ssw0rd/extra@splunk.example.com:8088", "p4ssw0rd"),
("https://admin:p4ss?w0rd@splunk.example.com:8088", "p4ss"),
("https://admin:p4ss#w0rd@splunk.example.com:8088", "p4ss"),
)
def test_invalid_url_raises_fixed_message_without_password(self):
saves = (
("aggregate", "save_aggregate_reports_to_splunk", _aggregate_report),
("failure", "save_failure_reports_to_splunk", _failure_report),
("smtp_tls", "save_smtp_tls_reports_to_splunk", _smtp_tls_report),
)
for url, secret in self.MALFORMED:
client = HECClient(url=url, access_token="tok", index="dmarc")
for name, method, make_report in saves:
with self.subTest(url=url, report=name):
with self.assertRaises(SplunkError) as ctx:
getattr(client, method)(make_report())
self.assertEqual(
str(ctx.exception),
"Invalid Splunk HEC URL (details withheld because they "
"may include credentials)",
)
self.assertIsNone(ctx.exception.__cause__)
self.assertTrue(ctx.exception.__suppress_context__)
trace = "".join(traceback.format_exception(ctx.exception))
self.assertNotIn(secret, trace)
def test_valid_url_is_the_one_posted(self):
client = HECClient(
url="https://admin:p4ssw0rd@splunk.example.com:8088",
access_token="tok",
index="dmarc",
)
client.session = MagicMock()
client.session.post.return_value = _ok_response()
client.save_aggregate_reports_to_splunk(_aggregate_report())
self.assertEqual(
client.session.post.call_args.args[0],
"https://admin:p4ssw0rd@splunk.example.com:8088"
"/services/collector/event/1.0",
)
class TestHECClientClose(unittest.TestCase):
def test_close_closes_session(self):
client = _client()