Merge commit from fork

* Evict expired dedup-cache entries on insert and hash the key (GHSA-9fw6-mwqr-p43m)

The seen-aggregate-report cache is an expiringdict.ExpiringDict, which
removes an expired entry only when that same key is read again, or drops
its oldest entry once it holds max_len (100 million) entries. A report
that is never received again is never read again, so in watch mode every
distinct aggregate report stayed in memory for the life of the process.
The key was the JSON of the sender-controlled reporter, Report-ID and
policy domain, so ten 5.6 KB attachments with a 5 MiB report_id held
64 MiB indefinitely.

SeenReportCache (parsedmarc/config.py) subclasses ExpiringDict and removes
expired entries from the head of its insertion order on every insert;
the cache's max_len is now 100,000. _aggregate_report_dedup_key() now
returns the SHA-256 hex digest of that JSON, so a key is 64 characters
whatever the report contains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a re-added dedup-cache key at the tail so the expiry sweep is not blocked

OrderedDict refreshes a re-set key's time in place, so a key added again while still stored kept its old position at the head and stopped the insert-time sweep before the expired entries behind it. SeenReportCache now moves such a key to the tail before storing it, keeping insertion order equal to age order.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Bound failed reverse-DNS lookups to 120 seconds per aggregate report (GHSA-rwfh-ww9c-jm9m)

Every record of an aggregate report made one blocking PTR lookup of a sender-chosen source IP, with no bound on count or time; an unanswering reverse zone costs the full 8 s DNS lifetime with the default nameservers, and a weak-fallback result is never cached, so one IP repeated in every record was looked up every time. A 3,500-record report could block processing for hours.

Lookups that return no PTR now share a 120-second budget per report; once it is spent, the remaining records are enriched from the local IP database only, their results are not cached, and one warning is logged. Each distinct source IP is looked up at most once per report, and lookups that find a PTR are not charged, so large legitimate reports keep their enrichment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep values PostgreSQL rejects out of its output, and let one rejected report not hold the batch (GHSA-fxr7-7h9q-v949)

An aggregate count above 2**31 - 1, an SMTP TLS sending-mta-ip or
receiving-ip that is not an IP address, and a NUL in a failure report's
sample each made the PostgreSQL save raise PostgreSQLError, which marked
the whole mailbox batch unsaved, so legitimate reports from other senders
were never archived. The parser now skips a record whose count is outside
0..MAX_MESSAGE_COUNT and rejects such an SMTP TLS session count (closing
the same rejection in Elasticsearch/OpenSearch), drops an invalid TLS IP
with a warning, and the PostgreSQL output replaces NUL with U+FFFD in
text, text array and JSONB values. Any other psycopg.DataError raises the
new PostgreSQLRejectedReport, which the CLI logs and skips for that one
report unless fail_on_output_error is set.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Check an aggregate record's count before its reverse-DNS lookup

A record whose count is outside 0..MAX_MESSAGE_COUNT is dropped, so its
source IP is no longer looked up first, which could cost up to 8 seconds
per record. A regression test mocks dnspython and asserts that the dropped
record's address is never queried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Require TLS for the PostgreSQL output by default and add [postgresql] sslmode and sslrootcert (GHSA-xvp7-7762-gx9g)

PostgreSQLClient connected without an sslmode, so libpq used its default,
prefer: when the server, or an on-path attacker, declined TLS, it continued
over plain TCP and answered a clear-text password request. The client now
passes sslmode=require unless the new sslmode argument, an sslmode in the
connection string, or PGSSLMODE chooses a mode, and sslrootcert names the
CA certificate for verify-ca and verify-full. A warning is logged at
startup when a TCP host other than localhost or a loopback address is used
with a mode that does not verify the server certificate. A malformed
connection string is now rejected by a parse before connecting, with the
same withheld-details error. A server without TLS fails with libpq's
"server does not support SSL, but SSL was required" until sslmode is set
to disable or prefer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Withhold the hosts URL from Elasticsearch and OpenSearch client-build errors (GHSA-74xv-rmpx-2pjm)

A password with an unencoded "/", "?" or "#" in a [elasticsearch] or
[opensearch] hosts URL makes elastic_transport raise "Could not parse URL
'<url>'" and opensearch-py (via urllib.parse) raise "Port could not be cast
to integer value as '<password prefix>'". The startup wrapper repeated that
text at ERROR without --debug and chained the original, which the SIGHUP
reload's logger.exception printed again.

set_hosts() now runs through _set_search_hosts(), which replaces any
ValueError with a fixed message raised outside the except block. Other
errors in the block keep their text with URL credentials masked and are
raised from None. This completes the GHSA-hjr3-rx4c-24cv fix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Cache the downloaded IP database in a private per-user directory and validate it (GHSA-5f56-8ph8-v9q5)

load_ip_db() cached the IPinfo Lite download at the fixed, shared
<tmp>/parsedmarc/ipinfo_lite.mmdb, where another local user could plant
a symlink to have the download overwrite a file the parsedmarc user can
write, or plant a database that was used whenever the download failed.

The cache now lives in $XDG_CACHE_HOME/parsedmarc or ~/.cache/parsedmarc
(%LOCALAPPDATA%\parsedmarc\cache on Windows), created 0700 and used only
if it is a real directory the user owns and can write that is not
group- or world-writable; otherwise a per-run mkdtemp() directory is
used. The download is written with mkstemp(), opened with maxminddb to
validate it, and moved into place with os.replace(). A cached copy is
used only if it is a regular file owned by the user.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Skip reports over the Logs Ingestion API size limit and publish the Log Analytics streams independently (GHSA-rg4h-58fm-8pc9)

The Azure Monitor ingestion SDK packs whole reports into chunks of up to
1 MiB and never splits one, so a report whose JSON is larger is sent as a
call of its own, and the Logs Ingestion API rejects any call over 1 MB
(https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits).
The rejected upload raised before the failure and SMTP TLS streams were
sent, and the CLI held the whole batch for retry and finally Unsaved.

publish_json() now measures each report as the SDK does and leaves out
one over 1,000,000 bytes with a warning naming it, and publish_results()
publishes each stream in its own try and raises one LogAnalyticsException
naming every stream that failed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Address review: redact the PostgreSQL warning host, drop the connect error chain, and fix Log Analytics and docs wording

libpq ends a URI's userinfo at the first "@", so the rest of a password
containing "@" becomes part of the host. The PostgreSQL startup warning
now credential-masks every host-like value it logs (host list entries,
hostaddr, PGHOST, PGHOSTADDR), checks hostaddr before host as libpq
does, and the general connect error is raised from None so the driver's
unredacted text no longer reaches a logged traceback. The CLI's
output-client Raises docs now say which outputs do not chain.

publish_json() returns the number of reports uploaded, and
publish_results() logs "Successfully pushed" only for a stream that
uploaded something.

The [postgresql] URI example uses the Unix socket and explains that TCP
to localhost needs TLS on the server or sslmode = disable, the
[log_analytics] docs note the SDK's 1 MiB call packing, the
reverse-DNS-budget wording says "local IP database and reverse DNS map"
everywhere, and the GHSA-fxr7 entry no longer claims the parser rejects
every unreadable start-datetime.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Require mailsuite 2.3.7

mailsuite 2.3.7 fixes parse_email() keeping a CRLF message's carriage
returns in address header values (seanthegeek/mailsuite#74). The floor
is raised in the base dependency and the msgraph, gmail and all extras,
and the usage docs' stated requirement is updated to match.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Drop Python 3.10 and add Python 3.15 to the supported versions and CI (#953)

Python 3.10 reaches end of life in October 2026. requires-python is now
>=3.11, the unused requires_python key under [build-system] (not read by
hatchling, and its <3.15 bound read as excluding 3.15) is removed, and
Python 3.15 joins the CI matrix, classifiers, and support tables. Docs,
AGENTS.md, and comments that named 3.10 as the floor are updated.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

* Keep aggregate identifiers and reasons within their schema (#950)

* Keep aggregate identifiers and reasons within their schema

AI assistance: OpenAI Codex.

* Expand the #948 CHANGELOG entry and tighten its tests

RFC 9990 Appendix A and RFC 7489 Appendix C define only header_from,
envelope_from and envelope_to under identifiers, and only type and
comment in a reason. Cite them in the test, check that envelope_from
and envelope_to survive next to a dropped nested unknown element, and
cover an unknown key on the legacy identities path. The CHANGELOG
entry now names the affected outputs and the compatibility change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Reject reports dated in the future and bound what parse_report_file() reads (#954)

* Reject reports whose index-naming date is more than two days in the future (#947)

The date that names a report's dated Elasticsearch/OpenSearch index (an
aggregate report's begin date, a failure report's Arrival-Date, an SMTP
TLS report's start-datetime) was never compared with the present, so a
report dated 2100 named an index for 2100. Each parser now rejects such a
report with its own Invalid*Report; past dates stay unbounded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Refuse an input larger than the report size cap before parse_report_file() reads it (#949)

The 32 MiB cap was applied only after the whole input was read. A path is
now checked with os.path.getsize(), bytes-like input with len(), and a
file object is read at most one byte past the cap. The error is the
ParserError with the message the content check already gives. Three
existing tests that fed over-cap content now see the earlier rejection.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Measure a memoryview input by its byte size

len() of a memoryview counts items, which are not bytes for every format,
so the size cap in parse_report_file() compares nbytes instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Bound the aggregate end date and read file objects to their end

Address the Copilot review of #954. A normalized report's per-day records
name their indexes from their own interval starts, so the report's end date
is bounded like its begin date. A raw stream may return fewer bytes than
asked for before its end, so parse_report_file() now reads in a loop until
end of file or one byte past the cap.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Accumulate bounded reads in one buffer

Address the second Copilot review of #954. Keeping every chunk in a list
cost one bytes object per read, about 85 bytes per byte for a stream that
returns a byte at a time (22.7 MiB for 256 KiB of input). One bytearray
keeps memory proportional to the bytes read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

* Skip a message with an invalid report attachment instead of aborting the mbox (#955)

* Skip a message with an invalid report attachment instead of aborting the mbox

parse_report_email() raised a bare ParserError when a report attachment was
invalid, which get_dmarc_reports_from_mbox() re-raises, so one such message
ended the whole file. Raise InvalidDMARCReport (a ParserError subclass), which
the mbox loop skips with a warning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Skip an invalid SMTP TLS attachment in an mbox too

Address the Copilot review of #955. An invalid SMTP TLS attachment raised
InvalidSMTPTLSReport, which is not an InvalidDMARCReport, so it still
stopped the mbox. parse_report_email() now names the message's subject in
that error, and both mbox loops skip it with a warning. A message that
cannot be parsed at all still raises a plain ParserError.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

* Address the second review: refuse an unencoded @ in a PostgreSQL URI, charge slow reverse-DNS lookups, and smaller fixes

- PostgreSQL: a connection string whose parsed host, hostaddr, port or
  dbname holds an "@" (the rest of a URI password with an unencoded "@",
  which libpq splits at the first "@") is refused with a fixed error
  before any warning is logged or any connection is attempted; _connect
  parses the string on every path. A leading "@" in a host entry (Linux
  abstract socket) is still allowed. _log_safe_host stays as defence in
  depth for the host argument and PGHOST/PGHOSTADDR.
- Log Analytics: publish_json catches AzureError, so a ServiceRequestError
  or ServiceResponseError no longer stops the remaining streams.
- MAX_MESSAGE_COUNT comment and CHANGELOG: the bound matches PostgreSQL's
  INTEGER columns and the Elasticsearch/OpenSearch integer mapping of the
  SMTP TLS session counts; the aggregate message_count is dynamically
  mapped as long.
- Reverse DNS budget: a lookup taking at least SLOW_REVERSE_DNS_SECONDS
  (1.0) is charged whether or not it returned a name; the warning now says
  "failed or were slow".
- An IPv6 zone ID in an aggregate record's source IP skips the record
  before any DNS lookup, and in a failure report's Source-IP raises
  InvalidFailureReport; one helper serves these and the SMTP TLS path.
- IP database cache: a relative candidate directory (HOME unset, no
  passwd entry) is rejected; offline / always_use_local_file creates no
  directory and no temporary directory and logs no "downloaded again".
- No startup warning for sslmode=require with an existing sslrootcert
  file, which libpq verifies as verify-ca.
- PostgreSQLRejectedReport truncates the driver text to 200 characters.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Address the third review: refuse a URI with more than one @, bound total reverse-DNS time, strip IPv6 zone IDs

- postgres: a URI connection string (postgresql:// or postgres://) with
  more than one literal "@" is refused on the raw string, before any log
  line or connect, closing the abstract-socket host-list and query
  parameter bypasses of the parsed-value check. libpq decodes %40 in a
  password, host, database name and query value, so a URI never needs a
  second "@". The parsed-value check remains for keyword/value strings.
- Reverse DNS: every online lookup is charged to a new 600 s per-report
  total budget (MAX_REVERSE_DNS_SECONDS_PER_REPORT); failed or slow ones
  still also use the 120 s budget. The warning names the limit reached.
- An IPv6 zone ID in an aggregate source_ip or a failure report's
  Source-IP is now stripped with a warning instead of rejecting the
  record or report; SMTP TLS still drops its optional IP fields.
- _rejection_detail truncates the driver text before escaping it.
- Docs and CHANGELOG entries updated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Withhold everything derived from a PostgreSQL connection string from logs and errors

Fourth review of GHSA-xvp7-7762-gx9g: libpq ends a URI's userinfo at the
first "@" or "/", so postgresql://u:SECRETX/x@db.invalid/d parses to host
"u", port "SECRETX" and dbname "x@db.invalid/d". It has one "@", so the
raw-count refusal let it through, and connecting raised 'invalid integer
value "SECRETX" for connection option "port"', a password fragment in the
error and its traceback. A variant put the fragment in sslmode, which the
startup warning printed. Three rounds of shape checks each missed a shape.

A. With a connection_string, nothing parsed from it is logged or raised:
   - A connection failure raises a fixed "PostgreSQL connection failed
     (details withheld ...)" from None, with " [SQLSTATE xxxxx]" appended
     only when the driver error carries a well-formed SQLSTATE
     (^[0-9A-Z]{5}$). A string libpq cannot parse keeps its own fixed
     message, and any error from the pre-parse is replaced by it.
   - The startup warning names no host and no value from the string; the
     mode is printed only when it is one of SSL_MODES.
   With the separate host/port/user/password/database arguments, the
   driver's message (credential-masked) and the host in the warning are
   kept.

B. The early refusal stays as a convenience that names the common
   mistake, mirroring libpq's scan: a URI is refused when the text after
   "://" holds more than one "@", or an "@" with a "/" before it, and the
   parsed-value check (_has_misplaced_at) runs on URIs again as well as
   keyword/value strings. Docstrings, usage.md and the CHANGELOG now say
   the refusal is a convenience and A is what keeps the password out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Wrap every PostgreSQL connect failure and narrow the connection-string refusal

- postgres._connect(): everything from parsing the settings to holding a
  configured connection (pre-parse, ssl kwargs, connect, autocommit,
  dumper registration) is protected. A non-driver error (psycopg's
  UnicodeEncodeError carrying the whole built connection string for a
  password with a lone surrogate; on Python 3.15 the unwrapped
  UnicodeEncodeError('idna', '<host>') for an over-long host label) is
  replaced with a fixed message from None: the withheld connection
  failure message with a connection string, a new fixed "a connection
  setting could not be passed to the driver" message with the separate
  settings. KeyboardInterrupt/SystemExit still propagate.
- cli._init_output_clients(): the PostgreSQL wrapper no longer chains
  (from None), and a non-PostgreSQLError gets a fixed message.
- _has_unencoded_at() now mirrors libpq's URI scan (userinfo up to the
  first "@" or "/") instead of counting "@", so a percent-encoded
  database name (d%40x) and an "@" in a query value after a "/" (a
  certificate path) are accepted, while every password-spill shape in
  the table is still refused except two that read like legitimate
  strings.
- _has_misplaced_at() is removed: a keyword/value password is its own
  keyword, so an "@" in a kv host, hostaddr or dbname is not a password
  fragment, and libpq does not percent-decode kv values, so the encoding
  advice was wrong. The refusal message now names the database name.
- Docstrings, usage.md and the GHSA-xvp7 CHANGELOG entry updated; tests
  for real psycopg (surrogate password, long host label), fake-psycopg
  non-driver errors at connect/autocommit/dumpers for both modes, a
  KeyboardInterrupt check, the rule table (also checked against the real
  parser), legitimate strings that must connect, and the CLI wrapper.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: jwalith4418-Mj <jwalith4418@gmail.com>
This commit is contained in:
authored and GitHub committed 2026-11-10 18:44:59 -05:00
1 parent 88899be94a
commit 02fcfd6e82
16 files changed
+4591 -178

No files matched your search

+493 -39
View File
@@ -2,6 +2,7 @@
import os
import shutil
import stat
import tempfile
import time
import traceback
@@ -739,8 +740,11 @@ class TestDownloadUrlCredentialsNotLogged(unittest.TestCase):
old_path = parsedmarc.utils._IP_DB_PATH
self.addCleanup(setattr, parsedmarc.utils, "_IP_DB_PATH", old_path)
parsedmarc.utils._IP_DB_PATH = None
old_cache_dir = parsedmarc.utils._IP_DB_CACHE_DIR
self.addCleanup(setattr, parsedmarc.utils, "_IP_DB_CACHE_DIR", old_cache_dir)
parsedmarc.utils._IP_DB_CACHE_DIR = None
with (
patch("parsedmarc.utils.tempfile.gettempdir", return_value=tmp_dir),
patch.dict(os.environ, {"XDG_CACHE_HOME": tmp_dir}),
self.assertLogs("parsedmarc.log", level="DEBUG") as logs,
):
parsedmarc.utils.load_ip_db(url=url or self.URL)
@@ -2249,27 +2253,86 @@ class TestNameserverCredentialsNotLeaked(unittest.TestCase):
class TestLoadIpDb(unittest.TestCase):
"""Tests for the load_ip_db() download/cache/bundled fallback chain,
mocking at the httpx SDK boundary."""
mocking at the httpx SDK boundary, and for the per-user cache directory
and cached-file checks added for GHSA-5f56-8ph8-v9q5 (the cache used to
live at the fixed, shared <tmp>/parsedmarc/ipinfo_lite.mmdb, where
another local user could plant a symlink or a database)."""
_bundled_bytes: bytes | None = None
@classmethod
def bundled_bytes(cls) -> bytes:
"""A real MMDB body for a download to return."""
if cls._bundled_bytes is None:
cls._bundled_bytes = (
files(parsedmarc.resources.ipinfo)
.joinpath("ipinfo_lite.mmdb")
.read_bytes()
)
return cls._bundled_bytes
def setUp(self):
old_ip_db_path = parsedmarc.utils._IP_DB_PATH
old_cache_dir = parsedmarc.utils._IP_DB_CACHE_DIR
parsedmarc.utils._IP_DB_PATH = None
parsedmarc.utils._IP_DB_CACHE_DIR = None
def restore():
parsedmarc.utils._IP_DB_PATH = old_ip_db_path
parsedmarc.utils._IP_DB_CACHE_DIR = old_cache_dir
self.addCleanup(restore)
# Redirect the download cache into a per-test directory so the
# tests never touch (or depend on) the real tempdir cache.
# Point the per-user cache into a per-test directory so the tests
# never touch (or depend on) the real ~/.cache/parsedmarc.
self.tmp_dir = tempfile.mkdtemp()
self.addCleanup(lambda: shutil.rmtree(self.tmp_dir, ignore_errors=True))
patcher = patch(
"parsedmarc.utils.tempfile.gettempdir", return_value=self.tmp_dir
)
self.cache_home = os.path.join(self.tmp_dir, "cache-home")
self.cache_dir = os.path.join(self.cache_home, "parsedmarc")
self.cached_path = os.path.join(self.cache_dir, "ipinfo_lite.mmdb")
patcher = patch.dict(os.environ, {"XDG_CACHE_HOME": self.cache_home})
patcher.start()
self.addCleanup(patcher.stop)
@property
def bundled(self) -> str:
return str(files(parsedmarc.resources.ipinfo).joinpath("ipinfo_lite.mmdb"))
def _response(self, content: bytes) -> MagicMock:
response = MagicMock()
response.content = content
response.raise_for_status.return_value = None
return response
def _track_temp_fallback(self) -> None:
"""Removes the temporary directory a rejected cache directory made
load_ip_db() create, once the test ends."""
def remove():
chosen = parsedmarc.utils._IP_DB_CACHE_DIR
if chosen is not None and chosen[1] != chosen[0]:
shutil.rmtree(chosen[1], ignore_errors=True)
self.addCleanup(remove)
def _assert_temp_fallback(self, cm) -> str:
"""Asserts the rejected-directory warning and that the cache moved
to a fresh parsedmarc- directory under the system temp dir, and
returns that directory."""
self.assertTrue(
any(
f"Not using {self.cache_dir} for the IP database cache" in m
for m in cm.output
),
cm.output,
)
chosen = parsedmarc.utils._IP_DB_CACHE_DIR
assert chosen is not None
fallback = chosen[1]
self.assertEqual(os.path.dirname(fallback), tempfile.gettempdir())
self.assertTrue(os.path.basename(fallback).startswith("parsedmarc-"))
return fallback
def testExistingLocalFileIsUsedDirectly(self):
"""An existing local_file_path wins without any network request"""
local_path = os.path.join(self.tmp_dir, "local.mmdb")
@@ -2281,24 +2344,25 @@ class TestLoadIpDb(unittest.TestCase):
self.assertEqual(parsedmarc.utils._IP_DB_PATH, local_path)
def testDownloadSuccessWritesCacheFile(self):
"""A successful download is written to the cache path and selected"""
response = MagicMock()
response.content = b"downloaded db bytes"
response.raise_for_status.return_value = None
with patch("parsedmarc.utils.httpx.get", return_value=response) as mock_get:
"""A successful download of a valid MMDB is written to the per-user
cache path, created 0700, and selected; no temporary file is left"""
body = self.bundled_bytes()
with patch(
"parsedmarc.utils.httpx.get", return_value=self._response(body)
) as mock_get:
parsedmarc.utils.load_ip_db(url="https://example.com/db.mmdb")
self.assertEqual(mock_get.call_args.args[0], "https://example.com/db.mmdb")
cached_path = os.path.join(self.tmp_dir, "parsedmarc", "ipinfo_lite.mmdb")
self.assertEqual(parsedmarc.utils._IP_DB_PATH, cached_path)
with open(cached_path, "rb") as f:
self.assertEqual(f.read(), b"downloaded db bytes")
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.cached_path)
with open(self.cached_path, "rb") as f:
self.assertEqual(f.read(), body)
self.assertEqual(os.listdir(self.cache_dir), ["ipinfo_lite.mmdb"])
if os.name != "nt":
self.assertEqual(os.stat(self.cache_dir).st_mode & 0o777, 0o700)
def testDownloadFailureFallsBackToCachedCopy(self):
"""On a network error, a previously cached copy is selected"""
cache_dir = os.path.join(self.tmp_dir, "parsedmarc")
os.makedirs(cache_dir)
cached_path = os.path.join(cache_dir, "ipinfo_lite.mmdb")
with open(cached_path, "wb") as f:
os.makedirs(self.cache_dir, mode=0o700)
with open(self.cached_path, "wb") as f:
f.write(b"stale cached db")
with patch(
"parsedmarc.utils.httpx.get",
@@ -2309,7 +2373,7 @@ class TestLoadIpDb(unittest.TestCase):
self.assertTrue(
any("Failed to fetch IP database" in message for message in cm.output)
)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, cached_path)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.cached_path)
def testDownloadFailureFallsBackToBundledCopy(self):
"""On a network error with no cached copy, the bundled db is used"""
@@ -2318,28 +2382,418 @@ class TestLoadIpDb(unittest.TestCase):
side_effect=httpx.ConnectError("no network"),
):
parsedmarc.utils.load_ip_db()
bundled = str(files(parsedmarc.resources.ipinfo).joinpath("ipinfo_lite.mmdb"))
self.assertEqual(parsedmarc.utils._IP_DB_PATH, bundled)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
def testSaveFailureFallsBackToBundledCopy(self):
"""A download that cannot be written to disk logs a warning and
falls back to the bundled db instead of crashing. The cache dir is
made uncreatable by pointing gettempdir at a regular file."""
blocker = os.path.join(self.tmp_dir, "blocker")
with open(blocker, "wb") as f:
f.write(b"not a directory")
response = MagicMock()
response.content = b"downloaded db bytes"
response.raise_for_status.return_value = None
with patch("parsedmarc.utils.tempfile.gettempdir", return_value=blocker):
with patch("parsedmarc.utils.httpx.get", return_value=response):
with self.assertLogs("parsedmarc.log", level="WARNING") as cm:
parsedmarc.utils.load_ip_db()
"""A download that cannot be written to disk logs a warning, leaves
no temporary file, and falls back to the bundled db instead of
crashing."""
with (
patch(
"parsedmarc.utils.httpx.get",
return_value=self._response(self.bundled_bytes()),
),
patch("parsedmarc.utils.os.replace", side_effect=OSError("disk full")),
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db()
self.assertTrue(
any("Failed to save IP database" in message for message in cm.output)
any(
"Failed to save IP database: disk full" in message
for message in cm.output
),
cm.output,
)
self.assertEqual(os.listdir(self.cache_dir), [])
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
def testUncreatableCacheDirUsesTemporaryDirectory(self):
"""When the per-user cache directory cannot be created (here its
path is a regular file), a warning names it, and the download goes
to a private temporary directory for this run instead."""
self._track_temp_fallback()
os.makedirs(self.cache_home)
with open(self.cache_dir, "wb") as f:
f.write(b"not a directory")
with (
patch(
"parsedmarc.utils.httpx.get",
return_value=self._response(self.bundled_bytes()),
),
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db()
fallback = self._assert_temp_fallback(cm)
self.assertIn("cannot be created", "\n".join(cm.output))
self.assertEqual(
parsedmarc.utils._IP_DB_PATH, os.path.join(fallback, "ipinfo_lite.mmdb")
)
if os.name != "nt":
self.assertEqual(os.stat(fallback).st_mode & 0o777, 0o700)
# A second call in the same process reuses that directory without
# repeating the warning.
with (
patch(
"parsedmarc.utils.httpx.get",
side_effect=httpx.ConnectError("no network"),
),
self.assertLogs("parsedmarc.log", level="INFO") as cm2,
):
parsedmarc.utils.load_ip_db()
self.assertFalse(any("Not using" in m for m in cm2.output), cm2.output)
self.assertEqual(
parsedmarc.utils._IP_DB_PATH, os.path.join(fallback, "ipinfo_lite.mmdb")
)
def testTemporaryDirectoryFailureFallsBackToBundledCopy(self):
"""If even the temporary directory cannot be created, load_ip_db()
warns and uses the bundled db rather than crashing startup."""
os.makedirs(self.cache_home)
with open(self.cache_dir, "wb") as f:
f.write(b"not a directory")
with (
patch(
"parsedmarc.utils.tempfile.mkdtemp",
side_effect=OSError("no temp space"),
),
patch("parsedmarc.utils.httpx.get") as mock_get,
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db()
mock_get.assert_not_called()
self.assertTrue(
any(
"Failed to create a temporary IP database cache: no temp space" in m
for m in cm.output
),
cm.output,
)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
@unittest.skipIf(os.name == "nt", "POSIX symlink semantics")
def testSymlinkAtCachedPathIsReplacedNotFollowed(self):
"""A symlink planted at the cached path does not redirect the
download into the file it points at: the victim file is unchanged
and the cached path becomes a regular file holding the download."""
os.makedirs(self.cache_dir, mode=0o700)
victim = os.path.join(self.tmp_dir, "victim.ini")
with open(victim, "w") as f:
f.write("[general]\noriginal\n")
os.symlink(victim, self.cached_path)
body = self.bundled_bytes()
with patch("parsedmarc.utils.httpx.get", return_value=self._response(body)):
parsedmarc.utils.load_ip_db()
with open(victim) as f:
self.assertEqual(f.read(), "[general]\noriginal\n")
self.assertTrue(stat.S_ISREG(os.lstat(self.cached_path).st_mode))
with open(self.cached_path, "rb") as f:
self.assertEqual(f.read(), body)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.cached_path)
@unittest.skipIf(os.name == "nt", "POSIX symlink semantics")
def testOfflineIgnoresSymlinkAtCachedPath(self):
"""Offline, a symlink at the cached path is not used; the bundled
db is."""
os.makedirs(self.cache_dir, mode=0o700)
planted = os.path.join(self.tmp_dir, "planted.mmdb")
with open(planted, "wb") as f:
f.write(self.bundled_bytes()[:16])
os.symlink(planted, self.cached_path)
with self.assertLogs("parsedmarc.log", level="WARNING") as cm:
parsedmarc.utils.load_ip_db(offline=True)
self.assertTrue(
any(
f"Ignoring the cached IP database at {self.cached_path}: "
"it is a symlink" in m
for m in cm.output
),
cm.output,
)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
def testOfflineIgnoresNonRegularCachedPath(self):
"""A directory at the cached path is not a usable cached copy."""
os.makedirs(self.cached_path, mode=0o700)
with self.assertLogs("parsedmarc.log", level="WARNING") as cm:
parsedmarc.utils.load_ip_db(offline=True)
self.assertTrue(
any("it is not a regular file" in m for m in cm.output), cm.output
)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
@unittest.skipIf(not hasattr(os, "geteuid"), "POSIX ownership")
def testOfflineIgnoresCachedCopyOwnedByAnotherUser(self):
"""A cached copy owned by another user is not used. Creating a file
owned by someone else needs root, so the effective uid is reported
as a different one instead: from the code's side the two are the
same observation (st_uid != geteuid()). The cache directory is
created first, under the real uid, so only the file check sees the
mismatch."""
os.makedirs(self.cache_dir, mode=0o700)
with open(self.cached_path, "wb") as f:
f.write(b"someone else's db")
parsedmarc.utils._IP_DB_CACHE_DIR = (self.cache_dir, self.cache_dir)
with (
patch("parsedmarc.utils.os.geteuid", return_value=os.geteuid() + 1),
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db(offline=True)
self.assertTrue(
any("it is owned by another user" in m for m in cm.output), cm.output
)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
def testInvalidDownloadIsDiscarded(self):
"""A download that is not an MMDB is not cached: a warning is
logged, no file is left in the cache directory, and the bundled db
is used."""
with (
patch(
"parsedmarc.utils.httpx.get",
return_value=self._response(b"not a database"),
),
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db()
self.assertTrue(
any(
"Downloaded IP database is not a valid MMDB file" in m
for m in cm.output
),
cm.output,
)
self.assertEqual(os.listdir(self.cache_dir), [])
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
def testInvalidDownloadKeepsPreviousCachedCopy(self):
"""An invalid download neither replaces nor hides a valid cached
copy from an earlier run."""
os.makedirs(self.cache_dir, mode=0o700)
with open(self.cached_path, "wb") as f:
f.write(b"earlier download")
with patch(
"parsedmarc.utils.httpx.get",
return_value=self._response(b"not a database"),
):
parsedmarc.utils.load_ip_db()
with open(self.cached_path, "rb") as f:
self.assertEqual(f.read(), b"earlier download")
self.assertEqual(os.listdir(self.cache_dir), ["ipinfo_lite.mmdb"])
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.cached_path)
@unittest.skipIf(not hasattr(os, "geteuid"), "POSIX permissions")
def testWorldWritableCacheDirIsNotUsed(self):
"""A world-writable cache directory (which another user could have
filled) is rejected and left as it is: a database planted in it is
not selected, and the download goes to a temporary directory."""
self._track_temp_fallback()
os.makedirs(self.cache_dir)
os.chmod(self.cache_dir, 0o777)
with open(self.cached_path, "wb") as f:
f.write(b"PLANTED")
with (
patch(
"parsedmarc.utils.httpx.get",
return_value=self._response(self.bundled_bytes()),
),
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db()
fallback = self._assert_temp_fallback(cm)
self.assertIn("writable by its group or by others", "\n".join(cm.output))
self.assertEqual(
parsedmarc.utils._IP_DB_PATH, os.path.join(fallback, "ipinfo_lite.mmdb")
)
# Not repaired: the directory may belong to someone else.
self.assertEqual(os.stat(self.cache_dir).st_mode & 0o777, 0o777)
with open(self.cached_path, "rb") as f:
self.assertEqual(f.read(), b"PLANTED")
def _parsedmarc_temp_dirs(self) -> set[str]:
return {
name
for name in os.listdir(tempfile.gettempdir())
if name.startswith("parsedmarc-")
}
def _load_with_failed_download(self):
"""Runs load_ip_db() online with the download failing, capturing
warnings; a rejected cache directory is still replaced by a
temporary one, because a download was attempted."""
with (
patch(
"parsedmarc.utils.httpx.get",
side_effect=httpx.ConnectError("no network"),
),
self.assertLogs("parsedmarc.log", level="WARNING") as cm,
):
parsedmarc.utils.load_ip_db()
return cm
@unittest.skipIf(not hasattr(os, "geteuid"), "POSIX permissions")
def testWorldWritableCacheDirOfflineUsesBundledCopy(self):
"""Offline (and with always_use_local_file) nothing is downloaded,
so a rejected cache directory is not replaced: the planted database
in it is not selected, the bundled db is, no temporary directory is
created, and no warning claims the database "will be downloaded
again on every run"."""
self._track_temp_fallback()
os.makedirs(self.cache_dir)
os.chmod(self.cache_dir, 0o777)
with open(self.cached_path, "wb") as f:
f.write(b"PLANTED")
for offline, always_use_local_file in ((True, False), (False, True)):
with self.subTest(
offline=offline, always_use_local_file=always_use_local_file
):
parsedmarc.utils._IP_DB_PATH = None
before = self._parsedmarc_temp_dirs()
with (
patch("parsedmarc.utils.httpx.get") as mock_get,
self.assertNoLogs("parsedmarc.log", level="WARNING"),
self.assertLogs("parsedmarc.log", level="DEBUG") as cm,
):
parsedmarc.utils.load_ip_db(
offline=offline, always_use_local_file=always_use_local_file
)
mock_get.assert_not_called()
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
self.assertIsNone(parsedmarc.utils._IP_DB_CACHE_DIR)
self.assertEqual(self._parsedmarc_temp_dirs(), before)
self.assertNotIn("downloaded again", "\n".join(cm.output))
def testOfflineDoesNotCreateTheCacheDir(self):
"""Offline, a missing per-user cache directory is not created."""
parsedmarc.utils.load_ip_db(offline=True)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
self.assertFalse(os.path.exists(self.cache_home))
def testOfflineReusesTheDirectoryChosenEarlier(self):
"""Offline after an online load in the same process, a copy in the
temporary directory that replaced a rejected candidate is used."""
chosen = os.path.join(self.tmp_dir, "chosen")
os.makedirs(chosen, mode=0o700)
with open(os.path.join(chosen, "ipinfo_lite.mmdb"), "wb") as f:
f.write(b"earlier download")
parsedmarc.utils._IP_DB_CACHE_DIR = (self.cache_dir, chosen)
parsedmarc.utils.load_ip_db(offline=True)
self.assertEqual(
parsedmarc.utils._IP_DB_PATH, os.path.join(chosen, "ipinfo_lite.mmdb")
)
@unittest.skipIf(not hasattr(os, "geteuid"), "POSIX ownership")
def testCacheDirOwnedByAnotherUserIsNotUsed(self):
"""A cache directory owned by another user is rejected. As in
testOfflineIgnoresCachedCopyOwnedByAnotherUser, a directory really
owned by someone else needs root, so the effective uid is reported
as a different one."""
self._track_temp_fallback()
os.makedirs(self.cache_dir, mode=0o700)
with open(self.cached_path, "wb") as f:
f.write(b"PLANTED")
with patch("parsedmarc.utils.os.geteuid", return_value=os.geteuid() + 1):
cm = self._load_with_failed_download()
self._assert_temp_fallback(cm)
self.assertIn("it is owned by another user", "\n".join(cm.output))
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
@unittest.skipIf(
not hasattr(os, "geteuid") or os.geteuid() == 0,
"POSIX permissions, which root bypasses",
)
def testReadOnlyCacheDirIsNotUsed(self):
"""A cache directory the user cannot write to is rejected."""
self._track_temp_fallback()
os.makedirs(self.cache_dir, mode=0o700)
os.chmod(self.cache_dir, 0o500)
self.addCleanup(os.chmod, self.cache_dir, 0o700)
cm = self._load_with_failed_download()
self._assert_temp_fallback(cm)
self.assertIn("it is not writable", "\n".join(cm.output))
@unittest.skipIf(os.name == "nt", "POSIX symlink semantics")
def testSymlinkedCacheDirIsNotUsed(self):
"""A cache directory that is a symlink (here to a directory holding
a planted database) is rejected, and the planted file not used,
online or offline."""
self._track_temp_fallback()
target = os.path.join(self.tmp_dir, "elsewhere")
os.makedirs(target, mode=0o700)
with open(os.path.join(target, "ipinfo_lite.mmdb"), "wb") as f:
f.write(b"PLANTED")
os.makedirs(self.cache_home)
os.symlink(target, self.cache_dir)
parsedmarc.utils.load_ip_db(offline=True)
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
cm = self._load_with_failed_download()
self._assert_temp_fallback(cm)
self.assertIn("it is not a directory", "\n".join(cm.output))
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
@unittest.skipIf(os.name == "nt", "POSIX cache location")
def testHomelessCacheDirUsesTemporaryDirectory(self):
"""With HOME unset and no passwd entry for the uid,
os.path.expanduser() returns "~/.cache/parsedmarc" unchanged, a
relative path, which would have put the cache under the current
directory. It is now rejected like any unsafe directory, and
nothing is created in the current directory."""
self._track_temp_fallback()
workdir = os.path.join(self.tmp_dir, "cwd")
os.makedirs(workdir)
old_cwd = os.getcwd()
os.chdir(workdir)
self.addCleanup(os.chdir, old_cwd)
env = {k: v for k, v in os.environ.items() if k != "XDG_CACHE_HOME"}
with (
patch.dict(os.environ, env, clear=True),
patch("parsedmarc.utils.os.path.expanduser", side_effect=lambda p: p),
):
cm = self._load_with_failed_download()
output = "\n".join(cm.output)
self.assertIn("for the IP database cache: it is not an absolute path", output)
self.assertIn("will be downloaded again on every run", output)
chosen = parsedmarc.utils._IP_DB_CACHE_DIR
assert chosen is not None
self.assertTrue(os.path.isabs(chosen[1]))
self.assertEqual(os.listdir(workdir), [])
self.assertEqual(parsedmarc.utils._IP_DB_PATH, self.bundled)
@unittest.skipIf(os.name == "nt", "POSIX cache location")
def testRelativeXdgCacheHomeIsIgnored(self):
"""A relative XDG_CACHE_HOME is invalid per the XDG Base Directory
specification and ignored in favor of ~/.cache/parsedmarc."""
home = os.path.join(self.tmp_dir, "home")
with patch.dict(os.environ, {"XDG_CACHE_HOME": "relative/cache", "HOME": home}):
with patch(
"parsedmarc.utils.httpx.get",
return_value=self._response(self.bundled_bytes()),
):
parsedmarc.utils.load_ip_db()
expected = os.path.join(home, ".cache", "parsedmarc", "ipinfo_lite.mmdb")
self.assertEqual(parsedmarc.utils._IP_DB_PATH, expected)
self.assertTrue(os.path.isfile(expected))
self.assertFalse(os.path.exists("relative"))
def testWindowsCacheLocation(self):
"""On Windows the cache lives under %LOCALAPPDATA%, or under
~/AppData/Local when that variable is unset."""
home = os.path.join(self.tmp_dir, "home")
with (
patch("parsedmarc.utils.os.name", "nt"),
patch.dict(
os.environ,
{"LOCALAPPDATA": "/lad", "HOME": home, "USERPROFILE": home},
),
):
with_var = parsedmarc.utils._ip_db_cache_dir_candidate()
del os.environ["LOCALAPPDATA"]
without_var = parsedmarc.utils._ip_db_cache_dir_candidate()
self.assertEqual(with_var, os.path.join("/lad", "parsedmarc", "cache"))
self.assertEqual(
without_var,
os.path.join(home, "AppData", "Local", "parsedmarc", "cache"),
)
bundled = str(files(parsedmarc.resources.ipinfo).joinpath("ipinfo_lite.mmdb"))
self.assertEqual(parsedmarc.utils._IP_DB_PATH, bundled)
class TestConfigureIpinfoApiProbe(unittest.TestCase):