diff --git a/CHANGELOG.md b/CHANGELOG.md index dcc0e8b1..45b4d64a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,11 +5,15 @@ ### New features - **`n_procs` parallel parsing now covers messages from mbox files and mailbox connections** ([#147](https://github.com/domainaware/parsedmarc/issues/147)), not just report files passed directly as CLI arguments: IMAP, Microsoft Graph, Gmail API, and Maildir connections, including watch mode. `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all gained an `n_procs` keyword argument. Parallel workers are now a reused process pool for the whole run, with a bounded submission window that keeps at most roughly `2 * n_procs` messages in flight at a time so memory stays bounded even for huge mboxes; the main process sends periodic IMAP keepalives while workers parse. +- **Added `ParserConfig`, a single frozen dataclass carrying every parsing/enrichment option** ([#503](https://github.com/domainaware/parsedmarc/issues/503)): offline mode, IP database path, reverse DNS map and PSL overrides paths/URLs, DNS nameservers/timeout/retries, `strip_attachment_payloads`, `normalize_timespan_threshold_hours`, and the three caches the parser and enrichment code share across calls (IP address info, seen aggregate report IDs, and the reverse DNS map). `parse_aggregate_report_xml()`, `parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`, `parse_report_file()`, `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all accept a keyword-only `config=` argument; when it's provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument continues to work unchanged when `config=` is omitted. Every explicitly constructed `ParserConfig` owns fresh, isolated caches; omitting `config=` falls back to the existing process-wide shared default caches, unchanged and identity-preserved (`parsedmarc.IP_ADDRESS_CACHE`, `parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, `parsedmarc.REVERSE_DNS_MAP`). Caches never cross multiprocessing worker boundaries: `ParserConfig.__getstate__` drops the three cache fields when a config is pickled for a worker, and each worker accumulates its own from that point on. `keep_alive` and `n_procs` remain separate keyword arguments — they control process/worker orchestration, not parsing or enrichment behavior, so they're deliberately not `ParserConfig` fields. See the new "Using parsedmarc as a library" section of the usage docs for an example. ### Bug fixes - **A report file whose parsing raised an unexpected non-parser exception no longer hangs the CLI forever.** The old direct-file parallel implementation spawned a fresh child process per file and blocked the parent on a pipe read; a child that crashed with anything other than a `ParserError` never wrote to that pipe, so the parent waited indefinitely. The new implementation returns the error as a value and logs `Failed to parse ` instead. - **Direct-file parallel parsing no longer stalls a whole batch on one slow file, and no longer spawns a fresh interpreter per file.** The old implementation processed files in hard batches of `n_procs`, so a single slow file delayed every other file in its batch; the new pooled-worker implementation streams files through a reused pool instead. +- **A one-shot mailbox run (no `--watch`) now honors `[general] dns_timeout` and `dns_retries`** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). The CLI's call to `get_dmarc_reports_from_mailbox()` never forwarded those two options, so every one-shot mailbox run silently used the library's own hardcoded default (`dns_timeout=6.0`, `dns_retries=0`) regardless of what the operator configured; watch-mode runs were unaffected since the `watch_inbox()` call site did pass them. The CLI now builds a single `ParserConfig` per run (via the new `_build_parser_config()` helper) from parsed options and passes it as `config=` to every parsing entry point — the direct-file, mbox, one-shot mailbox, and watch call sites, plus the SIGHUP config-reload path — instead of forwarding option keyword arguments by hand at each call site. +- **Lazily-triggered reverse DNS map loads no longer silently clobber configured PSL overrides with the bundled defaults** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). `get_ip_address_info()` and `get_service_from_reverse_dns_base_domain()` load the reverse DNS map on first use rather than eagerly (this also happens independently in each `n_procs` worker process, since a worker starts with an empty map), and `load_reverse_dns_map()` reloads `psl_overrides.txt` at the same time so map entries that depend on the current overrides fold correctly. That reload previously called `load_psl_overrides()` with no path/URL arguments, so the first lazy map load in a run overwrote any operator-configured PSL overrides file with the bundled defaults. `get_ip_address_info()` and `get_service_from_reverse_dns_base_domain()` now accept `psl_overrides_path`/`psl_overrides_url` parameters and thread them through to `load_reverse_dns_map()`, so a lazy load applies the same configured overrides an eager one would. +- **`get_dmarc_reports_from_mailbox()` and `watch_inbox()`'s `dns_timeout` default changed from a stray `6.0` to `DEFAULT_DNS_TIMEOUT`** (2.0 seconds, `parsedmarc/constants.py`), matching every other parsing entry point; `normalize_timespan_threshold_hours` also now defaults to the float `24.0` on both, rather than the int `24`. This is a minor behavior change for library callers who invoke either function directly without passing `dns_timeout`: DNS queries now time out after 2 seconds by default instead of 6, matching `parse_report_file()` and the CLI's own default. ## 10.3.0 diff --git a/docs/source/api.md b/docs/source/api.md index 0db79f0a..b5909533 100644 --- a/docs/source/api.md +++ b/docs/source/api.md @@ -7,6 +7,13 @@ :members: ``` +## parsedmarc.config + +```{eval-rst} +.. automodule:: parsedmarc.config + :members: +``` + ## parsedmarc.elastic ```{eval-rst} diff --git a/docs/source/usage.md b/docs/source/usage.md index 41a1d7d0..5100e0ae 100644 --- a/docs/source/usage.md +++ b/docs/source/usage.md @@ -944,6 +944,50 @@ For sections with underscores in the name, the full section name is used: | `gelf` | `PARSEDMARC_GELF_` | | `webhook` | `PARSEDMARC_WEBHOOK_` | +## Using parsedmarc as a library + +`parsedmarc` is also importable as a regular Python package, not just a CLI +tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`, +`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`, +`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and +`watch_inbox()` — are all importable +directly from the `parsedmarc` package. See the [API reference](api.md) for +the full set of modules and members. + +Each of these functions accepts either individual option keyword arguments +(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword +argument carrying a `ParserConfig` instance: + +```python +from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox +from parsedmarc.mail import IMAPConnection + +config = ParserConfig( + offline=False, + nameservers=["1.1.1.1", "1.0.0.1"], + dns_timeout=5.0, +) + +report = parse_report_file("aggregate_report.xml.gz", config=config) + +connection = IMAPConnection( + host="imap.example.com", user="dmarc@example.com", password="..." +) +results = get_dmarc_reports_from_mailbox(connection, config=config) +``` + +A few things to keep in mind: + +- When `config=` is passed, the individual option keyword arguments are + ignored in favor of the values carried on the `ParserConfig` instance. +- Each explicitly constructed `ParserConfig` owns its own isolated caches + (IP address info, seen aggregate report IDs, and the reverse DNS map). + Omitting `config=` falls back to the process-wide caches shared by every + call that doesn't pass one. +- `keep_alive` and `n_procs` are not part of `ParserConfig` — they control + process/worker orchestration rather than parsing or enrichment behavior, + so they are always passed as separate keyword arguments. + ## Performance tuning For large mailbox imports or backfills, parsedmarc can consume a noticeable amount diff --git a/parsedmarc/__init__.py b/parsedmarc/__init__.py index bc53f4d1..4148d679 100644 --- a/parsedmarc/__init__.py +++ b/parsedmarc/__init__.py @@ -20,11 +20,11 @@ import xml.parsers.expat as expat import zipfile import zlib from base64 import b64decode +from collections import deque +from collections.abc import Callable, Sequence from csv import DictWriter from datetime import date, datetime, timedelta, timezone, tzinfo from io import BytesIO, StringIO -from collections import deque -from collections.abc import Callable, Sequence from typing import ( Any, BinaryIO, @@ -38,6 +38,12 @@ from expiringdict import ExpiringDict from mailsuite.smtp import send_email from tqdm import tqdm +from parsedmarc.config import ( + IP_ADDRESS_CACHE, + REVERSE_DNS_MAP, + SEEN_AGGREGATE_REPORT_IDS, + ParserConfig, +) from parsedmarc.constants import ( DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT, @@ -116,10 +122,6 @@ EMAIL_SAMPLE_CONTENT_TYPES = ( "message/rfc-822-headers", ) -IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=14400) -SEEN_AGGREGATE_REPORT_IDS = ExpiringDict(max_len=100000000, max_age_seconds=3600) -REVERSE_DNS_MAP = dict() - class ParserError(RuntimeError): """Raised whenever the parser fails for some reason""" @@ -145,6 +147,66 @@ class InvalidFailureReport(InvalidDMARCReport): InvalidForensicReport = InvalidFailureReport +def _resolve_config( + config: ParserConfig | None, + *, + offline: bool = False, + ip_db_path: str | None = None, + always_use_local_files: bool = False, + reverse_dns_map_path: str | None = None, + reverse_dns_map_url: str | None = None, + nameservers: list[str] | None = None, + dns_timeout: float = DEFAULT_DNS_TIMEOUT, + dns_retries: int = DEFAULT_DNS_MAX_RETRIES, + strip_attachment_payloads: bool = False, + normalize_timespan_threshold_hours: float = 24.0, +) -> ParserConfig: + """Resolve the effective :class:`~parsedmarc.config.ParserConfig` for a + public parsing call, from either an explicit ``config`` or the caller's + individual option keyword arguments. + + If ``config`` is not ``None``, it is returned unchanged: per the + documented ``config=`` contract, the individual option keyword arguments + are ignored in favor of the config's own values, so no merging happens + here. + + Otherwise, a new ``ParserConfig`` is built from the given keyword + arguments. Its three cache fields are deliberately *not* left to their + ``default_factory`` -- doing so would hand back a config with brand new, + empty caches on every call, silently defeating cross-call IP-info + caching and aggregate-report dedup. Instead, the module-default caches + (:data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`, + :data:`REVERSE_DNS_MAP`) are injected explicitly, so kwargs-style calls + keep observing and mutating the same shared caches they always have. + + ``dataclasses.replace`` is deliberately not used here: it would need an + existing ``ParserConfig`` to start from, and there isn't one on the + kwargs path -- this function's job is to build the first one. + + ``psl_overrides_path`` / ``psl_overrides_url`` have no corresponding + keyword arguments on the public functions (see AGENTS.md's guidance on + justifying new config options), so they stay ``None`` on this path; they + are only ever set via an explicitly constructed ``config=``. + """ + if config is not None: + return config + return ParserConfig( + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + normalize_timespan_threshold_hours=float(normalize_timespan_threshold_hours), + ip_address_cache=IP_ADDRESS_CACHE, + seen_aggregate_report_ids=SEEN_AGGREGATE_REPORT_IDS, + reverse_dns_map=REVERSE_DNS_MAP, + ) + + def _exc_origin(error: BaseException) -> str: """Returns a ``" (raised at :)"`` suffix pointing at where an unexpected exception actually originated, but only when the parsedmarc @@ -378,14 +440,7 @@ def _append_parsed_record( def _parse_report_record( record: dict[str, Any], *, - ip_db_path: str | None = None, - always_use_local_files: bool = False, - reverse_dns_map_path: str | None = None, - reverse_dns_map_url: str | None = None, - offline: bool = False, - nameservers: list[str] | None = None, - dns_timeout: float = DEFAULT_DNS_TIMEOUT, - dns_retries: int = DEFAULT_DNS_MAX_RETRIES, + config: ParserConfig, is_rfc_9990: bool = False, ) -> dict[str, Any]: """ @@ -394,16 +449,9 @@ def _parse_report_record( Args: record (dict): The record to convert - always_use_local_files (bool): Do not download files - reverse_dns_map_path (str): Path to a reverse DNS map file - reverse_dns_map_url (str): URL to a reverse DNS map file - ip_db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP - offline (bool): Do not query online for geolocation or DNS - nameservers (list): A list of one or more nameservers to use - (Cloudflare's public DNS resolvers by default) - dns_timeout (float): Sets the DNS timeout in seconds - dns_retries (int): Number of times to retry DNS queries on timeout - or other transient errors + config (ParserConfig): Parsing and enrichment options, plus caches + is_rfc_9990 (bool): Whether the enclosing report was detected as + RFC 9990-shaped, for RFC 9990-aware validation warnings Returns: dict: The converted record @@ -414,16 +462,18 @@ def _parse_report_record( raise ValueError("Source IP address is empty") new_record_source = get_ip_address_info( record["row"]["source_ip"], - cache=IP_ADDRESS_CACHE, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - reverse_dns_map=REVERSE_DNS_MAP, - offline=offline, - nameservers=nameservers, - timeout=dns_timeout, - retries=dns_retries, + cache=config.ip_address_cache, + ip_db_path=config.ip_db_path, + always_use_local_files=config.always_use_local_files, + reverse_dns_map_path=config.reverse_dns_map_path, + reverse_dns_map_url=config.reverse_dns_map_url, + reverse_dns_map=config.reverse_dns_map, + offline=config.offline, + nameservers=config.nameservers, + timeout=config.dns_timeout, + retries=config.dns_retries, + psl_overrides_path=config.psl_overrides_path, + psl_overrides_url=config.psl_overrides_url, ) new_record["source"] = new_record_source new_record["count"] = int(record["row"]["count"]) @@ -783,6 +833,7 @@ def parse_aggregate_report_xml( retries: int = DEFAULT_DNS_MAX_RETRIES, keep_alive: Callable | None = None, normalize_timespan_threshold_hours: float = 24.0, + config: ParserConfig | None = None, ) -> AggregateReport: """Parses a DMARC XML report string and returns a consistent dict @@ -799,12 +850,29 @@ def parse_aggregate_report_xml( timeout (float): Sets the DNS timeout in seconds retries (int): Number of times to retry DNS queries on timeout or other transient errors - keep_alive (callable): Keep alive function + keep_alive (callable): Keep alive function. Not part of ``config``; + always applies. normalize_timespan_threshold_hours (float): Normalize timespans beyond this + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: The parsed aggregate DMARC report """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=timeout, + dns_retries=retries, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) errors = [] # Parse XML and recover from errors if isinstance(xml, bytes): @@ -896,7 +964,9 @@ def parse_aggregate_report_xml( end_ts = int(date_range["end"].split(".")[0]) span_seconds = end_ts - begin_ts - normalize_timespan = span_seconds > normalize_timespan_threshold_hours * 3600 + normalize_timespan = ( + span_seconds > cfg.normalize_timespan_threshold_hours * 3600 + ) date_range["begin"] = timestamp_to_human(begin_ts) date_range["end"] = timestamp_to_human(end_ts) @@ -1006,14 +1076,7 @@ def parse_aggregate_report_xml( try: report_record = _parse_report_record( report["record"][i], - ip_db_path=ip_db_path, - offline=offline, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - nameservers=nameservers, - dns_timeout=timeout, - dns_retries=retries, + config=cfg, is_rfc_9990=is_rfc_9990, ) _append_parsed_record( @@ -1029,14 +1092,7 @@ def parse_aggregate_report_xml( else: report_record = _parse_report_record( report["record"], - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - nameservers=nameservers, - dns_timeout=timeout, - dns_retries=retries, + config=cfg, is_rfc_9990=is_rfc_9990, ) _append_parsed_record( @@ -1175,6 +1231,7 @@ def parse_aggregate_report_file( dns_retries: int = DEFAULT_DNS_MAX_RETRIES, keep_alive: Callable | None = None, normalize_timespan_threshold_hours: float = 24.0, + config: ParserConfig | None = None, ) -> AggregateReport: """Parses a file at the given path, a file-like object. or bytes as an aggregate DMARC report @@ -1191,12 +1248,29 @@ def parse_aggregate_report_file( dns_timeout (float): Sets the DNS timeout in seconds dns_retries (int): Number of times to retry DNS queries on timeout or other transient errors - keep_alive (callable): Keep alive function + keep_alive (callable): Keep alive function. Not part of ``config``; + always applies. normalize_timespan_threshold_hours (float): Normalize timespans beyond this + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: The parsed DMARC aggregate report """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) try: xml = extract_report(_input) @@ -1205,16 +1279,8 @@ def parse_aggregate_report_file( return parse_aggregate_report_xml( xml, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - ip_db_path=ip_db_path, - offline=offline, - nameservers=nameservers, - timeout=dns_timeout, - retries=dns_retries, + config=cfg, keep_alive=keep_alive, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, ) @@ -1441,6 +1507,7 @@ def parse_failure_report( dns_timeout: float = DEFAULT_DNS_TIMEOUT, dns_retries: int = DEFAULT_DNS_MAX_RETRIES, strip_attachment_payloads: bool = False, + config: ParserConfig | None = None, ) -> FailureReport: """ Converts a DMARC failure report and sample to a dict @@ -1461,10 +1528,26 @@ def parse_failure_report( or other transient errors strip_attachment_payloads (bool): Remove attachment payloads from failure report results + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: A parsed report and sample """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + ) delivery_results = ["delivered", "spam", "policy", "reject", "other"] try: @@ -1504,16 +1587,18 @@ def parse_failure_report( ip_address = re.split(r"\s", parsed_report["source_ip"]).pop(0) parsed_report_source = get_ip_address_info( ip_address, - cache=IP_ADDRESS_CACHE, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - reverse_dns_map=REVERSE_DNS_MAP, - offline=offline, - nameservers=nameservers, - timeout=dns_timeout, - retries=dns_retries, + cache=cfg.ip_address_cache, + ip_db_path=cfg.ip_db_path, + always_use_local_files=cfg.always_use_local_files, + reverse_dns_map_path=cfg.reverse_dns_map_path, + reverse_dns_map_url=cfg.reverse_dns_map_url, + reverse_dns_map=cfg.reverse_dns_map, + offline=cfg.offline, + nameservers=cfg.nameservers, + timeout=cfg.dns_timeout, + retries=cfg.dns_retries, + psl_overrides_path=cfg.psl_overrides_path, + psl_overrides_url=cfg.psl_overrides_url, ) parsed_report["source"] = parsed_report_source del parsed_report["source_ip"] @@ -1582,7 +1667,7 @@ def parse_failure_report( parsed_report[optional_field] = None parsed_sample = parse_email( - sample, strip_attachment_payloads=strip_attachment_payloads + sample, strip_attachment_payloads=cfg.strip_attachment_payloads ) if "reported_domain" not in parsed_report: @@ -1723,6 +1808,7 @@ def parse_report_email( strip_attachment_payloads: bool = False, keep_alive: Callable | None = None, normalize_timespan_threshold_hours: float = 24.0, + config: ParserConfig | None = None, ) -> ParsedReport: """ Parses a DMARC report from an email @@ -1740,14 +1826,32 @@ def parse_report_email( or other transient errors strip_attachment_payloads (bool): Remove attachment payloads from failure report results - keep_alive (callable): keep alive function + keep_alive (callable): keep alive function. Not part of ``config``; + always applies. normalize_timespan_threshold_hours (float): Normalize timespans beyond this + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: * ``report_type``: ``aggregate`` or ``failure`` * ``report``: The parsed report """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) result: ParsedReport | None = None msg_date: datetime = datetime.now(timezone.utc) @@ -1855,16 +1959,8 @@ def parse_report_email( elif payload_text.strip().startswith("<"): aggregate_report = parse_aggregate_report_xml( payload_text, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - nameservers=nameservers, - timeout=dns_timeout, - retries=dns_retries, + config=cfg, keep_alive=keep_alive, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, ) result = {"report_type": "aggregate", "report": aggregate_report} @@ -1889,15 +1985,7 @@ def parse_report_email( feedback_report, sample, msg_date, - offline=offline, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - nameservers=nameservers, - dns_timeout=dns_timeout, - dns_retries=dns_retries, - strip_attachment_payloads=strip_attachment_payloads, + config=cfg, ) except InvalidFailureReport as e: error = ( @@ -1978,7 +2066,8 @@ def parse_report_file( reverse_dns_map_url: str | None = None, offline: bool = False, keep_alive: Callable | None = None, - normalize_timespan_threshold_hours: float = 24, + normalize_timespan_threshold_hours: float = 24.0, + config: ParserConfig | None = None, ) -> ParsedReport: """Parses a DMARC aggregate or failure file at the given path, a file-like object. or bytes @@ -1998,11 +2087,30 @@ def parse_report_file( reverse_dns_map_path (str): Path to a reverse DNS map reverse_dns_map_url (str): URL to a reverse DNS map offline (bool): Do not make online queries for geolocation or DNS - keep_alive (callable): Keep alive function + keep_alive (callable): Keep alive function. Not part of ``config``; + always applies. + normalize_timespan_threshold_hours (float): Normalize timespans beyond this + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: The parsed DMARC report """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) file_object: BinaryIO if isinstance(input_, (str, os.PathLike)): file_path = os.fspath(input_) @@ -2027,16 +2135,8 @@ def parse_report_file( try: report = parse_aggregate_report_file( content, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - nameservers=nameservers, - dns_timeout=dns_timeout, - dns_retries=dns_retries, + config=cfg, keep_alive=keep_alive, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, ) results = {"report_type": "aggregate", "report": report} except InvalidAggregateReport as aggregate_error: @@ -2047,17 +2147,8 @@ def parse_report_file( try: results = parse_report_email( content, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - nameservers=nameservers, - dns_timeout=dns_timeout, - dns_retries=dns_retries, - strip_attachment_payloads=strip_attachment_payloads, + config=cfg, keep_alive=keep_alive, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, ) except InvalidDMARCReport as email_error: raise ParserError( @@ -2076,14 +2167,19 @@ def _classify_parsed_email( aggregate_reports: list[AggregateReport], failure_reports: list[FailureReport], smtp_tls_reports: list[SMTPTLSReport], + *, + seen_aggregate_report_ids: ExpiringDict, ) -> ReportType: """Classify a parsed report email, appending it to the matching list. - Owns the ``SEEN_AGGREGATE_REPORT_IDS`` dedup check: an aggregate report - already seen (keyed on ``{org_name}_{report_id}``) is logged and - dropped instead of appended. Shared, unmodified, by the sequential and - parallel branches of ``get_dmarc_reports_from_mbox`` and - ``get_dmarc_reports_from_mailbox`` so both dedup identically. + Owns the seen-aggregate-report-ID dedup check against + ``seen_aggregate_report_ids``: an aggregate report already seen (keyed + on ``{org_name}_{report_id}``) is logged and dropped instead of + appended. Shared, unmodified, by the sequential and parallel branches + of ``get_dmarc_reports_from_mbox`` and ``get_dmarc_reports_from_mailbox`` + so both dedup identically -- callers pass the config's + ``seen_aggregate_report_ids`` cache so dedup state stays scoped to + whichever ``ParserConfig`` (explicit or module-default) is in effect. Returns the report type so mailbox callers know which UID list to append the source message's UID to. @@ -2096,8 +2192,8 @@ def _classify_parsed_email( report_org = parsed_email["report"]["report_metadata"]["org_name"] report_id = parsed_email["report"]["report_metadata"]["report_id"] report_key = f"{report_org}_{report_id}" - if report_key not in SEEN_AGGREGATE_REPORT_IDS: - SEEN_AGGREGATE_REPORT_IDS[report_key] = True + if report_key not in seen_aggregate_report_ids: + seen_aggregate_report_ids[report_key] = True aggregate_reports.append(parsed_email["report"]) else: logger.debug( @@ -2179,6 +2275,7 @@ def get_dmarc_reports_from_mbox( offline: bool = False, normalize_timespan_threshold_hours: float = 24.0, n_procs: int = 1, + config: ParserConfig | None = None, ) -> ParsingResults: """Parses a mailbox in mbox format containing e-mails with attached DMARC reports @@ -2200,12 +2297,30 @@ def get_dmarc_reports_from_mbox( normalize_timespan_threshold_hours (float): Normalize timespans beyond this n_procs (int): Number of processes to use for parsing messages in parallel. Message reading, deduplication, and result assembly - stay in the calling process; only parsing is parallelized. + stay in the calling process; only parsing is parallelized. Not + part of ``config``; always applies. + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: Lists of ``aggregate_reports``, ``failure_reports``, and ``smtp_tls_reports`` """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) aggregate_reports: list[AggregateReport] = [] failure_reports: list[FailureReport] = [] smtp_tls_reports: list[SMTPTLSReport] = [] @@ -2218,21 +2333,7 @@ def get_dmarc_reports_from_mbox( if n_procs > 1 and total_messages > 1: from parsedmarc.parallel import _parse_report_email_job, parallel_map - parse_kwargs = { - "ip_db_path": ip_db_path, - "always_use_local_files": always_use_local_files, - "reverse_dns_map_path": reverse_dns_map_path, - "reverse_dns_map_url": reverse_dns_map_url, - "offline": offline, - "nameservers": nameservers, - "dns_timeout": dns_timeout, - "dns_retries": dns_retries, - "strip_attachment_payloads": strip_attachment_payloads, - "normalize_timespan_threshold_hours": ( - normalize_timespan_threshold_hours - ), - } - func = functools.partial(_parse_report_email_job, kwargs=parse_kwargs) + func = functools.partial(_parse_report_email_job, config=cfg) def _jobs(): for i in range(total_messages): @@ -2251,7 +2352,11 @@ def get_dmarc_reports_from_mbox( raise result else: _classify_parsed_email( - result, aggregate_reports, failure_reports, smtp_tls_reports + result, + aggregate_reports, + failure_reports, + smtp_tls_reports, + seen_aggregate_report_ids=cfg.seen_aggregate_report_ids, ) else: for i in tqdm(range(total_messages), disable=None): @@ -2259,25 +2364,13 @@ def get_dmarc_reports_from_mbox( logger.info(f"Processing message {i + 1} of {total_messages}") msg_content = mbox.get_string(message_key) try: - sa = strip_attachment_payloads - parsed_email = parse_report_email( - msg_content, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - nameservers=nameservers, - dns_timeout=dns_timeout, - dns_retries=dns_retries, - strip_attachment_payloads=sa, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, - ) + parsed_email = parse_report_email(msg_content, config=cfg) _classify_parsed_email( parsed_email, aggregate_reports, failure_reports, smtp_tls_reports, + seen_aggregate_report_ids=cfg.seen_aggregate_report_ids, ) except InvalidDMARCReport as error: logger.warning(error.__str__()) @@ -2341,15 +2434,16 @@ def get_dmarc_reports_from_mailbox( reverse_dns_map_url: str | None = None, offline: bool = False, nameservers: list[str] | None = None, - dns_timeout: float = 6.0, + dns_timeout: float = DEFAULT_DNS_TIMEOUT, dns_retries: int = DEFAULT_DNS_MAX_RETRIES, strip_attachment_payloads: bool = False, results: ParsingResults | None = None, batch_size: int = 10, since: datetime | date | str | None = None, create_folders: bool = True, - normalize_timespan_threshold_hours: float = 24, + normalize_timespan_threshold_hours: float = 24.0, n_procs: int = 1, + config: ParserConfig | None = None, ) -> ParsingResults: """ Fetches and parses DMARC reports from a mailbox @@ -2385,6 +2479,11 @@ def get_dmarc_reports_from_mailbox( parallelized. With ``n_procs > 1``, invalid-message disposition happens after the parsing phase completes, rather than interleaved message-by-message as it is when ``n_procs`` is 1. + Not part of ``config``; always applies. + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. Returns: dict: Lists of ``aggregate_reports``, ``failure_reports``, and ``smtp_tls_reports`` @@ -2395,6 +2494,20 @@ def get_dmarc_reports_from_mailbox( if connection is None: raise ValueError("Must supply a connection") + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) + # current_time useful to fetch_messages later in the program current_time: datetime | date | str | None = None @@ -2479,23 +2592,15 @@ def get_dmarc_reports_from_mailbox( if n_procs > 1 and message_limit > 1: from parsedmarc.parallel import _parse_report_email_job, parallel_map - # keep_alive is a bound method of the live connection object and is - # not picklable, so it must never cross the process boundary; the - # heartbeat passed to parallel_map below keeps the connection alive - # instead. - parse_kwargs = { - "nameservers": nameservers, - "dns_timeout": dns_timeout, - "dns_retries": dns_retries, - "ip_db_path": ip_db_path, - "always_use_local_files": always_use_local_files, - "reverse_dns_map_path": reverse_dns_map_path, - "reverse_dns_map_url": reverse_dns_map_url, - "offline": offline, - "strip_attachment_payloads": strip_attachment_payloads, - "normalize_timespan_threshold_hours": (normalize_timespan_threshold_hours), - } - func = functools.partial(_parse_report_email_job, kwargs=parse_kwargs) + # The config's caches (ip_address_cache, seen_aggregate_report_ids, + # reverse_dns_map) never cross the process boundary -- + # ParserConfig.__getstate__ drops them, and each worker accumulates + # its own via the module defaults it rebinds to on unpickling (see + # ParserConfig.__setstate__). keep_alive is a bound method of the + # live connection object and is not a ParserConfig field, so it is + # never submitted to the pool either; the heartbeat passed to + # parallel_map below keeps the connection alive instead. + func = functools.partial(_parse_report_email_job, config=cfg) # parallel_map yields results in submission order, so the oldest # queued id always belongs to the next yielded result; popping as @@ -2525,7 +2630,11 @@ def get_dmarc_reports_from_mailbox( invalid_msg_ids.append(message_id) else: report_type = _classify_parsed_email( - result, aggregate_reports, failure_reports, smtp_tls_reports + result, + aggregate_reports, + failure_reports, + smtp_tls_reports, + seen_aggregate_report_ids=cfg.seen_aggregate_report_ids, ) if report_type == "aggregate": aggregate_report_msg_uids.append(message_id) @@ -2547,23 +2656,17 @@ def get_dmarc_reports_from_mailbox( ) message_id, msg_content = _fetch_mailbox_message(connection, msg_uid, test) try: - sa = strip_attachment_payloads parsed_email = parse_report_email( msg_content, - nameservers=nameservers, - dns_timeout=dns_timeout, - dns_retries=dns_retries, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - strip_attachment_payloads=sa, + config=cfg, keep_alive=connection.keepalive, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, ) report_type = _classify_parsed_email( - parsed_email, aggregate_reports, failure_reports, smtp_tls_reports + parsed_email, + aggregate_reports, + failure_reports, + smtp_tls_reports, + seen_aggregate_report_ids=cfg.seen_aggregate_report_ids, ) if report_type == "aggregate": aggregate_report_msg_uids.append(message_id) @@ -2669,19 +2772,10 @@ def get_dmarc_reports_from_mailbox( archive_folder=archive_folder, delete=delete, test=test, - nameservers=nameservers, - dns_timeout=dns_timeout, - dns_retries=dns_retries, - strip_attachment_payloads=strip_attachment_payloads, results=results, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, since=current_time, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, n_procs=n_procs, + config=cfg, ) return results @@ -2702,14 +2796,15 @@ def watch_inbox( reverse_dns_map_url: str | None = None, offline: bool = False, nameservers: list[str] | None = None, - dns_timeout: float = 6.0, + dns_timeout: float = DEFAULT_DNS_TIMEOUT, dns_retries: int = DEFAULT_DNS_MAX_RETRIES, strip_attachment_payloads: bool = False, batch_size: int = 10, since: datetime | date | str | None = None, - normalize_timespan_threshold_hours: float = 24, + normalize_timespan_threshold_hours: float = 24.0, config_reloading: Callable | None = None, n_procs: int = 1, + config: ParserConfig | None = None, ): """ Watches the mailbox for new messages and @@ -2745,8 +2840,25 @@ def watch_inbox( IDLE loop, so the watcher exits cleanly at a safe boundary. n_procs (int): Number of processes to use for parsing messages in parallel. Passed through to ``get_dmarc_reports_from_mailbox`` - on each check. + on each check. Not part of ``config``; always applies. + config (ParserConfig): a single object carrying all parsing and + enrichment options plus the caches; when provided, the + individual option keyword arguments listed above are ignored in + favor of the config's values. """ + cfg = _resolve_config( + config, + offline=offline, + ip_db_path=ip_db_path, + always_use_local_files=always_use_local_files, + reverse_dns_map_path=reverse_dns_map_path, + reverse_dns_map_url=reverse_dns_map_url, + nameservers=nameservers, + dns_timeout=dns_timeout, + dns_retries=dns_retries, + strip_attachment_payloads=strip_attachment_payloads, + normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + ) def check_callback(connection): res = get_dmarc_reports_from_mailbox( @@ -2755,20 +2867,11 @@ def watch_inbox( archive_folder=archive_folder, delete=delete, test=test, - ip_db_path=ip_db_path, - always_use_local_files=always_use_local_files, - reverse_dns_map_path=reverse_dns_map_path, - reverse_dns_map_url=reverse_dns_map_url, - offline=offline, - nameservers=nameservers, - dns_timeout=dns_timeout, - n_procs=n_procs, - dns_retries=dns_retries, - strip_attachment_payloads=strip_attachment_payloads, batch_size=batch_size, since=since, create_folders=False, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours, + n_procs=n_procs, + config=cfg, ) callback(res) diff --git a/parsedmarc/cli.py b/parsedmarc/cli.py index 1133a38d..47ddabd6 100644 --- a/parsedmarc/cli.py +++ b/parsedmarc/cli.py @@ -24,9 +24,11 @@ from kiota_abstractions.api_error import APIError from tqdm import tqdm from parsedmarc import ( + IP_ADDRESS_CACHE, REVERSE_DNS_MAP, SEEN_AGGREGATE_REPORT_IDS, InvalidDMARCReport, + ParserConfig, ParserError, __version__, elastic, @@ -46,6 +48,7 @@ from parsedmarc import ( watch_inbox, webhook, ) +from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT from parsedmarc.log import logger from parsedmarc.mail import ( AuthMethod, @@ -1458,6 +1461,41 @@ def _close_output_clients(clients): logger.warning("Error closing %s", name, exc_info=True) +def _build_parser_config(opts: Namespace) -> ParserConfig: + """Builds the single ParserConfig for this run from parsed opts, bound to + the process-wide default caches (the parsedmarc module globals). + """ + return ParserConfig( + offline=opts.offline, + ip_db_path=opts.ip_db_path, + always_use_local_files=opts.always_use_local_files, + reverse_dns_map_path=opts.reverse_dns_map_path, + reverse_dns_map_url=opts.reverse_dns_map_url, + psl_overrides_path=opts.psl_overrides_path, + psl_overrides_url=opts.psl_overrides_url, + nameservers=opts.nameservers, + dns_timeout=( + float(opts.dns_timeout) + if opts.dns_timeout is not None + else DEFAULT_DNS_TIMEOUT + ), + dns_retries=( + int(opts.dns_retries) + if opts.dns_retries is not None + else DEFAULT_DNS_MAX_RETRIES + ), + strip_attachment_payloads=opts.strip_attachment_payloads, + normalize_timespan_threshold_hours=( + float(opts.normalize_timespan_threshold_hours) + if opts.normalize_timespan_threshold_hours is not None + else 24.0 + ), + ip_address_cache=IP_ADDRESS_CACHE, + seen_aggregate_report_ids=SEEN_AGGREGATE_REPORT_IDS, + reverse_dns_map=REVERSE_DNS_MAP, + ) + + def _main(): """Called when the module is executed""" @@ -2289,19 +2327,9 @@ def _main(): if n_procs < 1: n_procs = 1 - parse_kwargs = dict( - offline=opts.offline, - ip_db_path=opts.ip_db_path, - always_use_local_files=opts.always_use_local_files, - reverse_dns_map_path=opts.reverse_dns_map_path, - reverse_dns_map_url=opts.reverse_dns_map_url, - nameservers=opts.nameservers, - dns_timeout=opts.dns_timeout, - dns_retries=opts.dns_retries, - strip_attachment_payloads=opts.strip_attachment_payloads, - normalize_timespan_threshold_hours=opts.normalize_timespan_threshold_hours, - ) - func = functools.partial(_parse_report_file_job, kwargs=parse_kwargs) + parser_config = _build_parser_config(opts) + + func = functools.partial(_parse_report_file_job, config=parser_config) for file_path, result in parallel_map( func, file_paths, n_procs, should_stop=lambda: _shutdown_requested ): @@ -2341,24 +2369,9 @@ def _main(): if _shutdown_requested: logger.info("Shutdown requested, skipping remaining mbox files") break - normalize_timespan_threshold_hours_value = ( - float(opts.normalize_timespan_threshold_hours) - if opts.normalize_timespan_threshold_hours is not None - else 24.0 - ) - strip = opts.strip_attachment_payloads reports = get_dmarc_reports_from_mbox( mbox_path, - nameservers=opts.nameservers, - dns_timeout=opts.dns_timeout, - dns_retries=opts.dns_retries, - strip_attachment_payloads=strip, - ip_db_path=opts.ip_db_path, - always_use_local_files=opts.always_use_local_files, - reverse_dns_map_path=opts.reverse_dns_map_path, - reverse_dns_map_url=opts.reverse_dns_map_url, - offline=opts.offline, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value, + config=parser_config, n_procs=n_procs, ) aggregate_reports += reports["aggregate_reports"] @@ -2369,7 +2382,6 @@ def _main(): msgraph_connection: MSGraphConnection | None = None mailbox_batch_size_value = 10 mailbox_check_timeout_value = 30 - normalize_timespan_threshold_hours_value = 24.0 if opts.imap_host: try: @@ -2525,11 +2537,6 @@ def _main(): if opts.mailbox_check_timeout is not None else 30 ) - normalize_timespan_threshold_hours_value = ( - float(opts.normalize_timespan_threshold_hours) - if opts.normalize_timespan_threshold_hours is not None - else 24.0 - ) if mailbox_connection and not _shutdown_requested: try: reports = get_dmarc_reports_from_mailbox( @@ -2538,17 +2545,9 @@ def _main(): batch_size=mailbox_batch_size_value, reports_folder=opts.mailbox_reports_folder, archive_folder=opts.mailbox_archive_folder, - ip_db_path=opts.ip_db_path, - always_use_local_files=opts.always_use_local_files, - reverse_dns_map_path=opts.reverse_dns_map_path, - reverse_dns_map_url=opts.reverse_dns_map_url, - offline=opts.offline, - nameservers=opts.nameservers, test=opts.mailbox_test, - strip_attachment_payloads=opts.strip_attachment_payloads, since=opts.mailbox_since, - dns_retries=opts.dns_retries, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value, + config=parser_config, n_procs=n_procs, ) @@ -2667,18 +2666,9 @@ def _main(): delete=opts.mailbox_delete, test=opts.mailbox_test, check_timeout=mailbox_check_timeout_value, - nameservers=opts.nameservers, - dns_timeout=opts.dns_timeout, - dns_retries=opts.dns_retries, - strip_attachment_payloads=opts.strip_attachment_payloads, batch_size=mailbox_batch_size_value, since=opts.mailbox_since, - ip_db_path=opts.ip_db_path, - always_use_local_files=opts.always_use_local_files, - reverse_dns_map_path=opts.reverse_dns_map_path, - reverse_dns_map_url=opts.reverse_dns_map_url, - offline=opts.offline, - normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value, + config=parser_config, config_reloading=lambda: _reload_requested or _shutdown_requested, n_procs=n_procs, ) @@ -2767,6 +2757,8 @@ def _main(): for k, v in vars(new_opts).items(): setattr(opts, k, v) + parser_config = _build_parser_config(opts) + # Update watch parameters from reloaded config mailbox_batch_size_value = ( int(opts.mailbox_batch_size) @@ -2778,11 +2770,6 @@ def _main(): if opts.mailbox_check_timeout is not None else 30 ) - normalize_timespan_threshold_hours_value = ( - float(opts.normalize_timespan_threshold_hours) - if opts.normalize_timespan_threshold_hours is not None - else 24.0 - ) # Update log level logger.setLevel(logging.ERROR) diff --git a/parsedmarc/config.py b/parsedmarc/config.py new file mode 100644 index 00000000..0a171f2f --- /dev/null +++ b/parsedmarc/config.py @@ -0,0 +1,183 @@ +# -*- coding: utf-8 -*- + +"""Centralized parser configuration (GitHub issue #503). + +This module defines :class:`ParserConfig`, a single frozen dataclass that +carries every parsing/enrichment option (offline mode, IP database path, +reverse DNS map location, PSL overrides location, DNS behavior, etc.) plus +the three caches that the parser and enrichment code share across calls: +the IP address info cache, the seen-aggregate-report-ID cache, and the +reverse DNS map. + +``parsedmarc/__init__.py`` re-exports :data:`IP_ADDRESS_CACHE`, +:data:`SEEN_AGGREGATE_REPORT_IDS`, and :data:`REVERSE_DNS_MAP` under the +same names for backward compatibility. That compatibility is +identity-based: ``parsedmarc.IP_ADDRESS_CACHE is parsedmarc.config.IP_ADDRESS_CACHE`` +must hold, not merely equal contents, since callers may mutate the caches +in place. + +Only :mod:`parsedmarc.constants` and :mod:`parsedmarc.utils` are imported +from the package here — importing :mod:`parsedmarc` itself would be +circular, since ``parsedmarc/__init__.py`` imports from this module. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field, fields + +from expiringdict import ExpiringDict + +from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT +from parsedmarc.utils import ReverseDNSMap + + +def _new_ip_address_cache() -> ExpiringDict: + """Build a fresh IP-address-info cache (4 hour expiry).""" + return ExpiringDict(max_len=10000, max_age_seconds=14400) + + +def _new_seen_aggregate_report_ids() -> ExpiringDict: + """Build a fresh seen-aggregate-report-ID cache (1 hour expiry).""" + return ExpiringDict(max_len=100000000, max_age_seconds=3600) + + +# Canonical default caches. parsedmarc/__init__.py re-exports these three +# objects under the same names (IP_ADDRESS_CACHE, SEEN_AGGREGATE_REPORT_IDS, +# REVERSE_DNS_MAP) for backward compatibility with code that imported them +# directly from the top-level package. That compatibility promise is +# identity-based — the re-exported names must point at these very same +# objects, not merely equivalent ones — so that library callers who obtained +# a reference before this refactor keep observing the same mutations as code +# that goes through a ParserConfig built from these defaults (e.g. during +# unpickling in a multiprocessing worker; see ParserConfig.__setstate__). +IP_ADDRESS_CACHE = _new_ip_address_cache() +SEEN_AGGREGATE_REPORT_IDS = _new_seen_aggregate_report_ids() +REVERSE_DNS_MAP: ReverseDNSMap = {} + +# The ParserConfig fields excluded from pickling; every other field must +# survive the round-trip, so __getstate__ derives its contents from +# dataclasses.fields() rather than enumerating option fields by hand. +_CACHE_FIELD_NAMES = ( + "ip_address_cache", + "seen_aggregate_report_ids", + "reverse_dns_map", +) + + +@dataclass(frozen=True) +class ParserConfig: + """Carries all parsing/enrichment options, plus the caches they share. + + When passed as ``config=`` to parsedmarc's public parsing functions, the + individual option keyword arguments (``offline``, ``ip_db_path``, + ``nameservers``, etc.) are ignored in favor of the values carried on this + object. + + Every explicitly constructed ``ParserConfig()`` owns fresh, isolated + cache objects (``ip_address_cache``, ``seen_aggregate_report_ids``, + ``reverse_dns_map``) via ``default_factory`` — two independently + constructed configs never share cache state. To derive a variant of an + existing config that *does* keep sharing its caches (e.g. to override + ``offline`` for one call while still benefiting from warm caches), use + ``dataclasses.replace(cfg, ...)``: since every field, including the three + cache fields, is an init field, ``replace()`` copies the source's cache + objects onto the new instance rather than constructing fresh ones. + + Pickling (as happens when a config is captured in a + ``functools.partial`` payload submitted to a multiprocessing worker) + intentionally drops cache *contents*: ``__getstate__`` omits the three + cache fields, and ``__setstate__`` rebinds them to this module's + :data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`, and + :data:`REVERSE_DNS_MAP` — the unpickling process's module-default + caches — rather than either the sender's cache contents (which would be + expensive and stale to serialize per task) or brand new empty caches per + unpickle (which would silently defeat per-worker caching, since a + ``functools.partial`` payload is re-pickled for every task submitted to a + worker). Binding the module defaults means a freshly spawned worker + interpreter starts with empty caches and accumulates hits across the + tasks it handles, matching pre-refactor behavior. + + Note that ``keep_alive`` and ``n_procs`` are deliberately **not** fields + on this class: those control process/worker orchestration, not parsing + or enrichment behavior. + + Attributes: + offline: Do not make online requests (DNS, IP database download, + reverse DNS map download, PSL overrides download). + ip_db_path: Path to a local MMDB file from IPinfo, MaxMind, or DBIP. + always_use_local_files: Always use local/bundled files instead of + downloading the IP database, reverse DNS map, or PSL overrides. + reverse_dns_map_path: Path to a local reverse DNS map file. + reverse_dns_map_url: URL to a reverse DNS map file. + psl_overrides_path: Path to a local PSL overrides file. + psl_overrides_url: URL to a PSL overrides file. + nameservers: A list of one or more nameservers to use for DNS + queries (Cloudflare's public DNS resolvers by default). + dns_timeout: DNS query timeout, in seconds. + dns_retries: Number of times to retry a DNS query after a timeout or + other transient error. + strip_attachment_payloads: Remove attachment payloads from parsed + email results. + normalize_timespan_threshold_hours: Aggregate reports whose + timespan exceeds this many hours are normalized/split. + ip_address_cache: Cache of IP address enrichment results. + seen_aggregate_report_ids: Cache of already-seen aggregate report + IDs, used for de-duplication. + reverse_dns_map: The reverse DNS map used to classify base domains + found via reverse DNS. + """ + + offline: bool = False + ip_db_path: str | None = None + always_use_local_files: bool = False + reverse_dns_map_path: str | None = None + reverse_dns_map_url: str | None = None + psl_overrides_path: str | None = None + psl_overrides_url: str | None = None + nameservers: list[str] | None = None + dns_timeout: float = DEFAULT_DNS_TIMEOUT + dns_retries: int = DEFAULT_DNS_MAX_RETRIES + strip_attachment_payloads: bool = False + normalize_timespan_threshold_hours: float = 24.0 + ip_address_cache: ExpiringDict = field( + default_factory=_new_ip_address_cache, repr=False, compare=False + ) + seen_aggregate_report_ids: ExpiringDict = field( + default_factory=_new_seen_aggregate_report_ids, repr=False, compare=False + ) + reverse_dns_map: ReverseDNSMap = field( + default_factory=dict, repr=False, compare=False + ) + + def __getstate__(self) -> dict[str, object]: + """Return picklable state, excluding the three cache fields. + + Cache contents are process-local and potentially huge; they are + never serialized. See the class docstring for the full rationale. + """ + return { + f.name: getattr(self, f.name) + for f in fields(self) + if f.name not in _CACHE_FIELD_NAMES + } + + def __setstate__(self, state: dict[str, object]) -> None: + """Restore option fields, then rebind caches to this module's + process-wide defaults (never fresh empty caches — see the class + docstring for why that would silently break per-worker caching). + + Non-cache fields are first initialized to their class defaults, so + unpickling a ``ParserConfig`` serialized by an older parsedmarc + version (whose state predates fields added since) leaves the newer + fields at their defaults instead of unset entirely — ``__init__`` + never runs during unpickling, so without this an absent field would + raise ``AttributeError`` on first access. + """ + for f in fields(self): + if f.name not in _CACHE_FIELD_NAMES: + object.__setattr__(self, f.name, f.default) + for key, value in state.items(): + object.__setattr__(self, key, value) + object.__setattr__(self, "ip_address_cache", IP_ADDRESS_CACHE) + object.__setattr__(self, "seen_aggregate_report_ids", SEEN_AGGREGATE_REPORT_IDS) + object.__setattr__(self, "reverse_dns_map", REVERSE_DNS_MAP) diff --git a/parsedmarc/parallel.py b/parsedmarc/parallel.py index 5e007afe..ec69de7e 100644 --- a/parsedmarc/parallel.py +++ b/parsedmarc/parallel.py @@ -6,10 +6,11 @@ import logging from collections import deque from collections.abc import Callable, Iterable, Iterator from concurrent.futures import Future, ProcessPoolExecutor, wait -from typing import TYPE_CHECKING, Any, TypeVar +from typing import TYPE_CHECKING, TypeVar if TYPE_CHECKING: from parsedmarc import ParserError + from parsedmarc.config import ParserConfig from parsedmarc.types import ParsedReport _J = TypeVar("_J") @@ -35,14 +36,20 @@ def _init_worker_logging(log_level: int, log_files: list[str]) -> None: def _parse_report_email_job( - msg_content: bytes | str, *, kwargs: dict[str, Any] + msg_content: bytes | str, *, config: ParserConfig ) -> ParsedReport | ParserError: """Worker job that parses a single report email. Module-level and picklable so it can run in a ``ProcessPoolExecutor`` - worker. ``kwargs`` is forwarded to ``parse_report_email`` and must - never contain ``keep_alive`` - it is a bound method of the live - mailbox connection in the parent process and is not picklable. + worker. ``config`` is forwarded to ``parse_report_email``. The config's + caches (``ip_address_cache``, ``seen_aggregate_report_ids``, + ``reverse_dns_map``) never cross the process boundary - + ``ParserConfig.__getstate__`` drops them, and each worker accumulates + its own via the module defaults it rebinds to on unpickling (see + ``ParserConfig.__setstate__``). ``keep_alive`` is not a ``ParserConfig`` + field - it is a bound method of the live mailbox connection in the + parent process and is not picklable - so nothing unpicklable is ever + submitted to the pool. Returns the parsed report on success. A ``ParserError`` raised by ``parse_report_email`` is caught and returned as a value (never @@ -53,18 +60,18 @@ def _parse_report_email_job( from parsedmarc import ParserError, parse_report_email try: - return parse_report_email(msg_content, **kwargs) + return parse_report_email(msg_content, config=config) except ParserError as e: return e def _parse_report_file_job( - file_path: str, *, kwargs: dict[str, Any] + file_path: str, *, config: ParserConfig ) -> tuple[str, ParsedReport | Exception]: """Worker job that parses a single report file. Module-level and picklable so it can run in a ``ProcessPoolExecutor`` - worker. ``kwargs`` is forwarded to ``parse_report_file``. + worker. ``config`` is forwarded to ``parse_report_file``. Catches any ``Exception`` (not just ``ParserError``) and returns it paired with ``file_path`` rather than letting it propagate. This is a @@ -77,7 +84,7 @@ def _parse_report_file_job( from parsedmarc import parse_report_file try: - return file_path, parse_report_file(file_path, **kwargs) + return file_path, parse_report_file(file_path, config=config) except Exception as e: return file_path, e diff --git a/parsedmarc/utils.py b/parsedmarc/utils.py index 775570cc..1737a141 100644 --- a/parsedmarc/utils.py +++ b/parsedmarc/utils.py @@ -842,6 +842,8 @@ def get_service_from_reverse_dns_base_domain( url: str | None = None, offline: bool = False, reverse_dns_map: ReverseDNSMap | None = None, + psl_overrides_path: str | None = None, + psl_overrides_url: str | None = None, ) -> ReverseDNSService: """ Returns the service name of a given base domain name from reverse DNS. @@ -850,9 +852,11 @@ def get_service_from_reverse_dns_base_domain( base_domain (str): The base domain of the reverse DNS lookup always_use_local_file (bool): Always use a local map file local_file_path (str): Path to a local map file - url (str): URL ro a reverse DNS map + url (str): URL to a reverse DNS map offline (bool): Use the built-in copy of the reverse DNS map reverse_dns_map (dict): A reverse DNS map + psl_overrides_path (str): Path to a local PSL overrides file + psl_overrides_url (str): URL to a PSL overrides file Returns: dict: A dictionary containing name and type. If the service is unknown, the name will be @@ -873,6 +877,8 @@ def get_service_from_reverse_dns_base_domain( local_file_path=local_file_path, url=url, offline=offline, + psl_overrides_path=psl_overrides_path, + psl_overrides_url=psl_overrides_url, ) service: ReverseDNSService @@ -897,6 +903,8 @@ def get_ip_address_info( nameservers: list[str] | None = None, timeout: float = DEFAULT_DNS_TIMEOUT, retries: int = DEFAULT_DNS_MAX_RETRIES, + psl_overrides_path: str | None = None, + psl_overrides_url: str | None = None, ) -> IPAddressInfo: """ Returns reverse DNS and country information for the given IP address @@ -915,6 +923,8 @@ def get_ip_address_info( timeout (float): Sets the DNS timeout in seconds retries (int): Number of times to retry on timeout or other transient errors + psl_overrides_path (str): Path to a local PSL overrides file + psl_overrides_url (str): URL to a PSL overrides file Returns: dict: ``ip_address``, ``reverse_dns``, ``country`` @@ -967,6 +977,8 @@ def get_ip_address_info( url=reverse_dns_map_url, always_use_local_file=always_use_local_files, reverse_dns_map=reverse_dns_map, + psl_overrides_path=psl_overrides_path, + psl_overrides_url=psl_overrides_url, ) info["base_domain"] = base_domain info["type"] = service["type"] @@ -986,6 +998,8 @@ def get_ip_address_info( local_file_path=reverse_dns_map_path, url=reverse_dns_map_url, offline=offline, + psl_overrides_path=psl_overrides_path, + psl_overrides_url=psl_overrides_url, ) if info["as_domain"] and info["as_domain"] in map_value: service = map_value[info["as_domain"]] diff --git a/tests/test_cli.py b/tests/test_cli.py index e5afd995..de067b76 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1236,6 +1236,108 @@ since = 2d self.assertEqual(mock_watch_inbox.call_args.kwargs.get("since"), "2d") +class TestCliParserConfigWiring(unittest.TestCase): + """Tests that _main() builds a single ParserConfig (via + _build_parser_config) from parsed opts and passes it as ``config=`` to + the library's mailbox-fetching functions, rather than forwarding + individual option kwargs (offline, dns_timeout, ip_db_path, etc.) by + hand at each call site.""" + + def setUp(self): + from parsedmarc.log import logger as _logger + + _logger.disabled = True + self._stdout_patch = patch("sys.stdout", new_callable=io.StringIO) + self._stderr_patch = patch("sys.stderr", new_callable=io.StringIO) + self._stdout_patch.start() + self._stderr_patch.start() + # SEEN_AGGREGATE_REPORT_IDS is a module-level ExpiringDict shared + # across tests in this process; clear it both ways so state from an + # earlier test class doesn't leak in, and so this class doesn't leak + # into a later one. Precedent: TestDirectoryFilePaths.setUp above. + parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear() + self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear) + + def tearDown(self): + from parsedmarc.log import logger as _logger + + _logger.disabled = False + self._stderr_patch.stop() + self._stdout_patch.stop() + + def _run_one_shot_mailbox( + self, dns_timeout: float | None = None, dns_retries: int | None = None + ) -> parsedmarc.ParserConfig: + """Runs a real one-shot _main() against a mocked IMAP connection and + mocked get_dmarc_reports_from_mailbox, and returns the ParserConfig + the CLI passed as ``config=``.""" + config_lines = ["[general]", "silent = true"] + if dns_timeout is not None: + config_lines.append(f"dns_timeout = {dns_timeout}") + if dns_retries is not None: + config_lines.append(f"dns_retries = {dns_retries}") + config_lines += [ + "", + "[imap]", + "host = imap.example.com", + "user = user", + "password = pass", + ] + config_text = "\n".join(config_lines) + "\n" + + with tempfile.NamedTemporaryFile("w", suffix=".ini", delete=False) as cfg: + cfg.write(config_text) + cfg_path = cfg.name + self.addCleanup(lambda: os.path.exists(cfg_path) and os.remove(cfg_path)) + + with ( + patch("parsedmarc.cli.get_dmarc_reports_from_mailbox") as mock_get_reports, + patch("parsedmarc.cli.IMAPConnection") as mock_imap, + ): + mock_imap.return_value = object() + mock_get_reports.return_value = { + "aggregate_reports": [], + "failure_reports": [], + "smtp_tls_reports": [], + } + with patch.object(sys, "argv", ["parsedmarc", "-c", cfg_path]): + parsedmarc.cli._main() + return mock_get_reports.call_args.kwargs["config"] + + def test_one_shot_mailbox_run_honors_general_dns_timeout(self): + """Regression test for the one-shot mailbox call site in _main(): + before this fix, that call to get_dmarc_reports_from_mailbox + omitted the dns_timeout/dns_retries kwargs entirely (and, before + config= existed, had no way to pass them at all), so a one-shot run + silently used the library's own hardcoded default + (``dns_timeout: float = 6.0`` on master's + get_dmarc_reports_from_mailbox signature) instead of the operator's + ``[general] dns_timeout`` / ``dns_retries`` config values. Watch-mode + runs were unaffected because the watch_inbox call site did pass + dns_timeout/dns_retries directly. + """ + cfg = self._run_one_shot_mailbox(dns_timeout=11.5, dns_retries=3) + self.assertEqual(cfg.dns_timeout, 11.5) + self.assertEqual(cfg.dns_retries, 3) + + def test_cli_config_binds_module_default_caches(self): + """The ParserConfig built by the CLI must bind the process-wide + default caches (parsedmarc.IP_ADDRESS_CACHE, + parsedmarc.SEEN_AGGREGATE_REPORT_IDS, parsedmarc.REVERSE_DNS_MAP) by + identity, not fresh/isolated caches — otherwise every CLI run would + get its own empty caches (defeating the point of the 4-hour IP + cache and the 1-hour dedup cache) even though ParserConfig's default + factories exist specifically to give library callers isolated + caches when they don't pass config=. + """ + cfg = self._run_one_shot_mailbox() + self.assertIs(cfg.ip_address_cache, parsedmarc.IP_ADDRESS_CACHE) + self.assertIs( + cfg.seen_aggregate_report_ids, parsedmarc.SEEN_AGGREGATE_REPORT_IDS + ) + self.assertIs(cfg.reverse_dns_map, parsedmarc.REVERSE_DNS_MAP) + + class TestMailboxPerformance(unittest.TestCase): def setUp(self): from parsedmarc.log import logger as _logger @@ -3035,6 +3137,91 @@ watch = true "Stale entry should have been cleared by reload", ) + @unittest.skipUnless( + hasattr(signal, "SIGHUP"), + "SIGHUP not available on this platform", + ) + @patch("parsedmarc.cli._init_output_clients") + @patch("parsedmarc.cli._parse_config") + @patch("parsedmarc.cli._load_config") + @patch("parsedmarc.cli.get_dmarc_reports_from_mailbox") + @patch("parsedmarc.cli.watch_inbox") + @patch("parsedmarc.cli.IMAPConnection") + def test_sighup_reload_rebuilds_parser_config( + self, + mock_imap, + mock_watch, + mock_get_reports, + mock_load_config, + mock_parse_config, + mock_init_clients, + ): + """After a SIGHUP reload, the ParserConfig passed to watch_inbox as + ``config=`` must reflect the reloaded ``[general] dns_timeout``, not + the value from the initial config load. + + Guards against _build_parser_config(opts) being called only once at + startup: opts itself is correctly refreshed in place by the existing + ``for k, v in vars(new_opts).items(): setattr(opts, k, v)`` loop, but + parser_config is a separate ParserConfig snapshot built from opts — + if the reload path forgot to rebuild it, watch_inbox would keep + receiving the stale pre-reload config object forever. + """ + import signal as signal_module + + mock_imap.return_value = object() + mock_get_reports.return_value = { + "aggregate_reports": [], + "failure_reports": [], + "smtp_tls_reports": [], + } + + mock_load_config.return_value = ConfigParser() + + parse_calls = [0] + + def parse_side_effect(config, opts): + parse_calls[0] += 1 + opts.imap_host = "imap.example.com" + opts.imap_user = "user" + opts.imap_password = "pass" + opts.mailbox_watch = True + opts.dns_timeout = 5.0 if parse_calls[0] == 1 else 42.0 + return None + + mock_parse_config.side_effect = parse_side_effect + mock_init_clients.return_value = {} + + watch_calls = [0] + + def watch_side_effect(*args, **kwargs): + watch_calls[0] += 1 + if watch_calls[0] == 1: + if hasattr(signal_module, "SIGHUP"): + import os + + os.kill(os.getpid(), signal_module.SIGHUP) + return + else: + raise FileExistsError("stop-watch-loop") + + mock_watch.side_effect = watch_side_effect + + with tempfile.NamedTemporaryFile("w", suffix=".ini", delete=False) as cfg: + cfg.write(self._BASE_CONFIG) + cfg_path = cfg.name + self.addCleanup(lambda: os.path.exists(cfg_path) and os.remove(cfg_path)) + + with patch.object(sys, "argv", ["parsedmarc", "-c", cfg_path]): + with self.assertRaises(SystemExit): + parsedmarc.cli._main() + + self.assertEqual(mock_watch.call_count, 2) + first_config = mock_watch.call_args_list[0].kwargs["config"] + second_config = mock_watch.call_args_list[1].kwargs["config"] + self.assertEqual(first_config.dns_timeout, 5.0) + self.assertEqual(second_config.dns_timeout, 42.0) + class TestSigtermShutdown(unittest.TestCase): """Tests for graceful SIGTERM/SIGINT shutdown.""" diff --git a/tests/test_config.py b/tests/test_config.py new file mode 100644 index 00000000..f721916b --- /dev/null +++ b/tests/test_config.py @@ -0,0 +1,169 @@ +"""Tests for parsedmarc.config""" + +import dataclasses +import pickle +import unittest + +import parsedmarc +import parsedmarc.config as parsedmarc_config + + +class TestParserConfigCaches(unittest.TestCase): + """Covers per-instance cache isolation for `ParserConfig`.""" + + def test_each_instance_gets_isolated_caches(self): + """Two independently constructed ParserConfig instances must never + share cache objects with each other or with the module defaults, and + mutating one instance's caches must not leak into the others.""" + cfg_a = parsedmarc_config.ParserConfig() + cfg_b = parsedmarc_config.ParserConfig() + + # All distinct objects. + self.assertIsNot(cfg_a.ip_address_cache, cfg_b.ip_address_cache) + self.assertIsNot( + cfg_a.seen_aggregate_report_ids, cfg_b.seen_aggregate_report_ids + ) + self.assertIsNot(cfg_a.reverse_dns_map, cfg_b.reverse_dns_map) + + self.assertIsNot(cfg_a.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE) + self.assertIsNot( + cfg_a.seen_aggregate_report_ids, + parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS, + ) + self.assertIsNot(cfg_a.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP) + + self.assertIsNot(cfg_b.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE) + self.assertIsNot( + cfg_b.seen_aggregate_report_ids, + parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS, + ) + self.assertIsNot(cfg_b.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP) + + # Mutating one instance's caches must not affect the other, nor the + # module defaults. + cfg_a.ip_address_cache["1.2.3.4"] = {"ip_address": "1.2.3.4"} + cfg_a.seen_aggregate_report_ids["report-id-a"] = True + cfg_a.reverse_dns_map["example.com"] = {"name": "Example", "type": None} + + self.assertNotIn("1.2.3.4", cfg_b.ip_address_cache) + self.assertNotIn("report-id-a", cfg_b.seen_aggregate_report_ids) + self.assertNotIn("example.com", cfg_b.reverse_dns_map) + + self.assertNotIn("1.2.3.4", parsedmarc_config.IP_ADDRESS_CACHE) + self.assertNotIn("report-id-a", parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS) + self.assertNotIn("example.com", parsedmarc_config.REVERSE_DNS_MAP) + + def test_module_default_caches_are_the_public_globals(self): + """The three module-default caches in `parsedmarc.config` must be the + very same objects re-exported as `parsedmarc.IP_ADDRESS_CACHE`, + `parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, and + `parsedmarc.REVERSE_DNS_MAP`, so that pre-refactor callers who + imported these names directly from the top-level package keep + observing the same cache objects as code that goes through + ParserConfig. + """ + self.assertIs(parsedmarc.IP_ADDRESS_CACHE, parsedmarc_config.IP_ADDRESS_CACHE) + self.assertIs( + parsedmarc.SEEN_AGGREGATE_REPORT_IDS, + parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS, + ) + self.assertIs(parsedmarc.REVERSE_DNS_MAP, parsedmarc_config.REVERSE_DNS_MAP) + + +class TestParserConfigPickling(unittest.TestCase): + """Covers the pickle strategy documented on `ParserConfig`: option fields + round-trip, but cache contents never cross process boundaries and the + unpickled object's caches rebind to the unpickling process's module + defaults.""" + + def test_pickle_drops_cache_contents_and_binds_process_defaults(self): + """Pickling and unpickling a ParserConfig must preserve option + fields, but must NOT carry cache contents across the round-trip, and + must leave the unpickled instance's caches bound to this module's + (the "unpickling process's") default cache objects rather than + fresh, empty ones. Fresh-per-unpickle caches would silently defeat + per-worker caching, since a functools.partial payload carrying a + ParserConfig is re-pickled for every task submitted to a + multiprocessing worker.""" + cfg = parsedmarc_config.ParserConfig( + dns_timeout=9.5, nameservers=["192.0.2.53"] + ) + cfg.ip_address_cache["sentinel-ip"] = "sentinel-ip-value" + cfg.seen_aggregate_report_ids["sentinel-report-id"] = True + cfg.reverse_dns_map["sentinel.example"] = { + "name": "Sentinel", + "type": None, + } + + restored = pickle.loads(pickle.dumps(cfg)) + + # Option fields compare equal (cache fields are compare=False). + self.assertEqual(cfg, restored) + self.assertEqual(restored.dns_timeout, 9.5) + self.assertEqual(restored.nameservers, ["192.0.2.53"]) + + # Caches rebind to this process's module defaults, not fresh copies. + self.assertIs(restored.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE) + self.assertIs( + restored.seen_aggregate_report_ids, + parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS, + ) + self.assertIs(restored.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP) + + # Sentinel entries placed on the original instance's caches must NOT + # have crossed into the module defaults. + self.assertNotIn("sentinel-ip", parsedmarc_config.IP_ADDRESS_CACHE) + self.assertNotIn( + "sentinel-report-id", parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS + ) + self.assertNotIn("sentinel.example", parsedmarc_config.REVERSE_DNS_MAP) + + def test_setstate_defaults_fields_missing_from_older_pickles(self): + """__setstate__ must initialize every non-cache field to its class + default before applying the pickled state, so a ParserConfig + serialized by an older parsedmarc version (whose state lacks fields + added since) unpickles with the newer fields at their defaults + instead of unset — __init__ never runs during unpickling, so an + absent field would otherwise raise AttributeError on first access. + Simulated by calling __setstate__ directly with a partial state + dict, exactly what pickle.loads does with an old payload.""" + restored = object.__new__(parsedmarc_config.ParserConfig) + restored.__setstate__({"offline": True, "dns_timeout": 7.5}) + + self.assertTrue(restored.offline) + self.assertEqual(restored.dns_timeout, 7.5) + # Fields absent from the old state get their class defaults. + self.assertIsNone(restored.ip_db_path) + self.assertIsNone(restored.nameservers) + self.assertEqual(restored.normalize_timespan_threshold_hours, 24.0) + # Caches still rebind to the module defaults. + self.assertIs(restored.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE) + self.assertIs( + restored.seen_aggregate_report_ids, + parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS, + ) + self.assertIs(restored.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP) + + +class TestParserConfigFrozenAndReplace(unittest.TestCase): + """Covers frozen-dataclass immutability and the `dataclasses.replace` + escape hatch for deriving variants that keep sharing caches.""" + + def test_frozen_and_replace_shares_caches(self): + """ParserConfig is frozen, so attribute assignment must raise + FrozenInstanceError. `dataclasses.replace(cfg, ...)` must return a + new config that shares the SAME cache objects as the source (all + fields, including the three cache fields, are init fields), since + that is the documented way to derive a variant that keeps warm + caches.""" + cfg = parsedmarc_config.ParserConfig() + + with self.assertRaises(dataclasses.FrozenInstanceError): + cfg.offline = True # type: ignore[misc] + + variant = dataclasses.replace(cfg, offline=True) + + self.assertTrue(variant.offline) + self.assertIs(variant.ip_address_cache, cfg.ip_address_cache) + self.assertIs(variant.seen_aggregate_report_ids, cfg.seen_aggregate_report_ids) + self.assertIs(variant.reverse_dns_map, cfg.reverse_dns_map) diff --git a/tests/test_init.py b/tests/test_init.py index ed8ec7a8..9a79e51f 100644 --- a/tests/test_init.py +++ b/tests/test_init.py @@ -7,6 +7,7 @@ extract_report, get_dmarc_reports_from_mbox, and the CSV / JSON renderers. import base64 import gzip +import inspect import json import logging import mailbox @@ -24,6 +25,7 @@ from unittest.mock import MagicMock, patch from lxml import etree # type: ignore[import-untyped] import parsedmarc +import parsedmarc.constants as constants from parsedmarc.mail import MaildirConnection, MSGraphConnection from parsedmarc.types import ( AggregateReport, @@ -749,7 +751,9 @@ class Test(unittest.TestCase): "auth_results": {"dkim": [], "spf": []}, } with self.assertRaises(ValueError): - parsedmarc._parse_report_record(record, offline=True) + parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) def testParseReportRecordMissingDkimSpf(self): """Record with missing dkim/spf auth results defaults correctly""" @@ -766,7 +770,9 @@ class Test(unittest.TestCase): "identifiers": {"header_from": "example.com"}, "auth_results": {}, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertEqual(result["auth_results"]["dkim"], []) self.assertEqual(result["auth_results"]["spf"], []) @@ -786,7 +792,9 @@ class Test(unittest.TestCase): "identifiers": {"header_from": "example.com"}, "auth_results": {"dkim": [], "spf": []}, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) reasons = result["policy_evaluated"]["policy_override_reasons"] self.assertEqual(len(reasons), 1) self.assertEqual(reasons[0]["type"], "forwarded") @@ -811,7 +819,9 @@ class Test(unittest.TestCase): "identifiers": {"header_from": "example.com"}, "auth_results": {"dkim": [], "spf": []}, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) reasons = result["policy_evaluated"]["policy_override_reasons"] self.assertEqual(len(reasons), 2) self.assertEqual(reasons[0]["comment"], "relay") @@ -835,7 +845,9 @@ class Test(unittest.TestCase): }, "auth_results": {"dkim": [], "spf": []}, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertIn("identifiers", result) self.assertEqual(result["identifiers"]["header_from"], "example.com") @@ -857,7 +869,9 @@ class Test(unittest.TestCase): "spf": [], }, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) dkim = result["auth_results"]["dkim"][0] self.assertEqual(dkim["selector"], "none") self.assertEqual(dkim["result"], "none") @@ -881,7 +895,9 @@ class Test(unittest.TestCase): "spf": {"domain": "example.com"}, }, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) spf = result["auth_results"]["spf"][0] self.assertEqual(spf["scope"], "mfrom") self.assertEqual(spf["result"], "none") @@ -919,7 +935,9 @@ class Test(unittest.TestCase): ], }, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertEqual(result["auth_results"]["dkim"][0]["human_result"], "good key") self.assertEqual( result["auth_results"]["spf"][0]["human_result"], "sender valid" @@ -945,7 +963,9 @@ class Test(unittest.TestCase): ], }, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertEqual(result["identifiers"]["envelope_from"], "bounce.example.com") def testParseReportRecordEnvelopeFromNullFallback(self): @@ -971,7 +991,9 @@ class Test(unittest.TestCase): ], }, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertEqual(result["identifiers"]["envelope_from"], "spf.example.com") def testParseReportRecordEnvelopeFromNullNoSpfDomain(self): @@ -998,7 +1020,9 @@ class Test(unittest.TestCase): "spf": [{"scope": "mfrom", "result": "pass"}], }, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertIsNone(result["identifiers"]["envelope_from"]) def testParseReportRecordEnvelopeTo(self): @@ -1020,7 +1044,9 @@ class Test(unittest.TestCase): }, "auth_results": {"dkim": [], "spf": []}, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertEqual(result["identifiers"]["envelope_to"], "recipient@example.com") def testParseReportRecordAlignment(self): @@ -1038,7 +1064,9 @@ class Test(unittest.TestCase): "identifiers": {"header_from": "example.com"}, "auth_results": {"dkim": [], "spf": []}, } - result = parsedmarc._parse_report_record(record, offline=True) + result = parsedmarc._parse_report_record( + record, config=parsedmarc.ParserConfig(offline=True) + ) self.assertTrue(result["alignment"]["dkim"]) self.assertFalse(result["alignment"]["spf"]) self.assertTrue(result["alignment"]["dmarc"]) @@ -2709,6 +2737,214 @@ class TestGetDmarcReportsFromMboxParallel(unittest.TestCase): ) +class TestCentralizedConfig(unittest.TestCase): + """Regression coverage for the centralize-config-503 refactor + (parsedmarc/config.py's ``ParserConfig``): the kwargs-style public API + must keep observing/mutating the same module-default caches it always + has (via ``_resolve_config`` injecting ``IP_ADDRESS_CACHE`` / + ``SEEN_AGGREGATE_REPORT_IDS`` / ``REVERSE_DNS_MAP`` rather than letting + a fresh ``ParserConfig()`` default-factory hand back empty ones), an + explicit ``config=`` must win over individual option keyword arguments + when both are given, and the DNS-timeout/retry/normalize-threshold + defaults must stay consistent between each public function's own + signature and ``ParserConfig``'s field defaults. + """ + + AGGREGATE = "samples/aggregate/twilight.eml" + + def _build_single_aggregate_mbox(self) -> str: + """Builds a temporary mbox containing one copy of AGGREGATE.""" + tmp = mkdtemp() + self.addCleanup(rmtree, tmp, ignore_errors=True) + path = os.path.join(tmp, "reports.mbox") + box = mailbox.mbox(path) + box.lock() + try: + with open(self.AGGREGATE, "rb") as source_file: + box.add(mailbox.mboxMessage(source_file.read())) + box.flush() + finally: + box.unlock() + box.close() + return path + + def test_kwargs_path_uses_module_default_caches_not_fresh_ones(self): + """Regression guard for _resolve_config: calling + get_dmarc_reports_from_mbox with plain kwargs (no config=) twice in + a row over the same mbox must dedup the second run's aggregate + report against the first run's, because both calls must resolve to + the SAME module-default parsedmarc.SEEN_AGGREGATE_REPORT_IDS cache. + If _resolve_config ever let the kwargs path fall through to + ParserConfig's default_factory instead of explicitly injecting the + module-default caches, each call would get a fresh, empty cache and + this dedup would silently stop working. + """ + parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear() + self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear) + path = self._build_single_aggregate_mbox() + + first = parsedmarc.get_dmarc_reports_from_mbox(path, offline=True) + second = parsedmarc.get_dmarc_reports_from_mbox(path, offline=True) + + self.assertEqual(len(first["aggregate_reports"]), 1) + self.assertEqual(len(second["aggregate_reports"]), 0) + + report_metadata = first["aggregate_reports"][0]["report_metadata"] + report_key = f"{report_metadata['org_name']}_{report_metadata['report_id']}" + self.assertIn(report_key, parsedmarc.SEEN_AGGREGATE_REPORT_IDS) + + def test_kwargs_and_config_equivalence(self): + """parse_report_file must produce identical results whether called + with individual option keyword arguments or an equivalent explicit + ParserConfig, for one sample of each report type. None of these + paths touch dedup (that's only in _classify_parsed_email / + get_dmarc_reports_from_mbox / get_dmarc_reports_from_mailbox), so + no cache clearing is needed between calls. + """ + sample_paths = [ + "samples/aggregate/rfc9990-sample.xml", + "samples/failure/dmarc_ruf_report_linkedin.eml", + "samples/smtp_tls/google.com_smtp_tls_report.eml", + ] + for sample_path in sample_paths: + with self.subTest(sample=sample_path): + kwargs_result = parsedmarc.parse_report_file( + sample_path, offline=True, always_use_local_files=True + ) + config_result = parsedmarc.parse_report_file( + sample_path, + config=parsedmarc.ParserConfig( + offline=True, always_use_local_files=True + ), + ) + self.assertEqual(kwargs_result, config_result) + + def test_config_wins_over_kwargs_normalize_threshold(self): + """When both config= and normalize_timespan_threshold_hours= are + given, the config's value must win -- per the documented contract, + an explicit config makes the individual option kwargs inert. + + samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml + spans exactly 86400 seconds (24h), so a 1.0-hour threshold + normalizes it and a 1000-hour threshold does not; this makes the + config-vs-kwarg outcome observable in normalized_timespan. + """ + sample_path = "samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml" + with open(sample_path, "rb") as f: + data = f.read() + + # Config's high threshold must win over the kwarg's low threshold: + # the report must NOT be normalized. + report = parsedmarc.parse_aggregate_report_file( + data, + offline=True, + config=parsedmarc.ParserConfig( + offline=True, normalize_timespan_threshold_hours=1000.0 + ), + normalize_timespan_threshold_hours=1.0, + ) + for record in report["records"]: + self.assertFalse(record["normalized_timespan"]) # type: ignore[typeddict-item] + + # Converse: config's low threshold must win over the kwarg's high + # threshold: the report MUST be normalized. + report = parsedmarc.parse_aggregate_report_file( + data, + offline=True, + config=parsedmarc.ParserConfig( + offline=True, normalize_timespan_threshold_hours=1.0 + ), + normalize_timespan_threshold_hours=1000.0, + ) + for record in report["records"]: + self.assertTrue(record["normalized_timespan"]) # type: ignore[typeddict-item] + + def test_separate_configs_isolate_dedup_state(self): + """Two independently constructed ParserConfig(offline=True) + instances must NOT share dedup state (each has its own + seen_aggregate_report_ids cache), but reusing the SAME instance + across two calls must dedup, exactly like the module-default-cache + kwargs path does. + """ + path = self._build_single_aggregate_mbox() + + cfg_a = parsedmarc.ParserConfig(offline=True) + cfg_b = parsedmarc.ParserConfig(offline=True) + result_a = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_a) + result_b = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_b) + self.assertEqual(len(result_a["aggregate_reports"]), 1) + self.assertEqual(len(result_b["aggregate_reports"]), 1) + + cfg_c = parsedmarc.ParserConfig(offline=True) + first = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_c) + second = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_c) + self.assertEqual(len(first["aggregate_reports"]), 1) + self.assertEqual(len(second["aggregate_reports"]), 0) + + def test_dns_and_normalize_defaults_match_constants_and_parser_config(self): + """DNS timeout/retries and normalize-timespan-threshold defaults + must match parsedmarc.constants (the authoritative source -- + DEFAULT_DNS_TIMEOUT, DEFAULT_DNS_MAX_RETRIES) and + parsedmarc.config.ParserConfig's own field defaults, across every + public function that accepts them. + + Regression guard: before this refactor, get_dmarc_reports_from_mailbox + and watch_inbox each had a stray literal ``dns_timeout=6.0`` (instead + of ``DEFAULT_DNS_TIMEOUT == 2.0``) and + ``normalize_timespan_threshold_hours=24`` (an int, instead of the + float ``24.0`` used everywhere else) -- exactly the kind of drift + _resolve_config's shared construction path is meant to prevent from + silently reappearing. + """ + functions_and_dns_params = [ + (parsedmarc.parse_aggregate_report_xml, "timeout", "retries"), + (parsedmarc.parse_aggregate_report_file, "dns_timeout", "dns_retries"), + (parsedmarc.parse_failure_report, "dns_timeout", "dns_retries"), + (parsedmarc.parse_report_email, "dns_timeout", "dns_retries"), + (parsedmarc.parse_report_file, "dns_timeout", "dns_retries"), + (parsedmarc.get_dmarc_reports_from_mbox, "dns_timeout", "dns_retries"), + ( + parsedmarc.get_dmarc_reports_from_mailbox, + "dns_timeout", + "dns_retries", + ), + (parsedmarc.watch_inbox, "dns_timeout", "dns_retries"), + ] + + default_config = parsedmarc.ParserConfig() + + for func, timeout_param, retries_param in functions_and_dns_params: + with self.subTest(func=func.__name__, param="dns"): + sig = inspect.signature(func) + timeout_default = sig.parameters[timeout_param].default + retries_default = sig.parameters[retries_param].default + self.assertEqual(timeout_default, constants.DEFAULT_DNS_TIMEOUT) + self.assertEqual(retries_default, constants.DEFAULT_DNS_MAX_RETRIES) + self.assertEqual(timeout_default, default_config.dns_timeout) + self.assertEqual(retries_default, default_config.dns_retries) + + # parse_failure_report has no normalize_timespan_threshold_hours + # parameter -- normalization is an aggregate-report-only concept. + normalize_funcs = [ + parsedmarc.parse_aggregate_report_xml, + parsedmarc.parse_aggregate_report_file, + parsedmarc.parse_report_email, + parsedmarc.parse_report_file, + parsedmarc.get_dmarc_reports_from_mbox, + parsedmarc.get_dmarc_reports_from_mailbox, + parsedmarc.watch_inbox, + ] + for func in normalize_funcs: + with self.subTest(func=func.__name__, param="normalize"): + sig = inspect.signature(func) + default = sig.parameters["normalize_timespan_threshold_hours"].default + self.assertEqual(default, 24.0) + self.assertIsInstance(default, float) + self.assertEqual( + default, default_config.normalize_timespan_threshold_hours + ) + + class TestGetDmarcReportsFromMailboxValidation(unittest.TestCase): """Input validation on get_dmarc_reports_from_mailbox. diff --git a/tests/test_parallel.py b/tests/test_parallel.py index fdedb48f..9e3e8b07 100644 --- a/tests/test_parallel.py +++ b/tests/test_parallel.py @@ -68,7 +68,9 @@ class TestParallelMapParseReportFile(_ParallelTestCase): for path in SAMPLE_PATHS ] - job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True)) + job = functools.partial( + _parse_report_file_job, config=parsedmarc.ParserConfig(offline=True) + ) results = list(parallel_map(job, SAMPLE_PATHS, n_procs=2)) self.assertEqual(len(results), len(SAMPLE_PATHS)) @@ -94,7 +96,9 @@ class TestParallelMapJunkFile(_ParallelTestCase): junk_path = tf.name self.addCleanup(os.remove, junk_path) - job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True)) + job = functools.partial( + _parse_report_file_job, config=parsedmarc.ParserConfig(offline=True) + ) results = list(parallel_map(job, [junk_path, junk_path], n_procs=2)) self.assertEqual(len(results), 2) @@ -208,7 +212,9 @@ class TestWorkerLogging(_ParallelTestCase): configure_logging(logging.DEBUG, log_path) sample = SAMPLE_PATHS[0] - job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True)) + job = functools.partial( + _parse_report_file_job, config=parsedmarc.ParserConfig(offline=True) + ) results = list(parallel_map(job, [sample], n_procs=2)) self.assertEqual(len(results), 1) @@ -275,7 +281,7 @@ class TestParseReportEmailJob(_ParallelTestCase): def test_invalid_email_returns_parser_error_value(self): result = _parse_report_email_job( - b"not a valid email", kwargs=dict(offline=True) + b"not a valid email", config=parsedmarc.ParserConfig(offline=True) ) self.assertIsInstance(result, parsedmarc.ParserError) diff --git a/tests/test_utils.py b/tests/test_utils.py index cd475e87..4e0fe8aa 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -490,6 +490,67 @@ class TestLoadReverseDnsMapReloadsPSLOverrides(unittest.TestCase): offline=True, ) + def _write_psl_overrides_file(self): + """Write a temp PSL overrides file with a unique suffix and register + cleanup, returning its path.""" + tf = tempfile.NamedTemporaryFile( + "w", suffix=".txt", delete=False, encoding="utf-8" + ) + tf.write(".internal.example-503.net\n") + tf.close() + self.addCleanup(os.unlink, tf.name) + return tf.name + + def test_lazy_reverse_dns_map_load_applies_psl_overrides(self): + """Regression test for GitHub issue #503: the lazy + ``load_reverse_dns_map()`` call inside + ``get_service_from_reverse_dns_base_domain`` previously omitted + ``psl_overrides_path``/``psl_overrides_url``, so + ``load_reverse_dns_map``'s unconditional ``load_psl_overrides()`` + call (utils.py) silently reloaded ``psl_overrides`` with the bundled + defaults, discarding an operator-configured overrides file. This + proves the lazy load now threads the caller's + ``psl_overrides_path`` through: after calling with an empty + ``reverse_dns_map`` (which forces the lazy load) and a custom + ``psl_overrides_path``, the custom override must be in effect. + """ + path = self._write_psl_overrides_file() + parsedmarc.utils.get_service_from_reverse_dns_base_domain( + "something.example", + reverse_dns_map={}, + always_use_local_file=True, + offline=True, + psl_overrides_path=path, + ) + self.assertEqual( + parsedmarc.utils.get_base_domain("deep.sub.internal.example-503.net"), + "internal.example-503.net", + ) + + def test_get_ip_address_info_lazy_load_applies_psl_overrides(self): + """Regression test for GitHub issue #503, exercising the + ASN-fallback lazy ``load_reverse_dns_map()`` call inside + ``get_ip_address_info`` (used when no PTR record resolves). Before + the fix, this lazy load also omitted ``psl_overrides_path``/ + ``psl_overrides_url``, so it clobbered operator-configured PSL + overrides with the bundled defaults the same way. ``offline=True`` + forces the no-PTR path so this call reaches the ASN-fallback lazy + load rather than the PTR-driven + ``get_service_from_reverse_dns_base_domain`` call. + """ + path = self._write_psl_overrides_file() + parsedmarc.utils.get_ip_address_info( + "192.0.2.1", + offline=True, + reverse_dns_map={}, + always_use_local_files=True, + psl_overrides_path=path, + ) + self.assertEqual( + parsedmarc.utils.get_base_domain("deep.sub.internal.example-503.net"), + "internal.example-503.net", + ) + class TestGetBaseDomainWithOverrides(unittest.TestCase): """`get_base_domain` must honour the current psl_overrides list."""