Centralize configuration handling with ParserConfig (#503) (#851)

* Centralize configuration handling with ParserConfig (#503)

Add parsedmarc/config.py with ParserConfig, a frozen dataclass carrying
every parsing/enrichment option plus the three shared caches (IP address
info, seen aggregate report IDs, reverse DNS map). All eight public
parsing/mailbox functions accept a keyword-only config= argument; when
provided, the individual option keyword arguments are ignored in favor
of the config's values, and every existing per-option keyword argument
keeps working unchanged. The three hand-copied parse_kwargs dicts and
the dns_timeout<->timeout rename chain are gone; the CLI builds one
ParserConfig per run (rebuilt on SIGHUP) and passes it everywhere.

Explicitly constructed configs own fresh isolated caches;
dataclasses.replace() shares them; pickling drops cache contents and
rebinds the unpickling process's module defaults, preserving the
per-worker cache behavior of n_procs parallel parsing. The module
globals IP_ADDRESS_CACHE / SEEN_AGGREGATE_REPORT_IDS / REVERSE_DNS_MAP
remain, identity-preserved, as re-exports of the default caches.

Bug fixes that ride along, each with a regression test:

- One-shot mailbox runs now honor [general] dns_timeout/dns_retries;
  the CLI call site never forwarded dns_timeout, so the library's
  stray 6.0 default silently applied.
- Lazily-triggered reverse DNS map loads (get_ip_address_info /
  get_service_from_reverse_dns_base_domain, including in n_procs
  workers) now thread psl_overrides_path/psl_overrides_url through to
  load_reverse_dns_map instead of clobbering operator-configured PSL
  overrides with the bundled defaults.
- get_dmarc_reports_from_mailbox() and watch_inbox() dns_timeout
  defaults unified to DEFAULT_DNS_TIMEOUT (2.0s) from a stray 6.0, and
  normalize_timespan_threshold_hours to the float 24.0 used everywhere
  else.

Closes #503

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address CI and review feedback on #851

- Wrap the IMAPConnection example in usage.md so ruff format is clean
  over the docs code blocks (CI runs ruff format --check on the whole
  repo; the local runs were scoped to parsedmarc/ and tests/ and
  missed it).
- Fix the pre-existing "URL ro a reverse DNS map" docstring typo in
  get_service_from_reverse_dns_base_domain, caught by Copilot on the
  adjacent hunk.
- Import parsedmarc.config once, as an aliased plain import, in
  tests/test_config.py instead of mixing import and import-from of the
  same module (flagged by code quality scanning); the aliased module
  import also keeps pyright able to resolve the submodule attribute
  access.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address second Copilot review round on #851

- Add parse_aggregate_report_file() to the library entry-point list in
  usage.md; the following paragraph describes the config= contract for
  "each of these functions", so the list must name all eight
  config-accepting entry points.
- ParserConfig.__setstate__ now initializes every non-cache field to
  its class default before applying the pickled state, so a config
  serialized by an older parsedmarc version (whose state predates
  fields added later) unpickles with the newer fields at their
  defaults instead of unset entirely (__init__ never runs during
  unpickling, so an absent field would raise AttributeError on first
  access). Covered by a regression test that feeds __setstate__ a
  partial state dict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-07-25 19:27:09 -04:00
committed by GitHub
co-authored by Claude Fable 5
parent 4f4733003c
commit 4e80047e68
13 changed files with 1301 additions and 293 deletions
+4
View File
@@ -5,11 +5,15 @@
### New features
- **`n_procs` parallel parsing now covers messages from mbox files and mailbox connections** ([#147](https://github.com/domainaware/parsedmarc/issues/147)), not just report files passed directly as CLI arguments: IMAP, Microsoft Graph, Gmail API, and Maildir connections, including watch mode. `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all gained an `n_procs` keyword argument. Parallel workers are now a reused process pool for the whole run, with a bounded submission window that keeps at most roughly `2 * n_procs` messages in flight at a time so memory stays bounded even for huge mboxes; the main process sends periodic IMAP keepalives while workers parse.
- **Added `ParserConfig`, a single frozen dataclass carrying every parsing/enrichment option** ([#503](https://github.com/domainaware/parsedmarc/issues/503)): offline mode, IP database path, reverse DNS map and PSL overrides paths/URLs, DNS nameservers/timeout/retries, `strip_attachment_payloads`, `normalize_timespan_threshold_hours`, and the three caches the parser and enrichment code share across calls (IP address info, seen aggregate report IDs, and the reverse DNS map). `parse_aggregate_report_xml()`, `parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`, `parse_report_file()`, `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all accept a keyword-only `config=` argument; when it's provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument continues to work unchanged when `config=` is omitted. Every explicitly constructed `ParserConfig` owns fresh, isolated caches; omitting `config=` falls back to the existing process-wide shared default caches, unchanged and identity-preserved (`parsedmarc.IP_ADDRESS_CACHE`, `parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, `parsedmarc.REVERSE_DNS_MAP`). Caches never cross multiprocessing worker boundaries: `ParserConfig.__getstate__` drops the three cache fields when a config is pickled for a worker, and each worker accumulates its own from that point on. `keep_alive` and `n_procs` remain separate keyword arguments — they control process/worker orchestration, not parsing or enrichment behavior, so they're deliberately not `ParserConfig` fields. See the new "Using parsedmarc as a library" section of the usage docs for an example.
### Bug fixes
- **A report file whose parsing raised an unexpected non-parser exception no longer hangs the CLI forever.** The old direct-file parallel implementation spawned a fresh child process per file and blocked the parent on a pipe read; a child that crashed with anything other than a `ParserError` never wrote to that pipe, so the parent waited indefinitely. The new implementation returns the error as a value and logs `Failed to parse <path>` instead.
- **Direct-file parallel parsing no longer stalls a whole batch on one slow file, and no longer spawns a fresh interpreter per file.** The old implementation processed files in hard batches of `n_procs`, so a single slow file delayed every other file in its batch; the new pooled-worker implementation streams files through a reused pool instead.
- **A one-shot mailbox run (no `--watch`) now honors `[general] dns_timeout` and `dns_retries`** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). The CLI's call to `get_dmarc_reports_from_mailbox()` never forwarded those two options, so every one-shot mailbox run silently used the library's own hardcoded default (`dns_timeout=6.0`, `dns_retries=0`) regardless of what the operator configured; watch-mode runs were unaffected since the `watch_inbox()` call site did pass them. The CLI now builds a single `ParserConfig` per run (via the new `_build_parser_config()` helper) from parsed options and passes it as `config=` to every parsing entry point — the direct-file, mbox, one-shot mailbox, and watch call sites, plus the SIGHUP config-reload path — instead of forwarding option keyword arguments by hand at each call site.
- **Lazily-triggered reverse DNS map loads no longer silently clobber configured PSL overrides with the bundled defaults** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). `get_ip_address_info()` and `get_service_from_reverse_dns_base_domain()` load the reverse DNS map on first use rather than eagerly (this also happens independently in each `n_procs` worker process, since a worker starts with an empty map), and `load_reverse_dns_map()` reloads `psl_overrides.txt` at the same time so map entries that depend on the current overrides fold correctly. That reload previously called `load_psl_overrides()` with no path/URL arguments, so the first lazy map load in a run overwrote any operator-configured PSL overrides file with the bundled defaults. `get_ip_address_info()` and `get_service_from_reverse_dns_base_domain()` now accept `psl_overrides_path`/`psl_overrides_url` parameters and thread them through to `load_reverse_dns_map()`, so a lazy load applies the same configured overrides an eager one would.
- **`get_dmarc_reports_from_mailbox()` and `watch_inbox()`'s `dns_timeout` default changed from a stray `6.0` to `DEFAULT_DNS_TIMEOUT`** (2.0 seconds, `parsedmarc/constants.py`), matching every other parsing entry point; `normalize_timespan_threshold_hours` also now defaults to the float `24.0` on both, rather than the int `24`. This is a minor behavior change for library callers who invoke either function directly without passing `dns_timeout`: DNS queries now time out after 2 seconds by default instead of 6, matching `parse_report_file()` and the CLI's own default.
## 10.3.0
+7
View File
@@ -7,6 +7,13 @@
:members:
```
## parsedmarc.config
```{eval-rst}
.. automodule:: parsedmarc.config
:members:
```
## parsedmarc.elastic
```{eval-rst}
+44
View File
@@ -944,6 +944,50 @@ For sections with underscores in the name, the full section name is used:
| `gelf` | `PARSEDMARC_GELF_` |
| `webhook` | `PARSEDMARC_WEBHOOK_` |
## Using parsedmarc as a library
`parsedmarc` is also importable as a regular Python package, not just a CLI
tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`,
`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`,
`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and
`watch_inbox()` — are all importable
directly from the `parsedmarc` package. See the [API reference](api.md) for
the full set of modules and members.
Each of these functions accepts either individual option keyword arguments
(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword
argument carrying a `ParserConfig` instance:
```python
from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
from parsedmarc.mail import IMAPConnection
config = ParserConfig(
offline=False,
nameservers=["1.1.1.1", "1.0.0.1"],
dns_timeout=5.0,
)
report = parse_report_file("aggregate_report.xml.gz", config=config)
connection = IMAPConnection(
host="imap.example.com", user="dmarc@example.com", password="..."
)
results = get_dmarc_reports_from_mailbox(connection, config=config)
```
A few things to keep in mind:
- When `config=` is passed, the individual option keyword arguments are
ignored in favor of the values carried on the `ParserConfig` instance.
- Each explicitly constructed `ParserConfig` owns its own isolated caches
(IP address info, seen aggregate report IDs, and the reverse DNS map).
Omitting `config=` falls back to the process-wide caches shared by every
call that doesn't pass one.
- `keep_alive` and `n_procs` are not part of `ParserConfig` — they control
process/worker orchestration rather than parsing or enrichment behavior,
so they are always passed as separate keyword arguments.
## Performance tuning
For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
+310 -207
View File
@@ -20,11 +20,11 @@ import xml.parsers.expat as expat
import zipfile
import zlib
from base64 import b64decode
from collections import deque
from collections.abc import Callable, Sequence
from csv import DictWriter
from datetime import date, datetime, timedelta, timezone, tzinfo
from io import BytesIO, StringIO
from collections import deque
from collections.abc import Callable, Sequence
from typing import (
Any,
BinaryIO,
@@ -38,6 +38,12 @@ from expiringdict import ExpiringDict
from mailsuite.smtp import send_email
from tqdm import tqdm
from parsedmarc.config import (
IP_ADDRESS_CACHE,
REVERSE_DNS_MAP,
SEEN_AGGREGATE_REPORT_IDS,
ParserConfig,
)
from parsedmarc.constants import (
DEFAULT_DNS_MAX_RETRIES,
DEFAULT_DNS_TIMEOUT,
@@ -116,10 +122,6 @@ EMAIL_SAMPLE_CONTENT_TYPES = (
"message/rfc-822-headers",
)
IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=14400)
SEEN_AGGREGATE_REPORT_IDS = ExpiringDict(max_len=100000000, max_age_seconds=3600)
REVERSE_DNS_MAP = dict()
class ParserError(RuntimeError):
"""Raised whenever the parser fails for some reason"""
@@ -145,6 +147,66 @@ class InvalidFailureReport(InvalidDMARCReport):
InvalidForensicReport = InvalidFailureReport
def _resolve_config(
config: ParserConfig | None,
*,
offline: bool = False,
ip_db_path: str | None = None,
always_use_local_files: bool = False,
reverse_dns_map_path: str | None = None,
reverse_dns_map_url: str | None = None,
nameservers: list[str] | None = None,
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
strip_attachment_payloads: bool = False,
normalize_timespan_threshold_hours: float = 24.0,
) -> ParserConfig:
"""Resolve the effective :class:`~parsedmarc.config.ParserConfig` for a
public parsing call, from either an explicit ``config`` or the caller's
individual option keyword arguments.
If ``config`` is not ``None``, it is returned unchanged: per the
documented ``config=`` contract, the individual option keyword arguments
are ignored in favor of the config's own values, so no merging happens
here.
Otherwise, a new ``ParserConfig`` is built from the given keyword
arguments. Its three cache fields are deliberately *not* left to their
``default_factory`` -- doing so would hand back a config with brand new,
empty caches on every call, silently defeating cross-call IP-info
caching and aggregate-report dedup. Instead, the module-default caches
(:data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`,
:data:`REVERSE_DNS_MAP`) are injected explicitly, so kwargs-style calls
keep observing and mutating the same shared caches they always have.
``dataclasses.replace`` is deliberately not used here: it would need an
existing ``ParserConfig`` to start from, and there isn't one on the
kwargs path -- this function's job is to build the first one.
``psl_overrides_path`` / ``psl_overrides_url`` have no corresponding
keyword arguments on the public functions (see AGENTS.md's guidance on
justifying new config options), so they stay ``None`` on this path; they
are only ever set via an explicitly constructed ``config=``.
"""
if config is not None:
return config
return ParserConfig(
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
normalize_timespan_threshold_hours=float(normalize_timespan_threshold_hours),
ip_address_cache=IP_ADDRESS_CACHE,
seen_aggregate_report_ids=SEEN_AGGREGATE_REPORT_IDS,
reverse_dns_map=REVERSE_DNS_MAP,
)
def _exc_origin(error: BaseException) -> str:
"""Returns a ``" (raised at <file>:<line>)"`` suffix pointing at where an
unexpected exception actually originated, but only when the parsedmarc
@@ -378,14 +440,7 @@ def _append_parsed_record(
def _parse_report_record(
record: dict[str, Any],
*,
ip_db_path: str | None = None,
always_use_local_files: bool = False,
reverse_dns_map_path: str | None = None,
reverse_dns_map_url: str | None = None,
offline: bool = False,
nameservers: list[str] | None = None,
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
config: ParserConfig,
is_rfc_9990: bool = False,
) -> dict[str, Any]:
"""
@@ -394,16 +449,9 @@ def _parse_report_record(
Args:
record (dict): The record to convert
always_use_local_files (bool): Do not download files
reverse_dns_map_path (str): Path to a reverse DNS map file
reverse_dns_map_url (str): URL to a reverse DNS map file
ip_db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP
offline (bool): Do not query online for geolocation or DNS
nameservers (list): A list of one or more nameservers to use
(Cloudflare's public DNS resolvers by default)
dns_timeout (float): Sets the DNS timeout in seconds
dns_retries (int): Number of times to retry DNS queries on timeout
or other transient errors
config (ParserConfig): Parsing and enrichment options, plus caches
is_rfc_9990 (bool): Whether the enclosing report was detected as
RFC 9990-shaped, for RFC 9990-aware validation warnings
Returns:
dict: The converted record
@@ -414,16 +462,18 @@ def _parse_report_record(
raise ValueError("Source IP address is empty")
new_record_source = get_ip_address_info(
record["row"]["source_ip"],
cache=IP_ADDRESS_CACHE,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
reverse_dns_map=REVERSE_DNS_MAP,
offline=offline,
nameservers=nameservers,
timeout=dns_timeout,
retries=dns_retries,
cache=config.ip_address_cache,
ip_db_path=config.ip_db_path,
always_use_local_files=config.always_use_local_files,
reverse_dns_map_path=config.reverse_dns_map_path,
reverse_dns_map_url=config.reverse_dns_map_url,
reverse_dns_map=config.reverse_dns_map,
offline=config.offline,
nameservers=config.nameservers,
timeout=config.dns_timeout,
retries=config.dns_retries,
psl_overrides_path=config.psl_overrides_path,
psl_overrides_url=config.psl_overrides_url,
)
new_record["source"] = new_record_source
new_record["count"] = int(record["row"]["count"])
@@ -783,6 +833,7 @@ def parse_aggregate_report_xml(
retries: int = DEFAULT_DNS_MAX_RETRIES,
keep_alive: Callable | None = None,
normalize_timespan_threshold_hours: float = 24.0,
config: ParserConfig | None = None,
) -> AggregateReport:
"""Parses a DMARC XML report string and returns a consistent dict
@@ -799,12 +850,29 @@ def parse_aggregate_report_xml(
timeout (float): Sets the DNS timeout in seconds
retries (int): Number of times to retry DNS queries on timeout or
other transient errors
keep_alive (callable): Keep alive function
keep_alive (callable): Keep alive function. Not part of ``config``;
always applies.
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict: The parsed aggregate DMARC report
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=timeout,
dns_retries=retries,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
errors = []
# Parse XML and recover from errors
if isinstance(xml, bytes):
@@ -896,7 +964,9 @@ def parse_aggregate_report_xml(
end_ts = int(date_range["end"].split(".")[0])
span_seconds = end_ts - begin_ts
normalize_timespan = span_seconds > normalize_timespan_threshold_hours * 3600
normalize_timespan = (
span_seconds > cfg.normalize_timespan_threshold_hours * 3600
)
date_range["begin"] = timestamp_to_human(begin_ts)
date_range["end"] = timestamp_to_human(end_ts)
@@ -1006,14 +1076,7 @@ def parse_aggregate_report_xml(
try:
report_record = _parse_report_record(
report["record"][i],
ip_db_path=ip_db_path,
offline=offline,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=timeout,
dns_retries=retries,
config=cfg,
is_rfc_9990=is_rfc_9990,
)
_append_parsed_record(
@@ -1029,14 +1092,7 @@ def parse_aggregate_report_xml(
else:
report_record = _parse_report_record(
report["record"],
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
nameservers=nameservers,
dns_timeout=timeout,
dns_retries=retries,
config=cfg,
is_rfc_9990=is_rfc_9990,
)
_append_parsed_record(
@@ -1175,6 +1231,7 @@ def parse_aggregate_report_file(
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
keep_alive: Callable | None = None,
normalize_timespan_threshold_hours: float = 24.0,
config: ParserConfig | None = None,
) -> AggregateReport:
"""Parses a file at the given path, a file-like object. or bytes as an
aggregate DMARC report
@@ -1191,12 +1248,29 @@ def parse_aggregate_report_file(
dns_timeout (float): Sets the DNS timeout in seconds
dns_retries (int): Number of times to retry DNS queries on timeout
or other transient errors
keep_alive (callable): Keep alive function
keep_alive (callable): Keep alive function. Not part of ``config``;
always applies.
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict: The parsed DMARC aggregate report
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
try:
xml = extract_report(_input)
@@ -1205,16 +1279,8 @@ def parse_aggregate_report_file(
return parse_aggregate_report_xml(
xml,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
ip_db_path=ip_db_path,
offline=offline,
nameservers=nameservers,
timeout=dns_timeout,
retries=dns_retries,
config=cfg,
keep_alive=keep_alive,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
@@ -1441,6 +1507,7 @@ def parse_failure_report(
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
strip_attachment_payloads: bool = False,
config: ParserConfig | None = None,
) -> FailureReport:
"""
Converts a DMARC failure report and sample to a dict
@@ -1461,10 +1528,26 @@ def parse_failure_report(
or other transient errors
strip_attachment_payloads (bool): Remove attachment payloads from
failure report results
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict: A parsed report and sample
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
)
delivery_results = ["delivered", "spam", "policy", "reject", "other"]
try:
@@ -1504,16 +1587,18 @@ def parse_failure_report(
ip_address = re.split(r"\s", parsed_report["source_ip"]).pop(0)
parsed_report_source = get_ip_address_info(
ip_address,
cache=IP_ADDRESS_CACHE,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
reverse_dns_map=REVERSE_DNS_MAP,
offline=offline,
nameservers=nameservers,
timeout=dns_timeout,
retries=dns_retries,
cache=cfg.ip_address_cache,
ip_db_path=cfg.ip_db_path,
always_use_local_files=cfg.always_use_local_files,
reverse_dns_map_path=cfg.reverse_dns_map_path,
reverse_dns_map_url=cfg.reverse_dns_map_url,
reverse_dns_map=cfg.reverse_dns_map,
offline=cfg.offline,
nameservers=cfg.nameservers,
timeout=cfg.dns_timeout,
retries=cfg.dns_retries,
psl_overrides_path=cfg.psl_overrides_path,
psl_overrides_url=cfg.psl_overrides_url,
)
parsed_report["source"] = parsed_report_source
del parsed_report["source_ip"]
@@ -1582,7 +1667,7 @@ def parse_failure_report(
parsed_report[optional_field] = None
parsed_sample = parse_email(
sample, strip_attachment_payloads=strip_attachment_payloads
sample, strip_attachment_payloads=cfg.strip_attachment_payloads
)
if "reported_domain" not in parsed_report:
@@ -1723,6 +1808,7 @@ def parse_report_email(
strip_attachment_payloads: bool = False,
keep_alive: Callable | None = None,
normalize_timespan_threshold_hours: float = 24.0,
config: ParserConfig | None = None,
) -> ParsedReport:
"""
Parses a DMARC report from an email
@@ -1740,14 +1826,32 @@ def parse_report_email(
or other transient errors
strip_attachment_payloads (bool): Remove attachment payloads from
failure report results
keep_alive (callable): keep alive function
keep_alive (callable): keep alive function. Not part of ``config``;
always applies.
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict:
* ``report_type``: ``aggregate`` or ``failure``
* ``report``: The parsed report
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
result: ParsedReport | None = None
msg_date: datetime = datetime.now(timezone.utc)
@@ -1855,16 +1959,8 @@ def parse_report_email(
elif payload_text.strip().startswith("<"):
aggregate_report = parse_aggregate_report_xml(
payload_text,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
nameservers=nameservers,
timeout=dns_timeout,
retries=dns_retries,
config=cfg,
keep_alive=keep_alive,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
result = {"report_type": "aggregate", "report": aggregate_report}
@@ -1889,15 +1985,7 @@ def parse_report_email(
feedback_report,
sample,
msg_date,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
config=cfg,
)
except InvalidFailureReport as e:
error = (
@@ -1978,7 +2066,8 @@ def parse_report_file(
reverse_dns_map_url: str | None = None,
offline: bool = False,
keep_alive: Callable | None = None,
normalize_timespan_threshold_hours: float = 24,
normalize_timespan_threshold_hours: float = 24.0,
config: ParserConfig | None = None,
) -> ParsedReport:
"""Parses a DMARC aggregate or failure file at the given path, a
file-like object. or bytes
@@ -1998,11 +2087,30 @@ def parse_report_file(
reverse_dns_map_path (str): Path to a reverse DNS map
reverse_dns_map_url (str): URL to a reverse DNS map
offline (bool): Do not make online queries for geolocation or DNS
keep_alive (callable): Keep alive function
keep_alive (callable): Keep alive function. Not part of ``config``;
always applies.
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict: The parsed DMARC report
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
file_object: BinaryIO
if isinstance(input_, (str, os.PathLike)):
file_path = os.fspath(input_)
@@ -2027,16 +2135,8 @@ def parse_report_file(
try:
report = parse_aggregate_report_file(
content,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
config=cfg,
keep_alive=keep_alive,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
results = {"report_type": "aggregate", "report": report}
except InvalidAggregateReport as aggregate_error:
@@ -2047,17 +2147,8 @@ def parse_report_file(
try:
results = parse_report_email(
content,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
config=cfg,
keep_alive=keep_alive,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
except InvalidDMARCReport as email_error:
raise ParserError(
@@ -2076,14 +2167,19 @@ def _classify_parsed_email(
aggregate_reports: list[AggregateReport],
failure_reports: list[FailureReport],
smtp_tls_reports: list[SMTPTLSReport],
*,
seen_aggregate_report_ids: ExpiringDict,
) -> ReportType:
"""Classify a parsed report email, appending it to the matching list.
Owns the ``SEEN_AGGREGATE_REPORT_IDS`` dedup check: an aggregate report
already seen (keyed on ``{org_name}_{report_id}``) is logged and
dropped instead of appended. Shared, unmodified, by the sequential and
parallel branches of ``get_dmarc_reports_from_mbox`` and
``get_dmarc_reports_from_mailbox`` so both dedup identically.
Owns the seen-aggregate-report-ID dedup check against
``seen_aggregate_report_ids``: an aggregate report already seen (keyed
on ``{org_name}_{report_id}``) is logged and dropped instead of
appended. Shared, unmodified, by the sequential and parallel branches
of ``get_dmarc_reports_from_mbox`` and ``get_dmarc_reports_from_mailbox``
so both dedup identically -- callers pass the config's
``seen_aggregate_report_ids`` cache so dedup state stays scoped to
whichever ``ParserConfig`` (explicit or module-default) is in effect.
Returns the report type so mailbox callers know which UID list to
append the source message's UID to.
@@ -2096,8 +2192,8 @@ def _classify_parsed_email(
report_org = parsed_email["report"]["report_metadata"]["org_name"]
report_id = parsed_email["report"]["report_metadata"]["report_id"]
report_key = f"{report_org}_{report_id}"
if report_key not in SEEN_AGGREGATE_REPORT_IDS:
SEEN_AGGREGATE_REPORT_IDS[report_key] = True
if report_key not in seen_aggregate_report_ids:
seen_aggregate_report_ids[report_key] = True
aggregate_reports.append(parsed_email["report"])
else:
logger.debug(
@@ -2179,6 +2275,7 @@ def get_dmarc_reports_from_mbox(
offline: bool = False,
normalize_timespan_threshold_hours: float = 24.0,
n_procs: int = 1,
config: ParserConfig | None = None,
) -> ParsingResults:
"""Parses a mailbox in mbox format containing e-mails with attached
DMARC reports
@@ -2200,12 +2297,30 @@ def get_dmarc_reports_from_mbox(
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
n_procs (int): Number of processes to use for parsing messages in
parallel. Message reading, deduplication, and result assembly
stay in the calling process; only parsing is parallelized.
stay in the calling process; only parsing is parallelized. Not
part of ``config``; always applies.
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict: Lists of ``aggregate_reports``, ``failure_reports``, and ``smtp_tls_reports``
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
aggregate_reports: list[AggregateReport] = []
failure_reports: list[FailureReport] = []
smtp_tls_reports: list[SMTPTLSReport] = []
@@ -2218,21 +2333,7 @@ def get_dmarc_reports_from_mbox(
if n_procs > 1 and total_messages > 1:
from parsedmarc.parallel import _parse_report_email_job, parallel_map
parse_kwargs = {
"ip_db_path": ip_db_path,
"always_use_local_files": always_use_local_files,
"reverse_dns_map_path": reverse_dns_map_path,
"reverse_dns_map_url": reverse_dns_map_url,
"offline": offline,
"nameservers": nameservers,
"dns_timeout": dns_timeout,
"dns_retries": dns_retries,
"strip_attachment_payloads": strip_attachment_payloads,
"normalize_timespan_threshold_hours": (
normalize_timespan_threshold_hours
),
}
func = functools.partial(_parse_report_email_job, kwargs=parse_kwargs)
func = functools.partial(_parse_report_email_job, config=cfg)
def _jobs():
for i in range(total_messages):
@@ -2251,7 +2352,11 @@ def get_dmarc_reports_from_mbox(
raise result
else:
_classify_parsed_email(
result, aggregate_reports, failure_reports, smtp_tls_reports
result,
aggregate_reports,
failure_reports,
smtp_tls_reports,
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
)
else:
for i in tqdm(range(total_messages), disable=None):
@@ -2259,25 +2364,13 @@ def get_dmarc_reports_from_mbox(
logger.info(f"Processing message {i + 1} of {total_messages}")
msg_content = mbox.get_string(message_key)
try:
sa = strip_attachment_payloads
parsed_email = parse_report_email(
msg_content,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=sa,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
parsed_email = parse_report_email(msg_content, config=cfg)
_classify_parsed_email(
parsed_email,
aggregate_reports,
failure_reports,
smtp_tls_reports,
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
)
except InvalidDMARCReport as error:
logger.warning(error.__str__())
@@ -2341,15 +2434,16 @@ def get_dmarc_reports_from_mailbox(
reverse_dns_map_url: str | None = None,
offline: bool = False,
nameservers: list[str] | None = None,
dns_timeout: float = 6.0,
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
strip_attachment_payloads: bool = False,
results: ParsingResults | None = None,
batch_size: int = 10,
since: datetime | date | str | None = None,
create_folders: bool = True,
normalize_timespan_threshold_hours: float = 24,
normalize_timespan_threshold_hours: float = 24.0,
n_procs: int = 1,
config: ParserConfig | None = None,
) -> ParsingResults:
"""
Fetches and parses DMARC reports from a mailbox
@@ -2385,6 +2479,11 @@ def get_dmarc_reports_from_mailbox(
parallelized. With ``n_procs > 1``, invalid-message disposition
happens after the parsing phase completes, rather than
interleaved message-by-message as it is when ``n_procs`` is 1.
Not part of ``config``; always applies.
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
Returns:
dict: Lists of ``aggregate_reports``, ``failure_reports``, and ``smtp_tls_reports``
@@ -2395,6 +2494,20 @@ def get_dmarc_reports_from_mailbox(
if connection is None:
raise ValueError("Must supply a connection")
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
# current_time useful to fetch_messages later in the program
current_time: datetime | date | str | None = None
@@ -2479,23 +2592,15 @@ def get_dmarc_reports_from_mailbox(
if n_procs > 1 and message_limit > 1:
from parsedmarc.parallel import _parse_report_email_job, parallel_map
# keep_alive is a bound method of the live connection object and is
# not picklable, so it must never cross the process boundary; the
# heartbeat passed to parallel_map below keeps the connection alive
# instead.
parse_kwargs = {
"nameservers": nameservers,
"dns_timeout": dns_timeout,
"dns_retries": dns_retries,
"ip_db_path": ip_db_path,
"always_use_local_files": always_use_local_files,
"reverse_dns_map_path": reverse_dns_map_path,
"reverse_dns_map_url": reverse_dns_map_url,
"offline": offline,
"strip_attachment_payloads": strip_attachment_payloads,
"normalize_timespan_threshold_hours": (normalize_timespan_threshold_hours),
}
func = functools.partial(_parse_report_email_job, kwargs=parse_kwargs)
# The config's caches (ip_address_cache, seen_aggregate_report_ids,
# reverse_dns_map) never cross the process boundary --
# ParserConfig.__getstate__ drops them, and each worker accumulates
# its own via the module defaults it rebinds to on unpickling (see
# ParserConfig.__setstate__). keep_alive is a bound method of the
# live connection object and is not a ParserConfig field, so it is
# never submitted to the pool either; the heartbeat passed to
# parallel_map below keeps the connection alive instead.
func = functools.partial(_parse_report_email_job, config=cfg)
# parallel_map yields results in submission order, so the oldest
# queued id always belongs to the next yielded result; popping as
@@ -2525,7 +2630,11 @@ def get_dmarc_reports_from_mailbox(
invalid_msg_ids.append(message_id)
else:
report_type = _classify_parsed_email(
result, aggregate_reports, failure_reports, smtp_tls_reports
result,
aggregate_reports,
failure_reports,
smtp_tls_reports,
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
)
if report_type == "aggregate":
aggregate_report_msg_uids.append(message_id)
@@ -2547,23 +2656,17 @@ def get_dmarc_reports_from_mailbox(
)
message_id, msg_content = _fetch_mailbox_message(connection, msg_uid, test)
try:
sa = strip_attachment_payloads
parsed_email = parse_report_email(
msg_content,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
strip_attachment_payloads=sa,
config=cfg,
keep_alive=connection.keepalive,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
report_type = _classify_parsed_email(
parsed_email, aggregate_reports, failure_reports, smtp_tls_reports
parsed_email,
aggregate_reports,
failure_reports,
smtp_tls_reports,
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
)
if report_type == "aggregate":
aggregate_report_msg_uids.append(message_id)
@@ -2669,19 +2772,10 @@ def get_dmarc_reports_from_mailbox(
archive_folder=archive_folder,
delete=delete,
test=test,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
results=results,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
since=current_time,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
n_procs=n_procs,
config=cfg,
)
return results
@@ -2702,14 +2796,15 @@ def watch_inbox(
reverse_dns_map_url: str | None = None,
offline: bool = False,
nameservers: list[str] | None = None,
dns_timeout: float = 6.0,
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
strip_attachment_payloads: bool = False,
batch_size: int = 10,
since: datetime | date | str | None = None,
normalize_timespan_threshold_hours: float = 24,
normalize_timespan_threshold_hours: float = 24.0,
config_reloading: Callable | None = None,
n_procs: int = 1,
config: ParserConfig | None = None,
):
"""
Watches the mailbox for new messages and
@@ -2745,8 +2840,25 @@ def watch_inbox(
IDLE loop, so the watcher exits cleanly at a safe boundary.
n_procs (int): Number of processes to use for parsing messages in
parallel. Passed through to ``get_dmarc_reports_from_mailbox``
on each check.
on each check. Not part of ``config``; always applies.
config (ParserConfig): a single object carrying all parsing and
enrichment options plus the caches; when provided, the
individual option keyword arguments listed above are ignored in
favor of the config's values.
"""
cfg = _resolve_config(
config,
offline=offline,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
nameservers=nameservers,
dns_timeout=dns_timeout,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
)
def check_callback(connection):
res = get_dmarc_reports_from_mailbox(
@@ -2755,20 +2867,11 @@ def watch_inbox(
archive_folder=archive_folder,
delete=delete,
test=test,
ip_db_path=ip_db_path,
always_use_local_files=always_use_local_files,
reverse_dns_map_path=reverse_dns_map_path,
reverse_dns_map_url=reverse_dns_map_url,
offline=offline,
nameservers=nameservers,
dns_timeout=dns_timeout,
n_procs=n_procs,
dns_retries=dns_retries,
strip_attachment_payloads=strip_attachment_payloads,
batch_size=batch_size,
since=since,
create_folders=False,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
n_procs=n_procs,
config=cfg,
)
callback(res)
+46 -59
View File
@@ -24,9 +24,11 @@ from kiota_abstractions.api_error import APIError
from tqdm import tqdm
from parsedmarc import (
IP_ADDRESS_CACHE,
REVERSE_DNS_MAP,
SEEN_AGGREGATE_REPORT_IDS,
InvalidDMARCReport,
ParserConfig,
ParserError,
__version__,
elastic,
@@ -46,6 +48,7 @@ from parsedmarc import (
watch_inbox,
webhook,
)
from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT
from parsedmarc.log import logger
from parsedmarc.mail import (
AuthMethod,
@@ -1458,6 +1461,41 @@ def _close_output_clients(clients):
logger.warning("Error closing %s", name, exc_info=True)
def _build_parser_config(opts: Namespace) -> ParserConfig:
"""Builds the single ParserConfig for this run from parsed opts, bound to
the process-wide default caches (the parsedmarc module globals).
"""
return ParserConfig(
offline=opts.offline,
ip_db_path=opts.ip_db_path,
always_use_local_files=opts.always_use_local_files,
reverse_dns_map_path=opts.reverse_dns_map_path,
reverse_dns_map_url=opts.reverse_dns_map_url,
psl_overrides_path=opts.psl_overrides_path,
psl_overrides_url=opts.psl_overrides_url,
nameservers=opts.nameservers,
dns_timeout=(
float(opts.dns_timeout)
if opts.dns_timeout is not None
else DEFAULT_DNS_TIMEOUT
),
dns_retries=(
int(opts.dns_retries)
if opts.dns_retries is not None
else DEFAULT_DNS_MAX_RETRIES
),
strip_attachment_payloads=opts.strip_attachment_payloads,
normalize_timespan_threshold_hours=(
float(opts.normalize_timespan_threshold_hours)
if opts.normalize_timespan_threshold_hours is not None
else 24.0
),
ip_address_cache=IP_ADDRESS_CACHE,
seen_aggregate_report_ids=SEEN_AGGREGATE_REPORT_IDS,
reverse_dns_map=REVERSE_DNS_MAP,
)
def _main():
"""Called when the module is executed"""
@@ -2289,19 +2327,9 @@ def _main():
if n_procs < 1:
n_procs = 1
parse_kwargs = dict(
offline=opts.offline,
ip_db_path=opts.ip_db_path,
always_use_local_files=opts.always_use_local_files,
reverse_dns_map_path=opts.reverse_dns_map_path,
reverse_dns_map_url=opts.reverse_dns_map_url,
nameservers=opts.nameservers,
dns_timeout=opts.dns_timeout,
dns_retries=opts.dns_retries,
strip_attachment_payloads=opts.strip_attachment_payloads,
normalize_timespan_threshold_hours=opts.normalize_timespan_threshold_hours,
)
func = functools.partial(_parse_report_file_job, kwargs=parse_kwargs)
parser_config = _build_parser_config(opts)
func = functools.partial(_parse_report_file_job, config=parser_config)
for file_path, result in parallel_map(
func, file_paths, n_procs, should_stop=lambda: _shutdown_requested
):
@@ -2341,24 +2369,9 @@ def _main():
if _shutdown_requested:
logger.info("Shutdown requested, skipping remaining mbox files")
break
normalize_timespan_threshold_hours_value = (
float(opts.normalize_timespan_threshold_hours)
if opts.normalize_timespan_threshold_hours is not None
else 24.0
)
strip = opts.strip_attachment_payloads
reports = get_dmarc_reports_from_mbox(
mbox_path,
nameservers=opts.nameservers,
dns_timeout=opts.dns_timeout,
dns_retries=opts.dns_retries,
strip_attachment_payloads=strip,
ip_db_path=opts.ip_db_path,
always_use_local_files=opts.always_use_local_files,
reverse_dns_map_path=opts.reverse_dns_map_path,
reverse_dns_map_url=opts.reverse_dns_map_url,
offline=opts.offline,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value,
config=parser_config,
n_procs=n_procs,
)
aggregate_reports += reports["aggregate_reports"]
@@ -2369,7 +2382,6 @@ def _main():
msgraph_connection: MSGraphConnection | None = None
mailbox_batch_size_value = 10
mailbox_check_timeout_value = 30
normalize_timespan_threshold_hours_value = 24.0
if opts.imap_host:
try:
@@ -2525,11 +2537,6 @@ def _main():
if opts.mailbox_check_timeout is not None
else 30
)
normalize_timespan_threshold_hours_value = (
float(opts.normalize_timespan_threshold_hours)
if opts.normalize_timespan_threshold_hours is not None
else 24.0
)
if mailbox_connection and not _shutdown_requested:
try:
reports = get_dmarc_reports_from_mailbox(
@@ -2538,17 +2545,9 @@ def _main():
batch_size=mailbox_batch_size_value,
reports_folder=opts.mailbox_reports_folder,
archive_folder=opts.mailbox_archive_folder,
ip_db_path=opts.ip_db_path,
always_use_local_files=opts.always_use_local_files,
reverse_dns_map_path=opts.reverse_dns_map_path,
reverse_dns_map_url=opts.reverse_dns_map_url,
offline=opts.offline,
nameservers=opts.nameservers,
test=opts.mailbox_test,
strip_attachment_payloads=opts.strip_attachment_payloads,
since=opts.mailbox_since,
dns_retries=opts.dns_retries,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value,
config=parser_config,
n_procs=n_procs,
)
@@ -2667,18 +2666,9 @@ def _main():
delete=opts.mailbox_delete,
test=opts.mailbox_test,
check_timeout=mailbox_check_timeout_value,
nameservers=opts.nameservers,
dns_timeout=opts.dns_timeout,
dns_retries=opts.dns_retries,
strip_attachment_payloads=opts.strip_attachment_payloads,
batch_size=mailbox_batch_size_value,
since=opts.mailbox_since,
ip_db_path=opts.ip_db_path,
always_use_local_files=opts.always_use_local_files,
reverse_dns_map_path=opts.reverse_dns_map_path,
reverse_dns_map_url=opts.reverse_dns_map_url,
offline=opts.offline,
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value,
config=parser_config,
config_reloading=lambda: _reload_requested or _shutdown_requested,
n_procs=n_procs,
)
@@ -2767,6 +2757,8 @@ def _main():
for k, v in vars(new_opts).items():
setattr(opts, k, v)
parser_config = _build_parser_config(opts)
# Update watch parameters from reloaded config
mailbox_batch_size_value = (
int(opts.mailbox_batch_size)
@@ -2778,11 +2770,6 @@ def _main():
if opts.mailbox_check_timeout is not None
else 30
)
normalize_timespan_threshold_hours_value = (
float(opts.normalize_timespan_threshold_hours)
if opts.normalize_timespan_threshold_hours is not None
else 24.0
)
# Update log level
logger.setLevel(logging.ERROR)
+183
View File
@@ -0,0 +1,183 @@
# -*- coding: utf-8 -*-
"""Centralized parser configuration (GitHub issue #503).
This module defines :class:`ParserConfig`, a single frozen dataclass that
carries every parsing/enrichment option (offline mode, IP database path,
reverse DNS map location, PSL overrides location, DNS behavior, etc.) plus
the three caches that the parser and enrichment code share across calls:
the IP address info cache, the seen-aggregate-report-ID cache, and the
reverse DNS map.
``parsedmarc/__init__.py`` re-exports :data:`IP_ADDRESS_CACHE`,
:data:`SEEN_AGGREGATE_REPORT_IDS`, and :data:`REVERSE_DNS_MAP` under the
same names for backward compatibility. That compatibility is
identity-based: ``parsedmarc.IP_ADDRESS_CACHE is parsedmarc.config.IP_ADDRESS_CACHE``
must hold, not merely equal contents, since callers may mutate the caches
in place.
Only :mod:`parsedmarc.constants` and :mod:`parsedmarc.utils` are imported
from the package here importing :mod:`parsedmarc` itself would be
circular, since ``parsedmarc/__init__.py`` imports from this module.
"""
from __future__ import annotations
from dataclasses import dataclass, field, fields
from expiringdict import ExpiringDict
from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT
from parsedmarc.utils import ReverseDNSMap
def _new_ip_address_cache() -> ExpiringDict:
"""Build a fresh IP-address-info cache (4 hour expiry)."""
return ExpiringDict(max_len=10000, max_age_seconds=14400)
def _new_seen_aggregate_report_ids() -> ExpiringDict:
"""Build a fresh seen-aggregate-report-ID cache (1 hour expiry)."""
return ExpiringDict(max_len=100000000, max_age_seconds=3600)
# Canonical default caches. parsedmarc/__init__.py re-exports these three
# objects under the same names (IP_ADDRESS_CACHE, SEEN_AGGREGATE_REPORT_IDS,
# REVERSE_DNS_MAP) for backward compatibility with code that imported them
# directly from the top-level package. That compatibility promise is
# identity-based — the re-exported names must point at these very same
# objects, not merely equivalent ones — so that library callers who obtained
# a reference before this refactor keep observing the same mutations as code
# that goes through a ParserConfig built from these defaults (e.g. during
# unpickling in a multiprocessing worker; see ParserConfig.__setstate__).
IP_ADDRESS_CACHE = _new_ip_address_cache()
SEEN_AGGREGATE_REPORT_IDS = _new_seen_aggregate_report_ids()
REVERSE_DNS_MAP: ReverseDNSMap = {}
# The ParserConfig fields excluded from pickling; every other field must
# survive the round-trip, so __getstate__ derives its contents from
# dataclasses.fields() rather than enumerating option fields by hand.
_CACHE_FIELD_NAMES = (
"ip_address_cache",
"seen_aggregate_report_ids",
"reverse_dns_map",
)
@dataclass(frozen=True)
class ParserConfig:
"""Carries all parsing/enrichment options, plus the caches they share.
When passed as ``config=`` to parsedmarc's public parsing functions, the
individual option keyword arguments (``offline``, ``ip_db_path``,
``nameservers``, etc.) are ignored in favor of the values carried on this
object.
Every explicitly constructed ``ParserConfig()`` owns fresh, isolated
cache objects (``ip_address_cache``, ``seen_aggregate_report_ids``,
``reverse_dns_map``) via ``default_factory`` two independently
constructed configs never share cache state. To derive a variant of an
existing config that *does* keep sharing its caches (e.g. to override
``offline`` for one call while still benefiting from warm caches), use
``dataclasses.replace(cfg, ...)``: since every field, including the three
cache fields, is an init field, ``replace()`` copies the source's cache
objects onto the new instance rather than constructing fresh ones.
Pickling (as happens when a config is captured in a
``functools.partial`` payload submitted to a multiprocessing worker)
intentionally drops cache *contents*: ``__getstate__`` omits the three
cache fields, and ``__setstate__`` rebinds them to this module's
:data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`, and
:data:`REVERSE_DNS_MAP` the unpickling process's module-default
caches rather than either the sender's cache contents (which would be
expensive and stale to serialize per task) or brand new empty caches per
unpickle (which would silently defeat per-worker caching, since a
``functools.partial`` payload is re-pickled for every task submitted to a
worker). Binding the module defaults means a freshly spawned worker
interpreter starts with empty caches and accumulates hits across the
tasks it handles, matching pre-refactor behavior.
Note that ``keep_alive`` and ``n_procs`` are deliberately **not** fields
on this class: those control process/worker orchestration, not parsing
or enrichment behavior.
Attributes:
offline: Do not make online requests (DNS, IP database download,
reverse DNS map download, PSL overrides download).
ip_db_path: Path to a local MMDB file from IPinfo, MaxMind, or DBIP.
always_use_local_files: Always use local/bundled files instead of
downloading the IP database, reverse DNS map, or PSL overrides.
reverse_dns_map_path: Path to a local reverse DNS map file.
reverse_dns_map_url: URL to a reverse DNS map file.
psl_overrides_path: Path to a local PSL overrides file.
psl_overrides_url: URL to a PSL overrides file.
nameservers: A list of one or more nameservers to use for DNS
queries (Cloudflare's public DNS resolvers by default).
dns_timeout: DNS query timeout, in seconds.
dns_retries: Number of times to retry a DNS query after a timeout or
other transient error.
strip_attachment_payloads: Remove attachment payloads from parsed
email results.
normalize_timespan_threshold_hours: Aggregate reports whose
timespan exceeds this many hours are normalized/split.
ip_address_cache: Cache of IP address enrichment results.
seen_aggregate_report_ids: Cache of already-seen aggregate report
IDs, used for de-duplication.
reverse_dns_map: The reverse DNS map used to classify base domains
found via reverse DNS.
"""
offline: bool = False
ip_db_path: str | None = None
always_use_local_files: bool = False
reverse_dns_map_path: str | None = None
reverse_dns_map_url: str | None = None
psl_overrides_path: str | None = None
psl_overrides_url: str | None = None
nameservers: list[str] | None = None
dns_timeout: float = DEFAULT_DNS_TIMEOUT
dns_retries: int = DEFAULT_DNS_MAX_RETRIES
strip_attachment_payloads: bool = False
normalize_timespan_threshold_hours: float = 24.0
ip_address_cache: ExpiringDict = field(
default_factory=_new_ip_address_cache, repr=False, compare=False
)
seen_aggregate_report_ids: ExpiringDict = field(
default_factory=_new_seen_aggregate_report_ids, repr=False, compare=False
)
reverse_dns_map: ReverseDNSMap = field(
default_factory=dict, repr=False, compare=False
)
def __getstate__(self) -> dict[str, object]:
"""Return picklable state, excluding the three cache fields.
Cache contents are process-local and potentially huge; they are
never serialized. See the class docstring for the full rationale.
"""
return {
f.name: getattr(self, f.name)
for f in fields(self)
if f.name not in _CACHE_FIELD_NAMES
}
def __setstate__(self, state: dict[str, object]) -> None:
"""Restore option fields, then rebind caches to this module's
process-wide defaults (never fresh empty caches see the class
docstring for why that would silently break per-worker caching).
Non-cache fields are first initialized to their class defaults, so
unpickling a ``ParserConfig`` serialized by an older parsedmarc
version (whose state predates fields added since) leaves the newer
fields at their defaults instead of unset entirely ``__init__``
never runs during unpickling, so without this an absent field would
raise ``AttributeError`` on first access.
"""
for f in fields(self):
if f.name not in _CACHE_FIELD_NAMES:
object.__setattr__(self, f.name, f.default)
for key, value in state.items():
object.__setattr__(self, key, value)
object.__setattr__(self, "ip_address_cache", IP_ADDRESS_CACHE)
object.__setattr__(self, "seen_aggregate_report_ids", SEEN_AGGREGATE_REPORT_IDS)
object.__setattr__(self, "reverse_dns_map", REVERSE_DNS_MAP)
+16 -9
View File
@@ -6,10 +6,11 @@ import logging
from collections import deque
from collections.abc import Callable, Iterable, Iterator
from concurrent.futures import Future, ProcessPoolExecutor, wait
from typing import TYPE_CHECKING, Any, TypeVar
from typing import TYPE_CHECKING, TypeVar
if TYPE_CHECKING:
from parsedmarc import ParserError
from parsedmarc.config import ParserConfig
from parsedmarc.types import ParsedReport
_J = TypeVar("_J")
@@ -35,14 +36,20 @@ def _init_worker_logging(log_level: int, log_files: list[str]) -> None:
def _parse_report_email_job(
msg_content: bytes | str, *, kwargs: dict[str, Any]
msg_content: bytes | str, *, config: ParserConfig
) -> ParsedReport | ParserError:
"""Worker job that parses a single report email.
Module-level and picklable so it can run in a ``ProcessPoolExecutor``
worker. ``kwargs`` is forwarded to ``parse_report_email`` and must
never contain ``keep_alive`` - it is a bound method of the live
mailbox connection in the parent process and is not picklable.
worker. ``config`` is forwarded to ``parse_report_email``. The config's
caches (``ip_address_cache``, ``seen_aggregate_report_ids``,
``reverse_dns_map``) never cross the process boundary -
``ParserConfig.__getstate__`` drops them, and each worker accumulates
its own via the module defaults it rebinds to on unpickling (see
``ParserConfig.__setstate__``). ``keep_alive`` is not a ``ParserConfig``
field - it is a bound method of the live mailbox connection in the
parent process and is not picklable - so nothing unpicklable is ever
submitted to the pool.
Returns the parsed report on success. A ``ParserError`` raised by
``parse_report_email`` is caught and returned as a value (never
@@ -53,18 +60,18 @@ def _parse_report_email_job(
from parsedmarc import ParserError, parse_report_email
try:
return parse_report_email(msg_content, **kwargs)
return parse_report_email(msg_content, config=config)
except ParserError as e:
return e
def _parse_report_file_job(
file_path: str, *, kwargs: dict[str, Any]
file_path: str, *, config: ParserConfig
) -> tuple[str, ParsedReport | Exception]:
"""Worker job that parses a single report file.
Module-level and picklable so it can run in a ``ProcessPoolExecutor``
worker. ``kwargs`` is forwarded to ``parse_report_file``.
worker. ``config`` is forwarded to ``parse_report_file``.
Catches any ``Exception`` (not just ``ParserError``) and returns it
paired with ``file_path`` rather than letting it propagate. This is a
@@ -77,7 +84,7 @@ def _parse_report_file_job(
from parsedmarc import parse_report_file
try:
return file_path, parse_report_file(file_path, **kwargs)
return file_path, parse_report_file(file_path, config=config)
except Exception as e:
return file_path, e
+15 -1
View File
@@ -842,6 +842,8 @@ def get_service_from_reverse_dns_base_domain(
url: str | None = None,
offline: bool = False,
reverse_dns_map: ReverseDNSMap | None = None,
psl_overrides_path: str | None = None,
psl_overrides_url: str | None = None,
) -> ReverseDNSService:
"""
Returns the service name of a given base domain name from reverse DNS.
@@ -850,9 +852,11 @@ def get_service_from_reverse_dns_base_domain(
base_domain (str): The base domain of the reverse DNS lookup
always_use_local_file (bool): Always use a local map file
local_file_path (str): Path to a local map file
url (str): URL ro a reverse DNS map
url (str): URL to a reverse DNS map
offline (bool): Use the built-in copy of the reverse DNS map
reverse_dns_map (dict): A reverse DNS map
psl_overrides_path (str): Path to a local PSL overrides file
psl_overrides_url (str): URL to a PSL overrides file
Returns:
dict: A dictionary containing name and type.
If the service is unknown, the name will be
@@ -873,6 +877,8 @@ def get_service_from_reverse_dns_base_domain(
local_file_path=local_file_path,
url=url,
offline=offline,
psl_overrides_path=psl_overrides_path,
psl_overrides_url=psl_overrides_url,
)
service: ReverseDNSService
@@ -897,6 +903,8 @@ def get_ip_address_info(
nameservers: list[str] | None = None,
timeout: float = DEFAULT_DNS_TIMEOUT,
retries: int = DEFAULT_DNS_MAX_RETRIES,
psl_overrides_path: str | None = None,
psl_overrides_url: str | None = None,
) -> IPAddressInfo:
"""
Returns reverse DNS and country information for the given IP address
@@ -915,6 +923,8 @@ def get_ip_address_info(
timeout (float): Sets the DNS timeout in seconds
retries (int): Number of times to retry on timeout or other transient
errors
psl_overrides_path (str): Path to a local PSL overrides file
psl_overrides_url (str): URL to a PSL overrides file
Returns:
dict: ``ip_address``, ``reverse_dns``, ``country``
@@ -967,6 +977,8 @@ def get_ip_address_info(
url=reverse_dns_map_url,
always_use_local_file=always_use_local_files,
reverse_dns_map=reverse_dns_map,
psl_overrides_path=psl_overrides_path,
psl_overrides_url=psl_overrides_url,
)
info["base_domain"] = base_domain
info["type"] = service["type"]
@@ -986,6 +998,8 @@ def get_ip_address_info(
local_file_path=reverse_dns_map_path,
url=reverse_dns_map_url,
offline=offline,
psl_overrides_path=psl_overrides_path,
psl_overrides_url=psl_overrides_url,
)
if info["as_domain"] and info["as_domain"] in map_value:
service = map_value[info["as_domain"]]
+187
View File
@@ -1236,6 +1236,108 @@ since = 2d
self.assertEqual(mock_watch_inbox.call_args.kwargs.get("since"), "2d")
class TestCliParserConfigWiring(unittest.TestCase):
"""Tests that _main() builds a single ParserConfig (via
_build_parser_config) from parsed opts and passes it as ``config=`` to
the library's mailbox-fetching functions, rather than forwarding
individual option kwargs (offline, dns_timeout, ip_db_path, etc.) by
hand at each call site."""
def setUp(self):
from parsedmarc.log import logger as _logger
_logger.disabled = True
self._stdout_patch = patch("sys.stdout", new_callable=io.StringIO)
self._stderr_patch = patch("sys.stderr", new_callable=io.StringIO)
self._stdout_patch.start()
self._stderr_patch.start()
# SEEN_AGGREGATE_REPORT_IDS is a module-level ExpiringDict shared
# across tests in this process; clear it both ways so state from an
# earlier test class doesn't leak in, and so this class doesn't leak
# into a later one. Precedent: TestDirectoryFilePaths.setUp above.
parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear()
self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear)
def tearDown(self):
from parsedmarc.log import logger as _logger
_logger.disabled = False
self._stderr_patch.stop()
self._stdout_patch.stop()
def _run_one_shot_mailbox(
self, dns_timeout: float | None = None, dns_retries: int | None = None
) -> parsedmarc.ParserConfig:
"""Runs a real one-shot _main() against a mocked IMAP connection and
mocked get_dmarc_reports_from_mailbox, and returns the ParserConfig
the CLI passed as ``config=``."""
config_lines = ["[general]", "silent = true"]
if dns_timeout is not None:
config_lines.append(f"dns_timeout = {dns_timeout}")
if dns_retries is not None:
config_lines.append(f"dns_retries = {dns_retries}")
config_lines += [
"",
"[imap]",
"host = imap.example.com",
"user = user",
"password = pass",
]
config_text = "\n".join(config_lines) + "\n"
with tempfile.NamedTemporaryFile("w", suffix=".ini", delete=False) as cfg:
cfg.write(config_text)
cfg_path = cfg.name
self.addCleanup(lambda: os.path.exists(cfg_path) and os.remove(cfg_path))
with (
patch("parsedmarc.cli.get_dmarc_reports_from_mailbox") as mock_get_reports,
patch("parsedmarc.cli.IMAPConnection") as mock_imap,
):
mock_imap.return_value = object()
mock_get_reports.return_value = {
"aggregate_reports": [],
"failure_reports": [],
"smtp_tls_reports": [],
}
with patch.object(sys, "argv", ["parsedmarc", "-c", cfg_path]):
parsedmarc.cli._main()
return mock_get_reports.call_args.kwargs["config"]
def test_one_shot_mailbox_run_honors_general_dns_timeout(self):
"""Regression test for the one-shot mailbox call site in _main():
before this fix, that call to get_dmarc_reports_from_mailbox
omitted the dns_timeout/dns_retries kwargs entirely (and, before
config= existed, had no way to pass them at all), so a one-shot run
silently used the library's own hardcoded default
(``dns_timeout: float = 6.0`` on master's
get_dmarc_reports_from_mailbox signature) instead of the operator's
``[general] dns_timeout`` / ``dns_retries`` config values. Watch-mode
runs were unaffected because the watch_inbox call site did pass
dns_timeout/dns_retries directly.
"""
cfg = self._run_one_shot_mailbox(dns_timeout=11.5, dns_retries=3)
self.assertEqual(cfg.dns_timeout, 11.5)
self.assertEqual(cfg.dns_retries, 3)
def test_cli_config_binds_module_default_caches(self):
"""The ParserConfig built by the CLI must bind the process-wide
default caches (parsedmarc.IP_ADDRESS_CACHE,
parsedmarc.SEEN_AGGREGATE_REPORT_IDS, parsedmarc.REVERSE_DNS_MAP) by
identity, not fresh/isolated caches otherwise every CLI run would
get its own empty caches (defeating the point of the 4-hour IP
cache and the 1-hour dedup cache) even though ParserConfig's default
factories exist specifically to give library callers isolated
caches when they don't pass config=.
"""
cfg = self._run_one_shot_mailbox()
self.assertIs(cfg.ip_address_cache, parsedmarc.IP_ADDRESS_CACHE)
self.assertIs(
cfg.seen_aggregate_report_ids, parsedmarc.SEEN_AGGREGATE_REPORT_IDS
)
self.assertIs(cfg.reverse_dns_map, parsedmarc.REVERSE_DNS_MAP)
class TestMailboxPerformance(unittest.TestCase):
def setUp(self):
from parsedmarc.log import logger as _logger
@@ -3035,6 +3137,91 @@ watch = true
"Stale entry should have been cleared by reload",
)
@unittest.skipUnless(
hasattr(signal, "SIGHUP"),
"SIGHUP not available on this platform",
)
@patch("parsedmarc.cli._init_output_clients")
@patch("parsedmarc.cli._parse_config")
@patch("parsedmarc.cli._load_config")
@patch("parsedmarc.cli.get_dmarc_reports_from_mailbox")
@patch("parsedmarc.cli.watch_inbox")
@patch("parsedmarc.cli.IMAPConnection")
def test_sighup_reload_rebuilds_parser_config(
self,
mock_imap,
mock_watch,
mock_get_reports,
mock_load_config,
mock_parse_config,
mock_init_clients,
):
"""After a SIGHUP reload, the ParserConfig passed to watch_inbox as
``config=`` must reflect the reloaded ``[general] dns_timeout``, not
the value from the initial config load.
Guards against _build_parser_config(opts) being called only once at
startup: opts itself is correctly refreshed in place by the existing
``for k, v in vars(new_opts).items(): setattr(opts, k, v)`` loop, but
parser_config is a separate ParserConfig snapshot built from opts
if the reload path forgot to rebuild it, watch_inbox would keep
receiving the stale pre-reload config object forever.
"""
import signal as signal_module
mock_imap.return_value = object()
mock_get_reports.return_value = {
"aggregate_reports": [],
"failure_reports": [],
"smtp_tls_reports": [],
}
mock_load_config.return_value = ConfigParser()
parse_calls = [0]
def parse_side_effect(config, opts):
parse_calls[0] += 1
opts.imap_host = "imap.example.com"
opts.imap_user = "user"
opts.imap_password = "pass"
opts.mailbox_watch = True
opts.dns_timeout = 5.0 if parse_calls[0] == 1 else 42.0
return None
mock_parse_config.side_effect = parse_side_effect
mock_init_clients.return_value = {}
watch_calls = [0]
def watch_side_effect(*args, **kwargs):
watch_calls[0] += 1
if watch_calls[0] == 1:
if hasattr(signal_module, "SIGHUP"):
import os
os.kill(os.getpid(), signal_module.SIGHUP)
return
else:
raise FileExistsError("stop-watch-loop")
mock_watch.side_effect = watch_side_effect
with tempfile.NamedTemporaryFile("w", suffix=".ini", delete=False) as cfg:
cfg.write(self._BASE_CONFIG)
cfg_path = cfg.name
self.addCleanup(lambda: os.path.exists(cfg_path) and os.remove(cfg_path))
with patch.object(sys, "argv", ["parsedmarc", "-c", cfg_path]):
with self.assertRaises(SystemExit):
parsedmarc.cli._main()
self.assertEqual(mock_watch.call_count, 2)
first_config = mock_watch.call_args_list[0].kwargs["config"]
second_config = mock_watch.call_args_list[1].kwargs["config"]
self.assertEqual(first_config.dns_timeout, 5.0)
self.assertEqual(second_config.dns_timeout, 42.0)
class TestSigtermShutdown(unittest.TestCase):
"""Tests for graceful SIGTERM/SIGINT shutdown."""
+169
View File
@@ -0,0 +1,169 @@
"""Tests for parsedmarc.config"""
import dataclasses
import pickle
import unittest
import parsedmarc
import parsedmarc.config as parsedmarc_config
class TestParserConfigCaches(unittest.TestCase):
"""Covers per-instance cache isolation for `ParserConfig`."""
def test_each_instance_gets_isolated_caches(self):
"""Two independently constructed ParserConfig instances must never
share cache objects with each other or with the module defaults, and
mutating one instance's caches must not leak into the others."""
cfg_a = parsedmarc_config.ParserConfig()
cfg_b = parsedmarc_config.ParserConfig()
# All distinct objects.
self.assertIsNot(cfg_a.ip_address_cache, cfg_b.ip_address_cache)
self.assertIsNot(
cfg_a.seen_aggregate_report_ids, cfg_b.seen_aggregate_report_ids
)
self.assertIsNot(cfg_a.reverse_dns_map, cfg_b.reverse_dns_map)
self.assertIsNot(cfg_a.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
self.assertIsNot(
cfg_a.seen_aggregate_report_ids,
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
)
self.assertIsNot(cfg_a.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
self.assertIsNot(cfg_b.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
self.assertIsNot(
cfg_b.seen_aggregate_report_ids,
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
)
self.assertIsNot(cfg_b.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
# Mutating one instance's caches must not affect the other, nor the
# module defaults.
cfg_a.ip_address_cache["1.2.3.4"] = {"ip_address": "1.2.3.4"}
cfg_a.seen_aggregate_report_ids["report-id-a"] = True
cfg_a.reverse_dns_map["example.com"] = {"name": "Example", "type": None}
self.assertNotIn("1.2.3.4", cfg_b.ip_address_cache)
self.assertNotIn("report-id-a", cfg_b.seen_aggregate_report_ids)
self.assertNotIn("example.com", cfg_b.reverse_dns_map)
self.assertNotIn("1.2.3.4", parsedmarc_config.IP_ADDRESS_CACHE)
self.assertNotIn("report-id-a", parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS)
self.assertNotIn("example.com", parsedmarc_config.REVERSE_DNS_MAP)
def test_module_default_caches_are_the_public_globals(self):
"""The three module-default caches in `parsedmarc.config` must be the
very same objects re-exported as `parsedmarc.IP_ADDRESS_CACHE`,
`parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, and
`parsedmarc.REVERSE_DNS_MAP`, so that pre-refactor callers who
imported these names directly from the top-level package keep
observing the same cache objects as code that goes through
ParserConfig.
"""
self.assertIs(parsedmarc.IP_ADDRESS_CACHE, parsedmarc_config.IP_ADDRESS_CACHE)
self.assertIs(
parsedmarc.SEEN_AGGREGATE_REPORT_IDS,
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
)
self.assertIs(parsedmarc.REVERSE_DNS_MAP, parsedmarc_config.REVERSE_DNS_MAP)
class TestParserConfigPickling(unittest.TestCase):
"""Covers the pickle strategy documented on `ParserConfig`: option fields
round-trip, but cache contents never cross process boundaries and the
unpickled object's caches rebind to the unpickling process's module
defaults."""
def test_pickle_drops_cache_contents_and_binds_process_defaults(self):
"""Pickling and unpickling a ParserConfig must preserve option
fields, but must NOT carry cache contents across the round-trip, and
must leave the unpickled instance's caches bound to this module's
(the "unpickling process's") default cache objects rather than
fresh, empty ones. Fresh-per-unpickle caches would silently defeat
per-worker caching, since a functools.partial payload carrying a
ParserConfig is re-pickled for every task submitted to a
multiprocessing worker."""
cfg = parsedmarc_config.ParserConfig(
dns_timeout=9.5, nameservers=["192.0.2.53"]
)
cfg.ip_address_cache["sentinel-ip"] = "sentinel-ip-value"
cfg.seen_aggregate_report_ids["sentinel-report-id"] = True
cfg.reverse_dns_map["sentinel.example"] = {
"name": "Sentinel",
"type": None,
}
restored = pickle.loads(pickle.dumps(cfg))
# Option fields compare equal (cache fields are compare=False).
self.assertEqual(cfg, restored)
self.assertEqual(restored.dns_timeout, 9.5)
self.assertEqual(restored.nameservers, ["192.0.2.53"])
# Caches rebind to this process's module defaults, not fresh copies.
self.assertIs(restored.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
self.assertIs(
restored.seen_aggregate_report_ids,
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
)
self.assertIs(restored.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
# Sentinel entries placed on the original instance's caches must NOT
# have crossed into the module defaults.
self.assertNotIn("sentinel-ip", parsedmarc_config.IP_ADDRESS_CACHE)
self.assertNotIn(
"sentinel-report-id", parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS
)
self.assertNotIn("sentinel.example", parsedmarc_config.REVERSE_DNS_MAP)
def test_setstate_defaults_fields_missing_from_older_pickles(self):
"""__setstate__ must initialize every non-cache field to its class
default before applying the pickled state, so a ParserConfig
serialized by an older parsedmarc version (whose state lacks fields
added since) unpickles with the newer fields at their defaults
instead of unset __init__ never runs during unpickling, so an
absent field would otherwise raise AttributeError on first access.
Simulated by calling __setstate__ directly with a partial state
dict, exactly what pickle.loads does with an old payload."""
restored = object.__new__(parsedmarc_config.ParserConfig)
restored.__setstate__({"offline": True, "dns_timeout": 7.5})
self.assertTrue(restored.offline)
self.assertEqual(restored.dns_timeout, 7.5)
# Fields absent from the old state get their class defaults.
self.assertIsNone(restored.ip_db_path)
self.assertIsNone(restored.nameservers)
self.assertEqual(restored.normalize_timespan_threshold_hours, 24.0)
# Caches still rebind to the module defaults.
self.assertIs(restored.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
self.assertIs(
restored.seen_aggregate_report_ids,
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
)
self.assertIs(restored.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
class TestParserConfigFrozenAndReplace(unittest.TestCase):
"""Covers frozen-dataclass immutability and the `dataclasses.replace`
escape hatch for deriving variants that keep sharing caches."""
def test_frozen_and_replace_shares_caches(self):
"""ParserConfig is frozen, so attribute assignment must raise
FrozenInstanceError. `dataclasses.replace(cfg, ...)` must return a
new config that shares the SAME cache objects as the source (all
fields, including the three cache fields, are init fields), since
that is the documented way to derive a variant that keeps warm
caches."""
cfg = parsedmarc_config.ParserConfig()
with self.assertRaises(dataclasses.FrozenInstanceError):
cfg.offline = True # type: ignore[misc]
variant = dataclasses.replace(cfg, offline=True)
self.assertTrue(variant.offline)
self.assertIs(variant.ip_address_cache, cfg.ip_address_cache)
self.assertIs(variant.seen_aggregate_report_ids, cfg.seen_aggregate_report_ids)
self.assertIs(variant.reverse_dns_map, cfg.reverse_dns_map)
+249 -13
View File
@@ -7,6 +7,7 @@ extract_report, get_dmarc_reports_from_mbox, and the CSV / JSON renderers.
import base64
import gzip
import inspect
import json
import logging
import mailbox
@@ -24,6 +25,7 @@ from unittest.mock import MagicMock, patch
from lxml import etree # type: ignore[import-untyped]
import parsedmarc
import parsedmarc.constants as constants
from parsedmarc.mail import MaildirConnection, MSGraphConnection
from parsedmarc.types import (
AggregateReport,
@@ -749,7 +751,9 @@ class Test(unittest.TestCase):
"auth_results": {"dkim": [], "spf": []},
}
with self.assertRaises(ValueError):
parsedmarc._parse_report_record(record, offline=True)
parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
def testParseReportRecordMissingDkimSpf(self):
"""Record with missing dkim/spf auth results defaults correctly"""
@@ -766,7 +770,9 @@ class Test(unittest.TestCase):
"identifiers": {"header_from": "example.com"},
"auth_results": {},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertEqual(result["auth_results"]["dkim"], [])
self.assertEqual(result["auth_results"]["spf"], [])
@@ -786,7 +792,9 @@ class Test(unittest.TestCase):
"identifiers": {"header_from": "example.com"},
"auth_results": {"dkim": [], "spf": []},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
reasons = result["policy_evaluated"]["policy_override_reasons"]
self.assertEqual(len(reasons), 1)
self.assertEqual(reasons[0]["type"], "forwarded")
@@ -811,7 +819,9 @@ class Test(unittest.TestCase):
"identifiers": {"header_from": "example.com"},
"auth_results": {"dkim": [], "spf": []},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
reasons = result["policy_evaluated"]["policy_override_reasons"]
self.assertEqual(len(reasons), 2)
self.assertEqual(reasons[0]["comment"], "relay")
@@ -835,7 +845,9 @@ class Test(unittest.TestCase):
},
"auth_results": {"dkim": [], "spf": []},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertIn("identifiers", result)
self.assertEqual(result["identifiers"]["header_from"], "example.com")
@@ -857,7 +869,9 @@ class Test(unittest.TestCase):
"spf": [],
},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
dkim = result["auth_results"]["dkim"][0]
self.assertEqual(dkim["selector"], "none")
self.assertEqual(dkim["result"], "none")
@@ -881,7 +895,9 @@ class Test(unittest.TestCase):
"spf": {"domain": "example.com"},
},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
spf = result["auth_results"]["spf"][0]
self.assertEqual(spf["scope"], "mfrom")
self.assertEqual(spf["result"], "none")
@@ -919,7 +935,9 @@ class Test(unittest.TestCase):
],
},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertEqual(result["auth_results"]["dkim"][0]["human_result"], "good key")
self.assertEqual(
result["auth_results"]["spf"][0]["human_result"], "sender valid"
@@ -945,7 +963,9 @@ class Test(unittest.TestCase):
],
},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertEqual(result["identifiers"]["envelope_from"], "bounce.example.com")
def testParseReportRecordEnvelopeFromNullFallback(self):
@@ -971,7 +991,9 @@ class Test(unittest.TestCase):
],
},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertEqual(result["identifiers"]["envelope_from"], "spf.example.com")
def testParseReportRecordEnvelopeFromNullNoSpfDomain(self):
@@ -998,7 +1020,9 @@ class Test(unittest.TestCase):
"spf": [{"scope": "mfrom", "result": "pass"}],
},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertIsNone(result["identifiers"]["envelope_from"])
def testParseReportRecordEnvelopeTo(self):
@@ -1020,7 +1044,9 @@ class Test(unittest.TestCase):
},
"auth_results": {"dkim": [], "spf": []},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertEqual(result["identifiers"]["envelope_to"], "recipient@example.com")
def testParseReportRecordAlignment(self):
@@ -1038,7 +1064,9 @@ class Test(unittest.TestCase):
"identifiers": {"header_from": "example.com"},
"auth_results": {"dkim": [], "spf": []},
}
result = parsedmarc._parse_report_record(record, offline=True)
result = parsedmarc._parse_report_record(
record, config=parsedmarc.ParserConfig(offline=True)
)
self.assertTrue(result["alignment"]["dkim"])
self.assertFalse(result["alignment"]["spf"])
self.assertTrue(result["alignment"]["dmarc"])
@@ -2709,6 +2737,214 @@ class TestGetDmarcReportsFromMboxParallel(unittest.TestCase):
)
class TestCentralizedConfig(unittest.TestCase):
"""Regression coverage for the centralize-config-503 refactor
(parsedmarc/config.py's ``ParserConfig``): the kwargs-style public API
must keep observing/mutating the same module-default caches it always
has (via ``_resolve_config`` injecting ``IP_ADDRESS_CACHE`` /
``SEEN_AGGREGATE_REPORT_IDS`` / ``REVERSE_DNS_MAP`` rather than letting
a fresh ``ParserConfig()`` default-factory hand back empty ones), an
explicit ``config=`` must win over individual option keyword arguments
when both are given, and the DNS-timeout/retry/normalize-threshold
defaults must stay consistent between each public function's own
signature and ``ParserConfig``'s field defaults.
"""
AGGREGATE = "samples/aggregate/twilight.eml"
def _build_single_aggregate_mbox(self) -> str:
"""Builds a temporary mbox containing one copy of AGGREGATE."""
tmp = mkdtemp()
self.addCleanup(rmtree, tmp, ignore_errors=True)
path = os.path.join(tmp, "reports.mbox")
box = mailbox.mbox(path)
box.lock()
try:
with open(self.AGGREGATE, "rb") as source_file:
box.add(mailbox.mboxMessage(source_file.read()))
box.flush()
finally:
box.unlock()
box.close()
return path
def test_kwargs_path_uses_module_default_caches_not_fresh_ones(self):
"""Regression guard for _resolve_config: calling
get_dmarc_reports_from_mbox with plain kwargs (no config=) twice in
a row over the same mbox must dedup the second run's aggregate
report against the first run's, because both calls must resolve to
the SAME module-default parsedmarc.SEEN_AGGREGATE_REPORT_IDS cache.
If _resolve_config ever let the kwargs path fall through to
ParserConfig's default_factory instead of explicitly injecting the
module-default caches, each call would get a fresh, empty cache and
this dedup would silently stop working.
"""
parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear()
self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear)
path = self._build_single_aggregate_mbox()
first = parsedmarc.get_dmarc_reports_from_mbox(path, offline=True)
second = parsedmarc.get_dmarc_reports_from_mbox(path, offline=True)
self.assertEqual(len(first["aggregate_reports"]), 1)
self.assertEqual(len(second["aggregate_reports"]), 0)
report_metadata = first["aggregate_reports"][0]["report_metadata"]
report_key = f"{report_metadata['org_name']}_{report_metadata['report_id']}"
self.assertIn(report_key, parsedmarc.SEEN_AGGREGATE_REPORT_IDS)
def test_kwargs_and_config_equivalence(self):
"""parse_report_file must produce identical results whether called
with individual option keyword arguments or an equivalent explicit
ParserConfig, for one sample of each report type. None of these
paths touch dedup (that's only in _classify_parsed_email /
get_dmarc_reports_from_mbox / get_dmarc_reports_from_mailbox), so
no cache clearing is needed between calls.
"""
sample_paths = [
"samples/aggregate/rfc9990-sample.xml",
"samples/failure/dmarc_ruf_report_linkedin.eml",
"samples/smtp_tls/google.com_smtp_tls_report.eml",
]
for sample_path in sample_paths:
with self.subTest(sample=sample_path):
kwargs_result = parsedmarc.parse_report_file(
sample_path, offline=True, always_use_local_files=True
)
config_result = parsedmarc.parse_report_file(
sample_path,
config=parsedmarc.ParserConfig(
offline=True, always_use_local_files=True
),
)
self.assertEqual(kwargs_result, config_result)
def test_config_wins_over_kwargs_normalize_threshold(self):
"""When both config= and normalize_timespan_threshold_hours= are
given, the config's value must win -- per the documented contract,
an explicit config makes the individual option kwargs inert.
samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml
spans exactly 86400 seconds (24h), so a 1.0-hour threshold
normalizes it and a 1000-hour threshold does not; this makes the
config-vs-kwarg outcome observable in normalized_timespan.
"""
sample_path = "samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml"
with open(sample_path, "rb") as f:
data = f.read()
# Config's high threshold must win over the kwarg's low threshold:
# the report must NOT be normalized.
report = parsedmarc.parse_aggregate_report_file(
data,
offline=True,
config=parsedmarc.ParserConfig(
offline=True, normalize_timespan_threshold_hours=1000.0
),
normalize_timespan_threshold_hours=1.0,
)
for record in report["records"]:
self.assertFalse(record["normalized_timespan"]) # type: ignore[typeddict-item]
# Converse: config's low threshold must win over the kwarg's high
# threshold: the report MUST be normalized.
report = parsedmarc.parse_aggregate_report_file(
data,
offline=True,
config=parsedmarc.ParserConfig(
offline=True, normalize_timespan_threshold_hours=1.0
),
normalize_timespan_threshold_hours=1000.0,
)
for record in report["records"]:
self.assertTrue(record["normalized_timespan"]) # type: ignore[typeddict-item]
def test_separate_configs_isolate_dedup_state(self):
"""Two independently constructed ParserConfig(offline=True)
instances must NOT share dedup state (each has its own
seen_aggregate_report_ids cache), but reusing the SAME instance
across two calls must dedup, exactly like the module-default-cache
kwargs path does.
"""
path = self._build_single_aggregate_mbox()
cfg_a = parsedmarc.ParserConfig(offline=True)
cfg_b = parsedmarc.ParserConfig(offline=True)
result_a = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_a)
result_b = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_b)
self.assertEqual(len(result_a["aggregate_reports"]), 1)
self.assertEqual(len(result_b["aggregate_reports"]), 1)
cfg_c = parsedmarc.ParserConfig(offline=True)
first = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_c)
second = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_c)
self.assertEqual(len(first["aggregate_reports"]), 1)
self.assertEqual(len(second["aggregate_reports"]), 0)
def test_dns_and_normalize_defaults_match_constants_and_parser_config(self):
"""DNS timeout/retries and normalize-timespan-threshold defaults
must match parsedmarc.constants (the authoritative source --
DEFAULT_DNS_TIMEOUT, DEFAULT_DNS_MAX_RETRIES) and
parsedmarc.config.ParserConfig's own field defaults, across every
public function that accepts them.
Regression guard: before this refactor, get_dmarc_reports_from_mailbox
and watch_inbox each had a stray literal ``dns_timeout=6.0`` (instead
of ``DEFAULT_DNS_TIMEOUT == 2.0``) and
``normalize_timespan_threshold_hours=24`` (an int, instead of the
float ``24.0`` used everywhere else) -- exactly the kind of drift
_resolve_config's shared construction path is meant to prevent from
silently reappearing.
"""
functions_and_dns_params = [
(parsedmarc.parse_aggregate_report_xml, "timeout", "retries"),
(parsedmarc.parse_aggregate_report_file, "dns_timeout", "dns_retries"),
(parsedmarc.parse_failure_report, "dns_timeout", "dns_retries"),
(parsedmarc.parse_report_email, "dns_timeout", "dns_retries"),
(parsedmarc.parse_report_file, "dns_timeout", "dns_retries"),
(parsedmarc.get_dmarc_reports_from_mbox, "dns_timeout", "dns_retries"),
(
parsedmarc.get_dmarc_reports_from_mailbox,
"dns_timeout",
"dns_retries",
),
(parsedmarc.watch_inbox, "dns_timeout", "dns_retries"),
]
default_config = parsedmarc.ParserConfig()
for func, timeout_param, retries_param in functions_and_dns_params:
with self.subTest(func=func.__name__, param="dns"):
sig = inspect.signature(func)
timeout_default = sig.parameters[timeout_param].default
retries_default = sig.parameters[retries_param].default
self.assertEqual(timeout_default, constants.DEFAULT_DNS_TIMEOUT)
self.assertEqual(retries_default, constants.DEFAULT_DNS_MAX_RETRIES)
self.assertEqual(timeout_default, default_config.dns_timeout)
self.assertEqual(retries_default, default_config.dns_retries)
# parse_failure_report has no normalize_timespan_threshold_hours
# parameter -- normalization is an aggregate-report-only concept.
normalize_funcs = [
parsedmarc.parse_aggregate_report_xml,
parsedmarc.parse_aggregate_report_file,
parsedmarc.parse_report_email,
parsedmarc.parse_report_file,
parsedmarc.get_dmarc_reports_from_mbox,
parsedmarc.get_dmarc_reports_from_mailbox,
parsedmarc.watch_inbox,
]
for func in normalize_funcs:
with self.subTest(func=func.__name__, param="normalize"):
sig = inspect.signature(func)
default = sig.parameters["normalize_timespan_threshold_hours"].default
self.assertEqual(default, 24.0)
self.assertIsInstance(default, float)
self.assertEqual(
default, default_config.normalize_timespan_threshold_hours
)
class TestGetDmarcReportsFromMailboxValidation(unittest.TestCase):
"""Input validation on get_dmarc_reports_from_mailbox.
+10 -4
View File
@@ -68,7 +68,9 @@ class TestParallelMapParseReportFile(_ParallelTestCase):
for path in SAMPLE_PATHS
]
job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True))
job = functools.partial(
_parse_report_file_job, config=parsedmarc.ParserConfig(offline=True)
)
results = list(parallel_map(job, SAMPLE_PATHS, n_procs=2))
self.assertEqual(len(results), len(SAMPLE_PATHS))
@@ -94,7 +96,9 @@ class TestParallelMapJunkFile(_ParallelTestCase):
junk_path = tf.name
self.addCleanup(os.remove, junk_path)
job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True))
job = functools.partial(
_parse_report_file_job, config=parsedmarc.ParserConfig(offline=True)
)
results = list(parallel_map(job, [junk_path, junk_path], n_procs=2))
self.assertEqual(len(results), 2)
@@ -208,7 +212,9 @@ class TestWorkerLogging(_ParallelTestCase):
configure_logging(logging.DEBUG, log_path)
sample = SAMPLE_PATHS[0]
job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True))
job = functools.partial(
_parse_report_file_job, config=parsedmarc.ParserConfig(offline=True)
)
results = list(parallel_map(job, [sample], n_procs=2))
self.assertEqual(len(results), 1)
@@ -275,7 +281,7 @@ class TestParseReportEmailJob(_ParallelTestCase):
def test_invalid_email_returns_parser_error_value(self):
result = _parse_report_email_job(
b"not a valid email", kwargs=dict(offline=True)
b"not a valid email", config=parsedmarc.ParserConfig(offline=True)
)
self.assertIsInstance(result, parsedmarc.ParserError)
+61
View File
@@ -490,6 +490,67 @@ class TestLoadReverseDnsMapReloadsPSLOverrides(unittest.TestCase):
offline=True,
)
def _write_psl_overrides_file(self):
"""Write a temp PSL overrides file with a unique suffix and register
cleanup, returning its path."""
tf = tempfile.NamedTemporaryFile(
"w", suffix=".txt", delete=False, encoding="utf-8"
)
tf.write(".internal.example-503.net\n")
tf.close()
self.addCleanup(os.unlink, tf.name)
return tf.name
def test_lazy_reverse_dns_map_load_applies_psl_overrides(self):
"""Regression test for GitHub issue #503: the lazy
``load_reverse_dns_map()`` call inside
``get_service_from_reverse_dns_base_domain`` previously omitted
``psl_overrides_path``/``psl_overrides_url``, so
``load_reverse_dns_map``'s unconditional ``load_psl_overrides()``
call (utils.py) silently reloaded ``psl_overrides`` with the bundled
defaults, discarding an operator-configured overrides file. This
proves the lazy load now threads the caller's
``psl_overrides_path`` through: after calling with an empty
``reverse_dns_map`` (which forces the lazy load) and a custom
``psl_overrides_path``, the custom override must be in effect.
"""
path = self._write_psl_overrides_file()
parsedmarc.utils.get_service_from_reverse_dns_base_domain(
"something.example",
reverse_dns_map={},
always_use_local_file=True,
offline=True,
psl_overrides_path=path,
)
self.assertEqual(
parsedmarc.utils.get_base_domain("deep.sub.internal.example-503.net"),
"internal.example-503.net",
)
def test_get_ip_address_info_lazy_load_applies_psl_overrides(self):
"""Regression test for GitHub issue #503, exercising the
ASN-fallback lazy ``load_reverse_dns_map()`` call inside
``get_ip_address_info`` (used when no PTR record resolves). Before
the fix, this lazy load also omitted ``psl_overrides_path``/
``psl_overrides_url``, so it clobbered operator-configured PSL
overrides with the bundled defaults the same way. ``offline=True``
forces the no-PTR path so this call reaches the ASN-fallback lazy
load rather than the PTR-driven
``get_service_from_reverse_dns_base_domain`` call.
"""
path = self._write_psl_overrides_file()
parsedmarc.utils.get_ip_address_info(
"192.0.2.1",
offline=True,
reverse_dns_map={},
always_use_local_files=True,
psl_overrides_path=path,
)
self.assertEqual(
parsedmarc.utils.get_base_domain("deep.sub.internal.example-503.net"),
"internal.example-503.net",
)
class TestGetBaseDomainWithOverrides(unittest.TestCase):
"""`get_base_domain` must honour the current psl_overrides list."""