mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-27 19:04:54 +00:00
* Centralize configuration handling with ParserConfig (#503) Add parsedmarc/config.py with ParserConfig, a frozen dataclass carrying every parsing/enrichment option plus the three shared caches (IP address info, seen aggregate report IDs, reverse DNS map). All eight public parsing/mailbox functions accept a keyword-only config= argument; when provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument keeps working unchanged. The three hand-copied parse_kwargs dicts and the dns_timeout<->timeout rename chain are gone; the CLI builds one ParserConfig per run (rebuilt on SIGHUP) and passes it everywhere. Explicitly constructed configs own fresh isolated caches; dataclasses.replace() shares them; pickling drops cache contents and rebinds the unpickling process's module defaults, preserving the per-worker cache behavior of n_procs parallel parsing. The module globals IP_ADDRESS_CACHE / SEEN_AGGREGATE_REPORT_IDS / REVERSE_DNS_MAP remain, identity-preserved, as re-exports of the default caches. Bug fixes that ride along, each with a regression test: - One-shot mailbox runs now honor [general] dns_timeout/dns_retries; the CLI call site never forwarded dns_timeout, so the library's stray 6.0 default silently applied. - Lazily-triggered reverse DNS map loads (get_ip_address_info / get_service_from_reverse_dns_base_domain, including in n_procs workers) now thread psl_overrides_path/psl_overrides_url through to load_reverse_dns_map instead of clobbering operator-configured PSL overrides with the bundled defaults. - get_dmarc_reports_from_mailbox() and watch_inbox() dns_timeout defaults unified to DEFAULT_DNS_TIMEOUT (2.0s) from a stray 6.0, and normalize_timespan_threshold_hours to the float 24.0 used everywhere else. Closes #503 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address CI and review feedback on #851 - Wrap the IMAPConnection example in usage.md so ruff format is clean over the docs code blocks (CI runs ruff format --check on the whole repo; the local runs were scoped to parsedmarc/ and tests/ and missed it). - Fix the pre-existing "URL ro a reverse DNS map" docstring typo in get_service_from_reverse_dns_base_domain, caught by Copilot on the adjacent hunk. - Import parsedmarc.config once, as an aliased plain import, in tests/test_config.py instead of mixing import and import-from of the same module (flagged by code quality scanning); the aliased module import also keeps pyright able to resolve the submodule attribute access. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address second Copilot review round on #851 - Add parse_aggregate_report_file() to the library entry-point list in usage.md; the following paragraph describes the config= contract for "each of these functions", so the list must name all eight config-accepting entry points. - ParserConfig.__setstate__ now initializes every non-cache field to its class default before applying the pickled state, so a config serialized by an older parsedmarc version (whose state predates fields added later) unpickles with the newer fields at their defaults instead of unset entirely (__init__ never runs during unpickling, so an absent field would raise AttributeError on first access). Covered by a regression test that feeds __setstate__ a partial state dict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
4f4733003c
commit
4e80047e68
@@ -5,11 +5,15 @@
|
||||
### New features
|
||||
|
||||
- **`n_procs` parallel parsing now covers messages from mbox files and mailbox connections** ([#147](https://github.com/domainaware/parsedmarc/issues/147)), not just report files passed directly as CLI arguments: IMAP, Microsoft Graph, Gmail API, and Maildir connections, including watch mode. `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all gained an `n_procs` keyword argument. Parallel workers are now a reused process pool for the whole run, with a bounded submission window that keeps at most roughly `2 * n_procs` messages in flight at a time so memory stays bounded even for huge mboxes; the main process sends periodic IMAP keepalives while workers parse.
|
||||
- **Added `ParserConfig`, a single frozen dataclass carrying every parsing/enrichment option** ([#503](https://github.com/domainaware/parsedmarc/issues/503)): offline mode, IP database path, reverse DNS map and PSL overrides paths/URLs, DNS nameservers/timeout/retries, `strip_attachment_payloads`, `normalize_timespan_threshold_hours`, and the three caches the parser and enrichment code share across calls (IP address info, seen aggregate report IDs, and the reverse DNS map). `parse_aggregate_report_xml()`, `parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`, `parse_report_file()`, `get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and `watch_inbox()` all accept a keyword-only `config=` argument; when it's provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument continues to work unchanged when `config=` is omitted. Every explicitly constructed `ParserConfig` owns fresh, isolated caches; omitting `config=` falls back to the existing process-wide shared default caches, unchanged and identity-preserved (`parsedmarc.IP_ADDRESS_CACHE`, `parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, `parsedmarc.REVERSE_DNS_MAP`). Caches never cross multiprocessing worker boundaries: `ParserConfig.__getstate__` drops the three cache fields when a config is pickled for a worker, and each worker accumulates its own from that point on. `keep_alive` and `n_procs` remain separate keyword arguments — they control process/worker orchestration, not parsing or enrichment behavior, so they're deliberately not `ParserConfig` fields. See the new "Using parsedmarc as a library" section of the usage docs for an example.
|
||||
|
||||
### Bug fixes
|
||||
|
||||
- **A report file whose parsing raised an unexpected non-parser exception no longer hangs the CLI forever.** The old direct-file parallel implementation spawned a fresh child process per file and blocked the parent on a pipe read; a child that crashed with anything other than a `ParserError` never wrote to that pipe, so the parent waited indefinitely. The new implementation returns the error as a value and logs `Failed to parse <path>` instead.
|
||||
- **Direct-file parallel parsing no longer stalls a whole batch on one slow file, and no longer spawns a fresh interpreter per file.** The old implementation processed files in hard batches of `n_procs`, so a single slow file delayed every other file in its batch; the new pooled-worker implementation streams files through a reused pool instead.
|
||||
- **A one-shot mailbox run (no `--watch`) now honors `[general] dns_timeout` and `dns_retries`** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). The CLI's call to `get_dmarc_reports_from_mailbox()` never forwarded those two options, so every one-shot mailbox run silently used the library's own hardcoded default (`dns_timeout=6.0`, `dns_retries=0`) regardless of what the operator configured; watch-mode runs were unaffected since the `watch_inbox()` call site did pass them. The CLI now builds a single `ParserConfig` per run (via the new `_build_parser_config()` helper) from parsed options and passes it as `config=` to every parsing entry point — the direct-file, mbox, one-shot mailbox, and watch call sites, plus the SIGHUP config-reload path — instead of forwarding option keyword arguments by hand at each call site.
|
||||
- **Lazily-triggered reverse DNS map loads no longer silently clobber configured PSL overrides with the bundled defaults** ([#503](https://github.com/domainaware/parsedmarc/issues/503)). `get_ip_address_info()` and `get_service_from_reverse_dns_base_domain()` load the reverse DNS map on first use rather than eagerly (this also happens independently in each `n_procs` worker process, since a worker starts with an empty map), and `load_reverse_dns_map()` reloads `psl_overrides.txt` at the same time so map entries that depend on the current overrides fold correctly. That reload previously called `load_psl_overrides()` with no path/URL arguments, so the first lazy map load in a run overwrote any operator-configured PSL overrides file with the bundled defaults. `get_ip_address_info()` and `get_service_from_reverse_dns_base_domain()` now accept `psl_overrides_path`/`psl_overrides_url` parameters and thread them through to `load_reverse_dns_map()`, so a lazy load applies the same configured overrides an eager one would.
|
||||
- **`get_dmarc_reports_from_mailbox()` and `watch_inbox()`'s `dns_timeout` default changed from a stray `6.0` to `DEFAULT_DNS_TIMEOUT`** (2.0 seconds, `parsedmarc/constants.py`), matching every other parsing entry point; `normalize_timespan_threshold_hours` also now defaults to the float `24.0` on both, rather than the int `24`. This is a minor behavior change for library callers who invoke either function directly without passing `dns_timeout`: DNS queries now time out after 2 seconds by default instead of 6, matching `parse_report_file()` and the CLI's own default.
|
||||
|
||||
## 10.3.0
|
||||
|
||||
|
||||
@@ -7,6 +7,13 @@
|
||||
:members:
|
||||
```
|
||||
|
||||
## parsedmarc.config
|
||||
|
||||
```{eval-rst}
|
||||
.. automodule:: parsedmarc.config
|
||||
:members:
|
||||
```
|
||||
|
||||
## parsedmarc.elastic
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
@@ -944,6 +944,50 @@ For sections with underscores in the name, the full section name is used:
|
||||
| `gelf` | `PARSEDMARC_GELF_` |
|
||||
| `webhook` | `PARSEDMARC_WEBHOOK_` |
|
||||
|
||||
## Using parsedmarc as a library
|
||||
|
||||
`parsedmarc` is also importable as a regular Python package, not just a CLI
|
||||
tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`,
|
||||
`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`,
|
||||
`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and
|
||||
`watch_inbox()` — are all importable
|
||||
directly from the `parsedmarc` package. See the [API reference](api.md) for
|
||||
the full set of modules and members.
|
||||
|
||||
Each of these functions accepts either individual option keyword arguments
|
||||
(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword
|
||||
argument carrying a `ParserConfig` instance:
|
||||
|
||||
```python
|
||||
from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
|
||||
from parsedmarc.mail import IMAPConnection
|
||||
|
||||
config = ParserConfig(
|
||||
offline=False,
|
||||
nameservers=["1.1.1.1", "1.0.0.1"],
|
||||
dns_timeout=5.0,
|
||||
)
|
||||
|
||||
report = parse_report_file("aggregate_report.xml.gz", config=config)
|
||||
|
||||
connection = IMAPConnection(
|
||||
host="imap.example.com", user="dmarc@example.com", password="..."
|
||||
)
|
||||
results = get_dmarc_reports_from_mailbox(connection, config=config)
|
||||
```
|
||||
|
||||
A few things to keep in mind:
|
||||
|
||||
- When `config=` is passed, the individual option keyword arguments are
|
||||
ignored in favor of the values carried on the `ParserConfig` instance.
|
||||
- Each explicitly constructed `ParserConfig` owns its own isolated caches
|
||||
(IP address info, seen aggregate report IDs, and the reverse DNS map).
|
||||
Omitting `config=` falls back to the process-wide caches shared by every
|
||||
call that doesn't pass one.
|
||||
- `keep_alive` and `n_procs` are not part of `ParserConfig` — they control
|
||||
process/worker orchestration rather than parsing or enrichment behavior,
|
||||
so they are always passed as separate keyword arguments.
|
||||
|
||||
## Performance tuning
|
||||
|
||||
For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
|
||||
|
||||
+310
-207
@@ -20,11 +20,11 @@ import xml.parsers.expat as expat
|
||||
import zipfile
|
||||
import zlib
|
||||
from base64 import b64decode
|
||||
from collections import deque
|
||||
from collections.abc import Callable, Sequence
|
||||
from csv import DictWriter
|
||||
from datetime import date, datetime, timedelta, timezone, tzinfo
|
||||
from io import BytesIO, StringIO
|
||||
from collections import deque
|
||||
from collections.abc import Callable, Sequence
|
||||
from typing import (
|
||||
Any,
|
||||
BinaryIO,
|
||||
@@ -38,6 +38,12 @@ from expiringdict import ExpiringDict
|
||||
from mailsuite.smtp import send_email
|
||||
from tqdm import tqdm
|
||||
|
||||
from parsedmarc.config import (
|
||||
IP_ADDRESS_CACHE,
|
||||
REVERSE_DNS_MAP,
|
||||
SEEN_AGGREGATE_REPORT_IDS,
|
||||
ParserConfig,
|
||||
)
|
||||
from parsedmarc.constants import (
|
||||
DEFAULT_DNS_MAX_RETRIES,
|
||||
DEFAULT_DNS_TIMEOUT,
|
||||
@@ -116,10 +122,6 @@ EMAIL_SAMPLE_CONTENT_TYPES = (
|
||||
"message/rfc-822-headers",
|
||||
)
|
||||
|
||||
IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=14400)
|
||||
SEEN_AGGREGATE_REPORT_IDS = ExpiringDict(max_len=100000000, max_age_seconds=3600)
|
||||
REVERSE_DNS_MAP = dict()
|
||||
|
||||
|
||||
class ParserError(RuntimeError):
|
||||
"""Raised whenever the parser fails for some reason"""
|
||||
@@ -145,6 +147,66 @@ class InvalidFailureReport(InvalidDMARCReport):
|
||||
InvalidForensicReport = InvalidFailureReport
|
||||
|
||||
|
||||
def _resolve_config(
|
||||
config: ParserConfig | None,
|
||||
*,
|
||||
offline: bool = False,
|
||||
ip_db_path: str | None = None,
|
||||
always_use_local_files: bool = False,
|
||||
reverse_dns_map_path: str | None = None,
|
||||
reverse_dns_map_url: str | None = None,
|
||||
nameservers: list[str] | None = None,
|
||||
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
strip_attachment_payloads: bool = False,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
) -> ParserConfig:
|
||||
"""Resolve the effective :class:`~parsedmarc.config.ParserConfig` for a
|
||||
public parsing call, from either an explicit ``config`` or the caller's
|
||||
individual option keyword arguments.
|
||||
|
||||
If ``config`` is not ``None``, it is returned unchanged: per the
|
||||
documented ``config=`` contract, the individual option keyword arguments
|
||||
are ignored in favor of the config's own values, so no merging happens
|
||||
here.
|
||||
|
||||
Otherwise, a new ``ParserConfig`` is built from the given keyword
|
||||
arguments. Its three cache fields are deliberately *not* left to their
|
||||
``default_factory`` -- doing so would hand back a config with brand new,
|
||||
empty caches on every call, silently defeating cross-call IP-info
|
||||
caching and aggregate-report dedup. Instead, the module-default caches
|
||||
(:data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`,
|
||||
:data:`REVERSE_DNS_MAP`) are injected explicitly, so kwargs-style calls
|
||||
keep observing and mutating the same shared caches they always have.
|
||||
|
||||
``dataclasses.replace`` is deliberately not used here: it would need an
|
||||
existing ``ParserConfig`` to start from, and there isn't one on the
|
||||
kwargs path -- this function's job is to build the first one.
|
||||
|
||||
``psl_overrides_path`` / ``psl_overrides_url`` have no corresponding
|
||||
keyword arguments on the public functions (see AGENTS.md's guidance on
|
||||
justifying new config options), so they stay ``None`` on this path; they
|
||||
are only ever set via an explicitly constructed ``config=``.
|
||||
"""
|
||||
if config is not None:
|
||||
return config
|
||||
return ParserConfig(
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=float(normalize_timespan_threshold_hours),
|
||||
ip_address_cache=IP_ADDRESS_CACHE,
|
||||
seen_aggregate_report_ids=SEEN_AGGREGATE_REPORT_IDS,
|
||||
reverse_dns_map=REVERSE_DNS_MAP,
|
||||
)
|
||||
|
||||
|
||||
def _exc_origin(error: BaseException) -> str:
|
||||
"""Returns a ``" (raised at <file>:<line>)"`` suffix pointing at where an
|
||||
unexpected exception actually originated, but only when the parsedmarc
|
||||
@@ -378,14 +440,7 @@ def _append_parsed_record(
|
||||
def _parse_report_record(
|
||||
record: dict[str, Any],
|
||||
*,
|
||||
ip_db_path: str | None = None,
|
||||
always_use_local_files: bool = False,
|
||||
reverse_dns_map_path: str | None = None,
|
||||
reverse_dns_map_url: str | None = None,
|
||||
offline: bool = False,
|
||||
nameservers: list[str] | None = None,
|
||||
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
config: ParserConfig,
|
||||
is_rfc_9990: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
@@ -394,16 +449,9 @@ def _parse_report_record(
|
||||
|
||||
Args:
|
||||
record (dict): The record to convert
|
||||
always_use_local_files (bool): Do not download files
|
||||
reverse_dns_map_path (str): Path to a reverse DNS map file
|
||||
reverse_dns_map_url (str): URL to a reverse DNS map file
|
||||
ip_db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP
|
||||
offline (bool): Do not query online for geolocation or DNS
|
||||
nameservers (list): A list of one or more nameservers to use
|
||||
(Cloudflare's public DNS resolvers by default)
|
||||
dns_timeout (float): Sets the DNS timeout in seconds
|
||||
dns_retries (int): Number of times to retry DNS queries on timeout
|
||||
or other transient errors
|
||||
config (ParserConfig): Parsing and enrichment options, plus caches
|
||||
is_rfc_9990 (bool): Whether the enclosing report was detected as
|
||||
RFC 9990-shaped, for RFC 9990-aware validation warnings
|
||||
|
||||
Returns:
|
||||
dict: The converted record
|
||||
@@ -414,16 +462,18 @@ def _parse_report_record(
|
||||
raise ValueError("Source IP address is empty")
|
||||
new_record_source = get_ip_address_info(
|
||||
record["row"]["source_ip"],
|
||||
cache=IP_ADDRESS_CACHE,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
reverse_dns_map=REVERSE_DNS_MAP,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
timeout=dns_timeout,
|
||||
retries=dns_retries,
|
||||
cache=config.ip_address_cache,
|
||||
ip_db_path=config.ip_db_path,
|
||||
always_use_local_files=config.always_use_local_files,
|
||||
reverse_dns_map_path=config.reverse_dns_map_path,
|
||||
reverse_dns_map_url=config.reverse_dns_map_url,
|
||||
reverse_dns_map=config.reverse_dns_map,
|
||||
offline=config.offline,
|
||||
nameservers=config.nameservers,
|
||||
timeout=config.dns_timeout,
|
||||
retries=config.dns_retries,
|
||||
psl_overrides_path=config.psl_overrides_path,
|
||||
psl_overrides_url=config.psl_overrides_url,
|
||||
)
|
||||
new_record["source"] = new_record_source
|
||||
new_record["count"] = int(record["row"]["count"])
|
||||
@@ -783,6 +833,7 @@ def parse_aggregate_report_xml(
|
||||
retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
keep_alive: Callable | None = None,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
config: ParserConfig | None = None,
|
||||
) -> AggregateReport:
|
||||
"""Parses a DMARC XML report string and returns a consistent dict
|
||||
|
||||
@@ -799,12 +850,29 @@ def parse_aggregate_report_xml(
|
||||
timeout (float): Sets the DNS timeout in seconds
|
||||
retries (int): Number of times to retry DNS queries on timeout or
|
||||
other transient errors
|
||||
keep_alive (callable): Keep alive function
|
||||
keep_alive (callable): Keep alive function. Not part of ``config``;
|
||||
always applies.
|
||||
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict: The parsed aggregate DMARC report
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=timeout,
|
||||
dns_retries=retries,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
errors = []
|
||||
# Parse XML and recover from errors
|
||||
if isinstance(xml, bytes):
|
||||
@@ -896,7 +964,9 @@ def parse_aggregate_report_xml(
|
||||
end_ts = int(date_range["end"].split(".")[0])
|
||||
span_seconds = end_ts - begin_ts
|
||||
|
||||
normalize_timespan = span_seconds > normalize_timespan_threshold_hours * 3600
|
||||
normalize_timespan = (
|
||||
span_seconds > cfg.normalize_timespan_threshold_hours * 3600
|
||||
)
|
||||
|
||||
date_range["begin"] = timestamp_to_human(begin_ts)
|
||||
date_range["end"] = timestamp_to_human(end_ts)
|
||||
@@ -1006,14 +1076,7 @@ def parse_aggregate_report_xml(
|
||||
try:
|
||||
report_record = _parse_report_record(
|
||||
report["record"][i],
|
||||
ip_db_path=ip_db_path,
|
||||
offline=offline,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=timeout,
|
||||
dns_retries=retries,
|
||||
config=cfg,
|
||||
is_rfc_9990=is_rfc_9990,
|
||||
)
|
||||
_append_parsed_record(
|
||||
@@ -1029,14 +1092,7 @@ def parse_aggregate_report_xml(
|
||||
else:
|
||||
report_record = _parse_report_record(
|
||||
report["record"],
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=timeout,
|
||||
dns_retries=retries,
|
||||
config=cfg,
|
||||
is_rfc_9990=is_rfc_9990,
|
||||
)
|
||||
_append_parsed_record(
|
||||
@@ -1175,6 +1231,7 @@ def parse_aggregate_report_file(
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
keep_alive: Callable | None = None,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
config: ParserConfig | None = None,
|
||||
) -> AggregateReport:
|
||||
"""Parses a file at the given path, a file-like object. or bytes as an
|
||||
aggregate DMARC report
|
||||
@@ -1191,12 +1248,29 @@ def parse_aggregate_report_file(
|
||||
dns_timeout (float): Sets the DNS timeout in seconds
|
||||
dns_retries (int): Number of times to retry DNS queries on timeout
|
||||
or other transient errors
|
||||
keep_alive (callable): Keep alive function
|
||||
keep_alive (callable): Keep alive function. Not part of ``config``;
|
||||
always applies.
|
||||
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict: The parsed DMARC aggregate report
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
|
||||
try:
|
||||
xml = extract_report(_input)
|
||||
@@ -1205,16 +1279,8 @@ def parse_aggregate_report_file(
|
||||
|
||||
return parse_aggregate_report_xml(
|
||||
xml,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
ip_db_path=ip_db_path,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
timeout=dns_timeout,
|
||||
retries=dns_retries,
|
||||
config=cfg,
|
||||
keep_alive=keep_alive,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
|
||||
|
||||
@@ -1441,6 +1507,7 @@ def parse_failure_report(
|
||||
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
strip_attachment_payloads: bool = False,
|
||||
config: ParserConfig | None = None,
|
||||
) -> FailureReport:
|
||||
"""
|
||||
Converts a DMARC failure report and sample to a dict
|
||||
@@ -1461,10 +1528,26 @@ def parse_failure_report(
|
||||
or other transient errors
|
||||
strip_attachment_payloads (bool): Remove attachment payloads from
|
||||
failure report results
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict: A parsed report and sample
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
)
|
||||
delivery_results = ["delivered", "spam", "policy", "reject", "other"]
|
||||
|
||||
try:
|
||||
@@ -1504,16 +1587,18 @@ def parse_failure_report(
|
||||
ip_address = re.split(r"\s", parsed_report["source_ip"]).pop(0)
|
||||
parsed_report_source = get_ip_address_info(
|
||||
ip_address,
|
||||
cache=IP_ADDRESS_CACHE,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
reverse_dns_map=REVERSE_DNS_MAP,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
timeout=dns_timeout,
|
||||
retries=dns_retries,
|
||||
cache=cfg.ip_address_cache,
|
||||
ip_db_path=cfg.ip_db_path,
|
||||
always_use_local_files=cfg.always_use_local_files,
|
||||
reverse_dns_map_path=cfg.reverse_dns_map_path,
|
||||
reverse_dns_map_url=cfg.reverse_dns_map_url,
|
||||
reverse_dns_map=cfg.reverse_dns_map,
|
||||
offline=cfg.offline,
|
||||
nameservers=cfg.nameservers,
|
||||
timeout=cfg.dns_timeout,
|
||||
retries=cfg.dns_retries,
|
||||
psl_overrides_path=cfg.psl_overrides_path,
|
||||
psl_overrides_url=cfg.psl_overrides_url,
|
||||
)
|
||||
parsed_report["source"] = parsed_report_source
|
||||
del parsed_report["source_ip"]
|
||||
@@ -1582,7 +1667,7 @@ def parse_failure_report(
|
||||
parsed_report[optional_field] = None
|
||||
|
||||
parsed_sample = parse_email(
|
||||
sample, strip_attachment_payloads=strip_attachment_payloads
|
||||
sample, strip_attachment_payloads=cfg.strip_attachment_payloads
|
||||
)
|
||||
|
||||
if "reported_domain" not in parsed_report:
|
||||
@@ -1723,6 +1808,7 @@ def parse_report_email(
|
||||
strip_attachment_payloads: bool = False,
|
||||
keep_alive: Callable | None = None,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
config: ParserConfig | None = None,
|
||||
) -> ParsedReport:
|
||||
"""
|
||||
Parses a DMARC report from an email
|
||||
@@ -1740,14 +1826,32 @@ def parse_report_email(
|
||||
or other transient errors
|
||||
strip_attachment_payloads (bool): Remove attachment payloads from
|
||||
failure report results
|
||||
keep_alive (callable): keep alive function
|
||||
keep_alive (callable): keep alive function. Not part of ``config``;
|
||||
always applies.
|
||||
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict:
|
||||
* ``report_type``: ``aggregate`` or ``failure``
|
||||
* ``report``: The parsed report
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
result: ParsedReport | None = None
|
||||
msg_date: datetime = datetime.now(timezone.utc)
|
||||
|
||||
@@ -1855,16 +1959,8 @@ def parse_report_email(
|
||||
elif payload_text.strip().startswith("<"):
|
||||
aggregate_report = parse_aggregate_report_xml(
|
||||
payload_text,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
timeout=dns_timeout,
|
||||
retries=dns_retries,
|
||||
config=cfg,
|
||||
keep_alive=keep_alive,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
result = {"report_type": "aggregate", "report": aggregate_report}
|
||||
|
||||
@@ -1889,15 +1985,7 @@ def parse_report_email(
|
||||
feedback_report,
|
||||
sample,
|
||||
msg_date,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
config=cfg,
|
||||
)
|
||||
except InvalidFailureReport as e:
|
||||
error = (
|
||||
@@ -1978,7 +2066,8 @@ def parse_report_file(
|
||||
reverse_dns_map_url: str | None = None,
|
||||
offline: bool = False,
|
||||
keep_alive: Callable | None = None,
|
||||
normalize_timespan_threshold_hours: float = 24,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
config: ParserConfig | None = None,
|
||||
) -> ParsedReport:
|
||||
"""Parses a DMARC aggregate or failure file at the given path, a
|
||||
file-like object. or bytes
|
||||
@@ -1998,11 +2087,30 @@ def parse_report_file(
|
||||
reverse_dns_map_path (str): Path to a reverse DNS map
|
||||
reverse_dns_map_url (str): URL to a reverse DNS map
|
||||
offline (bool): Do not make online queries for geolocation or DNS
|
||||
keep_alive (callable): Keep alive function
|
||||
keep_alive (callable): Keep alive function. Not part of ``config``;
|
||||
always applies.
|
||||
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict: The parsed DMARC report
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
file_object: BinaryIO
|
||||
if isinstance(input_, (str, os.PathLike)):
|
||||
file_path = os.fspath(input_)
|
||||
@@ -2027,16 +2135,8 @@ def parse_report_file(
|
||||
try:
|
||||
report = parse_aggregate_report_file(
|
||||
content,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
config=cfg,
|
||||
keep_alive=keep_alive,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
results = {"report_type": "aggregate", "report": report}
|
||||
except InvalidAggregateReport as aggregate_error:
|
||||
@@ -2047,17 +2147,8 @@ def parse_report_file(
|
||||
try:
|
||||
results = parse_report_email(
|
||||
content,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
config=cfg,
|
||||
keep_alive=keep_alive,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
except InvalidDMARCReport as email_error:
|
||||
raise ParserError(
|
||||
@@ -2076,14 +2167,19 @@ def _classify_parsed_email(
|
||||
aggregate_reports: list[AggregateReport],
|
||||
failure_reports: list[FailureReport],
|
||||
smtp_tls_reports: list[SMTPTLSReport],
|
||||
*,
|
||||
seen_aggregate_report_ids: ExpiringDict,
|
||||
) -> ReportType:
|
||||
"""Classify a parsed report email, appending it to the matching list.
|
||||
|
||||
Owns the ``SEEN_AGGREGATE_REPORT_IDS`` dedup check: an aggregate report
|
||||
already seen (keyed on ``{org_name}_{report_id}``) is logged and
|
||||
dropped instead of appended. Shared, unmodified, by the sequential and
|
||||
parallel branches of ``get_dmarc_reports_from_mbox`` and
|
||||
``get_dmarc_reports_from_mailbox`` so both dedup identically.
|
||||
Owns the seen-aggregate-report-ID dedup check against
|
||||
``seen_aggregate_report_ids``: an aggregate report already seen (keyed
|
||||
on ``{org_name}_{report_id}``) is logged and dropped instead of
|
||||
appended. Shared, unmodified, by the sequential and parallel branches
|
||||
of ``get_dmarc_reports_from_mbox`` and ``get_dmarc_reports_from_mailbox``
|
||||
so both dedup identically -- callers pass the config's
|
||||
``seen_aggregate_report_ids`` cache so dedup state stays scoped to
|
||||
whichever ``ParserConfig`` (explicit or module-default) is in effect.
|
||||
|
||||
Returns the report type so mailbox callers know which UID list to
|
||||
append the source message's UID to.
|
||||
@@ -2096,8 +2192,8 @@ def _classify_parsed_email(
|
||||
report_org = parsed_email["report"]["report_metadata"]["org_name"]
|
||||
report_id = parsed_email["report"]["report_metadata"]["report_id"]
|
||||
report_key = f"{report_org}_{report_id}"
|
||||
if report_key not in SEEN_AGGREGATE_REPORT_IDS:
|
||||
SEEN_AGGREGATE_REPORT_IDS[report_key] = True
|
||||
if report_key not in seen_aggregate_report_ids:
|
||||
seen_aggregate_report_ids[report_key] = True
|
||||
aggregate_reports.append(parsed_email["report"])
|
||||
else:
|
||||
logger.debug(
|
||||
@@ -2179,6 +2275,7 @@ def get_dmarc_reports_from_mbox(
|
||||
offline: bool = False,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
n_procs: int = 1,
|
||||
config: ParserConfig | None = None,
|
||||
) -> ParsingResults:
|
||||
"""Parses a mailbox in mbox format containing e-mails with attached
|
||||
DMARC reports
|
||||
@@ -2200,12 +2297,30 @@ def get_dmarc_reports_from_mbox(
|
||||
normalize_timespan_threshold_hours (float): Normalize timespans beyond this
|
||||
n_procs (int): Number of processes to use for parsing messages in
|
||||
parallel. Message reading, deduplication, and result assembly
|
||||
stay in the calling process; only parsing is parallelized.
|
||||
stay in the calling process; only parsing is parallelized. Not
|
||||
part of ``config``; always applies.
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict: Lists of ``aggregate_reports``, ``failure_reports``, and ``smtp_tls_reports``
|
||||
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
aggregate_reports: list[AggregateReport] = []
|
||||
failure_reports: list[FailureReport] = []
|
||||
smtp_tls_reports: list[SMTPTLSReport] = []
|
||||
@@ -2218,21 +2333,7 @@ def get_dmarc_reports_from_mbox(
|
||||
if n_procs > 1 and total_messages > 1:
|
||||
from parsedmarc.parallel import _parse_report_email_job, parallel_map
|
||||
|
||||
parse_kwargs = {
|
||||
"ip_db_path": ip_db_path,
|
||||
"always_use_local_files": always_use_local_files,
|
||||
"reverse_dns_map_path": reverse_dns_map_path,
|
||||
"reverse_dns_map_url": reverse_dns_map_url,
|
||||
"offline": offline,
|
||||
"nameservers": nameservers,
|
||||
"dns_timeout": dns_timeout,
|
||||
"dns_retries": dns_retries,
|
||||
"strip_attachment_payloads": strip_attachment_payloads,
|
||||
"normalize_timespan_threshold_hours": (
|
||||
normalize_timespan_threshold_hours
|
||||
),
|
||||
}
|
||||
func = functools.partial(_parse_report_email_job, kwargs=parse_kwargs)
|
||||
func = functools.partial(_parse_report_email_job, config=cfg)
|
||||
|
||||
def _jobs():
|
||||
for i in range(total_messages):
|
||||
@@ -2251,7 +2352,11 @@ def get_dmarc_reports_from_mbox(
|
||||
raise result
|
||||
else:
|
||||
_classify_parsed_email(
|
||||
result, aggregate_reports, failure_reports, smtp_tls_reports
|
||||
result,
|
||||
aggregate_reports,
|
||||
failure_reports,
|
||||
smtp_tls_reports,
|
||||
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
|
||||
)
|
||||
else:
|
||||
for i in tqdm(range(total_messages), disable=None):
|
||||
@@ -2259,25 +2364,13 @@ def get_dmarc_reports_from_mbox(
|
||||
logger.info(f"Processing message {i + 1} of {total_messages}")
|
||||
msg_content = mbox.get_string(message_key)
|
||||
try:
|
||||
sa = strip_attachment_payloads
|
||||
parsed_email = parse_report_email(
|
||||
msg_content,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=sa,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
parsed_email = parse_report_email(msg_content, config=cfg)
|
||||
_classify_parsed_email(
|
||||
parsed_email,
|
||||
aggregate_reports,
|
||||
failure_reports,
|
||||
smtp_tls_reports,
|
||||
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
|
||||
)
|
||||
except InvalidDMARCReport as error:
|
||||
logger.warning(error.__str__())
|
||||
@@ -2341,15 +2434,16 @@ def get_dmarc_reports_from_mailbox(
|
||||
reverse_dns_map_url: str | None = None,
|
||||
offline: bool = False,
|
||||
nameservers: list[str] | None = None,
|
||||
dns_timeout: float = 6.0,
|
||||
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
strip_attachment_payloads: bool = False,
|
||||
results: ParsingResults | None = None,
|
||||
batch_size: int = 10,
|
||||
since: datetime | date | str | None = None,
|
||||
create_folders: bool = True,
|
||||
normalize_timespan_threshold_hours: float = 24,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
n_procs: int = 1,
|
||||
config: ParserConfig | None = None,
|
||||
) -> ParsingResults:
|
||||
"""
|
||||
Fetches and parses DMARC reports from a mailbox
|
||||
@@ -2385,6 +2479,11 @@ def get_dmarc_reports_from_mailbox(
|
||||
parallelized. With ``n_procs > 1``, invalid-message disposition
|
||||
happens after the parsing phase completes, rather than
|
||||
interleaved message-by-message as it is when ``n_procs`` is 1.
|
||||
Not part of ``config``; always applies.
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
|
||||
Returns:
|
||||
dict: Lists of ``aggregate_reports``, ``failure_reports``, and ``smtp_tls_reports``
|
||||
@@ -2395,6 +2494,20 @@ def get_dmarc_reports_from_mailbox(
|
||||
if connection is None:
|
||||
raise ValueError("Must supply a connection")
|
||||
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
|
||||
# current_time useful to fetch_messages later in the program
|
||||
current_time: datetime | date | str | None = None
|
||||
|
||||
@@ -2479,23 +2592,15 @@ def get_dmarc_reports_from_mailbox(
|
||||
if n_procs > 1 and message_limit > 1:
|
||||
from parsedmarc.parallel import _parse_report_email_job, parallel_map
|
||||
|
||||
# keep_alive is a bound method of the live connection object and is
|
||||
# not picklable, so it must never cross the process boundary; the
|
||||
# heartbeat passed to parallel_map below keeps the connection alive
|
||||
# instead.
|
||||
parse_kwargs = {
|
||||
"nameservers": nameservers,
|
||||
"dns_timeout": dns_timeout,
|
||||
"dns_retries": dns_retries,
|
||||
"ip_db_path": ip_db_path,
|
||||
"always_use_local_files": always_use_local_files,
|
||||
"reverse_dns_map_path": reverse_dns_map_path,
|
||||
"reverse_dns_map_url": reverse_dns_map_url,
|
||||
"offline": offline,
|
||||
"strip_attachment_payloads": strip_attachment_payloads,
|
||||
"normalize_timespan_threshold_hours": (normalize_timespan_threshold_hours),
|
||||
}
|
||||
func = functools.partial(_parse_report_email_job, kwargs=parse_kwargs)
|
||||
# The config's caches (ip_address_cache, seen_aggregate_report_ids,
|
||||
# reverse_dns_map) never cross the process boundary --
|
||||
# ParserConfig.__getstate__ drops them, and each worker accumulates
|
||||
# its own via the module defaults it rebinds to on unpickling (see
|
||||
# ParserConfig.__setstate__). keep_alive is a bound method of the
|
||||
# live connection object and is not a ParserConfig field, so it is
|
||||
# never submitted to the pool either; the heartbeat passed to
|
||||
# parallel_map below keeps the connection alive instead.
|
||||
func = functools.partial(_parse_report_email_job, config=cfg)
|
||||
|
||||
# parallel_map yields results in submission order, so the oldest
|
||||
# queued id always belongs to the next yielded result; popping as
|
||||
@@ -2525,7 +2630,11 @@ def get_dmarc_reports_from_mailbox(
|
||||
invalid_msg_ids.append(message_id)
|
||||
else:
|
||||
report_type = _classify_parsed_email(
|
||||
result, aggregate_reports, failure_reports, smtp_tls_reports
|
||||
result,
|
||||
aggregate_reports,
|
||||
failure_reports,
|
||||
smtp_tls_reports,
|
||||
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
|
||||
)
|
||||
if report_type == "aggregate":
|
||||
aggregate_report_msg_uids.append(message_id)
|
||||
@@ -2547,23 +2656,17 @@ def get_dmarc_reports_from_mailbox(
|
||||
)
|
||||
message_id, msg_content = _fetch_mailbox_message(connection, msg_uid, test)
|
||||
try:
|
||||
sa = strip_attachment_payloads
|
||||
parsed_email = parse_report_email(
|
||||
msg_content,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
strip_attachment_payloads=sa,
|
||||
config=cfg,
|
||||
keep_alive=connection.keepalive,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
report_type = _classify_parsed_email(
|
||||
parsed_email, aggregate_reports, failure_reports, smtp_tls_reports
|
||||
parsed_email,
|
||||
aggregate_reports,
|
||||
failure_reports,
|
||||
smtp_tls_reports,
|
||||
seen_aggregate_report_ids=cfg.seen_aggregate_report_ids,
|
||||
)
|
||||
if report_type == "aggregate":
|
||||
aggregate_report_msg_uids.append(message_id)
|
||||
@@ -2669,19 +2772,10 @@ def get_dmarc_reports_from_mailbox(
|
||||
archive_folder=archive_folder,
|
||||
delete=delete,
|
||||
test=test,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
results=results,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
since=current_time,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
n_procs=n_procs,
|
||||
config=cfg,
|
||||
)
|
||||
|
||||
return results
|
||||
@@ -2702,14 +2796,15 @@ def watch_inbox(
|
||||
reverse_dns_map_url: str | None = None,
|
||||
offline: bool = False,
|
||||
nameservers: list[str] | None = None,
|
||||
dns_timeout: float = 6.0,
|
||||
dns_timeout: float = DEFAULT_DNS_TIMEOUT,
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
strip_attachment_payloads: bool = False,
|
||||
batch_size: int = 10,
|
||||
since: datetime | date | str | None = None,
|
||||
normalize_timespan_threshold_hours: float = 24,
|
||||
normalize_timespan_threshold_hours: float = 24.0,
|
||||
config_reloading: Callable | None = None,
|
||||
n_procs: int = 1,
|
||||
config: ParserConfig | None = None,
|
||||
):
|
||||
"""
|
||||
Watches the mailbox for new messages and
|
||||
@@ -2745,8 +2840,25 @@ def watch_inbox(
|
||||
IDLE loop, so the watcher exits cleanly at a safe boundary.
|
||||
n_procs (int): Number of processes to use for parsing messages in
|
||||
parallel. Passed through to ``get_dmarc_reports_from_mailbox``
|
||||
on each check.
|
||||
on each check. Not part of ``config``; always applies.
|
||||
config (ParserConfig): a single object carrying all parsing and
|
||||
enrichment options plus the caches; when provided, the
|
||||
individual option keyword arguments listed above are ignored in
|
||||
favor of the config's values.
|
||||
"""
|
||||
cfg = _resolve_config(
|
||||
config,
|
||||
offline=offline,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
)
|
||||
|
||||
def check_callback(connection):
|
||||
res = get_dmarc_reports_from_mailbox(
|
||||
@@ -2755,20 +2867,11 @@ def watch_inbox(
|
||||
archive_folder=archive_folder,
|
||||
delete=delete,
|
||||
test=test,
|
||||
ip_db_path=ip_db_path,
|
||||
always_use_local_files=always_use_local_files,
|
||||
reverse_dns_map_path=reverse_dns_map_path,
|
||||
reverse_dns_map_url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
nameservers=nameservers,
|
||||
dns_timeout=dns_timeout,
|
||||
n_procs=n_procs,
|
||||
dns_retries=dns_retries,
|
||||
strip_attachment_payloads=strip_attachment_payloads,
|
||||
batch_size=batch_size,
|
||||
since=since,
|
||||
create_folders=False,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours,
|
||||
n_procs=n_procs,
|
||||
config=cfg,
|
||||
)
|
||||
callback(res)
|
||||
|
||||
|
||||
+46
-59
@@ -24,9 +24,11 @@ from kiota_abstractions.api_error import APIError
|
||||
from tqdm import tqdm
|
||||
|
||||
from parsedmarc import (
|
||||
IP_ADDRESS_CACHE,
|
||||
REVERSE_DNS_MAP,
|
||||
SEEN_AGGREGATE_REPORT_IDS,
|
||||
InvalidDMARCReport,
|
||||
ParserConfig,
|
||||
ParserError,
|
||||
__version__,
|
||||
elastic,
|
||||
@@ -46,6 +48,7 @@ from parsedmarc import (
|
||||
watch_inbox,
|
||||
webhook,
|
||||
)
|
||||
from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT
|
||||
from parsedmarc.log import logger
|
||||
from parsedmarc.mail import (
|
||||
AuthMethod,
|
||||
@@ -1458,6 +1461,41 @@ def _close_output_clients(clients):
|
||||
logger.warning("Error closing %s", name, exc_info=True)
|
||||
|
||||
|
||||
def _build_parser_config(opts: Namespace) -> ParserConfig:
|
||||
"""Builds the single ParserConfig for this run from parsed opts, bound to
|
||||
the process-wide default caches (the parsedmarc module globals).
|
||||
"""
|
||||
return ParserConfig(
|
||||
offline=opts.offline,
|
||||
ip_db_path=opts.ip_db_path,
|
||||
always_use_local_files=opts.always_use_local_files,
|
||||
reverse_dns_map_path=opts.reverse_dns_map_path,
|
||||
reverse_dns_map_url=opts.reverse_dns_map_url,
|
||||
psl_overrides_path=opts.psl_overrides_path,
|
||||
psl_overrides_url=opts.psl_overrides_url,
|
||||
nameservers=opts.nameservers,
|
||||
dns_timeout=(
|
||||
float(opts.dns_timeout)
|
||||
if opts.dns_timeout is not None
|
||||
else DEFAULT_DNS_TIMEOUT
|
||||
),
|
||||
dns_retries=(
|
||||
int(opts.dns_retries)
|
||||
if opts.dns_retries is not None
|
||||
else DEFAULT_DNS_MAX_RETRIES
|
||||
),
|
||||
strip_attachment_payloads=opts.strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=(
|
||||
float(opts.normalize_timespan_threshold_hours)
|
||||
if opts.normalize_timespan_threshold_hours is not None
|
||||
else 24.0
|
||||
),
|
||||
ip_address_cache=IP_ADDRESS_CACHE,
|
||||
seen_aggregate_report_ids=SEEN_AGGREGATE_REPORT_IDS,
|
||||
reverse_dns_map=REVERSE_DNS_MAP,
|
||||
)
|
||||
|
||||
|
||||
def _main():
|
||||
"""Called when the module is executed"""
|
||||
|
||||
@@ -2289,19 +2327,9 @@ def _main():
|
||||
if n_procs < 1:
|
||||
n_procs = 1
|
||||
|
||||
parse_kwargs = dict(
|
||||
offline=opts.offline,
|
||||
ip_db_path=opts.ip_db_path,
|
||||
always_use_local_files=opts.always_use_local_files,
|
||||
reverse_dns_map_path=opts.reverse_dns_map_path,
|
||||
reverse_dns_map_url=opts.reverse_dns_map_url,
|
||||
nameservers=opts.nameservers,
|
||||
dns_timeout=opts.dns_timeout,
|
||||
dns_retries=opts.dns_retries,
|
||||
strip_attachment_payloads=opts.strip_attachment_payloads,
|
||||
normalize_timespan_threshold_hours=opts.normalize_timespan_threshold_hours,
|
||||
)
|
||||
func = functools.partial(_parse_report_file_job, kwargs=parse_kwargs)
|
||||
parser_config = _build_parser_config(opts)
|
||||
|
||||
func = functools.partial(_parse_report_file_job, config=parser_config)
|
||||
for file_path, result in parallel_map(
|
||||
func, file_paths, n_procs, should_stop=lambda: _shutdown_requested
|
||||
):
|
||||
@@ -2341,24 +2369,9 @@ def _main():
|
||||
if _shutdown_requested:
|
||||
logger.info("Shutdown requested, skipping remaining mbox files")
|
||||
break
|
||||
normalize_timespan_threshold_hours_value = (
|
||||
float(opts.normalize_timespan_threshold_hours)
|
||||
if opts.normalize_timespan_threshold_hours is not None
|
||||
else 24.0
|
||||
)
|
||||
strip = opts.strip_attachment_payloads
|
||||
reports = get_dmarc_reports_from_mbox(
|
||||
mbox_path,
|
||||
nameservers=opts.nameservers,
|
||||
dns_timeout=opts.dns_timeout,
|
||||
dns_retries=opts.dns_retries,
|
||||
strip_attachment_payloads=strip,
|
||||
ip_db_path=opts.ip_db_path,
|
||||
always_use_local_files=opts.always_use_local_files,
|
||||
reverse_dns_map_path=opts.reverse_dns_map_path,
|
||||
reverse_dns_map_url=opts.reverse_dns_map_url,
|
||||
offline=opts.offline,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value,
|
||||
config=parser_config,
|
||||
n_procs=n_procs,
|
||||
)
|
||||
aggregate_reports += reports["aggregate_reports"]
|
||||
@@ -2369,7 +2382,6 @@ def _main():
|
||||
msgraph_connection: MSGraphConnection | None = None
|
||||
mailbox_batch_size_value = 10
|
||||
mailbox_check_timeout_value = 30
|
||||
normalize_timespan_threshold_hours_value = 24.0
|
||||
|
||||
if opts.imap_host:
|
||||
try:
|
||||
@@ -2525,11 +2537,6 @@ def _main():
|
||||
if opts.mailbox_check_timeout is not None
|
||||
else 30
|
||||
)
|
||||
normalize_timespan_threshold_hours_value = (
|
||||
float(opts.normalize_timespan_threshold_hours)
|
||||
if opts.normalize_timespan_threshold_hours is not None
|
||||
else 24.0
|
||||
)
|
||||
if mailbox_connection and not _shutdown_requested:
|
||||
try:
|
||||
reports = get_dmarc_reports_from_mailbox(
|
||||
@@ -2538,17 +2545,9 @@ def _main():
|
||||
batch_size=mailbox_batch_size_value,
|
||||
reports_folder=opts.mailbox_reports_folder,
|
||||
archive_folder=opts.mailbox_archive_folder,
|
||||
ip_db_path=opts.ip_db_path,
|
||||
always_use_local_files=opts.always_use_local_files,
|
||||
reverse_dns_map_path=opts.reverse_dns_map_path,
|
||||
reverse_dns_map_url=opts.reverse_dns_map_url,
|
||||
offline=opts.offline,
|
||||
nameservers=opts.nameservers,
|
||||
test=opts.mailbox_test,
|
||||
strip_attachment_payloads=opts.strip_attachment_payloads,
|
||||
since=opts.mailbox_since,
|
||||
dns_retries=opts.dns_retries,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value,
|
||||
config=parser_config,
|
||||
n_procs=n_procs,
|
||||
)
|
||||
|
||||
@@ -2667,18 +2666,9 @@ def _main():
|
||||
delete=opts.mailbox_delete,
|
||||
test=opts.mailbox_test,
|
||||
check_timeout=mailbox_check_timeout_value,
|
||||
nameservers=opts.nameservers,
|
||||
dns_timeout=opts.dns_timeout,
|
||||
dns_retries=opts.dns_retries,
|
||||
strip_attachment_payloads=opts.strip_attachment_payloads,
|
||||
batch_size=mailbox_batch_size_value,
|
||||
since=opts.mailbox_since,
|
||||
ip_db_path=opts.ip_db_path,
|
||||
always_use_local_files=opts.always_use_local_files,
|
||||
reverse_dns_map_path=opts.reverse_dns_map_path,
|
||||
reverse_dns_map_url=opts.reverse_dns_map_url,
|
||||
offline=opts.offline,
|
||||
normalize_timespan_threshold_hours=normalize_timespan_threshold_hours_value,
|
||||
config=parser_config,
|
||||
config_reloading=lambda: _reload_requested or _shutdown_requested,
|
||||
n_procs=n_procs,
|
||||
)
|
||||
@@ -2767,6 +2757,8 @@ def _main():
|
||||
for k, v in vars(new_opts).items():
|
||||
setattr(opts, k, v)
|
||||
|
||||
parser_config = _build_parser_config(opts)
|
||||
|
||||
# Update watch parameters from reloaded config
|
||||
mailbox_batch_size_value = (
|
||||
int(opts.mailbox_batch_size)
|
||||
@@ -2778,11 +2770,6 @@ def _main():
|
||||
if opts.mailbox_check_timeout is not None
|
||||
else 30
|
||||
)
|
||||
normalize_timespan_threshold_hours_value = (
|
||||
float(opts.normalize_timespan_threshold_hours)
|
||||
if opts.normalize_timespan_threshold_hours is not None
|
||||
else 24.0
|
||||
)
|
||||
|
||||
# Update log level
|
||||
logger.setLevel(logging.ERROR)
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
"""Centralized parser configuration (GitHub issue #503).
|
||||
|
||||
This module defines :class:`ParserConfig`, a single frozen dataclass that
|
||||
carries every parsing/enrichment option (offline mode, IP database path,
|
||||
reverse DNS map location, PSL overrides location, DNS behavior, etc.) plus
|
||||
the three caches that the parser and enrichment code share across calls:
|
||||
the IP address info cache, the seen-aggregate-report-ID cache, and the
|
||||
reverse DNS map.
|
||||
|
||||
``parsedmarc/__init__.py`` re-exports :data:`IP_ADDRESS_CACHE`,
|
||||
:data:`SEEN_AGGREGATE_REPORT_IDS`, and :data:`REVERSE_DNS_MAP` under the
|
||||
same names for backward compatibility. That compatibility is
|
||||
identity-based: ``parsedmarc.IP_ADDRESS_CACHE is parsedmarc.config.IP_ADDRESS_CACHE``
|
||||
must hold, not merely equal contents, since callers may mutate the caches
|
||||
in place.
|
||||
|
||||
Only :mod:`parsedmarc.constants` and :mod:`parsedmarc.utils` are imported
|
||||
from the package here — importing :mod:`parsedmarc` itself would be
|
||||
circular, since ``parsedmarc/__init__.py`` imports from this module.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field, fields
|
||||
|
||||
from expiringdict import ExpiringDict
|
||||
|
||||
from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT
|
||||
from parsedmarc.utils import ReverseDNSMap
|
||||
|
||||
|
||||
def _new_ip_address_cache() -> ExpiringDict:
|
||||
"""Build a fresh IP-address-info cache (4 hour expiry)."""
|
||||
return ExpiringDict(max_len=10000, max_age_seconds=14400)
|
||||
|
||||
|
||||
def _new_seen_aggregate_report_ids() -> ExpiringDict:
|
||||
"""Build a fresh seen-aggregate-report-ID cache (1 hour expiry)."""
|
||||
return ExpiringDict(max_len=100000000, max_age_seconds=3600)
|
||||
|
||||
|
||||
# Canonical default caches. parsedmarc/__init__.py re-exports these three
|
||||
# objects under the same names (IP_ADDRESS_CACHE, SEEN_AGGREGATE_REPORT_IDS,
|
||||
# REVERSE_DNS_MAP) for backward compatibility with code that imported them
|
||||
# directly from the top-level package. That compatibility promise is
|
||||
# identity-based — the re-exported names must point at these very same
|
||||
# objects, not merely equivalent ones — so that library callers who obtained
|
||||
# a reference before this refactor keep observing the same mutations as code
|
||||
# that goes through a ParserConfig built from these defaults (e.g. during
|
||||
# unpickling in a multiprocessing worker; see ParserConfig.__setstate__).
|
||||
IP_ADDRESS_CACHE = _new_ip_address_cache()
|
||||
SEEN_AGGREGATE_REPORT_IDS = _new_seen_aggregate_report_ids()
|
||||
REVERSE_DNS_MAP: ReverseDNSMap = {}
|
||||
|
||||
# The ParserConfig fields excluded from pickling; every other field must
|
||||
# survive the round-trip, so __getstate__ derives its contents from
|
||||
# dataclasses.fields() rather than enumerating option fields by hand.
|
||||
_CACHE_FIELD_NAMES = (
|
||||
"ip_address_cache",
|
||||
"seen_aggregate_report_ids",
|
||||
"reverse_dns_map",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParserConfig:
|
||||
"""Carries all parsing/enrichment options, plus the caches they share.
|
||||
|
||||
When passed as ``config=`` to parsedmarc's public parsing functions, the
|
||||
individual option keyword arguments (``offline``, ``ip_db_path``,
|
||||
``nameservers``, etc.) are ignored in favor of the values carried on this
|
||||
object.
|
||||
|
||||
Every explicitly constructed ``ParserConfig()`` owns fresh, isolated
|
||||
cache objects (``ip_address_cache``, ``seen_aggregate_report_ids``,
|
||||
``reverse_dns_map``) via ``default_factory`` — two independently
|
||||
constructed configs never share cache state. To derive a variant of an
|
||||
existing config that *does* keep sharing its caches (e.g. to override
|
||||
``offline`` for one call while still benefiting from warm caches), use
|
||||
``dataclasses.replace(cfg, ...)``: since every field, including the three
|
||||
cache fields, is an init field, ``replace()`` copies the source's cache
|
||||
objects onto the new instance rather than constructing fresh ones.
|
||||
|
||||
Pickling (as happens when a config is captured in a
|
||||
``functools.partial`` payload submitted to a multiprocessing worker)
|
||||
intentionally drops cache *contents*: ``__getstate__`` omits the three
|
||||
cache fields, and ``__setstate__`` rebinds them to this module's
|
||||
:data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`, and
|
||||
:data:`REVERSE_DNS_MAP` — the unpickling process's module-default
|
||||
caches — rather than either the sender's cache contents (which would be
|
||||
expensive and stale to serialize per task) or brand new empty caches per
|
||||
unpickle (which would silently defeat per-worker caching, since a
|
||||
``functools.partial`` payload is re-pickled for every task submitted to a
|
||||
worker). Binding the module defaults means a freshly spawned worker
|
||||
interpreter starts with empty caches and accumulates hits across the
|
||||
tasks it handles, matching pre-refactor behavior.
|
||||
|
||||
Note that ``keep_alive`` and ``n_procs`` are deliberately **not** fields
|
||||
on this class: those control process/worker orchestration, not parsing
|
||||
or enrichment behavior.
|
||||
|
||||
Attributes:
|
||||
offline: Do not make online requests (DNS, IP database download,
|
||||
reverse DNS map download, PSL overrides download).
|
||||
ip_db_path: Path to a local MMDB file from IPinfo, MaxMind, or DBIP.
|
||||
always_use_local_files: Always use local/bundled files instead of
|
||||
downloading the IP database, reverse DNS map, or PSL overrides.
|
||||
reverse_dns_map_path: Path to a local reverse DNS map file.
|
||||
reverse_dns_map_url: URL to a reverse DNS map file.
|
||||
psl_overrides_path: Path to a local PSL overrides file.
|
||||
psl_overrides_url: URL to a PSL overrides file.
|
||||
nameservers: A list of one or more nameservers to use for DNS
|
||||
queries (Cloudflare's public DNS resolvers by default).
|
||||
dns_timeout: DNS query timeout, in seconds.
|
||||
dns_retries: Number of times to retry a DNS query after a timeout or
|
||||
other transient error.
|
||||
strip_attachment_payloads: Remove attachment payloads from parsed
|
||||
email results.
|
||||
normalize_timespan_threshold_hours: Aggregate reports whose
|
||||
timespan exceeds this many hours are normalized/split.
|
||||
ip_address_cache: Cache of IP address enrichment results.
|
||||
seen_aggregate_report_ids: Cache of already-seen aggregate report
|
||||
IDs, used for de-duplication.
|
||||
reverse_dns_map: The reverse DNS map used to classify base domains
|
||||
found via reverse DNS.
|
||||
"""
|
||||
|
||||
offline: bool = False
|
||||
ip_db_path: str | None = None
|
||||
always_use_local_files: bool = False
|
||||
reverse_dns_map_path: str | None = None
|
||||
reverse_dns_map_url: str | None = None
|
||||
psl_overrides_path: str | None = None
|
||||
psl_overrides_url: str | None = None
|
||||
nameservers: list[str] | None = None
|
||||
dns_timeout: float = DEFAULT_DNS_TIMEOUT
|
||||
dns_retries: int = DEFAULT_DNS_MAX_RETRIES
|
||||
strip_attachment_payloads: bool = False
|
||||
normalize_timespan_threshold_hours: float = 24.0
|
||||
ip_address_cache: ExpiringDict = field(
|
||||
default_factory=_new_ip_address_cache, repr=False, compare=False
|
||||
)
|
||||
seen_aggregate_report_ids: ExpiringDict = field(
|
||||
default_factory=_new_seen_aggregate_report_ids, repr=False, compare=False
|
||||
)
|
||||
reverse_dns_map: ReverseDNSMap = field(
|
||||
default_factory=dict, repr=False, compare=False
|
||||
)
|
||||
|
||||
def __getstate__(self) -> dict[str, object]:
|
||||
"""Return picklable state, excluding the three cache fields.
|
||||
|
||||
Cache contents are process-local and potentially huge; they are
|
||||
never serialized. See the class docstring for the full rationale.
|
||||
"""
|
||||
return {
|
||||
f.name: getattr(self, f.name)
|
||||
for f in fields(self)
|
||||
if f.name not in _CACHE_FIELD_NAMES
|
||||
}
|
||||
|
||||
def __setstate__(self, state: dict[str, object]) -> None:
|
||||
"""Restore option fields, then rebind caches to this module's
|
||||
process-wide defaults (never fresh empty caches — see the class
|
||||
docstring for why that would silently break per-worker caching).
|
||||
|
||||
Non-cache fields are first initialized to their class defaults, so
|
||||
unpickling a ``ParserConfig`` serialized by an older parsedmarc
|
||||
version (whose state predates fields added since) leaves the newer
|
||||
fields at their defaults instead of unset entirely — ``__init__``
|
||||
never runs during unpickling, so without this an absent field would
|
||||
raise ``AttributeError`` on first access.
|
||||
"""
|
||||
for f in fields(self):
|
||||
if f.name not in _CACHE_FIELD_NAMES:
|
||||
object.__setattr__(self, f.name, f.default)
|
||||
for key, value in state.items():
|
||||
object.__setattr__(self, key, value)
|
||||
object.__setattr__(self, "ip_address_cache", IP_ADDRESS_CACHE)
|
||||
object.__setattr__(self, "seen_aggregate_report_ids", SEEN_AGGREGATE_REPORT_IDS)
|
||||
object.__setattr__(self, "reverse_dns_map", REVERSE_DNS_MAP)
|
||||
+16
-9
@@ -6,10 +6,11 @@ import logging
|
||||
from collections import deque
|
||||
from collections.abc import Callable, Iterable, Iterator
|
||||
from concurrent.futures import Future, ProcessPoolExecutor, wait
|
||||
from typing import TYPE_CHECKING, Any, TypeVar
|
||||
from typing import TYPE_CHECKING, TypeVar
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from parsedmarc import ParserError
|
||||
from parsedmarc.config import ParserConfig
|
||||
from parsedmarc.types import ParsedReport
|
||||
|
||||
_J = TypeVar("_J")
|
||||
@@ -35,14 +36,20 @@ def _init_worker_logging(log_level: int, log_files: list[str]) -> None:
|
||||
|
||||
|
||||
def _parse_report_email_job(
|
||||
msg_content: bytes | str, *, kwargs: dict[str, Any]
|
||||
msg_content: bytes | str, *, config: ParserConfig
|
||||
) -> ParsedReport | ParserError:
|
||||
"""Worker job that parses a single report email.
|
||||
|
||||
Module-level and picklable so it can run in a ``ProcessPoolExecutor``
|
||||
worker. ``kwargs`` is forwarded to ``parse_report_email`` and must
|
||||
never contain ``keep_alive`` - it is a bound method of the live
|
||||
mailbox connection in the parent process and is not picklable.
|
||||
worker. ``config`` is forwarded to ``parse_report_email``. The config's
|
||||
caches (``ip_address_cache``, ``seen_aggregate_report_ids``,
|
||||
``reverse_dns_map``) never cross the process boundary -
|
||||
``ParserConfig.__getstate__`` drops them, and each worker accumulates
|
||||
its own via the module defaults it rebinds to on unpickling (see
|
||||
``ParserConfig.__setstate__``). ``keep_alive`` is not a ``ParserConfig``
|
||||
field - it is a bound method of the live mailbox connection in the
|
||||
parent process and is not picklable - so nothing unpicklable is ever
|
||||
submitted to the pool.
|
||||
|
||||
Returns the parsed report on success. A ``ParserError`` raised by
|
||||
``parse_report_email`` is caught and returned as a value (never
|
||||
@@ -53,18 +60,18 @@ def _parse_report_email_job(
|
||||
from parsedmarc import ParserError, parse_report_email
|
||||
|
||||
try:
|
||||
return parse_report_email(msg_content, **kwargs)
|
||||
return parse_report_email(msg_content, config=config)
|
||||
except ParserError as e:
|
||||
return e
|
||||
|
||||
|
||||
def _parse_report_file_job(
|
||||
file_path: str, *, kwargs: dict[str, Any]
|
||||
file_path: str, *, config: ParserConfig
|
||||
) -> tuple[str, ParsedReport | Exception]:
|
||||
"""Worker job that parses a single report file.
|
||||
|
||||
Module-level and picklable so it can run in a ``ProcessPoolExecutor``
|
||||
worker. ``kwargs`` is forwarded to ``parse_report_file``.
|
||||
worker. ``config`` is forwarded to ``parse_report_file``.
|
||||
|
||||
Catches any ``Exception`` (not just ``ParserError``) and returns it
|
||||
paired with ``file_path`` rather than letting it propagate. This is a
|
||||
@@ -77,7 +84,7 @@ def _parse_report_file_job(
|
||||
from parsedmarc import parse_report_file
|
||||
|
||||
try:
|
||||
return file_path, parse_report_file(file_path, **kwargs)
|
||||
return file_path, parse_report_file(file_path, config=config)
|
||||
except Exception as e:
|
||||
return file_path, e
|
||||
|
||||
|
||||
+15
-1
@@ -842,6 +842,8 @@ def get_service_from_reverse_dns_base_domain(
|
||||
url: str | None = None,
|
||||
offline: bool = False,
|
||||
reverse_dns_map: ReverseDNSMap | None = None,
|
||||
psl_overrides_path: str | None = None,
|
||||
psl_overrides_url: str | None = None,
|
||||
) -> ReverseDNSService:
|
||||
"""
|
||||
Returns the service name of a given base domain name from reverse DNS.
|
||||
@@ -850,9 +852,11 @@ def get_service_from_reverse_dns_base_domain(
|
||||
base_domain (str): The base domain of the reverse DNS lookup
|
||||
always_use_local_file (bool): Always use a local map file
|
||||
local_file_path (str): Path to a local map file
|
||||
url (str): URL ro a reverse DNS map
|
||||
url (str): URL to a reverse DNS map
|
||||
offline (bool): Use the built-in copy of the reverse DNS map
|
||||
reverse_dns_map (dict): A reverse DNS map
|
||||
psl_overrides_path (str): Path to a local PSL overrides file
|
||||
psl_overrides_url (str): URL to a PSL overrides file
|
||||
Returns:
|
||||
dict: A dictionary containing name and type.
|
||||
If the service is unknown, the name will be
|
||||
@@ -873,6 +877,8 @@ def get_service_from_reverse_dns_base_domain(
|
||||
local_file_path=local_file_path,
|
||||
url=url,
|
||||
offline=offline,
|
||||
psl_overrides_path=psl_overrides_path,
|
||||
psl_overrides_url=psl_overrides_url,
|
||||
)
|
||||
|
||||
service: ReverseDNSService
|
||||
@@ -897,6 +903,8 @@ def get_ip_address_info(
|
||||
nameservers: list[str] | None = None,
|
||||
timeout: float = DEFAULT_DNS_TIMEOUT,
|
||||
retries: int = DEFAULT_DNS_MAX_RETRIES,
|
||||
psl_overrides_path: str | None = None,
|
||||
psl_overrides_url: str | None = None,
|
||||
) -> IPAddressInfo:
|
||||
"""
|
||||
Returns reverse DNS and country information for the given IP address
|
||||
@@ -915,6 +923,8 @@ def get_ip_address_info(
|
||||
timeout (float): Sets the DNS timeout in seconds
|
||||
retries (int): Number of times to retry on timeout or other transient
|
||||
errors
|
||||
psl_overrides_path (str): Path to a local PSL overrides file
|
||||
psl_overrides_url (str): URL to a PSL overrides file
|
||||
|
||||
Returns:
|
||||
dict: ``ip_address``, ``reverse_dns``, ``country``
|
||||
@@ -967,6 +977,8 @@ def get_ip_address_info(
|
||||
url=reverse_dns_map_url,
|
||||
always_use_local_file=always_use_local_files,
|
||||
reverse_dns_map=reverse_dns_map,
|
||||
psl_overrides_path=psl_overrides_path,
|
||||
psl_overrides_url=psl_overrides_url,
|
||||
)
|
||||
info["base_domain"] = base_domain
|
||||
info["type"] = service["type"]
|
||||
@@ -986,6 +998,8 @@ def get_ip_address_info(
|
||||
local_file_path=reverse_dns_map_path,
|
||||
url=reverse_dns_map_url,
|
||||
offline=offline,
|
||||
psl_overrides_path=psl_overrides_path,
|
||||
psl_overrides_url=psl_overrides_url,
|
||||
)
|
||||
if info["as_domain"] and info["as_domain"] in map_value:
|
||||
service = map_value[info["as_domain"]]
|
||||
|
||||
@@ -1236,6 +1236,108 @@ since = 2d
|
||||
self.assertEqual(mock_watch_inbox.call_args.kwargs.get("since"), "2d")
|
||||
|
||||
|
||||
class TestCliParserConfigWiring(unittest.TestCase):
|
||||
"""Tests that _main() builds a single ParserConfig (via
|
||||
_build_parser_config) from parsed opts and passes it as ``config=`` to
|
||||
the library's mailbox-fetching functions, rather than forwarding
|
||||
individual option kwargs (offline, dns_timeout, ip_db_path, etc.) by
|
||||
hand at each call site."""
|
||||
|
||||
def setUp(self):
|
||||
from parsedmarc.log import logger as _logger
|
||||
|
||||
_logger.disabled = True
|
||||
self._stdout_patch = patch("sys.stdout", new_callable=io.StringIO)
|
||||
self._stderr_patch = patch("sys.stderr", new_callable=io.StringIO)
|
||||
self._stdout_patch.start()
|
||||
self._stderr_patch.start()
|
||||
# SEEN_AGGREGATE_REPORT_IDS is a module-level ExpiringDict shared
|
||||
# across tests in this process; clear it both ways so state from an
|
||||
# earlier test class doesn't leak in, and so this class doesn't leak
|
||||
# into a later one. Precedent: TestDirectoryFilePaths.setUp above.
|
||||
parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear()
|
||||
self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear)
|
||||
|
||||
def tearDown(self):
|
||||
from parsedmarc.log import logger as _logger
|
||||
|
||||
_logger.disabled = False
|
||||
self._stderr_patch.stop()
|
||||
self._stdout_patch.stop()
|
||||
|
||||
def _run_one_shot_mailbox(
|
||||
self, dns_timeout: float | None = None, dns_retries: int | None = None
|
||||
) -> parsedmarc.ParserConfig:
|
||||
"""Runs a real one-shot _main() against a mocked IMAP connection and
|
||||
mocked get_dmarc_reports_from_mailbox, and returns the ParserConfig
|
||||
the CLI passed as ``config=``."""
|
||||
config_lines = ["[general]", "silent = true"]
|
||||
if dns_timeout is not None:
|
||||
config_lines.append(f"dns_timeout = {dns_timeout}")
|
||||
if dns_retries is not None:
|
||||
config_lines.append(f"dns_retries = {dns_retries}")
|
||||
config_lines += [
|
||||
"",
|
||||
"[imap]",
|
||||
"host = imap.example.com",
|
||||
"user = user",
|
||||
"password = pass",
|
||||
]
|
||||
config_text = "\n".join(config_lines) + "\n"
|
||||
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".ini", delete=False) as cfg:
|
||||
cfg.write(config_text)
|
||||
cfg_path = cfg.name
|
||||
self.addCleanup(lambda: os.path.exists(cfg_path) and os.remove(cfg_path))
|
||||
|
||||
with (
|
||||
patch("parsedmarc.cli.get_dmarc_reports_from_mailbox") as mock_get_reports,
|
||||
patch("parsedmarc.cli.IMAPConnection") as mock_imap,
|
||||
):
|
||||
mock_imap.return_value = object()
|
||||
mock_get_reports.return_value = {
|
||||
"aggregate_reports": [],
|
||||
"failure_reports": [],
|
||||
"smtp_tls_reports": [],
|
||||
}
|
||||
with patch.object(sys, "argv", ["parsedmarc", "-c", cfg_path]):
|
||||
parsedmarc.cli._main()
|
||||
return mock_get_reports.call_args.kwargs["config"]
|
||||
|
||||
def test_one_shot_mailbox_run_honors_general_dns_timeout(self):
|
||||
"""Regression test for the one-shot mailbox call site in _main():
|
||||
before this fix, that call to get_dmarc_reports_from_mailbox
|
||||
omitted the dns_timeout/dns_retries kwargs entirely (and, before
|
||||
config= existed, had no way to pass them at all), so a one-shot run
|
||||
silently used the library's own hardcoded default
|
||||
(``dns_timeout: float = 6.0`` on master's
|
||||
get_dmarc_reports_from_mailbox signature) instead of the operator's
|
||||
``[general] dns_timeout`` / ``dns_retries`` config values. Watch-mode
|
||||
runs were unaffected because the watch_inbox call site did pass
|
||||
dns_timeout/dns_retries directly.
|
||||
"""
|
||||
cfg = self._run_one_shot_mailbox(dns_timeout=11.5, dns_retries=3)
|
||||
self.assertEqual(cfg.dns_timeout, 11.5)
|
||||
self.assertEqual(cfg.dns_retries, 3)
|
||||
|
||||
def test_cli_config_binds_module_default_caches(self):
|
||||
"""The ParserConfig built by the CLI must bind the process-wide
|
||||
default caches (parsedmarc.IP_ADDRESS_CACHE,
|
||||
parsedmarc.SEEN_AGGREGATE_REPORT_IDS, parsedmarc.REVERSE_DNS_MAP) by
|
||||
identity, not fresh/isolated caches — otherwise every CLI run would
|
||||
get its own empty caches (defeating the point of the 4-hour IP
|
||||
cache and the 1-hour dedup cache) even though ParserConfig's default
|
||||
factories exist specifically to give library callers isolated
|
||||
caches when they don't pass config=.
|
||||
"""
|
||||
cfg = self._run_one_shot_mailbox()
|
||||
self.assertIs(cfg.ip_address_cache, parsedmarc.IP_ADDRESS_CACHE)
|
||||
self.assertIs(
|
||||
cfg.seen_aggregate_report_ids, parsedmarc.SEEN_AGGREGATE_REPORT_IDS
|
||||
)
|
||||
self.assertIs(cfg.reverse_dns_map, parsedmarc.REVERSE_DNS_MAP)
|
||||
|
||||
|
||||
class TestMailboxPerformance(unittest.TestCase):
|
||||
def setUp(self):
|
||||
from parsedmarc.log import logger as _logger
|
||||
@@ -3035,6 +3137,91 @@ watch = true
|
||||
"Stale entry should have been cleared by reload",
|
||||
)
|
||||
|
||||
@unittest.skipUnless(
|
||||
hasattr(signal, "SIGHUP"),
|
||||
"SIGHUP not available on this platform",
|
||||
)
|
||||
@patch("parsedmarc.cli._init_output_clients")
|
||||
@patch("parsedmarc.cli._parse_config")
|
||||
@patch("parsedmarc.cli._load_config")
|
||||
@patch("parsedmarc.cli.get_dmarc_reports_from_mailbox")
|
||||
@patch("parsedmarc.cli.watch_inbox")
|
||||
@patch("parsedmarc.cli.IMAPConnection")
|
||||
def test_sighup_reload_rebuilds_parser_config(
|
||||
self,
|
||||
mock_imap,
|
||||
mock_watch,
|
||||
mock_get_reports,
|
||||
mock_load_config,
|
||||
mock_parse_config,
|
||||
mock_init_clients,
|
||||
):
|
||||
"""After a SIGHUP reload, the ParserConfig passed to watch_inbox as
|
||||
``config=`` must reflect the reloaded ``[general] dns_timeout``, not
|
||||
the value from the initial config load.
|
||||
|
||||
Guards against _build_parser_config(opts) being called only once at
|
||||
startup: opts itself is correctly refreshed in place by the existing
|
||||
``for k, v in vars(new_opts).items(): setattr(opts, k, v)`` loop, but
|
||||
parser_config is a separate ParserConfig snapshot built from opts —
|
||||
if the reload path forgot to rebuild it, watch_inbox would keep
|
||||
receiving the stale pre-reload config object forever.
|
||||
"""
|
||||
import signal as signal_module
|
||||
|
||||
mock_imap.return_value = object()
|
||||
mock_get_reports.return_value = {
|
||||
"aggregate_reports": [],
|
||||
"failure_reports": [],
|
||||
"smtp_tls_reports": [],
|
||||
}
|
||||
|
||||
mock_load_config.return_value = ConfigParser()
|
||||
|
||||
parse_calls = [0]
|
||||
|
||||
def parse_side_effect(config, opts):
|
||||
parse_calls[0] += 1
|
||||
opts.imap_host = "imap.example.com"
|
||||
opts.imap_user = "user"
|
||||
opts.imap_password = "pass"
|
||||
opts.mailbox_watch = True
|
||||
opts.dns_timeout = 5.0 if parse_calls[0] == 1 else 42.0
|
||||
return None
|
||||
|
||||
mock_parse_config.side_effect = parse_side_effect
|
||||
mock_init_clients.return_value = {}
|
||||
|
||||
watch_calls = [0]
|
||||
|
||||
def watch_side_effect(*args, **kwargs):
|
||||
watch_calls[0] += 1
|
||||
if watch_calls[0] == 1:
|
||||
if hasattr(signal_module, "SIGHUP"):
|
||||
import os
|
||||
|
||||
os.kill(os.getpid(), signal_module.SIGHUP)
|
||||
return
|
||||
else:
|
||||
raise FileExistsError("stop-watch-loop")
|
||||
|
||||
mock_watch.side_effect = watch_side_effect
|
||||
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".ini", delete=False) as cfg:
|
||||
cfg.write(self._BASE_CONFIG)
|
||||
cfg_path = cfg.name
|
||||
self.addCleanup(lambda: os.path.exists(cfg_path) and os.remove(cfg_path))
|
||||
|
||||
with patch.object(sys, "argv", ["parsedmarc", "-c", cfg_path]):
|
||||
with self.assertRaises(SystemExit):
|
||||
parsedmarc.cli._main()
|
||||
|
||||
self.assertEqual(mock_watch.call_count, 2)
|
||||
first_config = mock_watch.call_args_list[0].kwargs["config"]
|
||||
second_config = mock_watch.call_args_list[1].kwargs["config"]
|
||||
self.assertEqual(first_config.dns_timeout, 5.0)
|
||||
self.assertEqual(second_config.dns_timeout, 42.0)
|
||||
|
||||
|
||||
class TestSigtermShutdown(unittest.TestCase):
|
||||
"""Tests for graceful SIGTERM/SIGINT shutdown."""
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
"""Tests for parsedmarc.config"""
|
||||
|
||||
import dataclasses
|
||||
import pickle
|
||||
import unittest
|
||||
|
||||
import parsedmarc
|
||||
import parsedmarc.config as parsedmarc_config
|
||||
|
||||
|
||||
class TestParserConfigCaches(unittest.TestCase):
|
||||
"""Covers per-instance cache isolation for `ParserConfig`."""
|
||||
|
||||
def test_each_instance_gets_isolated_caches(self):
|
||||
"""Two independently constructed ParserConfig instances must never
|
||||
share cache objects with each other or with the module defaults, and
|
||||
mutating one instance's caches must not leak into the others."""
|
||||
cfg_a = parsedmarc_config.ParserConfig()
|
||||
cfg_b = parsedmarc_config.ParserConfig()
|
||||
|
||||
# All distinct objects.
|
||||
self.assertIsNot(cfg_a.ip_address_cache, cfg_b.ip_address_cache)
|
||||
self.assertIsNot(
|
||||
cfg_a.seen_aggregate_report_ids, cfg_b.seen_aggregate_report_ids
|
||||
)
|
||||
self.assertIsNot(cfg_a.reverse_dns_map, cfg_b.reverse_dns_map)
|
||||
|
||||
self.assertIsNot(cfg_a.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertIsNot(
|
||||
cfg_a.seen_aggregate_report_ids,
|
||||
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
|
||||
)
|
||||
self.assertIsNot(cfg_a.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
self.assertIsNot(cfg_b.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertIsNot(
|
||||
cfg_b.seen_aggregate_report_ids,
|
||||
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
|
||||
)
|
||||
self.assertIsNot(cfg_b.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
# Mutating one instance's caches must not affect the other, nor the
|
||||
# module defaults.
|
||||
cfg_a.ip_address_cache["1.2.3.4"] = {"ip_address": "1.2.3.4"}
|
||||
cfg_a.seen_aggregate_report_ids["report-id-a"] = True
|
||||
cfg_a.reverse_dns_map["example.com"] = {"name": "Example", "type": None}
|
||||
|
||||
self.assertNotIn("1.2.3.4", cfg_b.ip_address_cache)
|
||||
self.assertNotIn("report-id-a", cfg_b.seen_aggregate_report_ids)
|
||||
self.assertNotIn("example.com", cfg_b.reverse_dns_map)
|
||||
|
||||
self.assertNotIn("1.2.3.4", parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertNotIn("report-id-a", parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS)
|
||||
self.assertNotIn("example.com", parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
def test_module_default_caches_are_the_public_globals(self):
|
||||
"""The three module-default caches in `parsedmarc.config` must be the
|
||||
very same objects re-exported as `parsedmarc.IP_ADDRESS_CACHE`,
|
||||
`parsedmarc.SEEN_AGGREGATE_REPORT_IDS`, and
|
||||
`parsedmarc.REVERSE_DNS_MAP`, so that pre-refactor callers who
|
||||
imported these names directly from the top-level package keep
|
||||
observing the same cache objects as code that goes through
|
||||
ParserConfig.
|
||||
"""
|
||||
self.assertIs(parsedmarc.IP_ADDRESS_CACHE, parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertIs(
|
||||
parsedmarc.SEEN_AGGREGATE_REPORT_IDS,
|
||||
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
|
||||
)
|
||||
self.assertIs(parsedmarc.REVERSE_DNS_MAP, parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
|
||||
class TestParserConfigPickling(unittest.TestCase):
|
||||
"""Covers the pickle strategy documented on `ParserConfig`: option fields
|
||||
round-trip, but cache contents never cross process boundaries and the
|
||||
unpickled object's caches rebind to the unpickling process's module
|
||||
defaults."""
|
||||
|
||||
def test_pickle_drops_cache_contents_and_binds_process_defaults(self):
|
||||
"""Pickling and unpickling a ParserConfig must preserve option
|
||||
fields, but must NOT carry cache contents across the round-trip, and
|
||||
must leave the unpickled instance's caches bound to this module's
|
||||
(the "unpickling process's") default cache objects rather than
|
||||
fresh, empty ones. Fresh-per-unpickle caches would silently defeat
|
||||
per-worker caching, since a functools.partial payload carrying a
|
||||
ParserConfig is re-pickled for every task submitted to a
|
||||
multiprocessing worker."""
|
||||
cfg = parsedmarc_config.ParserConfig(
|
||||
dns_timeout=9.5, nameservers=["192.0.2.53"]
|
||||
)
|
||||
cfg.ip_address_cache["sentinel-ip"] = "sentinel-ip-value"
|
||||
cfg.seen_aggregate_report_ids["sentinel-report-id"] = True
|
||||
cfg.reverse_dns_map["sentinel.example"] = {
|
||||
"name": "Sentinel",
|
||||
"type": None,
|
||||
}
|
||||
|
||||
restored = pickle.loads(pickle.dumps(cfg))
|
||||
|
||||
# Option fields compare equal (cache fields are compare=False).
|
||||
self.assertEqual(cfg, restored)
|
||||
self.assertEqual(restored.dns_timeout, 9.5)
|
||||
self.assertEqual(restored.nameservers, ["192.0.2.53"])
|
||||
|
||||
# Caches rebind to this process's module defaults, not fresh copies.
|
||||
self.assertIs(restored.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertIs(
|
||||
restored.seen_aggregate_report_ids,
|
||||
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
|
||||
)
|
||||
self.assertIs(restored.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
# Sentinel entries placed on the original instance's caches must NOT
|
||||
# have crossed into the module defaults.
|
||||
self.assertNotIn("sentinel-ip", parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertNotIn(
|
||||
"sentinel-report-id", parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS
|
||||
)
|
||||
self.assertNotIn("sentinel.example", parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
def test_setstate_defaults_fields_missing_from_older_pickles(self):
|
||||
"""__setstate__ must initialize every non-cache field to its class
|
||||
default before applying the pickled state, so a ParserConfig
|
||||
serialized by an older parsedmarc version (whose state lacks fields
|
||||
added since) unpickles with the newer fields at their defaults
|
||||
instead of unset — __init__ never runs during unpickling, so an
|
||||
absent field would otherwise raise AttributeError on first access.
|
||||
Simulated by calling __setstate__ directly with a partial state
|
||||
dict, exactly what pickle.loads does with an old payload."""
|
||||
restored = object.__new__(parsedmarc_config.ParserConfig)
|
||||
restored.__setstate__({"offline": True, "dns_timeout": 7.5})
|
||||
|
||||
self.assertTrue(restored.offline)
|
||||
self.assertEqual(restored.dns_timeout, 7.5)
|
||||
# Fields absent from the old state get their class defaults.
|
||||
self.assertIsNone(restored.ip_db_path)
|
||||
self.assertIsNone(restored.nameservers)
|
||||
self.assertEqual(restored.normalize_timespan_threshold_hours, 24.0)
|
||||
# Caches still rebind to the module defaults.
|
||||
self.assertIs(restored.ip_address_cache, parsedmarc_config.IP_ADDRESS_CACHE)
|
||||
self.assertIs(
|
||||
restored.seen_aggregate_report_ids,
|
||||
parsedmarc_config.SEEN_AGGREGATE_REPORT_IDS,
|
||||
)
|
||||
self.assertIs(restored.reverse_dns_map, parsedmarc_config.REVERSE_DNS_MAP)
|
||||
|
||||
|
||||
class TestParserConfigFrozenAndReplace(unittest.TestCase):
|
||||
"""Covers frozen-dataclass immutability and the `dataclasses.replace`
|
||||
escape hatch for deriving variants that keep sharing caches."""
|
||||
|
||||
def test_frozen_and_replace_shares_caches(self):
|
||||
"""ParserConfig is frozen, so attribute assignment must raise
|
||||
FrozenInstanceError. `dataclasses.replace(cfg, ...)` must return a
|
||||
new config that shares the SAME cache objects as the source (all
|
||||
fields, including the three cache fields, are init fields), since
|
||||
that is the documented way to derive a variant that keeps warm
|
||||
caches."""
|
||||
cfg = parsedmarc_config.ParserConfig()
|
||||
|
||||
with self.assertRaises(dataclasses.FrozenInstanceError):
|
||||
cfg.offline = True # type: ignore[misc]
|
||||
|
||||
variant = dataclasses.replace(cfg, offline=True)
|
||||
|
||||
self.assertTrue(variant.offline)
|
||||
self.assertIs(variant.ip_address_cache, cfg.ip_address_cache)
|
||||
self.assertIs(variant.seen_aggregate_report_ids, cfg.seen_aggregate_report_ids)
|
||||
self.assertIs(variant.reverse_dns_map, cfg.reverse_dns_map)
|
||||
+249
-13
@@ -7,6 +7,7 @@ extract_report, get_dmarc_reports_from_mbox, and the CSV / JSON renderers.
|
||||
|
||||
import base64
|
||||
import gzip
|
||||
import inspect
|
||||
import json
|
||||
import logging
|
||||
import mailbox
|
||||
@@ -24,6 +25,7 @@ from unittest.mock import MagicMock, patch
|
||||
from lxml import etree # type: ignore[import-untyped]
|
||||
|
||||
import parsedmarc
|
||||
import parsedmarc.constants as constants
|
||||
from parsedmarc.mail import MaildirConnection, MSGraphConnection
|
||||
from parsedmarc.types import (
|
||||
AggregateReport,
|
||||
@@ -749,7 +751,9 @@ class Test(unittest.TestCase):
|
||||
"auth_results": {"dkim": [], "spf": []},
|
||||
}
|
||||
with self.assertRaises(ValueError):
|
||||
parsedmarc._parse_report_record(record, offline=True)
|
||||
parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
|
||||
def testParseReportRecordMissingDkimSpf(self):
|
||||
"""Record with missing dkim/spf auth results defaults correctly"""
|
||||
@@ -766,7 +770,9 @@ class Test(unittest.TestCase):
|
||||
"identifiers": {"header_from": "example.com"},
|
||||
"auth_results": {},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertEqual(result["auth_results"]["dkim"], [])
|
||||
self.assertEqual(result["auth_results"]["spf"], [])
|
||||
|
||||
@@ -786,7 +792,9 @@ class Test(unittest.TestCase):
|
||||
"identifiers": {"header_from": "example.com"},
|
||||
"auth_results": {"dkim": [], "spf": []},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
reasons = result["policy_evaluated"]["policy_override_reasons"]
|
||||
self.assertEqual(len(reasons), 1)
|
||||
self.assertEqual(reasons[0]["type"], "forwarded")
|
||||
@@ -811,7 +819,9 @@ class Test(unittest.TestCase):
|
||||
"identifiers": {"header_from": "example.com"},
|
||||
"auth_results": {"dkim": [], "spf": []},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
reasons = result["policy_evaluated"]["policy_override_reasons"]
|
||||
self.assertEqual(len(reasons), 2)
|
||||
self.assertEqual(reasons[0]["comment"], "relay")
|
||||
@@ -835,7 +845,9 @@ class Test(unittest.TestCase):
|
||||
},
|
||||
"auth_results": {"dkim": [], "spf": []},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertIn("identifiers", result)
|
||||
self.assertEqual(result["identifiers"]["header_from"], "example.com")
|
||||
|
||||
@@ -857,7 +869,9 @@ class Test(unittest.TestCase):
|
||||
"spf": [],
|
||||
},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
dkim = result["auth_results"]["dkim"][0]
|
||||
self.assertEqual(dkim["selector"], "none")
|
||||
self.assertEqual(dkim["result"], "none")
|
||||
@@ -881,7 +895,9 @@ class Test(unittest.TestCase):
|
||||
"spf": {"domain": "example.com"},
|
||||
},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
spf = result["auth_results"]["spf"][0]
|
||||
self.assertEqual(spf["scope"], "mfrom")
|
||||
self.assertEqual(spf["result"], "none")
|
||||
@@ -919,7 +935,9 @@ class Test(unittest.TestCase):
|
||||
],
|
||||
},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertEqual(result["auth_results"]["dkim"][0]["human_result"], "good key")
|
||||
self.assertEqual(
|
||||
result["auth_results"]["spf"][0]["human_result"], "sender valid"
|
||||
@@ -945,7 +963,9 @@ class Test(unittest.TestCase):
|
||||
],
|
||||
},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertEqual(result["identifiers"]["envelope_from"], "bounce.example.com")
|
||||
|
||||
def testParseReportRecordEnvelopeFromNullFallback(self):
|
||||
@@ -971,7 +991,9 @@ class Test(unittest.TestCase):
|
||||
],
|
||||
},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertEqual(result["identifiers"]["envelope_from"], "spf.example.com")
|
||||
|
||||
def testParseReportRecordEnvelopeFromNullNoSpfDomain(self):
|
||||
@@ -998,7 +1020,9 @@ class Test(unittest.TestCase):
|
||||
"spf": [{"scope": "mfrom", "result": "pass"}],
|
||||
},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertIsNone(result["identifiers"]["envelope_from"])
|
||||
|
||||
def testParseReportRecordEnvelopeTo(self):
|
||||
@@ -1020,7 +1044,9 @@ class Test(unittest.TestCase):
|
||||
},
|
||||
"auth_results": {"dkim": [], "spf": []},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertEqual(result["identifiers"]["envelope_to"], "recipient@example.com")
|
||||
|
||||
def testParseReportRecordAlignment(self):
|
||||
@@ -1038,7 +1064,9 @@ class Test(unittest.TestCase):
|
||||
"identifiers": {"header_from": "example.com"},
|
||||
"auth_results": {"dkim": [], "spf": []},
|
||||
}
|
||||
result = parsedmarc._parse_report_record(record, offline=True)
|
||||
result = parsedmarc._parse_report_record(
|
||||
record, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertTrue(result["alignment"]["dkim"])
|
||||
self.assertFalse(result["alignment"]["spf"])
|
||||
self.assertTrue(result["alignment"]["dmarc"])
|
||||
@@ -2709,6 +2737,214 @@ class TestGetDmarcReportsFromMboxParallel(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestCentralizedConfig(unittest.TestCase):
|
||||
"""Regression coverage for the centralize-config-503 refactor
|
||||
(parsedmarc/config.py's ``ParserConfig``): the kwargs-style public API
|
||||
must keep observing/mutating the same module-default caches it always
|
||||
has (via ``_resolve_config`` injecting ``IP_ADDRESS_CACHE`` /
|
||||
``SEEN_AGGREGATE_REPORT_IDS`` / ``REVERSE_DNS_MAP`` rather than letting
|
||||
a fresh ``ParserConfig()`` default-factory hand back empty ones), an
|
||||
explicit ``config=`` must win over individual option keyword arguments
|
||||
when both are given, and the DNS-timeout/retry/normalize-threshold
|
||||
defaults must stay consistent between each public function's own
|
||||
signature and ``ParserConfig``'s field defaults.
|
||||
"""
|
||||
|
||||
AGGREGATE = "samples/aggregate/twilight.eml"
|
||||
|
||||
def _build_single_aggregate_mbox(self) -> str:
|
||||
"""Builds a temporary mbox containing one copy of AGGREGATE."""
|
||||
tmp = mkdtemp()
|
||||
self.addCleanup(rmtree, tmp, ignore_errors=True)
|
||||
path = os.path.join(tmp, "reports.mbox")
|
||||
box = mailbox.mbox(path)
|
||||
box.lock()
|
||||
try:
|
||||
with open(self.AGGREGATE, "rb") as source_file:
|
||||
box.add(mailbox.mboxMessage(source_file.read()))
|
||||
box.flush()
|
||||
finally:
|
||||
box.unlock()
|
||||
box.close()
|
||||
return path
|
||||
|
||||
def test_kwargs_path_uses_module_default_caches_not_fresh_ones(self):
|
||||
"""Regression guard for _resolve_config: calling
|
||||
get_dmarc_reports_from_mbox with plain kwargs (no config=) twice in
|
||||
a row over the same mbox must dedup the second run's aggregate
|
||||
report against the first run's, because both calls must resolve to
|
||||
the SAME module-default parsedmarc.SEEN_AGGREGATE_REPORT_IDS cache.
|
||||
If _resolve_config ever let the kwargs path fall through to
|
||||
ParserConfig's default_factory instead of explicitly injecting the
|
||||
module-default caches, each call would get a fresh, empty cache and
|
||||
this dedup would silently stop working.
|
||||
"""
|
||||
parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear()
|
||||
self.addCleanup(parsedmarc.SEEN_AGGREGATE_REPORT_IDS.clear)
|
||||
path = self._build_single_aggregate_mbox()
|
||||
|
||||
first = parsedmarc.get_dmarc_reports_from_mbox(path, offline=True)
|
||||
second = parsedmarc.get_dmarc_reports_from_mbox(path, offline=True)
|
||||
|
||||
self.assertEqual(len(first["aggregate_reports"]), 1)
|
||||
self.assertEqual(len(second["aggregate_reports"]), 0)
|
||||
|
||||
report_metadata = first["aggregate_reports"][0]["report_metadata"]
|
||||
report_key = f"{report_metadata['org_name']}_{report_metadata['report_id']}"
|
||||
self.assertIn(report_key, parsedmarc.SEEN_AGGREGATE_REPORT_IDS)
|
||||
|
||||
def test_kwargs_and_config_equivalence(self):
|
||||
"""parse_report_file must produce identical results whether called
|
||||
with individual option keyword arguments or an equivalent explicit
|
||||
ParserConfig, for one sample of each report type. None of these
|
||||
paths touch dedup (that's only in _classify_parsed_email /
|
||||
get_dmarc_reports_from_mbox / get_dmarc_reports_from_mailbox), so
|
||||
no cache clearing is needed between calls.
|
||||
"""
|
||||
sample_paths = [
|
||||
"samples/aggregate/rfc9990-sample.xml",
|
||||
"samples/failure/dmarc_ruf_report_linkedin.eml",
|
||||
"samples/smtp_tls/google.com_smtp_tls_report.eml",
|
||||
]
|
||||
for sample_path in sample_paths:
|
||||
with self.subTest(sample=sample_path):
|
||||
kwargs_result = parsedmarc.parse_report_file(
|
||||
sample_path, offline=True, always_use_local_files=True
|
||||
)
|
||||
config_result = parsedmarc.parse_report_file(
|
||||
sample_path,
|
||||
config=parsedmarc.ParserConfig(
|
||||
offline=True, always_use_local_files=True
|
||||
),
|
||||
)
|
||||
self.assertEqual(kwargs_result, config_result)
|
||||
|
||||
def test_config_wins_over_kwargs_normalize_threshold(self):
|
||||
"""When both config= and normalize_timespan_threshold_hours= are
|
||||
given, the config's value must win -- per the documented contract,
|
||||
an explicit config makes the individual option kwargs inert.
|
||||
|
||||
samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml
|
||||
spans exactly 86400 seconds (24h), so a 1.0-hour threshold
|
||||
normalizes it and a 1000-hour threshold does not; this makes the
|
||||
config-vs-kwarg outcome observable in normalized_timespan.
|
||||
"""
|
||||
sample_path = "samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml"
|
||||
with open(sample_path, "rb") as f:
|
||||
data = f.read()
|
||||
|
||||
# Config's high threshold must win over the kwarg's low threshold:
|
||||
# the report must NOT be normalized.
|
||||
report = parsedmarc.parse_aggregate_report_file(
|
||||
data,
|
||||
offline=True,
|
||||
config=parsedmarc.ParserConfig(
|
||||
offline=True, normalize_timespan_threshold_hours=1000.0
|
||||
),
|
||||
normalize_timespan_threshold_hours=1.0,
|
||||
)
|
||||
for record in report["records"]:
|
||||
self.assertFalse(record["normalized_timespan"]) # type: ignore[typeddict-item]
|
||||
|
||||
# Converse: config's low threshold must win over the kwarg's high
|
||||
# threshold: the report MUST be normalized.
|
||||
report = parsedmarc.parse_aggregate_report_file(
|
||||
data,
|
||||
offline=True,
|
||||
config=parsedmarc.ParserConfig(
|
||||
offline=True, normalize_timespan_threshold_hours=1.0
|
||||
),
|
||||
normalize_timespan_threshold_hours=1000.0,
|
||||
)
|
||||
for record in report["records"]:
|
||||
self.assertTrue(record["normalized_timespan"]) # type: ignore[typeddict-item]
|
||||
|
||||
def test_separate_configs_isolate_dedup_state(self):
|
||||
"""Two independently constructed ParserConfig(offline=True)
|
||||
instances must NOT share dedup state (each has its own
|
||||
seen_aggregate_report_ids cache), but reusing the SAME instance
|
||||
across two calls must dedup, exactly like the module-default-cache
|
||||
kwargs path does.
|
||||
"""
|
||||
path = self._build_single_aggregate_mbox()
|
||||
|
||||
cfg_a = parsedmarc.ParserConfig(offline=True)
|
||||
cfg_b = parsedmarc.ParserConfig(offline=True)
|
||||
result_a = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_a)
|
||||
result_b = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_b)
|
||||
self.assertEqual(len(result_a["aggregate_reports"]), 1)
|
||||
self.assertEqual(len(result_b["aggregate_reports"]), 1)
|
||||
|
||||
cfg_c = parsedmarc.ParserConfig(offline=True)
|
||||
first = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_c)
|
||||
second = parsedmarc.get_dmarc_reports_from_mbox(path, config=cfg_c)
|
||||
self.assertEqual(len(first["aggregate_reports"]), 1)
|
||||
self.assertEqual(len(second["aggregate_reports"]), 0)
|
||||
|
||||
def test_dns_and_normalize_defaults_match_constants_and_parser_config(self):
|
||||
"""DNS timeout/retries and normalize-timespan-threshold defaults
|
||||
must match parsedmarc.constants (the authoritative source --
|
||||
DEFAULT_DNS_TIMEOUT, DEFAULT_DNS_MAX_RETRIES) and
|
||||
parsedmarc.config.ParserConfig's own field defaults, across every
|
||||
public function that accepts them.
|
||||
|
||||
Regression guard: before this refactor, get_dmarc_reports_from_mailbox
|
||||
and watch_inbox each had a stray literal ``dns_timeout=6.0`` (instead
|
||||
of ``DEFAULT_DNS_TIMEOUT == 2.0``) and
|
||||
``normalize_timespan_threshold_hours=24`` (an int, instead of the
|
||||
float ``24.0`` used everywhere else) -- exactly the kind of drift
|
||||
_resolve_config's shared construction path is meant to prevent from
|
||||
silently reappearing.
|
||||
"""
|
||||
functions_and_dns_params = [
|
||||
(parsedmarc.parse_aggregate_report_xml, "timeout", "retries"),
|
||||
(parsedmarc.parse_aggregate_report_file, "dns_timeout", "dns_retries"),
|
||||
(parsedmarc.parse_failure_report, "dns_timeout", "dns_retries"),
|
||||
(parsedmarc.parse_report_email, "dns_timeout", "dns_retries"),
|
||||
(parsedmarc.parse_report_file, "dns_timeout", "dns_retries"),
|
||||
(parsedmarc.get_dmarc_reports_from_mbox, "dns_timeout", "dns_retries"),
|
||||
(
|
||||
parsedmarc.get_dmarc_reports_from_mailbox,
|
||||
"dns_timeout",
|
||||
"dns_retries",
|
||||
),
|
||||
(parsedmarc.watch_inbox, "dns_timeout", "dns_retries"),
|
||||
]
|
||||
|
||||
default_config = parsedmarc.ParserConfig()
|
||||
|
||||
for func, timeout_param, retries_param in functions_and_dns_params:
|
||||
with self.subTest(func=func.__name__, param="dns"):
|
||||
sig = inspect.signature(func)
|
||||
timeout_default = sig.parameters[timeout_param].default
|
||||
retries_default = sig.parameters[retries_param].default
|
||||
self.assertEqual(timeout_default, constants.DEFAULT_DNS_TIMEOUT)
|
||||
self.assertEqual(retries_default, constants.DEFAULT_DNS_MAX_RETRIES)
|
||||
self.assertEqual(timeout_default, default_config.dns_timeout)
|
||||
self.assertEqual(retries_default, default_config.dns_retries)
|
||||
|
||||
# parse_failure_report has no normalize_timespan_threshold_hours
|
||||
# parameter -- normalization is an aggregate-report-only concept.
|
||||
normalize_funcs = [
|
||||
parsedmarc.parse_aggregate_report_xml,
|
||||
parsedmarc.parse_aggregate_report_file,
|
||||
parsedmarc.parse_report_email,
|
||||
parsedmarc.parse_report_file,
|
||||
parsedmarc.get_dmarc_reports_from_mbox,
|
||||
parsedmarc.get_dmarc_reports_from_mailbox,
|
||||
parsedmarc.watch_inbox,
|
||||
]
|
||||
for func in normalize_funcs:
|
||||
with self.subTest(func=func.__name__, param="normalize"):
|
||||
sig = inspect.signature(func)
|
||||
default = sig.parameters["normalize_timespan_threshold_hours"].default
|
||||
self.assertEqual(default, 24.0)
|
||||
self.assertIsInstance(default, float)
|
||||
self.assertEqual(
|
||||
default, default_config.normalize_timespan_threshold_hours
|
||||
)
|
||||
|
||||
|
||||
class TestGetDmarcReportsFromMailboxValidation(unittest.TestCase):
|
||||
"""Input validation on get_dmarc_reports_from_mailbox.
|
||||
|
||||
|
||||
+10
-4
@@ -68,7 +68,9 @@ class TestParallelMapParseReportFile(_ParallelTestCase):
|
||||
for path in SAMPLE_PATHS
|
||||
]
|
||||
|
||||
job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True))
|
||||
job = functools.partial(
|
||||
_parse_report_file_job, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
results = list(parallel_map(job, SAMPLE_PATHS, n_procs=2))
|
||||
|
||||
self.assertEqual(len(results), len(SAMPLE_PATHS))
|
||||
@@ -94,7 +96,9 @@ class TestParallelMapJunkFile(_ParallelTestCase):
|
||||
junk_path = tf.name
|
||||
self.addCleanup(os.remove, junk_path)
|
||||
|
||||
job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True))
|
||||
job = functools.partial(
|
||||
_parse_report_file_job, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
results = list(parallel_map(job, [junk_path, junk_path], n_procs=2))
|
||||
|
||||
self.assertEqual(len(results), 2)
|
||||
@@ -208,7 +212,9 @@ class TestWorkerLogging(_ParallelTestCase):
|
||||
configure_logging(logging.DEBUG, log_path)
|
||||
|
||||
sample = SAMPLE_PATHS[0]
|
||||
job = functools.partial(_parse_report_file_job, kwargs=dict(offline=True))
|
||||
job = functools.partial(
|
||||
_parse_report_file_job, config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
results = list(parallel_map(job, [sample], n_procs=2))
|
||||
|
||||
self.assertEqual(len(results), 1)
|
||||
@@ -275,7 +281,7 @@ class TestParseReportEmailJob(_ParallelTestCase):
|
||||
|
||||
def test_invalid_email_returns_parser_error_value(self):
|
||||
result = _parse_report_email_job(
|
||||
b"not a valid email", kwargs=dict(offline=True)
|
||||
b"not a valid email", config=parsedmarc.ParserConfig(offline=True)
|
||||
)
|
||||
self.assertIsInstance(result, parsedmarc.ParserError)
|
||||
|
||||
|
||||
@@ -490,6 +490,67 @@ class TestLoadReverseDnsMapReloadsPSLOverrides(unittest.TestCase):
|
||||
offline=True,
|
||||
)
|
||||
|
||||
def _write_psl_overrides_file(self):
|
||||
"""Write a temp PSL overrides file with a unique suffix and register
|
||||
cleanup, returning its path."""
|
||||
tf = tempfile.NamedTemporaryFile(
|
||||
"w", suffix=".txt", delete=False, encoding="utf-8"
|
||||
)
|
||||
tf.write(".internal.example-503.net\n")
|
||||
tf.close()
|
||||
self.addCleanup(os.unlink, tf.name)
|
||||
return tf.name
|
||||
|
||||
def test_lazy_reverse_dns_map_load_applies_psl_overrides(self):
|
||||
"""Regression test for GitHub issue #503: the lazy
|
||||
``load_reverse_dns_map()`` call inside
|
||||
``get_service_from_reverse_dns_base_domain`` previously omitted
|
||||
``psl_overrides_path``/``psl_overrides_url``, so
|
||||
``load_reverse_dns_map``'s unconditional ``load_psl_overrides()``
|
||||
call (utils.py) silently reloaded ``psl_overrides`` with the bundled
|
||||
defaults, discarding an operator-configured overrides file. This
|
||||
proves the lazy load now threads the caller's
|
||||
``psl_overrides_path`` through: after calling with an empty
|
||||
``reverse_dns_map`` (which forces the lazy load) and a custom
|
||||
``psl_overrides_path``, the custom override must be in effect.
|
||||
"""
|
||||
path = self._write_psl_overrides_file()
|
||||
parsedmarc.utils.get_service_from_reverse_dns_base_domain(
|
||||
"something.example",
|
||||
reverse_dns_map={},
|
||||
always_use_local_file=True,
|
||||
offline=True,
|
||||
psl_overrides_path=path,
|
||||
)
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.get_base_domain("deep.sub.internal.example-503.net"),
|
||||
"internal.example-503.net",
|
||||
)
|
||||
|
||||
def test_get_ip_address_info_lazy_load_applies_psl_overrides(self):
|
||||
"""Regression test for GitHub issue #503, exercising the
|
||||
ASN-fallback lazy ``load_reverse_dns_map()`` call inside
|
||||
``get_ip_address_info`` (used when no PTR record resolves). Before
|
||||
the fix, this lazy load also omitted ``psl_overrides_path``/
|
||||
``psl_overrides_url``, so it clobbered operator-configured PSL
|
||||
overrides with the bundled defaults the same way. ``offline=True``
|
||||
forces the no-PTR path so this call reaches the ASN-fallback lazy
|
||||
load rather than the PTR-driven
|
||||
``get_service_from_reverse_dns_base_domain`` call.
|
||||
"""
|
||||
path = self._write_psl_overrides_file()
|
||||
parsedmarc.utils.get_ip_address_info(
|
||||
"192.0.2.1",
|
||||
offline=True,
|
||||
reverse_dns_map={},
|
||||
always_use_local_files=True,
|
||||
psl_overrides_path=path,
|
||||
)
|
||||
self.assertEqual(
|
||||
parsedmarc.utils.get_base_domain("deep.sub.internal.example-503.net"),
|
||||
"internal.example-503.net",
|
||||
)
|
||||
|
||||
|
||||
class TestGetBaseDomainWithOverrides(unittest.TestCase):
|
||||
"""`get_base_domain` must honour the current psl_overrides list."""
|
||||
|
||||
Reference in New Issue
Block a user