amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-08-02 13:42:19 +00:00
Code Issues Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Sean Whalen
2026-07-26 18:20:03 -04:00
parent d2d4c980ae
commit 39037ba327
31 changed files with 2160 additions and 397 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_modules/index.html
+4 -3
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Overview: module code &mdash; parsedmarc 10.3.0 documentation</title>
<title>Overview: module code &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=9edc463e" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=a126560f"></script>
<script src="../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../_static/js/theme.js"></script>
@@ -79,7 +79,8 @@
<h1>All modules for which code is available</h1>
<ul><li><a href="parsedmarc.html">parsedmarc</a></li>
<ul><li><a href="parsedmarc/elastic.html">parsedmarc.elastic</a></li>
<ul><li><a href="parsedmarc/config.html">parsedmarc.config</a></li>
<li><a href="parsedmarc/elastic.html">parsedmarc.elastic</a></li>
<li><a href="parsedmarc/opensearch.html">parsedmarc.opensearch</a></li>
<li><a href="parsedmarc/splunk.html">parsedmarc.splunk</a></li>
<li><a href="parsedmarc/types.html">parsedmarc.types</a></li>
_modules/parsedmarc.html
+901 -300
View File
File diff suppressed because it is too large Load Diff
_modules/parsedmarc/config.html
+298
View File
@@ -0,0 +1,298 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.config &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../index.html" class="icon icon-home">
parsedmarc
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../usage.html">Using parsedmarc</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../output.html">Sample outputs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../elasticsearch.html">Elasticsearch and Kibana</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../opensearch.html">OpenSearch and Grafana</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../kibana.html">Using the Kibana dashboards</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../splunk.html">Splunk</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../davmail.html">Accessing an inbox using OWA/EWS</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dmarc.html">Understanding DMARC</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributing.html">Contributing to parsedmarc</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api.html">API reference</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../index.html">parsedmarc</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../index.html">Module code</a></li>
<li class="breadcrumb-item"><a href="../parsedmarc.html">parsedmarc</a></li>
<li class="breadcrumb-item active">parsedmarc.config</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for parsedmarc.config</h1><div class="highlight"><pre>
<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="sd">&quot;&quot;&quot;Centralized parser configuration (GitHub issue #503).</span>
<span class="sd">This module defines :class:`ParserConfig`, a single frozen dataclass that</span>
<span class="sd">carries every parsing/enrichment option (offline mode, IP database path,</span>
<span class="sd">reverse DNS map location, PSL overrides location, DNS behavior, etc.) plus</span>
<span class="sd">the three caches that the parser and enrichment code share across calls:</span>
<span class="sd">the IP address info cache, the seen-aggregate-report-ID cache, and the</span>
<span class="sd">reverse DNS map.</span>
<span class="sd">``parsedmarc/__init__.py`` re-exports :data:`IP_ADDRESS_CACHE`,</span>
<span class="sd">:data:`SEEN_AGGREGATE_REPORT_IDS`, and :data:`REVERSE_DNS_MAP` under the</span>
<span class="sd">same names for backward compatibility. That compatibility is</span>
<span class="sd">identity-based: ``parsedmarc.IP_ADDRESS_CACHE is parsedmarc.config.IP_ADDRESS_CACHE``</span>
<span class="sd">must hold, not merely equal contents, since callers may mutate the caches</span>
<span class="sd">in place.</span>
<span class="sd">Only :mod:`parsedmarc.constants` and :mod:`parsedmarc.utils` are imported</span>
<span class="sd">from the package here — importing :mod:`parsedmarc` itself would be</span>
<span class="sd">circular, since ``parsedmarc/__init__.py`` imports from this module.</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">__future__</span><span class="w"> </span><span class="kn">import</span> <span class="n">annotations</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">dataclasses</span><span class="w"> </span><span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span><span class="p">,</span> <span class="n">fields</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">expiringdict</span><span class="w"> </span><span class="kn">import</span> <span class="n">ExpiringDict</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc.constants</span><span class="w"> </span><span class="kn">import</span> <span class="n">DEFAULT_DNS_MAX_RETRIES</span><span class="p">,</span> <span class="n">DEFAULT_DNS_TIMEOUT</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">ReverseDNSMap</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_new_ip_address_cache</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">ExpiringDict</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Build a fresh IP-address-info cache (4 hour expiry).&quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">ExpiringDict</span><span class="p">(</span><span class="n">max_len</span><span class="o">=</span><span class="mi">10000</span><span class="p">,</span> <span class="n">max_age_seconds</span><span class="o">=</span><span class="mi">14400</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_new_seen_aggregate_report_ids</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">ExpiringDict</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Build a fresh seen-aggregate-report-ID cache (1 hour expiry).&quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">ExpiringDict</span><span class="p">(</span><span class="n">max_len</span><span class="o">=</span><span class="mi">100000000</span><span class="p">,</span> <span class="n">max_age_seconds</span><span class="o">=</span><span class="mi">3600</span><span class="p">)</span>
<span class="c1"># Canonical default caches. parsedmarc/__init__.py re-exports these three</span>
<span class="c1"># objects under the same names (IP_ADDRESS_CACHE, SEEN_AGGREGATE_REPORT_IDS,</span>
<span class="c1"># REVERSE_DNS_MAP) for backward compatibility with code that imported them</span>
<span class="c1"># directly from the top-level package. That compatibility promise is</span>
<span class="c1"># identity-based — the re-exported names must point at these very same</span>
<span class="c1"># objects, not merely equivalent ones — so that library callers who obtained</span>
<span class="c1"># a reference before this refactor keep observing the same mutations as code</span>
<span class="c1"># that goes through a ParserConfig built from these defaults (e.g. during</span>
<span class="c1"># unpickling in a multiprocessing worker; see ParserConfig.__setstate__).</span>
<span class="n">IP_ADDRESS_CACHE</span> <span class="o">=</span> <span class="n">_new_ip_address_cache</span><span class="p">()</span>
<span class="n">SEEN_AGGREGATE_REPORT_IDS</span> <span class="o">=</span> <span class="n">_new_seen_aggregate_report_ids</span><span class="p">()</span>
<span class="n">REVERSE_DNS_MAP</span><span class="p">:</span> <span class="n">ReverseDNSMap</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># The ParserConfig fields excluded from pickling; every other field must</span>
<span class="c1"># survive the round-trip, so __getstate__ derives its contents from</span>
<span class="c1"># dataclasses.fields() rather than enumerating option fields by hand.</span>
<span class="n">_CACHE_FIELD_NAMES</span> <span class="o">=</span> <span class="p">(</span>
<span class="s2">&quot;ip_address_cache&quot;</span><span class="p">,</span>
<span class="s2">&quot;seen_aggregate_report_ids&quot;</span><span class="p">,</span>
<span class="s2">&quot;reverse_dns_map&quot;</span><span class="p">,</span>
<span class="p">)</span>
<div class="viewcode-block" id="ParserConfig">
<a class="viewcode-back" href="../../api.html#parsedmarc.config.ParserConfig">[docs]</a>
<span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">ParserConfig</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Carries all parsing/enrichment options, plus the caches they share.</span>
<span class="sd"> When passed as ``config=`` to parsedmarc&#39;s public parsing functions, the</span>
<span class="sd"> individual option keyword arguments (``offline``, ``ip_db_path``,</span>
<span class="sd"> ``nameservers``, etc.) are ignored in favor of the values carried on this</span>
<span class="sd"> object.</span>
<span class="sd"> Every explicitly constructed ``ParserConfig()`` owns fresh, isolated</span>
<span class="sd"> cache objects (``ip_address_cache``, ``seen_aggregate_report_ids``,</span>
<span class="sd"> ``reverse_dns_map``) via ``default_factory`` — two independently</span>
<span class="sd"> constructed configs never share cache state. To derive a variant of an</span>
<span class="sd"> existing config that *does* keep sharing its caches (e.g. to override</span>
<span class="sd"> ``offline`` for one call while still benefiting from warm caches), use</span>
<span class="sd"> ``dataclasses.replace(cfg, ...)``: since every field, including the three</span>
<span class="sd"> cache fields, is an init field, ``replace()`` copies the source&#39;s cache</span>
<span class="sd"> objects onto the new instance rather than constructing fresh ones.</span>
<span class="sd"> Pickling (as happens when a config is captured in a</span>
<span class="sd"> ``functools.partial`` payload submitted to a multiprocessing worker)</span>
<span class="sd"> intentionally drops cache *contents*: ``__getstate__`` omits the three</span>
<span class="sd"> cache fields, and ``__setstate__`` rebinds them to this module&#39;s</span>
<span class="sd"> :data:`IP_ADDRESS_CACHE`, :data:`SEEN_AGGREGATE_REPORT_IDS`, and</span>
<span class="sd"> :data:`REVERSE_DNS_MAP` — the unpickling process&#39;s module-default</span>
<span class="sd"> caches — rather than either the sender&#39;s cache contents (which would be</span>
<span class="sd"> expensive and stale to serialize per task) or brand new empty caches per</span>
<span class="sd"> unpickle (which would silently defeat per-worker caching, since a</span>
<span class="sd"> ``functools.partial`` payload is re-pickled for every task submitted to a</span>
<span class="sd"> worker). Binding the module defaults means a freshly spawned worker</span>
<span class="sd"> interpreter starts with empty caches and accumulates hits across the</span>
<span class="sd"> tasks it handles, matching pre-refactor behavior.</span>
<span class="sd"> Note that ``keep_alive`` and ``n_procs`` are deliberately **not** fields</span>
<span class="sd"> on this class: those control process/worker orchestration, not parsing</span>
<span class="sd"> or enrichment behavior.</span>
<span class="sd"> Attributes:</span>
<span class="sd"> offline: Do not make online requests (DNS, IP database download,</span>
<span class="sd"> reverse DNS map download, PSL overrides download).</span>
<span class="sd"> ip_db_path: Path to a local MMDB file from IPinfo, MaxMind, or DBIP.</span>
<span class="sd"> always_use_local_files: Always use local/bundled files instead of</span>
<span class="sd"> downloading the IP database, reverse DNS map, or PSL overrides.</span>
<span class="sd"> reverse_dns_map_path: Path to a local reverse DNS map file.</span>
<span class="sd"> reverse_dns_map_url: URL to a reverse DNS map file.</span>
<span class="sd"> psl_overrides_path: Path to a local PSL overrides file.</span>
<span class="sd"> psl_overrides_url: URL to a PSL overrides file.</span>
<span class="sd"> nameservers: A list of one or more nameservers to use for DNS</span>
<span class="sd"> queries (Cloudflare&#39;s public DNS resolvers by default).</span>
<span class="sd"> dns_timeout: DNS query timeout, in seconds.</span>
<span class="sd"> dns_retries: Number of times to retry a DNS query after a timeout or</span>
<span class="sd"> other transient error.</span>
<span class="sd"> strip_attachment_payloads: Remove attachment payloads from parsed</span>
<span class="sd"> email results.</span>
<span class="sd"> normalize_timespan_threshold_hours: Aggregate reports whose</span>
<span class="sd"> timespan exceeds this many hours are normalized/split.</span>
<span class="sd"> ip_address_cache: Cache of IP address enrichment results.</span>
<span class="sd"> seen_aggregate_report_ids: Cache of already-seen aggregate report</span>
<span class="sd"> IDs, used for de-duplication.</span>
<span class="sd"> reverse_dns_map: The reverse DNS map used to classify base domains</span>
<span class="sd"> found via reverse DNS.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">offline</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">ip_db_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">always_use_local_files</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">reverse_dns_map_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">reverse_dns_map_url</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">psl_overrides_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">psl_overrides_url</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">nameservers</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">dns_timeout</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="n">DEFAULT_DNS_TIMEOUT</span>
<span class="n">dns_retries</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">DEFAULT_DNS_MAX_RETRIES</span>
<span class="n">strip_attachment_payloads</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">normalize_timespan_threshold_hours</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">24.0</span>
<span class="n">ip_address_cache</span><span class="p">:</span> <span class="n">ExpiringDict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span>
<span class="n">default_factory</span><span class="o">=</span><span class="n">_new_ip_address_cache</span><span class="p">,</span> <span class="nb">repr</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="kc">False</span>
<span class="p">)</span>
<span class="n">seen_aggregate_report_ids</span><span class="p">:</span> <span class="n">ExpiringDict</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span>
<span class="n">default_factory</span><span class="o">=</span><span class="n">_new_seen_aggregate_report_ids</span><span class="p">,</span> <span class="nb">repr</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="kc">False</span>
<span class="p">)</span>
<span class="n">reverse_dns_map</span><span class="p">:</span> <span class="n">ReverseDNSMap</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span>
<span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">,</span> <span class="nb">repr</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">compare</span><span class="o">=</span><span class="kc">False</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">__getstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">object</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Return picklable state, excluding the three cache fields.</span>
<span class="sd"> Cache contents are process-local and potentially huge; they are</span>
<span class="sd"> never serialized. See the class docstring for the full rationale.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="p">{</span>
<span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="k">if</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">_CACHE_FIELD_NAMES</span>
<span class="p">}</span>
<span class="k">def</span><span class="w"> </span><span class="nf">__setstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">object</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Restore option fields, then rebind caches to this module&#39;s</span>
<span class="sd"> process-wide defaults (never fresh empty caches — see the class</span>
<span class="sd"> docstring for why that would silently break per-worker caching).</span>
<span class="sd"> Non-cache fields are first initialized to their class defaults, so</span>
<span class="sd"> unpickling a ``ParserConfig`` serialized by an older parsedmarc</span>
<span class="sd"> version (whose state predates fields added since) leaves the newer</span>
<span class="sd"> fields at their defaults instead of unset entirely — ``__init__``</span>
<span class="sd"> never runs during unpickling, so without this an absent field would</span>
<span class="sd"> raise ``AttributeError`` on first access.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">_CACHE_FIELD_NAMES</span><span class="p">:</span>
<span class="nb">object</span><span class="o">.</span><span class="fm">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">default</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">state</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="nb">object</span><span class="o">.</span><span class="fm">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nb">object</span><span class="o">.</span><span class="fm">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s2">&quot;ip_address_cache&quot;</span><span class="p">,</span> <span class="n">IP_ADDRESS_CACHE</span><span class="p">)</span>
<span class="nb">object</span><span class="o">.</span><span class="fm">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s2">&quot;seen_aggregate_report_ids&quot;</span><span class="p">,</span> <span class="n">SEEN_AGGREGATE_REPORT_IDS</span><span class="p">)</span>
<span class="nb">object</span><span class="o">.</span><span class="fm">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s2">&quot;reverse_dns_map&quot;</span><span class="p">,</span> <span class="n">REVERSE_DNS_MAP</span><span class="p">)</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2018 - 2025, Sean Whalen and contributors.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
_modules/parsedmarc/elastic.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.elastic &mdash; parsedmarc 10.3.0 documentation</title>
<title>parsedmarc.elastic &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=a126560f"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/opensearch.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.opensearch &mdash; parsedmarc 10.3.0 documentation</title>
<title>parsedmarc.opensearch &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=a126560f"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/splunk.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.splunk &mdash; parsedmarc 10.3.0 documentation</title>
<title>parsedmarc.splunk &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=a126560f"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/types.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.types &mdash; parsedmarc 10.3.0 documentation</title>
<title>parsedmarc.types &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=a126560f"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/utils.html
+17 -3
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.utils &mdash; parsedmarc 10.3.0 documentation</title>
<title>parsedmarc.utils &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=a126560f"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
@@ -983,6 +983,8 @@
<span class="n">url</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">offline</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">reverse_dns_map</span><span class="p">:</span> <span class="n">ReverseDNSMap</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">psl_overrides_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">psl_overrides_url</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">ReverseDNSService</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Returns the service name of a given base domain name from reverse DNS.</span>
@@ -991,9 +993,11 @@
<span class="sd"> base_domain (str): The base domain of the reverse DNS lookup</span>
<span class="sd"> always_use_local_file (bool): Always use a local map file</span>
<span class="sd"> local_file_path (str): Path to a local map file</span>
<span class="sd"> url (str): URL ro a reverse DNS map</span>
<span class="sd"> url (str): URL to a reverse DNS map</span>
<span class="sd"> offline (bool): Use the built-in copy of the reverse DNS map</span>
<span class="sd"> reverse_dns_map (dict): A reverse DNS map</span>
<span class="sd"> psl_overrides_path (str): Path to a local PSL overrides file</span>
<span class="sd"> psl_overrides_url (str): URL to a PSL overrides file</span>
<span class="sd"> Returns:</span>
<span class="sd"> dict: A dictionary containing name and type.</span>
<span class="sd"> If the service is unknown, the name will be</span>
@@ -1014,6 +1018,8 @@
<span class="n">local_file_path</span><span class="o">=</span><span class="n">local_file_path</span><span class="p">,</span>
<span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span>
<span class="n">offline</span><span class="o">=</span><span class="n">offline</span><span class="p">,</span>
<span class="n">psl_overrides_path</span><span class="o">=</span><span class="n">psl_overrides_path</span><span class="p">,</span>
<span class="n">psl_overrides_url</span><span class="o">=</span><span class="n">psl_overrides_url</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">service</span><span class="p">:</span> <span class="n">ReverseDNSService</span>
@@ -1041,6 +1047,8 @@
<span class="n">nameservers</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">timeout</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="n">DEFAULT_DNS_TIMEOUT</span><span class="p">,</span>
<span class="n">retries</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">DEFAULT_DNS_MAX_RETRIES</span><span class="p">,</span>
<span class="n">psl_overrides_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">psl_overrides_url</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">IPAddressInfo</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Returns reverse DNS and country information for the given IP address</span>
@@ -1059,6 +1067,8 @@
<span class="sd"> timeout (float): Sets the DNS timeout in seconds</span>
<span class="sd"> retries (int): Number of times to retry on timeout or other transient</span>
<span class="sd"> errors</span>
<span class="sd"> psl_overrides_path (str): Path to a local PSL overrides file</span>
<span class="sd"> psl_overrides_url (str): URL to a PSL overrides file</span>
<span class="sd"> Returns:</span>
<span class="sd"> dict: ``ip_address``, ``reverse_dns``, ``country``</span>
@@ -1111,6 +1121,8 @@
<span class="n">url</span><span class="o">=</span><span class="n">reverse_dns_map_url</span><span class="p">,</span>
<span class="n">always_use_local_file</span><span class="o">=</span><span class="n">always_use_local_files</span><span class="p">,</span>
<span class="n">reverse_dns_map</span><span class="o">=</span><span class="n">reverse_dns_map</span><span class="p">,</span>
<span class="n">psl_overrides_path</span><span class="o">=</span><span class="n">psl_overrides_path</span><span class="p">,</span>
<span class="n">psl_overrides_url</span><span class="o">=</span><span class="n">psl_overrides_url</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">info</span><span class="p">[</span><span class="s2">&quot;base_domain&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">base_domain</span>
<span class="n">info</span><span class="p">[</span><span class="s2">&quot;type&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">service</span><span class="p">[</span><span class="s2">&quot;type&quot;</span><span class="p">]</span>
@@ -1130,6 +1142,8 @@
<span class="n">local_file_path</span><span class="o">=</span><span class="n">reverse_dns_map_path</span><span class="p">,</span>
<span class="n">url</span><span class="o">=</span><span class="n">reverse_dns_map_url</span><span class="p">,</span>
<span class="n">offline</span><span class="o">=</span><span class="n">offline</span><span class="p">,</span>
<span class="n">psl_overrides_path</span><span class="o">=</span><span class="n">psl_overrides_path</span><span class="p">,</span>
<span class="n">psl_overrides_url</span><span class="o">=</span><span class="n">psl_overrides_url</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">info</span><span class="p">[</span><span class="s2">&quot;as_domain&quot;</span><span class="p">]</span> <span class="ow">and</span> <span class="n">info</span><span class="p">[</span><span class="s2">&quot;as_domain&quot;</span><span class="p">]</span> <span class="ow">in</span> <span class="n">map_value</span><span class="p">:</span>
<span class="n">service</span> <span class="o">=</span> <span class="n">map_value</span><span class="p">[</span><span class="n">info</span><span class="p">[</span><span class="s2">&quot;as_domain&quot;</span><span class="p">]]</span>
_sources/api.md.txt
+7
View File
@@ -7,6 +7,13 @@
:members:
```
## parsedmarc.config
```{eval-rst}
.. automodule:: parsedmarc.config
:members:
```
## parsedmarc.elastic
```{eval-rst}
_sources/kibana.md.txt
+1 -1
View File
@@ -38,7 +38,7 @@ valid when a message is forwarded without changing the from address, which is
often caused by a mailbox forwarding rule. This is because DKIM signatures are
part of the message headers, whereas SPF relies on SMTP session headers.
Underneath the pie charts. you can see graphs of DMARC passage and message
Underneath the pie charts, you can see graphs of DMARC compliance and message
disposition over time.
Under the graphs you will find the most useful data tables on the dashboard. On
_sources/splunk.md.txt
+1 -1
View File
@@ -18,5 +18,5 @@ The Splunk dashboards display the same content and layout as the
Kibana dashboards, although the Kibana dashboards have slightly
easier and more flexible filtering options.
[xml files]: https://github.com/domainaware/parsedmarc/tree/master/splunk
[xml files]: https://github.com/domainaware/parsedmarc/tree/master/dashboards/splunk
[http event collector (hec)]: http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC
_sources/usage.md.txt
+192 -9
View File
@@ -132,6 +132,24 @@ The full set of configuration options are:
payloads from results
- `silent` - bool: Set this to `False` to output results to STDOUT
- `output` - str: Directory to place JSON and CSV files in. This is required if you set either of the JSON output file options.
- `archive_directory` - str: Optional. When set, successfully
processed report files given as local file/directory path
arguments are moved into
`<archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/`
(year and month come from the report's own begin/arrival date, with
the month zero-padded). A successfully parsed report whose archive
date can't be determined is left in place with a logged warning.
Files that fail to parse as a report are moved to
`<archive_directory>/Invalid/`; files that fail for other reasons,
such as transient I/O errors, are left in place so a later run can
retry them. An existing destination file is never overwritten; a
numeric suffix is appended before the extension (e.g.
`report-1.xml`). This applies only to direct local file input —
reports fetched from mailboxes (IMAP, Microsoft Graph, Gmail API,
Maildir) use `[mailbox] archive_folder` instead, and mbox files are
never moved. Files already inside `archive_directory` are excluded
from processing, so the archive may safely live inside an input
directory. A failed move is logged and does not stop the run.
- `aggregate_json_filename` - str: filename for the aggregate
JSON output file
- `failure_json_filename` - str: filename for the failure
@@ -168,21 +186,32 @@ The full set of configuration options are:
- `fail_on_output_error` - bool: Exit with a non-zero status code if
any configured output destination fails while saving/publishing
reports (Default: `False`)
:::{note}
This option only controls the process exit code. Retaining mailbox
messages whose reports could not be saved is automatic and happens
either way — see
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved).
:::
- `log_file` - str: Write log messages to a file at this path
- `n_procs` - int: Number of processes to run in parallel when
parsing report files passed directly as CLI arguments
parsing report files passed directly as CLI arguments, messages
in mbox files, and messages from mailbox connections (IMAP,
Microsoft Graph, Gmail API, Maildir), including watch mode
(Default: `1`)
:::{note}
Setting this to a number larger than one can improve
performance when processing thousands of files
performance when processing thousands of files or messages
:::
:::{note}
`n_procs` only applies to report files passed directly on the
command line. Messages from mbox files and from mailbox
connections (IMAP, Microsoft Graph, Gmail API, Maildir) are
always processed sequentially.
Only parsing is parallelized across worker processes. Fetching
messages, deduplicating reports, archiving/deleting mailbox
messages, and saving/publishing to outputs all stay sequential
in the main process. Each worker process keeps its own DNS/GeoIP
cache.
:::
- `mailbox`
@@ -195,12 +224,46 @@ The full set of configuration options are:
messages as they arrive or poll MS Graph for new messages
- `delete` - bool: Delete messages after processing them,
instead of archiving them
- `delete_aggregate` - bool: Delete aggregate report messages
after processing them, instead of archiving them
(Default: the value of `delete`)
- `delete_failure` - bool: Delete failure report messages
after processing them, instead of archiving them
(Default: the value of `delete`)
- `delete_smtp_tls` - bool: Delete SMTP TLS report messages
after processing them, instead of archiving them
(Default: the value of `delete`)
- `delete_invalid` - bool: Delete messages that could not be
parsed, instead of archiving them in the `Invalid`
subfolder, where they can be inspected for debugging
(Default: the value of `delete`)
:::{note}
Each of these four options overrides `delete` for one kind of
message only, and the other three keep inheriting `delete`. So
`delete = True` combined with `delete_failure = False` archives
failure report messages while deleting processed aggregate and
SMTP TLS report messages — and unparseable ones, unless
`delete_invalid = False` is set as well.
:::
- `test` - bool: Do not move or delete messages
- `batch_size` - int: Number of messages to read and process
before saving. Default `10`. Use `0` for no limit.
- `check_timeout` - int: Number of seconds to wait for a IMAP
- `check_timeout` - int: Number of seconds to wait for an IMAP
IDLE response or the number of seconds until the next
mail check (Default: `30`)
- `max_unsaved_retries` - int: How many times a batch of messages
whose reports could not be saved is retried before its messages
are moved to the `Unsaved` archive subfolder instead of being
retried again (Default: `2`, i.e. the initial attempt plus two
retries). Use `0` to move messages on the first failed save;
negative values are rejected.
Failures are counted in memory, so the cap applies across watch-mode
checks within one long-running process, not across separate one-shot
runs. See
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
below.
- `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
Defaults to `1d` if incorrect value is provided.
@@ -806,6 +869,78 @@ PUT _cluster/settings
Increasing this value increases resource usage.
:::
### Mailbox messages are only archived once the reports are saved
parsedmarc processes a mailbox in batches of `batch_size` messages. Each
batch is written to every configured output destination *before* any of
that batch's messages are archived or deleted. If any destination reports a
failure — an Elasticsearch outage, an expired Splunk HEC token, an
unreachable Kafka broker, a full `--output` disk — the whole batch is left
in the reports folder and retried on the next run or watch-mode check, so a
report is never removed from the mailbox while it exists nowhere else
([issue #242](https://github.com/domainaware/parsedmarc/issues/242)).
This is all-or-nothing per batch: archiving a batch because most
destinations accepted it would still permanently lose the data for the one
that didn't. It also applies regardless of `fail_on_output_error`, which
only controls the process exit code. Messages that could not be parsed at
all carry no report data, so they are filed under `Invalid` (or deleted per
`delete_invalid`) as usual.
A destination that is broken rather than briefly unavailable would
otherwise be retried forever, so retries are capped. Once a message's batch
has failed `max_unsaved_retries + 1` times (three times by default), that
message is moved to `<archive_folder>/Unsaved` and stops being retried. **A
message is never deleted on this path, whatever the `delete` options say.**
To recover after fixing the output destination, either move the messages
from `Archive/Unsaved` back into the reports folder, or run parsedmarc once
with `reports_folder = Archive/Unsaved`.
:::{note}
The failure counts live in memory, so they are counted per parsedmarc
process. In watch mode — a long-running process that checks the mailbox
repeatedly — the cap works as described across checks. A one-shot run
(`cron`, `systemd` timers) attempts each message exactly once and then
exits, so its counts start over every time and the default cap is never
reached: messages simply keep being retried on every run, which is the
safe direction. Set `max_unsaved_retries = 0` if you want one-shot runs to
move unsavable messages to `Unsaved` immediately instead.
:::
:::{warning}
Retrying a batch means re-sending it. Output destinations that
deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize
an already-saved report — are unaffected, and S3 is idempotent because
each report is written to an object key built from its type, date, and
report ID, so a retry overwrites the same object. Kafka,
Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the
`--output` JSON/CSV files all append unconditionally, so each retry adds
another copy of every report in the batch. That is why the default retry
cap is deliberately low: at most three deliveries per report before its
message is set aside in `Unsaved`. The summary email covers everything
parsed in a run, including reports whose batch failed to save, so a report
retried across runs can also appear in more than one summary email.
:::
:::{note}
Not every destination can report a failed delivery. The webhook output
deliberately logs and swallows its own HTTP and network errors, and the
syslog and GELF outputs send through Python logging handlers, which
swallow delivery errors by design — so an unreachable webhook, syslog, or
GELF endpoint is *not* treated as a failed save and does not hold a
batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
Kafka, S3, PostgreSQL, Azure Log Analytics, and the `--output` files are
all detected and do.
:::
:::{note}
`since` interacts with retries: a message that ages out of the configured
`since` window stops being fetched, and therefore stops being retried
automatically. It is never deleted or moved — it simply stays in the
reports folder until it is processed by a run with a wider (or no)
`since` window.
:::
## Environment variable configuration
Any configuration option can be set via environment variables using the
@@ -941,6 +1076,50 @@ For sections with underscores in the name, the full section name is used:
| `gelf` | `PARSEDMARC_GELF_` |
| `webhook` | `PARSEDMARC_WEBHOOK_` |
## Using parsedmarc as a library
`parsedmarc` is also importable as a regular Python package, not just a CLI
tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`,
`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`,
`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and
`watch_inbox()` — are all importable
directly from the `parsedmarc` package. See the [API reference](api.md) for
the full set of modules and members.
Each of these functions accepts either individual option keyword arguments
(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword
argument carrying a `ParserConfig` instance:
```python
from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
from parsedmarc.mail import IMAPConnection
config = ParserConfig(
offline=False,
nameservers=["1.1.1.1", "1.0.0.1"],
dns_timeout=5.0,
)
report = parse_report_file("aggregate_report.xml.gz", config=config)
connection = IMAPConnection(
host="imap.example.com", user="dmarc@example.com", password="..."
)
results = get_dmarc_reports_from_mailbox(connection, config=config)
```
A few things to keep in mind:
- When `config=` is passed, the individual option keyword arguments are
ignored in favor of the values carried on the `ParserConfig` instance.
- Each explicitly constructed `ParserConfig` owns its own isolated caches
(IP address info, seen aggregate report IDs, and the reverse DNS map).
Omitting `config=` falls back to the process-wide caches shared by every
call that doesn't pass one.
- `keep_alive` and `n_procs` are not part of `ParserConfig` — they control
process/worker orchestration rather than parsing or enrichment behavior,
so they are always passed as separate keyword arguments.
## Performance tuning
For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
@@ -951,8 +1130,12 @@ imports more predictable:
- Reduce `mailbox.batch_size` to smaller values such as `100-500` instead of
processing a very large message set at once. Smaller batches trade throughput
for lower peak memory use and less sink pressure.
- Keep `n_procs` low for mailbox-heavy runs. In practice, `1-2` workers is often
a safer starting point for large backfills than aggressive parallelism.
- `n_procs` now parallelizes parsing for mailbox and mbox runs too, not just
report files passed directly as CLI arguments. It pays off most when a run
is bound by DNS/GeoIP enrichment rather than fetching or output. The
trade-off is memory and DNS load: at most roughly `2 * n_procs` messages
are held in flight at once, each worker process keeps its own DNS/GeoIP
cache, and DNS query volume can multiply by up to `n_procs`.
- Use `mailbox.since` to process reports in smaller time windows such as `1d`,
`7d`, or another interval that fits the backlog. This makes it easier to catch
up incrementally instead of loading an entire mailbox history in one run.
_static/documentation_options.js
+1 -1
View File
@@ -1,5 +1,5 @@
const DOCUMENTATION_OPTIONS = {
VERSION: '10.3.0',
VERSION: '10.4.0',
LANGUAGE: 'en',
COLLAPSE_INDEX: false,
BUILDER: 'html',
api.html
+436 -21
View File
File diff suppressed because one or more lines are too long
contributing.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Contributing to parsedmarc &mdash; parsedmarc 10.3.0 documentation</title>
<title>Contributing to parsedmarc &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
davmail.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 10.3.0 documentation</title>
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
dmarc.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Understanding DMARC &mdash; parsedmarc 10.3.0 documentation</title>
<title>Understanding DMARC &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
elasticsearch.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Elasticsearch and Kibana &mdash; parsedmarc 10.3.0 documentation</title>
<title>Elasticsearch and Kibana &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
genindex.html
+61 -7
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Index &mdash; parsedmarc 10.3.0 documentation</title>
<title>Index &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -91,6 +91,7 @@
| <a href="#I"><strong>I</strong></a>
| <a href="#L"><strong>L</strong></a>
| <a href="#M"><strong>M</strong></a>
| <a href="#N"><strong>N</strong></a>
| <a href="#O"><strong>O</strong></a>
| <a href="#P"><strong>P</strong></a>
| <a href="#Q"><strong>Q</strong></a>
@@ -130,6 +131,8 @@
<li><a href="api.html#parsedmarc.types.AggregateReportMetadata">AggregateReportMetadata (class in parsedmarc.types)</a>
</li>
<li><a href="api.html#parsedmarc.elastic.AlreadySaved">AlreadySaved</a>, <a href="api.html#parsedmarc.opensearch.AlreadySaved">[1]</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.always_use_local_files">always_use_local_files (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.append_json">append_json() (in module parsedmarc)</a>
</li>
@@ -160,9 +163,13 @@
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.utils.decode_base64">decode_base64() (in module parsedmarc.utils)</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.dns_retries">dns_retries (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.config.ParserConfig.dns_timeout">dns_timeout (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.utils.DownloadError">DownloadError</a>
</li>
</ul></td>
@@ -262,10 +269,14 @@
<li><a href="api.html#parsedmarc.InvalidForensicReport">InvalidForensicReport (in module parsedmarc)</a>
</li>
<li><a href="api.html#parsedmarc.utils.InvalidIPinfoAPIKey">InvalidIPinfoAPIKey</a>
</li>
<li><a href="api.html#parsedmarc.InvalidSMTPTLSReport">InvalidSMTPTLSReport</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.InvalidSMTPTLSReport">InvalidSMTPTLSReport</a>
<li><a href="api.html#parsedmarc.config.ParserConfig.ip_address_cache">ip_address_cache (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.ip_db_path">ip_db_path (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.utils.IPAddressInfo">IPAddressInfo (class in parsedmarc.utils)</a>
</li>
@@ -306,6 +317,8 @@
<ul>
<li><a href="api.html#module-parsedmarc">parsedmarc</a>
</li>
<li><a href="api.html#module-parsedmarc.config">parsedmarc.config</a>
</li>
<li><a href="api.html#module-parsedmarc.elastic">parsedmarc.elastic</a>
</li>
@@ -321,8 +334,24 @@
</ul></td>
</tr></table>
<h2 id="N">N</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.config.ParserConfig.nameservers">nameservers (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.config.ParserConfig.normalize_timespan_threshold_hours">normalize_timespan_threshold_hours (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
</tr></table>
<h2 id="O">O</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.config.ParserConfig.offline">offline (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.opensearch.OpenSearchError">OpenSearchError</a>
</li>
@@ -362,8 +391,6 @@
</li>
<li><a href="api.html#parsedmarc.parsed_smtp_tls_reports_to_csv">parsed_smtp_tls_reports_to_csv() (in module parsedmarc)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.parsed_smtp_tls_reports_to_csv_rows">parsed_smtp_tls_reports_to_csv_rows() (in module parsedmarc)</a>
</li>
<li><a href="api.html#parsedmarc.types.ParsedEmail">ParsedEmail (class in parsedmarc.types)</a>
@@ -373,6 +400,15 @@
<ul>
<li><a href="api.html#module-parsedmarc">module</a>
</li>
</ul></li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li>
parsedmarc.config
<ul>
<li><a href="api.html#module-parsedmarc.config">module</a>
</li>
</ul></li>
<li>
@@ -410,9 +446,15 @@
<li><a href="api.html#module-parsedmarc.utils">module</a>
</li>
</ul></li>
<li><a href="api.html#parsedmarc.config.ParserConfig">ParserConfig (class in parsedmarc.config)</a>
</li>
<li><a href="api.html#parsedmarc.ParserError">ParserError</a>
</li>
<li><a href="api.html#parsedmarc.types.ParsingResults">ParsingResults (class in parsedmarc.types)</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.psl_overrides_path">psl_overrides_path (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.psl_overrides_url">psl_overrides_url (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
</tr></table>
@@ -428,6 +470,14 @@
<h2 id="R">R</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.config.ParserConfig.reverse_dns_map">reverse_dns_map (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.reverse_dns_map_path">reverse_dns_map_path (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.config.ParserConfig.reverse_dns_map_url">reverse_dns_map_url (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.utils.ReverseDNSService">ReverseDNSService (class in parsedmarc.utils)</a>
</li>
</ul></td>
@@ -458,11 +508,13 @@
</li>
<li><a href="api.html#parsedmarc.elastic.save_smtp_tls_report_to_elasticsearch">save_smtp_tls_report_to_elasticsearch() (in module parsedmarc.elastic)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.opensearch.save_smtp_tls_report_to_opensearch">save_smtp_tls_report_to_opensearch() (in module parsedmarc.opensearch)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#parsedmarc.splunk.HECClient.save_smtp_tls_reports_to_splunk">save_smtp_tls_reports_to_splunk() (parsedmarc.splunk.HECClient method)</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.seen_aggregate_report_ids">seen_aggregate_report_ids (parsedmarc.config.ParserConfig attribute)</a>
</li>
<li><a href="api.html#parsedmarc.elastic.set_hosts">set_hosts() (in module parsedmarc.elastic)</a>
@@ -483,6 +535,8 @@
<li><a href="api.html#parsedmarc.types.SMTPTLSReport">SMTPTLSReport (class in parsedmarc.types)</a>
</li>
<li><a href="api.html#parsedmarc.splunk.SplunkError">SplunkError</a>
</li>
<li><a href="api.html#parsedmarc.config.ParserConfig.strip_attachment_payloads">strip_attachment_payloads (parsedmarc.config.ParserConfig attribute)</a>
</li>
</ul></td>
</tr></table>
index.html
+4 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 10.3.0 documentation</title>
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -187,6 +187,7 @@ for RHEL or Debian.</p>
<li class="toctree-l2"><a class="reference internal" href="usage.html#cli-help">CLI help</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#configuration-file">Configuration file</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#environment-variable-configuration">Environment variable configuration</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#using-parsedmarc-as-a-library">Using parsedmarc as a library</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#performance-tuning">Performance tuning</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#multi-tenant-support">Multi-tenant support</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#running-parsedmarc-as-a-systemd-service">Running parsedmarc as a systemd service</a></li>
@@ -234,6 +235,7 @@ for RHEL or Debian.</p>
</li>
<li class="toctree-l1"><a class="reference internal" href="api.html">API reference</a><ul>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-parsedmarc">parsedmarc</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-parsedmarc.config">parsedmarc.config</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-parsedmarc.elastic">parsedmarc.elastic</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-parsedmarc.opensearch">parsedmarc.opensearch</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-parsedmarc.splunk">parsedmarc.splunk</a></li>
installation.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Installation &mdash; parsedmarc 10.3.0 documentation</title>
<title>Installation &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
kibana.html
+3 -3
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using the Kibana dashboards &mdash; parsedmarc 10.3.0 documentation</title>
<title>Using the Kibana dashboards &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -120,7 +120,7 @@ passes if a message passes SPF or DKIM alignment, only DKIM alignment remains
valid when a message is forwarded without changing the from address, which is
often caused by a mailbox forwarding rule. This is because DKIM signatures are
part of the message headers, whereas SPF relies on SMTP session headers.</p>
<p>Underneath the pie charts. you can see graphs of DMARC passage and message
<p>Underneath the pie charts, you can see graphs of DMARC compliance and message
disposition over time.</p>
<p>Under the graphs you will find the most useful data tables on the dashboard. On
the left, there is a list of organizations that are sending you DMARC reports.
mailing-lists.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>What about mailing lists? &mdash; parsedmarc 10.3.0 documentation</title>
<title>What about mailing lists? &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
objects.inv
BIN
View File
Binary file not shown.
opensearch.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>OpenSearch and Grafana &mdash; parsedmarc 10.3.0 documentation</title>
<title>OpenSearch and Grafana &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
output.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Sample outputs &mdash; parsedmarc 10.3.0 documentation</title>
<title>Sample outputs &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
py-modindex.html
+7 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Python Module Index &mdash; parsedmarc 10.3.0 documentation</title>
<title>Python Module Index &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -97,6 +97,11 @@
<td>
<a href="api.html#module-parsedmarc"><code class="xref">parsedmarc</code></a></td><td>
<em></em></td></tr>
<tr class="cg-1">
<td></td>
<td>&#160;&#160;&#160;
<a href="api.html#module-parsedmarc.config"><code class="xref">parsedmarc.config</code></a></td><td>
<em></em></td></tr>
<tr class="cg-1">
<td></td>
<td>&#160;&#160;&#160;
search.html
+2 -2
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Search &mdash; parsedmarc 10.3.0 documentation</title>
<title>Search &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
@@ -13,7 +13,7 @@
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
splunk.html
+3 -3
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Splunk &mdash; parsedmarc 10.3.0 documentation</title>
<title>Splunk &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -85,7 +85,7 @@
<h1>Splunk<a class="headerlink" href="#splunk" title="Link to this heading"></a></h1>
<p>Starting in version 4.3.0 <code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> supports sending aggregate and/or
failure DMARC data to a Splunk <a class="reference external" href="http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC">HTTP Event collector (HEC)</a>.</p>
<p>The project repository contains <a class="reference external" href="https://github.com/domainaware/parsedmarc/tree/master/splunk">XML files</a> for premade Splunk
<p>The project repository contains <a class="reference external" href="https://github.com/domainaware/parsedmarc/tree/master/dashboards/splunk">XML files</a> for premade Splunk
dashboards for aggregate and failure DMARC reports.</p>
<p>Copy and paste the contents of each file into a separate Splunk
dashboard XML editor.</p>
usage.html
+197 -14
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using parsedmarc &mdash; parsedmarc 10.3.0 documentation</title>
<title>Using parsedmarc &mdash; parsedmarc 10.4.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=a126560f"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -47,7 +47,10 @@
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Using parsedmarc</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#cli-help">CLI help</a></li>
<li class="toctree-l2"><a class="reference internal" href="#configuration-file">Configuration file</a></li>
<li class="toctree-l2"><a class="reference internal" href="#configuration-file">Configuration file</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#mailbox-messages-are-only-archived-once-the-reports-are-saved">Mailbox messages are only archived once the reports are saved</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#environment-variable-configuration">Environment variable configuration</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#examples">Examples</a></li>
<li class="toctree-l3"><a class="reference internal" href="#specifying-the-config-file-via-environment-variable">Specifying the config file via environment variable</a></li>
@@ -57,6 +60,7 @@
<li class="toctree-l3"><a class="reference internal" href="#section-name-mapping">Section name mapping</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#using-parsedmarc-as-a-library">Using parsedmarc as a library</a></li>
<li class="toctree-l2"><a class="reference internal" href="#performance-tuning">Performance tuning</a></li>
<li class="toctree-l2"><a class="reference internal" href="#multi-tenant-support">Multi-tenant support</a></li>
<li class="toctree-l2"><a class="reference internal" href="#running-parsedmarc-as-a-systemd-service">Running parsedmarc as a systemd service</a><ul>
@@ -231,6 +235,24 @@ Elasticsearch, Splunk and/or S3</p></li>
payloads from results</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">silent</span></code> - bool: Set this to <code class="docutils literal notranslate"><span class="pre">False</span></code> to output results to STDOUT</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">output</span></code> - str: Directory to place JSON and CSV files in. This is required if you set either of the JSON output file options.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">archive_directory</span></code> - str: Optional. When set, successfully
processed report files given as local file/directory path
arguments are moved into
<code class="docutils literal notranslate"><span class="pre">&lt;archive_directory&gt;/&lt;year&gt;/&lt;month&gt;/&lt;Aggregate|Failure|SMTP-TLS&gt;/</span></code>
(year and month come from the report’s own begin/arrival date, with
the month zero-padded). A successfully parsed report whose archive
date can’t be determined is left in place with a logged warning.
Files that fail to parse as a report are moved to
<code class="docutils literal notranslate"><span class="pre">&lt;archive_directory&gt;/Invalid/</span></code>; files that fail for other reasons,
such as transient I/O errors, are left in place so a later run can
retry them. An existing destination file is never overwritten; a
numeric suffix is appended before the extension (e.g.
<code class="docutils literal notranslate"><span class="pre">report-1.xml</span></code>). This applies only to direct local file input —
reports fetched from mailboxes (IMAP, Microsoft Graph, Gmail API,
Maildir) use <code class="docutils literal notranslate"><span class="pre">[mailbox]</span> <span class="pre">archive_folder</span></code> instead, and mbox files are
never moved. Files already inside <code class="docutils literal notranslate"><span class="pre">archive_directory</span></code> are excluded
from processing, so the archive may safely live inside an input
directory. A failed move is logged and does not stop the run.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">aggregate_json_filename</span></code> - str: filename for the aggregate
JSON output file</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">failure_json_filename</span></code> - str: filename for the failure
@@ -266,28 +288,39 @@ DNS resolvers (Default: <code class="docutils literal notranslate"><span class="
<li><p><code class="docutils literal notranslate"><span class="pre">silent</span></code> - bool: Only print errors (Default: <code class="docutils literal notranslate"><span class="pre">True</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fail_on_output_error</span></code> - bool: Exit with a non-zero status code if
any configured output destination fails while saving/publishing
reports (Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>)</p></li>
reports (Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>)</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This option only controls the process exit code. Retaining mailbox
messages whose reports could not be saved is automatic and happens
either way — see
<a class="reference internal" href="#mailbox-messages-are-only-archived-once-the-reports-are-saved">Mailbox messages are only archived once the reports are saved</a>.</p>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">log_file</span></code> - str: Write log messages to a file at this path</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">n_procs</span></code> - int: Number of processes to run in parallel when
parsing report files passed directly as CLI arguments
parsing report files passed directly as CLI arguments, messages
in mbox files, and messages from mailbox connections (IMAP,
Microsoft Graph, Gmail API, Maildir), including watch mode
(Default: <code class="docutils literal notranslate"><span class="pre">1</span></code>)</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Setting this to a number larger than one can improve
performance when processing thousands of files</p>
performance when processing thousands of files or messages</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><code class="docutils literal notranslate"><span class="pre">n_procs</span></code> only applies to report files passed directly on the
command line. Messages from mbox files and from mailbox
connections (IMAP, Microsoft Graph, Gmail API, Maildir) are
always processed sequentially.</p>
<p>Only parsing is parallelized across worker processes. Fetching
messages, deduplicating reports, archiving/deleting mailbox
messages, and saving/publishing to outputs all stay sequential
in the main process. Each worker process keeps its own DNS/GeoIP
cache.</p>
</div>
</li>
</ul>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">mailbox</span></code></p>
<ul class="simple">
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">reports_folder</span></code> - str: The mailbox folder (or label for
Gmail) where the incoming reports can be found
(Default: <code class="docutils literal notranslate"><span class="pre">INBOX</span></code>)</p></li>
@@ -297,12 +330,46 @@ Gmail) to sort processed emails into (Default: <code class="docutils literal not
messages as they arrive or poll MS Graph for new messages</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">delete</span></code> - bool: Delete messages after processing them,
instead of archiving them</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">delete_aggregate</span></code> - bool: Delete aggregate report messages
after processing them, instead of archiving them
(Default: the value of <code class="docutils literal notranslate"><span class="pre">delete</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">delete_failure</span></code> - bool: Delete failure report messages
after processing them, instead of archiving them
(Default: the value of <code class="docutils literal notranslate"><span class="pre">delete</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">delete_smtp_tls</span></code> - bool: Delete SMTP TLS report messages
after processing them, instead of archiving them
(Default: the value of <code class="docutils literal notranslate"><span class="pre">delete</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">delete_invalid</span></code> - bool: Delete messages that could not be
parsed, instead of archiving them in the <code class="docutils literal notranslate"><span class="pre">Invalid</span></code>
subfolder, where they can be inspected for debugging
(Default: the value of <code class="docutils literal notranslate"><span class="pre">delete</span></code>)</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Each of these four options overrides <code class="docutils literal notranslate"><span class="pre">delete</span></code> for one kind of
message only, and the other three keep inheriting <code class="docutils literal notranslate"><span class="pre">delete</span></code>. So
<code class="docutils literal notranslate"><span class="pre">delete</span> <span class="pre">=</span> <span class="pre">True</span></code> combined with <code class="docutils literal notranslate"><span class="pre">delete_failure</span> <span class="pre">=</span> <span class="pre">False</span></code> archives
failure report messages while deleting processed aggregate and
SMTP TLS report messages — and unparseable ones, unless
<code class="docutils literal notranslate"><span class="pre">delete_invalid</span> <span class="pre">=</span> <span class="pre">False</span></code> is set as well.</p>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">test</span></code> - bool: Do not move or delete messages</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">batch_size</span></code> - int: Number of messages to read and process
before saving. Default <code class="docutils literal notranslate"><span class="pre">10</span></code>. Use <code class="docutils literal notranslate"><span class="pre">0</span></code> for no limit.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">check_timeout</span></code> - int: Number of seconds to wait for a IMAP
<li><p><code class="docutils literal notranslate"><span class="pre">check_timeout</span></code> - int: Number of seconds to wait for an IMAP
IDLE response or the number of seconds until the next
mail check (Default: <code class="docutils literal notranslate"><span class="pre">30</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">max_unsaved_retries</span></code> - int: How many times a batch of messages
whose reports could not be saved is retried before its messages
are moved to the <code class="docutils literal notranslate"><span class="pre">Unsaved</span></code> archive subfolder instead of being
retried again (Default: <code class="docutils literal notranslate"><span class="pre">2</span></code>, i.e. the initial attempt plus two
retries). Use <code class="docutils literal notranslate"><span class="pre">0</span></code> to move messages on the first failed save;
negative values are rejected.
Failures are counted in memory, so the cap applies across watch-mode
checks within one long-running process, not across separate one-shot
runs. See
<a class="reference internal" href="#mailbox-messages-are-only-archived-once-the-reports-are-saved">Mailbox messages are only archived once the reports are saved</a>
below.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">since</span></code> - str: Search for messages since certain time. (Examples: <code class="docutils literal notranslate"><span class="pre">5m|3h|2d|1w</span></code>)
Acceptable units - {“m”:“minutes”, “h”:“hours”, “d”:“days”, “w”:“weeks”}.
Defaults to <code class="docutils literal notranslate"><span class="pre">1d</span></code> if incorrect value is provided.</p></li>
@@ -986,6 +1053,76 @@ Check current usage (from Management -&gt; Dev Tools -&gt; Console):</p>
</div>
<p>Increasing this value increases resource usage.</p>
</div>
<section id="mailbox-messages-are-only-archived-once-the-reports-are-saved">
<h3>Mailbox messages are only archived once the reports are saved<a class="headerlink" href="#mailbox-messages-are-only-archived-once-the-reports-are-saved" title="Link to this heading"></a></h3>
<p>parsedmarc processes a mailbox in batches of <code class="docutils literal notranslate"><span class="pre">batch_size</span></code> messages. Each
batch is written to every configured output destination <em>before</em> any of
that batch’s messages are archived or deleted. If any destination reports a
failure — an Elasticsearch outage, an expired Splunk HEC token, an
unreachable Kafka broker, a full <code class="docutils literal notranslate"><span class="pre">--output</span></code> disk — the whole batch is left
in the reports folder and retried on the next run or watch-mode check, so a
report is never removed from the mailbox while it exists nowhere else
(<a class="reference external" href="https://github.com/domainaware/parsedmarc/issues/242">issue #242</a>).</p>
<p>This is all-or-nothing per batch: archiving a batch because most
destinations accepted it would still permanently lose the data for the one
that didn’t. It also applies regardless of <code class="docutils literal notranslate"><span class="pre">fail_on_output_error</span></code>, which
only controls the process exit code. Messages that could not be parsed at
all carry no report data, so they are filed under <code class="docutils literal notranslate"><span class="pre">Invalid</span></code> (or deleted per
<code class="docutils literal notranslate"><span class="pre">delete_invalid</span></code>) as usual.</p>
<p>A destination that is broken rather than briefly unavailable would
otherwise be retried forever, so retries are capped. Once a message’s batch
has failed <code class="docutils literal notranslate"><span class="pre">max_unsaved_retries</span> <span class="pre">+</span> <span class="pre">1</span></code> times (three times by default), that
message is moved to <code class="docutils literal notranslate"><span class="pre">&lt;archive_folder&gt;/Unsaved</span></code> and stops being retried. <strong>A
message is never deleted on this path, whatever the <code class="docutils literal notranslate"><span class="pre">delete</span></code> options say.</strong>
To recover after fixing the output destination, either move the messages
from <code class="docutils literal notranslate"><span class="pre">Archive/Unsaved</span></code> back into the reports folder, or run parsedmarc once
with <code class="docutils literal notranslate"><span class="pre">reports_folder</span> <span class="pre">=</span> <span class="pre">Archive/Unsaved</span></code>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The failure counts live in memory, so they are counted per parsedmarc
process. In watch mode — a long-running process that checks the mailbox
repeatedly — the cap works as described across checks. A one-shot run
(<code class="docutils literal notranslate"><span class="pre">cron</span></code>, <code class="docutils literal notranslate"><span class="pre">systemd</span></code> timers) attempts each message exactly once and then
exits, so its counts start over every time and the default cap is never
reached: messages simply keep being retried on every run, which is the
safe direction. Set <code class="docutils literal notranslate"><span class="pre">max_unsaved_retries</span> <span class="pre">=</span> <span class="pre">0</span></code> if you want one-shot runs to
move unsavable messages to <code class="docutils literal notranslate"><span class="pre">Unsaved</span></code> immediately instead.</p>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Retrying a batch means re-sending it. Output destinations that
deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize
an already-saved report — are unaffected, and S3 is idempotent because
each report is written to an object key built from its type, date, and
report ID, so a retry overwrites the same object. Kafka,
Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the
<code class="docutils literal notranslate"><span class="pre">--output</span></code> JSON/CSV files all append unconditionally, so each retry adds
another copy of every report in the batch. That is why the default retry
cap is deliberately low: at most three deliveries per report before its
message is set aside in <code class="docutils literal notranslate"><span class="pre">Unsaved</span></code>. The summary email covers everything
parsed in a run, including reports whose batch failed to save, so a report
retried across runs can also appear in more than one summary email.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Not every destination can report a failed delivery. The webhook output
deliberately logs and swallows its own HTTP and network errors, and the
syslog and GELF outputs send through Python logging handlers, which
swallow delivery errors by design — so an unreachable webhook, syslog, or
GELF endpoint is <em>not</em> treated as a failed save and does not hold a
batch’s messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
Kafka, S3, PostgreSQL, Azure Log Analytics, and the <code class="docutils literal notranslate"><span class="pre">--output</span></code> files are
all detected and do.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><code class="docutils literal notranslate"><span class="pre">since</span></code> interacts with retries: a message that ages out of the configured
<code class="docutils literal notranslate"><span class="pre">since</span></code> window stops being fetched, and therefore stops being retried
automatically. It is never deleted or moved — it simply stays in the
reports folder until it is processed by a run with a wider (or no)
<code class="docutils literal notranslate"><span class="pre">since</span></code> window.</p>
</div>
</section>
</section>
<section id="environment-variable-configuration">
<h2>Environment variable configuration<a class="headerlink" href="#environment-variable-configuration" title="Link to this heading"></a></h2>
@@ -1154,6 +1291,48 @@ then read and stored as the <code class="docutils literal notranslate"><span cla
</table>
</section>
</section>
<section id="using-parsedmarc-as-a-library">
<h2>Using parsedmarc as a library<a class="headerlink" href="#using-parsedmarc-as-a-library" title="Link to this heading"></a></h2>
<p><code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> is also importable as a regular Python package, not just a CLI
tool. The main entry points — <code class="docutils literal notranslate"><span class="pre">parse_report_file()</span></code>, <code class="docutils literal notranslate"><span class="pre">parse_aggregate_report_xml()</span></code>,
<code class="docutils literal notranslate"><span class="pre">parse_aggregate_report_file()</span></code>, <code class="docutils literal notranslate"><span class="pre">parse_failure_report()</span></code>, <code class="docutils literal notranslate"><span class="pre">parse_report_email()</span></code>,
<code class="docutils literal notranslate"><span class="pre">get_dmarc_reports_from_mbox()</span></code>, <code class="docutils literal notranslate"><span class="pre">get_dmarc_reports_from_mailbox()</span></code>, and
<code class="docutils literal notranslate"><span class="pre">watch_inbox()</span></code> — are all importable
directly from the <code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> package. See the <a class="reference internal" href="api.html"><span class="std std-doc">API reference</span></a> for
the full set of modules and members.</p>
<p>Each of these functions accepts either individual option keyword arguments
(<code class="docutils literal notranslate"><span class="pre">offline</span></code>, <code class="docutils literal notranslate"><span class="pre">nameservers</span></code>, <code class="docutils literal notranslate"><span class="pre">dns_timeout</span></code>, etc.) or a single <code class="docutils literal notranslate"><span class="pre">config=</span></code> keyword
argument carrying a <code class="docutils literal notranslate"><span class="pre">ParserConfig</span></code> instance:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc</span><span class="w"> </span><span class="kn">import</span> <span class="n">ParserConfig</span><span class="p">,</span> <span class="n">parse_report_file</span><span class="p">,</span> <span class="n">get_dmarc_reports_from_mailbox</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc.mail</span><span class="w"> </span><span class="kn">import</span> <span class="n">IMAPConnection</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">ParserConfig</span><span class="p">(</span>
<span class="n">offline</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">nameservers</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;1.1.1.1&quot;</span><span class="p">,</span> <span class="s2">&quot;1.0.0.1&quot;</span><span class="p">],</span>
<span class="n">dns_timeout</span><span class="o">=</span><span class="mf">5.0</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">report</span> <span class="o">=</span> <span class="n">parse_report_file</span><span class="p">(</span><span class="s2">&quot;aggregate_report.xml.gz&quot;</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">)</span>
<span class="n">connection</span> <span class="o">=</span> <span class="n">IMAPConnection</span><span class="p">(</span>
<span class="n">host</span><span class="o">=</span><span class="s2">&quot;imap.example.com&quot;</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="s2">&quot;dmarc@example.com&quot;</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="s2">&quot;...&quot;</span>
<span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">get_dmarc_reports_from_mailbox</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">)</span>
</pre></div>
</div>
<p>A few things to keep in mind:</p>
<ul class="simple">
<li><p>When <code class="docutils literal notranslate"><span class="pre">config=</span></code> is passed, the individual option keyword arguments are
ignored in favor of the values carried on the <code class="docutils literal notranslate"><span class="pre">ParserConfig</span></code> instance.</p></li>
<li><p>Each explicitly constructed <code class="docutils literal notranslate"><span class="pre">ParserConfig</span></code> owns its own isolated caches
(IP address info, seen aggregate report IDs, and the reverse DNS map).
Omitting <code class="docutils literal notranslate"><span class="pre">config=</span></code> falls back to the process-wide caches shared by every
call that doesn’t pass one.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">keep_alive</span></code> and <code class="docutils literal notranslate"><span class="pre">n_procs</span></code> are not part of <code class="docutils literal notranslate"><span class="pre">ParserConfig</span></code> — they control
process/worker orchestration rather than parsing or enrichment behavior,
so they are always passed as separate keyword arguments.</p></li>
</ul>
</section>
<section id="performance-tuning">
<h2>Performance tuning<a class="headerlink" href="#performance-tuning" title="Link to this heading"></a></h2>
<p>For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
@@ -1164,8 +1343,12 @@ imports more predictable:</p>
<li><p>Reduce <code class="docutils literal notranslate"><span class="pre">mailbox.batch_size</span></code> to smaller values such as <code class="docutils literal notranslate"><span class="pre">100-500</span></code> instead of
processing a very large message set at once. Smaller batches trade throughput
for lower peak memory use and less sink pressure.</p></li>
<li><p>Keep <code class="docutils literal notranslate"><span class="pre">n_procs</span></code> low for mailbox-heavy runs. In practice, <code class="docutils literal notranslate"><span class="pre">1-2</span></code> workers is often
a safer starting point for large backfills than aggressive parallelism.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">n_procs</span></code> now parallelizes parsing for mailbox and mbox runs too, not just
report files passed directly as CLI arguments. It pays off most when a run
is bound by DNS/GeoIP enrichment rather than fetching or output. The
trade-off is memory and DNS load: at most roughly <code class="docutils literal notranslate"><span class="pre">2</span> <span class="pre">*</span> <span class="pre">n_procs</span></code> messages
are held in flight at once, each worker process keeps its own DNS/GeoIP
cache, and DNS query volume can multiply by up to <code class="docutils literal notranslate"><span class="pre">n_procs</span></code>.</p></li>
<li><p>Use <code class="docutils literal notranslate"><span class="pre">mailbox.since</span></code> to process reports in smaller time windows such as <code class="docutils literal notranslate"><span class="pre">1d</span></code>,
<code class="docutils literal notranslate"><span class="pre">7d</span></code>, or another interval that fits the backlog. This makes it easier to catch
up incrementally instead of loading an entire mailbox history in one run.</p></li>