amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-06-09 12:19:42 +00:00
Code Issues Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Sean Whalen
2026-03-10 20:05:02 -04:00
parent 100f24c578
commit 10cc0d1bad
27 changed files with 221 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_modules/index.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Overview: module code &mdash; parsedmarc 9.2.0 documentation</title>
<title>Overview: module code &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=e59714d7" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=e8b940be"></script>
<script src="../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/js/theme.js"></script>
_modules/parsedmarc.html
+25 -15
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=e59714d7" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=e8b940be"></script>
<script src="../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../_static/doctools.js?v=9bcbadda"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/js/theme.js"></script>
@@ -1075,10 +1075,12 @@
<div class="viewcode-block" id="extract_report_from_file_path">
<a class="viewcode-back" href="../api.html#parsedmarc.extract_report_from_file_path">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">extract_report_from_file_path</span><span class="p">(</span><span class="n">file_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="k">def</span><span class="w"> </span><span class="nf">extract_report_from_file_path</span><span class="p">(</span>
<span class="n">file_path</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">PathLike</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">os</span><span class="o">.</span><span class="n">PathLike</span><span class="p">[</span><span class="nb">bytes</span><span class="p">]],</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Extracts report from a file at the given file_path&quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_path</span><span class="p">,</span> <span class="s2">&quot;rb&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">report_file</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">fspath</span><span class="p">(</span><span class="n">file_path</span><span class="p">),</span> <span class="s2">&quot;rb&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">report_file</span><span class="p">:</span>
<span class="k">return</span> <span class="n">extract_report</span><span class="p">(</span><span class="n">report_file</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
<span class="k">except</span> <span class="ne">FileNotFoundError</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ParserError</span><span class="p">(</span><span class="s2">&quot;File was not found&quot;</span><span class="p">)</span></div>
@@ -1797,7 +1799,7 @@
<div class="viewcode-block" id="parse_report_file">
<a class="viewcode-back" href="../api.html#parsedmarc.parse_report_file">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">parse_report_file</span><span class="p">(</span>
<span class="n">input_</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">bytes</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">BinaryIO</span><span class="p">],</span>
<span class="n">input_</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">bytes</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">PathLike</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">os</span><span class="o">.</span><span class="n">PathLike</span><span class="p">[</span><span class="nb">bytes</span><span class="p">],</span> <span class="n">BinaryIO</span><span class="p">],</span>
<span class="o">*</span><span class="p">,</span>
<span class="n">nameservers</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">dns_timeout</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">2.0</span><span class="p">,</span>
@@ -1814,7 +1816,8 @@
<span class="sd"> file-like object. or bytes</span>
<span class="sd"> Args:</span>
<span class="sd"> input_ (str | bytes | BinaryIO): A path to a file, a file like object, or bytes</span>
<span class="sd"> input_ (str | os.PathLike | bytes | BinaryIO): A path to a file,</span>
<span class="sd"> a file-like object, or bytes</span>
<span class="sd"> nameservers (list): A list of one or more nameservers to use</span>
<span class="sd"> (Cloudflare&#39;s public DNS resolvers by default)</span>
<span class="sd"> dns_timeout (float): Sets the DNS timeout in seconds</span>
@@ -1831,9 +1834,10 @@
<span class="sd"> dict: The parsed DMARC report</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">file_object</span><span class="p">:</span> <span class="n">BinaryIO</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">input_</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Parsing </span><span class="si">{0}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">input_</span><span class="p">))</span>
<span class="n">file_object</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">input_</span><span class="p">,</span> <span class="s2">&quot;rb&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">input_</span><span class="p">,</span> <span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">PathLike</span><span class="p">)):</span>
<span class="n">file_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fspath</span><span class="p">(</span><span class="n">input_</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Parsing </span><span class="si">{0}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">file_path</span><span class="p">))</span>
<span class="n">file_object</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_path</span><span class="p">,</span> <span class="s2">&quot;rb&quot;</span><span class="p">)</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">input_</span><span class="p">,</span> <span class="p">(</span><span class="nb">bytes</span><span class="p">,</span> <span class="nb">bytearray</span><span class="p">,</span> <span class="nb">memoryview</span><span class="p">)):</span>
<span class="n">file_object</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">input_</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
@@ -2280,14 +2284,17 @@
<span class="s2">&quot;smtp_tls_reports&quot;</span><span class="p">:</span> <span class="n">smtp_tls_reports</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">if</span> <span class="n">current_time</span><span class="p">:</span>
<span class="n">total_messages</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span>
<span class="n">connection</span><span class="o">.</span><span class="n">fetch_messages</span><span class="p">(</span><span class="n">reports_folder</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="n">current_time</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">test</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">batch_size</span><span class="p">:</span>
<span class="k">if</span> <span class="n">current_time</span><span class="p">:</span>
<span class="n">total_messages</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span>
<span class="n">connection</span><span class="o">.</span><span class="n">fetch_messages</span><span class="p">(</span><span class="n">reports_folder</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="n">current_time</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">total_messages</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">connection</span><span class="o">.</span><span class="n">fetch_messages</span><span class="p">(</span><span class="n">reports_folder</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">total_messages</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">connection</span><span class="o">.</span><span class="n">fetch_messages</span><span class="p">(</span><span class="n">reports_folder</span><span class="p">))</span>
<span class="n">total_messages</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">test</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">batch_size</span> <span class="ow">and</span> <span class="n">total_messages</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">if</span> <span class="n">total_messages</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Process emails that came in during the last run</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">get_dmarc_reports_from_mailbox</span><span class="p">(</span>
<span class="n">connection</span><span class="o">=</span><span class="n">connection</span><span class="p">,</span>
@@ -2332,6 +2339,7 @@
<span class="n">dns_timeout</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">6.0</span><span class="p">,</span>
<span class="n">strip_attachment_payloads</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">batch_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span>
<span class="n">since</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">Union</span><span class="p">[</span><span class="n">datetime</span><span class="p">,</span> <span class="n">date</span><span class="p">,</span> <span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">normalize_timespan_threshold_hours</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mi">24</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
@@ -2358,6 +2366,7 @@
<span class="sd"> strip_attachment_payloads (bool): Replace attachment payloads in</span>
<span class="sd"> forensic report samples with None</span>
<span class="sd"> batch_size (int): Number of messages to read and process before saving</span>
<span class="sd"> since: Search for messages since certain time</span>
<span class="sd"> normalize_timespan_threshold_hours (float): Normalize timespans beyond this</span>
<span class="sd"> &quot;&quot;&quot;</span>
@@ -2377,6 +2386,7 @@
<span class="n">dns_timeout</span><span class="o">=</span><span class="n">dns_timeout</span><span class="p">,</span>
<span class="n">strip_attachment_payloads</span><span class="o">=</span><span class="n">strip_attachment_payloads</span><span class="p">,</span>
<span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span>
<span class="n">since</span><span class="o">=</span><span class="n">since</span><span class="p">,</span>
<span class="n">create_folders</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">normalize_timespan_threshold_hours</span><span class="o">=</span><span class="n">normalize_timespan_threshold_hours</span><span class="p">,</span>
<span class="p">)</span>
_modules/parsedmarc/elastic.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.elastic &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc.elastic &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=e59714d7" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=e8b940be"></script>
<script src="../../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/opensearch.html
+35 -6
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.opensearch &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc.opensearch &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=e59714d7" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=e8b940be"></script>
<script src="../../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../_static/js/theme.js"></script>
@@ -86,7 +86,9 @@
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Union</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">boto3</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">opensearchpy</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">AWSV4SignerAuth</span><span class="p">,</span>
<span class="n">Boolean</span><span class="p">,</span>
<span class="n">Date</span><span class="p">,</span>
<span class="n">Document</span><span class="p">,</span>
@@ -97,6 +99,7 @@
<span class="n">Nested</span><span class="p">,</span>
<span class="n">Object</span><span class="p">,</span>
<span class="n">Q</span><span class="p">,</span>
<span class="n">RequestsHttpConnection</span><span class="p">,</span>
<span class="n">Search</span><span class="p">,</span>
<span class="n">Text</span><span class="p">,</span>
<span class="n">connections</span><span class="p">,</span>
@@ -362,6 +365,9 @@
<span class="n">password</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">api_key</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">timeout</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">float</span><span class="p">]</span> <span class="o">=</span> <span class="mf">60.0</span><span class="p">,</span>
<span class="n">auth_type</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;basic&quot;</span><span class="p">,</span>
<span class="n">aws_region</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">aws_service</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;es&quot;</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Sets the OpenSearch hosts to use</span>
@@ -374,6 +380,9 @@
<span class="sd"> password (str): The password to use for authentication</span>
<span class="sd"> api_key (str): The Base64 encoded API key to use for authentication</span>
<span class="sd"> timeout (float): Timeout in seconds</span>
<span class="sd"> auth_type (str): OpenSearch auth mode: basic (default) or awssigv4</span>
<span class="sd"> aws_region (str): AWS region for SigV4 auth (required for awssigv4)</span>
<span class="sd"> aws_service (str): AWS service for SigV4 signing (default: es)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hosts</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="n">hosts</span><span class="p">]</span>
@@ -385,10 +394,30 @@
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;ca_certs&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">ssl_cert_path</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;verify_certs&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">if</span> <span class="n">username</span> <span class="ow">and</span> <span class="n">password</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;http_auth&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">username</span> <span class="o">+</span> <span class="s2">&quot;:&quot;</span> <span class="o">+</span> <span class="n">password</span>
<span class="k">if</span> <span class="n">api_key</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;api_key&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">api_key</span>
<span class="n">normalized_auth_type</span> <span class="o">=</span> <span class="p">(</span><span class="n">auth_type</span> <span class="ow">or</span> <span class="s2">&quot;basic&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
<span class="k">if</span> <span class="n">normalized_auth_type</span> <span class="o">==</span> <span class="s2">&quot;awssigv4&quot;</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aws_region</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">OpenSearchError</span><span class="p">(</span>
<span class="s2">&quot;OpenSearch AWS SigV4 auth requires &#39;aws_region&#39; to be set&quot;</span>
<span class="p">)</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
<span class="n">credentials</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">get_credentials</span><span class="p">()</span>
<span class="k">if</span> <span class="n">credentials</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">OpenSearchError</span><span class="p">(</span>
<span class="s2">&quot;Unable to load AWS credentials for OpenSearch SigV4 authentication&quot;</span>
<span class="p">)</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;http_auth&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">AWSV4SignerAuth</span><span class="p">(</span><span class="n">credentials</span><span class="p">,</span> <span class="n">aws_region</span><span class="p">,</span> <span class="n">aws_service</span><span class="p">)</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;connection_class&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">RequestsHttpConnection</span>
<span class="k">elif</span> <span class="n">normalized_auth_type</span> <span class="o">==</span> <span class="s2">&quot;basic&quot;</span><span class="p">:</span>
<span class="k">if</span> <span class="n">username</span> <span class="ow">and</span> <span class="n">password</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;http_auth&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">username</span> <span class="o">+</span> <span class="s2">&quot;:&quot;</span> <span class="o">+</span> <span class="n">password</span>
<span class="k">if</span> <span class="n">api_key</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;api_key&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">api_key</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">OpenSearchError</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Unsupported OpenSearch auth_type &#39;</span><span class="si">{</span><span class="n">auth_type</span><span class="si">}</span><span class="s2">&#39;. &quot;</span>
<span class="s2">&quot;Expected &#39;basic&#39; or &#39;awssigv4&#39;.&quot;</span>
<span class="p">)</span>
<span class="n">connections</span><span class="o">.</span><span class="n">create_connection</span><span class="p">(</span><span class="o">**</span><span class="n">conn_params</span><span class="p">)</span></div>
_modules/parsedmarc/splunk.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.splunk &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc.splunk &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=e59714d7" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=e8b940be"></script>
<script src="../../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/types.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.types &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc.types &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=e59714d7" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=e8b940be"></script>
<script src="../../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/utils.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.utils &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc.utils &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=e59714d7" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=e8b940be"></script>
<script src="../../_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../_static/js/theme.js"></script>
_sources/usage.md.txt
+52 -1
View File
@@ -146,6 +146,9 @@ The full set of configuration options are:
- `dns_timeout` - float: DNS timeout period
- `debug` - bool: Print debugging messages
- `silent` - bool: Only print errors (Default: `True`)
- `fail_on_output_error` - bool: Exit with a non-zero status code if
any configured output destination fails while saving/publishing
reports (Default: `False`)
- `log_file` - str: Write log messages to a file at this path
- `n_procs` - int: Number of process to run in parallel when
parsing in CLI mode (Default: `1`)
@@ -200,7 +203,7 @@ The full set of configuration options are:
- `password` - str: The IMAP password
- `msgraph`
- `auth_method` - str: Authentication method, valid types are
`UsernamePassword`, `DeviceCode`, or `ClientSecret`
`UsernamePassword`, `DeviceCode`, `ClientSecret`, or `Certificate`
(Default: `UsernamePassword`).
- `user` - str: The M365 user, required when the auth method is
UsernamePassword
@@ -208,6 +211,11 @@ The full set of configuration options are:
method is UsernamePassword
- `client_id` - str: The app registration's client ID
- `client_secret` - str: The app registration's secret
- `certificate_path` - str: Path to a PEM or PKCS12 certificate
including the private key. Required when the auth method is
`Certificate`
- `certificate_password` - str: Optional password for the
certificate file when using `Certificate` auth
- `tenant_id` - str: The Azure AD tenant ID. This is required
for all auth methods except UsernamePassword.
- `mailbox` - str: The mailbox name. This defaults to the
@@ -245,6 +253,9 @@ The full set of configuration options are:
-Description "Restrict access to dmarc reports mailbox."
```
The same application permission and mailbox scoping guidance
applies to the `Certificate` auth method.
:::
- `elasticsearch`
- `hosts` - str: A comma separated list of hostnames and ports
@@ -281,6 +292,10 @@ The full set of configuration options are:
- `user` - str: Basic auth username
- `password` - str: Basic auth password
- `api_key` - str: API key
- `auth_type` - str: Authentication type: `basic` (default) or `awssigv4` (the key `authentication_type` is accepted as an alias for this option)
- `aws_region` - str: AWS region for SigV4 authentication
(required when `auth_type = awssigv4`)
- `aws_service` - str: AWS service for SigV4 signing (Default: `es`)
- `ssl` - bool: Use an encrypted SSL/TLS connection
(Default: `True`)
- `timeout` - float: Timeout in seconds (Default: 60)
@@ -394,10 +409,19 @@ The full set of configuration options are:
credentials, None to disable (Default: `None`)
- `token_file` - str: Path to save the token file
(Default: `.token`)
- `auth_mode` - str: Authentication mode, `installed_app` (default)
or `service_account`
- `service_account_user` - str: Delegated mailbox user for Gmail
service account auth (required for domain-wide delegation). Also
accepted as `delegated_user` for backward compatibility.
:::{note}
credentials_file and token_file can be got with [quickstart](https://developers.google.com/gmail/api/quickstart/python).Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
:::
:::{note}
When `auth_mode = service_account`, `credentials_file` must point to a
Google service account key JSON file, and `token_file` is not used.
:::
- `include_spam_trash` - bool: Include messages in Spam and
Trash when searching reports (Default: `False`)
- `scopes` - str: Comma separated list of scopes to use when
@@ -502,6 +526,33 @@ PUT _cluster/settings
Increasing this value increases resource usage.
:::
## Performance tuning
For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
of memory, especially when it runs on the same host as Elasticsearch or
OpenSearch. The following settings can reduce peak memory usage and make long
imports more predictable:
- Reduce `mailbox.batch_size` to smaller values such as `100-500` instead of
processing a very large message set at once. Smaller batches trade throughput
for lower peak memory use and less sink pressure.
- Keep `n_procs` low for mailbox-heavy runs. In practice, `1-2` workers is often
a safer starting point for large backfills than aggressive parallelism.
- Use `mailbox.since` to process reports in smaller time windows such as `1d`,
`7d`, or another interval that fits the backlog. This makes it easier to catch
up incrementally instead of loading an entire mailbox history in one run.
- Set `strip_attachment_payloads = True` when forensic reports contain large
attachments and you do not need to retain the raw payloads in the parsed
output.
- Prefer running parsedmarc separately from Elasticsearch or OpenSearch, or
reserve enough RAM for both services if they must share a host.
- For very large imports, prefer incremental supervised runs, such as a
scheduler or systemd service, over infrequent massive backfills.
These are operational tuning recommendations rather than hard requirements, but
they are often enough to avoid memory pressure and reduce failures during
high-volume mailbox processing.
## Multi-tenant support
Starting in `8.19.0`, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file that is formatted where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:
_static/documentation_options.js
+1 -1
View File
@@ -1,5 +1,5 @@
const DOCUMENTATION_OPTIONS = {
VERSION: '9.2.0',
VERSION: '9.2.1',
LANGUAGE: 'en',
COLLAPSE_INDEX: false,
BUILDER: 'html',
api.html
+12 -7
View File
File diff suppressed because one or more lines are too long
contributing.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Contributing to parsedmarc &mdash; parsedmarc 9.2.0 documentation</title>
<title>Contributing to parsedmarc &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
davmail.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 9.2.0 documentation</title>
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
dmarc.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Understanding DMARC &mdash; parsedmarc 9.2.0 documentation</title>
<title>Understanding DMARC &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
elasticsearch.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Elasticsearch and Kibana &mdash; parsedmarc 9.2.0 documentation</title>
<title>Elasticsearch and Kibana &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
genindex.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Index &mdash; parsedmarc 9.2.0 documentation</title>
<title>Index &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
index.html
+3 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 9.2.0 documentation</title>
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
@@ -183,6 +183,7 @@ for RHEL or Debian.</p>
<li class="toctree-l1"><a class="reference internal" href="usage.html">Using parsedmarc</a><ul>
<li class="toctree-l2"><a class="reference internal" href="usage.html#cli-help">CLI help</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#configuration-file">Configuration file</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#performance-tuning">Performance tuning</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#multi-tenant-support">Multi-tenant support</a></li>
<li class="toctree-l2"><a class="reference internal" href="usage.html#running-parsedmarc-as-a-systemd-service">Running parsedmarc as a systemd service</a></li>
</ul>
installation.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Installation &mdash; parsedmarc 9.2.0 documentation</title>
<title>Installation &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
kibana.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using the Kibana dashboards &mdash; parsedmarc 9.2.0 documentation</title>
<title>Using the Kibana dashboards &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
mailing-lists.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>What about mailing lists? &mdash; parsedmarc 9.2.0 documentation</title>
<title>What about mailing lists? &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
objects.inv
BIN
View File
Binary file not shown.
opensearch.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>OpenSearch and Grafana &mdash; parsedmarc 9.2.0 documentation</title>
<title>OpenSearch and Grafana &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
output.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Sample outputs &mdash; parsedmarc 9.2.0 documentation</title>
<title>Sample outputs &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
py-modindex.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Python Module Index &mdash; parsedmarc 9.2.0 documentation</title>
<title>Python Module Index &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
search.html
+2 -2
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Search &mdash; parsedmarc 9.2.0 documentation</title>
<title>Search &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
@@ -13,7 +13,7 @@
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
splunk.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Splunk &mdash; parsedmarc 9.2.0 documentation</title>
<title>Splunk &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
usage.html
+56 -4
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using parsedmarc &mdash; parsedmarc 9.2.0 documentation</title>
<title>Using parsedmarc &mdash; parsedmarc 9.2.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=e8b940be"></script>
<script src="_static/documentation_options.js?v=4cc1c4a7"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
@@ -48,6 +48,7 @@
<li class="toctree-l1 current"><a class="current reference internal" href="#">Using parsedmarc</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#cli-help">CLI help</a></li>
<li class="toctree-l2"><a class="reference internal" href="#configuration-file">Configuration file</a></li>
<li class="toctree-l2"><a class="reference internal" href="#performance-tuning">Performance tuning</a></li>
<li class="toctree-l2"><a class="reference internal" href="#multi-tenant-support">Multi-tenant support</a></li>
<li class="toctree-l2"><a class="reference internal" href="#running-parsedmarc-as-a-systemd-service">Running parsedmarc as a systemd service</a></li>
</ul>
@@ -232,6 +233,9 @@ DNS resolvers (Default: <code class="docutils literal notranslate"><span class="
<li><p><code class="docutils literal notranslate"><span class="pre">dns_timeout</span></code> - float: DNS timeout period</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">debug</span></code> - bool: Print debugging messages</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">silent</span></code> - bool: Only print errors (Default: <code class="docutils literal notranslate"><span class="pre">True</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fail_on_output_error</span></code> - bool: Exit with a non-zero status code if
any configured output destination fails while saving/publishing
reports (Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">log_file</span></code> - str: Write log messages to a file at this path</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">n_procs</span></code> - int: Number of process to run in parallel when
parsing in CLI mode (Default: <code class="docutils literal notranslate"><span class="pre">1</span></code>)</p>
@@ -295,7 +299,7 @@ verification (not recommended)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">msgraph</span></code></p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">auth_method</span></code> - str: Authentication method, valid types are
<code class="docutils literal notranslate"><span class="pre">UsernamePassword</span></code>, <code class="docutils literal notranslate"><span class="pre">DeviceCode</span></code>, or <code class="docutils literal notranslate"><span class="pre">ClientSecret</span></code>
<code class="docutils literal notranslate"><span class="pre">UsernamePassword</span></code>, <code class="docutils literal notranslate"><span class="pre">DeviceCode</span></code>, <code class="docutils literal notranslate"><span class="pre">ClientSecret</span></code>, or <code class="docutils literal notranslate"><span class="pre">Certificate</span></code>
(Default: <code class="docutils literal notranslate"><span class="pre">UsernamePassword</span></code>).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">user</span></code> - str: The M365 user, required when the auth method is
UsernamePassword</p></li>
@@ -303,6 +307,11 @@ UsernamePassword</p></li>
method is UsernamePassword</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">client_id</span></code> - str: The app registration’s client ID</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">client_secret</span></code> - str: The app registration’s secret</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">certificate_path</span></code> - str: Path to a PEM or PKCS12 certificate
including the private key. Required when the auth method is
<code class="docutils literal notranslate"><span class="pre">Certificate</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">certificate_password</span></code> - str: Optional password for the
certificate file when using <code class="docutils literal notranslate"><span class="pre">Certificate</span></code> auth</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">tenant_id</span></code> - str: The Azure AD tenant ID. This is required
for all auth methods except UsernamePassword.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mailbox</span></code> - str: The mailbox name. This defaults to the
@@ -338,6 +347,8 @@ group and use that as the group id.</p>
<span class="n">-Description</span> <span class="s2">&quot;Restrict access to dmarc reports mailbox.&quot;</span>
</pre></div>
</div>
<p>The same application permission and mailbox scoping guidance
applies to the <code class="docutils literal notranslate"><span class="pre">Certificate</span></code> auth method.</p>
</div>
</li>
</ul>
@@ -383,6 +394,10 @@ or URLs (e.g. <code class="docutils literal notranslate"><span class="pre">127.0
<li><p><code class="docutils literal notranslate"><span class="pre">user</span></code> - str: Basic auth username</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">password</span></code> - str: Basic auth password</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">api_key</span></code> - str: API key</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">auth_type</span></code> - str: Authentication type: <code class="docutils literal notranslate"><span class="pre">basic</span></code> (default) or <code class="docutils literal notranslate"><span class="pre">awssigv4</span></code> (the key <code class="docutils literal notranslate"><span class="pre">authentication_type</span></code> is accepted as an alias for this option)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">aws_region</span></code> - str: AWS region for SigV4 authentication
(required when <code class="docutils literal notranslate"><span class="pre">auth_type</span> <span class="pre">=</span> <span class="pre">awssigv4</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">aws_service</span></code> - str: AWS service for SigV4 signing (Default: <code class="docutils literal notranslate"><span class="pre">es</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ssl</span></code> - bool: Use an encrypted SSL/TLS connection
(Default: <code class="docutils literal notranslate"><span class="pre">True</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">timeout</span></code> - float: Timeout in seconds (Default: 60)</p></li>
@@ -506,11 +521,21 @@ so use <code class="docutils literal notranslate"><span class="pre">%%</span></c
<li><p><code class="docutils literal notranslate"><span class="pre">credentials_file</span></code> - str: Path to file containing the
credentials, None to disable (Default: <code class="docutils literal notranslate"><span class="pre">None</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">token_file</span></code> - str: Path to save the token file
(Default: <code class="docutils literal notranslate"><span class="pre">.token</span></code>)</p>
(Default: <code class="docutils literal notranslate"><span class="pre">.token</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">auth_mode</span></code> - str: Authentication mode, <code class="docutils literal notranslate"><span class="pre">installed_app</span></code> (default)
or <code class="docutils literal notranslate"><span class="pre">service_account</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">service_account_user</span></code> - str: Delegated mailbox user for Gmail
service account auth (required for domain-wide delegation). Also
accepted as <code class="docutils literal notranslate"><span class="pre">delegated_user</span></code> for backward compatibility.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>credentials_file and token_file can be got with <a class="reference external" href="https://developers.google.com/gmail/api/quickstart/python">quickstart</a>.Please change the scope to <code class="docutils literal notranslate"><span class="pre">https://www.googleapis.com/auth/gmail.modify</span></code>.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>When <code class="docutils literal notranslate"><span class="pre">auth_mode</span> <span class="pre">=</span> <span class="pre">service_account</span></code>, <code class="docutils literal notranslate"><span class="pre">credentials_file</span></code> must point to a
Google service account key JSON file, and <code class="docutils literal notranslate"><span class="pre">token_file</span></code> is not used.</p>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">include_spam_trash</span></code> - bool: Include messages in Spam and
Trash when searching reports (Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>)</p></li>
@@ -621,6 +646,33 @@ Check current usage (from Management -&gt; Dev Tools -&gt; Console):</p>
<p>Increasing this value increases resource usage.</p>
</div>
</section>
<section id="performance-tuning">
<h2>Performance tuning<a class="headerlink" href="#performance-tuning" title="Link to this heading"></a></h2>
<p>For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
of memory, especially when it runs on the same host as Elasticsearch or
OpenSearch. The following settings can reduce peak memory usage and make long
imports more predictable:</p>
<ul class="simple">
<li><p>Reduce <code class="docutils literal notranslate"><span class="pre">mailbox.batch_size</span></code> to smaller values such as <code class="docutils literal notranslate"><span class="pre">100-500</span></code> instead of
processing a very large message set at once. Smaller batches trade throughput
for lower peak memory use and less sink pressure.</p></li>
<li><p>Keep <code class="docutils literal notranslate"><span class="pre">n_procs</span></code> low for mailbox-heavy runs. In practice, <code class="docutils literal notranslate"><span class="pre">1-2</span></code> workers is often
a safer starting point for large backfills than aggressive parallelism.</p></li>
<li><p>Use <code class="docutils literal notranslate"><span class="pre">mailbox.since</span></code> to process reports in smaller time windows such as <code class="docutils literal notranslate"><span class="pre">1d</span></code>,
<code class="docutils literal notranslate"><span class="pre">7d</span></code>, or another interval that fits the backlog. This makes it easier to catch
up incrementally instead of loading an entire mailbox history in one run.</p></li>
<li><p>Set <code class="docutils literal notranslate"><span class="pre">strip_attachment_payloads</span> <span class="pre">=</span> <span class="pre">True</span></code> when forensic reports contain large
attachments and you do not need to retain the raw payloads in the parsed
output.</p></li>
<li><p>Prefer running parsedmarc separately from Elasticsearch or OpenSearch, or
reserve enough RAM for both services if they must share a host.</p></li>
<li><p>For very large imports, prefer incremental supervised runs, such as a
scheduler or systemd service, over infrequent massive backfills.</p></li>
</ul>
<p>These are operational tuning recommendations rather than hard requirements, but
they are often enough to avoid memory pressure and reduce failures during
high-volume mailbox processing.</p>
</section>
<section id="multi-tenant-support">
<h2>Multi-tenant support<a class="headerlink" href="#multi-tenant-support" title="Link to this heading"></a></h2>
<p>Starting in <code class="docutils literal notranslate"><span class="pre">8.19.0</span></code>, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file that is formatted where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:</p>