amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-08-02 13:42:19 +00:00
Code Issues Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Sean Whalen
2026-07-27 20:51:09 -04:00
parent 39037ba327
commit 0bf39b019a
29 changed files with 366 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_modules/index.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Overview: module code &mdash; parsedmarc 10.4.0 documentation</title>
<title>Overview: module code &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=9edc463e" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../_static/js/theme.js"></script>
_modules/parsedmarc.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=9edc463e" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../_static/js/theme.js"></script>
_modules/parsedmarc/config.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.config &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc.config &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/elastic.html
+121 -6
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.elastic &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc.elastic &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
@@ -102,6 +102,7 @@
<span class="n">Text</span><span class="p">,</span>
<span class="n">connections</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">elasticsearch.helpers</span><span class="w"> </span><span class="kn">import</span> <span class="n">reindex</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc</span><span class="w"> </span><span class="kn">import</span> <span class="n">InvalidFailureReport</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc.log</span><span class="w"> </span><span class="kn">import</span> <span class="n">logger</span>
@@ -771,18 +772,67 @@
<span class="n">_LEGACY_FO_FIELD</span> <span class="o">=</span> <span class="s2">&quot;published_policy.fo&quot;</span>
<span class="c1"># The same field split into the object/leaf names a mapping body nests it</span>
<span class="c1"># under. Derived from the dotted name so the mapping written below cannot</span>
<span class="c1"># drift from the field _legacy_fo_field_type() reads.</span>
<span class="n">_LEGACY_FO_OBJECT</span><span class="p">,</span> <span class="n">_LEGACY_FO_LEAF</span> <span class="o">=</span> <span class="n">_LEGACY_FO_FIELD</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_legacy_fo_field_type</span><span class="p">(</span><span class="n">index</span><span class="p">:</span> <span class="n">Index</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Return the mapped type of ``published_policy.fo`` in *index*.</span>
<span class="sd"> Returns ``None`` when the index does not map the field at all.</span>
<span class="sd"> Elasticsearch 6-era clusters keyed field mappings by the mapping type</span>
<span class="sd"> name (``doc``); mapping types are gone from Elasticsearch 8, whose</span>
<span class="sd"> responses put the field directly under ``mappings``. The type-keyed</span>
<span class="sd"> shape is only descended into when such a key is actually present, so</span>
<span class="sd"> this reads either shape.</span>
<span class="sd"> Args:</span>
<span class="sd"> index (Index): The index to inspect.</span>
<span class="sd"> Returns:</span>
<span class="sd"> str | None: The mapped field type, e.g. ``&quot;long&quot;`` or ``&quot;text&quot;``.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">index</span><span class="o">.</span><span class="n">get_field_mapping</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="n">_LEGACY_FO_FIELD</span><span class="p">])</span>
<span class="n">mappings</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="nb">list</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]][</span><span class="s2">&quot;mappings&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="n">_LEGACY_FO_FIELD</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">mappings</span><span class="p">:</span>
<span class="k">for</span> <span class="n">type_name</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;doc&quot;</span><span class="p">,</span> <span class="s2">&quot;_doc&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="n">type_name</span> <span class="ow">in</span> <span class="n">mappings</span><span class="p">:</span>
<span class="n">mappings</span> <span class="o">=</span> <span class="n">mappings</span><span class="p">[</span><span class="n">type_name</span><span class="p">]</span>
<span class="k">break</span>
<span class="n">field_mapping</span> <span class="o">=</span> <span class="n">mappings</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">_LEGACY_FO_FIELD</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">field_mapping</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span>
<span class="k">return</span> <span class="n">field_mapping</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;mapping&quot;</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">_LEGACY_FO_LEAF</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;type&quot;</span><span class="p">)</span>
<div class="viewcode-block" id="migrate_indexes">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.migrate_indexes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">migrate_indexes</span><span class="p">(</span>
<span class="n">aggregate_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">failure_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">smtp_tls_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">legacy_fo_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Backfills the ``dkim_results_combined``/``spf_results_combined`` fields</span>
<span class="sd"> (added for issue #169) on aggregate report documents, and the</span>
<span class="sd"> ``policies_combined``/``failure_details_combined`` fields on SMTP TLS</span>
<span class="sd"> report documents, that were saved before those fields existed.</span>
<span class="sd"> Runs index migrations and backfills.</span>
<span class="sd"> First, the legacy ``published_policy.fo`` migration, for each name in</span>
<span class="sd"> ``legacy_fo_indexes``: parsedmarc releases before 5.0.0 declared that</span>
<span class="sd"> field as an integer, so those indexes mapped it as ``long``, which</span>
<span class="sd"> cannot hold the multi-value ``fo`` settings reports carry (``0:1``,</span>
<span class="sd"> ``d:s``). Such an index is rebuilt as a ``-v2`` index with the</span>
<span class="sd"> text/keyword shape, the documents are reindexed into it, and the</span>
<span class="sd"> original is deleted.</span>
<span class="sd"> Second, it backfills the ``dkim_results_combined``/</span>
<span class="sd"> ``spf_results_combined`` fields (added for issue #169) on aggregate</span>
<span class="sd"> report documents, and the ``policies_combined``/</span>
<span class="sd"> ``failure_details_combined`` fields on SMTP TLS report documents, that</span>
<span class="sd"> were saved before those fields existed.</span>
<span class="sd"> For each name in ``aggregate_indexes``/``smtp_tls_indexes``, this</span>
<span class="sd"> submits an ``update_by_query`` against the ``f&quot;{name}*&quot;`` index pattern</span>
@@ -802,7 +852,72 @@
<span class="sd"> failure_indexes (list): A list of failure index names</span>
<span class="sd"> (accepted for API compatibility; unused)</span>
<span class="sd"> smtp_tls_indexes (list): A list of SMTP TLS index names</span>
<span class="sd"> legacy_fo_indexes (list): A list of index names to check for the</span>
<span class="sd"> pre-5.0.0 ``published_policy.fo`` ``long`` mapping. Unlike the</span>
<span class="sd"> backfill arguments these are exact names, not patterns:</span>
<span class="sd"> 5.0.0 introduced date-suffixed index names in the same release</span>
<span class="sd"> that fixed the mapping, so an affected index has no date</span>
<span class="sd"> component. It may still be prefixed or suffixed -- both</span>
<span class="sd"> options date back to 4.1.0 -- so callers should pass the</span>
<span class="sd"> names their own ``index_prefix``/``index_suffix``</span>
<span class="sd"> configuration produces.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aggregate_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">smtp_tls_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">legacy_fo_indexes</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">version</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">for</span> <span class="n">legacy_index_name</span> <span class="ow">in</span> <span class="n">legacy_fo_indexes</span> <span class="ow">or</span> <span class="p">[]:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">legacy_index</span> <span class="o">=</span> <span class="n">Index</span><span class="p">(</span><span class="n">legacy_index_name</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">legacy_index</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">_legacy_fo_field_type</span><span class="p">(</span><span class="n">legacy_index</span><span class="p">)</span> <span class="o">!=</span> <span class="s2">&quot;long&quot;</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">new_index_name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2">-v</span><span class="si">{</span><span class="n">version</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="c1"># Nested object form rather than the dotted key this used to</span>
<span class="c1"># send. Both are accepted and produce an identical mapping</span>
<span class="c1"># (verified against Elasticsearch 8.19 and OpenSearch 3), but</span>
<span class="c1"># dot expansion is conditional on the object field&#39;s</span>
<span class="c1"># `subobjects` setting, and this shape never is.</span>
<span class="n">properties</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">_LEGACY_FO_OBJECT</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;properties&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="n">_LEGACY_FO_LEAF</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;text&quot;</span><span class="p">,</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;keyword&quot;</span><span class="p">,</span> <span class="s2">&quot;ignore_above&quot;</span><span class="p">:</span> <span class="mi">256</span><span class="p">}</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Migrating </span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2"> to </span><span class="si">{</span><span class="n">new_index_name</span><span class="si">}</span><span class="s2">: &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_LEGACY_FO_FIELD</span><span class="si">}</span><span class="s2"> is mapped as long, as parsedmarc &quot;</span>
<span class="s2">&quot;releases before 5.0.0 declared it&quot;</span>
<span class="p">)</span>
<span class="c1"># Reaching here means the original index still holds the data:</span>
<span class="c1"># it is deleted only after the reindex below succeeds. So a</span>
<span class="c1"># leftover target index is the debris of an earlier attempt</span>
<span class="c1"># that died between create() and delete(), and keeping it would</span>
<span class="c1"># fail every later attempt on &quot;resource already exists&quot;.</span>
<span class="k">if</span> <span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Discarding </span><span class="si">{</span><span class="n">new_index_name</span><span class="si">}</span><span class="s2"> left behind by an earlier &quot;</span>
<span class="sa">f</span><span class="s2">&quot;interrupted migration of </span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">put_mapping</span><span class="p">(</span><span class="n">properties</span><span class="o">=</span><span class="n">properties</span><span class="p">)</span>
<span class="n">reindex</span><span class="p">(</span><span class="n">connections</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(),</span> <span class="n">legacy_index_name</span><span class="p">,</span> <span class="n">new_index_name</span><span class="p">)</span>
<span class="n">Index</span><span class="p">(</span><span class="n">legacy_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="s2">&quot;Failed the legacy published_policy.fo migration for &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. This will be retried at the &quot;</span>
<span class="s2">&quot;next startup.&quot;</span>
<span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aggregate_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">smtp_tls_indexes</span><span class="p">:</span>
<span class="k">return</span>
_modules/parsedmarc/opensearch.html
+101 -34
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.opensearch &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc.opensearch &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
@@ -693,20 +693,61 @@
<span class="n">_LEGACY_FO_FIELD</span> <span class="o">=</span> <span class="s2">&quot;published_policy.fo&quot;</span>
<span class="c1"># The same field split into the object/leaf names a mapping body nests it</span>
<span class="c1"># under. Derived from the dotted name so the mapping written below cannot</span>
<span class="c1"># drift from the field _legacy_fo_field_type() reads.</span>
<span class="n">_LEGACY_FO_OBJECT</span><span class="p">,</span> <span class="n">_LEGACY_FO_LEAF</span> <span class="o">=</span> <span class="n">_LEGACY_FO_FIELD</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_legacy_fo_field_type</span><span class="p">(</span><span class="n">index</span><span class="p">:</span> <span class="n">Index</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Return the mapped type of ``published_policy.fo`` in *index*.</span>
<span class="sd"> Returns ``None`` when the index does not map the field at all.</span>
<span class="sd"> Elasticsearch 6-era clusters keyed field mappings by the mapping type</span>
<span class="sd"> name (``doc``); mapping types are gone from both OpenSearch and</span>
<span class="sd"> Elasticsearch 8, whose responses put the field directly under</span>
<span class="sd"> ``mappings``. The type-keyed shape is only descended into when such a</span>
<span class="sd"> key is actually present, so this reads either shape.</span>
<span class="sd"> Args:</span>
<span class="sd"> index (Index): The index to inspect.</span>
<span class="sd"> Returns:</span>
<span class="sd"> str | None: The mapped field type, e.g. ``&quot;long&quot;`` or ``&quot;text&quot;``.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">index</span><span class="o">.</span><span class="n">get_field_mapping</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="n">_LEGACY_FO_FIELD</span><span class="p">])</span>
<span class="n">mappings</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="nb">list</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]][</span><span class="s2">&quot;mappings&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="n">_LEGACY_FO_FIELD</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">mappings</span><span class="p">:</span>
<span class="k">for</span> <span class="n">type_name</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;doc&quot;</span><span class="p">,</span> <span class="s2">&quot;_doc&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="n">type_name</span> <span class="ow">in</span> <span class="n">mappings</span><span class="p">:</span>
<span class="n">mappings</span> <span class="o">=</span> <span class="n">mappings</span><span class="p">[</span><span class="n">type_name</span><span class="p">]</span>
<span class="k">break</span>
<span class="n">field_mapping</span> <span class="o">=</span> <span class="n">mappings</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">_LEGACY_FO_FIELD</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">field_mapping</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span>
<span class="k">return</span> <span class="n">field_mapping</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;mapping&quot;</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">_LEGACY_FO_LEAF</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;type&quot;</span><span class="p">)</span>
<div class="viewcode-block" id="migrate_indexes">
<a class="viewcode-back" href="../../api.html#parsedmarc.opensearch.migrate_indexes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">migrate_indexes</span><span class="p">(</span>
<span class="n">aggregate_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">failure_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">smtp_tls_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">legacy_fo_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Runs index migrations and backfills.</span>
<span class="sd"> First, the legacy ``published_policy.fo`` migration: indexes where that</span>
<span class="sd"> field was mapped as ``long`` (data indexed by very old parsedmarc</span>
<span class="sd"> releases under the Elasticsearch 6-era ``doc`` mapping type) are rebuilt</span>
<span class="sd"> as a ``-v2`` index with the text/keyword shape.</span>
<span class="sd"> First, the legacy ``published_policy.fo`` migration, for each name in</span>
<span class="sd"> ``legacy_fo_indexes``: parsedmarc releases before 5.0.0 declared that</span>
<span class="sd"> field as an integer, so those indexes mapped it as ``long``, which</span>
<span class="sd"> cannot hold the multi-value ``fo`` settings reports carry (``0:1``,</span>
<span class="sd"> ``d:s``). Such an index is rebuilt as a ``-v2`` index with the</span>
<span class="sd"> text/keyword shape, the documents are reindexed into it, and the</span>
<span class="sd"> original is deleted.</span>
<span class="sd"> Second, the ``dkim_results_combined``/``spf_results_combined`` backfill</span>
<span class="sd"> (added for issue #169) for aggregate report documents that were saved</span>
@@ -733,51 +774,77 @@
<span class="sd"> (accepted for API compatibility; no migrations are</span>
<span class="sd"> currently needed for failure indexes)</span>
<span class="sd"> smtp_tls_indexes (list): A list of SMTP TLS index names</span>
<span class="sd"> legacy_fo_indexes (list): A list of index names to check for the</span>
<span class="sd"> pre-5.0.0 ``published_policy.fo`` ``long`` mapping. Unlike the</span>
<span class="sd"> backfill arguments these are exact names, not patterns:</span>
<span class="sd"> 5.0.0 introduced date-suffixed index names in the same release</span>
<span class="sd"> that fixed the mapping, so an affected index has no date</span>
<span class="sd"> component. It may still be prefixed or suffixed -- both</span>
<span class="sd"> options date back to 4.1.0 -- so callers should pass the</span>
<span class="sd"> names their own ``index_prefix``/``index_suffix``</span>
<span class="sd"> configuration produces.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aggregate_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">smtp_tls_indexes</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aggregate_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">smtp_tls_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">legacy_fo_indexes</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">version</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">for</span> <span class="n">aggregate_index_name</span> <span class="ow">in</span> <span class="n">aggregate_indexes</span> <span class="ow">or</span> <span class="p">[]:</span>
<span class="k">for</span> <span class="n">legacy_index_name</span> <span class="ow">in</span> <span class="n">legacy_fo_indexes</span> <span class="ow">or</span> <span class="p">[]:</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">Index</span><span class="p">(</span><span class="n">aggregate_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">legacy_index</span> <span class="o">=</span> <span class="n">Index</span><span class="p">(</span><span class="n">legacy_index_name</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">legacy_index</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="k">continue</span>
<span class="n">aggregate_index</span> <span class="o">=</span> <span class="n">Index</span><span class="p">(</span><span class="n">aggregate_index_name</span><span class="p">)</span>
<span class="n">doc</span> <span class="o">=</span> <span class="s2">&quot;doc&quot;</span>
<span class="n">fo_field</span> <span class="o">=</span> <span class="s2">&quot;published_policy.fo&quot;</span>
<span class="n">fo</span> <span class="o">=</span> <span class="s2">&quot;fo&quot;</span>
<span class="n">fo_mapping</span> <span class="o">=</span> <span class="n">aggregate_index</span><span class="o">.</span><span class="n">get_field_mapping</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="n">fo_field</span><span class="p">])</span>
<span class="n">fo_mapping</span> <span class="o">=</span> <span class="n">fo_mapping</span><span class="p">[</span><span class="nb">list</span><span class="p">(</span><span class="n">fo_mapping</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]][</span><span class="s2">&quot;mappings&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="n">doc</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">fo_mapping</span><span class="p">:</span>
<span class="k">if</span> <span class="n">_legacy_fo_field_type</span><span class="p">(</span><span class="n">legacy_index</span><span class="p">)</span> <span class="o">!=</span> <span class="s2">&quot;long&quot;</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">fo_mapping</span> <span class="o">=</span> <span class="n">fo_mapping</span><span class="p">[</span><span class="n">doc</span><span class="p">][</span><span class="n">fo_field</span><span class="p">][</span><span class="s2">&quot;mapping&quot;</span><span class="p">][</span><span class="n">fo</span><span class="p">]</span>
<span class="n">fo_type</span> <span class="o">=</span> <span class="n">fo_mapping</span><span class="p">[</span><span class="s2">&quot;type&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="n">fo_type</span> <span class="o">==</span> <span class="s2">&quot;long&quot;</span><span class="p">:</span>
<span class="n">new_index_name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">aggregate_index_name</span><span class="si">}</span><span class="s2">-v</span><span class="si">{</span><span class="n">version</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">body</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;properties&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;published_policy.fo&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;text&quot;</span><span class="p">,</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;keyword&quot;</span><span class="p">,</span> <span class="s2">&quot;ignore_above&quot;</span><span class="p">:</span> <span class="mi">256</span><span class="p">}</span>
<span class="p">},</span>
<span class="n">new_index_name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2">-v</span><span class="si">{</span><span class="n">version</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="c1"># Nested object form rather than the dotted key this used to</span>
<span class="c1"># send. Both are accepted and produce an identical mapping</span>
<span class="c1"># (verified against Elasticsearch 8.19 and OpenSearch 3), but</span>
<span class="c1"># dot expansion is conditional on the object field&#39;s</span>
<span class="c1"># `subobjects` setting, and this shape never is.</span>
<span class="n">body</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;properties&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="n">_LEGACY_FO_OBJECT</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;properties&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="n">_LEGACY_FO_LEAF</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;text&quot;</span><span class="p">,</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;keyword&quot;</span><span class="p">,</span> <span class="s2">&quot;ignore_above&quot;</span><span class="p">:</span> <span class="mi">256</span><span class="p">}</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">put_mapping</span><span class="p">(</span><span class="n">doc_type</span><span class="o">=</span><span class="n">doc</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span>
<span class="n">reindex</span><span class="p">(</span>
<span class="n">connections</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(),</span> <span class="n">aggregate_index_name</span><span class="p">,</span> <span class="n">new_index_name</span>
<span class="p">}</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Migrating </span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2"> to </span><span class="si">{</span><span class="n">new_index_name</span><span class="si">}</span><span class="s2">: &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_LEGACY_FO_FIELD</span><span class="si">}</span><span class="s2"> is mapped as long, as parsedmarc &quot;</span>
<span class="s2">&quot;releases before 5.0.0 declared it&quot;</span>
<span class="p">)</span>
<span class="c1"># Reaching here means the original index still holds the data:</span>
<span class="c1"># it is deleted only after the reindex below succeeds. So a</span>
<span class="c1"># leftover target index is the debris of an earlier attempt</span>
<span class="c1"># that died between create() and delete(), and keeping it would</span>
<span class="c1"># fail every later attempt on &quot;resource already exists&quot;.</span>
<span class="k">if</span> <span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Discarding </span><span class="si">{</span><span class="n">new_index_name</span><span class="si">}</span><span class="s2"> left behind by an earlier &quot;</span>
<span class="sa">f</span><span class="s2">&quot;interrupted migration of </span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">Index</span><span class="p">(</span><span class="n">aggregate_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="n">Index</span><span class="p">(</span><span class="n">new_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">put_mapping</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span>
<span class="n">reindex</span><span class="p">(</span><span class="n">connections</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(),</span> <span class="n">legacy_index_name</span><span class="p">,</span> <span class="n">new_index_name</span><span class="p">)</span>
<span class="n">Index</span><span class="p">(</span><span class="n">legacy_index_name</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="s2">&quot;Failed the legacy published_policy.fo migration for &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">aggregate_index_name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. This will be retried at the &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">legacy_index_name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. This will be retried at the &quot;</span>
<span class="s2">&quot;next startup.&quot;</span>
<span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aggregate_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">smtp_tls_indexes</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">connections</span><span class="o">.</span><span class="n">get_connection</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
_modules/parsedmarc/splunk.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.splunk &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc.splunk &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/types.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.types &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc.types &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_modules/parsedmarc/utils.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.utils &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc.utils &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=2f7da7c1"></script>
<script src="../../_static/documentation_options.js?v=8b0523d8"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
_sources/elasticsearch.md.txt
+23 -1
View File
@@ -247,6 +247,25 @@ OpenSearch. Any error talking to the cluster (for example, no indexes yet
on a fresh install) is logged as a warning and retried on the next startup,
rather than aborting parsedmarc.
Which index patterns it targets follows the ones parsedmarc writes to, and
is logged at debug level on startup:
- With `index_prefix_domain_map` configured in `[general]` and no
`index_prefix` set, every tenant prefix in the map gets its own index
pattern, alongside the unprefixed one — aggregate and failure reports for
a domain that is not in the map are still saved without a prefix. A
configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded
tenant is covered without a restart.
- With an `index_prefix` set in `[elasticsearch]`/`[opensearch]`, only that
prefix is targeted, and the map is not consulted. That is deliberate: such
a deployment writes only under its own prefix, and an `_update_by_query`
against a pattern it does not write to could reach another deployment's
data on a shared cluster.
- With an `index_suffix` set, both the suffixed and the unsuffixed pattern
are targeted, so documents indexed before the suffix was configured are
backfilled too. Note that the unsuffixed pattern also matches any *other*
suffix on the same cluster.
If you upgrade the dashboards without pointing the new parsedmarc version
at the cluster, or you'd rather control when the write load happens, you
can still run the backfill manually. It is idempotent (documents that
@@ -314,7 +333,10 @@ curl -X POST "http://localhost:9200/dmarc_aggregate*/_update_by_query?conflicts=
`wait_for_completion=false` returns a task ID — check progress with
`GET _tasks/<task id>`. Adjust the index pattern if you use a custom
`index_prefix`/`index_suffix`. After backfilling, re-import the updated
`index_prefix`/`index_suffix`; with `index_prefix_domain_map`, run the
command once per tenant prefix (`acme_corp_dmarc_aggregate*`) plus once for
the unprefixed pattern, or widen it to `*dmarc_aggregate*` to cover every
tenant in one pass. After backfilling, re-import the updated
dashboards ndjson (the index pattern saved object changed too) per the
import instructions above.
_sources/usage.md.txt
+7 -2
View File
@@ -127,7 +127,8 @@ The full set of configuration options are:
Elasticsearch, Splunk and/or S3
- `save_smtp_tls` - bool: Save SMTP-STS report data to
Elasticsearch, Splunk and/or S3
- `index_prefix_domain_map` - bool: A path mapping of Opensearch/Elasticsearch index prefixes to domain names
- `index_prefix_domain_map` - str: Path to a YAML file mapping
OpenSearch/Elasticsearch index prefixes to domain names
- `strip_attachment_payloads` - bool: Remove attachment
payloads from results
- `silent` - bool: Set this to `False` to output results to STDOUT
@@ -1167,12 +1168,16 @@ whalensolutions:
Save it to disk where the user running ParseDMARC can read it, then set `index_prefix_domain_map` to that filepath in the `[general]` section of the ParseDMARC configuration file and do not set an `index_prefix` option in the `[elasticsearch]` or `[opensearch]` sections.
When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of Opensearch or the ELK stack to only grant users access to the indexes that they need.
When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
:::{note}
A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
:::
Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See [Backfilling the combined DKIM/SPF result fields](elasticsearch.md#backfilling-the-combined-dkimspf-result-fields) for the details.
## Running parsedmarc as a systemd service
Use systemd to run `parsedmarc` as a service and process reports as
_static/documentation_options.js
+1 -1
View File
@@ -1,5 +1,5 @@
const DOCUMENTATION_OPTIONS = {
VERSION: '10.4.0',
VERSION: '10.4.1',
LANGUAGE: 'en',
COLLAPSE_INDEX: false,
BUILDER: 'html',
api.html
+42 -12
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>API reference &mdash; parsedmarc 10.4.0 documentation</title>
<title>API reference &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -1283,11 +1283,20 @@ remaining keys are passed through; defaults are skipped entirely.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="parsedmarc.elastic.migrate_indexes">
<span class="sig-prename descclassname"><span class="pre">parsedmarc.elastic.</span></span><span class="sig-name descname"><span class="pre">migrate_indexes</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">aggregate_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">failure_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">smtp_tls_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/parsedmarc/elastic.html#migrate_indexes"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#parsedmarc.elastic.migrate_indexes" title="Link to this definition"></a></dt>
<dd><p>Backfills the <code class="docutils literal notranslate"><span class="pre">dkim_results_combined</span></code>/<code class="docutils literal notranslate"><span class="pre">spf_results_combined</span></code> fields
(added for issue #169) on aggregate report documents, and the
<code class="docutils literal notranslate"><span class="pre">policies_combined</span></code>/<code class="docutils literal notranslate"><span class="pre">failure_details_combined</span></code> fields on SMTP TLS
report documents, that were saved before those fields existed.</p>
<span class="sig-prename descclassname"><span class="pre">parsedmarc.elastic.</span></span><span class="sig-name descname"><span class="pre">migrate_indexes</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">aggregate_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">failure_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">smtp_tls_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">legacy_fo_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/parsedmarc/elastic.html#migrate_indexes"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#parsedmarc.elastic.migrate_indexes" title="Link to this definition"></a></dt>
<dd><p>Runs index migrations and backfills.</p>
<p>First, the legacy <code class="docutils literal notranslate"><span class="pre">published_policy.fo</span></code> migration, for each name in
<code class="docutils literal notranslate"><span class="pre">legacy_fo_indexes</span></code>: parsedmarc releases before 5.0.0 declared that
field as an integer, so those indexes mapped it as <code class="docutils literal notranslate"><span class="pre">long</span></code>, which
cannot hold the multi-value <code class="docutils literal notranslate"><span class="pre">fo</span></code> settings reports carry (<code class="docutils literal notranslate"><span class="pre">0:1</span></code>,
<code class="docutils literal notranslate"><span class="pre">d:s</span></code>). Such an index is rebuilt as a <code class="docutils literal notranslate"><span class="pre">-v2</span></code> index with the
text/keyword shape, the documents are reindexed into it, and the
original is deleted.</p>
<p>Second, it backfills the <code class="docutils literal notranslate"><span class="pre">dkim_results_combined</span></code>/
<code class="docutils literal notranslate"><span class="pre">spf_results_combined</span></code> fields (added for issue #169) on aggregate
report documents, and the <code class="docutils literal notranslate"><span class="pre">policies_combined</span></code>/
<code class="docutils literal notranslate"><span class="pre">failure_details_combined</span></code> fields on SMTP TLS report documents, that
were saved before those fields existed.</p>
<p>For each name in <code class="docutils literal notranslate"><span class="pre">aggregate_indexes</span></code>/<code class="docutils literal notranslate"><span class="pre">smtp_tls_indexes</span></code>, this
submits an <code class="docutils literal notranslate"><span class="pre">update_by_query</span></code> against the <code class="docutils literal notranslate"><span class="pre">f&quot;{name}*&quot;</span></code> index pattern
(the real indexes are date-suffixed) as a non-blocking background task
@@ -1307,6 +1316,15 @@ next startup, and the manual <code class="docutils literal notranslate"><span cl
<li><p><strong>failure_indexes</strong> (<em>list</em>) – A list of failure index names
(accepted for API compatibility; unused)</p></li>
<li><p><strong>smtp_tls_indexes</strong> (<em>list</em>) – A list of SMTP TLS index names</p></li>
<li><p><strong>legacy_fo_indexes</strong> (<em>list</em>) – A list of index names to check for the
pre-5.0.0 <code class="docutils literal notranslate"><span class="pre">published_policy.fo</span></code> <code class="docutils literal notranslate"><span class="pre">long</span></code> mapping. Unlike the
backfill arguments these are exact names, not patterns:
5.0.0 introduced date-suffixed index names in the same release
that fixed the mapping, so an affected index has no date
component. It may still be prefixed or suffixed – both
options date back to 4.1.0 – so callers should pass the
names their own <code class="docutils literal notranslate"><span class="pre">index_prefix</span></code>/<code class="docutils literal notranslate"><span class="pre">index_suffix</span></code>
configuration produces.</p></li>
</ul>
</dd>
</dl>
@@ -1457,12 +1475,15 @@ any other settings through unchanged.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="parsedmarc.opensearch.migrate_indexes">
<span class="sig-prename descclassname"><span class="pre">parsedmarc.opensearch.</span></span><span class="sig-name descname"><span class="pre">migrate_indexes</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">aggregate_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">failure_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">smtp_tls_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/parsedmarc/opensearch.html#migrate_indexes"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#parsedmarc.opensearch.migrate_indexes" title="Link to this definition"></a></dt>
<span class="sig-prename descclassname"><span class="pre">parsedmarc.opensearch.</span></span><span class="sig-name descname"><span class="pre">migrate_indexes</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">aggregate_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">failure_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">smtp_tls_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">legacy_fo_indexes</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/parsedmarc/opensearch.html#migrate_indexes"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#parsedmarc.opensearch.migrate_indexes" title="Link to this definition"></a></dt>
<dd><p>Runs index migrations and backfills.</p>
<p>First, the legacy <code class="docutils literal notranslate"><span class="pre">published_policy.fo</span></code> migration: indexes where that
field was mapped as <code class="docutils literal notranslate"><span class="pre">long</span></code> (data indexed by very old parsedmarc
releases under the Elasticsearch 6-era <code class="docutils literal notranslate"><span class="pre">doc</span></code> mapping type) are rebuilt
as a <code class="docutils literal notranslate"><span class="pre">-v2</span></code> index with the text/keyword shape.</p>
<p>First, the legacy <code class="docutils literal notranslate"><span class="pre">published_policy.fo</span></code> migration, for each name in
<code class="docutils literal notranslate"><span class="pre">legacy_fo_indexes</span></code>: parsedmarc releases before 5.0.0 declared that
field as an integer, so those indexes mapped it as <code class="docutils literal notranslate"><span class="pre">long</span></code>, which
cannot hold the multi-value <code class="docutils literal notranslate"><span class="pre">fo</span></code> settings reports carry (<code class="docutils literal notranslate"><span class="pre">0:1</span></code>,
<code class="docutils literal notranslate"><span class="pre">d:s</span></code>). Such an index is rebuilt as a <code class="docutils literal notranslate"><span class="pre">-v2</span></code> index with the
text/keyword shape, the documents are reindexed into it, and the
original is deleted.</p>
<p>Second, the <code class="docutils literal notranslate"><span class="pre">dkim_results_combined</span></code>/<code class="docutils literal notranslate"><span class="pre">spf_results_combined</span></code> backfill
(added for issue #169) for aggregate report documents that were saved
before those fields existed. For each name in <code class="docutils literal notranslate"><span class="pre">aggregate_indexes</span></code>,
@@ -1488,6 +1509,15 @@ report documents, for each name in <code class="docutils literal notranslate"><s
(accepted for API compatibility; no migrations are
currently needed for failure indexes)</p></li>
<li><p><strong>smtp_tls_indexes</strong> (<em>list</em>) – A list of SMTP TLS index names</p></li>
<li><p><strong>legacy_fo_indexes</strong> (<em>list</em>) – A list of index names to check for the
pre-5.0.0 <code class="docutils literal notranslate"><span class="pre">published_policy.fo</span></code> <code class="docutils literal notranslate"><span class="pre">long</span></code> mapping. Unlike the
backfill arguments these are exact names, not patterns:
5.0.0 introduced date-suffixed index names in the same release
that fixed the mapping, so an affected index has no date
component. It may still be prefixed or suffixed – both
options date back to 4.1.0 – so callers should pass the
names their own <code class="docutils literal notranslate"><span class="pre">index_prefix</span></code>/<code class="docutils literal notranslate"><span class="pre">index_suffix</span></code>
configuration produces.</p></li>
</ul>
</dd>
</dl>
contributing.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Contributing to parsedmarc &mdash; parsedmarc 10.4.0 documentation</title>
<title>Contributing to parsedmarc &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
davmail.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 10.4.0 documentation</title>
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
dmarc.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Understanding DMARC &mdash; parsedmarc 10.4.0 documentation</title>
<title>Understanding DMARC &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
elasticsearch.html
+25 -3
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Elasticsearch and Kibana &mdash; parsedmarc 10.4.0 documentation</title>
<title>Elasticsearch and Kibana &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -284,6 +284,25 @@ count of 0 and log nothing further — and it works the same way on
OpenSearch. Any error talking to the cluster (for example, no indexes yet
on a fresh install) is logged as a warning and retried on the next startup,
rather than aborting parsedmarc.</p>
<p>Which index patterns it targets follows the ones parsedmarc writes to, and
is logged at debug level on startup:</p>
<ul class="simple">
<li><p>With <code class="docutils literal notranslate"><span class="pre">index_prefix_domain_map</span></code> configured in <code class="docutils literal notranslate"><span class="pre">[general]</span></code> and no
<code class="docutils literal notranslate"><span class="pre">index_prefix</span></code> set, every tenant prefix in the map gets its own index
pattern, alongside the unprefixed one — aggregate and failure reports for
a domain that is not in the map are still saved without a prefix. A
configuration reload (<code class="docutils literal notranslate"><span class="pre">SIGHUP</span></code>) re-reads the map, so a newly onboarded
tenant is covered without a restart.</p></li>
<li><p>With an <code class="docutils literal notranslate"><span class="pre">index_prefix</span></code> set in <code class="docutils literal notranslate"><span class="pre">[elasticsearch]</span></code>/<code class="docutils literal notranslate"><span class="pre">[opensearch]</span></code>, only that
prefix is targeted, and the map is not consulted. That is deliberate: such
a deployment writes only under its own prefix, and an <code class="docutils literal notranslate"><span class="pre">_update_by_query</span></code>
against a pattern it does not write to could reach another deployment’s
data on a shared cluster.</p></li>
<li><p>With an <code class="docutils literal notranslate"><span class="pre">index_suffix</span></code> set, both the suffixed and the unsuffixed pattern
are targeted, so documents indexed before the suffix was configured are
backfilled too. Note that the unsuffixed pattern also matches any <em>other</em>
suffix on the same cluster.</p></li>
</ul>
<p>If you upgrade the dashboards without pointing the new parsedmarc version
at the cluster, or you’d rather control when the write load happens, you
can still run the backfill manually. It is idempotent (documents that
@@ -349,7 +368,10 @@ no backfillable document is skipped.</p>
</div>
<p><code class="docutils literal notranslate"><span class="pre">wait_for_completion=false</span></code> returns a task ID — check progress with
<code class="docutils literal notranslate"><span class="pre">GET</span> <span class="pre">_tasks/&lt;task</span> <span class="pre">id&gt;</span></code>. Adjust the index pattern if you use a custom
<code class="docutils literal notranslate"><span class="pre">index_prefix</span></code>/<code class="docutils literal notranslate"><span class="pre">index_suffix</span></code>. After backfilling, re-import the updated
<code class="docutils literal notranslate"><span class="pre">index_prefix</span></code>/<code class="docutils literal notranslate"><span class="pre">index_suffix</span></code>; with <code class="docutils literal notranslate"><span class="pre">index_prefix_domain_map</span></code>, run the
command once per tenant prefix (<code class="docutils literal notranslate"><span class="pre">acme_corp_dmarc_aggregate*</span></code>) plus once for
the unprefixed pattern, or widen it to <code class="docutils literal notranslate"><span class="pre">*dmarc_aggregate*</span></code> to cover every
tenant in one pass. After backfilling, re-import the updated
dashboards ndjson (the index pattern saved object changed too) per the
import instructions above.</p>
<p>SMTP TLS documents have the same class of defect one level deeper:
genindex.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Index &mdash; parsedmarc 10.4.0 documentation</title>
<title>Index &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
index.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 10.4.0 documentation</title>
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
installation.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Installation &mdash; parsedmarc 10.4.0 documentation</title>
<title>Installation &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
kibana.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using the Kibana dashboards &mdash; parsedmarc 10.4.0 documentation</title>
<title>Using the Kibana dashboards &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
mailing-lists.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>What about mailing lists? &mdash; parsedmarc 10.4.0 documentation</title>
<title>What about mailing lists? &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
objects.inv
BIN
View File
Binary file not shown.
opensearch.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>OpenSearch and Grafana &mdash; parsedmarc 10.4.0 documentation</title>
<title>OpenSearch and Grafana &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
output.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Sample outputs &mdash; parsedmarc 10.4.0 documentation</title>
<title>Sample outputs &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
py-modindex.html
+2 -2
View File
@@ -5,14 +5,14 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Python Module Index &mdash; parsedmarc 10.4.0 documentation</title>
<title>Python Module Index &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
search.html
+2 -2
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Search &mdash; parsedmarc 10.4.0 documentation</title>
<title>Search &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
@@ -13,7 +13,7 @@
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
splunk.html
+2 -2
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Splunk &mdash; parsedmarc 10.4.0 documentation</title>
<title>Splunk &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
usage.html
+7 -4
View File
@@ -6,14 +6,14 @@
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using parsedmarc &mdash; parsedmarc 10.4.0 documentation</title>
<title>Using parsedmarc &mdash; parsedmarc 10.4.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=9edc463e" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=2f7da7c1"></script>
<script src="_static/documentation_options.js?v=8b0523d8"></script>
<script src="_static/doctools.js?v=fd6eb6e6"></script>
<script src="_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="_static/js/theme.js"></script>
@@ -230,7 +230,8 @@ Elasticsearch, Splunk and/or S3</p></li>
Elasticsearch, Splunk and/or S3</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">save_smtp_tls</span></code> - bool: Save SMTP-STS report data to
Elasticsearch, Splunk and/or S3</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">index_prefix_domain_map</span></code> - bool: A path mapping of Opensearch/Elasticsearch index prefixes to domain names</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">index_prefix_domain_map</span></code> - str: Path to a YAML file mapping
OpenSearch/Elasticsearch index prefixes to domain names</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">strip_attachment_payloads</span></code> - bool: Remove attachment
payloads from results</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">silent</span></code> - bool: Set this to <code class="docutils literal notranslate"><span class="pre">False</span></code> to output results to STDOUT</p></li>
@@ -1377,11 +1378,13 @@ high-volume mailbox processing.</p>
</pre></div>
</div>
<p>Save it to disk where the user running ParseDMARC can read it, then set <code class="docutils literal notranslate"><span class="pre">index_prefix_domain_map</span></code> to that filepath in the <code class="docutils literal notranslate"><span class="pre">[general]</span></code> section of the ParseDMARC configuration file and do not set an <code class="docutils literal notranslate"><span class="pre">index_prefix</span></code> option in the <code class="docutils literal notranslate"><span class="pre">[elasticsearch]</span></code> or <code class="docutils literal notranslate"><span class="pre">[opensearch]</span></code> sections.</p>
<p>When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of Opensearch or the ELK stack to only grant users access to the indexes that they need.</p>
<p>When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.</p>
</div>
<p>Each key must be a tenant name and each value a <em>list</em> of domain names, all strings; a file of any other shape is rejected at startup.</p>
<p>The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (<code class="docutils literal notranslate"><span class="pre">SIGHUP</span></code>) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See <a class="reference internal" href="elasticsearch.html#backfilling-the-combined-dkim-spf-result-fields"><span class="std std-ref">Backfilling the combined DKIM/SPF result fields</span></a> for the details.</p>
</section>
<section id="running-parsedmarc-as-a-systemd-service">
<h2>Running parsedmarc as a systemd service<a class="headerlink" href="#running-parsedmarc-as-a-systemd-service" title="Link to this heading"></a></h2>