amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-09-18 20:07:57 +00:00
Code Issues Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Sean Whalen
2026-07-21 21:20:15 -04:00
parent d3dee042cc
commit f7518b43d2
11 changed files with 231 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_modules/parsedmarc.html
+17 -7
View File
@@ -277,6 +277,15 @@
<span class="k">return</span> <span class="n">value</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_normalize_result_word</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Any</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Lowercase a reporter-supplied enum word; RFC 7489 Appendix C and</span>
<span class="sd"> RFC 9990 define result/disposition types as lowercase tokens. Non-string</span>
<span class="sd"> values (e.g. xmltodict dicts from attribute-bearing elements) pass</span>
<span class="sd"> through unchanged.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">value</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="n">value</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_bucket_interval_by_day</span><span class="p">(</span>
<span class="n">begin</span><span class="p">:</span> <span class="n">datetime</span><span class="p">,</span>
<span class="n">end</span><span class="p">:</span> <span class="n">datetime</span><span class="p">,</span>
@@ -518,11 +527,12 @@
<span class="s2">&quot;policy_override_reasons&quot;</span><span class="p">:</span> <span class="p">[],</span>
<span class="p">}</span>
<span class="k">if</span> <span class="s2">&quot;disposition&quot;</span> <span class="ow">in</span> <span class="n">policy_evaluated</span><span class="p">:</span>
<span class="n">new_policy_evaluated</span><span class="p">[</span><span class="s2">&quot;disposition&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">policy_evaluated</span><span class="p">[</span><span class="s2">&quot;disposition&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;dkim&quot;</span> <span class="ow">in</span> <span class="n">policy_evaluated</span><span class="p">:</span>
<span class="n">new_policy_evaluated</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">policy_evaluated</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;spf&quot;</span> <span class="ow">in</span> <span class="n">policy_evaluated</span><span class="p">:</span>
<span class="n">new_policy_evaluated</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">policy_evaluated</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">]</span>
<span class="n">new_policy_evaluated</span><span class="p">[</span><span class="s2">&quot;disposition&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">_normalize_result_word</span><span class="p">(</span>
<span class="n">policy_evaluated</span><span class="p">[</span><span class="s2">&quot;disposition&quot;</span><span class="p">]</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;dkim&quot;</span><span class="p">,</span> <span class="s2">&quot;spf&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">policy_evaluated</span><span class="p">:</span>
<span class="n">new_policy_evaluated</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">_normalize_result_word</span><span class="p">(</span><span class="n">policy_evaluated</span><span class="p">[</span><span class="n">key</span><span class="p">])</span>
<span class="n">reasons</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">spf_aligned</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">policy_evaluated</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
@@ -601,7 +611,7 @@
<span class="p">)</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;selector&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot;none&quot;</span>
<span class="k">if</span> <span class="s2">&quot;result&quot;</span> <span class="ow">in</span> <span class="n">result</span> <span class="ow">and</span> <span class="n">result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">_normalize_result_word</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot;none&quot;</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;human_result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">_text</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;human_result&quot;</span><span class="p">))</span>
@@ -617,7 +627,7 @@
<span class="k">else</span><span class="p">:</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;scope&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot;mfrom&quot;</span>
<span class="k">if</span> <span class="s2">&quot;result&quot;</span> <span class="ow">in</span> <span class="n">result</span> <span class="ow">and</span> <span class="n">result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">_normalize_result_word</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot;none&quot;</span>
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;human_result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">_text</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;human_result&quot;</span><span class="p">))</span>
_modules/parsedmarc/elastic.html
+26 -3
View File
@@ -184,7 +184,7 @@
<span class="n">domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">scope</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">human_result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
@@ -231,6 +231,18 @@
<span class="n">envelope_to</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">dkim_results</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_DKIMResult</span><span class="p">)</span>
<span class="n">spf_results</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_SPFResult</span><span class="p">)</span>
<span class="c1"># One &quot;{selector} / {domain} / {result}&quot; (DKIM) or &quot;{scope} / {domain} /</span>
<span class="c1"># {result}&quot; (SPF) string per auth result. Kibana/Grafana tables cannot</span>
<span class="c1"># terms-aggregate the subfields of an object array without producing a</span>
<span class="c1"># cross-product of values (issue #169), so dashboards aggregate these</span>
<span class="c1"># composed keywords instead. Declared to match what dynamic mapping</span>
<span class="c1"># produces for a string array (text + .keyword).</span>
<span class="n">dkim_results_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">spf_results_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">np</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">testing</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">discovery_method</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
@@ -243,7 +255,7 @@
<span class="bp">self</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">selector</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="n">_DKIMResult</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">human_result</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dkim_results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
@@ -254,12 +266,15 @@
<span class="n">human_result</span><span class="o">=</span><span class="n">human_result</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dkim_results_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">{0}</span><span class="s2"> / </span><span class="si">{1}</span><span class="s2"> / </span><span class="si">{2}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">selector</span><span class="p">,</span> <span class="n">domain</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_spf_result</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">scope</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="n">_SPFResult</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">human_result</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spf_results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
@@ -270,6 +285,9 @@
<span class="n">human_result</span><span class="o">=</span><span class="n">human_result</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spf_results_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">{0}</span><span class="s2"> / </span><span class="si">{1}</span><span class="s2"> / </span><span class="si">{2}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">domain</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">save</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="c1"># pyright: ignore[reportIncompatibleMethodOverride]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">passed_dmarc</span> <span class="o">=</span> <span class="kc">False</span>
@@ -543,6 +561,11 @@
<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="n">Index</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Deliberately no Index.document() registration: Kibana/OpenSearch</span>
<span class="c1"># Dashboards/Grafana cannot terms-aggregate fields inside a</span>
<span class="c1"># `nested` mapping, so the dynamic `object` mapping produced by a</span>
<span class="c1"># bare create is load-bearing for the shipped dashboards (issue</span>
<span class="c1"># #169; see the *_combined fields on _AggregateReportDoc).</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">index</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Creating Elasticsearch index: </span><span class="si">{0}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">))</span>
<span class="k">if</span> <span class="n">effective_settings</span><span class="p">:</span>
_modules/parsedmarc/opensearch.html
+26 -3
View File
@@ -147,7 +147,7 @@
<span class="k">class</span><span class="w"> </span><span class="nc">_SPFResult</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">scope</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">human_result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
@@ -188,6 +188,18 @@
<span class="n">envelope_to</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">dkim_results</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_DKIMResult</span><span class="p">)</span>
<span class="n">spf_results</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_SPFResult</span><span class="p">)</span>
<span class="c1"># One &quot;{selector} / {domain} / {result}&quot; (DKIM) or &quot;{scope} / {domain} /</span>
<span class="c1"># {result}&quot; (SPF) string per auth result. Kibana/Grafana tables cannot</span>
<span class="c1"># terms-aggregate the subfields of an object array without producing a</span>
<span class="c1"># cross-product of values (issue #169), so dashboards aggregate these</span>
<span class="c1"># composed keywords instead. Declared to match what dynamic mapping</span>
<span class="c1"># produces for a string array (text + .keyword).</span>
<span class="n">dkim_results_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">spf_results_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">np</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">testing</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">discovery_method</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
@@ -200,7 +212,7 @@
<span class="bp">self</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">selector</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="n">_DKIMResult</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">human_result</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dkim_results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
@@ -211,12 +223,15 @@
<span class="n">human_result</span><span class="o">=</span><span class="n">human_result</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dkim_results_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">{0}</span><span class="s2"> / </span><span class="si">{1}</span><span class="s2"> / </span><span class="si">{2}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">selector</span><span class="p">,</span> <span class="n">domain</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_spf_result</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">scope</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="n">_SPFResult</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">human_result</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spf_results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
@@ -227,6 +242,9 @@
<span class="n">human_result</span><span class="o">=</span><span class="n">human_result</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spf_results_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">{0}</span><span class="s2"> / </span><span class="si">{1}</span><span class="s2"> / </span><span class="si">{2}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">domain</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">save</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="c1"># pyright: ignore[reportIncompatibleMethodOverride]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">passed_dmarc</span> <span class="o">=</span> <span class="kc">False</span>
@@ -460,6 +478,11 @@
<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="n">Index</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Deliberately no Index.document() registration: Kibana/OpenSearch</span>
<span class="c1"># Dashboards/Grafana cannot terms-aggregate fields inside a</span>
<span class="c1"># `nested` mapping, so the dynamic `object` mapping produced by a</span>
<span class="c1"># bare create is load-bearing for the shipped dashboards (issue</span>
<span class="c1"># #169; see the *_combined fields on _AggregateReportDoc).</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">index</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Creating OpenSearch index: </span><span class="si">{0}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">))</span>
<span class="k">if</span> <span class="n">settings</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
_sources/elasticsearch.md.txt
+54
View File
@@ -226,6 +226,60 @@ Kibana index patterns with versions that match the upgraded indexes:
7. Import `export.ndjson` by clicking Import from the Kibana
Saved Objects page
## Backfilling the combined DKIM/SPF result fields
As of the version fixing [#169](https://github.com/domainaware/parsedmarc/issues/169),
aggregate documents include `dkim_results_combined` and `spf_results_combined` —
scalar string arrays that keep each auth result's selector/scope, domain, and
result paired, which the dashboards' alignment-detail tables aggregate on.
Reports saved by older versions lack these fields and will not appear in
those tables.
Running the following once per cluster backfills the fields on existing
documents. It is idempotent (documents that already have the fields are
skipped), so it is safe to re-run. It works identically on OpenSearch;
just adjust the URL and credentials. The query matches only documents
that have at least one DKIM or SPF auth result and lack the corresponding
combined field; documents with no auth results are skipped, because an
`exists` query cannot see an empty array, and for search purposes an
empty `dkim_results_combined` is identical to an absent one.
```bash
curl -X POST "http://localhost:9200/dmarc_aggregate*/_update_by_query?conflicts=proceed&wait_for_completion=false" \
-H "Content-Type: application/json" -d '
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"bool": {
"must": [{"exists": {"field": "dkim_results.domain"}}],
"must_not": [{"exists": {"field": "dkim_results_combined"}}]
}
},
{
"bool": {
"must": [{"exists": {"field": "spf_results.domain"}}],
"must_not": [{"exists": {"field": "spf_results_combined"}}]
}
}
]
}
},
"script": {
"lang": "painless",
"source": "List dk = new ArrayList(); def dr = ctx._source.dkim_results; if (dr != null) { if (!(dr instanceof List)) { dr = [dr]; } for (e in dr) { if (e == null) { continue; } def sel = e.selector != null ? e.selector : \"none\"; def dom = e.domain != null ? e.domain : \"none\"; def res = e.result != null ? e.result : \"none\"; dk.add(sel + \" / \" + dom + \" / \" + res); } } ctx._source.dkim_results_combined = dk; List sp = new ArrayList(); def sr = ctx._source.spf_results; if (sr != null) { if (!(sr instanceof List)) { sr = [sr]; } for (e in sr) { if (e == null) { continue; } def sc = e.scope != null ? e.scope : \"mfrom\"; def dom = e.domain != null ? e.domain : \"none\"; def res = e.result != null ? e.result : (e.results != null ? e.results : \"none\"); sp.add(sc + \" / \" + dom + \" / \" + res); } } ctx._source.spf_results_combined = sp;"
}
}'
```
`wait_for_completion=false` returns a task ID — check progress with
`GET _tasks/<task id>`. Adjust the index pattern if you use a custom
`index_prefix`/`index_suffix`. After backfilling, re-import the updated
dashboards ndjson (the index pattern saved object changed too) per the
import instructions above.
## Records retention
Starting in version 5.0.0, `parsedmarc` stores data in a separate
_sources/kibana.md.txt
+23 -8
View File
@@ -44,18 +44,29 @@ disposition over time.
Under the graphs you will find the most useful data tables on the dashboard. On
the left, there is a list of organizations that are sending you DMARC reports.
In the center, there is a list of sending servers grouped by the base domain
in their reverse DNS. On the right, there is a list of email from domains,
sorted by message volume.
in their reverse DNS. On the right, there is the "Message volume and DMARC
compliance by from domain" table, which lists email from domains with their
message volume and a percentage of those messages that passed DMARC.
By hovering your mouse over a data table value and using the magnifying glass
icons, you can filter on our filter out different values. Start by looking at
icons, you can filter on or filter out different values. Start by looking at
the Message Sources by Reverse DNS table. Find a sender that you recognize,
such as an email marketing service, hover over it, and click on the plus (+)
magnifying glass icon, to add a filter that only shows results for that sender.
Now, look at the Message From Header table to the right. That shows you the
domains that a sender is sending as, which might tell you which brand/business
is using a particular service. With that information, you can contact them and
have them set up DKIM.
Now, look at the Message volume and DMARC compliance by from domain table to
the right. That shows you the domains that a sender is sending as, and what
share of that traffic is passing DMARC, which might tell you which
brand/business is using a particular service. With that information, you can
contact them and have them set up DKIM.
:::{note}
The "Message volume and DMARC compliance by from domain" table is a TSVB
visualization, used because per-domain compliance percentages require a
Filter Ratio metric that agg-based data tables can't compute. It renders
correctly on Kibana 8.x as imported, but *editing* it requires first enabling
the `metrics:allowStringIndices` advanced setting, since it references the
`dmarc_aggregate*` index as a string pattern, which Elastic has deprecated.
:::
:::{note}
If you have a lot of B2C customers, you may see a high volume of emails as
@@ -71,7 +82,11 @@ Further down the dashboard, you can filter by source country or source IP
address.
Tables showing SPF and DKIM alignment details are located under the IP address
table.
table. Each row of the DKIM details table is one real DKIM signature, shown
as a combined `selector / domain / result` value; the SPF details table
shows `scope / domain / result` the same way. Combining the values into one
column keeps each signature's selector, domain, and result paired together,
rather than aggregating them as separate columns.
:::{note}
The alignment tables (SPF details, DKIM details) and the per-IP source
_sources/usage.md.txt
+5
View File
@@ -527,6 +527,11 @@ The full set of configuration options are:
- `aggregate_topic` - str: The Kafka topic for aggregate reports
- `failure_topic` - str: The Kafka topic for failure reports
- `smtp`
The results email is only sent when at least one aggregate, failure,
or SMTP TLS report was parsed during the run; an empty run (e.g. an
empty inbox) skips the email instead of sending headers-only CSVs.
- `host` - str: The SMTP hostname. Required unless `[msgraph]` is
configured, in which case omitting it sends the summary via
Microsoft Graph instead — see "Sending the summary email via
elasticsearch.html
+52
View File
@@ -50,6 +50,7 @@
<li class="toctree-l1 current"><a class="current reference internal" href="#">Elasticsearch and Kibana</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#installation">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="#upgrading-kibana-index-patterns">Upgrading Kibana index patterns</a></li>
<li class="toctree-l2"><a class="reference internal" href="#backfilling-the-combined-dkim-spf-result-fields">Backfilling the combined DKIM/SPF result fields</a></li>
<li class="toctree-l2"><a class="reference internal" href="#records-retention">Records retention</a></li>
</ul>
</li>
@@ -264,6 +265,57 @@ the latest version of <a class="reference external" href="https://raw.githubuser
Saved Objects page</p></li>
</ol>
</section>
<section id="backfilling-the-combined-dkim-spf-result-fields">
<h2>Backfilling the combined DKIM/SPF result fields<a class="headerlink" href="#backfilling-the-combined-dkim-spf-result-fields" title="Link to this heading"></a></h2>
<p>As of the version fixing <a class="reference external" href="https://github.com/domainaware/parsedmarc/issues/169">#169</a>,
aggregate documents include <code class="docutils literal notranslate"><span class="pre">dkim_results_combined</span></code> and <code class="docutils literal notranslate"><span class="pre">spf_results_combined</span></code> —
scalar string arrays that keep each auth result’s selector/scope, domain, and
result paired, which the dashboards’ alignment-detail tables aggregate on.
Reports saved by older versions lack these fields and will not appear in
those tables.</p>
<p>Running the following once per cluster backfills the fields on existing
documents. It is idempotent (documents that already have the fields are
skipped), so it is safe to re-run. It works identically on OpenSearch;
just adjust the URL and credentials. The query matches only documents
that have at least one DKIM or SPF auth result and lack the corresponding
combined field; documents with no auth results are skipped, because an
<code class="docutils literal notranslate"><span class="pre">exists</span></code> query cannot see an empty array, and for search purposes an
empty <code class="docutils literal notranslate"><span class="pre">dkim_results_combined</span></code> is identical to an absent one.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>curl<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;http://localhost:9200/dmarc_aggregate*/_update_by_query?conflicts=proceed&amp;wait_for_completion=false&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;</span>
<span class="s1">{</span>
<span class="s1"> &quot;query&quot;: {</span>
<span class="s1"> &quot;bool&quot;: {</span>
<span class="s1"> &quot;minimum_should_match&quot;: 1,</span>
<span class="s1"> &quot;should&quot;: [</span>
<span class="s1"> {</span>
<span class="s1"> &quot;bool&quot;: {</span>
<span class="s1"> &quot;must&quot;: [{&quot;exists&quot;: {&quot;field&quot;: &quot;dkim_results.domain&quot;}}],</span>
<span class="s1"> &quot;must_not&quot;: [{&quot;exists&quot;: {&quot;field&quot;: &quot;dkim_results_combined&quot;}}]</span>
<span class="s1"> }</span>
<span class="s1"> },</span>
<span class="s1"> {</span>
<span class="s1"> &quot;bool&quot;: {</span>
<span class="s1"> &quot;must&quot;: [{&quot;exists&quot;: {&quot;field&quot;: &quot;spf_results.domain&quot;}}],</span>
<span class="s1"> &quot;must_not&quot;: [{&quot;exists&quot;: {&quot;field&quot;: &quot;spf_results_combined&quot;}}]</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> ]</span>
<span class="s1"> }</span>
<span class="s1"> },</span>
<span class="s1"> &quot;script&quot;: {</span>
<span class="s1"> &quot;lang&quot;: &quot;painless&quot;,</span>
<span class="s1"> &quot;source&quot;: &quot;List dk = new ArrayList(); def dr = ctx._source.dkim_results; if (dr != null) { if (!(dr instanceof List)) { dr = [dr]; } for (e in dr) { if (e == null) { continue; } def sel = e.selector != null ? e.selector : \&quot;none\&quot;; def dom = e.domain != null ? e.domain : \&quot;none\&quot;; def res = e.result != null ? e.result : \&quot;none\&quot;; dk.add(sel + \&quot; / \&quot; + dom + \&quot; / \&quot; + res); } } ctx._source.dkim_results_combined = dk; List sp = new ArrayList(); def sr = ctx._source.spf_results; if (sr != null) { if (!(sr instanceof List)) { sr = [sr]; } for (e in sr) { if (e == null) { continue; } def sc = e.scope != null ? e.scope : \&quot;mfrom\&quot;; def dom = e.domain != null ? e.domain : \&quot;none\&quot;; def res = e.result != null ? e.result : (e.results != null ? e.results : \&quot;none\&quot;); sp.add(sc + \&quot; / \&quot; + dom + \&quot; / \&quot; + res); } } ctx._source.spf_results_combined = sp;&quot;</span>
<span class="s1"> }</span>
<span class="s1">}&#39;</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">wait_for_completion=false</span></code> returns a task ID — check progress with
<code class="docutils literal notranslate"><span class="pre">GET</span> <span class="pre">_tasks/&lt;task</span> <span class="pre">id&gt;</span></code>. Adjust the index pattern if you use a custom
<code class="docutils literal notranslate"><span class="pre">index_prefix</span></code>/<code class="docutils literal notranslate"><span class="pre">index_suffix</span></code>. After backfilling, re-import the updated
dashboards ndjson (the index pattern saved object changed too) per the
import instructions above.</p>
</section>
<section id="records-retention">
<h2>Records retention<a class="headerlink" href="#records-retention" title="Link to this heading"></a></h2>
<p>Starting in version 5.0.0, <code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> stores data in a separate
index.html
+1
View File
@@ -200,6 +200,7 @@ for RHEL or Debian.</p>
<li class="toctree-l1"><a class="reference internal" href="elasticsearch.html">Elasticsearch and Kibana</a><ul>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#installation">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#upgrading-kibana-index-patterns">Upgrading Kibana index patterns</a></li>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#backfilling-the-combined-dkim-spf-result-fields">Backfilling the combined DKIM/SPF result fields</a></li>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#records-retention">Records retention</a></li>
</ul>
</li>
kibana.html
+23 -8
View File
@@ -125,17 +125,28 @@ disposition over time.</p>
<p>Under the graphs you will find the most useful data tables on the dashboard. On
the left, there is a list of organizations that are sending you DMARC reports.
In the center, there is a list of sending servers grouped by the base domain
in their reverse DNS. On the right, there is a list of email from domains,
sorted by message volume.</p>
in their reverse DNS. On the right, there is the “Message volume and DMARC
compliance by from domain” table, which lists email from domains with their
message volume and a percentage of those messages that passed DMARC.</p>
<p>By hovering your mouse over a data table value and using the magnifying glass
icons, you can filter on our filter out different values. Start by looking at
icons, you can filter on or filter out different values. Start by looking at
the Message Sources by Reverse DNS table. Find a sender that you recognize,
such as an email marketing service, hover over it, and click on the plus (+)
magnifying glass icon, to add a filter that only shows results for that sender.
Now, look at the Message From Header table to the right. That shows you the
domains that a sender is sending as, which might tell you which brand/business
is using a particular service. With that information, you can contact them and
have them set up DKIM.</p>
Now, look at the Message volume and DMARC compliance by from domain table to
the right. That shows you the domains that a sender is sending as, and what
share of that traffic is passing DMARC, which might tell you which
brand/business is using a particular service. With that information, you can
contact them and have them set up DKIM.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The “Message volume and DMARC compliance by from domain” table is a TSVB
visualization, used because per-domain compliance percentages require a
Filter Ratio metric that agg-based data tables can’t compute. It renders
correctly on Kibana 8.x as imported, but <em>editing</em> it requires first enabling
the <code class="docutils literal notranslate"><span class="pre">metrics:allowStringIndices</span></code> advanced setting, since it references the
<code class="docutils literal notranslate"><span class="pre">dmarc_aggregate*</span></code> index as a string pattern, which Elastic has deprecated.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you have a lot of B2C customers, you may see a high volume of emails as
@@ -149,7 +160,11 @@ parent, subsidiary, and outdated brands.</p>
<p>Further down the dashboard, you can filter by source country or source IP
address.</p>
<p>Tables showing SPF and DKIM alignment details are located under the IP address
table.</p>
table. Each row of the DKIM details table is one real DKIM signature, shown
as a combined <code class="docutils literal notranslate"><span class="pre">selector</span> <span class="pre">/</span> <span class="pre">domain</span> <span class="pre">/</span> <span class="pre">result</span></code> value; the SPF details table
shows <code class="docutils literal notranslate"><span class="pre">scope</span> <span class="pre">/</span> <span class="pre">domain</span> <span class="pre">/</span> <span class="pre">result</span></code> the same way. Combining the values into one
column keeps each signature’s selector, domain, and result paired together,
rather than aggregating them as separate columns.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The alignment tables (SPF details, DKIM details) and the per-IP source
searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
usage.html
+3
View File
@@ -706,6 +706,9 @@ verification (not recommended)</p></li>
</ul>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">smtp</span></code></p>
<p>The results email is only sent when at least one aggregate, failure,
or SMTP TLS report was parsed during the run; an empty run (e.g. an
empty inbox) skips the email instead of sending headers-only CSVs.</p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">host</span></code> - str: The SMTP hostname. Required unless <code class="docutils literal notranslate"><span class="pre">[msgraph]</span></code> is
configured, in which case omitting it sends the summary via