amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-08-03 06:02:17 +00:00
Code Issues Projects Releases Wiki Activity
Files
d2d4c980ae5d8d35bc2e6a506e4588079354a91e
parsedmarc/_modules/parsedmarc/elastic.html
T
Sean Whalen d2d4c980ae Update docs
2026-07-25 13:53:18 -04:00

1479 lines
190 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.elastic &mdash; parsedmarc 10.3.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
<script src="../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../_static/documentation_options.js?v=a126560f"></script>
<script src="../../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../../_static/sphinx_highlight.js?v=6ffebe34"></script>
<script src="../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../index.html" class="icon icon-home">
parsedmarc
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../usage.html">Using parsedmarc</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../output.html">Sample outputs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../elasticsearch.html">Elasticsearch and Kibana</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../opensearch.html">OpenSearch and Grafana</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../kibana.html">Using the Kibana dashboards</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../splunk.html">Splunk</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../davmail.html">Accessing an inbox using OWA/EWS</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dmarc.html">Understanding DMARC</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributing.html">Contributing to parsedmarc</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api.html">API reference</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../index.html">parsedmarc</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../index.html">Module code</a></li>
<li class="breadcrumb-item"><a href="../parsedmarc.html">parsedmarc</a></li>
<li class="breadcrumb-item active">parsedmarc.elastic</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for parsedmarc.elastic</h1><div class="highlight"><pre>
<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">__future__</span><span class="w"> </span><span class="kn">import</span> <span class="n">annotations</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">TYPE_CHECKING</span><span class="p">,</span> <span class="n">Any</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">elasticsearch.dsl</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">Boolean</span><span class="p">,</span>
<span class="n">Date</span><span class="p">,</span>
<span class="n">Document</span><span class="p">,</span>
<span class="n">Index</span><span class="p">,</span>
<span class="n">InnerDoc</span><span class="p">,</span>
<span class="n">Integer</span><span class="p">,</span>
<span class="n">Ip</span><span class="p">,</span>
<span class="n">Keyword</span><span class="p">,</span>
<span class="n">Nested</span><span class="p">,</span>
<span class="n">Object</span><span class="p">,</span>
<span class="n">Q</span><span class="p">,</span>
<span class="n">Search</span><span class="p">,</span>
<span class="n">Text</span><span class="p">,</span>
<span class="n">connections</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc</span><span class="w"> </span><span class="kn">import</span> <span class="n">InvalidFailureReport</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc.log</span><span class="w"> </span><span class="kn">import</span> <span class="n">logger</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">parsedmarc.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">human_timestamp_to_datetime</span>
<div class="viewcode-block" id="ElasticsearchError">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.ElasticsearchError">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">ElasticsearchError</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Raised when an Elasticsearch error occurs&quot;&quot;&quot;</span></div>
<span class="c1"># Mirror of the ``serverless`` flag passed to ``set_hosts``; consulted by</span>
<span class="c1"># ``create_indexes`` to strip settings Elastic Cloud Serverless rejects.</span>
<span class="c1"># Module-level state is consistent with the existing ``connections.create_connection``</span>
<span class="c1"># global the rest of this module relies on — there is a single default ES</span>
<span class="c1"># connection per process.</span>
<span class="n">_SERVERLESS</span> <span class="o">=</span> <span class="kc">False</span>
<span class="c1"># Index settings rejected by Elastic Cloud Serverless with HTTP 400. Other</span>
<span class="c1"># settings (e.g. ``refresh_interval``) are accepted and pass through.</span>
<span class="n">_SERVERLESS_REJECTED_SETTINGS</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s2">&quot;number_of_shards&quot;</span><span class="p">,</span> <span class="s2">&quot;number_of_replicas&quot;</span><span class="p">})</span>
<span class="c1"># Guard query for the dkim_results_combined/spf_results_combined backfill</span>
<span class="c1"># (see ``migrate_indexes``). Matches only documents that have at least one</span>
<span class="c1"># DKIM or SPF auth result and are missing the corresponding combined field.</span>
<span class="c1"># Empty arrays are invisible to ``exists``, so documents with zero</span>
<span class="c1"># DKIM/SPF results are correctly skipped (verified against real data;</span>
<span class="c1"># this also makes the query idempotent — a backfilled document no longer</span>
<span class="c1"># matches). Each result is matched on an OR of its ``domain``/``result``</span>
<span class="c1"># subfields as defense in depth: the parsers we audited never store a</span>
<span class="c1"># result without both, but an empty string indexes no text tokens and is</span>
<span class="c1"># invisible to ``exists``, and the storage shape of every historical</span>
<span class="c1"># parsedmarc version can&#39;t be audited — matching either subfield costs</span>
<span class="c1"># nothing and cannot skip a document that has something to backfill.</span>
<span class="n">_COMBINED_BACKFILL_QUERY</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;minimum_should_match&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;should&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;must&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;minimum_should_match&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;should&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;dkim_results.domain&quot;</span><span class="p">}},</span>
<span class="p">{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;dkim_results.result&quot;</span><span class="p">}},</span>
<span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;must_not&quot;</span><span class="p">:</span> <span class="p">[{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;dkim_results_combined&quot;</span><span class="p">}}],</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;must&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;minimum_should_match&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;should&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;spf_results.domain&quot;</span><span class="p">}},</span>
<span class="p">{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;spf_results.result&quot;</span><span class="p">}},</span>
<span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;must_not&quot;</span><span class="p">:</span> <span class="p">[{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;spf_results_combined&quot;</span><span class="p">}}],</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1"># Painless script that (re)derives dkim_results_combined/spf_results_combined</span>
<span class="c1"># from dkim_results/spf_results, matching the format written by</span>
<span class="c1"># save_aggregate_report_to_elasticsearch(): &quot;{selector} / {domain} / {result}&quot;</span>
<span class="c1"># per DKIM result and &quot;{scope} / {domain} / {result}&quot; per SPF result.</span>
<span class="n">_COMBINED_BACKFILL_SCRIPT</span> <span class="o">=</span> <span class="p">(</span>
<span class="s2">&quot;List dk = new ArrayList(); &quot;</span>
<span class="s2">&quot;def dr = ctx._source.dkim_results; &quot;</span>
<span class="s2">&quot;if (dr != null) { &quot;</span>
<span class="s2">&quot;if (!(dr instanceof List)) { dr = [dr]; } &quot;</span>
<span class="s2">&quot;for (e in dr) { &quot;</span>
<span class="s2">&quot;if (e == null) { continue; } &quot;</span>
<span class="s1">&#39;def sel = e.selector != null ? e.selector : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def dom = e.domain != null ? e.domain : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def res = e.result != null ? e.result : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;dk.add(sel + &quot; / &quot; + dom + &quot; / &quot; + res); &#39;</span>
<span class="s2">&quot;} } &quot;</span>
<span class="s2">&quot;ctx._source.dkim_results_combined = dk; &quot;</span>
<span class="s2">&quot;List sp = new ArrayList(); &quot;</span>
<span class="s2">&quot;def sr = ctx._source.spf_results; &quot;</span>
<span class="s2">&quot;if (sr != null) { &quot;</span>
<span class="s2">&quot;if (!(sr instanceof List)) { sr = [sr]; } &quot;</span>
<span class="s2">&quot;for (e in sr) { &quot;</span>
<span class="s2">&quot;if (e == null) { continue; } &quot;</span>
<span class="s1">&#39;def sc = e.scope != null ? e.scope : &quot;mfrom&quot;; &#39;</span>
<span class="s1">&#39;def dom = e.domain != null ? e.domain : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def res = e.result != null ? e.result : (e.results != null ? e.results : &quot;none&quot;); &#39;</span>
<span class="s1">&#39;sp.add(sc + &quot; / &quot; + dom + &quot; / &quot; + res); &#39;</span>
<span class="s2">&quot;} } &quot;</span>
<span class="s2">&quot;ctx._source.spf_results_combined = sp;&quot;</span>
<span class="p">)</span>
<span class="c1"># Guard query for the policies_combined/failure_details_combined backfill</span>
<span class="c1"># (see ``migrate_indexes``). Matches only SMTP TLS documents that have at</span>
<span class="c1"># least one policy or failure detail and are missing the corresponding</span>
<span class="c1"># combined field. Empty arrays are invisible to ``exists``, so documents</span>
<span class="c1"># with zero policies/failure details are correctly skipped (this also</span>
<span class="c1"># makes the query idempotent — a backfilled document no longer matches).</span>
<span class="c1"># Each result is matched on an OR of its relevant subfields as defense in</span>
<span class="c1"># depth: the parsers we audited never store a policy/failure detail</span>
<span class="c1"># without these fields, but an empty string indexes no text tokens and is</span>
<span class="c1"># invisible to ``exists``, and the storage shape of every historical</span>
<span class="c1"># parsedmarc version can&#39;t be audited — matching either subfield costs</span>
<span class="c1"># nothing and cannot skip a document that has something to backfill.</span>
<span class="n">_SMTP_TLS_COMBINED_BACKFILL_QUERY</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;minimum_should_match&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;should&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;must&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;minimum_should_match&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;should&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;policies.policy_domain&quot;</span><span class="p">}},</span>
<span class="p">{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;policies.policy_type&quot;</span><span class="p">}},</span>
<span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;must_not&quot;</span><span class="p">:</span> <span class="p">[{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;policies_combined&quot;</span><span class="p">}}],</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;must&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;bool&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;minimum_should_match&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;should&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;policies.failure_details.result_type&quot;</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;policies.failure_details.sending_mta_ip&quot;</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;must_not&quot;</span><span class="p">:</span> <span class="p">[{</span><span class="s2">&quot;exists&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;field&quot;</span><span class="p">:</span> <span class="s2">&quot;failure_details_combined&quot;</span><span class="p">}}],</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1"># Painless script that (re)derives policies_combined/failure_details_combined</span>
<span class="c1"># from policies/policies.failure_details, matching the format written by</span>
<span class="c1"># save_smtp_tls_report_to_elasticsearch(): &quot;{policy_domain} / {policy_type}&quot;</span>
<span class="c1"># per policy and &quot;{policy_domain} / {policy_type} / {result_type} /</span>
<span class="c1"># {sending_mta_ip} / {receiving_ip} / {receiving_mx_hostname}&quot; per failure</span>
<span class="c1"># detail.</span>
<span class="n">_SMTP_TLS_COMBINED_BACKFILL_SCRIPT</span> <span class="o">=</span> <span class="p">(</span>
<span class="s2">&quot;List pols = new ArrayList(); &quot;</span>
<span class="s2">&quot;List dets = new ArrayList(); &quot;</span>
<span class="s2">&quot;def ps = ctx._source.policies; &quot;</span>
<span class="s2">&quot;if (ps != null) { &quot;</span>
<span class="s2">&quot;if (!(ps instanceof List)) { ps = [ps]; } &quot;</span>
<span class="s2">&quot;for (p in ps) { &quot;</span>
<span class="s2">&quot;if (p == null) { continue; } &quot;</span>
<span class="s1">&#39;def dom = p.policy_domain != null ? p.policy_domain : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def typ = p.policy_type != null ? p.policy_type : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;pols.add(dom + &quot; / &quot; + typ); &#39;</span>
<span class="s2">&quot;def fds = p.failure_details; &quot;</span>
<span class="s2">&quot;if (fds != null) { &quot;</span>
<span class="s2">&quot;if (!(fds instanceof List)) { fds = [fds]; } &quot;</span>
<span class="s2">&quot;for (f in fds) { &quot;</span>
<span class="s2">&quot;if (f == null) { continue; } &quot;</span>
<span class="s1">&#39;def rt = f.result_type != null ? f.result_type : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def smi = f.sending_mta_ip != null ? f.sending_mta_ip : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def ri = f.receiving_ip != null ? f.receiving_ip : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;def rmh = f.receiving_mx_hostname != null ? f.receiving_mx_hostname : &quot;none&quot;; &#39;</span>
<span class="s1">&#39;dets.add(dom + &quot; / &quot; + typ + &quot; / &quot; + rt + &quot; / &quot; + smi + &quot; / &quot; + ri + &quot; / &quot; + rmh); &#39;</span>
<span class="s2">&quot;} } } } &quot;</span>
<span class="s2">&quot;ctx._source.policies_combined = pols; &quot;</span>
<span class="s2">&quot;ctx._source.failure_details_combined = dets;&quot;</span>
<span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_PolicyOverride</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># The elasticsearch.dsl 8.x type stubs use dataclass_transform and only</span>
<span class="c1"># surface dataclass-style annotated fields (``name: M[...] =</span>
<span class="c1"># mapped_field(...)``) as constructor parameters. This module declares</span>
<span class="c1"># fields the pre-8.x way (``name = Text()``), which the runtime fully</span>
<span class="c1"># supports via ObjectBase.__init__(**kwargs), so this TYPE_CHECKING-only</span>
<span class="c1"># declaration restores the real runtime signature for the type checker.</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="nb">type</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_PublishedPolicy</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">adkim</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">aspf</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">sp</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">pct</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">fo</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">np</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">testing</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">discovery_method</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_DKIMResult</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">selector</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">human_result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_SPFResult</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">scope</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">human_result</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_AggregateReportDoc</span><span class="p">(</span><span class="n">Document</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">class</span><span class="w"> </span><span class="nc">Index</span><span class="p">:</span>
<span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;dmarc_aggregate&quot;</span>
<span class="n">xml_schema</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">xml_namespace</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">org_name</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">org_email</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">org_extra_contact_info</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">report_id</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">date_range</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">date_begin</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">date_end</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">normalized_timespan</span> <span class="o">=</span> <span class="n">Boolean</span><span class="p">()</span>
<span class="n">original_timespan_seconds</span> <span class="o">=</span> <span class="n">Integer</span>
<span class="n">errors</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">published_policy</span> <span class="o">=</span> <span class="n">Object</span><span class="p">(</span><span class="n">_PublishedPolicy</span><span class="p">)</span>
<span class="n">source_ip_address</span> <span class="o">=</span> <span class="n">Ip</span><span class="p">()</span>
<span class="n">source_country</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_reverse_dns</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_base_domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_type</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_name</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_asn</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">source_as_name</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_as_domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">message_count</span> <span class="o">=</span> <span class="n">Integer</span>
<span class="n">disposition</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">dkim_aligned</span> <span class="o">=</span> <span class="n">Boolean</span><span class="p">()</span>
<span class="n">spf_aligned</span> <span class="o">=</span> <span class="n">Boolean</span><span class="p">()</span>
<span class="n">passed_dmarc</span> <span class="o">=</span> <span class="n">Boolean</span><span class="p">()</span>
<span class="n">policy_overrides</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_PolicyOverride</span><span class="p">)</span>
<span class="n">header_from</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">envelope_from</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">envelope_to</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="c1"># Nested(...) on the two auth-result fields below is only the DSL&#39;s</span>
<span class="c1"># in-memory document shape; it is never installed as a mapping.</span>
<span class="c1"># create_indexes() deliberately skips Index.document() registration so</span>
<span class="c1"># these fields stay dynamic-mapped as plain `object` in the cluster</span>
<span class="c1"># (see the comment there and issue #169).</span>
<span class="n">dkim_results</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_DKIMResult</span><span class="p">)</span>
<span class="n">spf_results</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_SPFResult</span><span class="p">)</span>
<span class="c1"># One &quot;{selector} / {domain} / {result}&quot; (DKIM) or &quot;{scope} / {domain} /</span>
<span class="c1"># {result}&quot; (SPF) string per auth result. Kibana/Grafana tables cannot</span>
<span class="c1"># terms-aggregate the subfields of an object array without producing a</span>
<span class="c1"># cross-product of values (issue #169), so dashboards aggregate these</span>
<span class="c1"># composed keywords instead. Declared to match what dynamic mapping</span>
<span class="c1"># produces for a string array (text + .keyword).</span>
<span class="n">dkim_results_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">spf_results_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">np</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">testing</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">discovery_method</span> <span class="o">=</span> <span class="n">Keyword</span><span class="p">()</span>
<span class="n">generator</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_policy_override</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">type_</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">comment</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">policy_overrides</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_PolicyOverride</span><span class="p">(</span><span class="nb">type</span><span class="o">=</span><span class="n">type_</span><span class="p">,</span> <span class="n">comment</span><span class="o">=</span><span class="n">comment</span><span class="p">))</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_dkim_result</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">selector</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">human_result</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dkim_results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="n">_DKIMResult</span><span class="p">(</span>
<span class="n">domain</span><span class="o">=</span><span class="n">domain</span><span class="p">,</span>
<span class="n">selector</span><span class="o">=</span><span class="n">selector</span><span class="p">,</span>
<span class="n">result</span><span class="o">=</span><span class="n">result</span><span class="p">,</span>
<span class="n">human_result</span><span class="o">=</span><span class="n">human_result</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dkim_results_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">selector</span><span class="si">}</span><span class="s2"> / </span><span class="si">{</span><span class="n">domain</span><span class="si">}</span><span class="s2"> / </span><span class="si">{</span><span class="n">result</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_spf_result</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">scope</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">human_result</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spf_results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="n">_SPFResult</span><span class="p">(</span>
<span class="n">domain</span><span class="o">=</span><span class="n">domain</span><span class="p">,</span>
<span class="n">scope</span><span class="o">=</span><span class="n">scope</span><span class="p">,</span>
<span class="n">result</span><span class="o">=</span><span class="n">result</span><span class="p">,</span>
<span class="n">human_result</span><span class="o">=</span><span class="n">human_result</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spf_results_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">scope</span><span class="si">}</span><span class="s2"> / </span><span class="si">{</span><span class="n">domain</span><span class="si">}</span><span class="s2"> / </span><span class="si">{</span><span class="n">result</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">save</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="c1"># pyright: ignore[reportIncompatibleMethodOverride]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">passed_dmarc</span> <span class="o">=</span> <span class="kc">False</span>
<span class="bp">self</span><span class="o">.</span><span class="n">passed_dmarc</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">spf_aligned</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dkim_aligned</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_EmailAddressDoc</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">display_name</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_EmailAttachmentDoc</span><span class="p">(</span><span class="n">Document</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">sha256</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_FailureSampleDoc</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">raw</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">headers</span> <span class="o">=</span> <span class="n">Object</span><span class="p">()</span>
<span class="n">headers_only</span> <span class="o">=</span> <span class="n">Boolean</span><span class="p">()</span>
<span class="n">to</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">)</span>
<span class="n">subject</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">filename_safe_subject</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">_from</span> <span class="o">=</span> <span class="n">Object</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">)</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">reply_to</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">)</span>
<span class="n">cc</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">)</span>
<span class="n">bcc</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">)</span>
<span class="n">body</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">attachments</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_EmailAttachmentDoc</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_to</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">display_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">address</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">display_name</span><span class="p">,</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">))</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_reply_to</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">display_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">address</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_to</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="n">_EmailAddressDoc</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">display_name</span><span class="p">,</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_cc</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">display_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">address</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cc</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">display_name</span><span class="p">,</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">))</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_bcc</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">display_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">address</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bcc</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_EmailAddressDoc</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">display_name</span><span class="p">,</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">))</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_attachment</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">sha256</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">attachments</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="n">_EmailAttachmentDoc</span><span class="p">(</span>
<span class="n">filename</span><span class="o">=</span><span class="n">filename</span><span class="p">,</span> <span class="n">content_type</span><span class="o">=</span><span class="n">content_type</span><span class="p">,</span> <span class="n">sha256</span><span class="o">=</span><span class="n">sha256</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_FailureReportDoc</span><span class="p">(</span><span class="n">Document</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">class</span><span class="w"> </span><span class="nc">Index</span><span class="p">:</span>
<span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;dmarc_failure&quot;</span>
<span class="n">feedback_type</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">user_agent</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">version</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">original_mail_from</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">arrival_date</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">original_envelope_id</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">authentication_results</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">delivery_results</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_ip_address</span> <span class="o">=</span> <span class="n">Ip</span><span class="p">()</span>
<span class="n">source_country</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_reverse_dns</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_asn</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">source_as_name</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_as_domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_authentication_mechanisms</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">source_auth_failures</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">dkim_domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">original_rcpt_to</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">sample</span> <span class="o">=</span> <span class="n">Object</span><span class="p">(</span><span class="n">_FailureSampleDoc</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_SMTPTLSFailureDetailsDoc</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">result_type</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">sending_mta_ip</span> <span class="o">=</span> <span class="n">Ip</span><span class="p">()</span>
<span class="n">receiving_mx_helo</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">receiving_mx_hostname</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">receiving_ip</span> <span class="o">=</span> <span class="n">Ip</span><span class="p">()</span>
<span class="n">failed_session_count</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">additional_information_uri</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">failure_reason_code</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_SMTPTLSPolicyDoc</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">policy_domain</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">policy_type</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">policy_strings</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">mx_host_patterns</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">successful_session_count</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">failed_session_count</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">failure_details</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_SMTPTLSFailureDetailsDoc</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">add_failure_details</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">result_type</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">ip_address</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">receiving_ip</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">receiving_mx_helo</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">failed_session_count</span><span class="p">:</span> <span class="nb">int</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">sending_mta_ip</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">receiving_mx_hostname</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">additional_information_uri</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">failure_reason_code</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="nb">int</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="n">_details</span> <span class="o">=</span> <span class="n">_SMTPTLSFailureDetailsDoc</span><span class="p">(</span>
<span class="n">result_type</span><span class="o">=</span><span class="n">result_type</span><span class="p">,</span>
<span class="n">ip_address</span><span class="o">=</span><span class="n">ip_address</span><span class="p">,</span>
<span class="n">sending_mta_ip</span><span class="o">=</span><span class="n">sending_mta_ip</span><span class="p">,</span>
<span class="n">receiving_mx_hostname</span><span class="o">=</span><span class="n">receiving_mx_hostname</span><span class="p">,</span>
<span class="n">receiving_mx_helo</span><span class="o">=</span><span class="n">receiving_mx_helo</span><span class="p">,</span>
<span class="n">receiving_ip</span><span class="o">=</span><span class="n">receiving_ip</span><span class="p">,</span>
<span class="n">failed_session_count</span><span class="o">=</span><span class="n">failed_session_count</span><span class="p">,</span>
<span class="n">additional_information_uri</span><span class="o">=</span><span class="n">additional_information_uri</span><span class="p">,</span>
<span class="n">failure_reason_code</span><span class="o">=</span><span class="n">failure_reason_code</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">failure_details</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_details</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">_SMTPTLSReportDoc</span><span class="p">(</span><span class="n">Document</span><span class="p">):</span>
<span class="c1"># TYPE_CHECKING __init__: see _PolicyOverride</span>
<span class="k">if</span> <span class="n">TYPE_CHECKING</span><span class="p">:</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">class</span><span class="w"> </span><span class="nc">Index</span><span class="p">:</span>
<span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;smtp_tls&quot;</span>
<span class="n">organization_name</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">date_range</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">date_begin</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">date_end</span> <span class="o">=</span> <span class="n">Date</span><span class="p">()</span>
<span class="n">contact_info</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">report_id</span> <span class="o">=</span> <span class="n">Text</span><span class="p">()</span>
<span class="n">policies</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">_SMTPTLSPolicyDoc</span><span class="p">)</span>
<span class="c1"># One &quot;{policy_domain} / {policy_type}&quot; string per policy. Kibana/</span>
<span class="c1"># Grafana tables cannot terms-aggregate the subfields of an object</span>
<span class="c1"># array without producing a cross-product of values (issue #169), so</span>
<span class="c1"># dashboards aggregate these composed keywords instead. Declared to</span>
<span class="c1"># match what dynamic mapping produces for a string array (text +</span>
<span class="c1"># .keyword).</span>
<span class="n">policies_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span><span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)})</span>
<span class="c1"># One &quot;{policy_domain} / {policy_type} / {result_type} /</span>
<span class="c1"># {sending_mta_ip} / {receiving_ip} / {receiving_mx_hostname}&quot; string</span>
<span class="c1"># per failure detail, across all policies.</span>
<span class="n">failure_details_combined</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span>
<span class="n">multi</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;keyword&quot;</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">(</span><span class="n">ignore_above</span><span class="o">=</span><span class="mi">256</span><span class="p">)}</span>
<span class="p">)</span>
<div class="viewcode-block" id="AlreadySaved">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.AlreadySaved">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">AlreadySaved</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Raised when a report to be saved matches an existing report&quot;&quot;&quot;</span></div>
<div class="viewcode-block" id="set_hosts">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.set_hosts">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">set_hosts</span><span class="p">(</span>
<span class="n">hosts</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span>
<span class="o">*</span><span class="p">,</span>
<span class="n">use_ssl</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">ssl_cert_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">skip_certificate_verification</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">username</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">password</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">api_key</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">timeout</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">60.0</span><span class="p">,</span>
<span class="n">serverless</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Sets the Elasticsearch hosts to use</span>
<span class="sd"> Args:</span>
<span class="sd"> hosts (str | list[str]): A single hostname or URL, or list of hostnames or URLs</span>
<span class="sd"> use_ssl (bool): Controls the scheme prepended to any host that doesn&#39;t</span>
<span class="sd"> already include one (``https://`` when True, ``http://`` when</span>
<span class="sd"> False). Hosts that already carry a scheme are left unchanged.</span>
<span class="sd"> ssl_cert_path (str): Path to the certificate chain</span>
<span class="sd"> skip_certificate_verification (bool): Skip certificate verification</span>
<span class="sd"> username (str): The username to use for authentication</span>
<span class="sd"> password (str): The password to use for authentication</span>
<span class="sd"> api_key (str): The Base64 encoded API key to use for authentication</span>
<span class="sd"> timeout (float): Timeout in seconds</span>
<span class="sd"> serverless (bool): Target an Elastic Cloud Serverless project. When True,</span>
<span class="sd"> ``create_indexes`` strips ``number_of_shards`` / ``number_of_replicas``</span>
<span class="sd"> from its settings (which Serverless rejects with HTTP 400) and passes</span>
<span class="sd"> any other settings through unchanged.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Module-global; see the _SERVERLESS comment at the top of the module.</span>
<span class="k">global</span> <span class="n">_SERVERLESS</span>
<span class="n">_SERVERLESS</span> <span class="o">=</span> <span class="n">serverless</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hosts</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="n">hosts</span><span class="p">]</span>
<span class="n">scheme</span> <span class="o">=</span> <span class="s2">&quot;https://&quot;</span> <span class="k">if</span> <span class="n">use_ssl</span> <span class="k">else</span> <span class="s2">&quot;http://&quot;</span>
<span class="n">normalized_hosts</span> <span class="o">=</span> <span class="p">[</span><span class="n">host</span> <span class="k">if</span> <span class="s2">&quot;://&quot;</span> <span class="ow">in</span> <span class="n">host</span> <span class="k">else</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">scheme</span><span class="si">}{</span><span class="n">host</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">host</span> <span class="ow">in</span> <span class="n">hosts</span><span class="p">]</span>
<span class="n">conn_params</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;hosts&quot;</span><span class="p">:</span> <span class="n">normalized_hosts</span><span class="p">,</span> <span class="s2">&quot;request_timeout&quot;</span><span class="p">:</span> <span class="n">timeout</span><span class="p">}</span>
<span class="k">if</span> <span class="n">use_ssl</span><span class="p">:</span>
<span class="k">if</span> <span class="n">ssl_cert_path</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;ca_certs&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">ssl_cert_path</span>
<span class="k">if</span> <span class="n">skip_certificate_verification</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;verify_certs&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;verify_certs&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">if</span> <span class="n">username</span> <span class="ow">and</span> <span class="n">password</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;basic_auth&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
<span class="k">if</span> <span class="n">api_key</span><span class="p">:</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">&quot;api_key&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">api_key</span>
<span class="n">connections</span><span class="o">.</span><span class="n">create_connection</span><span class="p">(</span><span class="o">**</span><span class="n">conn_params</span><span class="p">)</span></div>
<div class="viewcode-block" id="create_indexes">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.create_indexes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">create_indexes</span><span class="p">(</span><span class="n">names</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">settings</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Create Elasticsearch indexes</span>
<span class="sd"> Args:</span>
<span class="sd"> names (list): A list of index names</span>
<span class="sd"> settings (dict): Index settings. In Serverless mode, keys in</span>
<span class="sd"> ``_SERVERLESS_REJECTED_SETTINGS`` are filtered out and the</span>
<span class="sd"> remaining keys are passed through; defaults are skipped entirely.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="n">settings</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">effective_settings</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span>
<span class="p">{}</span> <span class="k">if</span> <span class="n">_SERVERLESS</span> <span class="k">else</span> <span class="p">{</span><span class="s2">&quot;number_of_shards&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s2">&quot;number_of_replicas&quot;</span><span class="p">:</span> <span class="mi">0</span><span class="p">}</span>
<span class="p">)</span>
<span class="k">elif</span> <span class="n">_SERVERLESS</span><span class="p">:</span>
<span class="n">effective_settings</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">k</span><span class="p">:</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">settings</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">_SERVERLESS_REJECTED_SETTINGS</span>
<span class="p">}</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">effective_settings</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">settings</span><span class="p">)</span>
<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="n">Index</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Deliberately no Index.document() registration: the shipped</span>
<span class="c1"># dashboards cannot rebuild their detail tables on a `nested`</span>
<span class="c1"># mapping — Kibana/OSD visual editors do not support nested</span>
<span class="c1"># fields, Vega can run nested aggregations but does not render</span>
<span class="c1"># tables, and Grafana&#39;s nested bucket aggregation (9.4+) lacks</span>
<span class="c1"># reverse_nested for parent-level metrics like message_count —</span>
<span class="c1"># so the dynamic `object` mapping produced by a bare create is</span>
<span class="c1"># load-bearing for the shipped dashboards. _AggregateReportDoc</span>
<span class="c1"># still declares dkim_results/spf_results with Nested(...), but</span>
<span class="c1"># that is only the DSL&#39;s in-memory shape for building documents</span>
<span class="c1"># — it is never installed as a mapping. See issue #169 and the</span>
<span class="c1"># *_combined fields on _AggregateReportDoc.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">index</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Creating Elasticsearch index: </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">effective_settings</span><span class="p">:</span>
<span class="n">index</span><span class="o">.</span><span class="n">settings</span><span class="p">(</span><span class="o">**</span><span class="n">effective_settings</span><span class="p">)</span>
<span class="n">index</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ElasticsearchError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Elasticsearch error: </span><span class="si">{</span><span class="n">e</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="migrate_indexes">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.migrate_indexes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">migrate_indexes</span><span class="p">(</span>
<span class="n">aggregate_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">failure_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">smtp_tls_indexes</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Backfills the ``dkim_results_combined``/``spf_results_combined`` fields</span>
<span class="sd"> (added for issue #169) on aggregate report documents, and the</span>
<span class="sd"> ``policies_combined``/``failure_details_combined`` fields on SMTP TLS</span>
<span class="sd"> report documents, that were saved before those fields existed.</span>
<span class="sd"> For each name in ``aggregate_indexes``/``smtp_tls_indexes``, this</span>
<span class="sd"> submits an ``update_by_query`` against the ``f&quot;{name}*&quot;`` index pattern</span>
<span class="sd"> (the real indexes are date-suffixed) as a non-blocking background task</span>
<span class="sd"> (``wait_for_completion=False``), so it never delays parsedmarc startup.</span>
<span class="sd"> Submission is guarded by a cheap ``count`` query that only matches</span>
<span class="sd"> documents with DKIM/SPF results (or policies/failure details) but no</span>
<span class="sd"> combined field, so once an index is fully backfilled, later calls are a</span>
<span class="sd"> fast no-op. Any error talking to the cluster (e.g. no indexes yet on a</span>
<span class="sd"> fresh install, or a transient connection issue) is caught and logged as</span>
<span class="sd"> a warning rather than raised; the backfill is simply retried on the</span>
<span class="sd"> next startup, and the manual ``_update_by_query`` commands documented in</span>
<span class="sd"> ``docs/source/elasticsearch.md`` remain available in the meantime.</span>
<span class="sd"> Args:</span>
<span class="sd"> aggregate_indexes (list): A list of aggregate index names</span>
<span class="sd"> failure_indexes (list): A list of failure index names</span>
<span class="sd"> (accepted for API compatibility; unused)</span>
<span class="sd"> smtp_tls_indexes (list): A list of SMTP TLS index names</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">aggregate_indexes</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">smtp_tls_indexes</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">connections</span><span class="o">.</span><span class="n">get_connection</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="s2">&quot;Skipping the dkim_results_combined/spf_results_combined/&quot;</span>
<span class="s2">&quot;policies_combined/failure_details_combined backfill: could &quot;</span>
<span class="sa">f</span><span class="s2">&quot;not get an Elasticsearch connection: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. This will be &quot;</span>
<span class="s2">&quot;retried at the next startup.&quot;</span>
<span class="p">)</span>
<span class="k">return</span>
<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">aggregate_indexes</span> <span class="ow">or</span> <span class="p">[]:</span>
<span class="n">pattern</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">*&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">count_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">count</span><span class="p">(</span>
<span class="n">index</span><span class="o">=</span><span class="n">pattern</span><span class="p">,</span>
<span class="n">query</span><span class="o">=</span><span class="n">_COMBINED_BACKFILL_QUERY</span><span class="p">,</span>
<span class="n">ignore_unavailable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">allow_no_indices</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">count_response</span><span class="p">[</span><span class="s2">&quot;count&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">count</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">update_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">update_by_query</span><span class="p">(</span>
<span class="n">index</span><span class="o">=</span><span class="n">pattern</span><span class="p">,</span>
<span class="n">query</span><span class="o">=</span><span class="n">_COMBINED_BACKFILL_QUERY</span><span class="p">,</span>
<span class="n">script</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;source&quot;</span><span class="p">:</span> <span class="n">_COMBINED_BACKFILL_SCRIPT</span><span class="p">,</span> <span class="s2">&quot;lang&quot;</span><span class="p">:</span> <span class="s2">&quot;painless&quot;</span><span class="p">},</span>
<span class="n">conflicts</span><span class="o">=</span><span class="s2">&quot;proceed&quot;</span><span class="p">,</span>
<span class="n">wait_for_completion</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">ignore_unavailable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">allow_no_indices</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">task_id</span> <span class="o">=</span> <span class="n">update_response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;task&quot;</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s2">&quot;Backfilling dkim_results_combined/spf_results_combined on &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">count</span><span class="si">}</span><span class="s2"> existing documents in </span><span class="si">{</span><span class="n">pattern</span><span class="si">}</span><span class="s2"> (task </span><span class="si">{</span><span class="n">task_id</span><span class="si">}</span><span class="s2">)&quot;</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="s2">&quot;Failed to check/submit the dkim_results_combined/&quot;</span>
<span class="sa">f</span><span class="s2">&quot;spf_results_combined backfill for </span><span class="si">{</span><span class="n">pattern</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. This &quot;</span>
<span class="s2">&quot;will be retried at the next startup; the manual &quot;</span>
<span class="s2">&quot;_update_by_query command in the documentation remains &quot;</span>
<span class="s2">&quot;available in the meantime.&quot;</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">smtp_tls_indexes</span> <span class="ow">or</span> <span class="p">[]:</span>
<span class="n">pattern</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">*&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">count_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">count</span><span class="p">(</span>
<span class="n">index</span><span class="o">=</span><span class="n">pattern</span><span class="p">,</span>
<span class="n">query</span><span class="o">=</span><span class="n">_SMTP_TLS_COMBINED_BACKFILL_QUERY</span><span class="p">,</span>
<span class="n">ignore_unavailable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">allow_no_indices</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">count_response</span><span class="p">[</span><span class="s2">&quot;count&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">count</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">update_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">update_by_query</span><span class="p">(</span>
<span class="n">index</span><span class="o">=</span><span class="n">pattern</span><span class="p">,</span>
<span class="n">query</span><span class="o">=</span><span class="n">_SMTP_TLS_COMBINED_BACKFILL_QUERY</span><span class="p">,</span>
<span class="n">script</span><span class="o">=</span><span class="p">{</span>
<span class="s2">&quot;source&quot;</span><span class="p">:</span> <span class="n">_SMTP_TLS_COMBINED_BACKFILL_SCRIPT</span><span class="p">,</span>
<span class="s2">&quot;lang&quot;</span><span class="p">:</span> <span class="s2">&quot;painless&quot;</span><span class="p">,</span>
<span class="p">},</span>
<span class="n">conflicts</span><span class="o">=</span><span class="s2">&quot;proceed&quot;</span><span class="p">,</span>
<span class="n">wait_for_completion</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">ignore_unavailable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">allow_no_indices</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">task_id</span> <span class="o">=</span> <span class="n">update_response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;task&quot;</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s2">&quot;Backfilling policies_combined/failure_details_combined on &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">count</span><span class="si">}</span><span class="s2"> existing documents in </span><span class="si">{</span><span class="n">pattern</span><span class="si">}</span><span class="s2"> (task </span><span class="si">{</span><span class="n">task_id</span><span class="si">}</span><span class="s2">)&quot;</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="s2">&quot;Failed to check/submit the policies_combined/&quot;</span>
<span class="sa">f</span><span class="s2">&quot;failure_details_combined backfill for </span><span class="si">{</span><span class="n">pattern</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. &quot;</span>
<span class="s2">&quot;This will be retried at the next startup; the manual &quot;</span>
<span class="s2">&quot;_update_by_query command in the documentation remains &quot;</span>
<span class="s2">&quot;available in the meantime.&quot;</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="save_aggregate_report_to_elasticsearch">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.save_aggregate_report_to_elasticsearch">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">save_aggregate_report_to_elasticsearch</span><span class="p">(</span>
<span class="n">aggregate_report</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span>
<span class="n">index_suffix</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">index_prefix</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">monthly_indexes</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">number_of_shards</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span>
<span class="n">number_of_replicas</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Saves a parsed DMARC aggregate report to Elasticsearch</span>
<span class="sd"> Args:</span>
<span class="sd"> aggregate_report (dict): A parsed aggregate report</span>
<span class="sd"> index_suffix (str): The suffix of the name of the index to save to</span>
<span class="sd"> index_prefix (str): The prefix of the name of the index to save to</span>
<span class="sd"> monthly_indexes (bool): Use monthly indexes instead of daily indexes</span>
<span class="sd"> number_of_shards (int): The number of shards to use in the index</span>
<span class="sd"> number_of_replicas (int): The number of replicas to use in the index</span>
<span class="sd"> Raises:</span>
<span class="sd"> AlreadySaved</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;Saving aggregate report to Elasticsearch&quot;</span><span class="p">)</span>
<span class="n">aggregate_report</span> <span class="o">=</span> <span class="n">aggregate_report</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">metadata</span> <span class="o">=</span> <span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;report_metadata&quot;</span><span class="p">]</span>
<span class="n">org_name</span> <span class="o">=</span> <span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;org_name&quot;</span><span class="p">]</span>
<span class="n">report_id</span> <span class="o">=</span> <span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;report_id&quot;</span><span class="p">]</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;domain&quot;</span><span class="p">]</span>
<span class="n">begin_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">],</span> <span class="n">to_utc</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">end_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">],</span> <span class="n">to_utc</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">if</span> <span class="n">monthly_indexes</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">org_name_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">org_name</span><span class="o">=</span><span class="n">org_name</span><span class="p">)))</span> <span class="c1"># type: ignore</span>
<span class="n">report_id_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">report_id</span><span class="o">=</span><span class="n">report_id</span><span class="p">)))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">domain_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;published_policy.domain&quot;</span><span class="p">:</span> <span class="n">domain</span><span class="p">}))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">begin_date_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="nb">range</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">date_begin</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">gte</span><span class="o">=</span><span class="n">begin_date</span><span class="p">))))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">end_date_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="nb">range</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">date_end</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">lte</span><span class="o">=</span><span class="n">end_date</span><span class="p">))))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="k">if</span> <span class="n">index_suffix</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;dmarc_aggregate_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">*&quot;</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="s2">&quot;dmarc_aggregate*&quot;</span>
<span class="k">if</span> <span class="n">index_prefix</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index_prefix</span><span class="si">}{</span><span class="n">search_index</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">search</span> <span class="o">=</span> <span class="n">Search</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="n">search_index</span><span class="p">)</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">org_name_query</span> <span class="o">&amp;</span> <span class="n">report_id_query</span> <span class="o">&amp;</span> <span class="n">domain_query</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">query</span> <span class="o">&amp;</span> <span class="n">begin_date_query</span> <span class="o">&amp;</span> <span class="n">end_date_query</span>
<span class="c1"># elasticsearch.dsl&#39;s own docs recommend ``search.query = Q(...)``, but</span>
<span class="c1"># the ProxyDescriptor.__set__ stub is typed to accept only</span>
<span class="c1"># Dict[str, Any], not the Query object the docs tell you to assign.</span>
<span class="n">search</span><span class="o">.</span><span class="n">query</span> <span class="o">=</span> <span class="n">query</span> <span class="c1"># pyright: ignore[reportAttributeAccessIssue]</span>
<span class="n">begin_date_human</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2"> %H:%M:%SZ&quot;</span><span class="p">)</span>
<span class="n">end_date_human</span> <span class="o">=</span> <span class="n">end_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2"> %H:%M:%SZ&quot;</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">existing</span> <span class="o">=</span> <span class="n">search</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">error_</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ElasticsearchError</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Elasticsearch&#39;s search for existing report error: </span><span class="si">{</span><span class="n">error_</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">existing</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AlreadySaved</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;An aggregate report ID </span><span class="si">{</span><span class="n">report_id</span><span class="si">}</span><span class="s2"> from </span><span class="si">{</span><span class="n">org_name</span><span class="si">}</span><span class="s2"> about </span><span class="si">{</span><span class="n">domain</span><span class="si">}</span><span class="s2"> &quot;</span>
<span class="sa">f</span><span class="s2">&quot;with a date range of </span><span class="si">{</span><span class="n">begin_date_human</span><span class="si">}</span><span class="s2"> UTC to </span><span class="si">{</span><span class="n">end_date_human</span><span class="si">}</span><span class="s2"> UTC already &quot;</span>
<span class="s2">&quot;exists in &quot;</span>
<span class="s2">&quot;Elasticsearch&quot;</span>
<span class="p">)</span>
<span class="n">published_policy</span> <span class="o">=</span> <span class="n">_PublishedPolicy</span><span class="p">(</span>
<span class="n">domain</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;domain&quot;</span><span class="p">],</span>
<span class="n">adkim</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;adkim&quot;</span><span class="p">],</span>
<span class="n">aspf</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;aspf&quot;</span><span class="p">],</span>
<span class="n">p</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;p&quot;</span><span class="p">],</span>
<span class="n">sp</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;sp&quot;</span><span class="p">],</span>
<span class="n">pct</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;pct&quot;</span><span class="p">],</span>
<span class="n">fo</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">][</span><span class="s2">&quot;fo&quot;</span><span class="p">],</span>
<span class="n">np</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;np&quot;</span><span class="p">),</span>
<span class="n">testing</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;testing&quot;</span><span class="p">),</span>
<span class="n">discovery_method</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;discovery_method&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;records&quot;</span><span class="p">]:</span>
<span class="n">begin_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span>
<span class="n">record</span><span class="p">[</span><span class="s2">&quot;interval_begin&quot;</span><span class="p">],</span> <span class="n">to_utc</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">assume_utc</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">end_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span>
<span class="n">record</span><span class="p">[</span><span class="s2">&quot;interval_end&quot;</span><span class="p">],</span> <span class="n">to_utc</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">assume_utc</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">normalized_timespan</span> <span class="o">=</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;normalized_timespan&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="n">monthly_indexes</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">begin_date</span>
<span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">end_date</span>
<span class="n">date_range</span> <span class="o">=</span> <span class="p">[</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">],</span> <span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">]]</span>
<span class="n">agg_doc</span> <span class="o">=</span> <span class="n">_AggregateReportDoc</span><span class="p">(</span>
<span class="n">xml_schema</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;xml_schema&quot;</span><span class="p">],</span>
<span class="n">xml_namespace</span><span class="o">=</span><span class="n">aggregate_report</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;xml_namespace&quot;</span><span class="p">),</span>
<span class="n">org_name</span><span class="o">=</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;org_name&quot;</span><span class="p">],</span>
<span class="n">org_email</span><span class="o">=</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;org_email&quot;</span><span class="p">],</span>
<span class="n">org_extra_contact_info</span><span class="o">=</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;org_extra_contact_info&quot;</span><span class="p">],</span>
<span class="n">report_id</span><span class="o">=</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;report_id&quot;</span><span class="p">],</span>
<span class="n">date_range</span><span class="o">=</span><span class="n">date_range</span><span class="p">,</span>
<span class="n">date_begin</span><span class="o">=</span><span class="n">begin_date</span><span class="p">,</span>
<span class="n">date_end</span><span class="o">=</span><span class="n">end_date</span><span class="p">,</span>
<span class="n">normalized_timespan</span><span class="o">=</span><span class="n">normalized_timespan</span><span class="p">,</span>
<span class="n">errors</span><span class="o">=</span><span class="n">metadata</span><span class="p">[</span><span class="s2">&quot;errors&quot;</span><span class="p">],</span>
<span class="n">published_policy</span><span class="o">=</span><span class="n">published_policy</span><span class="p">,</span>
<span class="n">source_ip_address</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;ip_address&quot;</span><span class="p">],</span>
<span class="n">source_country</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;country&quot;</span><span class="p">],</span>
<span class="n">source_reverse_dns</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;reverse_dns&quot;</span><span class="p">],</span>
<span class="n">source_base_domain</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;base_domain&quot;</span><span class="p">],</span>
<span class="n">source_type</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;type&quot;</span><span class="p">],</span>
<span class="n">source_name</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;name&quot;</span><span class="p">],</span>
<span class="n">source_asn</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;asn&quot;</span><span class="p">],</span>
<span class="n">source_as_name</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;as_name&quot;</span><span class="p">],</span>
<span class="n">source_as_domain</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;as_domain&quot;</span><span class="p">],</span>
<span class="n">message_count</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;count&quot;</span><span class="p">],</span>
<span class="n">disposition</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;policy_evaluated&quot;</span><span class="p">][</span><span class="s2">&quot;disposition&quot;</span><span class="p">],</span>
<span class="n">dkim_aligned</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;policy_evaluated&quot;</span><span class="p">][</span><span class="s2">&quot;dkim&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="ow">and</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;policy_evaluated&quot;</span><span class="p">][</span><span class="s2">&quot;dkim&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;pass&quot;</span><span class="p">,</span>
<span class="n">spf_aligned</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;policy_evaluated&quot;</span><span class="p">][</span><span class="s2">&quot;spf&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="ow">and</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;policy_evaluated&quot;</span><span class="p">][</span><span class="s2">&quot;spf&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;pass&quot;</span><span class="p">,</span>
<span class="n">header_from</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;identifiers&quot;</span><span class="p">][</span><span class="s2">&quot;header_from&quot;</span><span class="p">],</span>
<span class="n">envelope_from</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;identifiers&quot;</span><span class="p">][</span><span class="s2">&quot;envelope_from&quot;</span><span class="p">],</span>
<span class="n">envelope_to</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s2">&quot;identifiers&quot;</span><span class="p">][</span><span class="s2">&quot;envelope_to&quot;</span><span class="p">],</span>
<span class="n">np</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;np&quot;</span><span class="p">),</span>
<span class="n">testing</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;testing&quot;</span><span class="p">),</span>
<span class="n">discovery_method</span><span class="o">=</span><span class="n">aggregate_report</span><span class="p">[</span><span class="s2">&quot;policy_published&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;discovery_method&quot;</span>
<span class="p">),</span>
<span class="n">generator</span><span class="o">=</span><span class="n">metadata</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;generator&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">override</span> <span class="ow">in</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;policy_evaluated&quot;</span><span class="p">][</span><span class="s2">&quot;policy_override_reasons&quot;</span><span class="p">]:</span>
<span class="n">agg_doc</span><span class="o">.</span><span class="n">add_policy_override</span><span class="p">(</span>
<span class="n">type_</span><span class="o">=</span><span class="n">override</span><span class="p">[</span><span class="s2">&quot;type&quot;</span><span class="p">],</span> <span class="n">comment</span><span class="o">=</span><span class="n">override</span><span class="p">[</span><span class="s2">&quot;comment&quot;</span><span class="p">]</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">dkim_result</span> <span class="ow">in</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;auth_results&quot;</span><span class="p">][</span><span class="s2">&quot;dkim&quot;</span><span class="p">]:</span>
<span class="n">agg_doc</span><span class="o">.</span><span class="n">add_dkim_result</span><span class="p">(</span>
<span class="n">domain</span><span class="o">=</span><span class="n">dkim_result</span><span class="p">[</span><span class="s2">&quot;domain&quot;</span><span class="p">],</span>
<span class="n">selector</span><span class="o">=</span><span class="n">dkim_result</span><span class="p">[</span><span class="s2">&quot;selector&quot;</span><span class="p">],</span>
<span class="n">result</span><span class="o">=</span><span class="n">dkim_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">],</span>
<span class="n">human_result</span><span class="o">=</span><span class="n">dkim_result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;human_result&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">spf_result</span> <span class="ow">in</span> <span class="n">record</span><span class="p">[</span><span class="s2">&quot;auth_results&quot;</span><span class="p">][</span><span class="s2">&quot;spf&quot;</span><span class="p">]:</span>
<span class="n">agg_doc</span><span class="o">.</span><span class="n">add_spf_result</span><span class="p">(</span>
<span class="n">domain</span><span class="o">=</span><span class="n">spf_result</span><span class="p">[</span><span class="s2">&quot;domain&quot;</span><span class="p">],</span>
<span class="n">scope</span><span class="o">=</span><span class="n">spf_result</span><span class="p">[</span><span class="s2">&quot;scope&quot;</span><span class="p">],</span>
<span class="n">result</span><span class="o">=</span><span class="n">spf_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">],</span>
<span class="n">human_result</span><span class="o">=</span><span class="n">spf_result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;human_result&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="n">index</span> <span class="o">=</span> <span class="s2">&quot;dmarc_aggregate&quot;</span>
<span class="k">if</span> <span class="n">index_suffix</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s2">_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="k">if</span> <span class="n">index_prefix</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index_prefix</span><span class="si">}{</span><span class="n">index</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">index_date</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">index_settings</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span>
<span class="n">number_of_shards</span><span class="o">=</span><span class="n">number_of_shards</span><span class="p">,</span> <span class="n">number_of_replicas</span><span class="o">=</span><span class="n">number_of_replicas</span>
<span class="p">)</span>
<span class="n">create_indexes</span><span class="p">([</span><span class="n">index</span><span class="p">],</span> <span class="n">index_settings</span><span class="p">)</span>
<span class="n">agg_doc</span><span class="o">.</span><span class="n">meta</span><span class="o">.</span><span class="n">index</span> <span class="o">=</span> <span class="n">index</span> <span class="c1"># pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">agg_doc</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ElasticsearchError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Elasticsearch error: </span><span class="si">{</span><span class="n">e</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="save_failure_report_to_elasticsearch">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.save_failure_report_to_elasticsearch">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">save_failure_report_to_elasticsearch</span><span class="p">(</span>
<span class="n">failure_report</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span>
<span class="n">index_suffix</span><span class="p">:</span> <span class="n">Any</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">index_prefix</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">monthly_indexes</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">number_of_shards</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span>
<span class="n">number_of_replicas</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Saves a parsed DMARC failure report to Elasticsearch</span>
<span class="sd"> Args:</span>
<span class="sd"> failure_report (dict): A parsed failure report</span>
<span class="sd"> index_suffix (str): The suffix of the name of the index to save to</span>
<span class="sd"> index_prefix (str): The prefix of the name of the index to save to</span>
<span class="sd"> monthly_indexes (bool): Use monthly indexes instead of daily</span>
<span class="sd"> indexes</span>
<span class="sd"> number_of_shards (int): The number of shards to use in the index</span>
<span class="sd"> number_of_replicas (int): The number of replicas to use in the</span>
<span class="sd"> index</span>
<span class="sd"> Raises:</span>
<span class="sd"> AlreadySaved</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;Saving failure report to Elasticsearch&quot;</span><span class="p">)</span>
<span class="n">failure_report</span> <span class="o">=</span> <span class="n">failure_report</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">sample_date</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;date&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">sample_date</span> <span class="o">=</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;date&quot;</span><span class="p">]</span>
<span class="n">sample_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span><span class="n">sample_date</span><span class="p">)</span>
<span class="n">original_headers</span> <span class="o">=</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;headers&quot;</span><span class="p">]</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">original_header</span> <span class="ow">in</span> <span class="n">original_headers</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="n">original_header</span><span class="o">.</span><span class="n">lower</span><span class="p">()]</span> <span class="o">=</span> <span class="n">original_headers</span><span class="p">[</span><span class="n">original_header</span><span class="p">]</span>
<span class="c1"># arrival_date_utc is a UTC wall-clock string; without assume_utc the</span>
<span class="c1"># naive .timestamp() below would interpret it as local time and skew</span>
<span class="c1"># the epoch by the host&#39;s UTC offset.</span>
<span class="n">arrival_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span>
<span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;arrival_date_utc&quot;</span><span class="p">],</span> <span class="n">assume_utc</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">arrival_date_epoch_milliseconds</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">arrival_date</span><span class="o">.</span><span class="n">timestamp</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">)</span>
<span class="k">if</span> <span class="n">index_suffix</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;dmarc_failure_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">*,dmarc_forensic_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">*&quot;</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="s2">&quot;dmarc_failure*,dmarc_forensic*&quot;</span>
<span class="k">if</span> <span class="n">index_prefix</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="s2">&quot;,&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index_prefix</span><span class="si">}{</span><span class="n">part</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">part</span> <span class="ow">in</span> <span class="n">search_index</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">search</span> <span class="o">=</span> <span class="n">Search</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="n">search_index</span><span class="p">)</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">arrival_date</span><span class="o">=</span><span class="n">arrival_date_epoch_milliseconds</span><span class="p">)))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">from_</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">to_</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">subject</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="s2">&quot;from&quot;</span> <span class="ow">in</span> <span class="n">headers</span><span class="p">:</span>
<span class="c1"># We convert the FROM header from a string list to a flat string.</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&quot;&quot;</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot; &lt;&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">])</span> <span class="o">+</span> <span class="s2">&quot;&gt;&quot;</span>
<span class="n">from_</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="n">from_</span><span class="p">[</span><span class="s2">&quot;sample.headers.from&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;from&quot;</span><span class="p">]</span>
<span class="n">from_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="n">from_</span><span class="p">))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">q</span> <span class="o">&amp;</span> <span class="n">from_query</span>
<span class="k">if</span> <span class="s2">&quot;to&quot;</span> <span class="ow">in</span> <span class="n">headers</span><span class="p">:</span>
<span class="c1"># We convert the TO header from a string list to a flat string.</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&quot;&quot;</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot; &lt;&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">])</span> <span class="o">+</span> <span class="s2">&quot;&gt;&quot;</span>
<span class="n">to_</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="n">to_</span><span class="p">[</span><span class="s2">&quot;sample.headers.to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;to&quot;</span><span class="p">]</span>
<span class="n">to_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="n">to_</span><span class="p">))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">q</span> <span class="o">&amp;</span> <span class="n">to_query</span>
<span class="k">if</span> <span class="s2">&quot;reply-to&quot;</span> <span class="ow">in</span> <span class="n">headers</span><span class="p">:</span>
<span class="c1"># Flatten the Reply-To header to a string so it can be displayed</span>
<span class="c1"># and aggregated like From/To. Only the first address is used,</span>
<span class="c1"># matching the From/To handling above. Not part of the dedup</span>
<span class="c1"># query.</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&quot;&quot;</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot; &lt;&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">headers</span><span class="p">[</span><span class="s2">&quot;reply-to&quot;</span><span class="p">])</span> <span class="o">+</span> <span class="s2">&quot;&gt;&quot;</span>
<span class="k">if</span> <span class="s2">&quot;subject&quot;</span> <span class="ow">in</span> <span class="n">headers</span><span class="p">:</span>
<span class="n">subject</span> <span class="o">=</span> <span class="n">headers</span><span class="p">[</span><span class="s2">&quot;subject&quot;</span><span class="p">]</span>
<span class="n">subject_query</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;match_phrase&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;sample.headers.subject&quot;</span><span class="p">:</span> <span class="n">subject</span><span class="p">}}</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">q</span> <span class="o">&amp;</span> <span class="n">Q</span><span class="p">(</span><span class="n">subject_query</span><span class="p">)</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">search</span><span class="o">.</span><span class="n">query</span> <span class="o">=</span> <span class="n">q</span> <span class="c1"># pyright: ignore[reportAttributeAccessIssue]</span>
<span class="n">existing</span> <span class="o">=</span> <span class="n">search</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">existing</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AlreadySaved</span><span class="p">(</span>
<span class="s2">&quot;A failure sample to </span><span class="si">{}</span><span class="s2"> from </span><span class="si">{}</span><span class="s2"> &quot;</span>
<span class="s2">&quot;with a subject of </span><span class="si">{}</span><span class="s2"> and arrival date of </span><span class="si">{}</span><span class="s2"> &quot;</span>
<span class="s2">&quot;already exists in &quot;</span>
<span class="s2">&quot;Elasticsearch&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">to_</span><span class="p">,</span> <span class="n">from_</span><span class="p">,</span> <span class="n">subject</span><span class="p">,</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;arrival_date_utc&quot;</span><span class="p">]</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="n">parsed_sample</span> <span class="o">=</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">]</span>
<span class="n">sample</span> <span class="o">=</span> <span class="n">_FailureSampleDoc</span><span class="p">(</span>
<span class="n">raw</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;sample&quot;</span><span class="p">],</span>
<span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span>
<span class="n">headers_only</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;sample_headers_only&quot;</span><span class="p">],</span>
<span class="n">date</span><span class="o">=</span><span class="n">sample_date</span><span class="p">,</span>
<span class="n">subject</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;subject&quot;</span><span class="p">],</span>
<span class="n">filename_safe_subject</span><span class="o">=</span><span class="n">parsed_sample</span><span class="p">[</span><span class="s2">&quot;filename_safe_subject&quot;</span><span class="p">],</span>
<span class="n">body</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;body&quot;</span><span class="p">],</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">address</span> <span class="ow">in</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;to&quot;</span><span class="p">]:</span>
<span class="n">sample</span><span class="o">.</span><span class="n">add_to</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;display_name&quot;</span><span class="p">],</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;address&quot;</span><span class="p">])</span>
<span class="k">for</span> <span class="n">address</span> <span class="ow">in</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;reply_to&quot;</span><span class="p">]:</span>
<span class="n">sample</span><span class="o">.</span><span class="n">add_reply_to</span><span class="p">(</span>
<span class="n">display_name</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;display_name&quot;</span><span class="p">],</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;address&quot;</span><span class="p">]</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">address</span> <span class="ow">in</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;cc&quot;</span><span class="p">]:</span>
<span class="n">sample</span><span class="o">.</span><span class="n">add_cc</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;display_name&quot;</span><span class="p">],</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;address&quot;</span><span class="p">])</span>
<span class="k">for</span> <span class="n">address</span> <span class="ow">in</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;bcc&quot;</span><span class="p">]:</span>
<span class="n">sample</span><span class="o">.</span><span class="n">add_bcc</span><span class="p">(</span><span class="n">display_name</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;display_name&quot;</span><span class="p">],</span> <span class="n">address</span><span class="o">=</span><span class="n">address</span><span class="p">[</span><span class="s2">&quot;address&quot;</span><span class="p">])</span>
<span class="k">for</span> <span class="n">attachment</span> <span class="ow">in</span> <span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;parsed_sample&quot;</span><span class="p">][</span><span class="s2">&quot;attachments&quot;</span><span class="p">]:</span>
<span class="n">sample</span><span class="o">.</span><span class="n">add_attachment</span><span class="p">(</span>
<span class="n">filename</span><span class="o">=</span><span class="n">attachment</span><span class="p">[</span><span class="s2">&quot;filename&quot;</span><span class="p">],</span>
<span class="n">content_type</span><span class="o">=</span><span class="n">attachment</span><span class="p">[</span><span class="s2">&quot;mail_content_type&quot;</span><span class="p">],</span>
<span class="n">sha256</span><span class="o">=</span><span class="n">attachment</span><span class="p">[</span><span class="s2">&quot;sha256&quot;</span><span class="p">],</span>
<span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">failure_doc</span> <span class="o">=</span> <span class="n">_FailureReportDoc</span><span class="p">(</span>
<span class="n">feedback_type</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;feedback_type&quot;</span><span class="p">],</span>
<span class="n">user_agent</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;user_agent&quot;</span><span class="p">],</span>
<span class="n">version</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;version&quot;</span><span class="p">],</span>
<span class="n">original_mail_from</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;original_mail_from&quot;</span><span class="p">],</span>
<span class="n">arrival_date</span><span class="o">=</span><span class="n">arrival_date_epoch_milliseconds</span><span class="p">,</span>
<span class="n">domain</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;reported_domain&quot;</span><span class="p">],</span>
<span class="n">original_envelope_id</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;original_envelope_id&quot;</span><span class="p">],</span>
<span class="n">authentication_results</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;authentication_results&quot;</span><span class="p">],</span>
<span class="n">delivery_results</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;delivery_result&quot;</span><span class="p">],</span>
<span class="n">source_ip_address</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;ip_address&quot;</span><span class="p">],</span>
<span class="n">source_country</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;country&quot;</span><span class="p">],</span>
<span class="n">source_reverse_dns</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;reverse_dns&quot;</span><span class="p">],</span>
<span class="n">source_base_domain</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;base_domain&quot;</span><span class="p">],</span>
<span class="n">source_asn</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;asn&quot;</span><span class="p">],</span>
<span class="n">source_as_name</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;as_name&quot;</span><span class="p">],</span>
<span class="n">source_as_domain</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;source&quot;</span><span class="p">][</span><span class="s2">&quot;as_domain&quot;</span><span class="p">],</span>
<span class="n">authentication_mechanisms</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;authentication_mechanisms&quot;</span><span class="p">],</span>
<span class="n">auth_failure</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;auth_failure&quot;</span><span class="p">],</span>
<span class="n">dkim_domain</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;dkim_domain&quot;</span><span class="p">],</span>
<span class="n">original_rcpt_to</span><span class="o">=</span><span class="n">failure_report</span><span class="p">[</span><span class="s2">&quot;original_rcpt_to&quot;</span><span class="p">],</span>
<span class="n">sample</span><span class="o">=</span><span class="n">sample</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">index</span> <span class="o">=</span> <span class="s2">&quot;dmarc_failure&quot;</span>
<span class="k">if</span> <span class="n">index_suffix</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s2">_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="k">if</span> <span class="n">index_prefix</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index_prefix</span><span class="si">}{</span><span class="n">index</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="k">if</span> <span class="n">monthly_indexes</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">arrival_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">arrival_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">index_date</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">index_settings</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span>
<span class="n">number_of_shards</span><span class="o">=</span><span class="n">number_of_shards</span><span class="p">,</span> <span class="n">number_of_replicas</span><span class="o">=</span><span class="n">number_of_replicas</span>
<span class="p">)</span>
<span class="n">create_indexes</span><span class="p">([</span><span class="n">index</span><span class="p">],</span> <span class="n">index_settings</span><span class="p">)</span>
<span class="n">failure_doc</span><span class="o">.</span><span class="n">meta</span><span class="o">.</span><span class="n">index</span> <span class="o">=</span> <span class="n">index</span> <span class="c1"># pyright: ignore[reportAttributeAccessIssue, reportOptionalMemberAccess]</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">failure_doc</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ElasticsearchError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Elasticsearch error: </span><span class="si">{</span><span class="n">e</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">KeyError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">InvalidFailureReport</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Failure report missing required field: </span><span class="si">{</span><span class="n">e</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="save_smtp_tls_report_to_elasticsearch">
<a class="viewcode-back" href="../../api.html#parsedmarc.elastic.save_smtp_tls_report_to_elasticsearch">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">save_smtp_tls_report_to_elasticsearch</span><span class="p">(</span>
<span class="n">report</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span>
<span class="n">index_suffix</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">index_prefix</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">monthly_indexes</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">number_of_shards</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span>
<span class="n">number_of_replicas</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Saves a parsed SMTP TLS report to Elasticsearch</span>
<span class="sd"> Args:</span>
<span class="sd"> report (dict): A parsed SMTP TLS report</span>
<span class="sd"> index_suffix (str): The suffix of the name of the index to save to</span>
<span class="sd"> index_prefix (str): The prefix of the name of the index to save to</span>
<span class="sd"> monthly_indexes (bool): Use monthly indexes instead of daily indexes</span>
<span class="sd"> number_of_shards (int): The number of shards to use in the index</span>
<span class="sd"> number_of_replicas (int): The number of replicas to use in the index</span>
<span class="sd"> Raises:</span>
<span class="sd"> AlreadySaved</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;Saving smtp tls report to Elasticsearch&quot;</span><span class="p">)</span>
<span class="n">org_name</span> <span class="o">=</span> <span class="n">report</span><span class="p">[</span><span class="s2">&quot;organization_name&quot;</span><span class="p">]</span>
<span class="n">report_id</span> <span class="o">=</span> <span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_id&quot;</span><span class="p">]</span>
<span class="n">begin_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">],</span> <span class="n">to_utc</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">end_date</span> <span class="o">=</span> <span class="n">human_timestamp_to_datetime</span><span class="p">(</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">],</span> <span class="n">to_utc</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">begin_date_human</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2"> %H:%M:%SZ&quot;</span><span class="p">)</span>
<span class="n">end_date_human</span> <span class="o">=</span> <span class="n">end_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2"> %H:%M:%SZ&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">monthly_indexes</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">index_date</span> <span class="o">=</span> <span class="n">begin_date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">report</span> <span class="o">=</span> <span class="n">report</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">report</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">begin_date</span>
<span class="n">report</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">end_date</span>
<span class="n">org_name_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">org_name</span><span class="o">=</span><span class="n">org_name</span><span class="p">)))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">report_id_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match_phrase</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">report_id</span><span class="o">=</span><span class="n">report_id</span><span class="p">)))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">begin_date_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">date_begin</span><span class="o">=</span><span class="n">begin_date</span><span class="p">)))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="n">end_date_query</span> <span class="o">=</span> <span class="n">Q</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">match</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">date_end</span><span class="o">=</span><span class="n">end_date</span><span class="p">)))</span> <span class="c1"># pyright: ignore[reportArgumentType]</span>
<span class="k">if</span> <span class="n">index_suffix</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;smtp_tls_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">*&quot;</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="s2">&quot;smtp_tls*&quot;</span>
<span class="k">if</span> <span class="n">index_prefix</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">search_index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index_prefix</span><span class="si">}{</span><span class="n">search_index</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">search</span> <span class="o">=</span> <span class="n">Search</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="n">search_index</span><span class="p">)</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">org_name_query</span> <span class="o">&amp;</span> <span class="n">report_id_query</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">query</span> <span class="o">&amp;</span> <span class="n">begin_date_query</span> <span class="o">&amp;</span> <span class="n">end_date_query</span>
<span class="n">search</span><span class="o">.</span><span class="n">query</span> <span class="o">=</span> <span class="n">query</span> <span class="c1"># pyright: ignore[reportAttributeAccessIssue]</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">existing</span> <span class="o">=</span> <span class="n">search</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">error_</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ElasticsearchError</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Elasticsearch&#39;s search for existing report error: </span><span class="si">{</span><span class="n">error_</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">existing</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AlreadySaved</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;An SMTP TLS report ID </span><span class="si">{</span><span class="n">report_id</span><span class="si">}</span><span class="s2"> from &quot;</span>
<span class="sa">f</span><span class="s2">&quot; </span><span class="si">{</span><span class="n">org_name</span><span class="si">}</span><span class="s2"> with a date range of &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">begin_date_human</span><span class="si">}</span><span class="s2"> UTC to &quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">end_date_human</span><span class="si">}</span><span class="s2"> UTC already &quot;</span>
<span class="s2">&quot;exists in Elasticsearch&quot;</span>
<span class="p">)</span>
<span class="n">index</span> <span class="o">=</span> <span class="s2">&quot;smtp_tls&quot;</span>
<span class="k">if</span> <span class="n">index_suffix</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s2">_</span><span class="si">{</span><span class="n">index_suffix</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="k">if</span> <span class="n">index_prefix</span><span class="p">:</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index_prefix</span><span class="si">}{</span><span class="n">index</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">index</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">index_date</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">index_settings</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span>
<span class="n">number_of_shards</span><span class="o">=</span><span class="n">number_of_shards</span><span class="p">,</span> <span class="n">number_of_replicas</span><span class="o">=</span><span class="n">number_of_replicas</span>
<span class="p">)</span>
<span class="n">smtp_tls_doc</span> <span class="o">=</span> <span class="n">_SMTPTLSReportDoc</span><span class="p">(</span>
<span class="n">org_name</span><span class="o">=</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;organization_name&quot;</span><span class="p">],</span>
<span class="n">date_range</span><span class="o">=</span><span class="p">[</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">],</span> <span class="n">report</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">]],</span>
<span class="n">date_begin</span><span class="o">=</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">],</span>
<span class="n">date_end</span><span class="o">=</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">],</span>
<span class="n">contact_info</span><span class="o">=</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;contact_info&quot;</span><span class="p">],</span>
<span class="n">report_id</span><span class="o">=</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_id&quot;</span><span class="p">],</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">policy</span> <span class="ow">in</span> <span class="n">report</span><span class="p">[</span><span class="s2">&quot;policies&quot;</span><span class="p">]:</span>
<span class="n">policy_strings</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">mx_host_patterns</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="s2">&quot;policy_strings&quot;</span> <span class="ow">in</span> <span class="n">policy</span><span class="p">:</span>
<span class="n">policy_strings</span> <span class="o">=</span> <span class="n">policy</span><span class="p">[</span><span class="s2">&quot;policy_strings&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;mx_host_patterns&quot;</span> <span class="ow">in</span> <span class="n">policy</span><span class="p">:</span>
<span class="n">mx_host_patterns</span> <span class="o">=</span> <span class="n">policy</span><span class="p">[</span><span class="s2">&quot;mx_host_patterns&quot;</span><span class="p">]</span>
<span class="c1"># policies_combined/failure_details_combined: see the field</span>
<span class="c1"># declarations on _SMTPTLSReportDoc and issue #169. policies and</span>
<span class="c1"># their failure_details are object arrays with the same</span>
<span class="c1"># cross-product problem as dkim_results/spf_results, so dashboards</span>
<span class="c1"># aggregate these composed strings instead of the raw subfields.</span>
<span class="n">policy_domain_combined</span> <span class="o">=</span> <span class="n">policy</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;policy_domain&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&quot;none&quot;</span>
<span class="n">policy_type_combined</span> <span class="o">=</span> <span class="n">policy</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;policy_type&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&quot;none&quot;</span>
<span class="n">smtp_tls_doc</span><span class="o">.</span><span class="n">policies_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">policy_domain_combined</span><span class="si">}</span><span class="s2"> / </span><span class="si">{</span><span class="n">policy_type_combined</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">policy_doc</span> <span class="o">=</span> <span class="n">_SMTPTLSPolicyDoc</span><span class="p">(</span>
<span class="n">policy_domain</span><span class="o">=</span><span class="n">policy</span><span class="p">[</span><span class="s2">&quot;policy_domain&quot;</span><span class="p">],</span>
<span class="n">policy_type</span><span class="o">=</span><span class="n">policy</span><span class="p">[</span><span class="s2">&quot;policy_type&quot;</span><span class="p">],</span>
<span class="n">successful_session_count</span><span class="o">=</span><span class="n">policy</span><span class="p">[</span><span class="s2">&quot;successful_session_count&quot;</span><span class="p">],</span>
<span class="n">failed_session_count</span><span class="o">=</span><span class="n">policy</span><span class="p">[</span><span class="s2">&quot;failed_session_count&quot;</span><span class="p">],</span>
<span class="n">policy_string</span><span class="o">=</span><span class="n">policy_strings</span><span class="p">,</span>
<span class="n">mx_host_patterns</span><span class="o">=</span><span class="n">mx_host_patterns</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">if</span> <span class="s2">&quot;failure_details&quot;</span> <span class="ow">in</span> <span class="n">policy</span><span class="p">:</span>
<span class="k">for</span> <span class="n">failure_detail</span> <span class="ow">in</span> <span class="n">policy</span><span class="p">[</span><span class="s2">&quot;failure_details&quot;</span><span class="p">]:</span>
<span class="n">receiving_mx_hostname</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">additional_information_uri</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">failure_reason_code</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">ip_address</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">receiving_ip</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">receiving_mx_helo</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">sending_mta_ip</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="s2">&quot;receiving_mx_hostname&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">receiving_mx_hostname</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;receiving_mx_hostname&quot;</span><span class="p">]</span>
<span class="c1"># The parser&#39;s key is additional_info_uri (see</span>
<span class="c1"># SMTPTLSFailureDetailsOptional in types.py); accept the</span>
<span class="c1"># long-form key too for dicts built by other callers.</span>
<span class="k">if</span> <span class="s2">&quot;additional_info_uri&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">additional_information_uri</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;additional_info_uri&quot;</span><span class="p">]</span>
<span class="k">elif</span> <span class="s2">&quot;additional_information_uri&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">additional_information_uri</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span>
<span class="s2">&quot;additional_information_uri&quot;</span>
<span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;failure_reason_code&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">failure_reason_code</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;failure_reason_code&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;ip_address&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">ip_address</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;ip_address&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;receiving_ip&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">receiving_ip</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;receiving_ip&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;receiving_mx_helo&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">receiving_mx_helo</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;receiving_mx_helo&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;sending_mta_ip&quot;</span> <span class="ow">in</span> <span class="n">failure_detail</span><span class="p">:</span>
<span class="n">sending_mta_ip</span> <span class="o">=</span> <span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;sending_mta_ip&quot;</span><span class="p">]</span>
<span class="n">policy_doc</span><span class="o">.</span><span class="n">add_failure_details</span><span class="p">(</span>
<span class="n">result_type</span><span class="o">=</span><span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;result_type&quot;</span><span class="p">],</span>
<span class="n">ip_address</span><span class="o">=</span><span class="n">ip_address</span><span class="p">,</span>
<span class="n">receiving_ip</span><span class="o">=</span><span class="n">receiving_ip</span><span class="p">,</span>
<span class="n">receiving_mx_helo</span><span class="o">=</span><span class="n">receiving_mx_helo</span><span class="p">,</span>
<span class="n">failed_session_count</span><span class="o">=</span><span class="n">failure_detail</span><span class="p">[</span><span class="s2">&quot;failed_session_count&quot;</span><span class="p">],</span>
<span class="n">sending_mta_ip</span><span class="o">=</span><span class="n">sending_mta_ip</span><span class="p">,</span>
<span class="n">receiving_mx_hostname</span><span class="o">=</span><span class="n">receiving_mx_hostname</span><span class="p">,</span>
<span class="n">additional_information_uri</span><span class="o">=</span><span class="n">additional_information_uri</span><span class="p">,</span>
<span class="n">failure_reason_code</span><span class="o">=</span><span class="n">failure_reason_code</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">smtp_tls_doc</span><span class="o">.</span><span class="n">failure_details_combined</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">{}</span><span class="s2"> / </span><span class="si">{}</span><span class="s2"> / </span><span class="si">{}</span><span class="s2"> / </span><span class="si">{}</span><span class="s2"> / </span><span class="si">{}</span><span class="s2"> / </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">policy_domain_combined</span><span class="p">,</span>
<span class="n">policy_type_combined</span><span class="p">,</span>
<span class="n">failure_detail</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;result_type&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&quot;none&quot;</span><span class="p">,</span>
<span class="n">sending_mta_ip</span> <span class="ow">or</span> <span class="s2">&quot;none&quot;</span><span class="p">,</span>
<span class="n">receiving_ip</span> <span class="ow">or</span> <span class="s2">&quot;none&quot;</span><span class="p">,</span>
<span class="n">receiving_mx_hostname</span> <span class="ow">or</span> <span class="s2">&quot;none&quot;</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="n">smtp_tls_doc</span><span class="o">.</span><span class="n">policies</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">policy_doc</span><span class="p">)</span>
<span class="n">create_indexes</span><span class="p">([</span><span class="n">index</span><span class="p">],</span> <span class="n">index_settings</span><span class="p">)</span>
<span class="n">smtp_tls_doc</span><span class="o">.</span><span class="n">meta</span><span class="o">.</span><span class="n">index</span> <span class="o">=</span> <span class="n">index</span> <span class="c1"># pyright: ignore[reportOptionalMemberAccess, reportAttributeAccessIssue]</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">smtp_tls_doc</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ElasticsearchError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Elasticsearch error: </span><span class="si">{</span><span class="n">e</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
<span class="c1"># Backward-compatible aliases</span>
<span class="n">_ForensicSampleDoc</span> <span class="o">=</span> <span class="n">_FailureSampleDoc</span>
<span class="n">_ForensicReportDoc</span> <span class="o">=</span> <span class="n">_FailureReportDoc</span>
<span class="n">save_forensic_report_to_elasticsearch</span> <span class="o">=</span> <span class="n">save_failure_report_to_elasticsearch</span>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2018 - 2025, Sean Whalen and contributors.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink