amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-04-14 17:48:55 +00:00
Code Issues Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Sean Whalen
2023-10-11 17:50:59 -04:00
parent b984803ae9
commit 9bc2cfd4d5
27 changed files with 148 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
_modules/index.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Overview: module code &mdash; parsedmarc 8.6.1 documentation</title>
<title>Overview: module code &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -33,7 +33,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">

19
_modules/parsedmarc.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc &mdash; parsedmarc 8.6.1 documentation</title>
<title>parsedmarc &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -33,7 +33,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
@@ -97,11 +97,11 @@
<span class="kn">import</span> <span class="nn">tempfile</span>
<span class="kn">import</span> <span class="nn">xml.parsers.expat</span> <span class="k">as</span> <span class="nn">expat</span>
<span class="kn">import</span> <span class="nn">zipfile</span>
<span class="kn">import</span> <span class="nn">zlib</span>
<span class="kn">from</span> <span class="nn">base64</span> <span class="kn">import</span> <span class="n">b64decode</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="kn">from</span> <span class="nn">csv</span> <span class="kn">import</span> <span class="n">DictWriter</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">from</span> <span class="nn">gzip</span> <span class="kn">import</span> <span class="n">GzipFile</span>
<span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">BytesIO</span><span class="p">,</span> <span class="n">StringIO</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Callable</span>
@@ -118,7 +118,7 @@
<span class="kn">from</span> <span class="nn">parsedmarc.utils</span> <span class="kn">import</span> <span class="n">parse_email</span>
<span class="kn">from</span> <span class="nn">parsedmarc.utils</span> <span class="kn">import</span> <span class="n">timestamp_to_human</span><span class="p">,</span> <span class="n">human_timestamp_to_datetime</span>
<span class="n">__version__</span> <span class="o">=</span> <span class="s2">&quot;8.6.1&quot;</span>
<span class="n">__version__</span> <span class="o">=</span> <span class="s2">&quot;8.6.2&quot;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;parsedmarc v</span><span class="si">{0}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">__version__</span><span class="p">))</span>
@@ -228,7 +228,7 @@
<span class="k">else</span><span class="p">:</span>
<span class="n">auth_results</span> <span class="o">=</span> <span class="n">new_record</span><span class="p">[</span><span class="s2">&quot;auth_results&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">])</span> <span class="o">!=</span> <span class="nb">list</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">],</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">]]</span>
<span class="k">for</span> <span class="n">result</span> <span class="ow">in</span> <span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;dkim&quot;</span><span class="p">]:</span>
<span class="k">if</span> <span class="s2">&quot;domain&quot;</span> <span class="ow">in</span> <span class="n">result</span> <span class="ow">and</span> <span class="n">result</span><span class="p">[</span><span class="s2">&quot;domain&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
@@ -243,7 +243,7 @@
<span class="n">new_result</span><span class="p">[</span><span class="s2">&quot;result&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&quot;none&quot;</span>
<span class="n">new_record</span><span class="p">[</span><span class="s2">&quot;auth_results&quot;</span><span class="p">][</span><span class="s2">&quot;dkim&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_result</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">])</span> <span class="o">!=</span> <span class="nb">list</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">],</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">]]</span>
<span class="k">for</span> <span class="n">result</span> <span class="ow">in</span> <span class="n">auth_results</span><span class="p">[</span><span class="s2">&quot;spf&quot;</span><span class="p">]:</span>
<span class="n">new_result</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s2">&quot;domain&quot;</span><span class="p">,</span> <span class="n">result</span><span class="p">[</span><span class="s2">&quot;domain&quot;</span><span class="p">])])</span>
@@ -366,7 +366,7 @@
<span class="n">new_report_metadata</span><span class="p">[</span><span class="s2">&quot;begin_date&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">date_range</span><span class="p">[</span><span class="s2">&quot;begin&quot;</span><span class="p">]</span>
<span class="n">new_report_metadata</span><span class="p">[</span><span class="s2">&quot;end_date&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">date_range</span><span class="p">[</span><span class="s2">&quot;end&quot;</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;error&quot;</span> <span class="ow">in</span> <span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_metadata&quot;</span><span class="p">]:</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_metadata&quot;</span><span class="p">][</span><span class="s2">&quot;error&quot;</span><span class="p">])</span> <span class="o">!=</span> <span class="nb">list</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_metadata&quot;</span><span class="p">][</span><span class="s2">&quot;error&quot;</span><span class="p">],</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">errors</span> <span class="o">=</span> <span class="p">[</span><span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_metadata&quot;</span><span class="p">][</span><span class="s2">&quot;error&quot;</span><span class="p">]]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">errors</span> <span class="o">=</span> <span class="n">report</span><span class="p">[</span><span class="s2">&quot;report_metadata&quot;</span><span class="p">][</span><span class="s2">&quot;error&quot;</span><span class="p">]</span>
@@ -473,7 +473,8 @@
<span class="n">_zip</span> <span class="o">=</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="n">file_object</span><span class="p">)</span>
<span class="n">xml</span> <span class="o">=</span> <span class="n">_zip</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">_zip</span><span class="o">.</span><span class="n">namelist</span><span class="p">()[</span><span class="mi">0</span><span class="p">])</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">errors</span><span class="o">=</span><span class="s1">&#39;ignore&#39;</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">header</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">MAGIC_GZIP</span><span class="p">):</span>
<span class="n">xml</span> <span class="o">=</span> <span class="n">GzipFile</span><span class="p">(</span><span class="n">fileobj</span><span class="o">=</span><span class="n">file_object</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">errors</span><span class="o">=</span><span class="s1">&#39;ignore&#39;</span><span class="p">)</span>
<span class="n">xml</span> <span class="o">=</span> <span class="n">zlib</span><span class="o">.</span><span class="n">decompress</span><span class="p">(</span><span class="n">file_object</span><span class="o">.</span><span class="n">getvalue</span><span class="p">(),</span>
<span class="n">zlib</span><span class="o">.</span><span class="n">MAX_WBITS</span> <span class="o">|</span> <span class="mi">16</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">errors</span><span class="o">=</span><span class="s1">&#39;ignore&#39;</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">header</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">MAGIC_XML</span><span class="p">):</span>
<span class="n">xml</span> <span class="o">=</span> <span class="n">file_object</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">errors</span><span class="o">=</span><span class="s1">&#39;ignore&#39;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
@@ -908,7 +909,7 @@
<span class="k">for</span> <span class="n">part</span> <span class="ow">in</span> <span class="n">msg</span><span class="o">.</span><span class="n">walk</span><span class="p">():</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">part</span><span class="o">.</span><span class="n">get_content_type</span><span class="p">()</span>
<span class="n">payload</span> <span class="o">=</span> <span class="n">part</span><span class="o">.</span><span class="n">get_payload</span><span class="p">()</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">list</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">payload</span> <span class="o">=</span> <span class="p">[</span><span class="n">payload</span><span class="p">]</span>
<span class="n">payload</span> <span class="o">=</span> <span class="n">payload</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
<span class="k">if</span> <span class="n">content_type</span> <span class="o">==</span> <span class="s2">&quot;message/feedback-report&quot;</span><span class="p">:</span>

6
_modules/parsedmarc/elastic.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.elastic &mdash; parsedmarc 8.6.1 documentation</title>
<title>parsedmarc.elastic &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -33,7 +33,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
@@ -266,7 +266,7 @@
<span class="sd"> password (str): The password to use for authentication</span>
<span class="sd"> timeout (float): Timeout in seconds</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">hosts</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">list</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hosts</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="n">hosts</span><span class="p">]</span>
<span class="n">conn_params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;hosts&quot;</span><span class="p">:</span> <span class="n">hosts</span><span class="p">,</span>

8
_modules/parsedmarc/splunk.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.splunk &mdash; parsedmarc 8.6.1 documentation</title>
<title>parsedmarc.splunk &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -33,7 +33,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
@@ -149,7 +149,7 @@
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Saving aggregate reports to Splunk&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">aggregate_reports</span><span class="p">)</span> <span class="o">==</span> <span class="nb">dict</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">aggregate_reports</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="n">aggregate_reports</span> <span class="o">=</span> <span class="p">[</span><span class="n">aggregate_reports</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">aggregate_reports</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">1</span><span class="p">:</span>
@@ -215,7 +215,7 @@
<span class="sd"> to save in Splunk</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Saving forensic reports to Splunk&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">forensic_reports</span><span class="p">)</span> <span class="o">==</span> <span class="nb">dict</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">forensic_reports</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="n">forensic_reports</span> <span class="o">=</span> <span class="p">[</span><span class="n">forensic_reports</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">forensic_reports</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">1</span><span class="p">:</span>

12
_modules/parsedmarc/utils.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc.utils &mdash; parsedmarc 8.6.1 documentation</title>
<title>parsedmarc.utils &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -33,7 +33,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
@@ -116,7 +116,7 @@
<span class="kn">import</span> <span class="nn">publicsuffixlist</span>
<span class="kn">from</span> <span class="nn">parsedmarc.log</span> <span class="kn">import</span> <span class="n">logger</span>
<span class="kn">import</span> <span class="nn">parsedmarc.resources</span>
<span class="kn">import</span> <span class="nn">parsedmarc.resources.dbip</span>
<span class="n">parenthesis_regex</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;\s*\(.*\)\s*&#39;</span><span class="p">)</span>
@@ -357,7 +357,7 @@
<span class="k">break</span>
<span class="k">if</span> <span class="n">db_path</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">with</span> <span class="n">pkg_resources</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="n">parsedmarc</span><span class="o">.</span><span class="n">resources</span><span class="p">,</span>
<span class="k">with</span> <span class="n">pkg_resources</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="n">parsedmarc</span><span class="o">.</span><span class="n">resources</span><span class="o">.</span><span class="n">dbip</span><span class="p">,</span>
<span class="s2">&quot;dbip-country-lite.mmdb&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">path</span><span class="p">:</span>
<span class="n">db_path</span> <span class="o">=</span> <span class="n">path</span>
@@ -494,7 +494,7 @@
<span class="sd"> Returns:</span>
<span class="sd"> bool: A flag that indicates if the file is an Outlook MSG file</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="nb">type</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="o">==</span> <span class="nb">bytes</span> <span class="ow">and</span> <span class="n">content</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span>
<span class="k">return</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">)</span> <span class="ow">and</span> <span class="n">content</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span>
<span class="sa">b</span><span class="s2">&quot;</span><span class="se">\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1</span><span class="s2">&quot;</span><span class="p">)</span></div>
@@ -544,7 +544,7 @@
<span class="sd"> dict: Parsed email data</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="o">==</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">):</span>
<span class="k">if</span> <span class="n">is_outlook_msg</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">convert_outlook_msg</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">,</span> <span class="n">errors</span><span class="o">=</span><span class="s2">&quot;replace&quot;</span><span class="p">)</span>

9
_sources/elasticsearch.md.txt
View File

@@ -1,15 +1,13 @@
# Elasticsearch and Kibana
:::{note}
Splunk is also supported starting with `parsedmarc` 4.3.0
:::
To set up visual dashboards of DMARC data, install Elasticsearch and Kibana.
:::{note}
Elasticsearch and Kibana 6 or later are required
:::
## Installation
On Debian/Ubuntu based systems, run:
```bash
@@ -126,7 +124,7 @@ server.ssl.certificate: /etc/kibana/kibana.crt
server.ssl.key: /etc/kibana/kibana.key
```
::{note}
:::{note}
For more security, you can configure Kibana to use a local network connexion
to elasticsearch :
```text
@@ -136,6 +134,7 @@ elasticsearch.hosts: ['https://SERVER_IP:9200']
```text
elasticsearch.hosts: ['https://127.0.0.1:9200']
```
:::
```bash
sudo systemctl restart kibana

33
_sources/installation.md.txt
View File

@@ -1,15 +1,17 @@
# Installation
## Testing multiple report analyzers
## Prerequisites
`parsedmarc` works with Python 3 only.
### Testing multiple report analyzers
If you would like to test parsedmarc and another report processing
solution at the same time, you can have up to two `mailto` URIs in each of the rua and ruf
tags in your DMARC record, separated by commas.
:::
`parsedmarc` works with Python 3 only.
### Using a web proxy
:::{note}
If your system is behind a web proxy, you need to configure your system
to use that proxy. To do this, edit `/etc/environment` and add your
proxy details there, for example:
@@ -29,18 +31,17 @@ ftp_proxy=http://prox-server:3128
```
This will set the proxy up for use system-wide, including for `parsedmarc`.
:::
:::{warning}
### Using Microsoft Exchange
If your mail server is Microsoft Exchange, ensure that it is patched to at
least:
- Exchange Server 2010 Update Rollup 22 ([KB4295699](https://support.microsoft.com/KB/4295699))
- Exchange Server 2013 Cumulative Update 21 ([KB4099855](https://support.microsoft.com/KB/4099855))
- Exchange Server 2016 Cumulative Update 11 ([KB4134118](https://support.microsoft.com/kb/4134118))
:::
- Exchange Server 2010 Update Rollup 22 ([KB4295699])
- Exchange Server 2013 Cumulative Update 21 ([KB4099855])
- Exchange Server 2016 Cumulative Update 11 ([KB4134118])
## geoipupdate setup
### geoipupdate setup
:::{note}
Starting in `parsedmarc` 7.1.0, a static copy of the
@@ -84,9 +85,8 @@ The latest builds for Linux, macOS, and Windows can be downloaded
from the [geoipupdate releases page on GitHub].
On December 30th, 2019, MaxMind started requiring free accounts to
access the free Geolite2 databases, in order [to
comply with various privacy
regulations][to comply with various privacy regulations].
access the free Geolite2 databases, in order
[to comply with various privacy regulations].
Start by [registering for a free GeoLite2 account], and signing in.
@@ -146,7 +146,7 @@ sudo dnf install -y python39 python3-virtualenv python3-setuptools python3-devel
```
Python 3 installers for Windows and macOS can be found at
<https://www.python.org/downloads/>
<https://www.python.org/downloads/>.
Create a system user
@@ -191,6 +191,9 @@ On Debian or Ubuntu systems, run:
sudo apt-get install libemail-outlook-message-perl
```
[KB4295699]: https://support.microsoft.com/KB/4295699
[KB4099855]: https://support.microsoft.com/KB/4099855
[KB4134118]: https://support.microsoft.com/kb/4134118
[Component "contrib"]: https://wiki.debian.org/SourcesList#Component
[geoipupdate]: https://github.com/maxmind/geoipupdate
[geoipupdate releases page on github]: https://github.com/maxmind/geoipupdate/releases

34
_sources/usage.md.txt
View File

@@ -111,17 +111,17 @@ The full set of configuration options are:
- `forensic_json_filename` - str: filename for the forensic
JSON output file
- `ip_db_path` - str: An optional custom path to a MMDB file
- from MaxMind or DBIP
from MaxMind or DBIP
- `offline` - bool: Do not use online queries for geolocation
or DNS
- `nameservers` - str: A comma separated list of
DNS resolvers (Default: [Cloudflare's public resolvers])
- `nameservers` - str: A comma separated list of
DNS resolvers (Default: `[Cloudflare's public resolvers]`)
- `dns_timeout` - float: DNS timeout period
- `debug` - bool: Print debugging messages
- `silent` - bool: Only print errors (Default: True)
- `silent` - bool: Only print errors (Default: `True`)
- `log_file` - str: Write log messages to a file at this path
- `n_procs` - int: Number of process to run in parallel when
parsing in CLI mode (Default: 1)
parsing in CLI mode (Default: `1`)
- `chunk_size` - int: Number of files to give to each process
when running in parallel.
@@ -134,7 +134,7 @@ The full set of configuration options are:
- `reports_folder` - str: The mailbox folder (or label for
Gmail) where the incoming reports can be found
(Default: `INBOX`)
- `archive_folder` - str: The mailbox folder (or label for
- `archive_folder` - str: The mailbox folder (or label for
Gmail) to sort processed emails into (Default: `Archive`)
- `watch` - bool: Use the IMAP `IDLE` command to process
- messages as they arrive or poll MS Graph for new messages
@@ -165,7 +165,7 @@ The full set of configuration options are:
:::
- `ssl` - bool: Use an encrypted SSL/TLS connection
(Default: True)
(Default: `True`)
- `skip_certificate_verification` - bool: Skip certificate
verification (not recommended)
- `user` - str: The IMAP user
@@ -188,7 +188,7 @@ The full set of configuration options are:
- `token_file` - str: Path to save the token file
(Default: `.token`)
- `allow_unencrypted_storage` - bool: Allows the Azure Identity
module to fall back to unencrypted token cache (Default: False).
module to fall back to unencrypted token cache (Default: `False`).
Even if enabled, the cache will always try encrypted storage first.
:::{note}
@@ -245,14 +245,14 @@ The full set of configuration options are:
- `hosts` - str: A comma separated list of Kafka hosts
- `user` - str: The Kafka user
- `passsword` - str: The Kafka password
- `ssl` - bool: Use an encrypted SSL/TLS connection (Default: True)
- `ssl` - bool: Use an encrypted SSL/TLS connection (Default: `True`)
- `skip_certificate_verification` - bool: Skip certificate
verification (not recommended)
- `aggregate_topic` - str: The Kafka topic for aggregate reports
- `forensic_topic` - str: The Kafka topic for forensic reports
- `smtp`
- `host` - str: The SMTP hostname
- `port` - int: The SMTP port (Default: 25)
- `port` - int: The SMTP port (Default: `25`)
- `ssl` - bool: Require SSL/TLS instead of using STARTTLS
- `skip_certificate_verification` - bool: Skip certificate
verification (not recommended)
@@ -272,26 +272,26 @@ The full set of configuration options are:
:::
- `s3`
- `bucket` - str: The S3 bucket name
- `path` - str: The path to upload reports to (Default: /)
- `path` - str: The path to upload reports to (Default: `/`)
- `region_name` - str: The region name (Optional)
- `endpoint_url` - str: The endpoint URL (Optional)
- `access_key_id` - str: The access key id (Optional)
- `secret_access_key` - str: The secret access key (Optional)
- `syslog`
- `server` - str: The Syslog server name or IP address
- `port` - int: The UDP port to use (Default: 514)
- `port` - int: The UDP port to use (Default: `514`)
- `gmail_api`
- `credentials_file` - str: Path to file containing the
credentials, None to disable (Default: None)
credentials, None to disable (Default: `None`)
- `token_file` - str: Path to save the token file
(Default: .token)
(Default: `.token`)
- `include_spam_trash` - bool: Include messages in Spam and
Trash when searching reports (Default: False)
Trash when searching reports (Default: `False`)
- `scopes` - str: Comma separated list of scopes to use when
acquiring credentials
(Default: `https://www.googleapis.com/auth/gmail.modify`)
- `oauth2_port` - int: The TCP port for the local server to
listen on for the OAuth2 response (Default: 8080)
listen on for the OAuth2 response (Default: `8080`)
- `log_analytics`
- `client_id` - str: The app registration's client ID
- `client_secret` - str: The app registration's client secret
@@ -315,7 +315,7 @@ The `nameservers` option should only be used if your network
blocks DNS requests to outside resolvers.
:::
:::{warning}
:::{note}
`save_aggregate` and `save_forensic` are separate options
because you may not want to save forensic reports
(also known as failure reports) to your Elasticsearch instance,

2
_static/documentation_options.js
View File

@@ -1,6 +1,6 @@
var DOCUMENTATION_OPTIONS = {
URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
VERSION: '8.6.1',
VERSION: '8.6.2',
LANGUAGE: 'en',
COLLAPSE_INDEX: false,
BUILDER: 'html',

1
_static/pygments.css
View File

@@ -17,6 +17,7 @@ span.linenos.special { color: #000000; background-color: #ffffc0; padding-left:
.highlight .cs { color: #3D7B7B; font-style: italic } /* Comment.Special */
.highlight .gd { color: #A00000 } /* Generic.Deleted */
.highlight .ge { font-style: italic } /* Generic.Emph */
.highlight .ges { font-weight: bold; font-style: italic } /* Generic.EmphStrong */
.highlight .gr { color: #E40000 } /* Generic.Error */
.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.highlight .gi { color: #008400 } /* Generic.Inserted */

4
api.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>API reference &mdash; parsedmarc 8.6.1 documentation</title>
<title>API reference &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -35,7 +35,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

4
contributing.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Contributing to parsedmarc &mdash; parsedmarc 8.6.1 documentation</title>
<title>Contributing to parsedmarc &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

4
davmail.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 8.6.1 documentation</title>
<title>Accessing an inbox using OWA/EWS &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

4
dmarc.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Understanding DMARC &mdash; parsedmarc 8.6.1 documentation</title>
<title>Understanding DMARC &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

18
elasticsearch.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Elasticsearch and Kibana &mdash; parsedmarc 8.6.1 documentation</title>
<title>Elasticsearch and Kibana &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
@@ -52,6 +52,7 @@
<li class="toctree-l1"><a class="reference internal" href="usage.html">Using parsedmarc</a></li>
<li class="toctree-l1"><a class="reference internal" href="output.html">Sample outputs</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Elasticsearch and Kibana</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#installation">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="#upgrading-kibana-index-patterns">Upgrading Kibana index patterns</a></li>
<li class="toctree-l2"><a class="reference internal" href="#records-retention">Records retention</a></li>
</ul>
@@ -90,15 +91,13 @@
<section class="tex2jax_ignore mathjax_ignore" id="elasticsearch-and-kibana">
<h1>Elasticsearch and Kibana<a class="headerlink" href="#elasticsearch-and-kibana" title="Permalink to this heading"></a></h1>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Splunk is also supported starting with <code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> 4.3.0</p>
</div>
<p>To set up visual dashboards of DMARC data, install Elasticsearch and Kibana.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Elasticsearch and Kibana 6 or later are required</p>
</div>
<section id="installation">
<h2>Installation<a class="headerlink" href="#installation" title="Permalink to this heading"></a></h2>
<p>On Debian/Ubuntu based systems, run:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>apt-transport-https
wget<span class="w"> </span>-qO<span class="w"> </span>-<span class="w"> </span>https://artifacts.elastic.co/GPG-KEY-elasticsearch<span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>gpg<span class="w"> </span>--dearmor<span class="w"> </span>-o<span class="w"> </span>/usr/share/keyrings/elasticsearch-keyring.gpg
@@ -188,8 +187,9 @@ server.ssl.certificate: /etc/kibana/kibana.crt
server.ssl.key: /etc/kibana/kibana.key
</pre></div>
</div>
<p>::{note}
For more security, you can configure Kibana to use a local network connexion
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>For more security, you can configure Kibana to use a local network connexion
to elasticsearch :</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>elasticsearch.hosts: [&#39;https://SERVER_IP:9200&#39;]
</pre></div>
@@ -198,6 +198,7 @@ to elasticsearch :</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>elasticsearch.hosts: [&#39;https://127.0.0.1:9200&#39;]
</pre></div>
</div>
</div>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo<span class="w"> </span>systemctl<span class="w"> </span>restart<span class="w"> </span>kibana
</pre></div>
</div>
@@ -241,6 +242,7 @@ breaks them, as there are no permissions/access controls in Kibana without
the commercial <a class="reference external" href="https://www.elastic.co/products/x-pack">X-Pack</a>.</p>
<a class="reference external image-reference" href="_static/screenshots/saved-objects.png"><img alt="A screenshot of setting the Saved Objects Stack management UI in Kibana" class="align-center" src="_images/saved-objects.png" /></a>
<a class="reference external image-reference" href="_static/screenshots/confirm-overwrite.png"><img alt="A screenshot of the overwrite conformation prompt" class="align-center" src="_images/confirm-overwrite.png" /></a>
</section>
<section id="upgrading-kibana-index-patterns">
<h2>Upgrading Kibana index patterns<a class="headerlink" href="#upgrading-kibana-index-patterns" title="Permalink to this heading"></a></h2>
<p><code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> 5.0.0 makes some changes to the way data is indexed in

4
genindex.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Index &mdash; parsedmarc 8.6.1 documentation</title>
<title>Index &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -33,7 +33,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

8
index.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 8.6.1 documentation</title>
<title>parsedmarc documentation - Open source DMARC report analyzer and visualizer &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -35,7 +35,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
@@ -120,8 +120,7 @@ premade dashboards</p></li>
<p class="caption" role="heading"><span class="caption-text">Contents</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a><ul>
<li class="toctree-l2"><a class="reference internal" href="installation.html#testing-multiple-report-analyzers">Testing multiple report analyzers</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html#geoipupdate-setup">geoipupdate setup</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html#prerequisites">Prerequisites</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html#installing-parsedmarc">Installing parsedmarc</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html#optional-dependencies">Optional dependencies</a></li>
</ul>
@@ -138,6 +137,7 @@ premade dashboards</p></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="elasticsearch.html">Elasticsearch and Kibana</a><ul>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#installation">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#upgrading-kibana-index-patterns">Upgrading Kibana index patterns</a></li>
<li class="toctree-l2"><a class="reference internal" href="elasticsearch.html#records-retention">Records retention</a></li>
</ul>

43
installation.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Installation &mdash; parsedmarc 8.6.1 documentation</title>
<title>Installation &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
@@ -49,8 +49,13 @@
<p class="caption" role="heading"><span class="caption-text">Contents</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Installation</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#testing-multiple-report-analyzers">Testing multiple report analyzers</a></li>
<li class="toctree-l2"><a class="reference internal" href="#geoipupdate-setup">geoipupdate setup</a></li>
<li class="toctree-l2"><a class="reference internal" href="#prerequisites">Prerequisites</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#testing-multiple-report-analyzers">Testing multiple report analyzers</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-a-web-proxy">Using a web proxy</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-microsoft-exchange">Using Microsoft Exchange</a></li>
<li class="toctree-l3"><a class="reference internal" href="#geoipupdate-setup">geoipupdate setup</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#installing-parsedmarc">Installing parsedmarc</a></li>
<li class="toctree-l2"><a class="reference internal" href="#optional-dependencies">Optional dependencies</a></li>
</ul>
@@ -92,15 +97,17 @@
<section class="tex2jax_ignore mathjax_ignore" id="installation">
<h1>Installation<a class="headerlink" href="#installation" title="Permalink to this heading"></a></h1>
<section id="prerequisites">
<h2>Prerequisites<a class="headerlink" href="#prerequisites" title="Permalink to this heading"></a></h2>
<p><code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> works with Python 3 only.</p>
<section id="testing-multiple-report-analyzers">
<h2>Testing multiple report analyzers<a class="headerlink" href="#testing-multiple-report-analyzers" title="Permalink to this heading"></a></h2>
<h3>Testing multiple report analyzers<a class="headerlink" href="#testing-multiple-report-analyzers" title="Permalink to this heading"></a></h3>
<p>If you would like to test parsedmarc and another report processing
solution at the same time, you can have up to two <code class="docutils literal notranslate"><span class="pre">mailto</span></code> URIs in each of the rua and ruf
tags in your DMARC record, separated by commas.</p>
<div class="docutils">
<p><code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> works with Python 3 only.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
</section>
<section id="using-a-web-proxy">
<h3>Using a web proxy<a class="headerlink" href="#using-a-web-proxy" title="Permalink to this heading"></a></h3>
<p>If your system is behind a web proxy, you need to configure your system
to use that proxy. To do this, edit <code class="docutils literal notranslate"><span class="pre">/etc/environment</span></code> and add your
proxy details there, for example:</p>
@@ -116,10 +123,9 @@ proxy details there, for example:</p>
</pre></div>
</div>
<p>This will set the proxy up for use system-wide, including for <code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code>.</p>
</div>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
</section>
<section id="using-microsoft-exchange">
<h3>Using Microsoft Exchange<a class="headerlink" href="#using-microsoft-exchange" title="Permalink to this heading"></a></h3>
<p>If your mail server is Microsoft Exchange, ensure that it is patched to at
least:</p>
<ul class="simple">
@@ -127,10 +133,9 @@ least:</p>
<li><p>Exchange Server 2013 Cumulative Update 21 (<a class="reference external" href="https://support.microsoft.com/KB/4099855">KB4099855</a>)</p></li>
<li><p>Exchange Server 2016 Cumulative Update 11 (<a class="reference external" href="https://support.microsoft.com/kb/4134118">KB4134118</a>)</p></li>
</ul>
</div>
</section>
<section id="geoipupdate-setup">
<h2>geoipupdate setup<a class="headerlink" href="#geoipupdate-setup" title="Permalink to this heading"></a></h2>
<h3>geoipupdate setup<a class="headerlink" href="#geoipupdate-setup" title="Permalink to this heading"></a></h3>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Starting in <code class="docutils literal notranslate"><span class="pre">parsedmarc</span></code> 7.1.0, a static copy of the
@@ -165,9 +170,8 @@ sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> <
<p>The latest builds for Linux, macOS, and Windows can be downloaded
from the <a class="reference external" href="https://github.com/maxmind/geoipupdate/releases">geoipupdate releases page on GitHub</a>.</p>
<p>On December 30th, 2019, MaxMind started requiring free accounts to
access the free Geolite2 databases, in order <a class="reference external" href="https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/">to
comply with various privacy
regulations</a>.</p>
access the free Geolite2 databases, in order
<a class="reference external" href="https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/">to comply with various privacy regulations</a>.</p>
<p>Start by <a class="reference external" href="https://www.maxmind.com/en/geolite2/signup">registering for a free GeoLite2 account</a>, and signing in.</p>
<p>Then, navigate to the <a class="reference external" href="https://www.maxmind.com/en/accounts/current/license-key">License Keys</a> page under your account,
and create a new license key for the version of
@@ -201,6 +205,7 @@ job or scheduled task.</p>
<p>More information about <code class="docutils literal notranslate"><span class="pre">geoipupdate</span></code> can be found at the
<a class="reference external" href="https://dev.maxmind.com/geoip/geoipupdate/">MaxMind geoipupdate page</a>.</p>
</section>
</section>
<section id="installing-parsedmarc">
<h2>Installing parsedmarc<a class="headerlink" href="#installing-parsedmarc" title="Permalink to this heading"></a></h2>
<p>On Debian or Ubuntu systems, run:</p>
@@ -212,7 +217,7 @@ job or scheduled task.</p>
</pre></div>
</div>
<p>Python 3 installers for Windows and macOS can be found at
<a class="reference external" href="https://www.python.org/downloads/">https://www.python.org/downloads/</a></p>
<a class="reference external" href="https://www.python.org/downloads/">https://www.python.org/downloads/</a>.</p>
<p>Create a system user</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo<span class="w"> </span>mkdir<span class="w"> </span>/opt
sudo<span class="w"> </span>useradd<span class="w"> </span>parsedmarc<span class="w"> </span>-r<span class="w"> </span>-s<span class="w"> </span>/bin/false<span class="w"> </span>-m<span class="w"> </span>-b<span class="w"> </span>/opt

4
kibana.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using the Kibana dashboards &mdash; parsedmarc 8.6.1 documentation</title>
<title>Using the Kibana dashboards &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

4
mailing-lists.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>What about mailing lists? &mdash; parsedmarc 8.6.1 documentation</title>
<title>What about mailing lists? &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -34,7 +34,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

BIN
objects.inv
View File

Binary file not shown.

4
output.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Sample outputs &mdash; parsedmarc 8.6.1 documentation</title>
<title>Sample outputs &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

4
py-modindex.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Python Module Index &mdash; parsedmarc 8.6.1 documentation</title>
<title>Python Module Index &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

4
search.html
View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Search &mdash; parsedmarc 8.6.1 documentation</title>
<title>Search &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="#" method="get">

2
searchindex.js
View File

File diff suppressed because one or more lines are too long

4
splunk.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Splunk &mdash; parsedmarc 8.6.1 documentation</title>
<title>Splunk &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">

42
usage.html
View File

@@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Using parsedmarc &mdash; parsedmarc 8.6.1 documentation</title>
<title>Using parsedmarc &mdash; parsedmarc 8.6.2 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
@@ -36,7 +36,7 @@
parsedmarc
</a>
<div class="version">
8.6.1
8.6.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
@@ -198,18 +198,18 @@ payloads from results</p></li>
JSON output file</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">forensic_json_filename</span></code> - str: filename for the forensic
JSON output file</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ip_db_path</span></code> - str: An optional custom path to a MMDB file</p></li>
<li><p>from MaxMind or DBIP</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ip_db_path</span></code> - str: An optional custom path to a MMDB file
from MaxMind or DBIP</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">offline</span></code> - bool: Do not use online queries for geolocation
or DNS</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">nameservers</span></code> - str: A comma separated list of
DNS resolvers (Default: <a class="reference external" href="https://1.1.1.1/">Cloudflare’s public resolvers</a>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">nameservers</span></code> - str: A comma separated list of
DNS resolvers (Default: <code class="docutils literal notranslate"><span class="pre">[Cloudflare's</span> <span class="pre">public</span> <span class="pre">resolvers]</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">dns_timeout</span></code> - float: DNS timeout period</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">debug</span></code> - bool: Print debugging messages</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">silent</span></code> - bool: Only print errors (Default: True)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">silent</span></code> - bool: Only print errors (Default: <code class="docutils literal notranslate"><span class="pre">True</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">log_file</span></code> - str: Write log messages to a file at this path</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">n_procs</span></code> - int: Number of process to run in parallel when
parsing in CLI mode (Default: 1)</p></li>
parsing in CLI mode (Default: <code class="docutils literal notranslate"><span class="pre">1</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">chunk_size</span></code> - int: Number of files to give to each process
when running in parallel.</p>
<div class="admonition note">
@@ -225,7 +225,7 @@ performance when processing thousands of files</p>
<li><p><code class="docutils literal notranslate"><span class="pre">reports_folder</span></code> - str: The mailbox folder (or label for
Gmail) where the incoming reports can be found
(Default: <code class="docutils literal notranslate"><span class="pre">INBOX</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">archive_folder</span></code> - str: The mailbox folder (or label for
<li><p><code class="docutils literal notranslate"><span class="pre">archive_folder</span></code> - str: The mailbox folder (or label for
Gmail) to sort processed emails into (Default: <code class="docutils literal notranslate"><span class="pre">Archive</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">watch</span></code> - bool: Use the IMAP <code class="docutils literal notranslate"><span class="pre">IDLE</span></code> command to process</p></li>
<li><p>messages as they arrive or poll MS Graph for new messages</p></li>
@@ -259,7 +259,7 @@ section have been moved to the <code class="docutils literal notranslate"><span
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">ssl</span></code> - bool: Use an encrypted SSL/TLS connection
(Default: True)</p></li>
(Default: <code class="docutils literal notranslate"><span class="pre">True</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">skip_certificate_verification</span></code> - bool: Skip certificate
verification (not recommended)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">user</span></code> - str: The IMAP user</p></li>
@@ -285,7 +285,7 @@ could be a shared mailbox if the user has access to the mailbox</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">token_file</span></code> - str: Path to save the token file
(Default: <code class="docutils literal notranslate"><span class="pre">.token</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">allow_unencrypted_storage</span></code> - bool: Allows the Azure Identity
module to fall back to unencrypted token cache (Default: False).
module to fall back to unencrypted token cache (Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>).
Even if enabled, the cache will always try encrypted storage first.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
@@ -350,7 +350,7 @@ verification (not recommended)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">hosts</span></code> - str: A comma separated list of Kafka hosts</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">user</span></code> - str: The Kafka user</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">passsword</span></code> - str: The Kafka password</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ssl</span></code> - bool: Use an encrypted SSL/TLS connection (Default: True)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ssl</span></code> - bool: Use an encrypted SSL/TLS connection (Default: <code class="docutils literal notranslate"><span class="pre">True</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">skip_certificate_verification</span></code> - bool: Skip certificate
verification (not recommended)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">aggregate_topic</span></code> - str: The Kafka topic for aggregate reports</p></li>
@@ -360,7 +360,7 @@ verification (not recommended)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">smtp</span></code></p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">host</span></code> - str: The SMTP hostname</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">port</span></code> - int: The SMTP port (Default: 25)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">port</span></code> - int: The SMTP port (Default: <code class="docutils literal notranslate"><span class="pre">25</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ssl</span></code> - bool: Require SSL/TLS instead of using STARTTLS</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">skip_certificate_verification</span></code> - bool: Skip certificate
verification (not recommended)</p></li>
@@ -384,7 +384,7 @@ so use <code class="docutils literal notranslate"><span class="pre">%%</span></c
<li><p><code class="docutils literal notranslate"><span class="pre">s3</span></code></p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">bucket</span></code> - str: The S3 bucket name</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">path</span></code> - str: The path to upload reports to (Default: /)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">path</span></code> - str: The path to upload reports to (Default: <code class="docutils literal notranslate"><span class="pre">/</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">region_name</span></code> - str: The region name (Optional)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">endpoint_url</span></code> - str: The endpoint URL (Optional)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">access_key_id</span></code> - str: The access key id (Optional)</p></li>
@@ -394,22 +394,22 @@ so use <code class="docutils literal notranslate"><span class="pre">%%</span></c
<li><p><code class="docutils literal notranslate"><span class="pre">syslog</span></code></p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">server</span></code> - str: The Syslog server name or IP address</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">port</span></code> - int: The UDP port to use (Default: 514)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">port</span></code> - int: The UDP port to use (Default: <code class="docutils literal notranslate"><span class="pre">514</span></code>)</p></li>
</ul>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">gmail_api</span></code></p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">credentials_file</span></code> - str: Path to file containing the
credentials, None to disable (Default: None)</p></li>
credentials, None to disable (Default: <code class="docutils literal notranslate"><span class="pre">None</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">token_file</span></code> - str: Path to save the token file
(Default: .token)</p></li>
(Default: <code class="docutils literal notranslate"><span class="pre">.token</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">include_spam_trash</span></code> - bool: Include messages in Spam and
Trash when searching reports (Default: False)</p></li>
Trash when searching reports (Default: <code class="docutils literal notranslate"><span class="pre">False</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">scopes</span></code> - str: Comma separated list of scopes to use when
acquiring credentials
(Default: <code class="docutils literal notranslate"><span class="pre">https://www.googleapis.com/auth/gmail.modify</span></code>)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">oauth2_port</span></code> - int: The TCP port for the local server to
listen on for the OAuth2 response (Default: 8080)</p></li>
listen on for the OAuth2 response (Default: <code class="docutils literal notranslate"><span class="pre">8080</span></code>)</p></li>
</ul>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">log_analytics</span></code></p>
@@ -437,8 +437,8 @@ reliable than Google, Cisco OpenDNS, or even most local resolvers.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">nameservers</span></code> option should only be used if your network
blocks DNS requests to outside resolvers.</p>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><code class="docutils literal notranslate"><span class="pre">save_aggregate</span></code> and <code class="docutils literal notranslate"><span class="pre">save_forensic</span></code> are separate options
because you may not want to save forensic reports
(also known as failure reports) to your Elasticsearch instance,