mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-09-06 05:57:58 +00:00
Make the output and mailbox integrations optional extras (#888)
* Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Sean Whalen
co-authored by
Claude Fable 5
Copilot
Claude Fable 5
Copilot
parent
52be8850b2
commit
07bca1ad28
@@ -2,12 +2,12 @@
|
||||
|
||||
:::{note}
|
||||
Starting in 8.0.0, parsedmarc supports accessing Microsoft/Office 365
|
||||
inboxes via the Microsoft Graph API, which is preferred over Davmail.
|
||||
inboxes via the Microsoft Graph API, which is preferred over DavMail.
|
||||
:::
|
||||
|
||||
Some organizations do not allow IMAP or the Microsoft Graph API,
|
||||
and only support Exchange Web Services (EWS)/Outlook Web Access (OWA).
|
||||
In that case, Davmail will need to be set up
|
||||
In that case, DavMail will need to be set up
|
||||
as a local EWS/OWA IMAP gateway. It can even work where
|
||||
[Modern Auth/multi-factor authentication] is required.
|
||||
|
||||
@@ -22,7 +22,7 @@ Install Java:
|
||||
sudo apt-get install default-jre-headless
|
||||
```
|
||||
|
||||
Configure Davmail by creating a `davmail.properties` file
|
||||
Configure DavMail by creating a `davmail.properties` file
|
||||
|
||||
```properties
|
||||
# DavMail settings, see http://davmail.sourceforge.net/ for documentation
|
||||
@@ -135,7 +135,7 @@ Then, enable the service
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable parsedmarc.service
|
||||
sudo systemctl enable davmail.service
|
||||
sudo service davmail restart
|
||||
```
|
||||
|
||||
@@ -163,7 +163,7 @@ service davmail status
|
||||
:::{note}
|
||||
In the event of a crash, systemd will restart the service after 5
|
||||
minutes, but the `service davmail status` command will only show the
|
||||
logs for the current process. To vew the logs for previous runs as
|
||||
logs for the current process. To view the logs for previous runs as
|
||||
well as the current process (newest to oldest), run:
|
||||
|
||||
```bash
|
||||
@@ -174,8 +174,8 @@ journalctl -u davmail.service -r
|
||||
|
||||
## Configuring parsedmarc for DavMail
|
||||
|
||||
Because you are interacting with DavMail server over the loopback
|
||||
(i.e. `127.0.0.1`), add the following options to `parsedmarc.ini`
|
||||
Because you are interacting with the DavMail server over the loopback
|
||||
(i.e. `127.0.0.1`), add the following options to the `parsedmarc.ini`
|
||||
config file:
|
||||
|
||||
```ini
|
||||
|
||||
@@ -16,12 +16,12 @@ check out the sister project,
|
||||
|
||||
### Lookalike domains
|
||||
|
||||
DMARC protects against domain spoofing, not lookalike domains. for open source
|
||||
DMARC protects against domain spoofing, not lookalike domains. For open source
|
||||
lookalike domain monitoring, check out [DomainAware](https://github.com/seanthegeek/domainaware).
|
||||
|
||||
## DMARC Alignment Guide
|
||||
|
||||
DMARC ensures that SPF and DKM authentication mechanisms actually authenticate
|
||||
DMARC ensures that SPF and DKIM authentication mechanisms actually authenticate
|
||||
against the same domain that the end user sees.
|
||||
|
||||
A message passes a DMARC check by passing DKIM or SPF, **as long as the related
|
||||
|
||||
@@ -55,13 +55,17 @@ sudo systemctl start elasticsearch.service
|
||||
sudo systemctl start kibana.service
|
||||
```
|
||||
|
||||
As of Elasticsearch 8.7, activate secure mode (xpack.security.*.ssl)
|
||||
Since Elasticsearch 8.0, security is enabled and auto-configured on
|
||||
first startup: TLS certificates are generated, the `xpack.security.*`
|
||||
settings below are written to `elasticsearch.yml`, and a password is
|
||||
generated for the `elastic` user. Verify the settings are present —
|
||||
and add them only if your install skipped auto-configuration:
|
||||
|
||||
```bash
|
||||
sudo vim /etc/elasticsearch/elasticsearch.yml
|
||||
```
|
||||
|
||||
Add the following configuration
|
||||
The security configuration looks like this:
|
||||
|
||||
```text
|
||||
# Enable security features
|
||||
@@ -92,11 +96,12 @@ openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout kibana.key -out kiba
|
||||
Or, to create a Certificate Signing Request (CSR) for a CA, run:
|
||||
|
||||
```bash
|
||||
openssl req -newkey rsa:4096-nodes -keyout kibana.key -out kibana.csr
|
||||
openssl req -newkey rsa:4096 -nodes -keyout kibana.key -out kibana.csr
|
||||
```
|
||||
|
||||
Fill in the prompts. Watch out for Common Name (e.g. server FQDN or YOUR
|
||||
domain name), which is the IP address or domain name that you will use to access Kibana. it is the most important field.
|
||||
domain name), which is the IP address or domain name that you will use to
|
||||
access Kibana. It is the most important field.
|
||||
|
||||
If you generated a CSR, remove the CSR after you have your certs
|
||||
|
||||
@@ -129,7 +134,7 @@ server.ssl.key: /etc/kibana/kibana.key
|
||||
|
||||
:::{note}
|
||||
For more security, you can configure Kibana to use a local network connection
|
||||
to elasticsearch :
|
||||
to Elasticsearch:
|
||||
```text
|
||||
elasticsearch.hosts: ['https://SERVER_IP:9200']
|
||||
```
|
||||
@@ -149,14 +154,14 @@ Enroll Kibana in Elasticsearch
|
||||
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
|
||||
```
|
||||
|
||||
Then access to your web server at `https://SERVER_IP:5601`, accept the self-signed
|
||||
certificate and paste the token in the "Enrollment token" field.
|
||||
Then access your web server at `https://SERVER_IP:5601`, accept the self-signed
|
||||
certificate, and paste the token in the "Enrollment token" field.
|
||||
|
||||
```bash
|
||||
sudo /usr/share/kibana/bin/kibana-verification-code
|
||||
```
|
||||
|
||||
Then put the verification code to your web browser.
|
||||
Then enter the verification code in your web browser.
|
||||
|
||||
End Kibana configuration
|
||||
|
||||
@@ -182,12 +187,12 @@ sudo systemctl restart elasticsearch
|
||||
Now that Elasticsearch is up and running, use `parsedmarc` to send data to
|
||||
it.
|
||||
|
||||
Download (right-click the link and click save as) [export.ndjson].
|
||||
Download (right-click the link and click save as) [opensearch_dashboards.ndjson].
|
||||
|
||||
Connect to kibana using the "elastic" user and the password you previously provide
|
||||
on the console ("End Kibana configuration" part).
|
||||
Connect to Kibana using the "elastic" user and the password you previously
|
||||
provided on the console ("End Kibana configuration" part).
|
||||
|
||||
Import `export.ndjson` the Saved Objects tab of the Stack management
|
||||
Import `opensearch_dashboards.ndjson` in the Saved Objects tab of the Stack Management
|
||||
page of Kibana. (Hamburger menu -> "Management" -> "Stack Management" ->
|
||||
"Kibana" -> "Saved Objects")
|
||||
|
||||
@@ -198,20 +203,20 @@ the commercial [X-Pack].
|
||||
|
||||
```{image} _static/screenshots/saved-objects.png
|
||||
:align: center
|
||||
:alt: A screenshot of setting the Saved Objects Stack management UI in Kibana
|
||||
:alt: A screenshot of the Saved Objects Stack Management UI in Kibana
|
||||
:target: _static/screenshots/saved-objects.png
|
||||
```
|
||||
|
||||
```{image} _static/screenshots/confirm-overwrite.png
|
||||
:align: center
|
||||
:alt: A screenshot of the overwrite conformation prompt
|
||||
:alt: A screenshot of the overwrite confirmation prompt
|
||||
:target: _static/screenshots/confirm-overwrite.png
|
||||
```
|
||||
|
||||
## Upgrading Kibana index patterns
|
||||
|
||||
`parsedmarc` 5.0.0 makes some changes to the way data is indexed in
|
||||
Elasticsearch. if you are upgrading from a previous release of
|
||||
Elasticsearch. If you are upgrading from a previous release of
|
||||
`parsedmarc`, you need to complete the following steps to replace the
|
||||
Kibana index patterns with versions that match the upgraded indexes:
|
||||
|
||||
@@ -220,10 +225,10 @@ Kibana index patterns with versions that match the upgraded indexes:
|
||||
3. Check the checkboxes for the `dmarc_aggregate` and `dmarc_failure`
|
||||
index patterns
|
||||
4. Click Delete
|
||||
5. Click Delete on the conformation message
|
||||
5. Click Delete on the confirmation message
|
||||
6. Download (right-click the link and click save as)
|
||||
the latest version of [export.ndjson]
|
||||
7. Import `export.ndjson` by clicking Import from the Kibana
|
||||
the latest version of [opensearch_dashboards.ndjson]
|
||||
7. Import `opensearch_dashboards.ndjson` by clicking Import from the Kibana
|
||||
Saved Objects page
|
||||
|
||||
## Backfilling the combined DKIM/SPF result fields
|
||||
@@ -408,6 +413,6 @@ retention regulations such as GDPR. For more information,
|
||||
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
|
||||
|
||||
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
|
||||
[export.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
|
||||
[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
|
||||
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
|
||||
[x-pack]: https://www.elastic.co/products/x-pack
|
||||
|
||||
@@ -61,7 +61,7 @@ On Debian or Ubuntu systems, run:
|
||||
sudo apt-get install -y python3-pip python3-venv python3-dev libxml2-dev libxslt-dev
|
||||
```
|
||||
|
||||
On CentOS, RHEL, oR Rocky Linux systems, run:
|
||||
On CentOS, RHEL, or Rocky Linux systems, run:
|
||||
|
||||
```bash
|
||||
sudo dnf install -y python3 python3-pip python3-devel libxml2-devel libxslt-devel
|
||||
@@ -89,13 +89,64 @@ any files created later are also owned by `parsedmarc`
|
||||
```bash
|
||||
sudo -u parsedmarc python3 -m venv /opt/parsedmarc/venv
|
||||
sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install --upgrade pip
|
||||
sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install --upgrade parsedmarc
|
||||
sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install --upgrade "parsedmarc[all]"
|
||||
```
|
||||
|
||||
To upgrade `parsedmarc` later, re-run the last command above and then
|
||||
restart the service.
|
||||
|
||||
## Optional dependencies
|
||||
### Choosing what to install
|
||||
|
||||
Starting with the next major release, the integrations that talk to
|
||||
external systems live in optional extras, so an install only carries the
|
||||
dependencies it actually uses.
|
||||
|
||||
`pip install parsedmarc` — the base install — provides:
|
||||
|
||||
- the parsing library (aggregate, failure, and SMTP TLS reports),
|
||||
- the CLI, reading reports from files, an IMAP mailbox, a Maildir, or an
|
||||
mbox file,
|
||||
- and CSV/JSON, Splunk HEC, webhook, and syslog output.
|
||||
|
||||
Everything else needs an extra:
|
||||
|
||||
| Extra | Enables |
|
||||
|---|---|
|
||||
| `elastic` | The `[elasticsearch]` output |
|
||||
| `opensearch` | The `[opensearch]` output |
|
||||
| `kafka` | The `[kafka]` output |
|
||||
| `s3` | The `[s3]` output |
|
||||
| `gelf` | The `[gelf]` output |
|
||||
| `loganalytics` | The `[log_analytics]` (Azure Monitor) output |
|
||||
| `msgraph` | The `[msgraph]` mailbox input (Microsoft 365) |
|
||||
| `gmail` | The `[gmail_api]` mailbox input |
|
||||
| `postgresql` | The `[postgresql]` output |
|
||||
|
||||
Extras can be combined: `pip install "parsedmarc[elastic,msgraph]"`.
|
||||
|
||||
`pip install "parsedmarc[all]"` installs every extra in the table
|
||||
**except** `postgresql`, which stays separate because `psycopg`'s
|
||||
prebuilt binary wheels are not available for every platform — folding it
|
||||
into `all` would make `parsedmarc[all]` fail to install there. Add it
|
||||
explicitly when you need it: `pip install "parsedmarc[all,postgresql]"`.
|
||||
|
||||
If a configuration file names a section whose extra is not installed,
|
||||
`parsedmarc` exits at startup with an error naming the exact
|
||||
`pip install` command to run.
|
||||
|
||||
:::{note}
|
||||
**Upgrading from 10.x:** every 10.x install carried the Elasticsearch,
|
||||
OpenSearch, Kafka, AWS, Azure, Gmail, and Microsoft Graph packages,
|
||||
whether or not it used them. They are no longer installed by
|
||||
`pip install parsedmarc`, so switch your install command to
|
||||
`pip install --upgrade "parsedmarc[all]"` (adding `postgresql` if you
|
||||
use it) to keep every integration available. An in-place upgrade does
|
||||
not uninstall packages you already have, but a rebuilt virtualenv — or
|
||||
any fresh install — gets only what the extras name. Users of the
|
||||
prebuilt Docker image are unaffected: it bundles `[all,postgresql]`.
|
||||
:::
|
||||
|
||||
## Optional system dependencies
|
||||
|
||||
If you would like to be able to parse emails saved from Microsoft
|
||||
Outlook (i.e. OLE .msg files), install `msgconvert`:
|
||||
|
||||
@@ -28,7 +28,7 @@ will filter for that value.
|
||||
|
||||
:::{note}
|
||||
Messages should not be considered malicious just because they failed to pass
|
||||
DMARC; especially if you have just started collecting data. It may be a
|
||||
DMARC, especially if you have just started collecting data. It may be a
|
||||
legitimate service that needs SPF and DKIM configured correctly.
|
||||
:::
|
||||
|
||||
@@ -36,7 +36,7 @@ Start by filtering the results to only show failed DKIM alignment. While DMARC
|
||||
passes if a message passes SPF or DKIM alignment, only DKIM alignment remains
|
||||
valid when a message is forwarded without changing the from address, which is
|
||||
often caused by a mailbox forwarding rule. This is because DKIM signatures are
|
||||
part of the message headers, whereas SPF relies on SMTP session headers.
|
||||
part of the message headers, whereas SPF relies on the SMTP envelope.
|
||||
|
||||
Underneath the pie charts, you can see graphs of DMARC compliance and message
|
||||
disposition over time.
|
||||
@@ -74,7 +74,7 @@ your domains coming from consumer email services, such as Google/Gmail and
|
||||
Yahoo! This occurs when customers have mailbox rules in place that forward
|
||||
emails from an old account to a new account, which is why DKIM
|
||||
authentication is so important, as mentioned earlier. Similar patterns may
|
||||
be observed with businesses who send from reverse DNS addressees of
|
||||
be observed with businesses who send from reverse DNS addresses of
|
||||
parent, subsidiary, and outdated brands.
|
||||
:::
|
||||
|
||||
|
||||
@@ -130,7 +130,7 @@ command line instead, for example:
|
||||
touch var/templates/lists/list.example.com/en/list:member:regular:footer
|
||||
```
|
||||
|
||||
Where `list.example.com` the list ID, and `en` is the language.
|
||||
Where `list.example.com` is the list ID, and `en` is the language.
|
||||
|
||||
Then restart mailman core.
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
Here are the results from parsing the [example](https://dmarc.org/wiki/FAQ#I_need_to_implement_aggregate_reports.2C_what_do_they_look_like.3F)
|
||||
report from the dmarc.org wiki. It's actually an older draft of
|
||||
the 1.0 report schema standardized in
|
||||
[RFC 7480 Appendix C](https://tools.ietf.org/html/rfc7489#appendix-C).
|
||||
[RFC 7489 Appendix C](https://tools.ietf.org/html/rfc7489#appendix-C).
|
||||
This draft schema is still in wide use.
|
||||
|
||||
`parsedmarc` produces consistent, normalized output, regardless
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Splunk
|
||||
|
||||
Starting in version 4.3.0 `parsedmarc` supports sending aggregate and/or
|
||||
failure DMARC data to a Splunk [HTTP Event collector (HEC)].
|
||||
failure DMARC data to a Splunk [HTTP Event Collector (HEC)].
|
||||
|
||||
The project repository contains [XML files] for premade Splunk
|
||||
dashboards for aggregate and failure DMARC reports.
|
||||
@@ -19,4 +19,4 @@ Kibana dashboards, although the Kibana dashboards have slightly
|
||||
easier and more flexible filtering options.
|
||||
|
||||
[xml files]: https://github.com/domainaware/parsedmarc/tree/master/dashboards/splunk
|
||||
[http event collector (hec)]: http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC
|
||||
[http event collector (hec)]: https://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC
|
||||
|
||||
+106
-51
@@ -14,8 +14,8 @@ usage: parsedmarc [-h] [-c CONFIG_FILE] [-r] [--strip-attachment-payloads] [-o O
|
||||
Parses DMARC reports
|
||||
|
||||
positional arguments:
|
||||
file_path one or more paths to aggregate or failure report files, emails, mbox files, or directories
|
||||
containing them
|
||||
file_path one or more paths to aggregate, failure, or SMTP TLS report files, emails, mbox files, or
|
||||
directories containing them
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
@@ -119,21 +119,23 @@ smtp_tls_url = https://smtp_tls_url.example.com
|
||||
timeout = 60
|
||||
```
|
||||
|
||||
The full set of configuration options are:
|
||||
The full set of configuration options is:
|
||||
|
||||
- `general`
|
||||
- `save_aggregate` - bool: Save aggregate report data to
|
||||
Elasticsearch, Splunk and/or S3
|
||||
- `save_failure` - bool: Save failure report data to
|
||||
Elasticsearch, Splunk and/or S3
|
||||
- `save_smtp_tls` - bool: Save SMTP-STS report data to
|
||||
- `save_smtp_tls` - bool: Save SMTP TLS report data to
|
||||
Elasticsearch, Splunk and/or S3
|
||||
- `index_prefix_domain_map` - str: Path to a YAML file mapping
|
||||
OpenSearch/Elasticsearch index prefixes to domain names
|
||||
- `strip_attachment_payloads` - bool: Remove attachment
|
||||
payloads from results
|
||||
- `silent` - bool: Set this to `False` to output results to STDOUT
|
||||
- `output` - str: Directory to place JSON and CSV files in. This is required if you set either of the JSON output file options.
|
||||
- `silent` - bool: Only print errors; set this to `False` to output
|
||||
results to STDOUT (Default: `True`)
|
||||
- `output` - str: Directory to place JSON and CSV files in. The JSON
|
||||
and CSV filename options below only take effect when this is set.
|
||||
- `archive_directory` - str: Optional. When set, successfully
|
||||
processed report files given as local file/directory path
|
||||
arguments are moved into
|
||||
@@ -156,6 +158,19 @@ The full set of configuration options are:
|
||||
JSON output file
|
||||
- `failure_json_filename` - str: filename for the failure
|
||||
JSON output file
|
||||
- `smtp_tls_json_filename` - str: filename for the SMTP TLS
|
||||
JSON output file
|
||||
- `aggregate_csv_filename` - str: filename for the aggregate
|
||||
CSV output file
|
||||
- `failure_csv_filename` - str: filename for the failure
|
||||
CSV output file
|
||||
- `smtp_tls_csv_filename` - str: filename for the SMTP TLS
|
||||
CSV output file
|
||||
- `prettify_json` - bool: Set this to `False` to output JSON in a
|
||||
single line without indentation (Default: `True`)
|
||||
- `normalize_timespan_threshold_hours` - float: Aggregate reports
|
||||
covering a longer time span than this many hours have their
|
||||
records normalized into per-day records (Default: `24`)
|
||||
- `ip_db_path` - str: An optional custom path to a MMDB file
|
||||
from IPinfo, MaxMind, or DBIP
|
||||
- `ipinfo_url` - str: Overrides the default download URL for the
|
||||
@@ -172,24 +187,25 @@ The full set of configuration options are:
|
||||
- `offline` - bool: Do not use online queries for geolocation
|
||||
or DNS. Also disables automatic downloading of the IP-to-country
|
||||
database and reverse DNS map.
|
||||
- `always_use_local_files` - Disables the download of the
|
||||
- `always_use_local_files` - bool: Disables the download of the
|
||||
IP-to-country database and reverse DNS map
|
||||
- `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
|
||||
- `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map
|
||||
- `local_psl_overrides_path` - Overrides the default local file path to use for the PSL overrides list
|
||||
- `psl_overrides_url` - Overrides the default download URL for the PSL overrides list
|
||||
- `local_reverse_dns_map_path` - str: Overrides the default local file path to use for the reverse DNS map
|
||||
- `reverse_dns_map_url` - str: Overrides the default download URL for the reverse DNS map
|
||||
- `local_psl_overrides_path` - str: Overrides the default local file path to use for the PSL overrides list
|
||||
- `psl_overrides_url` - str: Overrides the default download URL for the PSL overrides list
|
||||
- `nameservers` - str: A comma separated list of
|
||||
DNS resolvers (Default: `[Cloudflare's public resolvers]`). Each entry
|
||||
is an IP address (DNS over UDP/TCP port 53), an `https://` URL
|
||||
(DNS over HTTPS), or `tls://ip[:port][#hostname]` (DNS over TLS) —
|
||||
see [Encrypted DNS](#encrypted-dns)
|
||||
- `dns_test_address` - str: a dummy address used for DNS pre-flight checks
|
||||
(Default: 1.1.1.1)
|
||||
- `dns_timeout` - float: DNS timeout period
|
||||
- `dns_test_address` - str: A dummy address used for the DNS pre-flight
|
||||
check that runs when `nameservers` is set (Default: `1.1.1.1`)
|
||||
- `dns_timeout` - float: DNS timeout period in seconds (Default: `2.0`)
|
||||
- `dns_retries` - int: Number of times to retry a DNS query after a
|
||||
timeout or other transient error (Default: 0)
|
||||
timeout or other transient error (Default: `0`)
|
||||
- `debug` - bool: Print debugging messages
|
||||
- `silent` - bool: Only print errors (Default: `True`)
|
||||
- `verbose` - bool: More verbose output
|
||||
- `warnings` - bool: Print warnings in addition to errors
|
||||
- `fail_on_output_error` - bool: Exit with a non-zero status code if
|
||||
any configured output destination fails while saving/publishing
|
||||
reports (Default: `False`)
|
||||
@@ -227,8 +243,9 @@ The full set of configuration options are:
|
||||
(Default: `INBOX`)
|
||||
- `archive_folder` - str: The mailbox folder (or label for
|
||||
Gmail) to sort processed emails into (Default: `Archive`)
|
||||
- `watch` - bool: Use the IMAP `IDLE` command to process
|
||||
messages as they arrive or poll MS Graph for new messages
|
||||
- `watch` - bool: Process new messages as they arrive, via the
|
||||
IMAP `IDLE` command or by polling the other mailbox types
|
||||
(Microsoft Graph, Gmail API, Maildir)
|
||||
- `delete` - bool: Delete messages after processing them,
|
||||
instead of archiving them
|
||||
- `delete_aggregate` - bool: Delete aggregate report messages
|
||||
@@ -271,9 +288,9 @@ The full set of configuration options are:
|
||||
runs. See
|
||||
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
|
||||
below.
|
||||
- `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
|
||||
- `since` - str: Search for messages since a certain time. (Examples: `5m|3h|2d|1w`)
|
||||
Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
|
||||
Defaults to `1d` if incorrect value is provided.
|
||||
Defaults to `1d` if an incorrect value is provided.
|
||||
- `imap`
|
||||
- `host` - str: The IMAP server hostname or IP address
|
||||
- `port` - int: The IMAP server port (Default: `993`)
|
||||
@@ -296,9 +313,16 @@ The full set of configuration options are:
|
||||
(Default: `True`)
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
verification (not recommended)
|
||||
- `timeout` - int: Number of seconds to wait for an IMAP operation
|
||||
(Default: `30`)
|
||||
- `max_retries` - int: Maximum number of retries after an IMAP
|
||||
timeout (Default: `4`)
|
||||
- `user` - str: The IMAP user
|
||||
- `password` - str: The IMAP password
|
||||
- `msgraph`
|
||||
|
||||
Requires the `msgraph` extra: `pip install "parsedmarc[msgraph]"`
|
||||
|
||||
- `auth_method` - str: Authentication method, valid types are
|
||||
`UsernamePassword`, `DeviceCode`, `ClientSecret`, `Certificate`, or
|
||||
`ClientAssertion` (Default: `UsernamePassword`).
|
||||
@@ -307,7 +331,8 @@ The full set of configuration options are:
|
||||
- `password` - str: The user password, required when the auth
|
||||
method is UsernamePassword
|
||||
- `client_id` - str: The app registration's client ID
|
||||
- `client_secret` - str: The app registration's secret
|
||||
- `client_secret` - str: The app registration's secret. Required when
|
||||
the auth method is `UsernamePassword` or `ClientSecret`
|
||||
- `certificate_path` - str: Path to a PEM or PKCS12 certificate
|
||||
including the private key. Required when the auth method is
|
||||
`Certificate`
|
||||
@@ -329,8 +354,9 @@ The full set of configuration options are:
|
||||
- `mailbox` - str: The mailbox name. This defaults to the
|
||||
current user if using the UsernamePassword auth method, but
|
||||
could be a shared mailbox if the user has access to the mailbox
|
||||
- `graph_url` - str: Microsoft Graph URL. Allows for use of National Clouds (ex Azure Gov)
|
||||
(Default: https://graph.microsoft.com)
|
||||
- `graph_url` - str: Microsoft Graph URL. Allows for use of national
|
||||
clouds (e.g. Azure Gov)
|
||||
(Default: `https://graph.microsoft.com`)
|
||||
|
||||
:::{warning}
|
||||
Setting `graph_url` alone is **not** sufficient for a national/sovereign
|
||||
@@ -534,6 +560,9 @@ The full set of configuration options are:
|
||||
| Invalid/rejected timestamp in the `since`/`receivedDateTime` filter | Historical bug (parsedmarc [#706](https://github.com/domainaware/parsedmarc/pull/706)/[#708](https://github.com/domainaware/parsedmarc/pull/708)): older versions appended a spurious `Z` to an already-UTC-offset ISO timestamp. Fixed since parsedmarc 9.5.1/9.5.5. | Upgrade parsedmarc if you're on a version older than 9.5.5. |
|
||||
:::
|
||||
- `elasticsearch`
|
||||
|
||||
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
@@ -548,7 +577,7 @@ The full set of configuration options are:
|
||||
- `ssl` - bool: Use an encrypted SSL/TLS connection
|
||||
(Default: `True`)
|
||||
- `timeout` - float: Timeout in seconds (Default: 60)
|
||||
- `cert_path` - str: Path to a trusted certificates
|
||||
- `cert_path` - str: Path to a trusted CA certificates file
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
verification (not recommended)
|
||||
- `index_suffix` - str: A suffix to apply to the index names
|
||||
@@ -565,6 +594,9 @@ The full set of configuration options are:
|
||||
settings sent at index creation; any other settings (e.g.
|
||||
`refresh_interval`) are passed through unchanged (Default: `False`)
|
||||
- `opensearch`
|
||||
|
||||
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of hostnames and ports
|
||||
or URLs (e.g. `127.0.0.1:9200` or
|
||||
`https://user:secret@localhost`)
|
||||
@@ -583,7 +615,7 @@ The full set of configuration options are:
|
||||
- `ssl` - bool: Use an encrypted SSL/TLS connection
|
||||
(Default: `True`)
|
||||
- `timeout` - float: Timeout in seconds (Default: 60)
|
||||
- `cert_path` - str: Path to a trusted certificates
|
||||
- `cert_path` - str: Path to a trusted CA certificates file
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
verification (not recommended)
|
||||
- `index_suffix` - str: A suffix to apply to the index names
|
||||
@@ -600,14 +632,18 @@ The full set of configuration options are:
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
verification (not recommended)
|
||||
- `kafka`
|
||||
|
||||
Requires the `kafka` extra: `pip install "parsedmarc[kafka]"`
|
||||
|
||||
- `hosts` - str: A comma separated list of Kafka hosts
|
||||
- `user` - str: The Kafka user
|
||||
- `passsword` - str: The Kafka password
|
||||
- `password` - str: The Kafka password
|
||||
- `ssl` - bool: Use an encrypted SSL/TLS connection (Default: `True`)
|
||||
- `skip_certificate_verification` - bool: Skip certificate
|
||||
verification (not recommended)
|
||||
- `aggregate_topic` - str: The Kafka topic for aggregate reports
|
||||
- `failure_topic` - str: The Kafka topic for failure reports
|
||||
- `smtp_tls_topic` - str: The Kafka topic for SMTP TLS reports
|
||||
- `smtp`
|
||||
|
||||
The results email is only sent when at least one aggregate, failure,
|
||||
@@ -632,7 +668,7 @@ The full set of configuration options are:
|
||||
- `to` - list: A list of email addresses to send to
|
||||
- `subject` - str: The Subject header to use in the email
|
||||
(Default: `parsedmarc report`)
|
||||
- `attachment` - str: The ZIP attachment filenames
|
||||
- `attachment` - str: The ZIP attachment filename
|
||||
(Default: `DMARC-<YYYY-MM-DD>.zip`)
|
||||
- `message` - str: The email message
|
||||
(Default: `Please see the attached DMARC results.`)
|
||||
@@ -653,13 +689,14 @@ The full set of configuration options are:
|
||||
all individual parameters above are ignored.
|
||||
|
||||
The PostgreSQL backend is an optional extra. Install it with
|
||||
`pip install parsedmarc[postgresql]` (it pulls in `psycopg`); the
|
||||
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
|
||||
prebuilt binary wheels are not available for every platform, which is
|
||||
why it is not a mandatory dependency. The prebuilt Docker image
|
||||
(`ghcr.io/domainaware/parsedmarc`) already bundles this extra, so the
|
||||
PostgreSQL backend works out of the box in the container — `psycopg`
|
||||
ships `amd64` and `arm64` binary wheels, both of which the image
|
||||
supports.
|
||||
why it is the one extra that `parsedmarc[all]` deliberately leaves out
|
||||
— combine the two with `pip install "parsedmarc[all,postgresql]"`. The
|
||||
prebuilt Docker image (`ghcr.io/domainaware/parsedmarc`) already
|
||||
bundles `[all,postgresql]`, so the PostgreSQL backend works out of the
|
||||
box in the container — `psycopg` ships `amd64` and `arm64` binary
|
||||
wheels, both of which the image supports.
|
||||
|
||||
Tables are created automatically on first run using
|
||||
`CREATE TABLE IF NOT EXISTS`, so no manual schema migration is needed
|
||||
@@ -692,6 +729,9 @@ The full set of configuration options are:
|
||||
this section is configured.
|
||||
|
||||
- `s3`
|
||||
|
||||
Requires the `s3` extra: `pip install "parsedmarc[s3]"`
|
||||
|
||||
- `bucket` - str: The S3 bucket name
|
||||
- `path` - str: The path to upload reports to (Default: `/`)
|
||||
- `region_name` - str: The region name (Optional)
|
||||
@@ -755,6 +795,9 @@ The full set of configuration options are:
|
||||
```
|
||||
|
||||
- `gmail_api`
|
||||
|
||||
Requires the `gmail` extra: `pip install "parsedmarc[gmail]"`
|
||||
|
||||
- `credentials_file` - str: Path to file containing the
|
||||
credentials, None to disable (Default: `None`)
|
||||
- `token_file` - str: Path to save the token file
|
||||
@@ -766,7 +809,9 @@ The full set of configuration options are:
|
||||
accepted as `delegated_user` for backward compatibility.
|
||||
|
||||
:::{note}
|
||||
credentials_file and token_file can be got with [quickstart](https://developers.google.com/gmail/api/quickstart/python).Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
|
||||
`credentials_file` and `token_file` can be obtained by following the
|
||||
Gmail API [quickstart](https://developers.google.com/gmail/api/quickstart/python).
|
||||
Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
|
||||
:::
|
||||
:::{note}
|
||||
When `auth_mode = service_account`, `credentials_file` must point to a
|
||||
@@ -782,6 +827,9 @@ The full set of configuration options are:
|
||||
- `paginate_messages` - bool: When `True`, fetch all applicable Gmail messages.
|
||||
When `False`, only fetch up to 100 new messages per run (Default: `True`)
|
||||
- `log_analytics`
|
||||
|
||||
Requires the `loganalytics` extra: `pip install "parsedmarc[loganalytics]"`
|
||||
|
||||
- `client_id` - str: The app registration's client ID
|
||||
- `client_secret` - str: The app registration's client secret
|
||||
- `tenant_id` - str: The tenant id where the app registration resides
|
||||
@@ -792,22 +840,27 @@ The full set of configuration options are:
|
||||
- `dcr_smtp_tls_stream` - str: The stream name for the SMTP TLS reports in the DCR
|
||||
|
||||
:::{note}
|
||||
Information regarding the setup of the Data Collection Rule can be found [in the Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal).
|
||||
:::
|
||||
Information regarding the setup of the Data Collection Rule can be found [in the Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal).
|
||||
:::
|
||||
- `gelf`
|
||||
|
||||
Requires the `gelf` extra: `pip install "parsedmarc[gelf]"`
|
||||
|
||||
- `host` - str: The GELF server name or IP address
|
||||
- `port` - int: The port to use
|
||||
- `mode` - str: The GELF transport type to use. Valid modes: `tcp`, `udp`, `tls`
|
||||
|
||||
- `maildir`
|
||||
- `maildir_path` - str: Full path for mailbox maildir location (Default: `INBOX`)
|
||||
- `maildir_create` - bool: Create maildir if not present (Default: False)
|
||||
- `maildir_path` - str: Full path to the maildir location (the key
|
||||
`path` is accepted as an alias). Required to read from a maildir.
|
||||
- `maildir_create` - bool: Create the maildir if not present
|
||||
(Default: `False`; the key `create` is accepted as an alias)
|
||||
|
||||
- `webhook` - Post the individual reports to a webhook url with the report as the JSON body
|
||||
- `webhook` - Post the individual reports to a webhook URL with the report as the JSON body
|
||||
- `aggregate_url` - str: URL of the webhook which should receive the aggregate reports
|
||||
- `failure_url` - str: URL of the webhook which should receive the failure reports
|
||||
- `smtp_tls_url` - str: URL of the webhook which should receive the smtp_tls reports
|
||||
- `timeout` - int: Interval in which the webhook call should timeout
|
||||
- `smtp_tls_url` - str: URL of the webhook which should receive the SMTP TLS reports
|
||||
- `timeout` - int: Timeout in seconds for webhook requests (Default: `60`)
|
||||
|
||||
:::{warning}
|
||||
It is **strongly recommended** to **not** use the `nameservers`
|
||||
@@ -847,8 +900,8 @@ known samples you want to save to that folder
|
||||
:::
|
||||
|
||||
:::{warning}
|
||||
Elasticsearch 8 change limits policy for shards, restricting by
|
||||
default to 1000. parsedmarc use a shard per analyzed day. If you
|
||||
Elasticsearch 8 changed the limits policy for shards, restricting them
|
||||
by default to 1000. parsedmarc uses a shard per analyzed day. If you
|
||||
have more than ~3 years of data, you will need to update this
|
||||
limit.
|
||||
Check current usage (from Management -> Dev Tools -> Console):
|
||||
@@ -863,7 +916,7 @@ GET /_cluster/health?pretty
|
||||
}
|
||||
```
|
||||
|
||||
Update the limit to 2k per example:
|
||||
For example, update the limit to 2000:
|
||||
|
||||
```text
|
||||
PUT _cluster/settings
|
||||
@@ -1123,6 +1176,7 @@ For sections with underscores in the name, the full section name is used:
|
||||
| `kafka` | `PARSEDMARC_KAFKA_` |
|
||||
| `smtp` | `PARSEDMARC_SMTP_` |
|
||||
| `s3` | `PARSEDMARC_S3_` |
|
||||
| `postgresql` | `PARSEDMARC_POSTGRESQL_` |
|
||||
| `syslog` | `PARSEDMARC_SYSLOG_` |
|
||||
| `gmail_api` | `PARSEDMARC_GMAIL_API_` |
|
||||
| `maildir` | `PARSEDMARC_MAILDIR_` |
|
||||
@@ -1207,7 +1261,7 @@ high-volume mailbox processing.
|
||||
|
||||
## Multi-tenant support
|
||||
|
||||
Starting in `8.19.0`, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file that is formatted where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:
|
||||
Starting in `8.19.0`, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:
|
||||
|
||||
```yaml
|
||||
example:
|
||||
@@ -1223,8 +1277,8 @@ Save it to disk where the user running ParseDMARC can read it, then set `index_p
|
||||
|
||||
When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
|
||||
|
||||
:::{note}
|
||||
A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
|
||||
:::{note}
|
||||
A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
|
||||
:::
|
||||
|
||||
Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
|
||||
@@ -1301,9 +1355,9 @@ sudo service parsedmarc restart
|
||||
On `systemctl stop`/`restart` (or Ctrl-C) `parsedmarc` finishes the
|
||||
current batch, flushes its outputs, and exits cleanly. Shutdown is
|
||||
observed at batch boundaries, so the worst-case delay is roughly
|
||||
`mailbox_check_timeout` (default 30s) plus the batch's processing and
|
||||
`[mailbox] check_timeout` (default 30s) plus the batch's processing and
|
||||
flush time. Keep `TimeoutStopSec` comfortably above
|
||||
`mailbox_check_timeout` (≈2×, and raise both together) or systemd will
|
||||
`check_timeout` (≈2×, and raise both together) or systemd will
|
||||
`SIGKILL` mid-batch. In the foreground, a second Ctrl-C force-quits
|
||||
immediately, skipping the output flush.
|
||||
:::
|
||||
@@ -1360,8 +1414,9 @@ service parsedmarc status
|
||||
```
|
||||
|
||||
:::{note}
|
||||
In the event of a crash, systemd will restart the service after 10
|
||||
minutes, but the `service parsedmarc status` command will only show
|
||||
In the event of a crash, systemd will restart the service after 5
|
||||
minutes (per the `RestartSec` setting above), but the
|
||||
`service parsedmarc status` command will only show
|
||||
the logs for the current process. To view the logs for previous runs
|
||||
as well as the current process (newest to oldest), run:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user