mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-06 16:59:23 +00:00
768bf75480
* feat: add grafana loki output Push aggregate, failure and SMTP TLS reports as JSON log lines to the Loki push API. Static labels and labels taken from report fields are configurable, host defaults to the FQDN. The timestamp is either the ingest time or the report begin time. Supports basic auth, bearer tokens, X-Scope-OrgID and batching. Push errors are reported as output errors so messages are not archived. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * feat(dashboards): add grafana dashboard for the loki output Signed-off-by: Sebastian Schubert <basti@schubert.digital> * Address Copilot review on the Loki output Responds to Copilot's review of #926: - Treat any push status other than 200/204 as a failure. Loki returns block_ingestion_status_code (260 by default) when block_ingestion_until blocks ingestion, and the logs are discarded (Loki configuration reference); the old >= 300 check counted that as a successful save, so the CLI could archive mailbox messages whose reports were never stored. - Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate CSV rows carry the record's interval_begin, which the parser builds as naive UTC wall-clock time (to_utc=True); reading it as local time shifted entries by the host's UTC offset (+4h on America/New_York). - Group the Published Policies dashboard dedup by org_name as well as report_id: report IDs are only unique per reporter (the PostgreSQL key is UNIQUE (org_name, report_id, ...)). - Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback with the automatic step yields overlapping rolling totals, so the sum counted the same sessions repeatedly. Adds regression tests for each, including a non-UTC timezone test against a real sample report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address second Copilot review on the Loki output Responds to Copilot's second review of #926 (previously-missed findings): - Strip each [loki] label_fields entry. _str_to_list() only lstrips, so "report_type , domain" produced "report_type " and LokiClient refused to start with "Invalid Loki label name 'report_type '". - Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat inside the ES-has-docs guard, so after the ES container was recreated a reseed appended a second copy of the samples to Loki, which does not deduplicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address third Copilot review on the Loki output Responds to Copilot's third review of #926 (previously-missed findings): - Keep a base URL's query string when appending the push path. The path was string-concatenated, so https://gw.example.org?route=logs became ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted query. The path is now set on the parsed httpx.URL. - Push each report's entries in timestamp order. With timestamp = report_begin, a normalized multi-day aggregate report yields rows source-major (every day of one source, then day 1 of the next), all in one stream. Loki rejects entries older than the newest line minus half of max_chunk_age (1h by default) with too_far_behind (Loki configuration docs, "Accept out-of-order writes"), so later sources were partly rejected. The whole entry list is now sorted before batching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(loki): do not follow redirects and drop forensic alias httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError. Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * fix(dashboards): use the existing Loki datasource uid in the dev bootstrap When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead. Signed-off-by: Sebastian Schubert <basti@schubert.digital> --------- Signed-off-by: Sebastian Schubert <basti@schubert.digital> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
83 lines
3.1 KiB
Markdown
83 lines
3.1 KiB
Markdown
# parsedmarc documentation - Open source DMARC report analyzer and visualizer
|
|
|
|
[](https://github.com/domainaware/parsedmarc/actions/workflows/python-tests.yml)
|
|
[](https://codecov.io/gh/domainaware/parsedmarc)
|
|
[](https://pypi.org/project/parsedmarc/)
|
|
[](https://pypistats.org/packages/parsedmarc)
|
|
|
|
:::{note}
|
|
|
|
This project is maintained by one developer.
|
|
Please consider [sponsoring my work](https://github.com/sponsors/seanthegeek) if you or your organization benefit from it.
|
|
:::
|
|
|
|
```{image} _static/screenshots/dmarc-summary-charts.png
|
|
:align: center
|
|
:alt: A screenshot of DMARC summary charts in Kibana
|
|
:scale: 50 %
|
|
:target: _static/screenshots/dmarc-summary-charts.png
|
|
```
|
|
|
|
`parsedmarc` is a Python module and CLI utility for parsing DMARC reports.
|
|
When used with Elasticsearch and Kibana (or Splunk), or with OpenSearch and Grafana, it works as a self-hosted
|
|
open source alternative to commercial DMARC report processing services such
|
|
as Agari Brand Protection, Dmarcian, OnDMARC, ProofPoint Email Fraud Defense,
|
|
and Valimail.
|
|
|
|
## Features
|
|
|
|
- Parses aggregate/rua DMARC reports: the legacy draft and 1.0 schemas
|
|
(RFC 7489) and the new RFC 9990 schema for the final DMARC standard
|
|
(RFC 9989)
|
|
- Parses failure/ruf DMARC reports (RFC 6591 and RFC 9991; formerly called
|
|
forensic reports)
|
|
- Parses reports from SMTP TLS Reporting (TLS-RPT, RFC 8460)
|
|
- Can parse reports from an inbox over IMAP, Microsoft Graph, or Gmail API
|
|
- Transparently handles gzip or zip compressed reports
|
|
- Consistent data structures
|
|
- Simple JSON and/or CSV output
|
|
- Optionally email the results
|
|
- Optionally send the results to Elasticsearch, OpenSearch, Splunk, or
|
|
PostgreSQL, for use with premade dashboards
|
|
- Optionally send the results to Apache Kafka, Amazon S3, Azure Log
|
|
Analytics (Microsoft Sentinel), a Graylog (GELF) endpoint, a syslog server,
|
|
Grafana Loki, or an HTTP webhook
|
|
|
|
## Python Compatibility
|
|
|
|
This project supports the following Python versions, which are either actively maintained or are the default versions
|
|
for RHEL or Debian.
|
|
|
|
| Version | Supported | Reason |
|
|
| --- | --- | --- |
|
|
| < 3.6 | ❌ | End of Life (EOL) |
|
|
| 3.6 | ❌ | Used in RHEL 8, but not supported by project dependencies |
|
|
| 3.7 | ❌ | End of Life (EOL) |
|
|
| 3.8 | ❌ | End of Life (EOL) |
|
|
| 3.9 | ❌ | Used in Debian 11 and RHEL 9, but not supported by project dependencies |
|
|
| 3.10 | ✅ | Actively maintained |
|
|
| 3.11 | ✅ | Actively maintained; supported until June 2028 (Debian 12) |
|
|
| 3.12 | ✅ | Actively maintained; supported until May 2035 (RHEL 10) |
|
|
| 3.13 | ✅ | Actively maintained; supported until June 2030 (Debian 13) |
|
|
| 3.14 | ✅ | Supported (requires `imapclient>=3.1.0`) |
|
|
|
|
```{toctree}
|
|
:caption: 'Contents'
|
|
:maxdepth: 2
|
|
|
|
installation
|
|
usage
|
|
output
|
|
elasticsearch
|
|
opensearch
|
|
kibana
|
|
splunk
|
|
davmail
|
|
dmarc
|
|
contributing
|
|
api
|
|
```
|