mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-09-06 14:07:59 +00:00
* Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
250 lines
9.4 KiB
Markdown
250 lines
9.4 KiB
Markdown
# Sample outputs
|
|
|
|
## Sample aggregate report output
|
|
|
|
Here are the results from parsing the [example](https://dmarc.org/wiki/FAQ#I_need_to_implement_aggregate_reports.2C_what_do_they_look_like.3F)
|
|
report from the dmarc.org wiki. It's actually an older draft of
|
|
the 1.0 report schema standardized in
|
|
[RFC 7489 Appendix C](https://tools.ietf.org/html/rfc7489#appendix-C).
|
|
This draft schema is still in wide use.
|
|
|
|
`parsedmarc` produces consistent, normalized output, regardless
|
|
of the report schema.
|
|
|
|
### JSON aggregate report
|
|
|
|
```json
|
|
{
|
|
"xml_schema": "draft",
|
|
"report_metadata": {
|
|
"org_name": "acme.com",
|
|
"org_email": "noreply-dmarc-support@acme.com",
|
|
"org_extra_contact_info": "http://acme.com/dmarc/support",
|
|
"report_id": "9391651994964116463",
|
|
"begin_date": "2012-04-27 20:00:00",
|
|
"end_date": "2012-04-28 19:59:59",
|
|
"timespan_requires_normalization": false,
|
|
"original_timespan_seconds": 86399,
|
|
"errors": []
|
|
},
|
|
"policy_published": {
|
|
"domain": "example.com",
|
|
"adkim": "r",
|
|
"aspf": "r",
|
|
"p": "none",
|
|
"sp": "none",
|
|
"pct": "100",
|
|
"fo": "0"
|
|
},
|
|
"records": [
|
|
{
|
|
"source": {
|
|
"ip_address": "72.150.241.94",
|
|
"country": "US",
|
|
"reverse_dns": null,
|
|
"base_domain": null,
|
|
"name": null,
|
|
"type": null,
|
|
"asn": 7018,
|
|
"as_name": "AT&T Services, Inc.",
|
|
"as_domain": "att.com"
|
|
},
|
|
"count": 2,
|
|
"alignment": {
|
|
"spf": true,
|
|
"dkim": false,
|
|
"dmarc": true
|
|
},
|
|
"policy_evaluated": {
|
|
"disposition": "none",
|
|
"dkim": "fail",
|
|
"spf": "pass",
|
|
"policy_override_reasons": []
|
|
},
|
|
"identifiers": {
|
|
"header_from": "example.com",
|
|
"envelope_from": "example.com",
|
|
"envelope_to": null
|
|
},
|
|
"auth_results": {
|
|
"dkim": [
|
|
{
|
|
"domain": "example.com",
|
|
"selector": "none",
|
|
"result": "fail"
|
|
}
|
|
],
|
|
"spf": [
|
|
{
|
|
"domain": "example.com",
|
|
"scope": "mfrom",
|
|
"result": "pass"
|
|
}
|
|
]
|
|
},
|
|
"normalized_timespan": false,
|
|
"interval_begin": "2012-04-28 00:00:00",
|
|
"interval_end": "2012-04-28 23:59:59"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### CSV aggregate report
|
|
|
|
```text
|
|
xml_schema,org_name,org_email,org_extra_contact_info,report_id,begin_date,end_date,normalized_timespan,errors,domain,adkim,aspf,p,sp,pct,fo,source_ip_address,source_country,source_reverse_dns,source_base_domain,source_name,source_type,source_asn,source_as_name,source_as_domain,count,spf_aligned,dkim_aligned,dmarc_aligned,disposition,policy_override_reasons,policy_override_comments,envelope_from,header_from,envelope_to,dkim_domains,dkim_selectors,dkim_results,spf_domains,spf_scopes,spf_results
|
|
draft,acme.com,noreply-dmarc-support@acme.com,http://acme.com/dmarc/support,9391651994964116463,2012-04-28 00:00:00,2012-04-28 23:59:59,False,,example.com,r,r,none,none,100,0,72.150.241.94,US,,,,,2,True,False,True,none,,,example.com,example.com,,example.com,none,fail,example.com,mfrom,pass
|
|
draft,acme.com,noreply-dmarc-support@acme.com,http://acme.com/dmarc/support,9391651994964116463,2012-04-28 00:00:00,2012-04-28 23:59:59,False,,example.com,r,r,none,none,100,0,72.150.241.94,US,,,,,2,True,False,True,none,,,example.com,example.com,,example.com,none,fail,example.com,mfrom,pass
|
|
|
|
```
|
|
|
|
## Sample failure report output
|
|
|
|
Thanks to GitHub user [xennn](https://github.com/xennn) for the anonymized
|
|
[failure report email sample](<https://github.com/domainaware/parsedmarc/raw/master/samples/failure/DMARC%20Failure%20Report%20for%20domain.de%20(mail-from%3Dsharepoint%40domain.de%2C%20ip%3D10.10.10.10).eml>).
|
|
|
|
### JSON failure report
|
|
|
|
```json
|
|
{
|
|
"feedback_type": "auth-failure",
|
|
"user_agent": "Lua/1.0",
|
|
"version": "1.0",
|
|
"original_mail_from": "sharepoint@domain.de",
|
|
"original_rcpt_to": "peter.pan@domain.de",
|
|
"arrival_date": "Mon, 01 Oct 2018 11:20:27 +0200",
|
|
"message_id": "<38.E7.30937.BD6E1BB5@ mailrelay.de>",
|
|
"authentication_results": "dmarc=fail (p=none, dis=none) header.from=domain.de",
|
|
"delivery_result": "policy",
|
|
"auth_failure": [
|
|
"dmarc"
|
|
],
|
|
"reported_domain": "domain.de",
|
|
"arrival_date_utc": "2018-10-01 09:20:27",
|
|
"source": {
|
|
"ip_address": "10.10.10.10",
|
|
"country": null,
|
|
"reverse_dns": null,
|
|
"base_domain": null,
|
|
"name": null,
|
|
"type": null,
|
|
"asn": null,
|
|
"as_name": null,
|
|
"as_domain": null
|
|
},
|
|
"authentication_mechanisms": [],
|
|
"original_envelope_id": null,
|
|
"dkim_domain": null,
|
|
"sample_headers_only": false,
|
|
"sample": "Received: from Servernameone.domain.local (Servernameone.domain.local [10.10.10.10])\n\tby mailrelay.de (mail.DOMAIN.de) with SMTP id 38.E7.30937.BD6E1BB5; Mon, 1 Oct 2018 11:20:27 +0200 (CEST)\nDate: 01 Oct 2018 11:20:27 +0200\nMessage-ID: <38.E7.30937.BD6E1BB5@ mailrelay.de>\nTo: <peter.pan@domain.de>\nfrom: \"=?utf-8?B?SW50ZXJha3RpdmUgV2V0dGJld2VyYmVyLcOcYmVyc2ljaHQ=?=\" <sharepoint@domain.de>\nSubject: Subject\nMIME-Version: 1.0\nX-Mailer: Microsoft SharePoint Foundation 2010\nContent-Type: text/html; charset=utf-8\nContent-Transfer-Encoding: quoted-printable\n\n<html><head><base href=3D'\nwettbewerb' /></head><body><!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\"=\n><HTML><HEAD><META NAME=3D\"Generator\" CONTENT=3D\"MS Exchange Server version=\n 08.01.0240.003\"></html>\n",
|
|
"parsed_sample": {
|
|
"from": {
|
|
"display_name": "Interaktive Wettbewerber-Übersicht",
|
|
"address": "sharepoint@domain.de",
|
|
"local": "sharepoint",
|
|
"domain": "domain.de"
|
|
},
|
|
"to_domains": [
|
|
"domain.de"
|
|
],
|
|
"to": [
|
|
{
|
|
"display_name": null,
|
|
"address": "peter.pan@domain.de",
|
|
"local": "peter.pan",
|
|
"domain": "domain.de"
|
|
}
|
|
],
|
|
"subject": "Subject",
|
|
"timezone": "+2",
|
|
"mime-version": "1.0",
|
|
"date": "2018-10-01 09:20:27",
|
|
"content-type": "text/html; charset=utf-8",
|
|
"x-mailer": "Microsoft SharePoint Foundation 2010",
|
|
"body": "<html><head><base href='\nwettbewerb' /></head><body><!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\"><HTML><HEAD><META NAME=\"Generator\" CONTENT=\"MS Exchange Server version 08.01.0240.003\"></html>",
|
|
"received": [
|
|
{
|
|
"from": "Servernameone.domain.local Servernameone.domain.local 10.10.10.10",
|
|
"by": "mailrelay.de mail.DOMAIN.de",
|
|
"with": "SMTP id 38.E7.30937.BD6E1BB5",
|
|
"date": "Mon, 1 Oct 2018 11:20:27 +0200 CEST",
|
|
"hop": 1,
|
|
"date_utc": "2018-10-01 09:20:27",
|
|
"delay": 0
|
|
}
|
|
],
|
|
"content-transfer-encoding": "quoted-printable",
|
|
"message-id": "<38.E7.30937.BD6E1BB5@ mailrelay.de>",
|
|
"has_defects": false,
|
|
"headers": {
|
|
"Received": "from Servernameone.domain.local (Servernameone.domain.local [10.10.10.10])\n\tby mailrelay.de (mail.DOMAIN.de) with SMTP id 38.E7.30937.BD6E1BB5; Mon, 1 Oct 2018 11:20:27 +0200 (CEST)",
|
|
"Date": "01 Oct 2018 11:20:27 +0200",
|
|
"Message-ID": "<38.E7.30937.BD6E1BB5@ mailrelay.de>",
|
|
"To": "<peter.pan@domain.de>",
|
|
"from": "\"Interaktive Wettbewerber-Übersicht\" <sharepoint@domain.de>",
|
|
"Subject": "Subject",
|
|
"MIME-Version": "1.0",
|
|
"X-Mailer": "Microsoft SharePoint Foundation 2010",
|
|
"Content-Type": "text/html; charset=utf-8",
|
|
"Content-Transfer-Encoding": "quoted-printable"
|
|
},
|
|
"reply_to": [],
|
|
"cc": [],
|
|
"bcc": [],
|
|
"attachments": [],
|
|
"filename_safe_subject": "Subject"
|
|
}
|
|
}
|
|
```
|
|
|
|
### CSV failure report
|
|
|
|
```text
|
|
feedback_type,user_agent,version,original_envelope_id,original_mail_from,original_rcpt_to,arrival_date,arrival_date_utc,subject,message_id,authentication_results,dkim_domain,source_ip_address,source_country,source_reverse_dns,source_base_domain,source_name,source_type,source_asn,source_as_name,source_as_domain,delivery_result,auth_failure,reported_domain,authentication_mechanisms,sample_headers_only
|
|
auth-failure,Lua/1.0,1.0,,sharepoint@domain.de,peter.pan@domain.de,"Mon, 01 Oct 2018 11:20:27 +0200",2018-10-01 09:20:27,Subject,<38.E7.30937.BD6E1BB5@ mailrelay.de>,"dmarc=fail (p=none, dis=none) header.from=domain.de",,10.10.10.10,,,,policy,dmarc,domain.de,,False
|
|
```
|
|
|
|
### JSON SMTP TLS report
|
|
|
|
```json
|
|
[
|
|
{
|
|
"organization_name": "Example Inc.",
|
|
"begin_date": "2024-01-09T00:00:00Z",
|
|
"end_date": "2024-01-09T23:59:59Z",
|
|
"report_id": "2024-01-09T00:00:00Z_example.com",
|
|
"policies": [
|
|
{
|
|
"policy_domain": "example.com",
|
|
"policy_type": "sts",
|
|
"policy_strings": [
|
|
"version: STSv1",
|
|
"mode: testing",
|
|
"mx: example.com",
|
|
"max_age: 86400"
|
|
],
|
|
"successful_session_count": 0,
|
|
"failed_session_count": 3,
|
|
"failure_details": [
|
|
{
|
|
"result_type": "validation-failure",
|
|
"failed_session_count": 2,
|
|
"sending_mta_ip": "209.85.222.201",
|
|
"receiving_ip": "173.212.201.41",
|
|
"receiving_mx_hostname": "example.com"
|
|
},
|
|
{
|
|
"result_type": "validation-failure",
|
|
"failed_session_count": 1,
|
|
"sending_mta_ip": "209.85.208.176",
|
|
"receiving_ip": "173.212.201.41",
|
|
"receiving_mx_hostname": "example.com"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
]
|
|
```
|