mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-04-24 06:19:29 +00:00
2cda5bf59b
* Surface ASN info and fall back to it when a PTR is absent Adds three new fields to every IP source record — ``asn`` (integer, e.g. 15169), ``asn_name`` (``"Google LLC"``), ``asn_domain`` (``"google.com"``) — sourced from the bundled IPinfo Lite MMDB. These flow through to CSV, JSON, Elasticsearch, OpenSearch, and Splunk outputs as ``source_asn``, ``source_asn_name``, ``source_asn_domain``. More importantly: when an IP has no reverse DNS (common for many large senders), source attribution now falls back to the ASN domain as a lookup key into the same ``reverse_dns_map``. Thanks to #712 and #714, ~85% of routed IPv4 space now has an ``as_domain`` that hits the map, so rows that were previously unattributable now get a ``source_name``/``source_type`` derived from the ASN. When the ASN domain misses the map, the raw AS name is used as ``source_name`` with ``source_type`` left null — still better than nothing. Crucially, ``source_reverse_dns`` and ``source_base_domain`` remain null on ASN-derived rows, so downstream consumers can still tell a PTR-resolved attribution apart from an ASN-derived one. ASN is stored as an integer at the schema level (Elasticsearch / OpenSearch mappings use ``Integer``) so consumers can do range queries and numeric sorts; dashboards can prepend ``AS`` at display time. The MMDB reader normalizes both IPinfo's ``"AS15169"`` string and MaxMind's ``autonomous_system_number`` int to the same int form. Also fixes a pre-existing caching bug in ``get_ip_address_info``: entries without reverse DNS were never written to the IP-info cache, so every no-PTR IP re-did the MMDB read and DNS attempt on every call. The cache write is now unconditional. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump to 9.9.0 and document the ASN fallback work Updates the changelog with a 9.9.0 entry covering the ASN-domain aliases (#712, #714), map-maintenance tooling fixes (#713), and the ASN-fallback source attribution added in this branch. Extends AGENTS.md to explain that ``base_reverse_dns_map.csv`` is now a mixed-namespace map (rDNS bases alongside ASN domains) and adds a short recipe for finding high-value ASN-domain misses against the bundled MMDB, so future contributors know where the map's second lookup path comes from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document project conventions previously held only in agent memory Promotes four conventions out of per-agent memory and into AGENTS.md so every contributor — human or agent — works from the same baseline: - Run ruff check + format before committing (Code Style). - Store natively numeric values as numbers, not pre-formatted strings (e.g. ASN as int 15169, not "AS15169"; ES/OS mappings as Integer) (Code Style). - Before rewriting a tracked list/data file from freshly-generated content, verify the existing content via git — these files accumulate manually-curated entries across sessions (Editing tracked data files). - A release isn't done until hatch-built sdist + wheel are attached to the GitHub release page; full 8-step sequence documented (Releases). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
250 lines
9.4 KiB
Markdown
250 lines
9.4 KiB
Markdown
# Sample outputs
|
|
|
|
## Sample aggregate report output
|
|
|
|
Here are the results from parsing the [example](https://dmarc.org/wiki/FAQ#I_need_to_implement_aggregate_reports.2C_what_do_they_look_like.3F)
|
|
report from the dmarc.org wiki. It's actually an older draft of
|
|
the 1.0 report schema standardized in
|
|
[RFC 7480 Appendix C](https://tools.ietf.org/html/rfc7489#appendix-C).
|
|
This draft schema is still in wide use.
|
|
|
|
`parsedmarc` produces consistent, normalized output, regardless
|
|
of the report schema.
|
|
|
|
### JSON aggregate report
|
|
|
|
```json
|
|
{
|
|
"xml_schema": "draft",
|
|
"report_metadata": {
|
|
"org_name": "acme.com",
|
|
"org_email": "noreply-dmarc-support@acme.com",
|
|
"org_extra_contact_info": "http://acme.com/dmarc/support",
|
|
"report_id": "9391651994964116463",
|
|
"begin_date": "2012-04-27 20:00:00",
|
|
"end_date": "2012-04-28 19:59:59",
|
|
"timespan_requires_normalization": false,
|
|
"original_timespan_seconds": 86399,
|
|
"errors": []
|
|
},
|
|
"policy_published": {
|
|
"domain": "example.com",
|
|
"adkim": "r",
|
|
"aspf": "r",
|
|
"p": "none",
|
|
"sp": "none",
|
|
"pct": "100",
|
|
"fo": "0"
|
|
},
|
|
"records": [
|
|
{
|
|
"source": {
|
|
"ip_address": "72.150.241.94",
|
|
"country": "US",
|
|
"reverse_dns": null,
|
|
"base_domain": null,
|
|
"name": null,
|
|
"type": null,
|
|
"asn": 7018,
|
|
"asn_name": "AT&T Services, Inc.",
|
|
"asn_domain": "att.com"
|
|
},
|
|
"count": 2,
|
|
"alignment": {
|
|
"spf": true,
|
|
"dkim": false,
|
|
"dmarc": true
|
|
},
|
|
"policy_evaluated": {
|
|
"disposition": "none",
|
|
"dkim": "fail",
|
|
"spf": "pass",
|
|
"policy_override_reasons": []
|
|
},
|
|
"identifiers": {
|
|
"header_from": "example.com",
|
|
"envelope_from": "example.com",
|
|
"envelope_to": null
|
|
},
|
|
"auth_results": {
|
|
"dkim": [
|
|
{
|
|
"domain": "example.com",
|
|
"selector": "none",
|
|
"result": "fail"
|
|
}
|
|
],
|
|
"spf": [
|
|
{
|
|
"domain": "example.com",
|
|
"scope": "mfrom",
|
|
"result": "pass"
|
|
}
|
|
]
|
|
},
|
|
"normalized_timespan": false,
|
|
"interval_begin": "2012-04-28 00:00:00",
|
|
"interval_end": "2012-04-28 23:59:59"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### CSV aggregate report
|
|
|
|
```text
|
|
xml_schema,org_name,org_email,org_extra_contact_info,report_id,begin_date,end_date,normalized_timespan,errors,domain,adkim,aspf,p,sp,pct,fo,source_ip_address,source_country,source_reverse_dns,source_base_domain,source_name,source_type,source_asn,source_asn_name,source_asn_domain,count,spf_aligned,dkim_aligned,dmarc_aligned,disposition,policy_override_reasons,policy_override_comments,envelope_from,header_from,envelope_to,dkim_domains,dkim_selectors,dkim_results,spf_domains,spf_scopes,spf_results
|
|
draft,acme.com,noreply-dmarc-support@acme.com,http://acme.com/dmarc/support,9391651994964116463,2012-04-28 00:00:00,2012-04-28 23:59:59,False,,example.com,r,r,none,none,100,0,72.150.241.94,US,,,,,2,True,False,True,none,,,example.com,example.com,,example.com,none,fail,example.com,mfrom,pass
|
|
draft,acme.com,noreply-dmarc-support@acme.com,http://acme.com/dmarc/support,9391651994964116463,2012-04-28 00:00:00,2012-04-28 23:59:59,False,,example.com,r,r,none,none,100,0,72.150.241.94,US,,,,,2,True,False,True,none,,,example.com,example.com,,example.com,none,fail,example.com,mfrom,pass
|
|
|
|
```
|
|
|
|
## Sample forensic report output
|
|
|
|
Thanks to GitHub user [xennn](https://github.com/xennn) for the anonymized
|
|
[forensic report email sample](<https://github.com/domainaware/parsedmarc/raw/master/samples/forensic/DMARC%20Failure%20Report%20for%20domain.de%20(mail-from%3Dsharepoint%40domain.de%2C%20ip%3D10.10.10.10).eml>).
|
|
|
|
### JSON forensic report
|
|
|
|
```json
|
|
{
|
|
"feedback_type": "auth-failure",
|
|
"user_agent": "Lua/1.0",
|
|
"version": "1.0",
|
|
"original_mail_from": "sharepoint@domain.de",
|
|
"original_rcpt_to": "peter.pan@domain.de",
|
|
"arrival_date": "Mon, 01 Oct 2018 11:20:27 +0200",
|
|
"message_id": "<38.E7.30937.BD6E1BB5@ mailrelay.de>",
|
|
"authentication_results": "dmarc=fail (p=none, dis=none) header.from=domain.de",
|
|
"delivery_result": "policy",
|
|
"auth_failure": [
|
|
"dmarc"
|
|
],
|
|
"reported_domain": "domain.de",
|
|
"arrival_date_utc": "2018-10-01 09:20:27",
|
|
"source": {
|
|
"ip_address": "10.10.10.10",
|
|
"country": null,
|
|
"reverse_dns": null,
|
|
"base_domain": null,
|
|
"name": null,
|
|
"type": null,
|
|
"asn": null,
|
|
"asn_name": null,
|
|
"asn_domain": null
|
|
},
|
|
"authentication_mechanisms": [],
|
|
"original_envelope_id": null,
|
|
"dkim_domain": null,
|
|
"sample_headers_only": false,
|
|
"sample": "Received: from Servernameone.domain.local (Servernameone.domain.local [10.10.10.10])\n\tby mailrelay.de (mail.DOMAIN.de) with SMTP id 38.E7.30937.BD6E1BB5; Mon, 1 Oct 2018 11:20:27 +0200 (CEST)\nDate: 01 Oct 2018 11:20:27 +0200\nMessage-ID: <38.E7.30937.BD6E1BB5@ mailrelay.de>\nTo: <peter.pan@domain.de>\nfrom: \"=?utf-8?B?SW50ZXJha3RpdmUgV2V0dGJld2VyYmVyLcOcYmVyc2ljaHQ=?=\" <sharepoint@domain.de>\nSubject: Subject\nMIME-Version: 1.0\nX-Mailer: Microsoft SharePoint Foundation 2010\nContent-Type: text/html; charset=utf-8\nContent-Transfer-Encoding: quoted-printable\n\n<html><head><base href=3D'\nwettbewerb' /></head><body><!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\"=\n><HTML><HEAD><META NAME=3D\"Generator\" CONTENT=3D\"MS Exchange Server version=\n 08.01.0240.003\"></html>\n",
|
|
"parsed_sample": {
|
|
"from": {
|
|
"display_name": "Interaktive Wettbewerber-Übersicht",
|
|
"address": "sharepoint@domain.de",
|
|
"local": "sharepoint",
|
|
"domain": "domain.de"
|
|
},
|
|
"to_domains": [
|
|
"domain.de"
|
|
],
|
|
"to": [
|
|
{
|
|
"display_name": null,
|
|
"address": "peter.pan@domain.de",
|
|
"local": "peter.pan",
|
|
"domain": "domain.de"
|
|
}
|
|
],
|
|
"subject": "Subject",
|
|
"timezone": "+2",
|
|
"mime-version": "1.0",
|
|
"date": "2018-10-01 09:20:27",
|
|
"content-type": "text/html; charset=utf-8",
|
|
"x-mailer": "Microsoft SharePoint Foundation 2010",
|
|
"body": "<html><head><base href='\nwettbewerb' /></head><body><!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\"><HTML><HEAD><META NAME=\"Generator\" CONTENT=\"MS Exchange Server version 08.01.0240.003\"></html>",
|
|
"received": [
|
|
{
|
|
"from": "Servernameone.domain.local Servernameone.domain.local 10.10.10.10",
|
|
"by": "mailrelay.de mail.DOMAIN.de",
|
|
"with": "SMTP id 38.E7.30937.BD6E1BB5",
|
|
"date": "Mon, 1 Oct 2018 11:20:27 +0200 CEST",
|
|
"hop": 1,
|
|
"date_utc": "2018-10-01 09:20:27",
|
|
"delay": 0
|
|
}
|
|
],
|
|
"content-transfer-encoding": "quoted-printable",
|
|
"message-id": "<38.E7.30937.BD6E1BB5@ mailrelay.de>",
|
|
"has_defects": false,
|
|
"headers": {
|
|
"Received": "from Servernameone.domain.local (Servernameone.domain.local [10.10.10.10])\n\tby mailrelay.de (mail.DOMAIN.de) with SMTP id 38.E7.30937.BD6E1BB5; Mon, 1 Oct 2018 11:20:27 +0200 (CEST)",
|
|
"Date": "01 Oct 2018 11:20:27 +0200",
|
|
"Message-ID": "<38.E7.30937.BD6E1BB5@ mailrelay.de>",
|
|
"To": "<peter.pan@domain.de>",
|
|
"from": "\"Interaktive Wettbewerber-Übersicht\" <sharepoint@domain.de>",
|
|
"Subject": "Subject",
|
|
"MIME-Version": "1.0",
|
|
"X-Mailer": "Microsoft SharePoint Foundation 2010",
|
|
"Content-Type": "text/html; charset=utf-8",
|
|
"Content-Transfer-Encoding": "quoted-printable"
|
|
},
|
|
"reply_to": [],
|
|
"cc": [],
|
|
"bcc": [],
|
|
"attachments": [],
|
|
"filename_safe_subject": "Subject"
|
|
}
|
|
}
|
|
```
|
|
|
|
### CSV forensic report
|
|
|
|
```text
|
|
feedback_type,user_agent,version,original_envelope_id,original_mail_from,original_rcpt_to,arrival_date,arrival_date_utc,subject,message_id,authentication_results,dkim_domain,source_ip_address,source_country,source_reverse_dns,source_base_domain,source_name,source_type,source_asn,source_asn_name,source_asn_domain,delivery_result,auth_failure,reported_domain,authentication_mechanisms,sample_headers_only
|
|
auth-failure,Lua/1.0,1.0,,sharepoint@domain.de,peter.pan@domain.de,"Mon, 01 Oct 2018 11:20:27 +0200",2018-10-01 09:20:27,Subject,<38.E7.30937.BD6E1BB5@ mailrelay.de>,"dmarc=fail (p=none, dis=none) header.from=domain.de",,10.10.10.10,,,,policy,dmarc,domain.de,,False
|
|
```
|
|
|
|
### JSON SMTP TLS report
|
|
|
|
```json
|
|
[
|
|
{
|
|
"organization_name": "Example Inc.",
|
|
"begin_date": "2024-01-09T00:00:00Z",
|
|
"end_date": "2024-01-09T23:59:59Z",
|
|
"report_id": "2024-01-09T00:00:00Z_example.com",
|
|
"policies": [
|
|
{
|
|
"policy_domain": "example.com",
|
|
"policy_type": "sts",
|
|
"policy_strings": [
|
|
"version: STSv1",
|
|
"mode: testing",
|
|
"mx: example.com",
|
|
"max_age: 86400"
|
|
],
|
|
"successful_session_count": 0,
|
|
"failed_session_count": 3,
|
|
"failure_details": [
|
|
{
|
|
"result_type": "validation-failure",
|
|
"failed_session_count": 2,
|
|
"sending_mta_ip": "209.85.222.201",
|
|
"receiving_ip": "173.212.201.41",
|
|
"receiving_mx_hostname": "example.com"
|
|
},
|
|
{
|
|
"result_type": "validation-failure",
|
|
"failed_session_count": 1,
|
|
"sending_mta_ip": "209.85.208.176",
|
|
"receiving_ip": "173.212.201.41",
|
|
"receiving_mx_hostname": "example.com"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
]
|
|
```
|