* Surface ASN info and fall back to it when a PTR is absent
Adds three new fields to every IP source record — ``asn`` (integer,
e.g. 15169), ``asn_name`` (``"Google LLC"``), ``asn_domain``
(``"google.com"``) — sourced from the bundled IPinfo Lite MMDB. These
flow through to CSV, JSON, Elasticsearch, OpenSearch, and Splunk
outputs as ``source_asn``, ``source_asn_name``, ``source_asn_domain``.
More importantly: when an IP has no reverse DNS (common for many
large senders), source attribution now falls back to the ASN domain
as a lookup key into the same ``reverse_dns_map``. Thanks to #712
and #714, ~85% of routed IPv4 space now has an ``as_domain`` that
hits the map, so rows that were previously unattributable now get a
``source_name``/``source_type`` derived from the ASN. When the ASN
domain misses the map, the raw AS name is used as ``source_name``
with ``source_type`` left null — still better than nothing.
Crucially, ``source_reverse_dns`` and ``source_base_domain`` remain
null on ASN-derived rows, so downstream consumers can still tell a
PTR-resolved attribution apart from an ASN-derived one.
ASN is stored as an integer at the schema level (Elasticsearch /
OpenSearch mappings use ``Integer``) so consumers can do range
queries and numeric sorts; dashboards can prepend ``AS`` at display
time. The MMDB reader normalizes both IPinfo's ``"AS15169"`` string
and MaxMind's ``autonomous_system_number`` int to the same int form.
Also fixes a pre-existing caching bug in ``get_ip_address_info``:
entries without reverse DNS were never written to the IP-info cache,
so every no-PTR IP re-did the MMDB read and DNS attempt on every
call. The cache write is now unconditional.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bump to 9.9.0 and document the ASN fallback work
Updates the changelog with a 9.9.0 entry covering the ASN-domain
aliases (#712, #714), map-maintenance tooling fixes (#713), and the
ASN-fallback source attribution added in this branch.
Extends AGENTS.md to explain that ``base_reverse_dns_map.csv`` is now
a mixed-namespace map (rDNS bases alongside ASN domains) and adds a
short recipe for finding high-value ASN-domain misses against the
bundled MMDB, so future contributors know where the map's second
lookup path comes from.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Document project conventions previously held only in agent memory
Promotes four conventions out of per-agent memory and into AGENTS.md
so every contributor — human or agent — works from the same baseline:
- Run ruff check + format before committing (Code Style).
- Store natively numeric values as numbers, not pre-formatted strings
(e.g. ASN as int 15169, not "AS15169"; ES/OS mappings as Integer)
(Code Style).
- Before rewriting a tracked list/data file from freshly-generated
content, verify the existing content via git — these files
accumulate manually-curated entries across sessions (Editing tracked
data files).
- A release isn't done until hatch-built sdist + wheel are attached to
the GitHub release page; full 8-step sequence documented (Releases).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 43 more high-confidence aliases from the top IPv4-weighted misses
remaining after #712. Bumps ASN-domain coverage of the bundled ipinfo
lite MMDB from 84.0% to 85.0% — modest, as expected; the tail is a
long list of small ASNs where diminishing returns kick in hard.
This is the last bulk alias pass. Any remaining gap should be filled
by falling back to the raw `as_name` from the MMDB at attribution
time, not by continuing to hand-classify thousands of small ASNs.
Also promotes nask.pl out of known_unknown_base_reverse_dns.txt —
NASK is the Polish national research and academic network, which is
unambiguous from ASN context.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sortlists.py had three bugs that let bad data through:
- The `type` column validator was keyed on "Type" (capital T) but the
CSV header is "type" (lowercase), so every row bypassed validation.
- `types` was read via `f.readlines()` without stripping, so even if
the key had matched, values like `"ISP\n"` would never equal `"ISP"`.
- The map was sorted case-sensitively, but README and AGENTS.md both
state the map is sorted alphabetically case-insensitive.
Fixing the validator surfaced eight pre-existing rows with invalid or
inconsistent `type` values. All are now corrected:
- Two types listed in README but missing from base_reverse_dns_types.txt
(Religion, Utilities) have been added so the README and authoritative
types file agree.
- dhl.com, ghm-grenoble.fr, regusnet.com had lowercase-casing type
values (`logistics`, `healthcare`, `Real estate`) corrected to match
the canonical spellings.
- lodestonegroup.com was typed `Insurance`, which is not a listed
industry; reclassified as `Finance` (the closest listed category
for an insurance brokerage).
Also fixes one stale map entry: `rt.ru` was listed as `RT,Government
Media`, conflating Rostelecom (the Russian telco that owns and uses
rt.ru) with RT / Russia Today (which uses rt.com). Corrected to
`Rostelecom,ISP`.
Switching to case-insensitive sort moves exactly one row — the sole
mixed-case key `United-domains.de` — from the top of the file (where
ASCII ordering placed it before all lowercase keys) into the "united"
range where human readers would expect it.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add ASN-domain aliases to base_reverse_dns_map.csv
Adds 457 entries keyed on the `as_domain` values that ship in
`ipinfo_lite.mmdb`, so that the existing reverse_dns_map can serve as
a lookup table for IPs that resolve no PTR — the common case for many
large senders.
Before this change only ~33.8% of routed IPv4 space had an `as_domain`
that matched a map key; after, ~84.0%. All additions are brands that
were already represented in the map under a different rDNS-base key
(e.g. `comcast.com` alongside the existing `comcast.net`), plus a
handful of well-known operators that previously had no representation
at all.
Also promotes 10 entries out of known_unknown_base_reverse_dns.txt
(a1.net, actcorp.in, ais.co.th, emirates.net.ae, eolo.it, fpt.vn,
ibm.com, movilnet.com.ve, ote.gr, singnet.com.sg) — each is a
well-known operator whose identity is unambiguous from ASN context
even if the original rDNS base alone was inconclusive.
No code changes; this is purely data, in preparation for a follow-up
that wires `as_domain` into the source-attribution fallback path when
a report row has no reverse DNS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Reclassify Zscaler as SaaS
Zscaler is consumed as a self-service security platform, not delivered
as a managed service, so SaaS fits better than MSSP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch the bundled IP-to-country database from DB-IP Country Lite to
IPinfo Lite for greater lookup accuracy. The download URL, cached
filename, and packaged module path all move from
dbip/dbip-country-lite.mmdb to ipinfo/ipinfo_lite.mmdb.
IPinfo Lite uses a different MMDB schema (flat country_code) that is
incompatible with geoip2's Reader.country() helper, so get_ip_address_country()
now uses maxminddb directly and handles both the IPinfo schema and
the MaxMind/DBIP nested country.iso_code schema so users who drop in
their own MMDB from any of these providers continue to work.
Drop the geoip2 dependency (it was only used for the incompatible
helper) and add maxminddb as a direct dependency — it was already
installed transitively through geoip2.
Callers that imported parsedmarc.resources.dbip directly need to switch
to parsedmarc.resources.ipinfo. Old parsedmarc versions downloading
from the dbip/ GitHub raw URL will 404 and fall back to their bundled
copy — this is the documented behavior of load_ip_db().
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Port DNS reliability fixes from checkdmarc 5.15.x: cap per-query UDP
timeout at min(1.0, timeout) so a single dropped datagram no longer
consumes the entire lifetime budget, scale lifetime by nameserver count
for proper failover, and add a retries kwarg that retries on
LifetimeTimeout, NoNameservers (SERVFAIL), and OSError during TCP
fallback (NXDOMAIN and NoAnswer remain non-retryable).
Thread dns_retries through the parser API and expose it via
--dns-retries / the dns_retries INI option. Centralize DNS defaults in
parsedmarc.constants and add RECOMMENDED_DNS_NAMESERVERS for opt-in
cross-provider failover.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Auto-download psl_overrides.txt at startup (and whenever the reverse DNS
map is reloaded) via load_psl_overrides(); add local_psl_overrides_path
and psl_overrides_url config options
- Add collect_domain_info.py and detect_psl_overrides.py for bulk WHOIS/HTTP
enrichment and automatic cluster-based PSL override detection
- Block full-IPv4 reverse-DNS entries from ever entering
base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, or
unknown_base_reverse_dns.csv, and sweep pre-existing IP entries
- Add Religion and Utilities to the allowed service_type values
- Document the full map-maintenance workflow in AGENTS.md
- Substantial expansion of base_reverse_dns_map.csv (net ~+1,000 entries)
- Add 26 tests covering the new loader, IP filter, PSL fold logic, and
cluster detection
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Download the latest DB-IP Country Lite mmdb from GitHub on startup and
SIGHUP, caching it locally, with fallback to a previously cached or
bundled copy. Skipped when the offline flag is set. Adds ip_db_url
config option (PARSEDMARC_GENERAL_IP_DB_URL) to override the download
URL. Bumps version to 9.6.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Should have caught this on previous fix for since. the current time is used on line 2145: connection.fetch_messages(reports_folder, since=current_time)
if that code is called and it usually won't be depending upon configuration it will fail with the time format being wrong: yyyy-mm-ddThh:mm:ss.zzzzzz+00:00Z --- this removes the extra "Z" that is not needed since utc offset is already specified and becomes invalid.
- Fixed `FileNotFoundError` when using Maildir with Docker volume mounts. Python's `mailbox.Maildir(create=True)` only creates `cur/new/tmp` subdirectories when the top-level directory doesn't exist; Docker volume mounts pre-create the directory as empty, skipping subdirectory creation. parsedmarc now explicitly creates the subdirectories when `maildir_create` is enabled.
- Maildir UID mismatch no longer crashes the process. In Docker containers where volume ownership differs from the container UID, parsedmarc now logs a warning instead of raising an exception. Also handles `os.setuid` failures gracefully in containers without `CAP_SETUID`.
- Token file writes (MS Graph and Gmail) now create parent directories automatically, preventing `FileNotFoundError` when the token path points to a directory that doesn't yet exist.
- File paths from config (`token_file`, `credentials_file`, `cert_path`, `log_file`, `output`, `ip_db_path`, `maildir_path`, syslog cert paths, etc.) now expand `~` and `$VAR` references via `os.path.expanduser`/`os.path
- Fixed `FileNotFoundError` when using Maildir with Docker volume mounts. Python's `mailbox.Maildir(create=True)` only creates `cur/new/tmp` subdirectories when the top-level directory doesn't exist; Docker volume mounts pre-create the directory as empty, skipping subdirectory creation. parsedmarc now explicitly creates the subdirectories when `maildir_create` is enabled.
Fix formatting of ISO 8601 date strings for MSGraphConnection. format yyyy-dd-mmThh:MM:SS.zzzzzz+00:00 already has a timezone indicated. The extra Z is invalid in this format. specifying a "since" in config file causes msgraph to error due to invalid time stamp.
Add environment variable configuration support and update documentation
- Introduced support for configuration via environment variables using the `PARSEDMARC_{SECTION}_{KEY}` format.
- Added `PARSEDMARC_CONFIG_FILE` variable to specify the config file path.
- Enabled env-only mode for file-less Docker deployments.
- Implemented explicit read permission checks on config files.
- Updated changelog and usage documentation to reflect these changes.
### Added
- Extracted `load_reverse_dns_map()` utility function in `utils.py` for loading the reverse DNS map independently of individual IP lookups.
- SIGHUP reload now re-downloads/reloads the reverse DNS map, so changes take effect without restarting.
- Add premade OpenSearch index patterns, visualizations, and dashboards
### Changed
- When `index_prefix_domain_map` is configured, SMTP TLS reports for domains not in the map are now silently dropped instead of being output. Unlike DMARC, TLS-RPT has no DNS authorization records, so this filtering prevents processing reports for unrelated domains.
- Bump OpenSearch support to `< 4`
### Fixed
- Fixed `get_index_prefix` using wrong key (`domain` instead of `policy_domain`) for SMTP TLS reports, which prevented domain map matching from working for TLS reports.
- Domain matching in `get_index_prefix` now lowercases the domain for case-insensitive comparison.
Elasticsearch and OpenSearch now verify SSL certificates by default when `ssl = True`, even without a `cert_path`
- Added `skip_certificate_verification` option to the `elasticsearch` and `opensearch` configuration sections for consistency with `splunk_hec`
- Splunk HEC `skip_certificate_verification` now works correctly with self-signed certificates
- SMTP TLS reports no longer fail when saving to multiple output targets (e.g. Elasticsearch and OpenSearch) due to in-place mutation of the report dict
- Output client initialization errors now identify which module failed (e.g. "OpenSearch: ConnectionError..." instead of generic "Output client error")
- Enhanced error handling for output client initialization
- Better checking of `msconfig` configuration (PR #695)
- Updated `dbip-country-lite` database to version `2026-03`
- Changed - DNS query error logging level from `warning` to `debug`
* Add MS Graph certificate authentication support
* Preserve MS Graph constructor compatibility
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
