parsedmarc/parsedmarc/resources/maps

About

A mapping is meant to make it easier to identify who or what a sending source is. Please consider contributing additional mappings in a GitHub Pull Request.

Do not open these CSV files in Excel. It will replace Unicode characters with question marks. Use LibreOffice Calc instead.

base_reverse_dns_map.csv

A CSV file with three fields: base_reverse_dns, name, and type. Most of the time the base reverse DNS of sending service is closely related to the name of the service, but not always. Sometimes services will use multiple reverse DNS domains for the same service. For example, Intuit Mailchimp uses the base domains mcdlv.net, mcsv.net, and rsgsv.net. Having all of these mapped to the same service name and type makes it easier to answer questions like: "How many emails is Intuit Mailchimp sending as my domains?"

The service_type is based on the following rule precedence:

1. All email security services are identified as Email Security, no matter how or where they are hosted.
2. All marketing services are identified as Marketing, no matter how or where they are hosted.
3. All telecommunications providers that offer internet access are identified as ISP, even if they also offer other services, such as web hosting or email hosting.
4. All web hosting providers are identified as Web Host, even if the service also offers email hosting.
5. All email account providers are identified as Email Provider, no matter how or where they are hosted
6. All legitimate platforms offering their Software as a Service (SaaS) are identified as SaaS, regardless of industry. This helps simplify metrics.
7. All other senders that use their own domain as a Reverse DNS base domain should be identified based on their industry

• Agriculture
• Automotive
• Beauty
• Conglomerate
• Construction
• Consulting
• Defense
• Education
• Email Provider
• Email Security
• Entertainment
• Event Planning
• Finance
• Food
• Government
• Government Media
• Healthcare
• IaaS
• Industrial
• ISP
• Legal
• Logistics
• Manufacturing
• Marketing
• MSP
• MSSP
• News
• Nonprofit
• PaaS
• Photography
• Physical Security
• Print
• Publishing
• Real Estate
• Religion
• Retail
• SaaS
• Science
• Search Engine
• Social Media
• Sports
• Staffing
• Technology
• Travel
• Utilities
• Web Host

The list above is the authoritative set of allowed type values; sortlists.py parses the bullet items between the <!-- types-list:start --> and <!-- types-list:end --> HTML comment markers and uses them to validate every row's type column. Before validating the map, it also normalizes the block in place: trims whitespace, deduplicates case-insensitively, and sorts the entries alphabetically — so adding a new type is just a matter of inserting a - New Type line anywhere inside the markers, and sortlists.py will tidy it on the next run. Keep the markers themselves intact when editing.

The file currently contains over 5,000 mappings from a wide variety of email sending sources.

License

base_reverse_dns_map.csv is a curated derivative work. Many entries are derived from the bundled IPinfo Lite MMDB (as_domain and as_name fields) by walking the database with find_unmapped_as_domains.py for unmapped operators and classifying them via the workflow described in AGENTS.md. Because IPinfo Lite is licensed under Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0), this CSV is also distributed under CC BY-SA 4.0 with attribution to IPinfo for the underlying network identification data.

known_unknown_base_reverse_dns.txt

A list of reverse DNS base domains that could not be identified as belonging to a particular organization, service, or industry.

base_reverse_dns.csv

A CSV with the fields source_name and optionally message_count. This CSV can be generated by exporting the base DNS data from the Kibana or Splunk dashboards provided by parsedmarc. This file is not tracked by Git.

find_unknown_base_reverse_dns.py reads its input via -i/--input (default base_reverse_dns.csv) and writes its output via -o/--output (default unknown_base_reverse_dns.csv). The input file may alternatively be a plain-text file with one source name per line — e.g. a dashboard export of uncategorized sources — containing a mix of raw MMDB as_name strings and base reverse-DNS domains; the script auto-detects which format it was given.

unknown_base_reverse_dns.csv

A CSV file with the fields source_name and message_count. This file is not tracked by Git.

psl_overrides.txt

A plaintext list of reverse-DNS suffixes used to fold noisy subdomain patterns down to a single base. Each line is a suffix with an optional leading separator:

• -foo.com — any domain ending with -foo.com (for example, 1-2-3-4-foo.com) folds to foo.com.
• .foo.com — any domain ending with .foo.com (for example, host01.foo.com) folds to foo.com.
• foo.com — any domain ending with foo.com regardless of separator folds to foo.com.

Used by both find_unknown_base_reverse_dns.py and collect_domain_info.py, and auto-populated by detect_psl_overrides.py when N+ distinct full-IP-containing entries share a brand suffix. The leading . / - is stripped when computing the folded base.

find_bad_utf8.py

Locates invalid UTF-8 bytes in files and optionally tries to current them. Generated by GPT5. Helped me find where I had introduced invalid bytes in base_reverse_dns_map.csv.

find_unknown_base_reverse_dns.py

Reads the domains in base_reverse_dns.csv and writes the domains that are not in base_reverse_dns_map.csv or known_unknown_base_reverse_dns.txt to unknown_base_reverse_dns.csv, useful for identifying potential additional domains to contribute to base_reverse_dns_map.csv and known_unknown_base_reverse_dns.txt. Applies psl_overrides.txt to fold noisy subdomain patterns to their bases, and drops any entry containing a full IPv4 address (four dotted or dashed octets) so customer IPs never enter the pipeline.

When a source_name is not domain-shaped (e.g. Vodafone Group PLC), parsedmarc's ASN-fallback path emitted the raw MMDB as_name because the IP had no PTR and the corresponding as_domain was not in the map. The script translates such rows by looking the as_name up in the bundled ipinfo_lite.mmdb and substituting the matching as_domain (the one with the largest aggregate IPv4 footprint when an as_name covers multiple). Translated rows then flow through the normal known/known-unknown filter, so already-mapped operators drop out automatically and only genuinely new as_domain candidates land in the unknown CSV. AS names with no MMDB match are skipped with a warning.

detect_psl_overrides.py

Scans unknown_base_reverse_dns.csv for full-IP-containing entries that share a common brand suffix. Any suffix repeated by N+ distinct domains (default 3, configurable via --threshold) is appended to psl_overrides.txt, and every affected entry across the unknown / known-unknown / map files is folded to that suffix's base. Any remaining full-IP entries — whether they clustered or not — are then removed for privacy. After running, the newly exposed base domains still need to be researched and classified via collect_domain_info.py and a classifier pass. Supports --dry-run to preview without writing.

find_unmapped_as_domains.py

Walks every IPv4 record in the bundled ipinfo_lite.mmdb, aggregates the routed IPv4 footprint per as_domain, and subtracts domains already covered by base_reverse_dns_map.csv or known_unknown_base_reverse_dns.txt, applying psl_overrides.txt folding and the same full-IP privacy filter as find_unknown_base_reverse_dns.py. Writes unmapped_as_domains.csv (domain,ipv4_count,as_name, sorted by descending footprint), which collect_domain_info.py -i reads directly.

Candidates below --min-ips (default 4096, a /20) are dropped as an anti-poisoning guard — ASN registration data is self-declared to the RIRs and as_domain comes from registrant-controlled WHOIS, so a tiny ASN is cheap to register under a brand-impersonating domain name. The dropped count is always printed, never silently discarded.

unmapped_as_domains.csv

A CSV file with the fields domain, ipv4_count, and as_name, produced by find_unmapped_as_domains.py. This file is not tracked by Git.

collect_domain_info.py

Bulk enrichment collector. For every domain in unknown_base_reverse_dns.csv that is not already in base_reverse_dns_map.csv, runs whois on the domain, fetches a size-capped https:// GET, resolves A/AAAA records, and runs whois on the first resolved IP. Writes a TSV (domain_info.tsv by default) with the registrant org/country/registrar, page <title>/<meta description>, resolved IPs, and IP-WHOIS org/netname/country — the compact metadata a classifier needs to decide each domain in one pass. Respects psl_overrides.txt, skips full-IP entries, and is resume-safe (re-running only fetches domains missing from the output file).

The TSV also carries two derived columns that surface drift signals (and double as classification hints when a homepage explicitly names its operator):

• rebrand_signal — first ~120-char excerpt of the page where one of two regexes hit. (a) Body-text phrases: now X, is now part of X, formerly known as X, we became X, rebranded as X, acquired by X, merged with X, joined the X. Common false-positive trailing words (Now Available, Now Hiring, etc.) are filtered, and the captured brand must start with an uppercase letter. (b) Path / alt-text phrases: rebrand, brand-launch, brand-announcement, brand-change, name-change, our-new-name, new-name-for, acquisition-announcement, merger-announcement. The path scan runs against the JSON-unescaped page bytes, so it sees URL slugs and image alt attributes embedded in script blobs. Real-world case: bankonitusa.com's "now Navanta" banner is image-only — <a href="https://navanta.com/brand-launch-..."><img alt="Brand announcement"></a> — and pure body-text scanning misses it; the path regex matches via the brand-launch slug and Brand announcement alt attribute.
• external_links — comma-separated list of up to 5 distinct outbound link hosts, after stripping the input domain (and its subdomains) and a small noise list (social, CDN, analytics, app stores). Useful as context when reviewing a flagged row, but a noisy flag — most external links are to partners / customers / vendors that have no operator relationship — so detect_rebrands.py does not treat this column as a flag trigger by default. Pass --flag-external-links for a thorough sweep.

domain_info.tsv

The output of collect_domain_info.py. Tab-separated, one row per researched domain. Not tracked by Git — it is regenerated on demand and contains transient third-party WHOIS/HTML data.

classify_unknown_domains.py

Regex-based multilingual classifier that consumes a domain_info.tsv (from collect_domain_info.py) and emits two outputs: a CSV of map additions (domain,name,type rows) and a text file of known-unknown additions.

Useful for either lookup path that reads base_reverse_dns_map.csv:

• The original PTR-side flow that classifies reverse-DNS base domains derived from DMARC report source IPs (base_reverse_dns.csv → unknown_base_reverse_dns.csv → domain_info.tsv → this classifier).
• The MMDB-coverage flow that classifies ASN domains lifted from the bundled IPinfo Lite MMDB (the b5–b13 batches that drove distinct AS-domain coverage from ~10% to ~50% used this classifier as their regex baseline).

Run it from this directory:

python classify_unknown_domains.py \
    -i /tmp/batch_info.tsv \
    --map-out /tmp/additions.csv \
    --ku-out /tmp/ku_additions.txt

Detectors cover all 46 industry types listed in base_reverse_dns_map.csv above. Multilingual coverage is broadest for the high-volume detectors — Healthcare, Travel, Government, Retail, Finance, ISP, Web Host, Manufacturing, Logistics, Real Estate, Automotive, Legal, Agriculture have concept-translation parity across ~30 languages with multiple synonyms per language. Smaller detectors (Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media, Defense, IaaS, PaaS, SaaS, Beauty, Print, Publishing, Religion, Science, Event Planning, Staffing, Email Security, Email Provider, Marketing, Construction, Industrial, Utilities, Energy, Government Media, Physical Security, News, Nonprofit, Entertainment, Technology, Consulting) have ~10–20 languages with 1–3 keywords each. Each successive batch is expected to refine multilingual coverage as new patterns surface in the unclassified pool.

Brand-name selection prefers (in order): the MMDB as_name for the domain; the page title's first segment; non-redacted WHOIS registrant org; domain-derived fallback. A clean_brand step strips common legal-form suffixes (LLC / GmbH / Ltda / EIRELI / sp. z o.o. / etc.) and prefixes (PT, OOO). When the title has multiple segments separated by | / - / — etc., the segment whose simplified form contains the domain root is preferred — so e.g. accessmontana.com whose as_name is "MONTANA WEST, L.L.C." but whose title is "Internet, Phone & TV Bundles | Access Montana" maps to "Access Montana", not "Montana West".

The classifier is the regex baseline of step 4 of the Workflow for classifying unknown domains — it catches obvious cases at scale and leaves only the genuinely ambiguous to manual / LLM review. The empty HAND dict at the top of the script is an extension point for batch-specific overrides (e.g. acquisition aliases, brand-name corrections that don't fit any detector); each domain → ("Brand", "Type") entry wins over the auto-classifier and bypasses the guard below.

A brand-collision guard also loads base_reverse_dns_map.csv (--map, defaulting to the bundled map) so that a candidate whose proposed name matches an existing map display name, but whose domain has no lexical relationship to any existing key filed under that name, is demoted from --map-out to --ambiguous-out (marked name-collision-with-existing-map-entry) instead of being auto-promoted. This defends against a low-footprint or brand-impersonating candidate being silently attributed to an established operator, and applies to both the PTR-side and MMDB-coverage flows.

detect_rebrands.py

Cadence: run roughly once a year. Not part of the standard mapping workflow — operator rebrands and acquisitions accumulate slowly, and a yearly sweep is sufficient to keep base_reverse_dns_map.csv from drifting out of date. There is no benefit to running it more often.

Drift sweep that re-fetches every key in base_reverse_dns_map.csv with the same machinery as collect_domain_info.py and writes a TSV (rebrand_drift.tsv by default) of rows where a drift signal fired. Two signals are flagged by default:

• rebrand_signal — the collector's body-text and path/alt-text regexes (see above) matched.
• redirect_changed — the homepage's final URL host is not the input domain or a subdomain of it (typical case-1 acquisition redirect, e.g. vodafone.is → syn.is).

external_links is captured into the output for context but is not a default trigger — most outbound links are to partners / customers / vendors and would generate noise. Pass --flag-external-links to also flag on this column during a thorough sweep where missing an image-only banner that lacks a rebrand-themed slug or alt text is worse than the noise.

The output is for periodic review, not automated map mutation. Each hit is one corroborating source; promoting a flagged row into the map still requires a second source per the two-corroborating-sources rule in AGENTS.md. Resume-safe: re-running only re-fetches keys not already in the output file. Use --limit N to spot-check a slice and --include-clean to also write non-flagged rows for inspection of the no-signal majority.

rebrand_drift.tsv

The output of detect_rebrands.py. Tab-separated, one row per flagged map key. Not tracked by Git — regenerated on demand.

sortlists.py

Validation and sorting helper invoked as a module. Alphabetically sorts base_reverse_dns_map.csv (case-insensitive by first column, preserving CRLF line endings), deduplicates entries, validates that every type appears in this README's authoritative type list (parsed from the <!-- types-list:start --> / <!-- types-list:end --> block above), and warns on names that contain unescaped commas or stray whitespace. Run it after any batch merge before committing.