# About

A mapping is meant to make it easier to identify who or what a sending source is. Please consider contributing
additional mappings in a GitHub Pull Request.

Do not open these CSV files in Excel. It will replace Unicode characters with question marks. Use LibreOffice Calc instead.

## base_reverse_dns_map.csv

A CSV file with three fields: `base_reverse_dns`, `name`, and `type`.
Most of the time the base reverse DNS of sending service is closely related to the name of the
service, but not always. Sometimes services will use multiple reverse DNS domains for the same service. For example,
Intuit Mailchimp uses the base domains `mcdlv.net`, `mcsv.net`,
and `rsgsv.net`. Having all of these mapped to the same service name and type makes it easier to answer questions like:
"How many emails is Intuit Mailchimp sending as my domains?"

The `service_type` is based on the following rule precedence:

1. All email security services are identified as `Email Security`, no matter how or where they are hosted.
2. All marketing services are identified as `Marketing`, no matter how or where they are hosted.
3. All telecommunications providers that offer internet access are identified as `ISP`, even if they also offer other services, such as web hosting or email hosting.
4. All web hosting providers are identified as `Web Host`, even if the service also offers email hosting.
5. All email account providers are identified as `Email Provider`, no matter how or where they are hosted
6. All legitimate platforms offering their Software as a Service (SaaS) are identified as `SaaS`, regardless of industry. This helps simplify metrics.
7. All other senders that use their own domain as a Reverse DNS base domain should be identified based on their industry

<!-- types-list:start -->
- Agriculture
- Automotive
- Beauty
- Conglomerate
- Construction
- Consulting
- Defense
- Education
- Email Provider
- Email Security
- Entertainment
- Event Planning
- Finance
- Food
- Government
- Government Media
- Healthcare
- IaaS
- Industrial
- ISP
- Legal
- Logistics
- Manufacturing
- Marketing
- MSP
- MSSP
- News
- Nonprofit
- PaaS
- Photography
- Physical Security
- Print
- Publishing
- Real Estate
- Religion
- Retail
- SaaS
- Science
- Search Engine
- Social Media
- Sports
- Staffing
- Technology
- Travel
- Utilities
- Web Host
<!-- types-list:end -->

The list above is the authoritative set of allowed `type` values; `sortlists.py` parses the bullet items between the `<!-- types-list:start -->` and `<!-- types-list:end -->` HTML comment markers and uses them to validate every row's `type` column. Before validating the map, it also normalizes the block in place: trims whitespace, deduplicates case-insensitively, and sorts the entries alphabetically — so adding a new type is just a matter of inserting a `- New Type` line anywhere inside the markers, and `sortlists.py` will tidy it on the next run. Keep the markers themselves intact when editing.

The file currently contains over 5,000 mappings from a wide variety of email sending sources.

### License

`base_reverse_dns_map.csv` is a curated derivative work. Many entries
are derived from the bundled IPinfo Lite MMDB (`as_domain` and
`as_name` fields) by walking the database with `find_unmapped_as_domains.py`
for unmapped operators and classifying them via the workflow described in
[AGENTS.md](AGENTS.md).
Because IPinfo Lite is licensed under
[Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/),
this CSV is also distributed under **CC BY-SA 4.0** with attribution to
[IPinfo](https://ipinfo.io/) for the underlying network identification
data.

## known_unknown_base_reverse_dns.txt

A list of reverse DNS base domains that could not be identified as belonging to a particular organization, service, or industry.

## base_reverse_dns.csv

A CSV with the fields `source_name` and optionally `message_count`. This CSV can be generated by exporting the base DNS data from the Kibana or Splunk dashboards provided by parsedmarc. This file is not tracked by Git.

`find_unknown_base_reverse_dns.py` reads its input via `-i`/`--input` (default `base_reverse_dns.csv`) and writes its output via `-o`/`--output` (default `unknown_base_reverse_dns.csv`). The input file may alternatively be a plain-text file with one source name per line — e.g. a dashboard export of uncategorized sources — containing a mix of raw MMDB `as_name` strings and base reverse-DNS domains; the script auto-detects which format it was given.

## unknown_base_reverse_dns.csv

A CSV file with the fields `source_name` and `message_count`. This file is not tracked by Git.

## psl_overrides.txt

A plaintext list of reverse-DNS suffixes used to fold noisy subdomain patterns down to a single base. Each line is a suffix with an optional leading separator:

- `-foo.com` — any domain ending with `-foo.com` (for example, `1-2-3-4-foo.com`) folds to `foo.com`.
- `.foo.com` — any domain ending with `.foo.com` (for example, `host01.foo.com`) folds to `foo.com`.
- `foo.com` — any domain ending with `foo.com` regardless of separator folds to `foo.com`.

Used by both `find_unknown_base_reverse_dns.py` and `collect_domain_info.py`, and auto-populated by `detect_psl_overrides.py` when N+ distinct full-IP-containing entries share a brand suffix. The leading `.` / `-` is stripped when computing the folded base.

## find_bad_utf8.py

Locates invalid UTF-8 bytes in files and optionally tries to current them. Generated by GPT5. Helped me find where I had introduced invalid bytes in `base_reverse_dns_map.csv`.

## find_unknown_base_reverse_dns.py

Reads the domains in `base_reverse_dns.csv` and writes the domains that are not in `base_reverse_dns_map.csv` or `known_unknown_base_reverse_dns.txt` to `unknown_base_reverse_dns.csv`, useful for identifying potential additional domains to contribute to `base_reverse_dns_map.csv` and `known_unknown_base_reverse_dns.txt`. Applies `psl_overrides.txt` to fold noisy subdomain patterns to their bases, and drops any entry containing a full IPv4 address (four dotted or dashed octets) so customer IPs never enter the pipeline.

When a `source_name` is not domain-shaped (e.g. `Vodafone Group PLC`), parsedmarc's ASN-fallback path emitted the raw MMDB `as_name` because the IP had no PTR and the corresponding `as_domain` was not in the map. The script translates such rows by looking the `as_name` up in the bundled `ipinfo_lite.mmdb` and substituting the matching `as_domain` (the one with the largest aggregate IPv4 footprint when an `as_name` covers multiple). Translated rows then flow through the normal known/known-unknown filter, so already-mapped operators drop out automatically and only genuinely new `as_domain` candidates land in the unknown CSV. AS names with no MMDB match are skipped with a warning.

## detect_psl_overrides.py

Scans `unknown_base_reverse_dns.csv` for full-IP-containing entries that share a common brand suffix. Any suffix repeated by N+ distinct domains (default 3, configurable via `--threshold`) is appended to `psl_overrides.txt`, and every affected entry across the unknown / known-unknown / map files is folded to that suffix's base. Any remaining full-IP entries — whether they clustered or not — are then removed for privacy. After running, the newly exposed base domains still need to be researched and classified via `collect_domain_info.py` and a classifier pass. Supports `--dry-run` to preview without writing.

## find_unmapped_as_domains.py

Walks every IPv4 record in the bundled `ipinfo_lite.mmdb`, aggregates the routed IPv4 footprint per `as_domain`, and subtracts domains already covered by `base_reverse_dns_map.csv` or `known_unknown_base_reverse_dns.txt`, applying `psl_overrides.txt` folding and the same full-IP privacy filter as `find_unknown_base_reverse_dns.py`. Writes `unmapped_as_domains.csv` (`domain,ipv4_count,as_name`, sorted by descending footprint), which `collect_domain_info.py -i` reads directly.

Candidates below `--min-ips` (default `4096`, a /20) are dropped as an anti-poisoning guard — ASN registration data is self-declared to the RIRs and `as_domain` comes from registrant-controlled WHOIS, so a tiny ASN is cheap to register under a brand-impersonating domain name. The dropped count is always printed, never silently discarded.

## unmapped_as_domains.csv

A CSV file with the fields `domain`, `ipv4_count`, and `as_name`, produced by `find_unmapped_as_domains.py`. This file is not tracked by Git.

## collect_domain_info.py

Bulk enrichment collector. For every domain in `unknown_base_reverse_dns.csv` that is not already in `base_reverse_dns_map.csv`, runs `whois` on the domain, fetches a size-capped `https://` GET, resolves A/AAAA records, and runs `whois` on the first resolved IP. Writes a TSV (`domain_info.tsv` by default) with the registrant org/country/registrar, page `<title>`/`<meta description>`, resolved IPs, and IP-WHOIS org/netname/country — the compact metadata a classifier needs to decide each domain in one pass. Respects `psl_overrides.txt`, skips full-IP entries, and is resume-safe (re-running only fetches domains missing from the output file).

The TSV also carries two derived columns that surface drift signals (and double as classification hints when a homepage explicitly names its operator):

- `rebrand_signal` — first ~120-char excerpt of the page where one of two regexes hit. (a) Body-text phrases: *now X*, *is now part of X*, *formerly known as X*, *we became X*, *rebranded as X*, *acquired by X*, *merged with X*, *joined the X*. Common false-positive trailing words (`Now Available`, `Now Hiring`, etc.) are filtered, and the captured brand must start with an uppercase letter. (b) Path / alt-text phrases: `rebrand`, `brand-launch`, `brand-announcement`, `brand-change`, `name-change`, `our-new-name`, `new-name-for`, `acquisition-announcement`, `merger-announcement`. The path scan runs against the JSON-unescaped page bytes, so it sees URL slugs and image alt attributes embedded in script blobs. Real-world case: bankonitusa.com's "now Navanta" banner is image-only — `<a href="https://navanta.com/brand-launch-..."><img alt="Brand announcement"></a>` — and pure body-text scanning misses it; the path regex matches via the `brand-launch` slug and `Brand announcement` alt attribute.
- `external_links` — comma-separated list of up to 5 distinct outbound link hosts, after stripping the input domain (and its subdomains) and a small noise list (social, CDN, analytics, app stores). Useful as context when reviewing a flagged row, but a noisy *flag* — most external links are to partners / customers / vendors that have no operator relationship — so `detect_rebrands.py` does not treat this column as a flag trigger by default. Pass `--flag-external-links` for a thorough sweep.

## domain_info.tsv

The output of `collect_domain_info.py`. Tab-separated, one row per researched domain. Not tracked by Git — it is regenerated on demand and contains transient third-party WHOIS/HTML data.

## classify_unknown_domains.py

Regex-based multilingual classifier that consumes a `domain_info.tsv` (from `collect_domain_info.py`) and emits two outputs: a CSV of map additions (`domain,name,type` rows) and a text file of known-unknown additions.

Useful for either lookup path that reads `base_reverse_dns_map.csv`:

- The original PTR-side flow that classifies reverse-DNS base domains derived from DMARC report source IPs (`base_reverse_dns.csv` → `unknown_base_reverse_dns.csv` → `domain_info.tsv` → this classifier).
- The MMDB-coverage flow that classifies ASN domains lifted from the bundled IPinfo Lite MMDB (the b5–b13 batches that drove distinct AS-domain coverage from ~10% to ~50% used this classifier as their regex baseline).

Run it from this directory:

```bash
python classify_unknown_domains.py \
    -i /tmp/batch_info.tsv \
    --map-out /tmp/additions.csv \
    --ku-out /tmp/ku_additions.txt
```

Detectors cover all 44 industry types listed in [base_reverse_dns_map.csv](#base_reverse_dns_mapcsv) above. Multilingual coverage is broadest for the high-volume detectors — Healthcare, Travel, Government, Retail, Finance, ISP, Web Host, Manufacturing, Logistics, Real Estate, Automotive, Legal, Agriculture have concept-translation parity across ~30 languages with multiple synonyms per language. Smaller detectors (Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media, Defense, IaaS, PaaS, SaaS, Beauty, Print, Publishing, Religion, Science, Event Planning, Staffing, Email Security, Email Provider, Marketing, Construction, Industrial, Utilities, Energy, Government Media, Physical Security, News, Nonprofit, Entertainment, Technology, Consulting) have ~10–20 languages with 1–3 keywords each. Each successive batch is expected to refine multilingual coverage as new patterns surface in the unclassified pool.

Brand-name selection prefers (in order): the MMDB `as_name` for the domain; the page title's first segment; non-redacted WHOIS registrant org; domain-derived fallback. A `clean_brand` step strips common legal-form suffixes (LLC / GmbH / Ltda / EIRELI / sp. z o.o. / etc.) and prefixes (PT, OOO). When the title has multiple segments separated by `|` / `-` / `—` etc., the segment whose simplified form contains the domain root is preferred — so e.g. accessmontana.com whose `as_name` is "MONTANA WEST, L.L.C." but whose title is "Internet, Phone & TV Bundles | Access Montana" maps to "Access Montana", not "Montana West".

The classifier is the regex baseline of step 4 of the [Workflow for classifying unknown domains](AGENTS.md#workflow-for-classifying-unknown-domains) — it catches obvious cases at scale and leaves only the genuinely ambiguous to manual / LLM review. The empty `HAND` dict at the top of the script is an extension point for batch-specific overrides (e.g. acquisition aliases, brand-name corrections that don't fit any detector); each `domain → ("Brand", "Type")` entry wins over the auto-classifier and bypasses the guard below.

A brand-collision guard also loads `base_reverse_dns_map.csv` (`--map`, defaulting to the bundled map) so that a candidate whose proposed name matches an *existing* map display name, but whose domain has no lexical relationship to any existing key filed under that name, is demoted from `--map-out` to `--ambiguous-out` (marked `name-collision-with-existing-map-entry`) instead of being auto-promoted. This defends against a low-footprint or brand-impersonating candidate being silently attributed to an established operator, and applies to both the PTR-side and MMDB-coverage flows.

## detect_rebrands.py

**Cadence: run roughly once a year.** Not part of the standard mapping workflow — operator rebrands and acquisitions accumulate slowly, and a yearly sweep is sufficient to keep `base_reverse_dns_map.csv` from drifting out of date. There is no benefit to running it more often.

Drift sweep that re-fetches every key in `base_reverse_dns_map.csv` with the same machinery as `collect_domain_info.py` and writes a TSV (`rebrand_drift.tsv` by default) of rows where a drift signal fired. Two signals are flagged by default:

- `rebrand_signal` — the collector's body-text and path/alt-text regexes (see above) matched.
- `redirect_changed` — the homepage's final URL host is not the input domain or a subdomain of it (typical case-1 acquisition redirect, e.g. vodafone.is → syn.is).

`external_links` is captured into the output for context but is not a default trigger — most outbound links are to partners / customers / vendors and would generate noise. Pass `--flag-external-links` to also flag on this column during a thorough sweep where missing an image-only banner that lacks a rebrand-themed slug or alt text is worse than the noise.

The output is for periodic review, not automated map mutation. Each hit is one corroborating source; promoting a flagged row into the map still requires a second source per the two-corroborating-sources rule in [AGENTS.md](AGENTS.md). Resume-safe: re-running only re-fetches keys not already in the output file. Use `--limit N` to spot-check a slice and `--include-clean` to also write non-flagged rows for inspection of the no-signal majority.

## rebrand_drift.tsv

The output of `detect_rebrands.py`. Tab-separated, one row per flagged map key. Not tracked by Git — regenerated on demand.

## sortlists.py

Validation and sorting helper invoked as a module. Alphabetically sorts `base_reverse_dns_map.csv` (case-insensitive by first column, preserving CRLF line endings), deduplicates entries, validates that every `type` appears in this README's authoritative type list (parsed from the `<!-- types-list:start -->` / `<!-- types-list:end -->` block above), and warns on names that contain unescaped commas or stray whitespace. Run it after any batch merge before committing.
