# About

A mapping is meant to make it easier to identify who or what a sending source is. Please consider contributing
additional mappings in a GitHub Pull Request.

Do not open these CSV files in Excel. It will replace Unicode characters with question marks. Use LibreOffice Calc instead.

## base_reverse_dns_map.csv

A CSV file with three fields: `base_reverse_dns`, `name`, and `type`.
Most of the time the base reverse DNS of sending service is closely related to the name of the
service, but not always. Sometimes services will use multiple reverse DNS domains for the same service. For example,
Intuit Mailchimp uses the base domains `mcdlv.net`, `mcsv.net`,
and `rsgsv.net`. Having all of these mapped to the same service name and type makes it easier to answer questions like:
"How many emails is Intuit Mailchimp sending as my domains?"

The `service_type` is based on the following rule precedence:

1. All email security services are identified as `Email Security`, no matter how or where they are hosted.
2. All marketing services are identified as `Marketing`, no matter how or where they are hosted.
3. All telecommunications providers that offer internet access are identified as `ISP`, even if they also offer other services, such as web hosting or email hosting.
4. All web hosting providers are identified as `Web Hosting`, even if the service also offers email hosting.
5. All email account providers are identified as `Email Provider`, no matter how or where they are hosted
6. All legitimate platforms offering their Software as a Service (SaaS) are identified as `SaaS`, regardless of industry. This helps simplify metrics.
7. All other senders that use their own domain as a Reverse DNS base domain should be identified based on their industry

- Agriculture
- Automotive
- Beauty
- Conglomerate
- Construction
- Consulting
- Defense
- Education
- Email Provider
- Email Security
- Entertainment
- Event Planning
- Finance
- Food
- Government
- Government Media
- Healthcare
- IaaS
- Industrial
- ISP
- Legal
- Logistics
- Manufacturing
- Marketing
- MSP
- MSSP
- News
- Nonprofit
- PaaS
- Photography
- Physical Security
- Print
- Publishing
- Real Estate
- Religion
- Retail
- SaaS
- Science
- Search Engine
- Social Media
- Sports
- Staffing
- Technology
- Travel
- Utilities
- Web Host

The file currently contains over 1,400 mappings from a wide variety of email sending sources.

## known_unknown_base_reverse_dns.txt

A list of reverse DNS base domains that could not be identified as belonging to a particular organization, service, or industry.

## base_reverse_dns.csv

A CSV with the fields `source_name` and optionally `message_count`. This CSV can be generated by exporting the base DNS data from the Kibana or Splunk dashboards provided by parsedmarc. This file is not tracked by Git.

## unknown_base_reverse_dns.csv

A CSV file with the fields `source_name` and `message_count`. This file is not tracked by Git.

## base_reverse_dns_types.txt

A plaintext list (one per line) of the allowed `type` values. Should match the industry list in this README; used by `sortlists.py` as the authoritative set for validation.

## psl_overrides.txt

A plaintext list of reverse-DNS suffixes used to fold noisy subdomain patterns down to a single base. Each line is a suffix with an optional leading separator:

- `-foo.com` — any domain ending with `-foo.com` (for example, `1-2-3-4-foo.com`) folds to `foo.com`.
- `.foo.com` — any domain ending with `.foo.com` (for example, `host01.foo.com`) folds to `foo.com`.
- `foo.com` — any domain ending with `foo.com` regardless of separator folds to `foo.com`.

Used by both `find_unknown_base_reverse_dns.py` and `collect_domain_info.py`, and auto-populated by `detect_psl_overrides.py` when N+ distinct full-IP-containing entries share a brand suffix. The leading `.` / `-` is stripped when computing the folded base.

## find_bad_utf8.py

Locates invalid UTF-8 bytes in files and optionally tries to current them. Generated by GPT5. Helped me find where I had introduced invalid bytes in `base_reverse_dns_map.csv`.

## find_unknown_base_reverse_dns.py

Reads the domains in `base_reverse_dns.csv` and writes the domains that are not in `base_reverse_dns_map.csv` or `known_unknown_base_reverse_dns.txt` to `unknown_base_reverse_dns.csv`, useful for identifying potential additional domains to contribute to `base_reverse_dns_map.csv` and `known_unknown_base_reverse_dns.txt`. Applies `psl_overrides.txt` to fold noisy subdomain patterns to their bases, and drops any entry containing a full IPv4 address (four dotted or dashed octets) so customer IPs never enter the pipeline.

## detect_psl_overrides.py

Scans `unknown_base_reverse_dns.csv` for full-IP-containing entries that share a common brand suffix. Any suffix repeated by N+ distinct domains (default 3, configurable via `--threshold`) is appended to `psl_overrides.txt`, and every affected entry across the unknown / known-unknown / map files is folded to that suffix's base. Any remaining full-IP entries — whether they clustered or not — are then removed for privacy. After running, the newly exposed base domains still need to be researched and classified via `collect_domain_info.py` and a classifier pass. Supports `--dry-run` to preview without writing.

## collect_domain_info.py

Bulk enrichment collector. For every domain in `unknown_base_reverse_dns.csv` that is not already in `base_reverse_dns_map.csv`, runs `whois` on the domain, fetches a size-capped `https://` GET, resolves A/AAAA records, and runs `whois` on the first resolved IP. Writes a TSV (`domain_info.tsv` by default) with the registrant org/country/registrar, page `<title>`/`<meta description>`, resolved IPs, and IP-WHOIS org/netname/country — the compact metadata a classifier needs to decide each domain in one pass. Respects `psl_overrides.txt`, skips full-IP entries, and is resume-safe (re-running only fetches domains missing from the output file).

## domain_info.tsv

The output of `collect_domain_info.py`. Tab-separated, one row per researched domain. Not tracked by Git — it is regenerated on demand and contains transient third-party WHOIS/HTML data.

## sortlists.py

Validation and sorting helper invoked as a module. Alphabetically sorts `base_reverse_dns_map.csv` (case-insensitive by first column, preserving CRLF line endings), deduplicates entries, validates that every `type` appears in `base_reverse_dns_types.txt`, and warns on names that contain unescaped commas or stray whitespace. Run it after any batch merge before committing.
