From 7ba078bff166d62c90d7dafa9e0bf2466b660203 Mon Sep 17 00:00:00 2001 From: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Date: Tue, 5 May 2026 13:17:43 -0400 Subject: [PATCH] Translate AS-name source rows via MMDB; classify reverse DNS batch (#745) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(maps): translate AS-name source rows via MMDB When parsedmarc's ASN-fallback path in utils.get_ip_address_info surfaces a raw MMDB as_name (e.g. "Vodafone Group PLC") for an IP that has no PTR and whose as_domain isn't in the map, find_unknown_base_reverse_dns.py now looks the as_name up in the bundled ipinfo_lite.mmdb and substitutes the matching as_domain so the row enters the unknown pipeline as a researchable domain instead of being dropped or polluting the list. Normalize non-breaking spaces (U+00A0) and runs of whitespace when building and querying the as_name index — the source CSV and MMDB disagree on NBSP placement for several names (e.g. "UDomain\xa0Web Hosting Company Ltd" in the CSV vs. "UDomain Web Hosting Company Ltd" in the MMDB), causing exact-match lookups to miss otherwise-identical entries. Co-Authored-By: Claude Opus 4.7 (1M context) * chore(maps): classify a batch of unknown reverse DNS base domains 40 map additions (35 source domains + 5 redirect-target/promotion aliases) and 35 known-unknown additions, covering the 71-entry unknown_base_reverse_dns.csv refresh. Newly mapped operators include several MMDB-AS-translated regional ISPs (Babilon-T/TJ, MegaFon Tajikistan, Ucell, Ufone, PinPro, Teraline Telecom, Transtelecom Kazakhstan, Satis, AlmaTV, Radius-NET, Burlington Telecom), aliases of existing brands (Telstra/bigpond.net.au, UDomain/udomain.hk, AG Telekom/katv1.net, EWE/ewe-ip-backbone.de, Hostinger/hstgr.cloud, Docusign/docusign.net, Brevo/sp2-brevo.net, MegaFon/megafon.tj, Beeline/beeline.uz), Tier-0 brands (Visa, Tripster, Verde Agritech), one healthcare entry (Sanwakai Hospital), one government entry (Special Communication Service of Azerbaijan), one education entry (KazRENA), and an MSP (Otava). Redirect-target aliases added for burlingtontelecom.com, alma.plus, cn.at, and teraline-telecom.net per the post-batch sweep rule. fea.net promoted out of known-unknown to West Coast Internet (WCI) after its homepage redirect-target was already mapped. Domains with single-source corroboration (privacy WHOIS plus unreachable site, parked-domain pages, ambiguous categorizations) went to known_unknown_base_reverse_dns.txt rather than the map. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Sean Whalen Co-authored-by: Claude Opus 4.7 (1M context) --- AGENTS.md | 2 +- parsedmarc/resources/maps/README.md | 2 + .../resources/maps/base_reverse_dns_map.csv | 40 +++++++++ .../maps/find_unknown_base_reverse_dns.py | 83 +++++++++++++++++++ .../maps/known_unknown_base_reverse_dns.txt | 35 ++++++++ 5 files changed, 161 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 0497d48..aa21f1d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -222,7 +222,7 @@ When `unknown_base_reverse_dns.csv` has new entries, follow this order rather th ### Related utility scripts (all in `parsedmarc/resources/maps/`) -- `find_unknown_base_reverse_dns.py` — regenerates `unknown_base_reverse_dns.csv` from `base_reverse_dns.csv` by subtracting what is already mapped or known-unknown. Enforces the no-full-IP privacy rule at ingest. Run after merging a batch. +- `find_unknown_base_reverse_dns.py` — regenerates `unknown_base_reverse_dns.csv` from `base_reverse_dns.csv` by subtracting what is already mapped or known-unknown. Enforces the no-full-IP privacy rule at ingest. Translates non-domain-shaped `source_name` rows (raw MMDB `as_name` strings surfaced by the ASN-fallback path in `utils.py:get_ip_address_info` when the IP had no PTR and the `as_domain` was uncategorized) to their corresponding `as_domain` via the bundled MMDB, so the row enters the pipeline as a researchable domain (and drops out automatically if that `as_domain` is already mapped). Run after merging a batch. - `detect_psl_overrides.py` — scans the lists for clustered IP-containing patterns, auto-adds brand suffixes to `psl_overrides.txt`, folds affected entries to their base, and removes any remaining full-IP entries. Run before the collector on any new batch. - `collect_domain_info.py` — the bulk enrichment collector described above. Respects `psl_overrides.txt` and skips full-IP entries. - `find_bad_utf8.py` — locates invalid UTF-8 bytes (used after past encoding corruption). diff --git a/parsedmarc/resources/maps/README.md b/parsedmarc/resources/maps/README.md index 39c1c83..03a2621 100644 --- a/parsedmarc/resources/maps/README.md +++ b/parsedmarc/resources/maps/README.md @@ -119,6 +119,8 @@ Locates invalid UTF-8 bytes in files and optionally tries to current them. Gener Reads the domains in `base_reverse_dns.csv` and writes the domains that are not in `base_reverse_dns_map.csv` or `known_unknown_base_reverse_dns.txt` to `unknown_base_reverse_dns.csv`, useful for identifying potential additional domains to contribute to `base_reverse_dns_map.csv` and `known_unknown_base_reverse_dns.txt`. Applies `psl_overrides.txt` to fold noisy subdomain patterns to their bases, and drops any entry containing a full IPv4 address (four dotted or dashed octets) so customer IPs never enter the pipeline. +When a `source_name` is not domain-shaped (e.g. `Vodafone Group PLC`), parsedmarc's ASN-fallback path emitted the raw MMDB `as_name` because the IP had no PTR and the corresponding `as_domain` was not in the map. The script translates such rows by looking the `as_name` up in the bundled `ipinfo_lite.mmdb` and substituting the matching `as_domain` (the one with the largest aggregate IPv4 footprint when an `as_name` covers multiple). Translated rows then flow through the normal known/known-unknown filter, so already-mapped operators drop out automatically and only genuinely new `as_domain` candidates land in the unknown CSV. AS names with no MMDB match are skipped with a warning. + ## detect_psl_overrides.py Scans `unknown_base_reverse_dns.csv` for full-IP-containing entries that share a common brand suffix. Any suffix repeated by N+ distinct domains (default 3, configurable via `--threshold`) is appended to `psl_overrides.txt`, and every affected entry across the unknown / known-unknown / map files is folded to that suffix's base. Any remaining full-IP entries — whether they clustered or not — are then removed for privacy. After running, the newly exposed base domains still need to be researched and classified via `collect_domain_info.py` and a classifier pass. Supports `--dry-run` to preview without writing. diff --git a/parsedmarc/resources/maps/base_reverse_dns_map.csv b/parsedmarc/resources/maps/base_reverse_dns_map.csv index ef3c9ac..f915332 100644 --- a/parsedmarc/resources/maps/base_reverse_dns_map.csv +++ b/parsedmarc/resources/maps/base_reverse_dns_map.csv @@ -267,8 +267,10 @@ allrede.tec.br,Allrede,ISP allstate.com,Allstate,Finance alltecinternet.com.br,Alltec Internet,ISP alltradebusiness.it,AllTrade Business,Marketing +alma.plus,AlmaTV,ISP almanet.net,Alma Communications,ISP almatel.ru,Almatel,ISP +almatv.kz,AlmaTV,ISP almeidaparente.eti.br,Almeida Parente,ISP almobile.com,AL Mobile,SaaS alorica.com,Alorica,Staffing @@ -288,6 +290,7 @@ altice.com.do,Altice Dominicana,ISP alticor.com,Alticor,Retail altimatel.com,Altima Telecom,ISP altmanadvisors.com,Altman Advisors,Finance +altnet.md,Altnet,ISP altovalenet.com.br,Alto Vale Net,ISP alwaysdata.com,alwaysdata,Web Host alwaysdata.net,alwaysdata,Web Host @@ -509,6 +512,7 @@ azure.com,Microsoft Azure,PaaS azureedge.net,Microsoft Azure,Technology azurestaticapps.net,Microsoft Azure,PaaS azurewebsites.net,Microsoft Azure,PaaS +babilon-t.com,Babilon-T,ISP backland.net,Backland Communications,MSP baesystems.com,BAE Systems,Defense bahnhof.se,Bahnhof,ISP @@ -564,6 +568,7 @@ bedge.com,BEDGE,Web Host beecreek.net,Bee Creek Communications,ISP beeline.kz,Beeline,ISP beeline.ru,Beeline,ISP +beeline.uz,Beeline,ISP beget.app,Beget,Web Host beget.com,Beget,Web Host beget.ru,Beget,Web Host @@ -601,6 +606,7 @@ bigleaf.net,Bigleaf Networks,ISP biglobe.co.jp,BIGLOBE,ISP biglobe.ne.jp,BIGLOBE,ISP bigpond.com,Telstra,ISP +bigpond.net.au,Telstra,ISP bigscoots.com,BigScoots,Web Host bilink.com.br,Bilink Telecomunicações,ISP binero.com,Binero,Web Host @@ -730,6 +736,7 @@ btopenworld.com,BT,ISP btussel.com,Bug Tussel,ISP bu.edu,Boston University,Education buckeyebroadband.com,Buckeye Broadband,ISP +buckeyebusiness.net,Buckeye Business Solutions,MSP buffalo.edu,University at Buffalo,Education bugtusselwireless.com,Bug Tussel,ISP buildingmaterials.com.my,Building Materials,Construction @@ -739,6 +746,8 @@ bumrungrad.com,Bumrungrad International Hospital,Healthcare bunkaren.or.jp,Japan Culture and Welfare Federation of Agricultural Cooperatives,Agriculture bunnycommunications.com,Bunny Communications,ISP bunnytechnology.net,Bunny Technology,ISP +burlingtontelecom.com,Burlington Telecom,ISP +burlingtontelecom.net,Burlington Telecom,ISP burnhamholdings.com,Burnham Holdings,Industrial buroserv.net.au,Brunoserv,ISP busse-computer.de,Busse Computertechnik,MSP @@ -754,6 +763,7 @@ c3po3090.com.br,Núcleo Brasil Servidores,Web Host ca.gov,State of California,Government cable.net.co,Cablenet,ISP cablebahamas.com,Cable Bahamas,ISP +cablecolor.com.gt,Cable Color,ISP cablecom.ch,UPC,ISP cablecomm.ie,CableComm,ISP cablelink.at,Salzburg AG,Utilities @@ -764,6 +774,7 @@ cableonda.net,Cable Onda,ISP cableone.biz,Sparklight,ISP cables.com.tw,Yueh Feng Industrial,Manufacturing cabletel.com.mk,A1,ISP +cablevision.at,Cablevision Nöhmer,ISP cablevision.com,Cablevision,ISP cablevision.net.mx,Izzi Telecom,ISP cableworld.es,Cableworld,ISP @@ -924,6 +935,7 @@ citictel-cpc.com,CITIC Telecom CPC,ISP citizensmemorial.com,Citizens Memorial Hospital,Healthcare cityemail.com,CityEmail,Email Provider cityfibre.com,CityFibre,ISP +citynet.kg,Citynet KG,ISP citynet.net,Citynet,ISP cityoftacoma.org,City of Tacoma,Government cityonlinebd.net,City Online,ISP @@ -999,6 +1011,7 @@ cmo.de,CMO,Web Host cmorlando.com.ar,Carpintería Metálica Orlando,Manufacturing cmtietong.com,China TieTong,ISP cmu.edu,Carnegie Mellon University,Education +cn.at,Cablevision Nöhmer,ISP cn4e.com,35.com,Web Host cnci.co.jp,Community Network Center,ISP cncm.ne.jp,Nagasaki Cable Media,ISP @@ -1434,6 +1447,7 @@ dobson.net,Dobson Fiber,ISP docomopacific.com,DOCOMO Pacific,ISP doctors.org.uk,Doctors.net.uk,Healthcare docusign.com,Docusign,SaaS +docusign.net,Docusign,SaaS dodea.edu,U.S. Department of Defense Education Activity,Education dogado.de,dogado,Web Host dokom21.de,DOKOM21,ISP @@ -1771,6 +1785,7 @@ evolus-it.com,Evolus IX,ISP evolus-ix.com,Evolus IX,ISP evolusfibre.com,Evolus IX,ISP evonik.de,Evonik,Manufacturing +ewe-ip-backbone.de,EWE,ISP ewe.com,EWE,ISP ewp.live,EasyWP,Web Host eww.at,eww,Utilities @@ -1844,6 +1859,7 @@ fccn.pt,FCCN,Education fcnet.fr,fcnet,ISP fdcservers.net,FDC Servers,Web Host fdu.edu,Fairleigh Dickinson University,Education +fea.net,West Coast Internet (WCI),ISP feasa.ie,Feasa,Industrial federalreserve.gov,Federal Reserve,Government fedex.com,FedEx,Logistics @@ -2405,6 +2421,7 @@ hslda.net,Home School Legal Defense Association (HSLDA),Education hslda.org,Home School Legal Defense Association (HSLDA),Education hslink.com.br,HSLINK Telecom,ISP hspherefilter.com,"DynamicNet, Inc. (DNI)",Web Host +hstgr.cloud,Hostinger,Web Host ht.hr,Hrvatski Telekom,ISP htc.net,HTC,ISP htcinc.net,HTC,ISP @@ -2852,8 +2869,10 @@ katelecom.com.br,KA Telecom,ISP kater.com.br,Kater Telecom,ISP kattare.com,Kattare,Web Host katv1.az,AG Telekom,ISP +katv1.net,AG Telekom,ISP kaunogrudai.lt,Kauno Grūdai,Food kazintercom.kz,Kaz InterCom,ISP +kazrena.kz,KazRENA,Education kbro.com.tw,kbro,ISP kbtelecom.net,Koos Broadband Telecom,ISP kcc.com,Kimberly-Clark,Manufacturing @@ -3328,6 +3347,7 @@ megaconnect.com.br,MegaConnect,ISP megacp.com,iWorx Host,Web Host megadata.net.id,Megadata ISP,ISP megafon.ru,MegaFon,ISP +megafon.tj,MegaFon,ISP megaline.kg,Maga-Line,ISP megalink.net.ru,Мегалинк,ISP megalinkinternet.com.br,MegaLink,ISP @@ -4133,6 +4153,7 @@ osogrande.net,Oso Grande Technologies,ISP osorio.rs.gov.br,Prefeitura Municipal de Osório,Government osu.edu,The Ohio State University,Education otago.ac.nz,University of Otago,Education +otava.com,Otava,MSP ote.gr,OTE,ISP otelco.net,GoNetSpeed,ISP otenet.gr,Cosmote,ISP @@ -4261,6 +4282,7 @@ pilot-ipek.ru,Izhevsk Industrial-Economic College,Education pilotfiber.com,Pilot Fiber,ISP pink.co.rs,Mink Media Group,Marketing pinnaclecart.com,PinnacleCart,SaaS +pinpro.by,PinPro,ISP pioneer.co.th,Pioneer Transportation,Logistics pip.com.au,PIP Total IT Solutions,Web Host pipefy.com,Pipefy Inc,SaaS @@ -4372,6 +4394,7 @@ promax.media.pl,Promax,ISP prontomarketing.com,Pronto Marketing,Marketing proofhubmail.com,ProofHub,SaaS prosites.com,ProSites,Web Host +prospect.pl,Prospect,ISP proton.ch,Proton,Email Provider proton.me,Proton,Email Provider protonmail.ch,Proton,Email Provider @@ -4477,6 +4500,7 @@ radarinternet.com.br,Radar Internet,ISP radford.edu,Radford University,Education radiant.net,TELUS,ISP radiantsolutions.net,Radient Solutions,Web Host +radius-net.com,Radius-NET,ISP radware.com,Radware,Email Security rafftechnologies.com,Raff Technologies,Web Host rain.co.za,Rain,ISP @@ -4700,6 +4724,7 @@ sangchaimeter.com,Sangchai Meter,Industrial sanofi.fr,Sanofi,Healthcare santehealth.net,Sante Health System,Healthcare sanwa.co.jp,Sanwa Supply,Retail +sanwakai.jp,Sanwakai Hospital,Healthcare sap.com,SAP,Technology saplbd.com,Summit Alliance Port,Logistics sapmed.ac.jp,Sapporo Medical University,Healthcare @@ -4708,6 +4733,7 @@ saremail.com,SAREnet,ISP sarenet.es,Sarenet,ISP sarkor.uz,Sarkor,ISP sasktel.com,SaskTel,ISP +satis-tl.ru,Satis,ISP satnet.net,Xtrim,ISP satorilabs.com,NextGen Healthcare,Healthcare sattrakt.com,Sat-Trakt,ISP @@ -4734,6 +4760,7 @@ sch.gr,Greek School Network,Education schibsted.no,Schibsted Media,News schmc.ac.kr,Soon Chun Hyang University Medical Center,Healthcare scinternet.net,South Central Communications,ISP +scis.gov.az,Special Communication Service of Azerbaijan,Government scnet.com.br,SCNet,ISP scorm.com,SCORM,SaaS scrio.com.br,S.C. RIO Telecom,ISP @@ -5062,6 +5089,7 @@ southlandind.com,Southland Industries,Construction souuni.com,Uni Telecom,ISP soverin.com,Soverin,Email Provider soverin.net,Soverin,Email Provider +sp2-brevo.net,Brevo (Sendinblue),Marketing space.net,SpaceNet,MSP spaceship.com,Spaceship,Web Host spaceship.net,Spaceship,Web Host @@ -5359,6 +5387,7 @@ tedforbes.com,Ted Forbes,Photography tees.ne.jp,Toyohashi Cable Network,ISP teksavvy.com,TekSavvy,ISP tel.net.ba,HT Eronet,ISP +telbo.net,TELBO,ISP telcocom.com.ar,Telcocom,ISP telcomaster.com,TelcoMaster,ISP telconet.ec,Telconet,ISP @@ -5461,6 +5490,8 @@ tennessee.edu,University of Tennessee,Education teol.net,mtel,ISP tera-byte.com,Tera-Byte,Web Host terago.ca,TeraGo,ISP +teraline-telecom.net,Teraline Telecom,ISP +teraline.kz,Teraline Telecom,ISP teralinktelecom.com.br,Teralink,ISP terra.com,Terra Networks,Web Host terra.com.br,Terra Mail,Email Provider @@ -5560,6 +5591,7 @@ tofinosoftware.com,Tofino,SaaS tohoku-mpu.ac.jp,Tohoku Medical and Pharmaceutical University,Education tohoku.ac.jp,Tohoku University,Education tohoyk.co.jp,Toho Pharmaceutical,Healthcare +tojikiston.com,Babilon-T,ISP tokai-com.co.jp,TOKAI Communications,ISP tokala-mobile.com,Tokala Communications,ISP top-tokyo.co.jp,TOP CORPORATION,Healthcare @@ -5616,6 +5648,7 @@ trifle.net,Trifle Network,ISP trinity.cz,AB-NET,ISP triolan.com,Triolan,ISP triolan.net,Triolan,ISP +tripster.com,Tripster,Travel triton.net,Triton Technologies,Technology trivenet.it,Planetel,ISP trooli.com,Trooli,ISP @@ -5633,6 +5666,7 @@ tstc.edu,Texas State Technical College,Education tstt.co.tt,TSTT,ISP tstt.net.tt,TSTT,ISP tsukuba.ac.jp,University of Tsukuba,Education +ttc.kz,Transtelecom Kazakhstan,ISP ttcl.co.tz,Tanzania Telecommunications,ISP tti.com,TTI,Industrial ttiinc.com,TTI,Industrial @@ -5708,6 +5742,7 @@ ucar.edu,University Corporation for Atmospheric Research,Education uccs.edu,University of Colorado Colorado Springs,Education ucdavis.edu,UC Davis,Education ucdenver.edu,University of Colorado Denver,Education +ucell.uz,Ucell,ISP ucf.edu,University of Central Florida,Education uchealth.com,UC Health,Healthcare uchicago.edu,University of Chicago,Education @@ -5732,11 +5767,13 @@ udc.hk,Hong Kong Communications,Web Host udel.edu,University of Delaware,Education udg.mx,Universidad de Guadalajara,Education udlap.mx,Universidad de las Americas Puebla,Education +udomain.hk,UDomain,Web Host uen.org,Utah Education Network,Education ufanet.ru,Ufanet,ISP ufinet.com,Ufinet,ISP ufl.edu,University of Florida,Education ufmg.br,UFMG,Education +ufone.com,Ufone,ISP ufrgs.br,UFRGS,Education ufsc.br,UFSC,Education uga.edu,University of Georgia,Education @@ -5973,6 +6010,7 @@ vercel.app,Vercel,PaaS vercel.com,Vercel,PaaS vercel.dev,Vercel,PaaS vercel.run,Vercel,PaaS +verde.ag,Verde Agritech,Agriculture verginia.edu,University of Virginia,Education veridyen.com,Veridyen,Web Host verisign.com,Verisign,Technology @@ -6042,6 +6080,7 @@ virtualhosting.hk,UDomain,Web Host virtualnetfibra.com.br,Virtual Net Fibra,ISP virtualserver.io,Zitcom,Web Host virtuaserver.com.br,VirtuaServer,Web Host +visa.com,Visa,Finance visabeira.co.ao,COMATEL,ISP visi.com,US Signal,MSP visitmiddlesex.ca,"Middlesex County, Canada",Government @@ -6264,6 +6303,7 @@ windriver.com,Wind River,SaaS windstream.net,Windstream,ISP windtre.it,WINDTRE,ISP wingnet.com.br,WINGNET,ISP +wingo.ch,Wingo,ISP winknet.ne.jp,Himeji Cable Television,ISP winnermedical.com,Winner Medical,Healthcare wiredblade.com,Wired Blade,Web Host diff --git a/parsedmarc/resources/maps/find_unknown_base_reverse_dns.py b/parsedmarc/resources/maps/find_unknown_base_reverse_dns.py index 0005bb0..9bd37da 100755 --- a/parsedmarc/resources/maps/find_unknown_base_reverse_dns.py +++ b/parsedmarc/resources/maps/find_unknown_base_reverse_dns.py @@ -3,6 +3,8 @@ import os import csv import re +import sys +from collections import defaultdict # Privacy filter: a reverse DNS entry containing a full IPv4 address (four @@ -13,6 +15,13 @@ _FULL_IP_RE = re.compile( r"(? bool: for m in _FULL_IP_RE.finditer(s): @@ -22,11 +31,61 @@ def _has_full_ip(s: str) -> bool: return False +def _looks_like_domain(s: str) -> bool: + return bool(_DOMAIN_RE.match(s)) + + +def _normalize_as_name(s: str) -> str: + # NBSP (U+00A0) appears in both MMDB as_names and CSV source_names but + # not always on the same side, so an exact match misses pairs that are + # otherwise identical. Fold NBSP to a regular space and collapse runs + # of whitespace before comparing. + return re.sub(r"\s+", " ", s.replace("\xa0", " ")).lower().strip() + + +def _load_as_name_index(mmdb_path: str) -> dict[str, str]: + """Build a normalized as_name -> as_domain index from the bundled MMDB. + + When a single as_name maps to multiple as_domains (about 1% of records), + the as_domain with the largest aggregate IPv4 footprint wins. + """ + try: + import maxminddb + except ImportError: + print( + "Error: maxminddb is required to translate AS-name source rows; " + "install parsedmarc's runtime dependencies (pip install maxminddb)", + file=sys.stderr, + ) + sys.exit(1) + + counts: dict[tuple[str, str], int] = defaultdict(int) + with maxminddb.open_database(mmdb_path) as reader: + for net, rec in reader: + if net.version != 4 or not isinstance(rec, dict): + continue + as_name = rec.get("as_name") + as_domain = rec.get("as_domain") + if not as_name or not as_domain: + continue + counts[(_normalize_as_name(as_name), as_domain.lower().strip())] += ( + net.num_addresses + ) + + best: dict[str, tuple[str, int]] = {} + for (as_name_lower, as_domain_lower), count in counts.items(): + existing = best.get(as_name_lower) + if existing is None or count > existing[1]: + best[as_name_lower] = (as_domain_lower, count) + return {k: v[0] for k, v in best.items()} + + def _main(): input_csv_file_path = "base_reverse_dns.csv" base_reverse_dns_map_file_path = "base_reverse_dns_map.csv" known_unknown_list_file_path = "known_unknown_base_reverse_dns.txt" psl_overrides_file_path = "psl_overrides.txt" + mmdb_file_path = "../ipinfo/ipinfo_lite.mmdb" output_csv_file_path = "unknown_base_reverse_dns.csv" csv_headers = ["source_name", "message_count"] @@ -51,6 +110,12 @@ def _main(): load_list(known_unknown_list_file_path, known_unknown_domains) load_list(psl_overrides_file_path, psl_overrides) + if not os.path.exists(mmdb_file_path): + print(f"Error: {mmdb_file_path} does not exist") + exit(1) + print(f"Loading {mmdb_file_path}") + as_name_index = _load_as_name_index(mmdb_file_path) + print(f"Indexed {len(as_name_index)} as_names from the MMDB") if not os.path.exists(base_reverse_dns_map_file_path): print(f"Error: {base_reverse_dns_map_file_path} does not exist") print(f"Loading {base_reverse_dns_map_file_path}") @@ -78,6 +143,24 @@ def _main(): domain = row["source_name"].lower().strip() if domain == "": continue + # If source_name is not domain-shaped, parsedmarc's ASN-fallback + # path (utils.py:get_ip_address_info) surfaced the raw MMDB + # ``as_name`` because the IP had no PTR and the as_domain wasn't + # in the map. Translate to the corresponding as_domain so the + # row enters the pipeline as a researchable domain. If the + # as_domain is already in the map, the row drops out below as a + # known domain — exactly what we want. + if not _looks_like_domain(domain): + translated = as_name_index.get(_normalize_as_name(domain)) + if translated is None: + print( + f"Skipping AS-name source with no MMDB match: " + f"{row['source_name']!r}" + ) + continue + print(f"Translating AS-name {row['source_name']!r} -> {translated}") + row["source_name"] = translated + domain = translated for psl_domain in psl_overrides: if domain.endswith(psl_domain): domain = psl_domain.strip(".").strip("-") diff --git a/parsedmarc/resources/maps/known_unknown_base_reverse_dns.txt b/parsedmarc/resources/maps/known_unknown_base_reverse_dns.txt index 5e32800..760f3fd 100644 --- a/parsedmarc/resources/maps/known_unknown_base_reverse_dns.txt +++ b/parsedmarc/resources/maps/known_unknown_base_reverse_dns.txt @@ -85,6 +85,7 @@ admin.corpivensa.gob.ve admincomp.com adriadns.com adsl.cat +adslplus.ch adsnet-telecom.net.br advance.com.ar advancedridingtechniques.info @@ -117,6 +118,7 @@ airsystem33.com airtel.net airyhosting.com aistechsolutions.com +aitacargas.uy ajorealestate.com ak888.ru akado-ural.ru @@ -335,6 +337,7 @@ bitter-echo.com bitternet.ua biz-ps.com bizhostingservices.com +biznetg.io bizylady.com blackhawk-net.com blackpointcyber.com @@ -342,6 +345,7 @@ blackwaterregion.com blguss.com blmpb.com blockchainlaw.info +blogwisefinance.com blondewigs.shop bloodfire9.com bloomingdales.com @@ -419,6 +423,7 @@ byethost31.org byronservices.com bzconcept.com c53dw7m24rj.com +c5ace.com ca-expressive.site cablecolor.com.sv caccaweb.com @@ -477,6 +482,7 @@ cdcservices.com cdn1000.com cdt.cz cegalvez.com.ar +ceiausa.com centralmalaysia.com centralnic.net centrum.dk @@ -485,6 +491,7 @@ cf-emailsecurity.net cfnationt.de.com cgate-global.com ch-abbeville.fr +ch-charleville-mezieres.fr ch-dns.net ch-tournon.fr chacoonline.net @@ -842,6 +849,7 @@ e-shops.jp e-tiger.net e3webmail.com eagleredes.net.br +eaglezip.com eappscdn.com earthcare.com easternkingspei.com @@ -854,6 +862,7 @@ ecdaoductsservi-com.icu ecofex-mx.icu econet.co.ls economiceagles.com +ecova.com ecuahosting.net edcor.com edit.ne.jp @@ -906,6 +915,7 @@ enticingofdisadvantage.com entretothom.net enzu.com epaycontrol.com +epchd.org epectelco.com.ar epicinvestmentsreview.co epicinvestmentsreview.com @@ -1051,6 +1061,7 @@ foxnew.de francecanterbury.com frauen-online.net fredi.shop +fredperryst.com free.ds freehypnosisdownloads.com freenet.de @@ -1123,6 +1134,7 @@ ginza-entre.com gis.net gist-th.com glamsio.com +glattnet.ch glenpondia.xyz glenporch.cam glenverra.shop @@ -1218,6 +1230,7 @@ harmonianhub.com haswu.com hawaiiantel.net hawkaccess.com +hderit.com hdomestic.com hdprosupply.com healthcareathome.ca @@ -1278,6 +1291,7 @@ hostedexchange2010.fr hostednetworks.net hostedvps.co.uk hostelsucre.com +hosterby.com hostflyby.net hostidadns.com hosting1337.com @@ -1357,6 +1371,7 @@ igmohji.com igppevents.org.uk igroupnet.com ihglobaldns.com +ijalti.org.mx ijetabtens-mx.icu ikexpress.com ileysinc.com @@ -1370,6 +1385,7 @@ imagica-imageworks.co.jp imazu-chemical.co.jp imjtmn.cn immenzaces.com +imobie.com imp.ch impey.shop improvation.us @@ -1490,6 +1506,7 @@ jaeres.com janikingks.com japbuk.com jarmuloviktazone.com +jasarekber.id jatimnet.com.br jaxa.jp jazhostingco.com @@ -1550,6 +1567,7 @@ kenkids.com kennport.com kenobianresearch.com kernel.org +kesha145216.vds kevalier.com keznews.com khwebsolutions.co.uk @@ -1948,6 +1966,7 @@ net-rubi.com.br netbitsfibra.com.br netboard.hu netbri.com +netbuild.net netcarrier.net netcell.inf.br netcentertelecom.net.br @@ -1980,6 +1999,7 @@ newlifeinformatica.com.br newnetprovedor.net.br newsv.jp newwallstreetcode.com +newwave.com newyorkhonorlodge.com newyorkmarketanalytics.com nex-tech.com @@ -2026,6 +2046,7 @@ nossanettelecom.net.br novanetnp.net.br novast.com novuscom.net +npbnl.net nqntv.com.ar ns360.net nsk-kk.com @@ -2067,6 +2088,7 @@ ogicom.net ogreserve.com okweb3.jp olb-access.net +olbitx.network olcx.net olivettilexikon.co.uk ollatelecom.net.br @@ -2222,6 +2244,7 @@ pr-m.eu pravnagroup-m.pl preblepride.shop precisionpros.com +prevail-portal.com priawesta.store prima.com.ar prima.net.ar @@ -2288,7 +2311,9 @@ raxa.host rcncustomer.com rdock.ru reachesbeast.com.de +readyidc.com realestateofdetroit.com +realssl.com reberte.com receita.pb.gov.br recruit-ms.co.jp @@ -2314,6 +2339,7 @@ reliablepanel.com reloadhard.com renchildbirth.com renkus-hainz.com +renner.uz renopie.com renttitllc.com researchmarketreports.com @@ -2322,6 +2348,7 @@ resellerex.com reso-tech.ca restocarpediem.com resurrectioncatholic.church +retemetis.net rethos.com.mx retirementconcepts.com retroflect.shop @@ -2455,6 +2482,7 @@ serversignin.com servicenetpro.com.br servidor2.net sevenmeadowsfarm.net +sevtele.com seymenkolej.k12.tr sfek.kz sfr-sh.net @@ -2534,6 +2562,7 @@ solucija.eu solusoftware.com solutionspal.com soon.com.ar +sopricom.fr soultelecom.net.br sounditaly.com sourcedns.com @@ -2616,6 +2645,7 @@ suksangroup.com sulnet.net.br super-dns.net supercanal.com.ar +superdata.vn superdnssite.com superdomainzone.com superior-host.com @@ -2690,6 +2720,7 @@ tenkids.net tenriyorozu.jp tenten.vn teraswitch.com +terawiz.com terminavalley.com tetrocapital.com texasgs.net @@ -2733,6 +2764,7 @@ tilaa.cloud timbrasil.com.br tipira.net.br titamnet.net.br +titandc.io tld-mx.com tm.net.my tmg.md @@ -2744,6 +2776,7 @@ tohknet.co.jp tokushukai.or.jp tokyorinkai.jp toledofibra.net.br +tomcity.net tomtelnet.ru tondyfree.vip tongx.com.tw @@ -2814,8 +2847,10 @@ twohoax.info twp-capital.com txw-taishin.com tylerfordonline.com +tymtalk.com tzulo.com u-ikoi.co.jp +u-team.by uaehostings.com ubicarme.com ubjp.cn