mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-06 00:35:09 +00:00
Translate AS-name source rows via MMDB; classify reverse DNS batch (#745)
* feat(maps): translate AS-name source rows via MMDB When parsedmarc's ASN-fallback path in utils.get_ip_address_info surfaces a raw MMDB as_name (e.g. "Vodafone Group PLC") for an IP that has no PTR and whose as_domain isn't in the map, find_unknown_base_reverse_dns.py now looks the as_name up in the bundled ipinfo_lite.mmdb and substitutes the matching as_domain so the row enters the unknown pipeline as a researchable domain instead of being dropped or polluting the list. Normalize non-breaking spaces (U+00A0) and runs of whitespace when building and querying the as_name index — the source CSV and MMDB disagree on NBSP placement for several names (e.g. "UDomain\xa0Web Hosting Company Ltd" in the CSV vs. "UDomain Web Hosting Company Ltd" in the MMDB), causing exact-match lookups to miss otherwise-identical entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(maps): classify a batch of unknown reverse DNS base domains 40 map additions (35 source domains + 5 redirect-target/promotion aliases) and 35 known-unknown additions, covering the 71-entry unknown_base_reverse_dns.csv refresh. Newly mapped operators include several MMDB-AS-translated regional ISPs (Babilon-T/TJ, MegaFon Tajikistan, Ucell, Ufone, PinPro, Teraline Telecom, Transtelecom Kazakhstan, Satis, AlmaTV, Radius-NET, Burlington Telecom), aliases of existing brands (Telstra/bigpond.net.au, UDomain/udomain.hk, AG Telekom/katv1.net, EWE/ewe-ip-backbone.de, Hostinger/hstgr.cloud, Docusign/docusign.net, Brevo/sp2-brevo.net, MegaFon/megafon.tj, Beeline/beeline.uz), Tier-0 brands (Visa, Tripster, Verde Agritech), one healthcare entry (Sanwakai Hospital), one government entry (Special Communication Service of Azerbaijan), one education entry (KazRENA), and an MSP (Otava). Redirect-target aliases added for burlingtontelecom.com, alma.plus, cn.at, and teraline-telecom.net per the post-batch sweep rule. fea.net promoted out of known-unknown to West Coast Internet (WCI) after its homepage redirect-target was already mapped. Domains with single-source corroboration (privacy WHOIS plus unreachable site, parked-domain pages, ambiguous categorizations) went to known_unknown_base_reverse_dns.txt rather than the map. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
GitHub
@@ -222,7 +222,7 @@ When `unknown_base_reverse_dns.csv` has new entries, follow this order rather th
|
||||
|
||||
### Related utility scripts (all in `parsedmarc/resources/maps/`)
|
||||
|
||||
- `find_unknown_base_reverse_dns.py` — regenerates `unknown_base_reverse_dns.csv` from `base_reverse_dns.csv` by subtracting what is already mapped or known-unknown. Enforces the no-full-IP privacy rule at ingest. Run after merging a batch.
|
||||
- `find_unknown_base_reverse_dns.py` — regenerates `unknown_base_reverse_dns.csv` from `base_reverse_dns.csv` by subtracting what is already mapped or known-unknown. Enforces the no-full-IP privacy rule at ingest. Translates non-domain-shaped `source_name` rows (raw MMDB `as_name` strings surfaced by the ASN-fallback path in `utils.py:get_ip_address_info` when the IP had no PTR and the `as_domain` was uncategorized) to their corresponding `as_domain` via the bundled MMDB, so the row enters the pipeline as a researchable domain (and drops out automatically if that `as_domain` is already mapped). Run after merging a batch.
|
||||
- `detect_psl_overrides.py` — scans the lists for clustered IP-containing patterns, auto-adds brand suffixes to `psl_overrides.txt`, folds affected entries to their base, and removes any remaining full-IP entries. Run before the collector on any new batch.
|
||||
- `collect_domain_info.py` — the bulk enrichment collector described above. Respects `psl_overrides.txt` and skips full-IP entries.
|
||||
- `find_bad_utf8.py` — locates invalid UTF-8 bytes (used after past encoding corruption).
|
||||
|
||||
@@ -119,6 +119,8 @@ Locates invalid UTF-8 bytes in files and optionally tries to current them. Gener
|
||||
|
||||
Reads the domains in `base_reverse_dns.csv` and writes the domains that are not in `base_reverse_dns_map.csv` or `known_unknown_base_reverse_dns.txt` to `unknown_base_reverse_dns.csv`, useful for identifying potential additional domains to contribute to `base_reverse_dns_map.csv` and `known_unknown_base_reverse_dns.txt`. Applies `psl_overrides.txt` to fold noisy subdomain patterns to their bases, and drops any entry containing a full IPv4 address (four dotted or dashed octets) so customer IPs never enter the pipeline.
|
||||
|
||||
When a `source_name` is not domain-shaped (e.g. `Vodafone Group PLC`), parsedmarc's ASN-fallback path emitted the raw MMDB `as_name` because the IP had no PTR and the corresponding `as_domain` was not in the map. The script translates such rows by looking the `as_name` up in the bundled `ipinfo_lite.mmdb` and substituting the matching `as_domain` (the one with the largest aggregate IPv4 footprint when an `as_name` covers multiple). Translated rows then flow through the normal known/known-unknown filter, so already-mapped operators drop out automatically and only genuinely new `as_domain` candidates land in the unknown CSV. AS names with no MMDB match are skipped with a warning.
|
||||
|
||||
## detect_psl_overrides.py
|
||||
|
||||
Scans `unknown_base_reverse_dns.csv` for full-IP-containing entries that share a common brand suffix. Any suffix repeated by N+ distinct domains (default 3, configurable via `--threshold`) is appended to `psl_overrides.txt`, and every affected entry across the unknown / known-unknown / map files is folded to that suffix's base. Any remaining full-IP entries — whether they clustered or not — are then removed for privacy. After running, the newly exposed base domains still need to be researched and classified via `collect_domain_info.py` and a classifier pass. Supports `--dry-run` to preview without writing.
|
||||
|
||||
@@ -267,8 +267,10 @@ allrede.tec.br,Allrede,ISP
|
||||
allstate.com,Allstate,Finance
|
||||
alltecinternet.com.br,Alltec Internet,ISP
|
||||
alltradebusiness.it,AllTrade Business,Marketing
|
||||
alma.plus,AlmaTV,ISP
|
||||
almanet.net,Alma Communications,ISP
|
||||
almatel.ru,Almatel,ISP
|
||||
almatv.kz,AlmaTV,ISP
|
||||
almeidaparente.eti.br,Almeida Parente,ISP
|
||||
almobile.com,AL Mobile,SaaS
|
||||
alorica.com,Alorica,Staffing
|
||||
@@ -288,6 +290,7 @@ altice.com.do,Altice Dominicana,ISP
|
||||
alticor.com,Alticor,Retail
|
||||
altimatel.com,Altima Telecom,ISP
|
||||
altmanadvisors.com,Altman Advisors,Finance
|
||||
altnet.md,Altnet,ISP
|
||||
altovalenet.com.br,Alto Vale Net,ISP
|
||||
alwaysdata.com,alwaysdata,Web Host
|
||||
alwaysdata.net,alwaysdata,Web Host
|
||||
@@ -509,6 +512,7 @@ azure.com,Microsoft Azure,PaaS
|
||||
azureedge.net,Microsoft Azure,Technology
|
||||
azurestaticapps.net,Microsoft Azure,PaaS
|
||||
azurewebsites.net,Microsoft Azure,PaaS
|
||||
babilon-t.com,Babilon-T,ISP
|
||||
backland.net,Backland Communications,MSP
|
||||
baesystems.com,BAE Systems,Defense
|
||||
bahnhof.se,Bahnhof,ISP
|
||||
@@ -564,6 +568,7 @@ bedge.com,BEDGE,Web Host
|
||||
beecreek.net,Bee Creek Communications,ISP
|
||||
beeline.kz,Beeline,ISP
|
||||
beeline.ru,Beeline,ISP
|
||||
beeline.uz,Beeline,ISP
|
||||
beget.app,Beget,Web Host
|
||||
beget.com,Beget,Web Host
|
||||
beget.ru,Beget,Web Host
|
||||
@@ -601,6 +606,7 @@ bigleaf.net,Bigleaf Networks,ISP
|
||||
biglobe.co.jp,BIGLOBE,ISP
|
||||
biglobe.ne.jp,BIGLOBE,ISP
|
||||
bigpond.com,Telstra,ISP
|
||||
bigpond.net.au,Telstra,ISP
|
||||
bigscoots.com,BigScoots,Web Host
|
||||
bilink.com.br,Bilink Telecomunicações,ISP
|
||||
binero.com,Binero,Web Host
|
||||
@@ -730,6 +736,7 @@ btopenworld.com,BT,ISP
|
||||
btussel.com,Bug Tussel,ISP
|
||||
bu.edu,Boston University,Education
|
||||
buckeyebroadband.com,Buckeye Broadband,ISP
|
||||
buckeyebusiness.net,Buckeye Business Solutions,MSP
|
||||
buffalo.edu,University at Buffalo,Education
|
||||
bugtusselwireless.com,Bug Tussel,ISP
|
||||
buildingmaterials.com.my,Building Materials,Construction
|
||||
@@ -739,6 +746,8 @@ bumrungrad.com,Bumrungrad International Hospital,Healthcare
|
||||
bunkaren.or.jp,Japan Culture and Welfare Federation of Agricultural Cooperatives,Agriculture
|
||||
bunnycommunications.com,Bunny Communications,ISP
|
||||
bunnytechnology.net,Bunny Technology,ISP
|
||||
burlingtontelecom.com,Burlington Telecom,ISP
|
||||
burlingtontelecom.net,Burlington Telecom,ISP
|
||||
burnhamholdings.com,Burnham Holdings,Industrial
|
||||
buroserv.net.au,Brunoserv,ISP
|
||||
busse-computer.de,Busse Computertechnik,MSP
|
||||
@@ -754,6 +763,7 @@ c3po3090.com.br,Núcleo Brasil Servidores,Web Host
|
||||
ca.gov,State of California,Government
|
||||
cable.net.co,Cablenet,ISP
|
||||
cablebahamas.com,Cable Bahamas,ISP
|
||||
cablecolor.com.gt,Cable Color,ISP
|
||||
cablecom.ch,UPC,ISP
|
||||
cablecomm.ie,CableComm,ISP
|
||||
cablelink.at,Salzburg AG,Utilities
|
||||
@@ -764,6 +774,7 @@ cableonda.net,Cable Onda,ISP
|
||||
cableone.biz,Sparklight,ISP
|
||||
cables.com.tw,Yueh Feng Industrial,Manufacturing
|
||||
cabletel.com.mk,A1,ISP
|
||||
cablevision.at,Cablevision Nöhmer,ISP
|
||||
cablevision.com,Cablevision,ISP
|
||||
cablevision.net.mx,Izzi Telecom,ISP
|
||||
cableworld.es,Cableworld,ISP
|
||||
@@ -924,6 +935,7 @@ citictel-cpc.com,CITIC Telecom CPC,ISP
|
||||
citizensmemorial.com,Citizens Memorial Hospital,Healthcare
|
||||
cityemail.com,CityEmail,Email Provider
|
||||
cityfibre.com,CityFibre,ISP
|
||||
citynet.kg,Citynet KG,ISP
|
||||
citynet.net,Citynet,ISP
|
||||
cityoftacoma.org,City of Tacoma,Government
|
||||
cityonlinebd.net,City Online,ISP
|
||||
@@ -999,6 +1011,7 @@ cmo.de,CMO,Web Host
|
||||
cmorlando.com.ar,Carpintería Metálica Orlando,Manufacturing
|
||||
cmtietong.com,China TieTong,ISP
|
||||
cmu.edu,Carnegie Mellon University,Education
|
||||
cn.at,Cablevision Nöhmer,ISP
|
||||
cn4e.com,35.com,Web Host
|
||||
cnci.co.jp,Community Network Center,ISP
|
||||
cncm.ne.jp,Nagasaki Cable Media,ISP
|
||||
@@ -1434,6 +1447,7 @@ dobson.net,Dobson Fiber,ISP
|
||||
docomopacific.com,DOCOMO Pacific,ISP
|
||||
doctors.org.uk,Doctors.net.uk,Healthcare
|
||||
docusign.com,Docusign,SaaS
|
||||
docusign.net,Docusign,SaaS
|
||||
dodea.edu,U.S. Department of Defense Education Activity,Education
|
||||
dogado.de,dogado,Web Host
|
||||
dokom21.de,DOKOM21,ISP
|
||||
@@ -1771,6 +1785,7 @@ evolus-it.com,Evolus IX,ISP
|
||||
evolus-ix.com,Evolus IX,ISP
|
||||
evolusfibre.com,Evolus IX,ISP
|
||||
evonik.de,Evonik,Manufacturing
|
||||
ewe-ip-backbone.de,EWE,ISP
|
||||
ewe.com,EWE,ISP
|
||||
ewp.live,EasyWP,Web Host
|
||||
eww.at,eww,Utilities
|
||||
@@ -1844,6 +1859,7 @@ fccn.pt,FCCN,Education
|
||||
fcnet.fr,fcnet,ISP
|
||||
fdcservers.net,FDC Servers,Web Host
|
||||
fdu.edu,Fairleigh Dickinson University,Education
|
||||
fea.net,West Coast Internet (WCI),ISP
|
||||
feasa.ie,Feasa,Industrial
|
||||
federalreserve.gov,Federal Reserve,Government
|
||||
fedex.com,FedEx,Logistics
|
||||
@@ -2405,6 +2421,7 @@ hslda.net,Home School Legal Defense Association (HSLDA),Education
|
||||
hslda.org,Home School Legal Defense Association (HSLDA),Education
|
||||
hslink.com.br,HSLINK Telecom,ISP
|
||||
hspherefilter.com,"DynamicNet, Inc. (DNI)",Web Host
|
||||
hstgr.cloud,Hostinger,Web Host
|
||||
ht.hr,Hrvatski Telekom,ISP
|
||||
htc.net,HTC,ISP
|
||||
htcinc.net,HTC,ISP
|
||||
@@ -2852,8 +2869,10 @@ katelecom.com.br,KA Telecom,ISP
|
||||
kater.com.br,Kater Telecom,ISP
|
||||
kattare.com,Kattare,Web Host
|
||||
katv1.az,AG Telekom,ISP
|
||||
katv1.net,AG Telekom,ISP
|
||||
kaunogrudai.lt,Kauno Grūdai,Food
|
||||
kazintercom.kz,Kaz InterCom,ISP
|
||||
kazrena.kz,KazRENA,Education
|
||||
kbro.com.tw,kbro,ISP
|
||||
kbtelecom.net,Koos Broadband Telecom,ISP
|
||||
kcc.com,Kimberly-Clark,Manufacturing
|
||||
@@ -3328,6 +3347,7 @@ megaconnect.com.br,MegaConnect,ISP
|
||||
megacp.com,iWorx Host,Web Host
|
||||
megadata.net.id,Megadata ISP,ISP
|
||||
megafon.ru,MegaFon,ISP
|
||||
megafon.tj,MegaFon,ISP
|
||||
megaline.kg,Maga-Line,ISP
|
||||
megalink.net.ru,Мегалинк,ISP
|
||||
megalinkinternet.com.br,MegaLink,ISP
|
||||
@@ -4133,6 +4153,7 @@ osogrande.net,Oso Grande Technologies,ISP
|
||||
osorio.rs.gov.br,Prefeitura Municipal de Osório,Government
|
||||
osu.edu,The Ohio State University,Education
|
||||
otago.ac.nz,University of Otago,Education
|
||||
otava.com,Otava,MSP
|
||||
ote.gr,OTE,ISP
|
||||
otelco.net,GoNetSpeed,ISP
|
||||
otenet.gr,Cosmote,ISP
|
||||
@@ -4261,6 +4282,7 @@ pilot-ipek.ru,Izhevsk Industrial-Economic College,Education
|
||||
pilotfiber.com,Pilot Fiber,ISP
|
||||
pink.co.rs,Mink Media Group,Marketing
|
||||
pinnaclecart.com,PinnacleCart,SaaS
|
||||
pinpro.by,PinPro,ISP
|
||||
pioneer.co.th,Pioneer Transportation,Logistics
|
||||
pip.com.au,PIP Total IT Solutions,Web Host
|
||||
pipefy.com,Pipefy Inc,SaaS
|
||||
@@ -4372,6 +4394,7 @@ promax.media.pl,Promax,ISP
|
||||
prontomarketing.com,Pronto Marketing,Marketing
|
||||
proofhubmail.com,ProofHub,SaaS
|
||||
prosites.com,ProSites,Web Host
|
||||
prospect.pl,Prospect,ISP
|
||||
proton.ch,Proton,Email Provider
|
||||
proton.me,Proton,Email Provider
|
||||
protonmail.ch,Proton,Email Provider
|
||||
@@ -4477,6 +4500,7 @@ radarinternet.com.br,Radar Internet,ISP
|
||||
radford.edu,Radford University,Education
|
||||
radiant.net,TELUS,ISP
|
||||
radiantsolutions.net,Radient Solutions,Web Host
|
||||
radius-net.com,Radius-NET,ISP
|
||||
radware.com,Radware,Email Security
|
||||
rafftechnologies.com,Raff Technologies,Web Host
|
||||
rain.co.za,Rain,ISP
|
||||
@@ -4700,6 +4724,7 @@ sangchaimeter.com,Sangchai Meter,Industrial
|
||||
sanofi.fr,Sanofi,Healthcare
|
||||
santehealth.net,Sante Health System,Healthcare
|
||||
sanwa.co.jp,Sanwa Supply,Retail
|
||||
sanwakai.jp,Sanwakai Hospital,Healthcare
|
||||
sap.com,SAP,Technology
|
||||
saplbd.com,Summit Alliance Port,Logistics
|
||||
sapmed.ac.jp,Sapporo Medical University,Healthcare
|
||||
@@ -4708,6 +4733,7 @@ saremail.com,SAREnet,ISP
|
||||
sarenet.es,Sarenet,ISP
|
||||
sarkor.uz,Sarkor,ISP
|
||||
sasktel.com,SaskTel,ISP
|
||||
satis-tl.ru,Satis,ISP
|
||||
satnet.net,Xtrim,ISP
|
||||
satorilabs.com,NextGen Healthcare,Healthcare
|
||||
sattrakt.com,Sat-Trakt,ISP
|
||||
@@ -4734,6 +4760,7 @@ sch.gr,Greek School Network,Education
|
||||
schibsted.no,Schibsted Media,News
|
||||
schmc.ac.kr,Soon Chun Hyang University Medical Center,Healthcare
|
||||
scinternet.net,South Central Communications,ISP
|
||||
scis.gov.az,Special Communication Service of Azerbaijan,Government
|
||||
scnet.com.br,SCNet,ISP
|
||||
scorm.com,SCORM,SaaS
|
||||
scrio.com.br,S.C. RIO Telecom,ISP
|
||||
@@ -5062,6 +5089,7 @@ southlandind.com,Southland Industries,Construction
|
||||
souuni.com,Uni Telecom,ISP
|
||||
soverin.com,Soverin,Email Provider
|
||||
soverin.net,Soverin,Email Provider
|
||||
sp2-brevo.net,Brevo (Sendinblue),Marketing
|
||||
space.net,SpaceNet,MSP
|
||||
spaceship.com,Spaceship,Web Host
|
||||
spaceship.net,Spaceship,Web Host
|
||||
@@ -5359,6 +5387,7 @@ tedforbes.com,Ted Forbes,Photography
|
||||
tees.ne.jp,Toyohashi Cable Network,ISP
|
||||
teksavvy.com,TekSavvy,ISP
|
||||
tel.net.ba,HT Eronet,ISP
|
||||
telbo.net,TELBO,ISP
|
||||
telcocom.com.ar,Telcocom,ISP
|
||||
telcomaster.com,TelcoMaster,ISP
|
||||
telconet.ec,Telconet,ISP
|
||||
@@ -5461,6 +5490,8 @@ tennessee.edu,University of Tennessee,Education
|
||||
teol.net,mtel,ISP
|
||||
tera-byte.com,Tera-Byte,Web Host
|
||||
terago.ca,TeraGo,ISP
|
||||
teraline-telecom.net,Teraline Telecom,ISP
|
||||
teraline.kz,Teraline Telecom,ISP
|
||||
teralinktelecom.com.br,Teralink,ISP
|
||||
terra.com,Terra Networks,Web Host
|
||||
terra.com.br,Terra Mail,Email Provider
|
||||
@@ -5560,6 +5591,7 @@ tofinosoftware.com,Tofino,SaaS
|
||||
tohoku-mpu.ac.jp,Tohoku Medical and Pharmaceutical University,Education
|
||||
tohoku.ac.jp,Tohoku University,Education
|
||||
tohoyk.co.jp,Toho Pharmaceutical,Healthcare
|
||||
tojikiston.com,Babilon-T,ISP
|
||||
tokai-com.co.jp,TOKAI Communications,ISP
|
||||
tokala-mobile.com,Tokala Communications,ISP
|
||||
top-tokyo.co.jp,TOP CORPORATION,Healthcare
|
||||
@@ -5616,6 +5648,7 @@ trifle.net,Trifle Network,ISP
|
||||
trinity.cz,AB-NET,ISP
|
||||
triolan.com,Triolan,ISP
|
||||
triolan.net,Triolan,ISP
|
||||
tripster.com,Tripster,Travel
|
||||
triton.net,Triton Technologies,Technology
|
||||
trivenet.it,Planetel,ISP
|
||||
trooli.com,Trooli,ISP
|
||||
@@ -5633,6 +5666,7 @@ tstc.edu,Texas State Technical College,Education
|
||||
tstt.co.tt,TSTT,ISP
|
||||
tstt.net.tt,TSTT,ISP
|
||||
tsukuba.ac.jp,University of Tsukuba,Education
|
||||
ttc.kz,Transtelecom Kazakhstan,ISP
|
||||
ttcl.co.tz,Tanzania Telecommunications,ISP
|
||||
tti.com,TTI,Industrial
|
||||
ttiinc.com,TTI,Industrial
|
||||
@@ -5708,6 +5742,7 @@ ucar.edu,University Corporation for Atmospheric Research,Education
|
||||
uccs.edu,University of Colorado Colorado Springs,Education
|
||||
ucdavis.edu,UC Davis,Education
|
||||
ucdenver.edu,University of Colorado Denver,Education
|
||||
ucell.uz,Ucell,ISP
|
||||
ucf.edu,University of Central Florida,Education
|
||||
uchealth.com,UC Health,Healthcare
|
||||
uchicago.edu,University of Chicago,Education
|
||||
@@ -5732,11 +5767,13 @@ udc.hk,Hong Kong Communications,Web Host
|
||||
udel.edu,University of Delaware,Education
|
||||
udg.mx,Universidad de Guadalajara,Education
|
||||
udlap.mx,Universidad de las Americas Puebla,Education
|
||||
udomain.hk,UDomain,Web Host
|
||||
uen.org,Utah Education Network,Education
|
||||
ufanet.ru,Ufanet,ISP
|
||||
ufinet.com,Ufinet,ISP
|
||||
ufl.edu,University of Florida,Education
|
||||
ufmg.br,UFMG,Education
|
||||
ufone.com,Ufone,ISP
|
||||
ufrgs.br,UFRGS,Education
|
||||
ufsc.br,UFSC,Education
|
||||
uga.edu,University of Georgia,Education
|
||||
@@ -5973,6 +6010,7 @@ vercel.app,Vercel,PaaS
|
||||
vercel.com,Vercel,PaaS
|
||||
vercel.dev,Vercel,PaaS
|
||||
vercel.run,Vercel,PaaS
|
||||
verde.ag,Verde Agritech,Agriculture
|
||||
verginia.edu,University of Virginia,Education
|
||||
veridyen.com,Veridyen,Web Host
|
||||
verisign.com,Verisign,Technology
|
||||
@@ -6042,6 +6080,7 @@ virtualhosting.hk,UDomain,Web Host
|
||||
virtualnetfibra.com.br,Virtual Net Fibra,ISP
|
||||
virtualserver.io,Zitcom,Web Host
|
||||
virtuaserver.com.br,VirtuaServer,Web Host
|
||||
visa.com,Visa,Finance
|
||||
visabeira.co.ao,COMATEL,ISP
|
||||
visi.com,US Signal,MSP
|
||||
visitmiddlesex.ca,"Middlesex County, Canada",Government
|
||||
@@ -6264,6 +6303,7 @@ windriver.com,Wind River,SaaS
|
||||
windstream.net,Windstream,ISP
|
||||
windtre.it,WINDTRE,ISP
|
||||
wingnet.com.br,WINGNET,ISP
|
||||
wingo.ch,Wingo,ISP
|
||||
winknet.ne.jp,Himeji Cable Television,ISP
|
||||
winnermedical.com,Winner Medical,Healthcare
|
||||
wiredblade.com,Wired Blade,Web Host
|
||||
|
||||
|
@@ -3,6 +3,8 @@
|
||||
import os
|
||||
import csv
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
|
||||
# Privacy filter: a reverse DNS entry containing a full IPv4 address (four
|
||||
@@ -13,6 +15,13 @@ _FULL_IP_RE = re.compile(
|
||||
r"(?<![\d])(\d{1,3})[-.](\d{1,3})[-.](\d{1,3})[-.](\d{1,3})(?![\d])"
|
||||
)
|
||||
|
||||
# A source_name can fail this match when parsedmarc's ASN-fallback path in
|
||||
# utils.py:get_ip_address_info surfaces the raw MMDB ``as_name`` (e.g. "VODAFONE
|
||||
# GROUP PLC") because the IP had no PTR and the as_domain wasn't in the map.
|
||||
_DOMAIN_RE = re.compile(
|
||||
r"^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)+$"
|
||||
)
|
||||
|
||||
|
||||
def _has_full_ip(s: str) -> bool:
|
||||
for m in _FULL_IP_RE.finditer(s):
|
||||
@@ -22,11 +31,61 @@ def _has_full_ip(s: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _looks_like_domain(s: str) -> bool:
|
||||
return bool(_DOMAIN_RE.match(s))
|
||||
|
||||
|
||||
def _normalize_as_name(s: str) -> str:
|
||||
# NBSP (U+00A0) appears in both MMDB as_names and CSV source_names but
|
||||
# not always on the same side, so an exact match misses pairs that are
|
||||
# otherwise identical. Fold NBSP to a regular space and collapse runs
|
||||
# of whitespace before comparing.
|
||||
return re.sub(r"\s+", " ", s.replace("\xa0", " ")).lower().strip()
|
||||
|
||||
|
||||
def _load_as_name_index(mmdb_path: str) -> dict[str, str]:
|
||||
"""Build a normalized as_name -> as_domain index from the bundled MMDB.
|
||||
|
||||
When a single as_name maps to multiple as_domains (about 1% of records),
|
||||
the as_domain with the largest aggregate IPv4 footprint wins.
|
||||
"""
|
||||
try:
|
||||
import maxminddb
|
||||
except ImportError:
|
||||
print(
|
||||
"Error: maxminddb is required to translate AS-name source rows; "
|
||||
"install parsedmarc's runtime dependencies (pip install maxminddb)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
counts: dict[tuple[str, str], int] = defaultdict(int)
|
||||
with maxminddb.open_database(mmdb_path) as reader:
|
||||
for net, rec in reader:
|
||||
if net.version != 4 or not isinstance(rec, dict):
|
||||
continue
|
||||
as_name = rec.get("as_name")
|
||||
as_domain = rec.get("as_domain")
|
||||
if not as_name or not as_domain:
|
||||
continue
|
||||
counts[(_normalize_as_name(as_name), as_domain.lower().strip())] += (
|
||||
net.num_addresses
|
||||
)
|
||||
|
||||
best: dict[str, tuple[str, int]] = {}
|
||||
for (as_name_lower, as_domain_lower), count in counts.items():
|
||||
existing = best.get(as_name_lower)
|
||||
if existing is None or count > existing[1]:
|
||||
best[as_name_lower] = (as_domain_lower, count)
|
||||
return {k: v[0] for k, v in best.items()}
|
||||
|
||||
|
||||
def _main():
|
||||
input_csv_file_path = "base_reverse_dns.csv"
|
||||
base_reverse_dns_map_file_path = "base_reverse_dns_map.csv"
|
||||
known_unknown_list_file_path = "known_unknown_base_reverse_dns.txt"
|
||||
psl_overrides_file_path = "psl_overrides.txt"
|
||||
mmdb_file_path = "../ipinfo/ipinfo_lite.mmdb"
|
||||
output_csv_file_path = "unknown_base_reverse_dns.csv"
|
||||
|
||||
csv_headers = ["source_name", "message_count"]
|
||||
@@ -51,6 +110,12 @@ def _main():
|
||||
|
||||
load_list(known_unknown_list_file_path, known_unknown_domains)
|
||||
load_list(psl_overrides_file_path, psl_overrides)
|
||||
if not os.path.exists(mmdb_file_path):
|
||||
print(f"Error: {mmdb_file_path} does not exist")
|
||||
exit(1)
|
||||
print(f"Loading {mmdb_file_path}")
|
||||
as_name_index = _load_as_name_index(mmdb_file_path)
|
||||
print(f"Indexed {len(as_name_index)} as_names from the MMDB")
|
||||
if not os.path.exists(base_reverse_dns_map_file_path):
|
||||
print(f"Error: {base_reverse_dns_map_file_path} does not exist")
|
||||
print(f"Loading {base_reverse_dns_map_file_path}")
|
||||
@@ -78,6 +143,24 @@ def _main():
|
||||
domain = row["source_name"].lower().strip()
|
||||
if domain == "":
|
||||
continue
|
||||
# If source_name is not domain-shaped, parsedmarc's ASN-fallback
|
||||
# path (utils.py:get_ip_address_info) surfaced the raw MMDB
|
||||
# ``as_name`` because the IP had no PTR and the as_domain wasn't
|
||||
# in the map. Translate to the corresponding as_domain so the
|
||||
# row enters the pipeline as a researchable domain. If the
|
||||
# as_domain is already in the map, the row drops out below as a
|
||||
# known domain — exactly what we want.
|
||||
if not _looks_like_domain(domain):
|
||||
translated = as_name_index.get(_normalize_as_name(domain))
|
||||
if translated is None:
|
||||
print(
|
||||
f"Skipping AS-name source with no MMDB match: "
|
||||
f"{row['source_name']!r}"
|
||||
)
|
||||
continue
|
||||
print(f"Translating AS-name {row['source_name']!r} -> {translated}")
|
||||
row["source_name"] = translated
|
||||
domain = translated
|
||||
for psl_domain in psl_overrides:
|
||||
if domain.endswith(psl_domain):
|
||||
domain = psl_domain.strip(".").strip("-")
|
||||
|
||||
@@ -85,6 +85,7 @@ admin.corpivensa.gob.ve
|
||||
admincomp.com
|
||||
adriadns.com
|
||||
adsl.cat
|
||||
adslplus.ch
|
||||
adsnet-telecom.net.br
|
||||
advance.com.ar
|
||||
advancedridingtechniques.info
|
||||
@@ -117,6 +118,7 @@ airsystem33.com
|
||||
airtel.net
|
||||
airyhosting.com
|
||||
aistechsolutions.com
|
||||
aitacargas.uy
|
||||
ajorealestate.com
|
||||
ak888.ru
|
||||
akado-ural.ru
|
||||
@@ -335,6 +337,7 @@ bitter-echo.com
|
||||
bitternet.ua
|
||||
biz-ps.com
|
||||
bizhostingservices.com
|
||||
biznetg.io
|
||||
bizylady.com
|
||||
blackhawk-net.com
|
||||
blackpointcyber.com
|
||||
@@ -342,6 +345,7 @@ blackwaterregion.com
|
||||
blguss.com
|
||||
blmpb.com
|
||||
blockchainlaw.info
|
||||
blogwisefinance.com
|
||||
blondewigs.shop
|
||||
bloodfire9.com
|
||||
bloomingdales.com
|
||||
@@ -419,6 +423,7 @@ byethost31.org
|
||||
byronservices.com
|
||||
bzconcept.com
|
||||
c53dw7m24rj.com
|
||||
c5ace.com
|
||||
ca-expressive.site
|
||||
cablecolor.com.sv
|
||||
caccaweb.com
|
||||
@@ -477,6 +482,7 @@ cdcservices.com
|
||||
cdn1000.com
|
||||
cdt.cz
|
||||
cegalvez.com.ar
|
||||
ceiausa.com
|
||||
centralmalaysia.com
|
||||
centralnic.net
|
||||
centrum.dk
|
||||
@@ -485,6 +491,7 @@ cf-emailsecurity.net
|
||||
cfnationt.de.com
|
||||
cgate-global.com
|
||||
ch-abbeville.fr
|
||||
ch-charleville-mezieres.fr
|
||||
ch-dns.net
|
||||
ch-tournon.fr
|
||||
chacoonline.net
|
||||
@@ -842,6 +849,7 @@ e-shops.jp
|
||||
e-tiger.net
|
||||
e3webmail.com
|
||||
eagleredes.net.br
|
||||
eaglezip.com
|
||||
eappscdn.com
|
||||
earthcare.com
|
||||
easternkingspei.com
|
||||
@@ -854,6 +862,7 @@ ecdaoductsservi-com.icu
|
||||
ecofex-mx.icu
|
||||
econet.co.ls
|
||||
economiceagles.com
|
||||
ecova.com
|
||||
ecuahosting.net
|
||||
edcor.com
|
||||
edit.ne.jp
|
||||
@@ -906,6 +915,7 @@ enticingofdisadvantage.com
|
||||
entretothom.net
|
||||
enzu.com
|
||||
epaycontrol.com
|
||||
epchd.org
|
||||
epectelco.com.ar
|
||||
epicinvestmentsreview.co
|
||||
epicinvestmentsreview.com
|
||||
@@ -1051,6 +1061,7 @@ foxnew.de
|
||||
francecanterbury.com
|
||||
frauen-online.net
|
||||
fredi.shop
|
||||
fredperryst.com
|
||||
free.ds
|
||||
freehypnosisdownloads.com
|
||||
freenet.de
|
||||
@@ -1123,6 +1134,7 @@ ginza-entre.com
|
||||
gis.net
|
||||
gist-th.com
|
||||
glamsio.com
|
||||
glattnet.ch
|
||||
glenpondia.xyz
|
||||
glenporch.cam
|
||||
glenverra.shop
|
||||
@@ -1218,6 +1230,7 @@ harmonianhub.com
|
||||
haswu.com
|
||||
hawaiiantel.net
|
||||
hawkaccess.com
|
||||
hderit.com
|
||||
hdomestic.com
|
||||
hdprosupply.com
|
||||
healthcareathome.ca
|
||||
@@ -1278,6 +1291,7 @@ hostedexchange2010.fr
|
||||
hostednetworks.net
|
||||
hostedvps.co.uk
|
||||
hostelsucre.com
|
||||
hosterby.com
|
||||
hostflyby.net
|
||||
hostidadns.com
|
||||
hosting1337.com
|
||||
@@ -1357,6 +1371,7 @@ igmohji.com
|
||||
igppevents.org.uk
|
||||
igroupnet.com
|
||||
ihglobaldns.com
|
||||
ijalti.org.mx
|
||||
ijetabtens-mx.icu
|
||||
ikexpress.com
|
||||
ileysinc.com
|
||||
@@ -1370,6 +1385,7 @@ imagica-imageworks.co.jp
|
||||
imazu-chemical.co.jp
|
||||
imjtmn.cn
|
||||
immenzaces.com
|
||||
imobie.com
|
||||
imp.ch
|
||||
impey.shop
|
||||
improvation.us
|
||||
@@ -1490,6 +1506,7 @@ jaeres.com
|
||||
janikingks.com
|
||||
japbuk.com
|
||||
jarmuloviktazone.com
|
||||
jasarekber.id
|
||||
jatimnet.com.br
|
||||
jaxa.jp
|
||||
jazhostingco.com
|
||||
@@ -1550,6 +1567,7 @@ kenkids.com
|
||||
kennport.com
|
||||
kenobianresearch.com
|
||||
kernel.org
|
||||
kesha145216.vds
|
||||
kevalier.com
|
||||
keznews.com
|
||||
khwebsolutions.co.uk
|
||||
@@ -1948,6 +1966,7 @@ net-rubi.com.br
|
||||
netbitsfibra.com.br
|
||||
netboard.hu
|
||||
netbri.com
|
||||
netbuild.net
|
||||
netcarrier.net
|
||||
netcell.inf.br
|
||||
netcentertelecom.net.br
|
||||
@@ -1980,6 +1999,7 @@ newlifeinformatica.com.br
|
||||
newnetprovedor.net.br
|
||||
newsv.jp
|
||||
newwallstreetcode.com
|
||||
newwave.com
|
||||
newyorkhonorlodge.com
|
||||
newyorkmarketanalytics.com
|
||||
nex-tech.com
|
||||
@@ -2026,6 +2046,7 @@ nossanettelecom.net.br
|
||||
novanetnp.net.br
|
||||
novast.com
|
||||
novuscom.net
|
||||
npbnl.net
|
||||
nqntv.com.ar
|
||||
ns360.net
|
||||
nsk-kk.com
|
||||
@@ -2067,6 +2088,7 @@ ogicom.net
|
||||
ogreserve.com
|
||||
okweb3.jp
|
||||
olb-access.net
|
||||
olbitx.network
|
||||
olcx.net
|
||||
olivettilexikon.co.uk
|
||||
ollatelecom.net.br
|
||||
@@ -2222,6 +2244,7 @@ pr-m.eu
|
||||
pravnagroup-m.pl
|
||||
preblepride.shop
|
||||
precisionpros.com
|
||||
prevail-portal.com
|
||||
priawesta.store
|
||||
prima.com.ar
|
||||
prima.net.ar
|
||||
@@ -2288,7 +2311,9 @@ raxa.host
|
||||
rcncustomer.com
|
||||
rdock.ru
|
||||
reachesbeast.com.de
|
||||
readyidc.com
|
||||
realestateofdetroit.com
|
||||
realssl.com
|
||||
reberte.com
|
||||
receita.pb.gov.br
|
||||
recruit-ms.co.jp
|
||||
@@ -2314,6 +2339,7 @@ reliablepanel.com
|
||||
reloadhard.com
|
||||
renchildbirth.com
|
||||
renkus-hainz.com
|
||||
renner.uz
|
||||
renopie.com
|
||||
renttitllc.com
|
||||
researchmarketreports.com
|
||||
@@ -2322,6 +2348,7 @@ resellerex.com
|
||||
reso-tech.ca
|
||||
restocarpediem.com
|
||||
resurrectioncatholic.church
|
||||
retemetis.net
|
||||
rethos.com.mx
|
||||
retirementconcepts.com
|
||||
retroflect.shop
|
||||
@@ -2455,6 +2482,7 @@ serversignin.com
|
||||
servicenetpro.com.br
|
||||
servidor2.net
|
||||
sevenmeadowsfarm.net
|
||||
sevtele.com
|
||||
seymenkolej.k12.tr
|
||||
sfek.kz
|
||||
sfr-sh.net
|
||||
@@ -2534,6 +2562,7 @@ solucija.eu
|
||||
solusoftware.com
|
||||
solutionspal.com
|
||||
soon.com.ar
|
||||
sopricom.fr
|
||||
soultelecom.net.br
|
||||
sounditaly.com
|
||||
sourcedns.com
|
||||
@@ -2616,6 +2645,7 @@ suksangroup.com
|
||||
sulnet.net.br
|
||||
super-dns.net
|
||||
supercanal.com.ar
|
||||
superdata.vn
|
||||
superdnssite.com
|
||||
superdomainzone.com
|
||||
superior-host.com
|
||||
@@ -2690,6 +2720,7 @@ tenkids.net
|
||||
tenriyorozu.jp
|
||||
tenten.vn
|
||||
teraswitch.com
|
||||
terawiz.com
|
||||
terminavalley.com
|
||||
tetrocapital.com
|
||||
texasgs.net
|
||||
@@ -2733,6 +2764,7 @@ tilaa.cloud
|
||||
timbrasil.com.br
|
||||
tipira.net.br
|
||||
titamnet.net.br
|
||||
titandc.io
|
||||
tld-mx.com
|
||||
tm.net.my
|
||||
tmg.md
|
||||
@@ -2744,6 +2776,7 @@ tohknet.co.jp
|
||||
tokushukai.or.jp
|
||||
tokyorinkai.jp
|
||||
toledofibra.net.br
|
||||
tomcity.net
|
||||
tomtelnet.ru
|
||||
tondyfree.vip
|
||||
tongx.com.tw
|
||||
@@ -2814,8 +2847,10 @@ twohoax.info
|
||||
twp-capital.com
|
||||
txw-taishin.com
|
||||
tylerfordonline.com
|
||||
tymtalk.com
|
||||
tzulo.com
|
||||
u-ikoi.co.jp
|
||||
u-team.by
|
||||
uaehostings.com
|
||||
ubicarme.com
|
||||
ubjp.cn
|
||||
|
||||
Reference in New Issue
Block a user