mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-02 13:42:19 +00:00
Sean WhalenandClaude Sonnet 5
3b53f836a6
Batch 1 — MMDB coverage-gap scan (find_unmapped_as_domains.py, default
4,096-IP floor): 132 candidates collected and classified. Manually
audited every auto-classified row rather than trusting the classifier
output directly, since it derives as_name straight from the MMDB and
will auto-classify off that single string even with no reachable
homepage — which fails the two-corroborating-sources bar. Demoted 10
rows to known-unknown on that basis (e.g. pellera.com's ASN registrant
is "Converge Technology Solutions", aerloop.in's is "Aerpace" — no
shared brand token, single source). Corrected several brand-quality
issues before they hit the map: sdtv.com.tw was about to ship as
"SDTV Webmail" (a login-page title) instead of "San Da Cable TV";
liyang.gov.cn was about to ship the hosting network's name instead of
the actual government operator (confirmed via Chinese-language WHOIS).
Also fixed a pre-existing bad entry noticed in passing: backwaves.net
was mapped under a garbled Chinese tagline instead of "Back Waves".
Result: 52 map rows, 82 known-unknown entries.
Batch 2 — plain-text uncategorized-sources export (842 source names via
find_unknown_base_reverse_dns.py -i, mix of raw MMDB as_name strings
and reverse-DNS domains) -> 377 unknown domains -> collected and
classified. This batch surfaced more data-quality traps: a parked
domain-registrar welcome page (serveur-vps.net) and a "Site under
maintenance" title (gou.go.ug, resolved via TLD instead) auto-classified
as if they were live operators; a WHOIS privacy-proxy service name
("GKG.NET Domain Proxy Service Administrator") slipped past the
privacy-org filter for atlasok.com; grohe.org.ru turned out to be a
Russian third-party reseller storefront, not the real Grohe corporation,
so it was held back from the map to avoid misattributing a reseller's
mail to the multinational. Refined the two-source test to distinguish
domain-WHOIS (ties directly to the domain; counts even without a
literal name match, e.g. portotelecom.net.br's WHOIS "Rondotech" backs
its self-described "Porto Telecom" brand) from MMDB as_name
(infrastructure-level signal, needs an actual lexical link, same
caution as IP-WHOIS). Used existing map entries as legitimate second
sources for redirects/aliases (vsys.host -> v-sys.org, metronet.net ->
metronet.com, mentari.net.id -> megahub.id) and cleaned up
megahub.id's own tagline-derived name to "Megahub" while there.
Result: 24 map rows, 353 known-unknown entries.
Also ignore the scratch/ working directory used to stage the
uncategorized-sources export for this pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
154 lines
1.8 KiB
Plaintext
154 lines
1.8 KiB
Plaintext
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
env/
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
_tmp*
|
|
|
|
# PyInstaller
|
|
# Usually these files are written by a python script from a template
|
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
*.manifest
|
|
*.spec
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
htmlcov/
|
|
.tox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
*.cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
|
|
# Translations
|
|
*.mo
|
|
*.pot
|
|
|
|
# Django stuff:
|
|
*.log
|
|
local_settings.py
|
|
|
|
# Flask stuff:
|
|
instance/
|
|
.webassets-cache
|
|
|
|
# Scrapy stuff:
|
|
.scrapy
|
|
|
|
# Sphinx documentation
|
|
docs/build/
|
|
|
|
# PyBuilder
|
|
target/
|
|
|
|
# Jupyter Notebook
|
|
.ipynb_checkpoints
|
|
|
|
# pyenv
|
|
.python-version
|
|
|
|
# celery beat schedule file
|
|
celerybeat-schedule
|
|
|
|
# SageMath parsed files
|
|
*.sage.py
|
|
|
|
# dotenv
|
|
.env
|
|
|
|
# virtualenv
|
|
.venv
|
|
venv/
|
|
ENV/
|
|
|
|
# Spyder project settings
|
|
.spyderproject
|
|
.spyproject
|
|
|
|
# Rope project settings
|
|
.ropeproject
|
|
|
|
# mkdocs documentation
|
|
/site
|
|
|
|
# mypy
|
|
.mypy_cache/
|
|
|
|
# PyCharm Project settings
|
|
.idea/
|
|
|
|
# VS Code launch config
|
|
#.vscode/launch.json
|
|
|
|
# Visual Studio Code settings
|
|
#.vscode/
|
|
|
|
# I/O files
|
|
output/
|
|
*.xls*
|
|
|
|
# LibreOffice lock files
|
|
.~*
|
|
|
|
# Data files
|
|
*.dat
|
|
GeoIP*
|
|
GeoLite*
|
|
|
|
# Temp files
|
|
tmp/
|
|
|
|
# Config files
|
|
prod*.ini
|
|
stage*.ini
|
|
dev*.ini
|
|
|
|
# Private samples
|
|
samples/private
|
|
|
|
*.html
|
|
*.sqlite-journal
|
|
|
|
parsedmarc*.ini
|
|
scratch.py
|
|
scratch/
|
|
|
|
parsedmarc/resources/maps/base_reverse_dns.csv
|
|
parsedmarc/resources/maps/unknown_base_reverse_dns.csv
|
|
parsedmarc/resources/maps/unmapped_as_domains.csv
|
|
parsedmarc/resources/maps/sus_domains.csv
|
|
parsedmarc/resources/maps/unknown_domains.txt
|
|
*.bak
|
|
*.lock
|
|
parsedmarc/resources/maps/domain_info.tsv
|
|
coverage.json
|
|
junit.xml
|