Batch 1 — MMDB coverage-gap scan (find_unmapped_as_domains.py, default
4,096-IP floor): 132 candidates collected and classified. Manually
audited every auto-classified row rather than trusting the classifier
output directly, since it derives as_name straight from the MMDB and
will auto-classify off that single string even with no reachable
homepage — which fails the two-corroborating-sources bar. Demoted 10
rows to known-unknown on that basis (e.g. pellera.com's ASN registrant
is "Converge Technology Solutions", aerloop.in's is "Aerpace" — no
shared brand token, single source). Corrected several brand-quality
issues before they hit the map: sdtv.com.tw was about to ship as
"SDTV Webmail" (a login-page title) instead of "San Da Cable TV";
liyang.gov.cn was about to ship the hosting network's name instead of
the actual government operator (confirmed via Chinese-language WHOIS).
Also fixed a pre-existing bad entry noticed in passing: backwaves.net
was mapped under a garbled Chinese tagline instead of "Back Waves".
Result: 52 map rows, 82 known-unknown entries.
Batch 2 — plain-text uncategorized-sources export (842 source names via
find_unknown_base_reverse_dns.py -i, mix of raw MMDB as_name strings
and reverse-DNS domains) -> 377 unknown domains -> collected and
classified. This batch surfaced more data-quality traps: a parked
domain-registrar welcome page (serveur-vps.net) and a "Site under
maintenance" title (gou.go.ug, resolved via TLD instead) auto-classified
as if they were live operators; a WHOIS privacy-proxy service name
("GKG.NET Domain Proxy Service Administrator") slipped past the
privacy-org filter for atlasok.com; grohe.org.ru turned out to be a
Russian third-party reseller storefront, not the real Grohe corporation,
so it was held back from the map to avoid misattributing a reseller's
mail to the multinational. Refined the two-source test to distinguish
domain-WHOIS (ties directly to the domain; counts even without a
literal name match, e.g. portotelecom.net.br's WHOIS "Rondotech" backs
its self-described "Porto Telecom" brand) from MMDB as_name
(infrastructure-level signal, needs an actual lexical link, same
caution as IP-WHOIS). Used existing map entries as legitimate second
sources for redirects/aliases (vsys.host -> v-sys.org, metronet.net ->
metronet.com, mentari.net.id -> megahub.id) and cleaned up
megahub.id's own tagline-derived name to "Megahub" while there.
Result: 24 map rows, 353 known-unknown entries.
Also ignore the scratch/ working directory used to stage the
uncategorized-sources export for this pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Accept plain-text uncategorized-sources lists in find_unknown_base_reverse_dns.py
Dashboard exports of uncategorized email sources are plain-text lists of
one source name per line — a mix of raw MMDB as_name strings (when the
source IP had no PTR and resolved via the IPinfo Lite MMDB) and base
reverse-DNS domains. The script already translates as_names to their
as_domain and subtracts mapped/known-unknown entries, but only read a
hardcoded source_name-headed CSV.
Add -i/--input and -o/--output flags (defaults preserve current
behavior) and auto-detect the input format from the first line: a
source_name CSV header selects the existing DictReader path, anything
else is read as plain text with each line taken verbatim (never
comma-split, since as_names contain commas) and deduped
case-insensitively. Fix the missing-input error message, which
reported the map path instead of the input path.
Document the new entry point in the maps README and AGENTS.md, and make
explicit in the brand-quality triage rule that map display names must be
human-friendly operator names — never raw as_name strings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add MMDB coverage scan script with anti-poisoning guards
find_unmapped_as_domains.py turns the manual "Checking ASN-domain
coverage of the MMDB" recipe into a maintainer script: walk every IPv4
record in the bundled IPinfo Lite MMDB, aggregate routed footprint per
as_domain, subtract mapped/known-unknown keys, apply PSL folding and
the full-IP privacy filter, and emit domain,ipv4_count,as_name sorted
by footprint for the collector -> classifier pipeline.
Because ASN registration data is self-declared to the RIRs and
as_domain derives from registrant-controlled WHOIS, bulk-categorizing
the MMDB needs poisoning defenses:
- An IPv4-footprint floor (--min-ips, default 4096, a /20) keeps tiny
self-described ASNs out of the auto-classification queue; dropped
counts are always printed.
- A brand-collision guard in classify_unknown_domains.py loads the
existing map (--map) and demotes any single-category candidate whose
proposed display name matches an existing map name without a lexical
relationship to that operator's keys into the ambiguous bucket
(marked name-collision-with-existing-map-entry) for human review.
HAND overrides bypass the guard. The guard protects the PTR-side
flow as well as the MMDB-coverage flow.
Verified: scan yields 132 candidates at the default floor (1512
dropped); collector accepts the output directly; a fixture titled as
Comcast under an unrelated domain lands in ambiguous while a
comcast-rooted sibling auto-promotes. Also fix the maps README links
that still pointed at the root AGENTS.md for the classification
workflow after its extraction to maps/AGENTS.md, and correct the
classify_tsv docstring's return signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Enhance planning guidance in CLAUDE.md by specifying auto mode activation after user approval
* Codify triage flagging for identified operators with no fitting type
An operator confidently identified from two corroborating sources but
matching none of the README's type values should be flagged during
triage with a proposed new type for the reviewer, not force-fitted and
not silently recorded as known-unknown — KU means "we couldn't
identify this", which would bury completed research. Extends workflow
rule 7 and the LLM low-confidence list in the maps AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Split tests.py into per-module tests/test_<module>.py
The 5174-line tests.py monolith is split into per-module files under
tests/, mirroring the checkdmarc layout:
tests/test_init.py parsedmarc/__init__.py parsing surface
tests/test_cli.py parsedmarc/cli.py + config / env-vars / SIGHUP
tests/test_utils.py parsedmarc/utils.py (DNS, IP info, PSL, etc.)
tests/test_webhook.py parsedmarc/webhook.py
tests/test_kafkaclient.py parsedmarc/kafkaclient.py
tests/test_splunk.py parsedmarc/splunk.py
tests/test_syslog.py parsedmarc/syslog.py
tests/test_loganalytics.py parsedmarc/loganalytics.py
tests/test_gelf.py parsedmarc/gelf.py
tests/test_s3.py parsedmarc/s3.py
tests/test_maps.py parsedmarc/resources/maps/ maintainer scripts
The split is purely a redistribution — no test bodies changed, no tests
added or removed. All 276 existing tests pass under the new layout.
The current tests.py contains two kitchen-sink classes (`Test` at line 54
and `TestEnvVarConfig` at line 2360) holding tests that span many
modules. Their methods are routed to the correct per-module file by name
prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx,
TestSighupReload, etc.) move whole. Each target file gets its own
`class Test(unittest.TestCase)` for the redistributed kitchen-sink
methods, plus the thematic classes verbatim.
Wiring updates:
- `.github/workflows/python-tests.yml`: `pytest ... tests.py` →
`python -m pytest ... tests/` (also switches to `python -m pytest` per
the checkdmarc convention so cwd lands on the project root).
- `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]`
and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for
`parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only
batch tooling that ships out of the wheel; excluding them from
coverage makes the headline number reflect only installed library
code. Runtime coverage on the new layout is 59% (was 45% with maps
counted), and PR-B will push it to 90%+.
- `AGENTS.md`: documents the new layout and how to run individual files
/ tests; tells future contributors not to reintroduce a monolithic
tests.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore 66.9% coverage baseline (count tests/ + parsedmarc)
Master's headline 66.9% number on Codecov includes the tests.py file
itself (99.35% covered) being measured alongside parsedmarc/*. The
original tests.py had no `[tool.coverage.run]` block, so coverage's
default — "measure every file imported during the run" — counted the
test code as if it were product code.
The split commit added `source = ["parsedmarc"]` which suppressed
measurement of the test files (correct in principle, since test files
aren't shipped code), and that alone made the headline number drop by
~8 percentage points without any actual loss of testing. This commit
swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]`
so both halves are measured the way they were on master. Verified:
276 tests, 66.96% line coverage (effectively unchanged from master's
66.90%).
If you want the shipped-code-only number (was the headline that this
commit overrides), run `pytest --cov=parsedmarc tests/`. That number
is currently 59% and is the focus of the upcoming coverage-expansion PR.
Also adds junit.xml to .gitignore so the CI artefact doesn't get
accidentally committed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restrict coverage to shipped code (`source = ["parsedmarc"]`)
Reverts the prior commit's `include = ["tests/*"]`. Counting the test
files toward coverage was wrong — it conflates "shipped code exercised
by tests" with "test code that pytest auto-runs", inflates the headline
number, and rewards writing more tests rather than tests that verify
more code. Master's apparent 66.9% was an artefact of the old
monolithic tests.py having no [tool.coverage.run] block at all; coverage's
default behaviour measured every imported file, including the test file
itself at ~99% "covered", which added ~8 percentage points to the
displayed number without any real testing signal.
Restricting to `source = ["parsedmarc"]` plus the existing maps omit
gives a meaningful baseline: 59% of shipped code is exercised by the
test suite today. That's the number the next PR is targeting to lift
to 90%+ before the 10.0.0 release; the Codecov "drop" here is a
measurement correction, not a regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Auto-download psl_overrides.txt at startup (and whenever the reverse DNS
map is reloaded) via load_psl_overrides(); add local_psl_overrides_path
and psl_overrides_url config options
- Add collect_domain_info.py and detect_psl_overrides.py for bulk WHOIS/HTTP
enrichment and automatic cluster-based PSL override detection
- Block full-IPv4 reverse-DNS entries from ever entering
base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, or
unknown_base_reverse_dns.csv, and sweep pre-existing IP entries
- Add Religion and Utilities to the allowed service_type values
- Document the full map-maintenance workflow in AGENTS.md
- Substantial expansion of base_reverse_dns_map.csv (net ~+1,000 entries)
- Add 26 tests covering the new loader, IP filter, PSL fold logic, and
cluster detection
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
- Add source name and type information based on static mapping of the reverse DNS base domain
- See [this documentation](https://github.com/domainaware/parsedmarc/tree/master/parsedmarc/resources/maps) for more information, and to learn how to help!
- Replace `multiprocessing.Pool` with `Pipe` + `Process` (PR #491 closes issue #489)
- Remove unused parallel arguments (PR #492 closes issue #490)
- A static copy of the DBIP database is now included for use when a copy of the MaxMind GeoLite2 Country database is not installed (Closes#275)
- Add `ip_db_path` to as a parameter and `general` setting for a custom IP geolocation database location (Closes#184)
- Search default Homebrew path when searching for a copy of the MaxMind GeoLite2 Country database (Closes#272)
- Fix log messages written to root logger (PR #276)
- Fix `--offline` option in CLI not being passed as a boolean (PR #265)
- Set Elasticsearch shard replication to `0` (PR #274)
- Add support for syslog output (PR #263closes#227)
- Do not print TQDDM progress bar when running in a no-interactive TTY (PR #264)
Sean WhalenandClaude Sonnet 5
Sean Whalen
Sean Whalen