- py/empty-except: add a short comment to each of the 10 flagged
`except <Type>: pass` blocks explaining why swallowing the
exception is correct there (best-effort cleanup/close, or
intentional fall-through to the next report format).
- py/unnecessary-lambda: replace `map(lambda x: parse_email_address(x), ...)`
with `map(parse_email_address, ...)` at the 5 flagged sites in
parsedmarc/utils.py; parse_email_address takes exactly one
positional argument, so this is behavior-preserving. Ran
`ruff format` afterward, which collapsed 3 of the now-shorter
calls onto single lines.
- py/imprecise-assert: replace `assertTrue(a > b)` / `assertTrue(a >= b)`
with `assertGreater(a, b)` / `assertGreaterEqual(a, b)` at the 9
flagged test sites; none carried a custom assertion message.
- .vscode/settings.json: drop three cSpell whitelist entries that are
pure misspellings ("passsword", "httpasswd", "unparasable") and
appear nowhere else in the tracked tree, so a recurrence of the typo
they used to hide (see #888) would be caught again. The correctly
spelled "htpasswd" and "unparseable" entries are untouched.
Verified clean: ruff check, ruff format --check, pyright (0
errors/warnings), and pytest tests/ (989 tests collected and passing
before and after, GITHUB_ACTIONS=true).
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two `if not os.path.exists(path): print(f"Error: ...")` checks printed
the intended error message but had no sys.exit(1) after it, so execution
fell through into the subsequent open() call on the same missing path
and raised an unhandled FileNotFoundError instead of the clean error
exit the code clearly intended. Sibling duplicate-entry checks in the
same function already did print-then-sys.exit(1); these two now match.
Sites fixed (both in _main()):
- the nested load_list() helper's missing-file check (used for
known_unknown_base_reverse_dns.txt and psl_overrides.txt)
- the base_reverse_dns_map.csv missing-file check
Both sites now have regression tests in a new TestFindUnknownBaseReverseDNS
class:
- the load_list() site: calling _main() in a temp directory with no
known_unknown_base_reverse_dns.txt now raises SystemExit(1) instead of
FileNotFoundError.
- the base_reverse_dns_map.csv site: _load_as_name_index() does its
external work entirely through maxminddb.open_database(), the actual
SDK boundary, so that call is mocked to a context manager over an
empty iterable rather than loading the real ~23MB bundled MMDB or
mocking an internal helper. Calling _main() in a temp directory with
no base_reverse_dns_map.csv now raises SystemExit(1) instead of
FileNotFoundError.
Both tests capture stdout and assert on the specific "Error: ... does
not exist" message, pinning the exit to the intended site rather than
any sys.exit(1) in the function. Cleanup uses two separate addCleanup
calls (rmtree registered before chdir, so LIFO order runs chdir first)
instead of one lambda wrapping both, so rmtree still runs even if
chdir were to raise.
Also fixed an adjacent prose bug a few lines from the second site: the
"is in known_unknown... and base_reverse_dns_map..." error message was
missing a space after "Error:" and was a backslash-continued f-string
that embedded the source's literal indentation in the printed output.
It now prints as a single clean line, consistent with the file's other
error messages.
Logged the user-facing symptom (clean error replaced by a
FileNotFoundError traceback) under CHANGELOG.md's Unreleased/Bug fixes
section, matching the project's precedent of logging maintainer-tooling
fixes (e.g. the sortlists.py entry).
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
py/use-of-exit-or-quit (29 sites): replace bare `exit(...)` calls with
`sys.exit(...)` in cli.py (20 sites), sortlists.py (4 sites), and
find_unknown_base_reverse_dns.py (5 sites). `exit` is injected into the
interactive namespace by the `site` module, so it is not guaranteed to
exist under `python -S` or an embedded interpreter — the very error
paths that called it would raise NameError instead of exiting cleanly.
`sys.exit` is always available and was already used elsewhere in these
files. sortlists.py did not import `sys`; the import was added to its
existing stdlib import block. find_unknown_base_reverse_dns.py already
imported `sys` (used at line 71), so only the call sites changed there.
Checked tests/test_cli.py for anything patching `exit` directly — none
do; the SystemExit-based assertions exercise the same behavior for
either spelling, so no test changes were needed.
py/multiple-definition (4 sites): removed the dead `index_date`
computation in save_aggregate_report_to_elasticsearch()
(elastic.py:944-947) and save_aggregate_report_to_opensearch()
(opensearch.py:875-878). Both blocks derived index_date from the
report-level begin_date, but the per-record loop immediately below
unconditionally recomputes index_date from each record's own
begin_date before the only read of the variable (the index name build
near the end of the loop body). Verified by reading both functions
that no code between the outer assignment and the loop's recompute
reads index_date, and that the recompute is unconditional on every
path that reaches the read, before deleting.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Make the output and mailbox integrations optional extras (#883)
Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).
cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.
The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.
Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Patch psycopg presence in the PostgreSQL CLI wiring tests
CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs
- The optional-integration and Graph error-type import guards now catch
ModuleNotFoundError instead of ImportError, so only a genuinely absent
package reads as a missing extra; a broken-but-present SDK fails
loudly with its real error instead of masquerading as one. The test
blocker raises ModuleNotFoundError accordingly — the exact exception a
missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
an output module (it also serves the msgraph/gmail_api mailbox
sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
INI key the code reads is password (cli.py _parse_config).
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Quote extras specs in copy-paste install commands
From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix upgrade command in the changelog
* Documentation review: accuracy, spelling, grammar, and clarity pass
A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:
- usage.md: documented six missing [general] options (the CSV/JSON
filename options, prettify_json, normalize_timespan_threshold_hours),
the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
the postgresql env-var prefix; corrected the maildir_path default
(None, not INBOX — cli.py Namespace defaults), the mailbox
check_timeout option name, the systemd restart interval (RestartSec
is 5m), and merged the duplicate silent entry; quoted every
copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
the dashboards filename (opensearch_dashboards.ndjson, matching the
file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
(was parsedmarc.service — a copy-paste error that left DavMail
unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
(7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
matching the feature list. CONTRIBUTING: pre-PR checks now include
ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
the PostgreSQL backend the compose stack runs.
Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Docstring review: accuracy, spelling, grammar, and clarity pass
Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:
- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
splunk/loganalytics save functions described aggregate or failure
reports they do not handle; LogAnalyticsException claimed to be an
Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
enumeration omitted smtp_tls; parse_failure_report typed msg_date as
str (it is datetime); strip_attachment_payloads claimed payloads are
replaced with None (the key is deleted); kafkaclient's failure and
SMTP TLS savers claimed per-record slicing while sending the whole
list in one message (docstrings now describe reality — whether
slicing was intended is flagged for follow-up); the postgres savers
claimed to take parse_report_file's return value but receive the
inner report dict; elastic/opensearch save functions' Raises listed
only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
get_ip_address_country), enumeration completeness fixed
(get_ip_address_info's 9 result keys, maps script outputs, TSV
columns), and the stale 44-industry-types count corrected to the
46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
including two that overstated coverage of the elastic/opensearch
address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
files alongside aggregate and failure, mirrored into usage.md's
CLI-help block; --offline's doubled spaces removed (rendered help
unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
security is enabled and auto-configured on first startup since 8.0
(not "8.7 secure mode"), so the settings are verified, not
hand-written.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Upgrade ruff to 0.16.0 and pyright to 1.1.411; convert to f-strings
ruff 0.16.0 expanded the default lint rule selection well beyond the
long-standing E4/E7/E9/F, so [tool.ruff.lint] now selects the rule set
explicitly: the pre-0.16 defaults plus the modern-type-hint UP rules and
the two f-string rules (UP030/UP032). All 352 UP030/UP032 findings were
auto-fixed; conversions requiring Python 3.12 f-string quote reuse were
conservatively left as .format() by ruff (verified: the whole package and
test suite byte-compile under CPython 3.10.20, the oldest CI version).
Adopting the other newly-default rule families (BLE, SIM, C4, DTZ, I, ...)
is deferred as a deliberate per-family decision.
ruff format with 0.16.0 also now formats Python code fences in Markdown,
which reformatted one block in parsedmarc/resources/maps/AGENTS.md.
ruff check, ruff format --check, pyright (0 errors/warnings), and the
full test suite (775 passed) are green on the new versions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review findings on the f-string conversion
- Rewrite messages that used backslash line continuations inside string
literals, which embedded the source indentation as literal whitespace
in the logged/raised text: the duplicate search-error messages in the
Elasticsearch and OpenSearch outputs, the 'since'-option warning (which
also implicitly concatenated "24hrs" and "SMTP" with no separator) and
the IMAP 'since' debug line, and the missing-org_name KeyError message.
- Build the Splunk HEC newline-delimited payloads by appending to a list
and joining once instead of quadratic string concatenation in a loop.
Declined: switching the webhook output's logger.error to
logger.exception — the single-line ERROR without a traceback is the
deliberate house pattern for batch-resilient sinks, and changing log
verbosity is out of scope for this refactor PR.
ruff check/format, pyright (0 errors/warnings), 775 tests, and a
CPython 3.10 compileall all pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Batch 1 — MMDB coverage-gap scan (find_unmapped_as_domains.py, default
4,096-IP floor): 132 candidates collected and classified. Manually
audited every auto-classified row rather than trusting the classifier
output directly, since it derives as_name straight from the MMDB and
will auto-classify off that single string even with no reachable
homepage — which fails the two-corroborating-sources bar. Demoted 10
rows to known-unknown on that basis (e.g. pellera.com's ASN registrant
is "Converge Technology Solutions", aerloop.in's is "Aerpace" — no
shared brand token, single source). Corrected several brand-quality
issues before they hit the map: sdtv.com.tw was about to ship as
"SDTV Webmail" (a login-page title) instead of "San Da Cable TV";
liyang.gov.cn was about to ship the hosting network's name instead of
the actual government operator (confirmed via Chinese-language WHOIS).
Also fixed a pre-existing bad entry noticed in passing: backwaves.net
was mapped under a garbled Chinese tagline instead of "Back Waves".
Result: 52 map rows, 82 known-unknown entries.
Batch 2 — plain-text uncategorized-sources export (842 source names via
find_unknown_base_reverse_dns.py -i, mix of raw MMDB as_name strings
and reverse-DNS domains) -> 377 unknown domains -> collected and
classified. This batch surfaced more data-quality traps: a parked
domain-registrar welcome page (serveur-vps.net) and a "Site under
maintenance" title (gou.go.ug, resolved via TLD instead) auto-classified
as if they were live operators; a WHOIS privacy-proxy service name
("GKG.NET Domain Proxy Service Administrator") slipped past the
privacy-org filter for atlasok.com; grohe.org.ru turned out to be a
Russian third-party reseller storefront, not the real Grohe corporation,
so it was held back from the map to avoid misattributing a reseller's
mail to the multinational. Refined the two-source test to distinguish
domain-WHOIS (ties directly to the domain; counts even without a
literal name match, e.g. portotelecom.net.br's WHOIS "Rondotech" backs
its self-described "Porto Telecom" brand) from MMDB as_name
(infrastructure-level signal, needs an actual lexical link, same
caution as IP-WHOIS). Used existing map entries as legitimate second
sources for redirects/aliases (vsys.host -> v-sys.org, metronet.net ->
metronet.com, mentari.net.id -> megahub.id) and cleaned up
megahub.id's own tagline-derived name to "Megahub" while there.
Result: 24 map rows, 353 known-unknown entries.
Also ignore the scratch/ working directory used to stage the
uncategorized-sources export for this pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Accept plain-text uncategorized-sources lists in find_unknown_base_reverse_dns.py
Dashboard exports of uncategorized email sources are plain-text lists of
one source name per line — a mix of raw MMDB as_name strings (when the
source IP had no PTR and resolved via the IPinfo Lite MMDB) and base
reverse-DNS domains. The script already translates as_names to their
as_domain and subtracts mapped/known-unknown entries, but only read a
hardcoded source_name-headed CSV.
Add -i/--input and -o/--output flags (defaults preserve current
behavior) and auto-detect the input format from the first line: a
source_name CSV header selects the existing DictReader path, anything
else is read as plain text with each line taken verbatim (never
comma-split, since as_names contain commas) and deduped
case-insensitively. Fix the missing-input error message, which
reported the map path instead of the input path.
Document the new entry point in the maps README and AGENTS.md, and make
explicit in the brand-quality triage rule that map display names must be
human-friendly operator names — never raw as_name strings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add MMDB coverage scan script with anti-poisoning guards
find_unmapped_as_domains.py turns the manual "Checking ASN-domain
coverage of the MMDB" recipe into a maintainer script: walk every IPv4
record in the bundled IPinfo Lite MMDB, aggregate routed footprint per
as_domain, subtract mapped/known-unknown keys, apply PSL folding and
the full-IP privacy filter, and emit domain,ipv4_count,as_name sorted
by footprint for the collector -> classifier pipeline.
Because ASN registration data is self-declared to the RIRs and
as_domain derives from registrant-controlled WHOIS, bulk-categorizing
the MMDB needs poisoning defenses:
- An IPv4-footprint floor (--min-ips, default 4096, a /20) keeps tiny
self-described ASNs out of the auto-classification queue; dropped
counts are always printed.
- A brand-collision guard in classify_unknown_domains.py loads the
existing map (--map) and demotes any single-category candidate whose
proposed display name matches an existing map name without a lexical
relationship to that operator's keys into the ambiguous bucket
(marked name-collision-with-existing-map-entry) for human review.
HAND overrides bypass the guard. The guard protects the PTR-side
flow as well as the MMDB-coverage flow.
Verified: scan yields 132 candidates at the default floor (1512
dropped); collector accepts the output directly; a fixture titled as
Comcast under an unrelated domain lands in ambiguous while a
comcast-rooted sibling auto-promotes. Also fix the maps README links
that still pointed at the root AGENTS.md for the classification
workflow after its extraction to maps/AGENTS.md, and correct the
classify_tsv docstring's return signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Enhance planning guidance in CLAUDE.md by specifying auto mode activation after user approval
* Codify triage flagging for identified operators with no fitting type
An operator confidently identified from two corroborating sources but
matching none of the README's type values should be flagged during
triage with a proposed new type for the reviewer, not force-fitted and
not silently recorded as known-unknown — KU means "we couldn't
identify this", which would bury completed research. Extends workflow
rule 7 and the LLM low-confidence list in the maps AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Move the "Maintaining the reverse DNS maps" section (two-thirds of the
root AGENTS.md) to parsedmarc/resources/maps/AGENTS.md, next to the
files and tooling it governs. Agents that follow the AGENTS.md spec
read the nearest file in the directory tree; a one-line CLAUDE.md
import alongside it covers Claude Code, which loads subdirectory
context lazily when files under maps/ are first read. A pointer stub
remains in the root file for agents that only read the repo root.
The moved content is verbatim except for one cross-reference that now
points at the root AGENTS.md instead of "earlier in this file".
Exclude AGENTS.md/CLAUDE.md from hatch build targets so the new files
don't ship in the wheel, matching the existing README.md exclusion
(verified with a scratch wheel build).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make the whole codebase pass pyright cleanly and enforce it in CI
Fix all 102 pyright (1.1.410, standard mode) errors across the library,
tests, and maps scripts, then pin and enforce the zero-errors bar:
- postgres.py: make the optional psycopg import TYPE_CHECKING-aware so
the module is properly typed while keeping the runtime install-hint
fallback; import psycopg.types.json explicitly as psycopg_json (the
old psycopg_types.json attribute access only worked because psycopg
imports the submodule eagerly); have _connect()/_ensure_connected()
return the live connection so save methods use a non-Optional local;
type the DDL list as list[LiteralString] to match psycopg's execute()
overloads.
- kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error
fallback statically via TYPE_CHECKING (kafka-python 3.0 removed
NoBrokersAvailable), which also fixes _BootstrapError's import
resolution in tests.
- syslog.py: go through getattr/setattr for SysLogHandler.socket
(absent from typeshed); type the save_* methods with the report
TypedDicts (single or list, matching cli.py call sites — gelf.py gets
the same signatures); raise ValueError when retry_attempts < 1
instead of falling through and registering a None handler (bug fix,
with a regression test and a CHANGELOG entry).
- elastic.py / opensearch.py: human_result params are Optional[str].
- maps scripts: sort_csv declared a return type but never returned
(now -> None); seen_sort_field_values was possibly unbound;
convert_to_utf8's src_encoding is Optional[str].
- tests: cast sample-report dict helpers to their TypedDicts; mark
deliberate wrong-type calls with targeted pyright ignores; add
narrowing asserts for Optional results; access the mocked
KafkaProducer through a cast helper; match the mailsuite
fetch_message base signature (**kwargs); patch the renamed
parsedmarc.postgres.psycopg_json in test_postgres's setUpModule.
Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc,
tests, docs; standard mode), pyright==1.1.410 pinned in the [build]
extra (pinned exactly so a new pyright release can't break CI without a
code change), and a "Check types" step in the lint CI job — which now
also runs ruff format --check and installs the [postgresql] extra so
the optional psycopg import resolves. Documented in AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Set session headers via update() instead of replacing the dict
requests 2.34 ships inline type annotations, and Session.headers is a
CaseInsensitiveDict[str] — assigning a plain dict fails pyright there
(the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid
it). headers.update() is correctly typed against both versions, and is
the documented requests idiom: it overrides User-Agent and the
client-specific headers while keeping the session's defaults
(Accept-Encoding, Connection) instead of wiping them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Drop base_reverse_dns_types.txt; sortlists.py now reads types from README.md
The .txt file duplicated the README's industry list and introduced
drift risk — twice in the project's history we had to add types to
the .txt only because the README had been updated independently.
Make the README the single source of truth.
- Add `<!-- types-list:start -->` / `<!-- types-list:end -->` HTML
comment markers around the bullet list in
parsedmarc/resources/maps/README.md. Markers don't render in
GitHub's preview.
- New `load_types_from_readme()` in sortlists.py parses the bullet
items between the markers and returns them. Errors clearly if the
README is missing or the markers are absent.
- Delete base_reverse_dns_types.txt.
- Fix a pre-existing typo in README precedence rule 4: `Web Hosting`
→ `Web Host` (matches the canonical type used in 4,176 map rows).
Smoke test: feeding a row with a bogus type still triggers the
validator (`'NotARealType' is not an allowed value for 'type'`),
confirming the README-derived list flows through identically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sortlists.py: normalize README types-list block in place
Before validating the map, the validator now sorts the
<!-- types-list:start --> / <!-- types-list:end --> block in
README.md alphabetically (case-insensitively), trims leading and
trailing whitespace from each item, and deduplicates case-
insensitively, rewriting the README in place if any of those need
fixing. Errors clearly when two entries differ only by casing
(which would otherwise silently lose one).
Adding a new category is now just inserting a `- New Type` line
anywhere inside the markers — `sortlists.py` will tidy it on the
next run. Same shape as how the validator already normalizes
known_unknown_base_reverse_dns.txt and psl_overrides.txt.
The pure read path is preserved as `load_types_from_readme()` for
callers that don't want a side-effecting rewrite (tests, downstream
tooling).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Stop shipping maintainer scripts; bump to 9.11.2
The exclude list in [tool.hatch.build] was originally meant to keep
maintainer-only batch tooling under parsedmarc/resources/maps/ out
of the wheel and sdist (it lists `find_bad_utf8.py`,
`find_unknown_base_reverse_dns.py`, the renamed-and-removed
`sortmaps.py`). The list never grew when new tools were added, so
`collect_domain_info.py`, `classify_unknown_domains.py`,
`detect_psl_overrides.py`, `detect_rebrands.py`, and `sortlists.py`
all started shipping in distributions despite contributing nothing
to runtime functionality.
Replace the per-file basename list with a single glob pattern:
parsedmarc/resources/maps/[!_]*.py
The leading-`_` exception keeps `__init__.py` shipping (required so
that `importlib.resources.files(parsedmarc.resources.maps)` can
locate the bundled CSV/TXT data files), while excluding any other
.py file under that directory — including future maintainer scripts
that haven't been written yet.
Drop the now-redundant per-file entries from the exclude list:
`find_bad_utf8.py`, `find_unknown_base_reverse_dns.py`, and the
already-removed `sortmaps.py`. The non-.py exclusions stay
(`base_reverse_dns.csv`, `unknown_base_reverse_dns.csv`,
`README.md`, `*.bak`).
Verified with `hatch build`:
- Wheel under parsedmarc/resources/maps/: __init__.py + 3 data files
(CSV/TXTs), no maintainer .py
- sdist matches
- Clean-venv install of the built wheel loads 298 PSL overrides and
`get_base_domain('host01.netlify.app')` returns `netlify.app`
Bump to 9.11.2 since this changes shipped artifacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* collect_domain_info.py: opt-in DuckDuckGo search fallback for bot-blocked rows
A meaningful share of KU domains return a Cloudflare / DDoS-Guard / "Are
you a robot?" / px-captcha interstitial instead of real homepage content
— even after the curl-style relaxed-TLS fallback runs. For those rows we
have neither homepage signal nor (often) a usable as_name, and they fall
through to KU even though the operator is a real (often well-known)
business that the classifier could trivially handle if it could just see
the page.
Added an opt-in `--use-search-fallback` flag that asks DuckDuckGo for
`site:<domain>` when the homepage fetch returned a bot-block / parking /
empty result, and uses the top result's title and description (only if
the result host belongs to the input domain — anti-SEO-spam guard).
Mechanism
- New optional `ddgs` dependency, listed under the `[build]` extras.
`from ddgs import DDGS` is wrapped in a try/except — the script runs
without ddgs installed as long as `--use-search-fallback` isn't
passed; the flag check exits with a helpful install message
otherwise.
- `_SEARCH_FALLBACK_TRIGGER_RE` — title/description patterns that look
like a bot-block / WAF interstitial / parked / placeholder. Triggers
the fallback. Same shape as the classifier's TITLE_NOISE_RE /
PARKED_PAGE_RE; the search fallback is the recovery path for
exactly the rows that filter excludes.
- `_looks_bot_blocked()` — combined check: trigger regex matches OR
title and description are both empty (typical of WAF interstitials
that strip <title>/<meta> entirely).
- `_hosts_match()` — same-domain SEO-spam guard. A search result is
accepted only when its host is exactly the input domain or a
subdomain of it. Third-party SEO-spam pages that scraped the domain
name are silently skipped.
- `_search_fallback_fetch()` — runs `site:<domain>` through DDG, walks
results in rank order, returns the first one whose host passes the
guard. Returns empty if no result matches (caller leaves the row's
homepage data alone in that case).
- `_collect_one()` now takes a `use_search_fallback` flag, calls the
fallback after the homepage fetch when the homepage looks
bot-blocked, and writes `title_source = "homepage"` or
`"search"` so reviewers can audit which rows came from where.
- New `title_source` column in the TSV.
Smoke test
Test set: bbc.com (real homepage, no fallback expected) plus 5 known
Cloudflare-walled rows (1800contacts.com, americaneagle.com,
broadwaytechnology.com, health.gov.il, mfa.gov.il).
Result: bbc.com classified via homepage; the other 5 all recovered
title + description via search and got `title_source=search`. The
same-domain guard validated independently — for broadwaytechnology.com
the guard correctly rejects bloomberg.com and accepts
support.broadwaytechnology.com (broadway was acquired by Bloomberg, but
the search fallback returns the broadway-domain snippet, not the
parent's bloomberg.com product page).
Caveats codified in AGENTS.md
- Search snippets are still untrusted text (data-not-instructions rule
applies the same way it does to homepage HTML).
- DDG's index can lag a homepage rebrand by months — when a row
classified via `title_source=search` disagrees with a fresh manual
fetch, prefer the manual verification. The fallback is a recovery
aid, not a tiebreaker against fresh content.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* collect/classify: link-following + alias map rows for placeholder DDG titles
When the search fallback ran on the original 6-domain smoke set, two of
the recovered titles were essentially placeholder pointers carrying no
classifier signal — DDG returned `Link to fcs.health.gov.il` for one
input and a bare `yangon.mfa.gov.il` for another. Those snippets are
DDG's way of saying "I have an indexed subdomain but no real abstract
to give you", and feeding them to the regex classifier produces no
better signal than the parking-page result we were already trying to
recover from.
This commit teaches the collector to recognize both placeholder shapes,
follow the pointer to the target hostname, and use *that* hostname's
real content for the row. The classifier then emits the original input
and the link target as **two map rows under the same (name, type)** so
both keys are looked up against future DMARC reports.
collect_domain_info.py
- New `_LINK_TO_TITLE_RE` / `_BARE_HOSTNAME_RE` and an
`_extract_link_target` helper that returns the target hostname when
the search title is `Link to <hostname>` or a bare hostname, "" when
the title carries real content.
- After the search-fallback path, if the title looks like a pointer
and the target differs from the input, `_fetch_homepage(target)` is
called once. When the target's fetch returns real (non-bot-blocked)
content, the row's title / description / final_url / rebrand_signal
/ external_links are replaced with the target's, and `title_source`
becomes `search→<target>` so reviewers can audit the path.
- New `link_target_domain` column records the followed target whether
or not its fetch succeeded.
classify_unknown_domains.py
- When a row's `link_target_domain` is set and differs from the input
domain, the classifier emits a second map row for the target with
the same `(name, type)`. The original input is the "og" domain; the
target is what DDG pointed us at — both end up in the map as
aliases. Same handling applies on the ambiguous-bucket path so a
single human adjudication covers both.
Smoke test on the original 6-domain set:
bbc.com homepage → BBC Home – Breaking News, …
1800contacts.com search → 1800contacts
health.gov.il search → Homepage – COVID Information Center
of the Israel Ministry of Health
americaneagle.com search → Americaneagle.com | Web Design …
broadwaytechnology.com search → Bloomberg Completes Acquisition of …
mfa.gov.il search→yangon.mfa.gov.il
→ Home | Ministry of Foreign Affairs
link_target_domain=yangon.mfa.gov.il
The mfa.gov.il row triggered the new path: DDG returned `yangon.mfa.gov.il`
as the title, the collector followed it, the target's homepage gave us
"Home | Ministry of Foreign Affairs", and the classifier emitted both
`mfa.gov.il, Ministry of foreign affairs, Government` and
`yangon.mfa.gov.il, Ministry of foreign affairs, Government`.
AGENTS.md updated with the link-following / alias rules under the
search-fallback subsection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Run --use-search-fallback against 10,544 bot-blocked KU rows; +473 promotions
Also expands the search-fallback trigger regex to recognize self-signed
TLS interception (firewall block via cert) and a wider class of
local-firewall block-page strings.
Mechanics
1. Identified 10,544 KU rows from the 34,647-row prior TSV that looked
bot-blocked (via the new `_looks_bot_blocked` detector).
2. Ran `collect_domain_info.py --use-search-fallback` against just
those rows. Throughput was ~3.4 rows/sec at 32 workers / 3s HTTP
timeout / 5s WHOIS timeout. ~50 min wall time.
3. Audited the resulting TSV and discovered 2,078 rows whose homepage
fetch had silently returned a corporate firewall's block page
(Fortinet "Web Filter Violation" being the most common, 1,419 of
them). The original `_SEARCH_FALLBACK_TRIGGER_RE` didn't recognize
those strings, so search-fallback wasn't firing — the firewall's
block-page text was being fed to the classifier as if it were the
operator's homepage. Almost no false promotions resulted (block-page
text doesn't match industry detectors), but the rows weren't
recovering either.
4. Expanded the trigger regex to catch web-filter block pages, then
re-fetched just the 2,078 affected rows.
5. Final classifier pass: 474 unambiguous map adds, 41 ambiguous, 1
silently dropped (adult content), 10,066 still in KU.
Self-signed-cert detection
A separate fix lands in this commit: when the primary fetch fails with
an SSL cert verification error matching "self-signed certificate", the
collector skips the verify=False browser fallback. Rationale: TLS-
intercepting firewalls (corporate or personal-network) present their
own self-signed cert specifically when blocking. The verify=False
fallback would happily retrieve the firewall's block page, which then
poisons the row's title/description. Skipping that path leaves the
row's metadata empty so search-fallback can recover real content.
Other cert errors (hostname mismatch, weak DH, legacy renegotiation)
keep the existing fallback path because they're typically real
operators with misconfigured TLS rather than firewall interception.
Numbers
Map: 37,640 → 38,114 (+474)
KU: 32,324 → 31,886 (−438)
Disjoint check: 0 shared keys
Unknown CSV: regenerated, just the header
Type distribution of the 474 promotions
162 ISP 17 MSP 4 MSSP / Marketing
72 Web Host 16 Technology 4 Beauty / Agriculture
41 Finance 14 Healthcare 3 IaaS / Science / Legal
19 Government 11 Travel 2 Search / Religion / SaaS
10 Logistics 8 Manufacturing 2 Email Sec / Email Provider
9 Education / Retail 8 News 2 Entertainment
7 Utilities / Phys Sec 6 Real Estate 1 Auto / Staff / PaaS
6 Food / Consulting / Industrial / Conglomerate / Nonprofit
Most of the gains are network operators (162 ISPs, 72 Web Hosts) —
the population that's most likely to be Cloudflare-walled or DDoS-
Guard-walled at the homepage layer but show up clearly in DDG
abstracts.
Smoke audit on a 30-row random sample of map adds: 28 plausible, 2
borderline (`es.graphicpkg.com → Food` could also be Industrial since
Graphic Packaging makes packaging *for* the food industry, but the
vertically-specialized rule applies; `annuairesante.ameli.fr` →
Finance via French health-insurance vocabulary, defensible). The 41
ambiguous rows stay in KU per the established workflow — they need
the same one-row-at-a-time human triage as PR #766 used.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Search-fallback batch (partial; outage-truncated): +226 promotions
Hotspot-bypass collector run was interrupted ~6,300/10,107 in when
the hotspot lost connectivity and the machine reverted to the
firewalled connection. Stopping here to commit what was
unambiguously classifiable; the remaining ~3,800 candidates
(plus any rows whose homepage fetch was tainted by the firewall
fallback during the transition) will be re-collected in a fresh
run after network stability is restored.
Promotions in this batch:
- 219 auto-classified by the regex classifier on the partial TSV
- 17 ambiguous rows resolved per LLM auto-resolution rules + user
manual review
- 5 KU rows the user adjudicated explicitly (Bielsko-Biała,
Douala-IX, Ekol Logistics, ICB, Marcus Corporation)
- 13 from earlier triage worklist with brands assigned
- Net 226 net-new map entries after dedupe, alias-leak filtering
(3 link-target subdomains dropped where the parent base was
already in the adds), full-IP privacy filtering (2 dropped),
and ~30 targeted brand/category cleanups for rows where the
search-fallback snippet had picked up a wrong page or the title
contained registrant cruft / corporate-suffix leaks.
AGENTS.md updates:
- Codifies the "LLM auto-resolution of high-confidence ambiguous
rows" workflow with R1-R5 high-confidence rules, low-confidence
surface-to-human criteria, and the one-line auto-decision output
format for reviewer overrule.
- Adds 7 triage lessons learned during this batch's bot-blocked-KU
review (Polish/IT/ES/GR/RO city domains, "Sports Club" venues,
vertically-specialized investment firms, sub-page fetch FPs,
Telecom-suffix brand pinning, Hospital/Health-System suffix,
IXP -ix brand pinning).
Map and KU files are disjoint after this commit. unknown_base_reverse_dns.csv
is empty (header-only) since every base_reverse_dns input is now
either mapped or in KU.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Search-fallback hotspot batch: +213 promotions
Fresh hotspot run on the 9,881 still-bot-blocked KU candidates left
after the prior outage-truncated batch. Classifier: 202 auto + 31
ambiguous (14 LLM auto-resolved per the R1-R5 high-confidence rules,
17 surfaced for interactive review) + 9,665 still KU + 1 dropped.
Net 213 net-new map entries after dedupe, alias-leak filtering
(13 link-target subdomains dropped where the parent base was
already in the map or in this batch's adds), 1 full-IP privacy
filter, 2 user-DROPs (1 alias of an as-numbered domain, 1 KU
because the only signal was a cross-vertical client list), and
~8 targeted brand cleanups for rows where the search snippet had
left a registrant-leak or domain-as-name placeholder.
LLM auto-resolutions (R1-R5):
africell.ao ISP wi-tribe.pk ISP
ags.school.nz Education vwfs.com.au Finance
allaria.com.ar Finance wanxp.com ISP
asturias.org Government varendraisp.com ISP
bdo.com.ph Finance titansi.com.my IaaS
bikada.kz ISP redeyenetworks.com MSSP
informatiq.org ISP plusinfo.ru ISP
User-decided rows:
admincomp.com Consulting korisp.com Web Host
anrb.ru Science linkexplorer.net.br ISP
arpc.ir Industrial novatech.bg MSP
as63031.net Consulting reliable-nets.com ISP
aviti.net Web Host satortech.com MSP
binaryelements.com.au MSP skyworld.co.ke Finance
juni.net.br ISP telegroup-ltd.com Technology
west-webworld.fr Technology
User KU/drops:
itatec.com.py KU (cross-vertical client list, no operator signal)
ns2.as63031.net DROP (alias of as63031.net)
AGENTS.md addition: codifies the "Web Host vs Email Provider —
bundled email-hosting is still Web Host" rule. Same shape as the
existing CCaaS/CPaaS-vs-ISP and MSP-vs-MSSP rules: classify by the
operator's primary product, not by every feature in their bundle.
Prompted by the korisp.com triage during this batch.
Map and KU files are disjoint after this commit. unknown_base_reverse_dns.csv
remains header-only (every base_reverse_dns input is now mapped or in KU).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Reclassify KU pool: 2,248 promotions; surface 78 ambiguous rows for review
Re-fetched homepage / WHOIS / DNS for all 34,647 domains in
known_unknown_base_reverse_dns.txt via collect_domain_info.py and re-ran the
classifier. The classifier itself was extended in several directions while
auditing the unclassified pool — the changes are listed below.
Numbers
- 2,248 KU rows promoted to base_reverse_dns_map.csv (unambiguous matches).
- 78 rows surfaced as ambiguous (two or more distinct detector categories
fired) — these are NOT auto-promoted; they need human adjudication.
- 32,399 rows remain in KU (genuinely no signal — most have privacy-only
WHOIS, parked / blocked / Cloudflare-walled homepages, or empty MMDB
enrichment).
- Disjoint invariant verified: comm -12 of map keys and KU prints nothing.
- Unknown-list regenerated via find_unknown_base_reverse_dns.py.
Classifier changes (classify_unknown_domains.py)
1. Three output buckets via new --ambiguous-out flag. Per-row outcome is now
one of: map (auto-promote), ambiguous (worklist for human review), or
ku (no signal). When ≥2 distinct detector categories fire on a row, the
classifier picks a primary in precedence order but does NOT auto-promote
— instead it writes the row to the ambiguous TSV with the alternatives
listed. Rationale: the operator-typology question ("is this a SaaS
company or an Energy company?") is a judgment call the classifier
shouldn't make on its own.
2. Plural-matching fix: outer `\b` boundary changed to `s?\b` across all 46
detectors so `dedicated server` matches `dedicated servers`,
`law firm` matches `law firms`, etc. This was silently dropping the
majority of English-text matches.
3. TLD-only signal classification: bare-TLD rows (gov.kh / ac.id / mil.bd /
.jus.br etc.) now classify even when title/desc/as_name are all empty.
Previously short-circuited at "need some signal".
4. TLD lists massively expanded:
- Education: ~85 TLDs (every gov-restricted edu / ac suffix worldwide)
- Government: ~110 TLDs incl. judicial branch (.jus.br) and legislative
(.leg.br); covers Eastern Europe, MENA, SE Asia, Africa, Caribbean,
Pacific
- Military: ~45 .mil.* suffixes
- Plus US K-12 regex (.k12.<state>.us)
5. New concrete-vocabulary patterns added based on KU-pool audit:
- cybersecurity / cyber security for business → MSSP
- autonomous system / asn owner / network operator / peering exchange
/ IXP → ISP
- ICANN registrar / domain registrar / domain name platform / CDN /
WAF / anti-DDoS → Web Host
- BPM platform / CXM / CCaaS / CPaaS / contact center platform /
compliance software → SaaS
- katılım bankası / pensioen en verzekeringen / empréstimo consignado
/ credit (scores|reports|cards|comparison|bureau) /
stock and commodity market → Finance
- aeroportos de / passagem de ônibus / bilişim şirketi / havacılık →
Travel & Tech variants
- acciaio inossidabile / laminati piani → Industrial
- Russian football-club declension forms (футбольного клуба, etc.)
- tv channel / movie streaming / video streaming platform →
Entertainment
- genetic sequencing / next-generation sequencing /
clinical diagnostic → Healthcare
- punto vendita → Italian Retail
- electrolyser / electrolyzer / green hydrogen → Energy
6. Mojibake table extended for Western European compounds: ã/â/ê/î/ô
(Portuguese ã, French/PT â/ê/ô) plus uppercase variants.
Bug fixes from cross-language collisions
The audit pass exposed three short tokens that meant one thing in the
language they were added for and something completely different in another
language the classifier also targets:
- `por` (added as Luxembourgish for "parish" → Religion). Also the Spanish
and Portuguese preposition "for / by", which appears on roughly every
Spanish-language page. Was producing ~34 Religion false positives on
Mexican ISPs, Brazilian utilities, etc.
- `pura` (added as Indonesian/Sundanese/Balinese for "Hindu temple" →
Religion). Also the feminine of "pure" in Portuguese / Spanish / Italian,
and a frequent brand-name fragment ("Pura Energia", "Angkasa Pura").
Was misclassifying Brazilian electric utilities and Indonesian aviation
services.
- bare `broker` (added as Luxembourgish for Finance). Matched any English
text containing "broker" / "brokers" — including Cushman & Wakefield's
"real estate brokers" line, which forced the row into Finance instead
of Real Estate.
All three removed; AGENTS.md now codifies the rule.
AGENTS.md additions
- "Three output buckets" subsection: documents map / ambiguous / ku output
and how PRs should call out ambiguous review counts.
- "No taglines / slogans" rule: marketing copy ("we make it easy",
"smarter decisions") doesn't belong in any detector.
- "No ambiguous signals" rule: cross-category bare words (gazette / academy
/ society / club / studio) are forbidden as classifier keywords; use the
pinning compound instead. Same rule applies in every language.
- "Cross-language grammar / lexical overlap" rule: short tokens that mean X
in language A often mean a function word / adjective / brand fragment in
language B. Cites the por / pura / broker incidents.
- "Classify by what the operator literally provides" rule: clusters by
acronym suffix (UCaaS / CCaaS / CPaaS) tempt mis-grouping; CCaaS is SaaS
not ISP, etc. Includes the root-cause analysis of the
contact-center-as-ISP mistake.
- "Genuinely-ambiguous-between-two-types" rule: phrases like
"energy management software" that fit equally on a SaaS startup, an
Industrial conglomerate, and a consultancy belong in NO detector — leave
the row unmapped and rely on more-specific compounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Triage 78 ambiguous rows + new classifier filters and rules
Interactive triage of the 78 rows the v1 classifier surfaced as ambiguous
(two or more distinct categories fired). Net result of this commit, on
top of the v1 promotions already in the branch:
- 74 ambiguous rows promoted to map with a human-adjudicated category
(and 10 of those with a corrected human-cleaned brand vs. the noisy
as_name / title-bleed the v1 classifier captured).
- 1 row dropped silently per the AGENTS.md adult-content rule.
- 3 rows kept in KU (personal projects, parked pages caught by the
classifier mid-triage that we then surface'd-then-confirmed).
Map: 37,566 → 37,640 (+74). KU: 32,399 → 32,324 (−75). Disjoint clean.
Three new classifier filters added during triage as recurring patterns
surfaced — these run before category detectors and short-circuit to KU
or DROP rather than letting the operator-typology detectors fire on
parking-page / personal-page / adult-page text:
1. PARKED_PAGE_RE — Media Temple "automatically generated default server
page", Hostinger Horizons, Apache default, parked-by-registrar pages,
"site has shut down", "has completed its journey". Cloudflare /
DDoS-Guard / "Are you a robot?" interstitials are explicitly NOT
filtered (they leave the TLD-signal path open for gov / edu / mil
sites that are bot-blocked).
2. PERSONAL_PROJECT_RE — "personal BGP project", "personal website and
CV", "homelab", "hobby project", "side project". Hobbyists running
their own ASN aren't commercial operators.
3. ADULT_CONTENT_RE — adult web design / adult-entertainment hosting /
xxx / escort directory etc. Returns a sentinel ("DROP", None) so the
caller drops the domain from both map and KU per the AGENTS.md
content rule.
The classifier API now also writes a fourth output file (--dropped-out)
listing domains the adult-content filter caught, so the caller can
remove them from any tracked list files they currently sit in.
Title-noise list extended to catch: "attention required" / "are you a
robot" / "checking your browser" / "please enable javascript" /
"ddos-guard" / "px-captcha" / "site is not available" / "page is not
available" / "access to this page has been denied". This stops these
strings from bleeding into the brand column when TLD-only classification
fires (the `health.gov.il → "Attention Required!"` shape of bug).
Several cross-language false positives caught during the triage — same
shape as the por / pura / broker incidents the previous commit fixed:
- bare French `e?mailing` matched "Mailing Solutions" (mail-server
infrastructure on a Cisco VAR's product list, not marketing). Required
to start with `e` to keep the email-marketing meaning while losing the
bare-mailing collision.
- Norwegian / Danish bare `avis` (newspaper) matched "Avis Romania" car
rental and any French text saying "avis" (notice/opinion). Replaced
with compound forms (`dagsavis`, `lokalavis`, `morgenavis`, etc.).
- Vietnamese bare `bộ` (ministry) matched "bộ phim" (movie set), "bộ
sưu tập" (collection), and the founding-text references on Vietnam
Eximbank's about page. Replaced with compound forms (`bộ trưởng`, `bộ
tài chính`, `bộ ngoại giao`, etc.).
- Russian bare `провайдер` (provider) matched "хостинг провайдер"
(hosting provider, Web Host) on a Tajikistan domain registrar. Removed
the bare form; only the internet-specific compounds remain.
- Luxembourgish bare `broker` (Finance) matched "real estate brokers"
on Cushman & Wakefield's homepage and any English page mentioning
brokers. Removed the bare form entirely.
- Turkish bare `vakıf` (foundation) matched "Vakıf Katılım Bankası" —
for-profit Islamic-finance bank whose brand uses the word. Replaced
with nonprofit-specific compounds (`yardım vakfı`, `hayır vakfı`,
`kamu yararına vakıf`).
New positive-classification keywords added based on triage gaps:
- MSP rescue path now matches the SMB-IT-shop idiom in Polish
(`usługi IT dla biznesu`, `obsługa informatyczna firm`,
`outsourcing IT`), Spanish (`servicios informáticos para empresas`),
German (`IT-Dienstleister für`, `managed-IT-services`), French
(`infogérance`, `prestataire de services informatiques`), Italian
(`servizi informatici gestiti`, `outsourcing informatico`),
Portuguese (`serviços de TI gerenciados`, `terceirização de TI`),
Dutch (`beheerd-IT`, `IT-beheer`), and Indonesian
(`penyedia solusi IT`, `solusi IT terpadu/berbasis`).
- Finance now matches `accounting firm` / `cpa firm` /
`certified public accountants` / `chartered accountants` /
`tax preparation` / `tax advisory` / `audit firm` plus equivalents in
Spanish, Portuguese, French, German, Italian, and Polish.
- SaaS now matches CCaaS / CPaaS / `contact-center-as-a-service` /
`communications-platform-as-a-service` / `compliance software` /
`regulatory management software` and CCaaS no longer lives in ISP
(carryover from the user-flagged "contact centers are not ISPs"
correction).
AGENTS.md additions:
- "Triage heuristics learned from the 78-row interactive review of
PR #766's ambiguous bucket" subsection codifying every adjudication
rule the user applied during the review:
* pick the main-focus category (first / most-mentioned)
* clients are not operator typology
* vertically-specialized firms take the vertical
* stream-hosting infrastructure is Web Host
* multi-service SMB IT shops are MSP
* VARs are Technology
* CCaaS / CPaaS / UCaaS are SaaS
* gov/edu/mil/jus TLD signal trumps Cloudflare interstitials
* esports tournament organizers are Entertainment
* personal projects / parked pages / adult content go to KU or DROP
* brand quality is its own dimension — capture corrected brand
during triage rather than shipping the noisy as_name
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multilingual detectors previously held English at full breadth (e.g. Healthcare =
hospital + clinic + pharmacy + healthcare + pharmaceutical industry + nursing
home + medical center) while many non-English sections covered the same concept
set with only one or two transliterated words. This left every language other
than English under-detecting against pages that used the operator's natural
compound terms.
Reworked every detector so each language now expresses the same English concept
set in idiomatic compounds — never inventing calques where no natural form
exists. Added ~32 new languages (Macedonian, Belarusian, Azerbaijani, Armenian,
Georgian, Kazakh, Uzbek, Mongolian, Khmer, Burmese, Lao, Nepali, Sinhala,
Amharic, Yoruba, Hausa, Igbo, Zulu, Pashto, Kurdish, Tajik, Kyrgyz, Maltese,
Luxembourgish, Haitian Creole, Frisian, Yiddish, Faroese, Tatar, Javanese,
Sundanese, Cebuano) on top of the existing pool, again applied per-concept
rather than as token presence.
Also added British / American spelling pairs where they diverge (`tire`/`tyre`,
`defense`/`defence`, `center`/`centre`, etc.) and a handful of new English
concepts that had been implicit (`tire shop`, `car parts`, `oil exploration`,
`olympic committee`, ...) — each with its multilingual equivalents in the same
edit.
AGENTS.md: codified the rule under "Maintaining the reverse DNS maps" so future
edits are bound by it: every language section must cover the same concept set
the English section covers, with idiomatic compounds rather than calques, skip
rather than invent when no natural form exists, and any new English keyword
must be added in parallel across the existing language set.
Final shape: 11,777 alternations / 175,556 chars across 45 detectors. Ruff
check + format clean. Module compiles.
Known limitation (pre-existing, unchanged): Python's `re` does not treat
Unicode Mn / Mc combining marks as word characters, so Brahmic-script words
ending in vowel signs / virama won't match the outer `\b…\b`. Affects
pre-existing and new entries equally; fixable later by switching to the
`regex` module.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Commit classify_unknown_domains.py: regex-based multilingual classifier
Promotes the transient `/tmp/classify_b<N>.py` script that grew across
the b5–b13 reverse-DNS-map batches into a tracked tool. The classifier
takes a `collect_domain_info.py` TSV and emits a CSV of map additions
plus a text file of known-unknown additions — the regex baseline that
makes step 4 of the unknown-domain workflow ("classify from the TSV, not
by re-fetching") tractable at scale.
Coverage:
- Detectors for all 44 industry types in the README.
- Concept-translation parity across ~30 languages on the high-volume
detectors (Healthcare, Travel, Government, Retail, Finance, ISP, Web
Host, Manufacturing, Logistics, Real Estate, Automotive, Legal,
Agriculture).
- ~10–20 languages with 1–3 keywords each on the smaller detectors
(Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media,
Defense, IaaS/PaaS/SaaS, Beauty, Print, Publishing, Religion, Science,
Event Planning, Staffing, Email Security/Provider, Marketing,
Construction, Industrial, Utilities, Energy, Government Media,
Physical Security, News, Nonprofit, Entertainment, Technology,
Consulting).
Brand-name selection prefers MMDB `as_name` → page title's first
segment → non-redacted WHOIS registrant → domain-derived fallback, with
a `clean_brand` pass that strips legal-form suffixes (LLC / GmbH / Ltda
/ EIRELI / sp. z o.o. / s.c.a r.l / UAB / etc.) and prefixes (PT, OOO).
When the title has multiple segments, the segment whose simplified form
contains the domain root is preferred — accessmontana.com with as_name
"MONTANA WEST, L.L.C." and title "Internet, Phone & TV Bundles | Access
Montana" maps to "Access Montana", not "Montana West".
A small mojibake fixer normalizes the most common UTF-8-as-Latin-1
re-encodings ("ó" → "ó", etc.) so Spanish/Portuguese/French homepages
that `collect_domain_info.py` mishandled still classify.
The empty HAND dict at the top of the file is an extension point for
batch-specific overrides — e.g. acquisition aliases or brand-name
corrections that don't fit any detector; each `domain → ("Brand",
"Type")` entry wins over the auto-classifier.
Wired into AGENTS.md's "Related utility scripts" section and documented
in `parsedmarc/resources/maps/README.md` alongside the rest of the
maps utilities.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* classify_unknown_domains.py: clarify dual-purpose framing
The classifier serves both lookup paths into base_reverse_dns_map.csv —
the original PTR-side flow (reverse-DNS base domains derived from DMARC
report source IPs) and the MMDB-coverage flow (AS domains lifted from
the bundled IPinfo Lite MMDB). The initial commit's docstring/comments
emphasized the MMDB-coverage flow because that's where the script grew
up across the b5–b13 batches, but it was always equally applicable to
PTR-side domains.
Updates:
- Top docstring rewritten to lead with the dual-purpose framing.
- README.md adds an explicit "useful for either lookup path" paragraph
referencing the original DMARC-report flow and the MMDB-coverage flow.
- AGENTS.md "Related utility scripts" entry updated to mention both
flows.
- Drops a stale "happen to have ASN registrations" aside in the
RETAIL_RE comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-ran the expanded-multilingual classifier (PR #762's classifier with
broader language coverage on Healthcare, Travel, Government, Retail,
Finance, ISP, Manufacturing, Logistics, Real Estate, Automotive, Legal,
Agriculture, plus Finance-via-body-text catching insurance/investment/
asset-management) against every cached TSV from prior batches (b6–b13).
3,171 domains that previously couldn't be auto-classified (and were
therefore added to known_unknown_base_reverse_dns.txt) now match the new
detectors.
These domains are promoted out of KU and into the map under their newly
classified `(name, type)` pairs.
Type distribution of promotions:
Finance 736 Logistics 179 Real Estate 105 Healthcare 68
ISP 323 Retail 159 Education 110 Marketing 66
Manufacturing 207 Technology 142 Consulting 99 Nonprofit 64
Government 136 Travel 123 Utilities 71 Legal 53
+ smaller volumes across ~25 other industry types
ASN-domain coverage of the bundled IPinfo Lite MMDB after these promotions:
- by domain count: 32,254 / 63,993 (50.40%, up from 45.45%)
- by IPv4 weight: 98.45%
Honest scope note: the multilingual classifier achieves "concept parity"
for the top-5 high-volume detectors (Healthcare, Travel, Government,
Retail, Finance) across ~30 languages. Smaller detectors (Photography,
Conglomerate, Sports, Defense, MSSP, IaaS/PaaS/SaaS, etc.) still have
~10-15 languages with 1-3 keywords each. Further per-detector
multilingual parity is a follow-up effort; each subsequent expansion
recovers fewer domains as the long tail of language-specific phrasings
shrinks.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final cleanup pass to clear the remaining MMDB AS-domain queue. Applied an
expanded multilingual classifier covering all 44 README industry types
plus an Energy concept (mapped to Utilities pending a README addition).
Per-detector keyword lists now include Spanish, Portuguese, French,
Italian, German, Dutch, Russian, Polish, Czech, Turkish, Greek, Chinese
(simplified and traditional), Japanese, Korean, Arabic, Hebrew, Hindi,
Vietnamese, Indonesian, and Thai where the concept has a recognizable
local-language equivalent.
- 980 added to base_reverse_dns_map.csv (ISP 193, Education 193, Finance
155, Government 109, Healthcare 93, Web Host 37, MSP 31, Manufacturing
22, Logistics 17, Real Estate 12, Travel 11, Consulting 10, Tech 9,
Nonprofit 9, Legal 9, Food 9, Retail 8, Religion 8, Utilities 7, plus
smaller volumes across 14 more types).
- 1,669 added to known_unknown_base_reverse_dns.txt — the residual
unfetchable / parked / Cloudflare-challenged / non-recognized-content
rows.
ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
- by domain count: 29,083 / 63,993 (45.45%)
- by IPv4 weight: 98.36%
Total since batch 5: ~16,400 map rows + ~17,400 known-unknown rows added
across 9 batches. Remaining unmapped pool size: 0 — every MMDB AS-domain
has now been processed (either classified or recorded in known-unknown).
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Batch 12. Auto-rate dropped to 23% (2330/9998) — significantly lower than
batch 11. The deeper into the long tail, the more candidates fall into
non-classifier-recognized industries (retail, manufacturing, hospitality,
local services) where the ISP/Web Host/MSP regex doesn't fire even though
the page is fetchable.
- 2,330 added to base_reverse_dns_map.csv (ISP 991, Education 295, Finance
290, Government 265, Web Host 229, Healthcare 134, MSP 126).
- 7,667 added to known_unknown_base_reverse_dns.txt.
ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
- by domain count: 27,824 / 63,993 (43.48%, up from 40.27%)
- by IPv4 weight: 98.36%
Same classifier as prior batches (no new code).
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Batch 11. Auto-rate 30.6% — slightly lower than batch 10's 35.6%, consistent
with continuing long-tail descent.
- 3,063 added to base_reverse_dns_map.csv (ISP 1,734, Finance 270, Web Host
254, Education 253, Government 249, MSP 153, Healthcare 150).
- 6,934 added to known_unknown_base_reverse_dns.txt.
ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
- by domain count: 25,773 / 63,993 (40.27%, up from 35.49%)
- by IPv4 weight: 98.32% (up from 98.27%)
Same classifier as prior batches (no new code).
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of the 10K-batch series toward complete coverage. Auto-rate 35.6%
this round, consistent with the long-tail descent.
- 3,556 added to base_reverse_dns_map.csv (ISP 2,410, Web Host 348,
Education 247, Finance 201, Government 124, MSP 121, Healthcare 105).
- 6,442 added to known_unknown_base_reverse_dns.txt.
ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
- by domain count: 22,710 / 63,993 (35.49%, up from 29.93%)
- by IPv4 weight: 98.27% (up from 98.18%)
Same classifier as batches 5-9 (no new code).
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Auto-classification rate jumped back to 50% (2502/4999) from 36.5% in
batch 8 — this slice happens to contain a higher proportion of small ISPs
with conventional homepages, lifting the regex hit rate.
- 2,502 added to base_reverse_dns_map.csv (ISP 2,065, Web Host 133,
Education 96, Finance 67, MSP 60, Government 58, Healthcare 23).
- 2,496 added to known_unknown_base_reverse_dns.txt.
ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
- by domain count: 19,154 / 63,993 (29.93%, up from 26.02%)
- by IPv4 weight: 98.18% (up from 98.09%)
Same classifier as batches 5-8 (no new code).
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>