Commit Graph
182 Commits
Author SHA1 Message Date
github-actions[bot]andseanthegeek c1e7662e9e chore: update IPinfo Lite MMDB (#927)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-10-26 10:06:44 -04:00
github-actions[bot]andseanthegeek 4d3afac0c9 chore: update IPinfo Lite MMDB (#922)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-10-19 11:12:00 -04:00
github-actions[bot]andseanthegeek d6e3ef8393 chore: update IPinfo Lite MMDB (#919)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-10-13 18:04:00 -04:00
github-actions[bot]andseanthegeek af63675aeb chore: update IPinfo Lite MMDB (#918)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-10-05 10:41:55 -04:00
github-actions[bot]andseanthegeek 4a6b467586 chore: update IPinfo Lite MMDB (#917)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-09-30 18:43:11 -04:00
github-actions[bot]andseanthegeek 6648be7d02 chore: update IPinfo Lite MMDB (#914)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-09-21 12:49:16 -04:00
github-actions[bot]andseanthegeek edcfd0063b chore: update IPinfo Lite MMDB (#912)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-09-16 18:23:58 -04:00
Sean WhalenandClaude Opus 5 77045c2df0 chore: address CodeQL code-quality findings (no behavior change) (#900)
- py/empty-except: add a short comment to each of the 10 flagged
  `except <Type>: pass` blocks explaining why swallowing the
  exception is correct there (best-effort cleanup/close, or
  intentional fall-through to the next report format).
- py/unnecessary-lambda: replace `map(lambda x: parse_email_address(x), ...)`
  with `map(parse_email_address, ...)` at the 5 flagged sites in
  parsedmarc/utils.py; parse_email_address takes exactly one
  positional argument, so this is behavior-preserving. Ran
  `ruff format` afterward, which collapsed 3 of the now-shorter
  calls onto single lines.
- py/imprecise-assert: replace `assertTrue(a > b)` / `assertTrue(a >= b)`
  with `assertGreater(a, b)` / `assertGreaterEqual(a, b)` at the 9
  flagged test sites; none carried a custom assertion message.
- .vscode/settings.json: drop three cSpell whitelist entries that are
  pure misspellings ("passsword", "httpasswd", "unparasable") and
  appear nowhere else in the tracked tree, so a recurrence of the typo
  they used to hide (see #888) would be caught again. The correctly
  spelled "htpasswd" and "unparseable" entries are untouched.

Verified clean: ruff check, ruff format --check, pyright (0
errors/warnings), and pytest tests/ (989 tests collected and passing
before and after, GITHUB_ACTIONS=true).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 19:06:34 -04:00
Sean WhalenandClaude Opus 5 95ddc8a323 Fix missing-file error paths falling through in find_unknown_base_reverse_dns.py (#903)
Two `if not os.path.exists(path): print(f"Error: ...")` checks printed
the intended error message but had no sys.exit(1) after it, so execution
fell through into the subsequent open() call on the same missing path
and raised an unhandled FileNotFoundError instead of the clean error
exit the code clearly intended. Sibling duplicate-entry checks in the
same function already did print-then-sys.exit(1); these two now match.

Sites fixed (both in _main()):
- the nested load_list() helper's missing-file check (used for
  known_unknown_base_reverse_dns.txt and psl_overrides.txt)
- the base_reverse_dns_map.csv missing-file check

Both sites now have regression tests in a new TestFindUnknownBaseReverseDNS
class:
- the load_list() site: calling _main() in a temp directory with no
  known_unknown_base_reverse_dns.txt now raises SystemExit(1) instead of
  FileNotFoundError.
- the base_reverse_dns_map.csv site: _load_as_name_index() does its
  external work entirely through maxminddb.open_database(), the actual
  SDK boundary, so that call is mocked to a context manager over an
  empty iterable rather than loading the real ~23MB bundled MMDB or
  mocking an internal helper. Calling _main() in a temp directory with
  no base_reverse_dns_map.csv now raises SystemExit(1) instead of
  FileNotFoundError.

Both tests capture stdout and assert on the specific "Error: ... does
not exist" message, pinning the exit to the intended site rather than
any sys.exit(1) in the function. Cleanup uses two separate addCleanup
calls (rmtree registered before chdir, so LIFO order runs chdir first)
instead of one lambda wrapping both, so rmtree still runs even if
chdir were to raise.

Also fixed an adjacent prose bug a few lines from the second site: the
"is in known_unknown... and base_reverse_dns_map..." error message was
missing a space after "Error:" and was a backslash-continued f-string
that embedded the source's literal indentation in the printed output.
It now prints as a single clean line, consistent with the file's other
error messages.

Logged the user-facing symptom (clean error replaced by a
FileNotFoundError traceback) under CHANGELOG.md's Unreleased/Bug fixes
section, matching the project's precedent of logging maintainer-tooling
fixes (e.g. the sortlists.py entry).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 22:44:15 +00:00
Sean WhalenandClaude Opus 5 4298603713 Fix CodeQL findings: use sys.exit(), remove dead index_date stores (#898)
py/use-of-exit-or-quit (29 sites): replace bare `exit(...)` calls with
`sys.exit(...)` in cli.py (20 sites), sortlists.py (4 sites), and
find_unknown_base_reverse_dns.py (5 sites). `exit` is injected into the
interactive namespace by the `site` module, so it is not guaranteed to
exist under `python -S` or an embedded interpreter — the very error
paths that called it would raise NameError instead of exiting cleanly.
`sys.exit` is always available and was already used elsewhere in these
files. sortlists.py did not import `sys`; the import was added to its
existing stdlib import block. find_unknown_base_reverse_dns.py already
imported `sys` (used at line 71), so only the call sites changed there.
Checked tests/test_cli.py for anything patching `exit` directly — none
do; the SystemExit-based assertions exercise the same behavior for
either spelling, so no test changes were needed.

py/multiple-definition (4 sites): removed the dead `index_date`
computation in save_aggregate_report_to_elasticsearch()
(elastic.py:944-947) and save_aggregate_report_to_opensearch()
(opensearch.py:875-878). Both blocks derived index_date from the
report-level begin_date, but the per-record loop immediately below
unconditionally recomputes index_date from each record's own
begin_date before the only read of the variable (the index name build
near the end of the loop body). Verified by reading both functions
that no code between the outer assignment and the loop's recompute
reads index_date, and that the recompute is unconditional on every
path that reaches the read, before deleting.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 22:44:15 +00:00
github-actions[bot]andseanthegeek de88cd893f chore: update IPinfo Lite MMDB (#896)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-09-08 12:02:41 -04:00
github-actions[bot]andseanthegeek 823b0a1811 chore: update IPinfo Lite MMDB (#894)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-09-01 18:30:07 -04:00
07bca1ad28 Make the output and mailbox integrations optional extras (#888)
* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-08-28 17:33:18 -04:00
github-actions[bot]andseanthegeek d52fca130d chore: update IPinfo Lite MMDB (#881)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-08-24 09:21:30 -04:00
github-actions[bot]andseanthegeek 7acfaa0cea chore: update IPinfo Lite MMDB (#873)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-08-17 14:00:59 -04:00
github-actions[bot]andseanthegeek 5b2dfa7d1a chore: update IPinfo Lite MMDB (#871)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-08-10 09:15:05 -04:00
github-actions[bot]andseanthegeek 6f56b92c44 chore: update IPinfo Lite MMDB (#870)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-08-03 18:47:52 -04:00
github-actions[bot]andseanthegeek 7500d48c9e chore: update IPinfo Lite MMDB (#867)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-27 08:48:13 -04:00
Sean WhalenandClaude Fable 5 0a95a0ceb2 Upgrade ruff to 0.16.0 and pyright to 1.1.411; convert to f-strings (#847)
* Upgrade ruff to 0.16.0 and pyright to 1.1.411; convert to f-strings

ruff 0.16.0 expanded the default lint rule selection well beyond the
long-standing E4/E7/E9/F, so [tool.ruff.lint] now selects the rule set
explicitly: the pre-0.16 defaults plus the modern-type-hint UP rules and
the two f-string rules (UP030/UP032). All 352 UP030/UP032 findings were
auto-fixed; conversions requiring Python 3.12 f-string quote reuse were
conservatively left as .format() by ruff (verified: the whole package and
test suite byte-compile under CPython 3.10.20, the oldest CI version).
Adopting the other newly-default rule families (BLE, SIM, C4, DTZ, I, ...)
is deferred as a deliberate per-family decision.

ruff format with 0.16.0 also now formats Python code fences in Markdown,
which reformatted one block in parsedmarc/resources/maps/AGENTS.md.

ruff check, ruff format --check, pyright (0 errors/warnings), and the
full test suite (775 passed) are green on the new versions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review findings on the f-string conversion

- Rewrite messages that used backslash line continuations inside string
  literals, which embedded the source indentation as literal whitespace
  in the logged/raised text: the duplicate search-error messages in the
  Elasticsearch and OpenSearch outputs, the 'since'-option warning (which
  also implicitly concatenated "24hrs" and "SMTP" with no separator) and
  the IMAP 'since' debug line, and the missing-org_name KeyError message.
- Build the Splunk HEC newline-delimited payloads by appending to a list
  and joining once instead of quadratic string concatenation in a loop.

Declined: switching the webhook output's logger.error to
logger.exception — the single-line ERROR without a traceback is the
deliberate house pattern for batch-resilient sinks, and changing log
verbosity is out of scope for this refactor PR.

ruff check/format, pyright (0 errors/warnings), 775 tests, and a
CPython 3.10 compileall all pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 13:46:36 -04:00
github-actions[bot]andseanthegeek 5dc83613e6 chore: update IPinfo Lite MMDB (#832)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-20 15:15:19 -04:00
Sean WhalenandClaude Sonnet 5 3b53f836a6 Map two domain batches: MMDB ASN-coverage gap and uncategorized-sources export
Batch 1 — MMDB coverage-gap scan (find_unmapped_as_domains.py, default
4,096-IP floor): 132 candidates collected and classified. Manually
audited every auto-classified row rather than trusting the classifier
output directly, since it derives as_name straight from the MMDB and
will auto-classify off that single string even with no reachable
homepage — which fails the two-corroborating-sources bar. Demoted 10
rows to known-unknown on that basis (e.g. pellera.com's ASN registrant
is "Converge Technology Solutions", aerloop.in's is "Aerpace" — no
shared brand token, single source). Corrected several brand-quality
issues before they hit the map: sdtv.com.tw was about to ship as
"SDTV Webmail" (a login-page title) instead of "San Da Cable TV";
liyang.gov.cn was about to ship the hosting network's name instead of
the actual government operator (confirmed via Chinese-language WHOIS).
Also fixed a pre-existing bad entry noticed in passing: backwaves.net
was mapped under a garbled Chinese tagline instead of "Back Waves".

Result: 52 map rows, 82 known-unknown entries.

Batch 2 — plain-text uncategorized-sources export (842 source names via
find_unknown_base_reverse_dns.py -i, mix of raw MMDB as_name strings
and reverse-DNS domains) -> 377 unknown domains -> collected and
classified. This batch surfaced more data-quality traps: a parked
domain-registrar welcome page (serveur-vps.net) and a "Site under
maintenance" title (gou.go.ug, resolved via TLD instead) auto-classified
as if they were live operators; a WHOIS privacy-proxy service name
("GKG.NET Domain Proxy Service Administrator") slipped past the
privacy-org filter for atlasok.com; grohe.org.ru turned out to be a
Russian third-party reseller storefront, not the real Grohe corporation,
so it was held back from the map to avoid misattributing a reseller's
mail to the multinational. Refined the two-source test to distinguish
domain-WHOIS (ties directly to the domain; counts even without a
literal name match, e.g. portotelecom.net.br's WHOIS "Rondotech" backs
its self-described "Porto Telecom" brand) from MMDB as_name
(infrastructure-level signal, needs an actual lexical link, same
caution as IP-WHOIS). Used existing map entries as legitimate second
sources for redirects/aliases (vsys.host -> v-sys.org, metronet.net ->
metronet.com, mentari.net.id -> megahub.id) and cleaned up
megahub.id's own tagline-derived name to "Megahub" while there.

Result: 24 map rows, 353 known-unknown entries.

Also ignore the scratch/ working directory used to stage the
uncategorized-sources export for this pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 19:29:21 -04:00
Sean WhalenandClaude Fable 5 7d87ba18be Ingest uncategorized-sources exports and add MMDB coverage scan with anti-poisoning guards (#829)
* Accept plain-text uncategorized-sources lists in find_unknown_base_reverse_dns.py

Dashboard exports of uncategorized email sources are plain-text lists of
one source name per line — a mix of raw MMDB as_name strings (when the
source IP had no PTR and resolved via the IPinfo Lite MMDB) and base
reverse-DNS domains. The script already translates as_names to their
as_domain and subtracts mapped/known-unknown entries, but only read a
hardcoded source_name-headed CSV.

Add -i/--input and -o/--output flags (defaults preserve current
behavior) and auto-detect the input format from the first line: a
source_name CSV header selects the existing DictReader path, anything
else is read as plain text with each line taken verbatim (never
comma-split, since as_names contain commas) and deduped
case-insensitively. Fix the missing-input error message, which
reported the map path instead of the input path.

Document the new entry point in the maps README and AGENTS.md, and make
explicit in the brand-quality triage rule that map display names must be
human-friendly operator names — never raw as_name strings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add MMDB coverage scan script with anti-poisoning guards

find_unmapped_as_domains.py turns the manual "Checking ASN-domain
coverage of the MMDB" recipe into a maintainer script: walk every IPv4
record in the bundled IPinfo Lite MMDB, aggregate routed footprint per
as_domain, subtract mapped/known-unknown keys, apply PSL folding and
the full-IP privacy filter, and emit domain,ipv4_count,as_name sorted
by footprint for the collector -> classifier pipeline.

Because ASN registration data is self-declared to the RIRs and
as_domain derives from registrant-controlled WHOIS, bulk-categorizing
the MMDB needs poisoning defenses:

- An IPv4-footprint floor (--min-ips, default 4096, a /20) keeps tiny
  self-described ASNs out of the auto-classification queue; dropped
  counts are always printed.
- A brand-collision guard in classify_unknown_domains.py loads the
  existing map (--map) and demotes any single-category candidate whose
  proposed display name matches an existing map name without a lexical
  relationship to that operator's keys into the ambiguous bucket
  (marked name-collision-with-existing-map-entry) for human review.
  HAND overrides bypass the guard. The guard protects the PTR-side
  flow as well as the MMDB-coverage flow.

Verified: scan yields 132 candidates at the default floor (1512
dropped); collector accepts the output directly; a fixture titled as
Comcast under an unrelated domain lands in ambiguous while a
comcast-rooted sibling auto-promotes. Also fix the maps README links
that still pointed at the root AGENTS.md for the classification
workflow after its extraction to maps/AGENTS.md, and correct the
classify_tsv docstring's return signature.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Enhance planning guidance in CLAUDE.md by specifying auto mode activation after user approval

* Codify triage flagging for identified operators with no fitting type

An operator confidently identified from two corroborating sources but
matching none of the README's type values should be flagged during
triage with a proposed new type for the reviewer, not force-fitted and
not silently recorded as known-unknown — KU means "we couldn't
identify this", which would bury completed research. Extends workflow
rule 7 and the LLM low-confidence list in the maps AGENTS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 18:21:47 -04:00
Sean Whalen 64ddb13f18 Clarify introductory guidance in AGENTS.md by removing redundant phrasing 2026-07-16 17:10:15 -04:00
Sean Whalen 51cd3bfa45 Fix file path reference in AGENTS.md for clarity 2026-07-16 17:09:37 -04:00
Sean Whalen 1f1a0f0a0e Refactor AGENTS.md to remove redundant file paths for clarity 2026-07-16 16:54:55 -04:00
Sean Whalen eb7f11798b Remove examples in privacy rule regarding full IPv4 addresses in AGENTS.md 2026-07-16 16:48:31 -04:00
Sean WhalenandClaude Fable 5 0e31dcc0b8 Extract reverse-DNS map guidance to a nested AGENTS.md
Move the "Maintaining the reverse DNS maps" section (two-thirds of the
root AGENTS.md) to parsedmarc/resources/maps/AGENTS.md, next to the
files and tooling it governs. Agents that follow the AGENTS.md spec
read the nearest file in the directory tree; a one-line CLAUDE.md
import alongside it covers Claude Code, which loads subdirectory
context lazily when files under maps/ are first read. A pointer stub
remains in the root file for agents that only read the repo root.

The moved content is verbatim except for one cross-reference that now
points at the root AGENTS.md instead of "earlier in this file".

Exclude AGENTS.md/CLAUDE.md from hatch build targets so the new files
don't ship in the wheel, matching the existing README.md exclusion
(verified with a scratch wheel build).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:42:59 -04:00
github-actions[bot]andseanthegeek 547e6d7113 chore: update IPinfo Lite MMDB (#818)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-13 09:17:41 -04:00
github-actions[bot]andseanthegeek 25638cc3cd chore: update IPinfo Lite MMDB (#807)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-07-06 09:29:52 -04:00
github-actions[bot]andseanthegeek 2547422f40 chore: update IPinfo Lite MMDB (#804)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-29 12:54:42 -04:00
Sean Whalen c423b8dfff Modernize type hints to PEP 585 / PEP 604 syntax (#803) 2026-06-25 15:21:48 -04:00
github-actions[bot]andseanthegeek bc17e6afb2 chore: update IPinfo Lite MMDB (#801)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-23 13:46:35 -04:00
github-actions[bot]andseanthegeek d40447ea91 chore: update IPinfo Lite MMDB (#800)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-15 08:54:43 -04:00
Sean WhalenandClaude Fable 5 eaeea4f53d Make the whole codebase pass pyright cleanly and enforce it in CI (#798)
* Make the whole codebase pass pyright cleanly and enforce it in CI

Fix all 102 pyright (1.1.410, standard mode) errors across the library,
tests, and maps scripts, then pin and enforce the zero-errors bar:

- postgres.py: make the optional psycopg import TYPE_CHECKING-aware so
  the module is properly typed while keeping the runtime install-hint
  fallback; import psycopg.types.json explicitly as psycopg_json (the
  old psycopg_types.json attribute access only worked because psycopg
  imports the submodule eagerly); have _connect()/_ensure_connected()
  return the live connection so save methods use a non-Optional local;
  type the DDL list as list[LiteralString] to match psycopg's execute()
  overloads.
- kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error
  fallback statically via TYPE_CHECKING (kafka-python 3.0 removed
  NoBrokersAvailable), which also fixes _BootstrapError's import
  resolution in tests.
- syslog.py: go through getattr/setattr for SysLogHandler.socket
  (absent from typeshed); type the save_* methods with the report
  TypedDicts (single or list, matching cli.py call sites — gelf.py gets
  the same signatures); raise ValueError when retry_attempts < 1
  instead of falling through and registering a None handler (bug fix,
  with a regression test and a CHANGELOG entry).
- elastic.py / opensearch.py: human_result params are Optional[str].
- maps scripts: sort_csv declared a return type but never returned
  (now -> None); seen_sort_field_values was possibly unbound;
  convert_to_utf8's src_encoding is Optional[str].
- tests: cast sample-report dict helpers to their TypedDicts; mark
  deliberate wrong-type calls with targeted pyright ignores; add
  narrowing asserts for Optional results; access the mocked
  KafkaProducer through a cast helper; match the mailsuite
  fetch_message base signature (**kwargs); patch the renamed
  parsedmarc.postgres.psycopg_json in test_postgres's setUpModule.

Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc,
tests, docs; standard mode), pyright==1.1.410 pinned in the [build]
extra (pinned exactly so a new pyright release can't break CI without a
code change), and a "Check types" step in the lint CI job — which now
also runs ruff format --check and installs the [postgresql] extra so
the optional psycopg import resolves. Documented in AGENTS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Set session headers via update() instead of replacing the dict

requests 2.34 ships inline type annotations, and Session.headers is a
CaseInsensitiveDict[str] — assigning a plain dict fails pyright there
(the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid
it). headers.update() is correctly typed against both versions, and is
the documented requests idiom: it overrides User-Agent and the
client-specific headers while keeping the session's defaults
(Accept-Encoding, Connection) instead of wiping them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 21:33:01 -04:00
github-actions[bot]andseanthegeek 8ec3cc9ce8 chore: update IPinfo Lite MMDB (#791)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-08 17:02:09 -04:00
github-actions[bot]andseanthegeek ec0a313669 chore: update IPinfo Lite MMDB (#788)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-06-01 13:26:09 -04:00
github-actions[bot]andseanthegeek f65eebc4d8 chore: update IPinfo Lite MMDB (#787)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-05-26 14:37:59 -04:00
github-actions[bot]andseanthegeek a6ea169df5 chore: update IPinfo Lite MMDB (#771)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-05-18 09:41:38 -04:00
github-actions[bot]andseanthegeek 1fc1134f77 chore: update IPinfo Lite MMDB (#769)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
2026-05-11 15:05:33 -04:00
ff6f75d740 Map-data build hygiene: README single source of truth, drop maintainer scripts from wheel (9.11.2) (#768)
* Drop base_reverse_dns_types.txt; sortlists.py now reads types from README.md

The .txt file duplicated the README's industry list and introduced
drift risk — twice in the project's history we had to add types to
the .txt only because the README had been updated independently.
Make the README the single source of truth.

- Add `<!-- types-list:start -->` / `<!-- types-list:end -->` HTML
  comment markers around the bullet list in
  parsedmarc/resources/maps/README.md. Markers don't render in
  GitHub's preview.
- New `load_types_from_readme()` in sortlists.py parses the bullet
  items between the markers and returns them. Errors clearly if the
  README is missing or the markers are absent.
- Delete base_reverse_dns_types.txt.
- Fix a pre-existing typo in README precedence rule 4: `Web Hosting`
  → `Web Host` (matches the canonical type used in 4,176 map rows).

Smoke test: feeding a row with a bogus type still triggers the
validator (`'NotARealType' is not an allowed value for 'type'`),
confirming the README-derived list flows through identically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* sortlists.py: normalize README types-list block in place

Before validating the map, the validator now sorts the
<!-- types-list:start --> / <!-- types-list:end --> block in
README.md alphabetically (case-insensitively), trims leading and
trailing whitespace from each item, and deduplicates case-
insensitively, rewriting the README in place if any of those need
fixing. Errors clearly when two entries differ only by casing
(which would otherwise silently lose one).

Adding a new category is now just inserting a `- New Type` line
anywhere inside the markers — `sortlists.py` will tidy it on the
next run. Same shape as how the validator already normalizes
known_unknown_base_reverse_dns.txt and psl_overrides.txt.

The pure read path is preserved as `load_types_from_readme()` for
callers that don't want a side-effecting rewrite (tests, downstream
tooling).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Stop shipping maintainer scripts; bump to 9.11.2

The exclude list in [tool.hatch.build] was originally meant to keep
maintainer-only batch tooling under parsedmarc/resources/maps/ out
of the wheel and sdist (it lists `find_bad_utf8.py`,
`find_unknown_base_reverse_dns.py`, the renamed-and-removed
`sortmaps.py`). The list never grew when new tools were added, so
`collect_domain_info.py`, `classify_unknown_domains.py`,
`detect_psl_overrides.py`, `detect_rebrands.py`, and `sortlists.py`
all started shipping in distributions despite contributing nothing
to runtime functionality.

Replace the per-file basename list with a single glob pattern:

    parsedmarc/resources/maps/[!_]*.py

The leading-`_` exception keeps `__init__.py` shipping (required so
that `importlib.resources.files(parsedmarc.resources.maps)` can
locate the bundled CSV/TXT data files), while excluding any other
.py file under that directory — including future maintainer scripts
that haven't been written yet.

Drop the now-redundant per-file entries from the exclude list:
`find_bad_utf8.py`, `find_unknown_base_reverse_dns.py`, and the
already-removed `sortmaps.py`. The non-.py exclusions stay
(`base_reverse_dns.csv`, `unknown_base_reverse_dns.csv`,
`README.md`, `*.bak`).

Verified with `hatch build`:
- Wheel under parsedmarc/resources/maps/: __init__.py + 3 data files
  (CSV/TXTs), no maintainer .py
- sdist matches
- Clean-venv install of the built wheel loads 298 PSL overrides and
  `get_base_domain('host01.netlify.app')` returns `netlify.app`

Bump to 9.11.2 since this changes shipped artifacts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:36:48 -04:00
053195581b collect_domain_info.py: opt-in DuckDuckGo search fallback for bot-blocked rows (#767)
* collect_domain_info.py: opt-in DuckDuckGo search fallback for bot-blocked rows

A meaningful share of KU domains return a Cloudflare / DDoS-Guard / "Are
you a robot?" / px-captcha interstitial instead of real homepage content
— even after the curl-style relaxed-TLS fallback runs. For those rows we
have neither homepage signal nor (often) a usable as_name, and they fall
through to KU even though the operator is a real (often well-known)
business that the classifier could trivially handle if it could just see
the page.

Added an opt-in `--use-search-fallback` flag that asks DuckDuckGo for
`site:<domain>` when the homepage fetch returned a bot-block / parking /
empty result, and uses the top result's title and description (only if
the result host belongs to the input domain — anti-SEO-spam guard).

Mechanism

- New optional `ddgs` dependency, listed under the `[build]` extras.
  `from ddgs import DDGS` is wrapped in a try/except — the script runs
  without ddgs installed as long as `--use-search-fallback` isn't
  passed; the flag check exits with a helpful install message
  otherwise.
- `_SEARCH_FALLBACK_TRIGGER_RE` — title/description patterns that look
  like a bot-block / WAF interstitial / parked / placeholder. Triggers
  the fallback. Same shape as the classifier's TITLE_NOISE_RE /
  PARKED_PAGE_RE; the search fallback is the recovery path for
  exactly the rows that filter excludes.
- `_looks_bot_blocked()` — combined check: trigger regex matches OR
  title and description are both empty (typical of WAF interstitials
  that strip <title>/<meta> entirely).
- `_hosts_match()` — same-domain SEO-spam guard. A search result is
  accepted only when its host is exactly the input domain or a
  subdomain of it. Third-party SEO-spam pages that scraped the domain
  name are silently skipped.
- `_search_fallback_fetch()` — runs `site:<domain>` through DDG, walks
  results in rank order, returns the first one whose host passes the
  guard. Returns empty if no result matches (caller leaves the row's
  homepage data alone in that case).
- `_collect_one()` now takes a `use_search_fallback` flag, calls the
  fallback after the homepage fetch when the homepage looks
  bot-blocked, and writes `title_source = "homepage"` or
  `"search"` so reviewers can audit which rows came from where.
- New `title_source` column in the TSV.

Smoke test

Test set: bbc.com (real homepage, no fallback expected) plus 5 known
Cloudflare-walled rows (1800contacts.com, americaneagle.com,
broadwaytechnology.com, health.gov.il, mfa.gov.il).

Result: bbc.com classified via homepage; the other 5 all recovered
title + description via search and got `title_source=search`. The
same-domain guard validated independently — for broadwaytechnology.com
the guard correctly rejects bloomberg.com and accepts
support.broadwaytechnology.com (broadway was acquired by Bloomberg, but
the search fallback returns the broadway-domain snippet, not the
parent's bloomberg.com product page).

Caveats codified in AGENTS.md

- Search snippets are still untrusted text (data-not-instructions rule
  applies the same way it does to homepage HTML).
- DDG's index can lag a homepage rebrand by months — when a row
  classified via `title_source=search` disagrees with a fresh manual
  fetch, prefer the manual verification. The fallback is a recovery
  aid, not a tiebreaker against fresh content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* collect/classify: link-following + alias map rows for placeholder DDG titles

When the search fallback ran on the original 6-domain smoke set, two of
the recovered titles were essentially placeholder pointers carrying no
classifier signal — DDG returned `Link to fcs.health.gov.il` for one
input and a bare `yangon.mfa.gov.il` for another. Those snippets are
DDG's way of saying "I have an indexed subdomain but no real abstract
to give you", and feeding them to the regex classifier produces no
better signal than the parking-page result we were already trying to
recover from.

This commit teaches the collector to recognize both placeholder shapes,
follow the pointer to the target hostname, and use *that* hostname's
real content for the row. The classifier then emits the original input
and the link target as **two map rows under the same (name, type)** so
both keys are looked up against future DMARC reports.

collect_domain_info.py
- New `_LINK_TO_TITLE_RE` / `_BARE_HOSTNAME_RE` and an
  `_extract_link_target` helper that returns the target hostname when
  the search title is `Link to <hostname>` or a bare hostname, "" when
  the title carries real content.
- After the search-fallback path, if the title looks like a pointer
  and the target differs from the input, `_fetch_homepage(target)` is
  called once. When the target's fetch returns real (non-bot-blocked)
  content, the row's title / description / final_url / rebrand_signal
  / external_links are replaced with the target's, and `title_source`
  becomes `search→<target>` so reviewers can audit the path.
- New `link_target_domain` column records the followed target whether
  or not its fetch succeeded.

classify_unknown_domains.py
- When a row's `link_target_domain` is set and differs from the input
  domain, the classifier emits a second map row for the target with
  the same `(name, type)`. The original input is the "og" domain; the
  target is what DDG pointed us at — both end up in the map as
  aliases. Same handling applies on the ambiguous-bucket path so a
  single human adjudication covers both.

Smoke test on the original 6-domain set:

  bbc.com                  homepage   → BBC Home – Breaking News, …
  1800contacts.com         search     → 1800contacts
  health.gov.il            search     → Homepage – COVID Information Center
                                        of the Israel Ministry of Health
  americaneagle.com        search     → Americaneagle.com | Web Design …
  broadwaytechnology.com   search     → Bloomberg Completes Acquisition of …
  mfa.gov.il               search→yangon.mfa.gov.il
                                      → Home | Ministry of Foreign Affairs
                                        link_target_domain=yangon.mfa.gov.il

The mfa.gov.il row triggered the new path: DDG returned `yangon.mfa.gov.il`
as the title, the collector followed it, the target's homepage gave us
"Home | Ministry of Foreign Affairs", and the classifier emitted both
`mfa.gov.il, Ministry of foreign affairs, Government` and
`yangon.mfa.gov.il, Ministry of foreign affairs, Government`.

AGENTS.md updated with the link-following / alias rules under the
search-fallback subsection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Run --use-search-fallback against 10,544 bot-blocked KU rows; +473 promotions

Also expands the search-fallback trigger regex to recognize self-signed
TLS interception (firewall block via cert) and a wider class of
local-firewall block-page strings.

Mechanics

1. Identified 10,544 KU rows from the 34,647-row prior TSV that looked
   bot-blocked (via the new `_looks_bot_blocked` detector).
2. Ran `collect_domain_info.py --use-search-fallback` against just
   those rows. Throughput was ~3.4 rows/sec at 32 workers / 3s HTTP
   timeout / 5s WHOIS timeout. ~50 min wall time.
3. Audited the resulting TSV and discovered 2,078 rows whose homepage
   fetch had silently returned a corporate firewall's block page
   (Fortinet "Web Filter Violation" being the most common, 1,419 of
   them). The original `_SEARCH_FALLBACK_TRIGGER_RE` didn't recognize
   those strings, so search-fallback wasn't firing — the firewall's
   block-page text was being fed to the classifier as if it were the
   operator's homepage. Almost no false promotions resulted (block-page
   text doesn't match industry detectors), but the rows weren't
   recovering either.
4. Expanded the trigger regex to catch web-filter block pages, then
   re-fetched just the 2,078 affected rows.
5. Final classifier pass: 474 unambiguous map adds, 41 ambiguous, 1
   silently dropped (adult content), 10,066 still in KU.

Self-signed-cert detection

A separate fix lands in this commit: when the primary fetch fails with
an SSL cert verification error matching "self-signed certificate", the
collector skips the verify=False browser fallback. Rationale: TLS-
intercepting firewalls (corporate or personal-network) present their
own self-signed cert specifically when blocking. The verify=False
fallback would happily retrieve the firewall's block page, which then
poisons the row's title/description. Skipping that path leaves the
row's metadata empty so search-fallback can recover real content.
Other cert errors (hostname mismatch, weak DH, legacy renegotiation)
keep the existing fallback path because they're typically real
operators with misconfigured TLS rather than firewall interception.

Numbers

  Map:  37,640 → 38,114 (+474)
  KU:   32,324 → 31,886 (−438)

  Disjoint check: 0 shared keys
  Unknown CSV: regenerated, just the header

Type distribution of the 474 promotions

  162  ISP                 17  MSP                 4   MSSP / Marketing
   72  Web Host             16  Technology          4   Beauty / Agriculture
   41  Finance              14  Healthcare          3   IaaS / Science / Legal
   19  Government           11  Travel              2   Search / Religion / SaaS
   10  Logistics            8   Manufacturing       2   Email Sec / Email Provider
    9  Education / Retail   8   News                2   Entertainment
    7  Utilities / Phys Sec 6   Real Estate         1   Auto / Staff / PaaS
                            6   Food / Consulting / Industrial / Conglomerate / Nonprofit

Most of the gains are network operators (162 ISPs, 72 Web Hosts) —
the population that's most likely to be Cloudflare-walled or DDoS-
Guard-walled at the homepage layer but show up clearly in DDG
abstracts.

Smoke audit on a 30-row random sample of map adds: 28 plausible, 2
borderline (`es.graphicpkg.com → Food` could also be Industrial since
Graphic Packaging makes packaging *for* the food industry, but the
vertically-specialized rule applies; `annuairesante.ameli.fr` →
Finance via French health-insurance vocabulary, defensible). The 41
ambiguous rows stay in KU per the established workflow — they need
the same one-row-at-a-time human triage as PR #766 used.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Search-fallback batch (partial; outage-truncated): +226 promotions

Hotspot-bypass collector run was interrupted ~6,300/10,107 in when
the hotspot lost connectivity and the machine reverted to the
firewalled connection. Stopping here to commit what was
unambiguously classifiable; the remaining ~3,800 candidates
(plus any rows whose homepage fetch was tainted by the firewall
fallback during the transition) will be re-collected in a fresh
run after network stability is restored.

Promotions in this batch:
- 219 auto-classified by the regex classifier on the partial TSV
- 17 ambiguous rows resolved per LLM auto-resolution rules + user
  manual review
- 5 KU rows the user adjudicated explicitly (Bielsko-Biała,
  Douala-IX, Ekol Logistics, ICB, Marcus Corporation)
- 13 from earlier triage worklist with brands assigned
- Net 226 net-new map entries after dedupe, alias-leak filtering
  (3 link-target subdomains dropped where the parent base was
  already in the adds), full-IP privacy filtering (2 dropped),
  and ~30 targeted brand/category cleanups for rows where the
  search-fallback snippet had picked up a wrong page or the title
  contained registrant cruft / corporate-suffix leaks.

AGENTS.md updates:
- Codifies the "LLM auto-resolution of high-confidence ambiguous
  rows" workflow with R1-R5 high-confidence rules, low-confidence
  surface-to-human criteria, and the one-line auto-decision output
  format for reviewer overrule.
- Adds 7 triage lessons learned during this batch's bot-blocked-KU
  review (Polish/IT/ES/GR/RO city domains, "Sports Club" venues,
  vertically-specialized investment firms, sub-page fetch FPs,
  Telecom-suffix brand pinning, Hospital/Health-System suffix,
  IXP -ix brand pinning).

Map and KU files are disjoint after this commit. unknown_base_reverse_dns.csv
is empty (header-only) since every base_reverse_dns input is now
either mapped or in KU.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Search-fallback hotspot batch: +213 promotions

Fresh hotspot run on the 9,881 still-bot-blocked KU candidates left
after the prior outage-truncated batch. Classifier: 202 auto + 31
ambiguous (14 LLM auto-resolved per the R1-R5 high-confidence rules,
17 surfaced for interactive review) + 9,665 still KU + 1 dropped.

Net 213 net-new map entries after dedupe, alias-leak filtering
(13 link-target subdomains dropped where the parent base was
already in the map or in this batch's adds), 1 full-IP privacy
filter, 2 user-DROPs (1 alias of an as-numbered domain, 1 KU
because the only signal was a cross-vertical client list), and
~8 targeted brand cleanups for rows where the search snippet had
left a registrant-leak or domain-as-name placeholder.

LLM auto-resolutions (R1-R5):
  africell.ao         ISP        wi-tribe.pk         ISP
  ags.school.nz       Education  vwfs.com.au         Finance
  allaria.com.ar      Finance    wanxp.com           ISP
  asturias.org        Government varendraisp.com     ISP
  bdo.com.ph          Finance    titansi.com.my      IaaS
  bikada.kz           ISP        redeyenetworks.com  MSSP
  informatiq.org      ISP        plusinfo.ru         ISP

User-decided rows:
  admincomp.com       Consulting  korisp.com           Web Host
  anrb.ru             Science     linkexplorer.net.br  ISP
  arpc.ir             Industrial  novatech.bg          MSP
  as63031.net         Consulting  reliable-nets.com    ISP
  aviti.net           Web Host    satortech.com        MSP
  binaryelements.com.au MSP       skyworld.co.ke       Finance
  juni.net.br         ISP         telegroup-ltd.com    Technology
                                  west-webworld.fr     Technology

User KU/drops:
  itatec.com.py       KU (cross-vertical client list, no operator signal)
  ns2.as63031.net     DROP (alias of as63031.net)

AGENTS.md addition: codifies the "Web Host vs Email Provider —
bundled email-hosting is still Web Host" rule. Same shape as the
existing CCaaS/CPaaS-vs-ISP and MSP-vs-MSSP rules: classify by the
operator's primary product, not by every feature in their bundle.
Prompted by the korisp.com triage during this batch.

Map and KU files are disjoint after this commit. unknown_base_reverse_dns.csv
remains header-only (every base_reverse_dns input is now mapped or in KU).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:33:10 -04:00
b31a9e022f Reclassify KU pool: 2,248 promotions + new ambiguous-output worklist (#766)
* Reclassify KU pool: 2,248 promotions; surface 78 ambiguous rows for review

Re-fetched homepage / WHOIS / DNS for all 34,647 domains in
known_unknown_base_reverse_dns.txt via collect_domain_info.py and re-ran the
classifier. The classifier itself was extended in several directions while
auditing the unclassified pool — the changes are listed below.

Numbers
- 2,248 KU rows promoted to base_reverse_dns_map.csv (unambiguous matches).
- 78 rows surfaced as ambiguous (two or more distinct detector categories
  fired) — these are NOT auto-promoted; they need human adjudication.
- 32,399 rows remain in KU (genuinely no signal — most have privacy-only
  WHOIS, parked / blocked / Cloudflare-walled homepages, or empty MMDB
  enrichment).
- Disjoint invariant verified: comm -12 of map keys and KU prints nothing.
- Unknown-list regenerated via find_unknown_base_reverse_dns.py.

Classifier changes (classify_unknown_domains.py)

1. Three output buckets via new --ambiguous-out flag. Per-row outcome is now
   one of: map (auto-promote), ambiguous (worklist for human review), or
   ku (no signal). When ≥2 distinct detector categories fire on a row, the
   classifier picks a primary in precedence order but does NOT auto-promote
   — instead it writes the row to the ambiguous TSV with the alternatives
   listed. Rationale: the operator-typology question ("is this a SaaS
   company or an Energy company?") is a judgment call the classifier
   shouldn't make on its own.

2. Plural-matching fix: outer `\b` boundary changed to `s?\b` across all 46
   detectors so `dedicated server` matches `dedicated servers`,
   `law firm` matches `law firms`, etc. This was silently dropping the
   majority of English-text matches.

3. TLD-only signal classification: bare-TLD rows (gov.kh / ac.id / mil.bd /
   .jus.br etc.) now classify even when title/desc/as_name are all empty.
   Previously short-circuited at "need some signal".

4. TLD lists massively expanded:
   - Education: ~85 TLDs (every gov-restricted edu / ac suffix worldwide)
   - Government: ~110 TLDs incl. judicial branch (.jus.br) and legislative
     (.leg.br); covers Eastern Europe, MENA, SE Asia, Africa, Caribbean,
     Pacific
   - Military: ~45 .mil.* suffixes
   - Plus US K-12 regex (.k12.<state>.us)

5. New concrete-vocabulary patterns added based on KU-pool audit:
   - cybersecurity / cyber security for business → MSSP
   - autonomous system / asn owner / network operator / peering exchange
     / IXP → ISP
   - ICANN registrar / domain registrar / domain name platform / CDN /
     WAF / anti-DDoS → Web Host
   - BPM platform / CXM / CCaaS / CPaaS / contact center platform /
     compliance software → SaaS
   - katılım bankası / pensioen en verzekeringen / empréstimo consignado
     / credit (scores|reports|cards|comparison|bureau) /
     stock and commodity market → Finance
   - aeroportos de / passagem de ônibus / bilişim şirketi / havacılık →
     Travel & Tech variants
   - acciaio inossidabile / laminati piani → Industrial
   - Russian football-club declension forms (футбольного клуба, etc.)
   - tv channel / movie streaming / video streaming platform →
     Entertainment
   - genetic sequencing / next-generation sequencing /
     clinical diagnostic → Healthcare
   - punto vendita → Italian Retail
   - electrolyser / electrolyzer / green hydrogen → Energy

6. Mojibake table extended for Western European compounds: ã/â/ê/î/ô
   (Portuguese ã, French/PT â/ê/ô) plus uppercase variants.

Bug fixes from cross-language collisions

The audit pass exposed three short tokens that meant one thing in the
language they were added for and something completely different in another
language the classifier also targets:

- `por` (added as Luxembourgish for "parish" → Religion). Also the Spanish
  and Portuguese preposition "for / by", which appears on roughly every
  Spanish-language page. Was producing ~34 Religion false positives on
  Mexican ISPs, Brazilian utilities, etc.
- `pura` (added as Indonesian/Sundanese/Balinese for "Hindu temple" →
  Religion). Also the feminine of "pure" in Portuguese / Spanish / Italian,
  and a frequent brand-name fragment ("Pura Energia", "Angkasa Pura").
  Was misclassifying Brazilian electric utilities and Indonesian aviation
  services.
- bare `broker` (added as Luxembourgish for Finance). Matched any English
  text containing "broker" / "brokers" — including Cushman & Wakefield's
  "real estate brokers" line, which forced the row into Finance instead
  of Real Estate.

All three removed; AGENTS.md now codifies the rule.

AGENTS.md additions

- "Three output buckets" subsection: documents map / ambiguous / ku output
  and how PRs should call out ambiguous review counts.
- "No taglines / slogans" rule: marketing copy ("we make it easy",
  "smarter decisions") doesn't belong in any detector.
- "No ambiguous signals" rule: cross-category bare words (gazette / academy
  / society / club / studio) are forbidden as classifier keywords; use the
  pinning compound instead. Same rule applies in every language.
- "Cross-language grammar / lexical overlap" rule: short tokens that mean X
  in language A often mean a function word / adjective / brand fragment in
  language B. Cites the por / pura / broker incidents.
- "Classify by what the operator literally provides" rule: clusters by
  acronym suffix (UCaaS / CCaaS / CPaaS) tempt mis-grouping; CCaaS is SaaS
  not ISP, etc. Includes the root-cause analysis of the
  contact-center-as-ISP mistake.
- "Genuinely-ambiguous-between-two-types" rule: phrases like
  "energy management software" that fit equally on a SaaS startup, an
  Industrial conglomerate, and a consultancy belong in NO detector — leave
  the row unmapped and rely on more-specific compounds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Triage 78 ambiguous rows + new classifier filters and rules

Interactive triage of the 78 rows the v1 classifier surfaced as ambiguous
(two or more distinct categories fired). Net result of this commit, on
top of the v1 promotions already in the branch:

- 74 ambiguous rows promoted to map with a human-adjudicated category
  (and 10 of those with a corrected human-cleaned brand vs. the noisy
  as_name / title-bleed the v1 classifier captured).
- 1 row dropped silently per the AGENTS.md adult-content rule.
- 3 rows kept in KU (personal projects, parked pages caught by the
  classifier mid-triage that we then surface'd-then-confirmed).

Map: 37,566 → 37,640 (+74). KU: 32,399 → 32,324 (−75). Disjoint clean.

Three new classifier filters added during triage as recurring patterns
surfaced — these run before category detectors and short-circuit to KU
or DROP rather than letting the operator-typology detectors fire on
parking-page / personal-page / adult-page text:

1. PARKED_PAGE_RE — Media Temple "automatically generated default server
   page", Hostinger Horizons, Apache default, parked-by-registrar pages,
   "site has shut down", "has completed its journey". Cloudflare /
   DDoS-Guard / "Are you a robot?" interstitials are explicitly NOT
   filtered (they leave the TLD-signal path open for gov / edu / mil
   sites that are bot-blocked).
2. PERSONAL_PROJECT_RE — "personal BGP project", "personal website and
   CV", "homelab", "hobby project", "side project". Hobbyists running
   their own ASN aren't commercial operators.
3. ADULT_CONTENT_RE — adult web design / adult-entertainment hosting /
   xxx / escort directory etc. Returns a sentinel ("DROP", None) so the
   caller drops the domain from both map and KU per the AGENTS.md
   content rule.

The classifier API now also writes a fourth output file (--dropped-out)
listing domains the adult-content filter caught, so the caller can
remove them from any tracked list files they currently sit in.

Title-noise list extended to catch: "attention required" / "are you a
robot" / "checking your browser" / "please enable javascript" /
"ddos-guard" / "px-captcha" / "site is not available" / "page is not
available" / "access to this page has been denied". This stops these
strings from bleeding into the brand column when TLD-only classification
fires (the `health.gov.il → "Attention Required!"` shape of bug).

Several cross-language false positives caught during the triage — same
shape as the por / pura / broker incidents the previous commit fixed:

- bare French `e?mailing` matched "Mailing Solutions" (mail-server
  infrastructure on a Cisco VAR's product list, not marketing). Required
  to start with `e` to keep the email-marketing meaning while losing the
  bare-mailing collision.
- Norwegian / Danish bare `avis` (newspaper) matched "Avis Romania" car
  rental and any French text saying "avis" (notice/opinion). Replaced
  with compound forms (`dagsavis`, `lokalavis`, `morgenavis`, etc.).
- Vietnamese bare `bộ` (ministry) matched "bộ phim" (movie set), "bộ
  sưu tập" (collection), and the founding-text references on Vietnam
  Eximbank's about page. Replaced with compound forms (`bộ trưởng`, `bộ
  tài chính`, `bộ ngoại giao`, etc.).
- Russian bare `провайдер` (provider) matched "хостинг провайдер"
  (hosting provider, Web Host) on a Tajikistan domain registrar. Removed
  the bare form; only the internet-specific compounds remain.
- Luxembourgish bare `broker` (Finance) matched "real estate brokers"
  on Cushman & Wakefield's homepage and any English page mentioning
  brokers. Removed the bare form entirely.
- Turkish bare `vakıf` (foundation) matched "Vakıf Katılım Bankası" —
  for-profit Islamic-finance bank whose brand uses the word. Replaced
  with nonprofit-specific compounds (`yardım vakfı`, `hayır vakfı`,
  `kamu yararına vakıf`).

New positive-classification keywords added based on triage gaps:

- MSP rescue path now matches the SMB-IT-shop idiom in Polish
  (`usługi IT dla biznesu`, `obsługa informatyczna firm`,
  `outsourcing IT`), Spanish (`servicios informáticos para empresas`),
  German (`IT-Dienstleister für`, `managed-IT-services`), French
  (`infogérance`, `prestataire de services informatiques`), Italian
  (`servizi informatici gestiti`, `outsourcing informatico`),
  Portuguese (`serviços de TI gerenciados`, `terceirização de TI`),
  Dutch (`beheerd-IT`, `IT-beheer`), and Indonesian
  (`penyedia solusi IT`, `solusi IT terpadu/berbasis`).
- Finance now matches `accounting firm` / `cpa firm` /
  `certified public accountants` / `chartered accountants` /
  `tax preparation` / `tax advisory` / `audit firm` plus equivalents in
  Spanish, Portuguese, French, German, Italian, and Polish.
- SaaS now matches CCaaS / CPaaS / `contact-center-as-a-service` /
  `communications-platform-as-a-service` / `compliance software` /
  `regulatory management software` and CCaaS no longer lives in ISP
  (carryover from the user-flagged "contact centers are not ISPs"
  correction).

AGENTS.md additions:

- "Triage heuristics learned from the 78-row interactive review of
  PR #766's ambiguous bucket" subsection codifying every adjudication
  rule the user applied during the review:
  * pick the main-focus category (first / most-mentioned)
  * clients are not operator typology
  * vertically-specialized firms take the vertical
  * stream-hosting infrastructure is Web Host
  * multi-service SMB IT shops are MSP
  * VARs are Technology
  * CCaaS / CPaaS / UCaaS are SaaS
  * gov/edu/mil/jus TLD signal trumps Cloudflare interstitials
  * esports tournament organizers are Entertainment
  * personal projects / parked pages / adult content go to KU or DROP
  * brand quality is its own dimension — capture corrected brand
    during triage rather than shipping the noisy as_name

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 00:03:45 -04:00
06d277686d classify_unknown_domains.py: enforce concept-parity across ~60 languages (#765)
Multilingual detectors previously held English at full breadth (e.g. Healthcare =
hospital + clinic + pharmacy + healthcare + pharmaceutical industry + nursing
home + medical center) while many non-English sections covered the same concept
set with only one or two transliterated words. This left every language other
than English under-detecting against pages that used the operator's natural
compound terms.

Reworked every detector so each language now expresses the same English concept
set in idiomatic compounds — never inventing calques where no natural form
exists. Added ~32 new languages (Macedonian, Belarusian, Azerbaijani, Armenian,
Georgian, Kazakh, Uzbek, Mongolian, Khmer, Burmese, Lao, Nepali, Sinhala,
Amharic, Yoruba, Hausa, Igbo, Zulu, Pashto, Kurdish, Tajik, Kyrgyz, Maltese,
Luxembourgish, Haitian Creole, Frisian, Yiddish, Faroese, Tatar, Javanese,
Sundanese, Cebuano) on top of the existing pool, again applied per-concept
rather than as token presence.

Also added British / American spelling pairs where they diverge (`tire`/`tyre`,
`defense`/`defence`, `center`/`centre`, etc.) and a handful of new English
concepts that had been implicit (`tire shop`, `car parts`, `oil exploration`,
`olympic committee`, ...) — each with its multilingual equivalents in the same
edit.

AGENTS.md: codified the rule under "Maintaining the reverse DNS maps" so future
edits are bound by it: every language section must cover the same concept set
the English section covers, with idiomatic compounds rather than calques, skip
rather than invent when no natural form exists, and any new English keyword
must be added in parallel across the existing language set.

Final shape: 11,777 alternations / 175,556 chars across 45 detectors. Ruff
check + format clean. Module compiles.

Known limitation (pre-existing, unchanged): Python's `re` does not treat
Unicode Mn / Mc combining marks as word characters, so Brahmic-script words
ending in vowel signs / virama won't match the outer `\b…\b`. Affects
pre-existing and new entries equally; fixable later by switching to the
`regex` module.

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 19:01:15 -04:00
3b705aeaa8 Commit classify_unknown_domains.py — regex-based multilingual classifier (#764)
* Commit classify_unknown_domains.py: regex-based multilingual classifier

Promotes the transient `/tmp/classify_b<N>.py` script that grew across
the b5–b13 reverse-DNS-map batches into a tracked tool. The classifier
takes a `collect_domain_info.py` TSV and emits a CSV of map additions
plus a text file of known-unknown additions — the regex baseline that
makes step 4 of the unknown-domain workflow ("classify from the TSV, not
by re-fetching") tractable at scale.

Coverage:

- Detectors for all 44 industry types in the README.
- Concept-translation parity across ~30 languages on the high-volume
  detectors (Healthcare, Travel, Government, Retail, Finance, ISP, Web
  Host, Manufacturing, Logistics, Real Estate, Automotive, Legal,
  Agriculture).
- ~10–20 languages with 1–3 keywords each on the smaller detectors
  (Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media,
  Defense, IaaS/PaaS/SaaS, Beauty, Print, Publishing, Religion, Science,
  Event Planning, Staffing, Email Security/Provider, Marketing,
  Construction, Industrial, Utilities, Energy, Government Media,
  Physical Security, News, Nonprofit, Entertainment, Technology,
  Consulting).

Brand-name selection prefers MMDB `as_name` → page title's first
segment → non-redacted WHOIS registrant → domain-derived fallback, with
a `clean_brand` pass that strips legal-form suffixes (LLC / GmbH / Ltda
/ EIRELI / sp. z o.o. / s.c.a r.l / UAB / etc.) and prefixes (PT, OOO).
When the title has multiple segments, the segment whose simplified form
contains the domain root is preferred — accessmontana.com with as_name
"MONTANA WEST, L.L.C." and title "Internet, Phone & TV Bundles | Access
Montana" maps to "Access Montana", not "Montana West".

A small mojibake fixer normalizes the most common UTF-8-as-Latin-1
re-encodings ("ó" → "ó", etc.) so Spanish/Portuguese/French homepages
that `collect_domain_info.py` mishandled still classify.

The empty HAND dict at the top of the file is an extension point for
batch-specific overrides — e.g. acquisition aliases or brand-name
corrections that don't fit any detector; each `domain → ("Brand",
"Type")` entry wins over the auto-classifier.

Wired into AGENTS.md's "Related utility scripts" section and documented
in `parsedmarc/resources/maps/README.md` alongside the rest of the
maps utilities.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* classify_unknown_domains.py: clarify dual-purpose framing

The classifier serves both lookup paths into base_reverse_dns_map.csv —
the original PTR-side flow (reverse-DNS base domains derived from DMARC
report source IPs) and the MMDB-coverage flow (AS domains lifted from
the bundled IPinfo Lite MMDB). The initial commit's docstring/comments
emphasized the MMDB-coverage flow because that's where the script grew
up across the b5–b13 batches, but it was always equally applicable to
PTR-side domains.

Updates:

- Top docstring rewritten to lead with the dual-purpose framing.
- README.md adds an explicit "useful for either lookup path" paragraph
  referencing the original DMARC-report flow and the MMDB-coverage flow.
- AGENTS.md "Related utility scripts" entry updated to mention both
  flows.
- Drops a stale "happen to have ASN registrations" aside in the
  RETAIL_RE comment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 17:16:23 -04:00
9aa930f7cc Retroactive promotions: 3,171 KU rows reclassified by expanded multilingual classifier (#763)
Re-ran the expanded-multilingual classifier (PR #762's classifier with
broader language coverage on Healthcare, Travel, Government, Retail,
Finance, ISP, Manufacturing, Logistics, Real Estate, Automotive, Legal,
Agriculture, plus Finance-via-body-text catching insurance/investment/
asset-management) against every cached TSV from prior batches (b6–b13).
3,171 domains that previously couldn't be auto-classified (and were
therefore added to known_unknown_base_reverse_dns.txt) now match the new
detectors.

These domains are promoted out of KU and into the map under their newly
classified `(name, type)` pairs.

Type distribution of promotions:
  Finance         736   Logistics        179   Real Estate     105   Healthcare    68
  ISP             323   Retail           159   Education       110   Marketing     66
  Manufacturing   207   Technology       142   Consulting       99   Nonprofit     64
  Government      136   Travel           123   Utilities        71   Legal         53
  + smaller volumes across ~25 other industry types

ASN-domain coverage of the bundled IPinfo Lite MMDB after these promotions:
  - by domain count:  32,254 / 63,993  (50.40%, up from 45.45%)
  - by IPv4 weight:   98.45%

Honest scope note: the multilingual classifier achieves "concept parity"
for the top-5 high-volume detectors (Healthcare, Travel, Government,
Retail, Finance) across ~30 languages. Smaller detectors (Photography,
Conglomerate, Sports, Defense, MSSP, IaaS/PaaS/SaaS, etc.) still have
~10-15 languages with 1-3 keywords each. Further per-detector
multilingual parity is a follow-up effort; each subsequent expansion
recovers fewer domains as the long tail of language-specific phrasings
shrinks.

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 17:00:25 -04:00
c25bf28c1c Classify reverse DNS map: final cleanup batch (~2,650 unmapped MMDB ASN domains) (#762)
Final cleanup pass to clear the remaining MMDB AS-domain queue. Applied an
expanded multilingual classifier covering all 44 README industry types
plus an Energy concept (mapped to Utilities pending a README addition).
Per-detector keyword lists now include Spanish, Portuguese, French,
Italian, German, Dutch, Russian, Polish, Czech, Turkish, Greek, Chinese
(simplified and traditional), Japanese, Korean, Arabic, Hebrew, Hindi,
Vietnamese, Indonesian, and Thai where the concept has a recognizable
local-language equivalent.

- 980 added to base_reverse_dns_map.csv (ISP 193, Education 193, Finance
  155, Government 109, Healthcare 93, Web Host 37, MSP 31, Manufacturing
  22, Logistics 17, Real Estate 12, Travel 11, Consulting 10, Tech 9,
  Nonprofit 9, Legal 9, Food 9, Retail 8, Religion 8, Utilities 7, plus
  smaller volumes across 14 more types).
- 1,669 added to known_unknown_base_reverse_dns.txt — the residual
  unfetchable / parked / Cloudflare-challenged / non-recognized-content
  rows.

ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
  - by domain count:  29,083 / 63,993  (45.45%)
  - by IPv4 weight:   98.36%

Total since batch 5: ~16,400 map rows + ~17,400 known-unknown rows added
across 9 batches. Remaining unmapped pool size: 0 — every MMDB AS-domain
has now been processed (either classified or recorded in known-unknown).

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:48:56 -04:00
fa03b8f2c2 Classify reverse DNS map: next 10000 unmapped MMDB ASN domains (#761)
Batch 12. Auto-rate dropped to 23% (2330/9998) — significantly lower than
batch 11. The deeper into the long tail, the more candidates fall into
non-classifier-recognized industries (retail, manufacturing, hospitality,
local services) where the ISP/Web Host/MSP regex doesn't fire even though
the page is fetchable.

- 2,330 added to base_reverse_dns_map.csv (ISP 991, Education 295, Finance
  290, Government 265, Web Host 229, Healthcare 134, MSP 126).
- 7,667 added to known_unknown_base_reverse_dns.txt.

ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
  - by domain count:  27,824 / 63,993  (43.48%, up from 40.27%)
  - by IPv4 weight:   98.36%

Same classifier as prior batches (no new code).

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:54:25 -04:00
e6716c9e80 Classify reverse DNS map: next 10000 unmapped MMDB ASN domains (#760)
Batch 11. Auto-rate 30.6% — slightly lower than batch 10's 35.6%, consistent
with continuing long-tail descent.

- 3,063 added to base_reverse_dns_map.csv (ISP 1,734, Finance 270, Web Host
  254, Education 253, Government 249, MSP 153, Healthcare 150).
- 6,934 added to known_unknown_base_reverse_dns.txt.

ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
  - by domain count:  25,773 / 63,993  (40.27%, up from 35.49%)
  - by IPv4 weight:   98.32%           (up from 98.27%)

Same classifier as prior batches (no new code).

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:13:26 -04:00
ee9bda7228 Classify reverse DNS map: next 10000 unmapped MMDB ASN domains (#759)
First of the 10K-batch series toward complete coverage. Auto-rate 35.6%
this round, consistent with the long-tail descent.

- 3,556 added to base_reverse_dns_map.csv (ISP 2,410, Web Host 348,
  Education 247, Finance 201, Government 124, MSP 121, Healthcare 105).
- 6,442 added to known_unknown_base_reverse_dns.txt.

ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
  - by domain count:  22,710 / 63,993  (35.49%, up from 29.93%)
  - by IPv4 weight:   98.27%           (up from 98.18%)

Same classifier as batches 5-9 (no new code).

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 14:33:56 -04:00
80a132801d Classify reverse DNS map: next 5000 unmapped MMDB ASN domains (#758)
Auto-classification rate jumped back to 50% (2502/4999) from 36.5% in
batch 8 — this slice happens to contain a higher proportion of small ISPs
with conventional homepages, lifting the regex hit rate.

- 2,502 added to base_reverse_dns_map.csv (ISP 2,065, Web Host 133,
  Education 96, Finance 67, MSP 60, Government 58, Healthcare 23).
- 2,496 added to known_unknown_base_reverse_dns.txt.

ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch:
  - by domain count:  19,154 / 63,993  (29.93%, up from 26.02%)
  - by IPv4 weight:   98.18%           (up from 98.09%)

Same classifier as batches 5-8 (no new code).

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 13:46:28 -04:00