mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-05-20 19:05:24 +00:00
2d3e896f6d39dc2eab272b714a691636a1dbb6dd
1521 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sean Whalen
|
2d3e896f6d | Fix pytest command line argument typo | ||
|
Sean Whalen
|
c5b2fcec54 |
Enhance CI with JUnit XML output and Codecov results
Added JUnit XML output for pytest and Codecov test results upload. |
||
 github-actions[bot]
|
a6ea169df5 |
chore: update IPinfo Lite MMDB (#771)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
github-actions[bot]
|
1fc1134f77 |
chore: update IPinfo Lite MMDB (#769)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean Whalen
|
ff6f75d740 |
Map-data build hygiene: README single source of truth, drop maintainer scripts from wheel (9.11.2) (#768)
* Drop base_reverse_dns_types.txt; sortlists.py now reads types from README.md The .txt file duplicated the README's industry list and introduced drift risk — twice in the project's history we had to add types to the .txt only because the README had been updated independently. Make the README the single source of truth. - Add `<!-- types-list:start -->` / `<!-- types-list:end -->` HTML comment markers around the bullet list in parsedmarc/resources/maps/README.md. Markers don't render in GitHub's preview. - New `load_types_from_readme()` in sortlists.py parses the bullet items between the markers and returns them. Errors clearly if the README is missing or the markers are absent. - Delete base_reverse_dns_types.txt. - Fix a pre-existing typo in README precedence rule 4: `Web Hosting` → `Web Host` (matches the canonical type used in 4,176 map rows). Smoke test: feeding a row with a bogus type still triggers the validator (`'NotARealType' is not an allowed value for 'type'`), confirming the README-derived list flows through identically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * sortlists.py: normalize README types-list block in place Before validating the map, the validator now sorts the <!-- types-list:start --> / <!-- types-list:end --> block in README.md alphabetically (case-insensitively), trims leading and trailing whitespace from each item, and deduplicates case- insensitively, rewriting the README in place if any of those need fixing. Errors clearly when two entries differ only by casing (which would otherwise silently lose one). Adding a new category is now just inserting a `- New Type` line anywhere inside the markers — `sortlists.py` will tidy it on the next run. Same shape as how the validator already normalizes known_unknown_base_reverse_dns.txt and psl_overrides.txt. The pure read path is preserved as `load_types_from_readme()` for callers that don't want a side-effecting rewrite (tests, downstream tooling). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Stop shipping maintainer scripts; bump to 9.11.2 The exclude list in [tool.hatch.build] was originally meant to keep maintainer-only batch tooling under parsedmarc/resources/maps/ out of the wheel and sdist (it lists `find_bad_utf8.py`, `find_unknown_base_reverse_dns.py`, the renamed-and-removed `sortmaps.py`). The list never grew when new tools were added, so `collect_domain_info.py`, `classify_unknown_domains.py`, `detect_psl_overrides.py`, `detect_rebrands.py`, and `sortlists.py` all started shipping in distributions despite contributing nothing to runtime functionality. Replace the per-file basename list with a single glob pattern: parsedmarc/resources/maps/[!_]*.py The leading-`_` exception keeps `__init__.py` shipping (required so that `importlib.resources.files(parsedmarc.resources.maps)` can locate the bundled CSV/TXT data files), while excluding any other .py file under that directory — including future maintainer scripts that haven't been written yet. Drop the now-redundant per-file entries from the exclude list: `find_bad_utf8.py`, `find_unknown_base_reverse_dns.py`, and the already-removed `sortmaps.py`. The non-.py exclusions stay (`base_reverse_dns.csv`, `unknown_base_reverse_dns.csv`, `README.md`, `*.bak`). Verified with `hatch build`: - Wheel under parsedmarc/resources/maps/: __init__.py + 3 data files (CSV/TXTs), no maintainer .py - sdist matches - Clean-venv install of the built wheel loads 298 PSL overrides and `get_base_domain('host01.netlify.app')` returns `netlify.app` Bump to 9.11.2 since this changes shipped artifacts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9.11.2 |
||
|
Sean Whalen
|
053195581b |
collect_domain_info.py: opt-in DuckDuckGo search fallback for bot-blocked rows (#767)
* collect_domain_info.py: opt-in DuckDuckGo search fallback for bot-blocked rows A meaningful share of KU domains return a Cloudflare / DDoS-Guard / "Are you a robot?" / px-captcha interstitial instead of real homepage content — even after the curl-style relaxed-TLS fallback runs. For those rows we have neither homepage signal nor (often) a usable as_name, and they fall through to KU even though the operator is a real (often well-known) business that the classifier could trivially handle if it could just see the page. Added an opt-in `--use-search-fallback` flag that asks DuckDuckGo for `site:<domain>` when the homepage fetch returned a bot-block / parking / empty result, and uses the top result's title and description (only if the result host belongs to the input domain — anti-SEO-spam guard). Mechanism - New optional `ddgs` dependency, listed under the `[build]` extras. `from ddgs import DDGS` is wrapped in a try/except — the script runs without ddgs installed as long as `--use-search-fallback` isn't passed; the flag check exits with a helpful install message otherwise. - `_SEARCH_FALLBACK_TRIGGER_RE` — title/description patterns that look like a bot-block / WAF interstitial / parked / placeholder. Triggers the fallback. Same shape as the classifier's TITLE_NOISE_RE / PARKED_PAGE_RE; the search fallback is the recovery path for exactly the rows that filter excludes. - `_looks_bot_blocked()` — combined check: trigger regex matches OR title and description are both empty (typical of WAF interstitials that strip <title>/<meta> entirely). - `_hosts_match()` — same-domain SEO-spam guard. A search result is accepted only when its host is exactly the input domain or a subdomain of it. Third-party SEO-spam pages that scraped the domain name are silently skipped. - `_search_fallback_fetch()` — runs `site:<domain>` through DDG, walks results in rank order, returns the first one whose host passes the guard. Returns empty if no result matches (caller leaves the row's homepage data alone in that case). - `_collect_one()` now takes a `use_search_fallback` flag, calls the fallback after the homepage fetch when the homepage looks bot-blocked, and writes `title_source = "homepage"` or `"search"` so reviewers can audit which rows came from where. - New `title_source` column in the TSV. Smoke test Test set: bbc.com (real homepage, no fallback expected) plus 5 known Cloudflare-walled rows (1800contacts.com, americaneagle.com, broadwaytechnology.com, health.gov.il, mfa.gov.il). Result: bbc.com classified via homepage; the other 5 all recovered title + description via search and got `title_source=search`. The same-domain guard validated independently — for broadwaytechnology.com the guard correctly rejects bloomberg.com and accepts support.broadwaytechnology.com (broadway was acquired by Bloomberg, but the search fallback returns the broadway-domain snippet, not the parent's bloomberg.com product page). Caveats codified in AGENTS.md - Search snippets are still untrusted text (data-not-instructions rule applies the same way it does to homepage HTML). - DDG's index can lag a homepage rebrand by months — when a row classified via `title_source=search` disagrees with a fresh manual fetch, prefer the manual verification. The fallback is a recovery aid, not a tiebreaker against fresh content. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * collect/classify: link-following + alias map rows for placeholder DDG titles When the search fallback ran on the original 6-domain smoke set, two of the recovered titles were essentially placeholder pointers carrying no classifier signal — DDG returned `Link to fcs.health.gov.il` for one input and a bare `yangon.mfa.gov.il` for another. Those snippets are DDG's way of saying "I have an indexed subdomain but no real abstract to give you", and feeding them to the regex classifier produces no better signal than the parking-page result we were already trying to recover from. This commit teaches the collector to recognize both placeholder shapes, follow the pointer to the target hostname, and use *that* hostname's real content for the row. The classifier then emits the original input and the link target as **two map rows under the same (name, type)** so both keys are looked up against future DMARC reports. collect_domain_info.py - New `_LINK_TO_TITLE_RE` / `_BARE_HOSTNAME_RE` and an `_extract_link_target` helper that returns the target hostname when the search title is `Link to <hostname>` or a bare hostname, "" when the title carries real content. - After the search-fallback path, if the title looks like a pointer and the target differs from the input, `_fetch_homepage(target)` is called once. When the target's fetch returns real (non-bot-blocked) content, the row's title / description / final_url / rebrand_signal / external_links are replaced with the target's, and `title_source` becomes `search→<target>` so reviewers can audit the path. - New `link_target_domain` column records the followed target whether or not its fetch succeeded. classify_unknown_domains.py - When a row's `link_target_domain` is set and differs from the input domain, the classifier emits a second map row for the target with the same `(name, type)`. The original input is the "og" domain; the target is what DDG pointed us at — both end up in the map as aliases. Same handling applies on the ambiguous-bucket path so a single human adjudication covers both. Smoke test on the original 6-domain set: bbc.com homepage → BBC Home – Breaking News, … 1800contacts.com search → 1800contacts health.gov.il search → Homepage – COVID Information Center of the Israel Ministry of Health americaneagle.com search → Americaneagle.com | Web Design … broadwaytechnology.com search → Bloomberg Completes Acquisition of … mfa.gov.il search→yangon.mfa.gov.il → Home | Ministry of Foreign Affairs link_target_domain=yangon.mfa.gov.il The mfa.gov.il row triggered the new path: DDG returned `yangon.mfa.gov.il` as the title, the collector followed it, the target's homepage gave us "Home | Ministry of Foreign Affairs", and the classifier emitted both `mfa.gov.il, Ministry of foreign affairs, Government` and `yangon.mfa.gov.il, Ministry of foreign affairs, Government`. AGENTS.md updated with the link-following / alias rules under the search-fallback subsection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Run --use-search-fallback against 10,544 bot-blocked KU rows; +473 promotions Also expands the search-fallback trigger regex to recognize self-signed TLS interception (firewall block via cert) and a wider class of local-firewall block-page strings. Mechanics 1. Identified 10,544 KU rows from the 34,647-row prior TSV that looked bot-blocked (via the new `_looks_bot_blocked` detector). 2. Ran `collect_domain_info.py --use-search-fallback` against just those rows. Throughput was ~3.4 rows/sec at 32 workers / 3s HTTP timeout / 5s WHOIS timeout. ~50 min wall time. 3. Audited the resulting TSV and discovered 2,078 rows whose homepage fetch had silently returned a corporate firewall's block page (Fortinet "Web Filter Violation" being the most common, 1,419 of them). The original `_SEARCH_FALLBACK_TRIGGER_RE` didn't recognize those strings, so search-fallback wasn't firing — the firewall's block-page text was being fed to the classifier as if it were the operator's homepage. Almost no false promotions resulted (block-page text doesn't match industry detectors), but the rows weren't recovering either. 4. Expanded the trigger regex to catch web-filter block pages, then re-fetched just the 2,078 affected rows. 5. Final classifier pass: 474 unambiguous map adds, 41 ambiguous, 1 silently dropped (adult content), 10,066 still in KU. Self-signed-cert detection A separate fix lands in this commit: when the primary fetch fails with an SSL cert verification error matching "self-signed certificate", the collector skips the verify=False browser fallback. Rationale: TLS- intercepting firewalls (corporate or personal-network) present their own self-signed cert specifically when blocking. The verify=False fallback would happily retrieve the firewall's block page, which then poisons the row's title/description. Skipping that path leaves the row's metadata empty so search-fallback can recover real content. Other cert errors (hostname mismatch, weak DH, legacy renegotiation) keep the existing fallback path because they're typically real operators with misconfigured TLS rather than firewall interception. Numbers Map: 37,640 → 38,114 (+474) KU: 32,324 → 31,886 (−438) Disjoint check: 0 shared keys Unknown CSV: regenerated, just the header Type distribution of the 474 promotions 162 ISP 17 MSP 4 MSSP / Marketing 72 Web Host 16 Technology 4 Beauty / Agriculture 41 Finance 14 Healthcare 3 IaaS / Science / Legal 19 Government 11 Travel 2 Search / Religion / SaaS 10 Logistics 8 Manufacturing 2 Email Sec / Email Provider 9 Education / Retail 8 News 2 Entertainment 7 Utilities / Phys Sec 6 Real Estate 1 Auto / Staff / PaaS 6 Food / Consulting / Industrial / Conglomerate / Nonprofit Most of the gains are network operators (162 ISPs, 72 Web Hosts) — the population that's most likely to be Cloudflare-walled or DDoS- Guard-walled at the homepage layer but show up clearly in DDG abstracts. Smoke audit on a 30-row random sample of map adds: 28 plausible, 2 borderline (`es.graphicpkg.com → Food` could also be Industrial since Graphic Packaging makes packaging *for* the food industry, but the vertically-specialized rule applies; `annuairesante.ameli.fr` → Finance via French health-insurance vocabulary, defensible). The 41 ambiguous rows stay in KU per the established workflow — they need the same one-row-at-a-time human triage as PR #766 used. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Search-fallback batch (partial; outage-truncated): +226 promotions Hotspot-bypass collector run was interrupted ~6,300/10,107 in when the hotspot lost connectivity and the machine reverted to the firewalled connection. Stopping here to commit what was unambiguously classifiable; the remaining ~3,800 candidates (plus any rows whose homepage fetch was tainted by the firewall fallback during the transition) will be re-collected in a fresh run after network stability is restored. Promotions in this batch: - 219 auto-classified by the regex classifier on the partial TSV - 17 ambiguous rows resolved per LLM auto-resolution rules + user manual review - 5 KU rows the user adjudicated explicitly (Bielsko-Biała, Douala-IX, Ekol Logistics, ICB, Marcus Corporation) - 13 from earlier triage worklist with brands assigned - Net 226 net-new map entries after dedupe, alias-leak filtering (3 link-target subdomains dropped where the parent base was already in the adds), full-IP privacy filtering (2 dropped), and ~30 targeted brand/category cleanups for rows where the search-fallback snippet had picked up a wrong page or the title contained registrant cruft / corporate-suffix leaks. AGENTS.md updates: - Codifies the "LLM auto-resolution of high-confidence ambiguous rows" workflow with R1-R5 high-confidence rules, low-confidence surface-to-human criteria, and the one-line auto-decision output format for reviewer overrule. - Adds 7 triage lessons learned during this batch's bot-blocked-KU review (Polish/IT/ES/GR/RO city domains, "Sports Club" venues, vertically-specialized investment firms, sub-page fetch FPs, Telecom-suffix brand pinning, Hospital/Health-System suffix, IXP -ix brand pinning). Map and KU files are disjoint after this commit. unknown_base_reverse_dns.csv is empty (header-only) since every base_reverse_dns input is now either mapped or in KU. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Search-fallback hotspot batch: +213 promotions Fresh hotspot run on the 9,881 still-bot-blocked KU candidates left after the prior outage-truncated batch. Classifier: 202 auto + 31 ambiguous (14 LLM auto-resolved per the R1-R5 high-confidence rules, 17 surfaced for interactive review) + 9,665 still KU + 1 dropped. Net 213 net-new map entries after dedupe, alias-leak filtering (13 link-target subdomains dropped where the parent base was already in the map or in this batch's adds), 1 full-IP privacy filter, 2 user-DROPs (1 alias of an as-numbered domain, 1 KU because the only signal was a cross-vertical client list), and ~8 targeted brand cleanups for rows where the search snippet had left a registrant-leak or domain-as-name placeholder. LLM auto-resolutions (R1-R5): africell.ao ISP wi-tribe.pk ISP ags.school.nz Education vwfs.com.au Finance allaria.com.ar Finance wanxp.com ISP asturias.org Government varendraisp.com ISP bdo.com.ph Finance titansi.com.my IaaS bikada.kz ISP redeyenetworks.com MSSP informatiq.org ISP plusinfo.ru ISP User-decided rows: admincomp.com Consulting korisp.com Web Host anrb.ru Science linkexplorer.net.br ISP arpc.ir Industrial novatech.bg MSP as63031.net Consulting reliable-nets.com ISP aviti.net Web Host satortech.com MSP binaryelements.com.au MSP skyworld.co.ke Finance juni.net.br ISP telegroup-ltd.com Technology west-webworld.fr Technology User KU/drops: itatec.com.py KU (cross-vertical client list, no operator signal) ns2.as63031.net DROP (alias of as63031.net) AGENTS.md addition: codifies the "Web Host vs Email Provider — bundled email-hosting is still Web Host" rule. Same shape as the existing CCaaS/CPaaS-vs-ISP and MSP-vs-MSSP rules: classify by the operator's primary product, not by every feature in their bundle. Prompted by the korisp.com triage during this batch. Map and KU files are disjoint after this commit. unknown_base_reverse_dns.csv remains header-only (every base_reverse_dns input is now mapped or in KU). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
b31a9e022f |
Reclassify KU pool: 2,248 promotions + new ambiguous-output worklist (#766)
* Reclassify KU pool: 2,248 promotions; surface 78 ambiguous rows for review
Re-fetched homepage / WHOIS / DNS for all 34,647 domains in
known_unknown_base_reverse_dns.txt via collect_domain_info.py and re-ran the
classifier. The classifier itself was extended in several directions while
auditing the unclassified pool — the changes are listed below.
Numbers
- 2,248 KU rows promoted to base_reverse_dns_map.csv (unambiguous matches).
- 78 rows surfaced as ambiguous (two or more distinct detector categories
fired) — these are NOT auto-promoted; they need human adjudication.
- 32,399 rows remain in KU (genuinely no signal — most have privacy-only
WHOIS, parked / blocked / Cloudflare-walled homepages, or empty MMDB
enrichment).
- Disjoint invariant verified: comm -12 of map keys and KU prints nothing.
- Unknown-list regenerated via find_unknown_base_reverse_dns.py.
Classifier changes (classify_unknown_domains.py)
1. Three output buckets via new --ambiguous-out flag. Per-row outcome is now
one of: map (auto-promote), ambiguous (worklist for human review), or
ku (no signal). When ≥2 distinct detector categories fire on a row, the
classifier picks a primary in precedence order but does NOT auto-promote
— instead it writes the row to the ambiguous TSV with the alternatives
listed. Rationale: the operator-typology question ("is this a SaaS
company or an Energy company?") is a judgment call the classifier
shouldn't make on its own.
2. Plural-matching fix: outer `\b` boundary changed to `s?\b` across all 46
detectors so `dedicated server` matches `dedicated servers`,
`law firm` matches `law firms`, etc. This was silently dropping the
majority of English-text matches.
3. TLD-only signal classification: bare-TLD rows (gov.kh / ac.id / mil.bd /
.jus.br etc.) now classify even when title/desc/as_name are all empty.
Previously short-circuited at "need some signal".
4. TLD lists massively expanded:
- Education: ~85 TLDs (every gov-restricted edu / ac suffix worldwide)
- Government: ~110 TLDs incl. judicial branch (.jus.br) and legislative
(.leg.br); covers Eastern Europe, MENA, SE Asia, Africa, Caribbean,
Pacific
- Military: ~45 .mil.* suffixes
- Plus US K-12 regex (.k12.<state>.us)
5. New concrete-vocabulary patterns added based on KU-pool audit:
- cybersecurity / cyber security for business → MSSP
- autonomous system / asn owner / network operator / peering exchange
/ IXP → ISP
- ICANN registrar / domain registrar / domain name platform / CDN /
WAF / anti-DDoS → Web Host
- BPM platform / CXM / CCaaS / CPaaS / contact center platform /
compliance software → SaaS
- katılım bankası / pensioen en verzekeringen / empréstimo consignado
/ credit (scores|reports|cards|comparison|bureau) /
stock and commodity market → Finance
- aeroportos de / passagem de ônibus / bilişim şirketi / havacılık →
Travel & Tech variants
- acciaio inossidabile / laminati piani → Industrial
- Russian football-club declension forms (футбольного клуба, etc.)
- tv channel / movie streaming / video streaming platform →
Entertainment
- genetic sequencing / next-generation sequencing /
clinical diagnostic → Healthcare
- punto vendita → Italian Retail
- electrolyser / electrolyzer / green hydrogen → Energy
6. Mojibake table extended for Western European compounds: ã/â/ê/î/ô
(Portuguese ã, French/PT â/ê/ô) plus uppercase variants.
Bug fixes from cross-language collisions
The audit pass exposed three short tokens that meant one thing in the
language they were added for and something completely different in another
language the classifier also targets:
- `por` (added as Luxembourgish for "parish" → Religion). Also the Spanish
and Portuguese preposition "for / by", which appears on roughly every
Spanish-language page. Was producing ~34 Religion false positives on
Mexican ISPs, Brazilian utilities, etc.
- `pura` (added as Indonesian/Sundanese/Balinese for "Hindu temple" →
Religion). Also the feminine of "pure" in Portuguese / Spanish / Italian,
and a frequent brand-name fragment ("Pura Energia", "Angkasa Pura").
Was misclassifying Brazilian electric utilities and Indonesian aviation
services.
- bare `broker` (added as Luxembourgish for Finance). Matched any English
text containing "broker" / "brokers" — including Cushman & Wakefield's
"real estate brokers" line, which forced the row into Finance instead
of Real Estate.
All three removed; AGENTS.md now codifies the rule.
AGENTS.md additions
- "Three output buckets" subsection: documents map / ambiguous / ku output
and how PRs should call out ambiguous review counts.
- "No taglines / slogans" rule: marketing copy ("we make it easy",
"smarter decisions") doesn't belong in any detector.
- "No ambiguous signals" rule: cross-category bare words (gazette / academy
/ society / club / studio) are forbidden as classifier keywords; use the
pinning compound instead. Same rule applies in every language.
- "Cross-language grammar / lexical overlap" rule: short tokens that mean X
in language A often mean a function word / adjective / brand fragment in
language B. Cites the por / pura / broker incidents.
- "Classify by what the operator literally provides" rule: clusters by
acronym suffix (UCaaS / CCaaS / CPaaS) tempt mis-grouping; CCaaS is SaaS
not ISP, etc. Includes the root-cause analysis of the
contact-center-as-ISP mistake.
- "Genuinely-ambiguous-between-two-types" rule: phrases like
"energy management software" that fit equally on a SaaS startup, an
Industrial conglomerate, and a consultancy belong in NO detector — leave
the row unmapped and rely on more-specific compounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Triage 78 ambiguous rows + new classifier filters and rules
Interactive triage of the 78 rows the v1 classifier surfaced as ambiguous
(two or more distinct categories fired). Net result of this commit, on
top of the v1 promotions already in the branch:
- 74 ambiguous rows promoted to map with a human-adjudicated category
(and 10 of those with a corrected human-cleaned brand vs. the noisy
as_name / title-bleed the v1 classifier captured).
- 1 row dropped silently per the AGENTS.md adult-content rule.
- 3 rows kept in KU (personal projects, parked pages caught by the
classifier mid-triage that we then surface'd-then-confirmed).
Map: 37,566 → 37,640 (+74). KU: 32,399 → 32,324 (−75). Disjoint clean.
Three new classifier filters added during triage as recurring patterns
surfaced — these run before category detectors and short-circuit to KU
or DROP rather than letting the operator-typology detectors fire on
parking-page / personal-page / adult-page text:
1. PARKED_PAGE_RE — Media Temple "automatically generated default server
page", Hostinger Horizons, Apache default, parked-by-registrar pages,
"site has shut down", "has completed its journey". Cloudflare /
DDoS-Guard / "Are you a robot?" interstitials are explicitly NOT
filtered (they leave the TLD-signal path open for gov / edu / mil
sites that are bot-blocked).
2. PERSONAL_PROJECT_RE — "personal BGP project", "personal website and
CV", "homelab", "hobby project", "side project". Hobbyists running
their own ASN aren't commercial operators.
3. ADULT_CONTENT_RE — adult web design / adult-entertainment hosting /
xxx / escort directory etc. Returns a sentinel ("DROP", None) so the
caller drops the domain from both map and KU per the AGENTS.md
content rule.
The classifier API now also writes a fourth output file (--dropped-out)
listing domains the adult-content filter caught, so the caller can
remove them from any tracked list files they currently sit in.
Title-noise list extended to catch: "attention required" / "are you a
robot" / "checking your browser" / "please enable javascript" /
"ddos-guard" / "px-captcha" / "site is not available" / "page is not
available" / "access to this page has been denied". This stops these
strings from bleeding into the brand column when TLD-only classification
fires (the `health.gov.il → "Attention Required!"` shape of bug).
Several cross-language false positives caught during the triage — same
shape as the por / pura / broker incidents the previous commit fixed:
- bare French `e?mailing` matched "Mailing Solutions" (mail-server
infrastructure on a Cisco VAR's product list, not marketing). Required
to start with `e` to keep the email-marketing meaning while losing the
bare-mailing collision.
- Norwegian / Danish bare `avis` (newspaper) matched "Avis Romania" car
rental and any French text saying "avis" (notice/opinion). Replaced
with compound forms (`dagsavis`, `lokalavis`, `morgenavis`, etc.).
- Vietnamese bare `bộ` (ministry) matched "bộ phim" (movie set), "bộ
sưu tập" (collection), and the founding-text references on Vietnam
Eximbank's about page. Replaced with compound forms (`bộ trưởng`, `bộ
tài chính`, `bộ ngoại giao`, etc.).
- Russian bare `провайдер` (provider) matched "хостинг провайдер"
(hosting provider, Web Host) on a Tajikistan domain registrar. Removed
the bare form; only the internet-specific compounds remain.
- Luxembourgish bare `broker` (Finance) matched "real estate brokers"
on Cushman & Wakefield's homepage and any English page mentioning
brokers. Removed the bare form entirely.
- Turkish bare `vakıf` (foundation) matched "Vakıf Katılım Bankası" —
for-profit Islamic-finance bank whose brand uses the word. Replaced
with nonprofit-specific compounds (`yardım vakfı`, `hayır vakfı`,
`kamu yararına vakıf`).
New positive-classification keywords added based on triage gaps:
- MSP rescue path now matches the SMB-IT-shop idiom in Polish
(`usługi IT dla biznesu`, `obsługa informatyczna firm`,
`outsourcing IT`), Spanish (`servicios informáticos para empresas`),
German (`IT-Dienstleister für`, `managed-IT-services`), French
(`infogérance`, `prestataire de services informatiques`), Italian
(`servizi informatici gestiti`, `outsourcing informatico`),
Portuguese (`serviços de TI gerenciados`, `terceirização de TI`),
Dutch (`beheerd-IT`, `IT-beheer`), and Indonesian
(`penyedia solusi IT`, `solusi IT terpadu/berbasis`).
- Finance now matches `accounting firm` / `cpa firm` /
`certified public accountants` / `chartered accountants` /
`tax preparation` / `tax advisory` / `audit firm` plus equivalents in
Spanish, Portuguese, French, German, Italian, and Polish.
- SaaS now matches CCaaS / CPaaS / `contact-center-as-a-service` /
`communications-platform-as-a-service` / `compliance software` /
`regulatory management software` and CCaaS no longer lives in ISP
(carryover from the user-flagged "contact centers are not ISPs"
correction).
AGENTS.md additions:
- "Triage heuristics learned from the 78-row interactive review of
PR #766's ambiguous bucket" subsection codifying every adjudication
rule the user applied during the review:
* pick the main-focus category (first / most-mentioned)
* clients are not operator typology
* vertically-specialized firms take the vertical
* stream-hosting infrastructure is Web Host
* multi-service SMB IT shops are MSP
* VARs are Technology
* CCaaS / CPaaS / UCaaS are SaaS
* gov/edu/mil/jus TLD signal trumps Cloudflare interstitials
* esports tournament organizers are Entertainment
* personal projects / parked pages / adult content go to KU or DROP
* brand quality is its own dimension — capture corrected brand
during triage rather than shipping the noisy as_name
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sean Whalen
|
06d277686d |
classify_unknown_domains.py: enforce concept-parity across ~60 languages (#765)
Multilingual detectors previously held English at full breadth (e.g. Healthcare = hospital + clinic + pharmacy + healthcare + pharmaceutical industry + nursing home + medical center) while many non-English sections covered the same concept set with only one or two transliterated words. This left every language other than English under-detecting against pages that used the operator's natural compound terms. Reworked every detector so each language now expresses the same English concept set in idiomatic compounds — never inventing calques where no natural form exists. Added ~32 new languages (Macedonian, Belarusian, Azerbaijani, Armenian, Georgian, Kazakh, Uzbek, Mongolian, Khmer, Burmese, Lao, Nepali, Sinhala, Amharic, Yoruba, Hausa, Igbo, Zulu, Pashto, Kurdish, Tajik, Kyrgyz, Maltese, Luxembourgish, Haitian Creole, Frisian, Yiddish, Faroese, Tatar, Javanese, Sundanese, Cebuano) on top of the existing pool, again applied per-concept rather than as token presence. Also added British / American spelling pairs where they diverge (`tire`/`tyre`, `defense`/`defence`, `center`/`centre`, etc.) and a handful of new English concepts that had been implicit (`tire shop`, `car parts`, `oil exploration`, `olympic committee`, ...) — each with its multilingual equivalents in the same edit. AGENTS.md: codified the rule under "Maintaining the reverse DNS maps" so future edits are bound by it: every language section must cover the same concept set the English section covers, with idiomatic compounds rather than calques, skip rather than invent when no natural form exists, and any new English keyword must be added in parallel across the existing language set. Final shape: 11,777 alternations / 175,556 chars across 45 detectors. Ruff check + format clean. Module compiles. Known limitation (pre-existing, unchanged): Python's `re` does not treat Unicode Mn / Mc combining marks as word characters, so Brahmic-script words ending in vowel signs / virama won't match the outer `\b…\b`. Affects pre-existing and new entries equally; fixable later by switching to the `regex` module. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
3b705aeaa8 |
Commit classify_unknown_domains.py — regex-based multilingual classifier (#764)
* Commit classify_unknown_domains.py: regex-based multilingual classifier
Promotes the transient `/tmp/classify_b<N>.py` script that grew across
the b5–b13 reverse-DNS-map batches into a tracked tool. The classifier
takes a `collect_domain_info.py` TSV and emits a CSV of map additions
plus a text file of known-unknown additions — the regex baseline that
makes step 4 of the unknown-domain workflow ("classify from the TSV, not
by re-fetching") tractable at scale.
Coverage:
- Detectors for all 44 industry types in the README.
- Concept-translation parity across ~30 languages on the high-volume
detectors (Healthcare, Travel, Government, Retail, Finance, ISP, Web
Host, Manufacturing, Logistics, Real Estate, Automotive, Legal,
Agriculture).
- ~10–20 languages with 1–3 keywords each on the smaller detectors
(Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media,
Defense, IaaS/PaaS/SaaS, Beauty, Print, Publishing, Religion, Science,
Event Planning, Staffing, Email Security/Provider, Marketing,
Construction, Industrial, Utilities, Energy, Government Media,
Physical Security, News, Nonprofit, Entertainment, Technology,
Consulting).
Brand-name selection prefers MMDB `as_name` → page title's first
segment → non-redacted WHOIS registrant → domain-derived fallback, with
a `clean_brand` pass that strips legal-form suffixes (LLC / GmbH / Ltda
/ EIRELI / sp. z o.o. / s.c.a r.l / UAB / etc.) and prefixes (PT, OOO).
When the title has multiple segments, the segment whose simplified form
contains the domain root is preferred — accessmontana.com with as_name
"MONTANA WEST, L.L.C." and title "Internet, Phone & TV Bundles | Access
Montana" maps to "Access Montana", not "Montana West".
A small mojibake fixer normalizes the most common UTF-8-as-Latin-1
re-encodings ("ó" → "ó", etc.) so Spanish/Portuguese/French homepages
that `collect_domain_info.py` mishandled still classify.
The empty HAND dict at the top of the file is an extension point for
batch-specific overrides — e.g. acquisition aliases or brand-name
corrections that don't fit any detector; each `domain → ("Brand",
"Type")` entry wins over the auto-classifier.
Wired into AGENTS.md's "Related utility scripts" section and documented
in `parsedmarc/resources/maps/README.md` alongside the rest of the
maps utilities.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* classify_unknown_domains.py: clarify dual-purpose framing
The classifier serves both lookup paths into base_reverse_dns_map.csv —
the original PTR-side flow (reverse-DNS base domains derived from DMARC
report source IPs) and the MMDB-coverage flow (AS domains lifted from
the bundled IPinfo Lite MMDB). The initial commit's docstring/comments
emphasized the MMDB-coverage flow because that's where the script grew
up across the b5–b13 batches, but it was always equally applicable to
PTR-side domains.
Updates:
- Top docstring rewritten to lead with the dual-purpose framing.
- README.md adds an explicit "useful for either lookup path" paragraph
referencing the original DMARC-report flow and the MMDB-coverage flow.
- AGENTS.md "Related utility scripts" entry updated to mention both
flows.
- Drops a stale "happen to have ASN registrations" aside in the
RETAIL_RE comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sean Whalen
|
9aa930f7cc |
Retroactive promotions: 3,171 KU rows reclassified by expanded multilingual classifier (#763)
Re-ran the expanded-multilingual classifier (PR #762's classifier with broader language coverage on Healthcare, Travel, Government, Retail, Finance, ISP, Manufacturing, Logistics, Real Estate, Automotive, Legal, Agriculture, plus Finance-via-body-text catching insurance/investment/ asset-management) against every cached TSV from prior batches (b6–b13). 3,171 domains that previously couldn't be auto-classified (and were therefore added to known_unknown_base_reverse_dns.txt) now match the new detectors. These domains are promoted out of KU and into the map under their newly classified `(name, type)` pairs. Type distribution of promotions: Finance 736 Logistics 179 Real Estate 105 Healthcare 68 ISP 323 Retail 159 Education 110 Marketing 66 Manufacturing 207 Technology 142 Consulting 99 Nonprofit 64 Government 136 Travel 123 Utilities 71 Legal 53 + smaller volumes across ~25 other industry types ASN-domain coverage of the bundled IPinfo Lite MMDB after these promotions: - by domain count: 32,254 / 63,993 (50.40%, up from 45.45%) - by IPv4 weight: 98.45% Honest scope note: the multilingual classifier achieves "concept parity" for the top-5 high-volume detectors (Healthcare, Travel, Government, Retail, Finance) across ~30 languages. Smaller detectors (Photography, Conglomerate, Sports, Defense, MSSP, IaaS/PaaS/SaaS, etc.) still have ~10-15 languages with 1-3 keywords each. Further per-detector multilingual parity is a follow-up effort; each subsequent expansion recovers fewer domains as the long tail of language-specific phrasings shrinks. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
c25bf28c1c |
Classify reverse DNS map: final cleanup batch (~2,650 unmapped MMDB ASN domains) (#762)
Final cleanup pass to clear the remaining MMDB AS-domain queue. Applied an expanded multilingual classifier covering all 44 README industry types plus an Energy concept (mapped to Utilities pending a README addition). Per-detector keyword lists now include Spanish, Portuguese, French, Italian, German, Dutch, Russian, Polish, Czech, Turkish, Greek, Chinese (simplified and traditional), Japanese, Korean, Arabic, Hebrew, Hindi, Vietnamese, Indonesian, and Thai where the concept has a recognizable local-language equivalent. - 980 added to base_reverse_dns_map.csv (ISP 193, Education 193, Finance 155, Government 109, Healthcare 93, Web Host 37, MSP 31, Manufacturing 22, Logistics 17, Real Estate 12, Travel 11, Consulting 10, Tech 9, Nonprofit 9, Legal 9, Food 9, Retail 8, Religion 8, Utilities 7, plus smaller volumes across 14 more types). - 1,669 added to known_unknown_base_reverse_dns.txt — the residual unfetchable / parked / Cloudflare-challenged / non-recognized-content rows. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 29,083 / 63,993 (45.45%) - by IPv4 weight: 98.36% Total since batch 5: ~16,400 map rows + ~17,400 known-unknown rows added across 9 batches. Remaining unmapped pool size: 0 — every MMDB AS-domain has now been processed (either classified or recorded in known-unknown). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
fa03b8f2c2 |
Classify reverse DNS map: next 10000 unmapped MMDB ASN domains (#761)
Batch 12. Auto-rate dropped to 23% (2330/9998) — significantly lower than batch 11. The deeper into the long tail, the more candidates fall into non-classifier-recognized industries (retail, manufacturing, hospitality, local services) where the ISP/Web Host/MSP regex doesn't fire even though the page is fetchable. - 2,330 added to base_reverse_dns_map.csv (ISP 991, Education 295, Finance 290, Government 265, Web Host 229, Healthcare 134, MSP 126). - 7,667 added to known_unknown_base_reverse_dns.txt. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 27,824 / 63,993 (43.48%, up from 40.27%) - by IPv4 weight: 98.36% Same classifier as prior batches (no new code). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
e6716c9e80 |
Classify reverse DNS map: next 10000 unmapped MMDB ASN domains (#760)
Batch 11. Auto-rate 30.6% — slightly lower than batch 10's 35.6%, consistent with continuing long-tail descent. - 3,063 added to base_reverse_dns_map.csv (ISP 1,734, Finance 270, Web Host 254, Education 253, Government 249, MSP 153, Healthcare 150). - 6,934 added to known_unknown_base_reverse_dns.txt. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 25,773 / 63,993 (40.27%, up from 35.49%) - by IPv4 weight: 98.32% (up from 98.27%) Same classifier as prior batches (no new code). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
ee9bda7228 |
Classify reverse DNS map: next 10000 unmapped MMDB ASN domains (#759)
First of the 10K-batch series toward complete coverage. Auto-rate 35.6% this round, consistent with the long-tail descent. - 3,556 added to base_reverse_dns_map.csv (ISP 2,410, Web Host 348, Education 247, Finance 201, Government 124, MSP 121, Healthcare 105). - 6,442 added to known_unknown_base_reverse_dns.txt. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 22,710 / 63,993 (35.49%, up from 29.93%) - by IPv4 weight: 98.27% (up from 98.18%) Same classifier as batches 5-9 (no new code). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
80a132801d |
Classify reverse DNS map: next 5000 unmapped MMDB ASN domains (#758)
Auto-classification rate jumped back to 50% (2502/4999) from 36.5% in batch 8 — this slice happens to contain a higher proportion of small ISPs with conventional homepages, lifting the regex hit rate. - 2,502 added to base_reverse_dns_map.csv (ISP 2,065, Web Host 133, Education 96, Finance 67, MSP 60, Government 58, Healthcare 23). - 2,496 added to known_unknown_base_reverse_dns.txt. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 19,154 / 63,993 (29.93%, up from 26.02%) - by IPv4 weight: 98.18% (up from 98.09%) Same classifier as batches 5-8 (no new code). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
c523d0da9c |
Classify reverse DNS map: next 5000 unmapped MMDB ASN domains (#757)
Continuing the long-tail sweep. Auto-classification rate dropped to 36.5% this round (1826/5000) from ~43% in prior batches — the further into the tail we go, the more candidates have parked / Cloudflare-challenged / sparse homepages where the regex can't match. - 1,826 added to base_reverse_dns_map.csv (ISP 1,187, Web Host 267, Education 112, MSP 80, Finance 65, Government 62, Healthcare 53). - 3,174 added to known_unknown_base_reverse_dns.txt. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 16,652 / 63,993 (26.02%, up from 23.17%) - by IPv4 weight: 98.09% (up from 98.01%) Same classifier as batches 5-7 (no new code). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
4446702b84 |
Classify reverse DNS map: next 5000 unmapped MMDB ASN domains (#756)
Continuing the long-tail sweep toward complete ASN-domain coverage. This batch's candidates each represent only 2,048-3,072 IPv4 addresses (smaller than batch 6's 3K-6K range) so by-weight gains are diminishing, but each classified operator is one more small ISP / web host the project can name in DMARC reports. - 2,148 added to base_reverse_dns_map.csv (ISP 1,605, Web Host 236, Education 126, Government 56, MSP 46, Finance 43, Healthcare 36). - 2,852 added to known_unknown_base_reverse_dns.txt — homepages that were parked / Cloudflare-challenged / generic-server-test pages, in obscure-language without telecom-keyword cognates the classifier recognized, or rows whose WHOIS / MMDB as_name / homepage couldn't combine into two corroborating sources. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 14,826 / 63,993 (23.17%, up from 19.81%) - by IPv4 weight: 98.01% (up from 97.85%) Same classifier as batch 6 (no new code). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
7ef153b4da |
Classify reverse DNS map: next 5000 unmapped MMDB ASN domains (#755)
5x the typical batch size to chase complete ASN-domain coverage. Small ISPs and web hosts are high-value targets for spam/phishing abuse, so the long tail of unmapped operators is worth investing review effort in. Each candidate at this depth represents 3,072–6,144 IPv4 addresses (well below the 100K+ that head-batches saw); auto-classification rate is 43.5%, similar to the prior batch. - 2,177 added to base_reverse_dns_map.csv (ISP 1,477, Web Host 296, Education 214, MSP 65, Government 56, Healthcare 40, Finance 29). - 2,823 added to known_unknown_base_reverse_dns.txt — parked / Cloudflare- challenged / generic-server-test pages, obscure-language homepages without telecom-keyword cognates the classifier recognized, or rows whose WHOIS / MMDB as_name / homepage couldn't combine into two corroborating sources. ASN-domain coverage of the bundled IPinfo Lite MMDB after this batch: - by domain count: 12,678 / 63,993 (19.81%, up from 15.86%) - by IPv4 weight: 97.85% (up from 97.55%) Reused the batch-5 classifier (MMDB as_name as primary brand source with domain-root-aware title-segment selection, multilingual ISP/Web Host/MSP keyword regex, government and education TLD lists, Communications-with- media-context-guard fallback, and the deep brand-suffix cleanup for EPP/EIRELI/UAB/Druzstvo/etc. plus the UTF-8-as-Latin-1 mojibake fix). No new classifier changes this batch. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
34518585b6 |
Classify reverse DNS map: next 1000 unmapped MMDB ASN domains (#754)
The next 1000 by aggregate IPv4 weight, all sitting in the long tail (each
candidate ASN holds ~7,400 IPv4 addresses, ~0.21% of total v4 weight), so
auto-classification rate is modest compared to head-batches:
- 460 added to base_reverse_dns_map.csv (ISP 344, Web Host 60, Education 21,
MSP 12, Healthcare 8, Government 8, Finance 7).
- 540 added to known_unknown_base_reverse_dns.txt — homepages that were
parked, behind a Cloudflare bot challenge, returning a generic-server test
page, in obscure languages with no telecom-keyword cognates the classifier
recognized, or whose WHOIS / MMDB as_name didn't combine with any
homepage signal to clear two corroborating sources.
Classifier improvements applied this batch (relative to prior batches' code):
- MMDB as_name is the primary brand source, with cleaned title as fallback
and domain-derived as last resort (WHOIS is mostly privacy-redacted at
this depth in the long tail).
- Title-segment selection now prefers the segment whose simplified form
contains the domain root, catching cases like accessmontana.com whose
as_name is the holding company "MONTANA WEST, L.L.C." but whose title
surfaces the operator brand "Access Montana".
- as_name fallback for ISP added "Communications" (with a media-context
guard so "Christian Broadcasting Network" doesn't hit) plus bare
"Internet" / "Cable" / "Telephone Co." patterns common in rural-US ISP
brands.
- Government TLD list expanded for .go.id, .gv.at, .gov.cn, .gob.cl/ar/gt,
.admin.ch, etc.; Education TLD list expanded for .ac.kr / .ac.za /
.ac.nz / .edu.cn / .edu.tw / .edu.sg / .edu.my / .edu.ph / .edu.eg.
- MSP detection re-added (`it solutions` / `managed it support` /
`managed tech` patterns) for marconet.com / odyssey.uk / vmi.se type
long-tail managed-IT shops.
- Brand cleanup deepened to handle Brazilian EPP / EIRELI ME, Italian
s.c.a r.l, Polish sp z o.o variants, Lithuanian UAB, Czech Druzstvo,
Venezuelan C.A., trailing-single-letter artifacts, and double-spaces.
- Encoding-mojibake fixer for the common UTF-8-as-Latin-1 cases
("Fibra óptica" → "Fibra óptica") so Spanish/Portuguese ISP pages
classify even when collect_domain_info.py mishandled the encoding.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sean Whalen
|
769b16bb03 |
Drift-detect rebrands: tighten regex; promote 11 verified rebrand-aliased map keys (#753)
* Tighten rebrand regex to drop CTA, third-party-mention, and CSS-asset FPs The first run of detect_rebrands.py against the live map surfaced systemic false-positive categories that drowned the real signals. Tightening over two rounds of FP triage: REBRAND_RE — drop bare "now <Cap>" and "joined the X" branches: - "Buy Now PROMO", "Apply Now Who", "Order Now Free Shipping" — modern marketing pages saturate body text with CTA fragments and ~95% of bare "now <Capital>" matches were these. Replaced with the linguistically meaningful pattern "(is|are|was|were|am) now (?:(?:a )?part of)?" which still catches "BankOnIT is now Navanta", "We are now Cencora", "is now part of Lumen", etc. - "joined the Festo Certified System Integrator Program", "joined the ClimateCAP Initiative", "joined the Fredonia Women's Rugby team" — the "joined the X" pattern was too generic; real "joined the X family" rebrand banners are rare enough that dropping the branch is the right trade. REBRAND_RE — add `\b` word boundary at the start so triggers don't match mid-word: "Stre*am* now Mystery" was matching `am now <Cap>` because the last two letters of "Stream" satisfied the verb alternation. REBRAND_PATH_RE — drop bare `rebrand`, `name change`, `new name for`, and `brand-update` / `brand-refresh` patterns. They appeared too often as CSS class names (`class="rebrand-page"`), CSS variables (`--rebrand-underline-color`), image filenames (`bms-rebrand-logo.svg`, `brand-update.css`), and JSON/JS strings (`"name change"` user-account labels). Adding `\b` boundaries doesn't help because dashes are non-word characters. The remaining narrow patterns (`brand-launch`, `brand-announcement`, `brand-reveal`, `our-new-name`, `our-new-brand`, `acquisition-announcement`, `merger-announcement`) still catch the canonical bankonitusa.com case via its `brand-launch-frequently-asked- questions` URL slug and `Brand announcement` alt text. _REBRAND_NOISE — make the comparison case-insensitive and add "included", "iso", "secure", "part" to suppress "is now ON" / "is now LIVE" / "is now ISO 27001 certified" / "is now Secure Managed Wi-Fi" / "is now Part of" patterns. Twitter/Facebook/Square (the social-platform rebrand mentions in footers like "X (formerly Twitter)") moved to lowercase since the comparison is now case-insensitive. Net effect on a full sweep over the ~13,100-key map: rebrand-signal flagged-row count dropped from ~270 (initial run) to 108 (round-3), clearing the dominant FP categories while every real signal — verified against the bankonitusa.com canonical case plus 11 other actual rebrands — still fires. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Promote 11 verified rebrands found by drift sweep; alias 4 acquirer domains Renames produced by `detect_rebrands.py` running against the full ~13,100-key map and verified by re-reading each operator's homepage. Type column unchanged for every row — only the canonical `name` shifts to the new operator. Where the new operator's primary domain wasn't already in the map, a case-1 alias row is added pointing to the same `(name, type)`. Renames: - amerisourcebergen.com: AMERISOURCEBERGEN → Cencora - aurorahealthcare.org: Aurora Health Care → Advocate Health - consolidated.com: Consolidated Communications → Fidium Fiber - databridgesites.com: Meridian Parkway Data Center Owner → TierPoint - emarsys.com: SAP Emarsys → SAP Engagement Cloud - rig.net: RigNet → Viasat - rxlightning.com: RxLightning → CoverMyMeds - telepoint.bg: Telepoint → Digital Realty - thehostgroup.com: The Host Group → HostGo - ultisat.com: Globecomm Services Maryland → UltiSat - unifiedpostgroup.com: Unifiedpost Group → Banqup New aliases (operator's primary domain not previously mapped): - cencora.com → Cencora, Healthcare - advocatehealth.com → Advocate Health, Healthcare - covermymeds.com → CoverMyMeds, Healthcare - banqup.com → Banqup, SaaS Five sweep hits intentionally deferred for lack of a clear second source: megatel.co.nz → Nova (`nova.co.nz` is for sale via a domain broker; unclear which Nova entity), pogozone.com → NeuBeam (NeuBeam's homepage doesn't acknowledge the PogoZone acquisition), prempub.com → Ingenious Media (ingeniousmedia.com fetch failed), voltagepark.com → ? (merger with Lightning AI rather than a clean rebrand), and a handful of more ambiguous Synopsys/Ansys/OmniAccess/Rakuten/Indigital/Synthite signals that need manual research. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document detect_rebrands.py cadence as run-once-a-year The drift sweep is for catching operator rebrands and acquisitions that accumulated since the previous run; M&A activity over the mapped operator set is slow enough that yearly is sufficient. Annotate the script's own docstring, the maps README, and the AGENTS.md "Related utility scripts" entry so a future contributor doesn't mistake it for a per-batch step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
c752e776de |
Detect map-key rebrands via homepage drift sweep (#752)
Adds two complementary pieces of M&A drift detection over base_reverse_dns_map.csv:
- `collect_domain_info.py` gains two derived columns. `rebrand_signal` combines
a body-text regex ("now X" / "formerly known as X" / "we became X" / ...)
with a narrow path-and-alt-text regex ("rebrand", "brand-launch",
"brand-announcement", "name-change", "our-new-name", ...) that runs against
the JSON-unescaped page bytes, so URL slugs and image alt attributes inside
Elementor / hydration script blobs are reachable. The two-regex split is
what catches image-only acquisition banners like bankonitusa.com's "now
Navanta" — a `<a href="https://navanta.com/brand-launch-..."><img
alt="Brand announcement"></a>` with no visible text — that pure body-text
scanning misses. `external_links` collects the homepage's non-self,
non-social outbound link hosts as review context only.
- `detect_rebrands.py` is a new sibling drift sweep. It re-fetches every key
in base_reverse_dns_map.csv with the same fetch machinery, evaluates two
default flag triggers (`rebrand_signal` matched, or final URL host doesn't
sit under the input domain), and writes a compact TSV of just the flagged
rows. `external_links` is captured into the row as context but is not a
default trigger — most outbound links are to partners / customers / vendors,
and flagging them would flood review with noise. `--flag-external-links`
opts into that signal for thorough sweeps. Resume-safe via `-o`.
Output is review fodder, not automated map mutation: a single signal is one
corroborating source, and promoting a flagged row into the map still requires
a second source per the two-corroborating-sources rule.
README and AGENTS.md updated to document the new columns and script.
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sean Whalen
|
6fa561d172 |
Classify reverse DNS map: ~2,100 unmapped MMDB ASN domains; bankonitusa.com → Navanta (#751)
Adds ~2,125 ASN-domain classifications carried out across four ~1,000-domain batches in a prior session that wasn't pushed before #748/#749 merged. The overlap with those merged batches is dropped — origin/master's classifications are kept as authoritative — and only the genuinely-new domains land here. 188 known-unknown rows are promoted out to the map for the same reason. Also updates bankonitusa.com from BankOnIT to Navanta and adds navanta.com as an alias after a spot check observed the operator's "now Navanta" rebrand banner. Two corroborating sources: the banner on bankonitusa.com itself (image-only `<a href="https://navanta.com/brand-launch-..."><img alt="Brand announcement"></a>`) and the rebrand explainer on navanta.com ("Why We Became Navanta", "MyBankonIT has been rebranded to MyBPC"). The MMDB still names the pre-rebrand entity (BankOnIT, L.L.C.) — typical years-of-lag pattern. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
bf526f4e12 |
docs(AGENTS.md): require fresh branch off origin/master per batch (#750)
* docs(AGENTS.md): require fresh branch off origin/master per batch Add a "Starting the next batch" subsection to the reverse-DNS-maps workflow. Each batch must start from a fresh checkout of origin/master, not from the previous batch's branch. The trap: if the previous batch's commit has already merged via a PR pushed from elsewhere (a co-worker's session, an unsynced laptop, an earlier session), the local copy of that commit still sits on the old branch. Stacking new work on top makes the new PR conflict with master, because the merged commit and the local copy insert identical map rows at identical sorted positions and the same lines collide. Hit live this batch (PR #749) and recovered via `git rebase --onto origin/master <stale-commit> <branch>` plus a force-push, then a PR-description trim. Documenting the failure mode and the recovery so the next contributor avoids the trap entirely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(AGENTS.md): also check for open map PRs before starting a batch Add a pre-flight `gh pr list --search` step ahead of the branch-fresh- off-master rule. Same scenario in mind: a previous batch's PR is still in flight, started from a different machine or session, and starting a new batch in parallel duplicates effort or splits attention across two competing PRs touching the same files. Cheap one-liner; cost of forgetting it is the kind of conflict #749 already documented at the branch-hygiene level. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
7ef31f8083 |
Classify reverse DNS map: next 1000 unmapped MMDB ASN domains (#749)
Continued the MMDB ASN-domain coverage walk into the 14k-10k IPv4-weight band. Added 883 new map entries and 117 new known-unknown entries from the top 1,000 unmapped candidates. ASN-domain coverage by IPv4 weight: 96.5% -> 96.8%. ASN-domain coverage by domain count: 11.0% -> 12.4%. Composition: ~50 globally-known brands (Vanguard, AIG, Aon, Equifax, Mercedes-Benz USA, BP, BHP, Bechtel, Tetra Pak, Anheuser-Busch, Air Canada, Maersk, NFL, NHL, MGM Resorts, Wolfram, Red Hat, Palo Alto Networks, New Relic, Travelport, Epicor, IQVIA, Dassault Systemes, Disney+, Valve, Seagate, Analog Devices, Renesas, Dow Jones, Lee Enterprises, IGN, Mondadori, AtkinsRealis, Eiffage, Ogilvy, Interpublic, Equifax, Ooredoo Maldives, MTN Zambia, Movistar Costa Rica, Telekom Romania Mobile, Sparkle, Vodafone Ireland, etc.); ~30 universities and government / state agencies (City of San Jose, City of Phoenix, Bulgarian gov, Region Uppsala, Weld County, Long Beach Unified, Escambia School District, Region 4 ESC, Merced COE, Santa Cruz COE, Politechnika Warszawska, Bogazici, KAIST-affiliated Korean universities, Ural Federal University, etc.); the long tail of regional ISPs / hosters / MSPs / data-center operators classified via MMDB as_name + homepage / WHOIS corroboration. 117 added to known-unknown where the two-corroborating-sources bar wasn't met (Cloudflare-blocked sites with privacy-redacted WHOIS, generic-token AS-names with empty homepages, parked domains, etc.). Files remain disjoint per the workflow guardrail. sortlists.py validates clean (types, sort, dedupe). CRLF preserved. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
ab9d4e93f5 |
Classify reverse DNS map: next 1000 unmapped MMDB ASN domains (#748)
Continued the MMDB ASN-domain coverage walk into the 18k–14k IPv4-weight band. Added 941 new map entries and 59 new known-unknown entries from the top 1,000 unmapped candidates. ASN-domain coverage by IPv4 weight: 96.0% → 96.5%. ASN-domain coverage by domain count: 9.5% → 11.0%. Composition: ~50 universities and government / state agencies (HMRC, SSA, DHS, DOJ, BART, Pittsburgh, Charlotte, NY courts, Bank of Canada, MTA, gov.si, gov.ru, KAUST, Sharif University, Karolinska Institutet, IIT, KTH, etc.), ~70 globally-known brands (Nvidia, AMD, BMW, Mastercard, Nasdaq, NetApp, Allianz, Honeywell, JPMorgan, Goldman Sachs, Mitel, Arista, Take-Two, Universal Music, Disney Go, Fox, Nike, Cigna, Aetna, Humana, AbbVie, Mitsubishi Electric, Saint-Gobain, Reliance Industries, Hyundai Autoever, Square Enix, NEXON, Riot Games, Mahidol University, Hong Kong HSBC, Standard Chartered, etc.), and the long tail of regional ISPs / hosters / MSPs / data center operators classified via MMDB as_name + homepage corroboration. 59 added to known-unknown where the two-corroborating-sources bar wasn't met. Files remain disjoint per the workflow guardrail. sortlists.py validates clean (types, sort, dedupe). CRLF preserved. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
1fd833bbf0 |
Classify reverse DNS map: next 1000 unmapped MMDB ASN domains (#747)
Continued the MMDB ASN-domain coverage walk into the 30k–18k IPv4-weight band. Added 971 new map entries and 32 new known-unknown entries from the top 1,000 unmapped candidates. ASN-domain coverage by IPv4 weight: 95.3% → 96.0%. ASN-domain coverage by domain count: 8.0% → 9.5%. Composition: ~30 universities and government / state agencies (maryland.gov, ok.gov, nj.gov, NIA Korea, NICTEC Thailand, etc.), ~80 globally-known brands (Nvidia, Tesla, Intel, Ford, GM, Volvo, Disney, EA, Roblox, Riot Games, Sony PlayStation, JPMorgan, Goldman Sachs, Morgan Stanley, Charles Schwab, AXA, Cigna, Cargill, Hallmark, Pepsi, Kroger, Random House, NBCUniversal, Qualcomm, Deutsche Bank, UBS, Citi, Lloyds Banking, Westpac, CommBank, Adobe, Broadcom, NXP, Schaeffler, Saint-Gobain, Hanwha, Doosan, Hyundai Autoever, Square Enix, Garena, etc.), and the long tail of regional ISPs / hosters / MSPs classified via MMDB as_name + homepage corroboration. 1 entry promoted out of known_unknown_base_reverse_dns.txt; 32 added where the two-corroborating-sources bar still wasn't met. Files remain disjoint per the workflow guardrail. sortlists.py validates clean (types, sort, dedupe). CRLF preserved. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
05adb9c831 |
Classify reverse DNS map: top ~1,950 unmapped MMDB ASN domains (#746)
Walked the bundled IPinfo Lite MMDB for ASN domains absent from base_reverse_dns_map.csv, processed the top ~1,950 by IPv4 weight across five batches (collect_domain_info.py + tier-based classification per AGENTS.md), and added 1,664 new map entries. ASN-domain coverage by IPv4 weight: ~92.9% → 95.3%. ASN-domain coverage by domain count: 5.4% → 8.0%. Composition: ~250 universities/government (Tier 0 — restricted TLD + MMDB as_name), ~80 globally-known brands (Saudi Telecom, JAXA, RailTel, LY Corporation, Tesla, Intel, Citi, Schwab, Disney, EA, Volvo, Mitsubishi Electric, Cargill, Hallmark, Medtronic, Banco do Brasil, Petrobras, etc.), direct aliases for already-mapped brands (HKBN, Tata Teleservices, Cox, NTT, T-Mobile, etc.), and the long tail of regional ISPs / hosters / DC operators classified via MMDB as_name + homepage corroboration. 66 entries promoted out of known_unknown_base_reverse_dns.txt where the new collector data cleared the two-corroborating-sources bar; 55 added where the bar still wasn't met. Files remain disjoint per the workflow guardrail. sortlists.py validates clean (types, sort, dedupe). CRLF preserved. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
7ba078bff1 |
Translate AS-name source rows via MMDB; classify reverse DNS batch (#745)
* feat(maps): translate AS-name source rows via MMDB When parsedmarc's ASN-fallback path in utils.get_ip_address_info surfaces a raw MMDB as_name (e.g. "Vodafone Group PLC") for an IP that has no PTR and whose as_domain isn't in the map, find_unknown_base_reverse_dns.py now looks the as_name up in the bundled ipinfo_lite.mmdb and substitutes the matching as_domain so the row enters the unknown pipeline as a researchable domain instead of being dropped or polluting the list. Normalize non-breaking spaces (U+00A0) and runs of whitespace when building and querying the as_name index — the source CSV and MMDB disagree on NBSP placement for several names (e.g. "UDomain\xa0Web Hosting Company Ltd" in the CSV vs. "UDomain Web Hosting Company Ltd" in the MMDB), causing exact-match lookups to miss otherwise-identical entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(maps): classify a batch of unknown reverse DNS base domains 40 map additions (35 source domains + 5 redirect-target/promotion aliases) and 35 known-unknown additions, covering the 71-entry unknown_base_reverse_dns.csv refresh. Newly mapped operators include several MMDB-AS-translated regional ISPs (Babilon-T/TJ, MegaFon Tajikistan, Ucell, Ufone, PinPro, Teraline Telecom, Transtelecom Kazakhstan, Satis, AlmaTV, Radius-NET, Burlington Telecom), aliases of existing brands (Telstra/bigpond.net.au, UDomain/udomain.hk, AG Telekom/katv1.net, EWE/ewe-ip-backbone.de, Hostinger/hstgr.cloud, Docusign/docusign.net, Brevo/sp2-brevo.net, MegaFon/megafon.tj, Beeline/beeline.uz), Tier-0 brands (Visa, Tripster, Verde Agritech), one healthcare entry (Sanwakai Hospital), one government entry (Special Communication Service of Azerbaijan), one education entry (KazRENA), and an MSP (Otava). Redirect-target aliases added for burlingtontelecom.com, alma.plus, cn.at, and teraline-telecom.net per the post-batch sweep rule. fea.net promoted out of known-unknown to West Coast Internet (WCI) after its homepage redirect-target was already mapped. Domains with single-source corroboration (privacy WHOIS plus unreachable site, parked-domain pages, ambiguous categorizations) went to known_unknown_base_reverse_dns.txt rather than the map. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
 Sean Whalen
|
6ff6261df9 | docs: update installation instructions for IPinfo Lite and MaxMind GeoLite2 databases | ||
|
Sean Whalen
|
06fd3f2b09 | docs: update installation instructions and usage notes for parsedmarc | ||
|
github-actions[bot]
|
7ba8a1d10f |
chore: update IPinfo Lite MMDB (#744)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean Whalen
|
02a8014893 | Fix splunk SMTP TLS dashboard: add additional renames for failure details and adjust stats query | ||
|
Sean Whalen
|
8317ffcde8 | Fix rename syntax for parsed_sample headers in Splunk DMARC forensic dashboard | ||
|
Sean Whalen
|
3b9e678533 |
Refactor SMTP TLS dashboard with base search
Refactored the SMTP TLS Splunk dashboard to use a base search for improved query efficiency and maintainability. Updated input token names and adjusted search queries for better organization and clarity. |
||
|
Sean Whalen
|
5ba72d2783 | Add source AS name to fillnull and search queries in DMARC aggregate dashboard | ||
|
Sean Whalen
|
e40b53da64 | Enhance Splunk DMARC aggregate dashboard: add source AS name dropdown and update search queries | ||
|
Sean Whalen
|
fe296ca869 |
Update dashboard documentation
- Introduced a new README.md for dashboard development with detailed instructions. - Removed outdated README files for Grafana and Splunk dashboards. |
||
|
Sean Whalen
|
397378de8e |
Bump mailsuite to >=2.0.2 for 9.11.1 release (#743)
Addresses RuntimeError: Event loop is closed in the MS Graph mailbox backend (#742). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9.11.1 |
||
|
Sean Whalen
|
5d816a4e56 |
Offload mailbox layer to mailsuite>=2.0.0 (#741)
mailsuite 2.0.0 extracted the IMAP, Microsoft Graph, Gmail, and Maildir connections out of parsedmarc into mailsuite.mailbox so other projects can reuse the same provider-agnostic interface. Replace the parsedmarc/mail submodules with a thin re-export of mailsuite.mailbox and drop the duplicated implementations. Per the migration note in seanthegeek/mailsuite#22, pass token_cache_name="parsedmarc" so existing AuthenticationRecord caches on disk continue to work without re-prompting users to authenticate. The existing graph_url config knob is forwarded unchanged. Drop direct dependencies that are now installed transitively via mailsuite[gmail,msgraph] (msgraph-core, imapclient, google-*). The extras are pulled in non-optionally so Gmail and Microsoft Graph support remain available out of the box. Drop nine test classes that were exercising mailsuite-side implementation internals (TestGmailConnection, TestGraphConnection, TestImapConnection, the _get_creds/_generate_credential half of TestGmailAuthModes, TestImapFallbacks, TestMSGraphFolderFallback, TestMaildirConnection, TestMaildirReportsFolder, TestMaildirUidHandling, TestTokenParentDirCreation); these are mailsuite's tests now. The CLI integration tests that mock parsedmarc.cli.{IMAP,Gmail,MSGraph}Connection are kept. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9.11.0 |
||
|
Sean Whalen
|
900ee22525 | Make map and country list side by side in the Splunk DMARC aggregate dashboard XML | ||
|
Sean Whalen
|
e709839f79 | Fix typo in source ip viz | ||
|
Sean Whalen
|
e7f6e1b5e7 | Update map files | ||
|
Sean Whalen
|
26f54b1269 | Add content rule to exclude adult websites from domain lists | ||
|
Sean Whalen
|
44fd1aa555 |
Coerce malformed <email> in aggregate report metadata to None (#740)
xmltodict turns stray angle brackets in <email> (e.g. "<bad-xml@bad-xml.net>") into a nested dict, which then flows through parse_aggregate_report_xml as the org_email value. Parsing succeeds, but Elasticsearch / OpenSearch reject the document at index time because the org_email mapping is text — observed as document_parsing_exception / mapper_parsing_exception with a "{#text=..., bad-xml=null}" preview. When report_metadata["email"] comes back as a dict, log it at debug and discard. The rest of the report still ingests with org_email=None instead of failing the whole document downstream. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
github-actions[bot]
|
f3a2e894e0 |
chore: update IPinfo Lite MMDB (#739)
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
Sean Whalen
|
265bf64240 |
Align Grafana dashboard with OpenSearch Dashboards source-of-truth (#738)
* Align Grafana dashboard with OpenSearch Dashboards source-of-truth Adds the two aggregate-DMARC panels that exist on the OSD dashboard but were missing from the bundled Grafana dashboard: - "Message sources by name and type" — buckets by source_name + source_type, sums message_count per (name, type) tuple. Mirrors the OSD viz from 9.4.x. - "Message sources by Autonomous System" — buckets by source_asn + source_as_name + source_as_domain, sums message_count per ASN. Mirrors the OSD viz added in 9.9.0 with the IPinfo Lite ASN integration. Both panels are patterned on the existing "Reporting Organisations" panel (same datasource $datasourceag, same sum(message_count) metric, same gradient-gauge "Messages" column with rename transforms). They sit at the bottom of the existing layout (gridPos y=129 and y=140) so the existing panel positions are unchanged. Verified against the bundled grafana/grafana:12.3.0: dashboard import returns status=success, both panels render with real data from the sample-corpus indexes, and the ES aggregations (terms on source_name + source_type, numeric terms on source_asn) return the expected results. Out of scope: - Extras in the Grafana dashboard that aren't on OSD (SPF/DKIM Results Over Time, Alignment Over Time, Stat overview, Published Policies, Forensic IP / country tables) are left in place. They were community-contributed and likely valued by some users. - Migrating the deprecated `graph` and `grafana-worldmap-panel` panel types to modern timeseries / geomap is a separate, larger task. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Grafana: migrate deprecated graph and worldmap panels Replaces the 6 legacy `graph` panels with `timeseries` panels and the 2 legacy `grafana-worldmap-panel` panels with `geomap` panels. Both deprecated plugins still rendered in Grafana 12 via auto-migration but were flagged for removal; this ships the modern saved shape. graph -> timeseries (6 panels): SPF Results Over Time, DKIM Results Over Time, SPF Alignment Over Time, DKIM Alignment Over Time, DMARC Passage Over Time, Message Disposition Over Time. Panel `aliasColors` (e.g. {true: dark-green, false: dark-red}) are translated into per-series `fieldConfig.overrides` so the green/red by-pass-fail colorings carry forward; legacy graph fields (lines, fill, yaxes, tooltip etc.) are dropped in favor of the new `fieldConfig.defaults.custom` block and `options.legend` / `options.tooltip`. worldmap -> geomap (2 panels): Map of Message Source Countries (aggregate), Forensic Sample Sources by Country (forensic). The legacy `locationData=countries` lookup-by-ISO becomes a geomap markers layer with `location.mode=lookup`, `gazetteer=public/gazetteer/countries.json`, and `lookup=source_country.keyword` — same input data, modern renderer. Drops the date_histogram bucket from the geomap targets since the map is a snapshot over the panel time range, not a time series. Verified against the bundled grafana/grafana:12.3.0: dashboard imports with status=success and `version=19`, live panel types now report `{timeseries: 6, geomap: 2, table: 14, grafana-piechart-panel: 3, stat: 1, row: 3}` — no more `graph` or `grafana-worldmap-panel` entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
4e8c28bbc0 |
Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737)
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Sean Whalen
|
826e78c390 |
Fix DMARC dashboard metrics (OSD + Splunk) and add dashboard-dev bootstrap (#736)
* OSD: fix aggregate dashboard metrics to sum(message_count) 13 panels on the DMARC aggregate dashboard were aggregating with `count` (number of OSD docs) when they should have been summing `message_count`. Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from the XML and carries an integer message_count, so doc-counting reports "distinct sources" rather than "messages". Panels with titles like "Message volume by header from", "DMARC passage over time", etc. were producing misleading numbers. Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and DKIM details. (DMARC failure email samples kept count — one OSD doc per RUF sample, so it's correct. SMTP TLS panels untouched — they sum the right session-count fields.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: align dashboards with OSD and fix query bugs Aggregate dashboard: - Add "Message sources by Autonomous System" panel (source_asn / as_name / as_domain), formatted "AS<n>" at render with eval, matching the OSD addition. - DKIM details: add the missing dkim_aligned column. - SPF details: reorder columns to OSD order (spf_aligned at end). - Map / country titles renamed to match OSD ("Map of message sources by country", "Message sources by country"). - Map widget: stats count by Country -> stats sum(message_count) by Country, so the choropleth shades by message volume not record count. - fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain, source_country to mirror OSD's missing-bucket labels. - charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart. Forensic dashboard: - Restructure to match OSD's two-panel layout (markdown + samples table). - Drop the country map / IP table / country-ISO table panels (not in OSD). - Samples table columns aligned to OSD: arrival_date_utc, source.ip_address, from, subject, reply_to, authentication_results. - Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=* required field to exist; LinkedIn RUF sample with null From was filtered out). SMTP TLS dashboard: - Reorder metrics to OSD order (successful before failed). - Domains panel: add policy_type bucket. - Failure details: replace search-time `failed_session_count>0` (which doesn't evaluate against multivalued JSON paths in Splunk) with `result_type=*` for presence + post-stats `where failed_sessions>0`. Drop _time/successful_sessions columns; reorder to match OSD. - Wire the existing policy_type input into all three searches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add dashboard-dev bootstrap script and VSCode task dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml, seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini, and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana, and Splunk. Idempotent: existence checks skip provisioning that's already done; only the dashboard imports re-run unconditionally on every invocation (that's the point of running it after a dashboard edit). Notable provisioning quirks the script handles: - Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships with indexes=[] and index=default; rewrites it to allow the email index. - ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily parsedmarc indexes via _cat/indices and deletes one at a time. - Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1 deletes and recreates the email index (then re-applies the HEC token). - OSD security plugin tenants: imports target global_tenant explicitly via the securitytenant header so they're visible to the shared workspace rather than landing in the API user's private tenant. Override with OSD_TENANT=<name>. - Splunk ships an in-product announcement view (scheduled_export_dashboard) with sharing=global; the script narrows it to sharing=app so it stops showing up in every app's dashboards list. Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump version to 9.10.3 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: warn against the "Create new objects with unique IDs" import mode OSD's import dialog has two modes: the default "Check for existing objects" (which honors saved-object IDs and overwrites in place when "Automatically overwrite conflicts" is on) and "Create new objects with unique IDs" (which imports under fresh UUIDs and leaves the buggy originals untouched). Picking the second one means the dashboards keep rendering the wrong numbers because the originals are never replaced. Spell that out so users don't fall into the trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: label the metric column "messages" instead of "Sum of message_count" OSD's table column header defaults to "Sum of message_count" when the metric agg has no customLabel. "messages" reads better and matches what the panels are actually counting. Applies to all 15 aggregate-DMARC visualizations that use sum(message_count). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable Trim the verbose technical exposition; lead each fix with the user-visible symptom. Move the action-required call out to its own header in upgrade notes so the re-import instructions don't get lost in a wall of text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move per-tool dashboard exports under a single dashboards/ directory Consolidates the four sibling top-level folders (kibana/, opensearch/, grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/. Updates the only path references in tracked files: bootstrap script (5 lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in docs/source/elasticsearch.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: restore the "DKIM alignment" panel title on the aggregate dashboard The DKIM alignment panel had no title override in panelsJSON, so OSD fell back to the visualization's own name ("Aggregate DMARC DKIM alignment"). Every other pie/table on the same dashboard sets a clean title (SPF alignment, Passed DMARC, etc.) — this was a stray regression. Set the panel title to "DKIM alignment" to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: color the message-disposition timechart by severity Reject is red, quarantine is yellow, none is green — same semantic mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart, applied via charting.fieldColors. Matches OSD's existing color overrides on the equivalent viz. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: clarify that "Create new objects with unique IDs" is the default The OSD import dialog defaults to that mode — users have to actively switch away from it, not just avoid picking it. Reword the upgrade note to lead with the switch and explain why the default would silently preserve the bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9.10.3 |
||
|
Sean Whalen
|
8cc017fe84 |
ASN-domain coverage sweep #3: 516 new map entries (#735)
* Add Tier 0 to the verification triage: globally-known brand at primary domain In the previous ASN-domain coverage sweep, the agent ran web searches for entries like `bestbuy.com → Best Buy`, `ups.com → United Parcel Service`, `usps.gov → US Postal Service`, `marriott.com → Marriott`, `henkel.cn → Henkel`, `experian.com → Experian`, `jd.com → JD.com`, `ing.com → ING`, `verisign.com → Verisign`. For each of these the domain ↔ brand pairing is encyclopedic — same outcome a few seconds slower. The two-corroborating-sources rule (rule 8) was being applied mechanically: "MMDB as_name alone is one source, must fetch a second." But for globally-known brands at their primary domain, the brand identity itself is the second source. Searching for confirmation that Best Buy owns bestbuy.com is the kind of busywork the tier system exists to avoid. Adds Tier 0 with explicit guardrails — must be globally known (multinational or top-tier-national, decades-old, single canonical entity), must be the entity's primary marketing/corporate domain (not a tracking subdomain or regional ccTLD where ownership is non-obvious), and no recent acquisition/rebrand status in question. Cross-references the existing parent-too-generic sub-rule and warns against stretching to mid-size brands the agent happens to recognize. When in doubt: drop to Tier 3 and search. Also generalizes the section's lead from "redirect-target candidates" to cover MMDB coverage-gap and PSL private-domain candidates — the tier logic transfers cleanly across all three workflows. Updates the Tier 1 description with an explicit MMDB-coverage-gap analog. Refreshes the held-back-review split stat to 0 / 109 / 2 / 34 / 35 (Tier 0 didn't apply to that batch because every candidate was a redirect target that needed to inherit the *source row's* existing canonical name, not its own brand identity). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ASN-domain coverage sweep #3: 516 new map entries Third pass against the IPinfo Lite MMDB coverage gap, processing the top ~500 unmapped as_domain entries by IPv4 weight after the prior two sweeps. Verifies each entry against AGENTS.md's tiered triage: - **Tier 0** (globally-known brand at primary domain, no search needed): Barclays, Liberty Mutual, Zurich Insurance, ABN AMRO, Swedbank, CIBC, Allstate, Julius Baer, MUFG, Travelers, USPS-Bank, ING, Florida Blue, AgriBank, Energy Transfer, FirstEnergy, Scania, Evonik, Merck KGaA, Agfa, Bosch, Iveco, Applied Materials, Micron, Andritz, Whirlpool, Leonardo, QinetiQ, Atlas Elektronik, Draper, Airbus, Jacobs Engineering, Teledyne, Dropbox, Autodesk, Wind River, Stratus, Unisys, ByteDance, Chevron, BBC, CDC, NEC, HPE, Kimberly-Clark, U.S. Bank, NATO, EUROCONTROL, Federal Reserve, NIST, NSF, DARPA, Library of Congress, IMF, FAO, IAEA, ITU, several US state/county/city governments, Australian state/federal departments, European national agencies, United Airlines, Alaska Airlines, Rakuten Mobile, Coles, Woolworths. - **Tier 1** (MMDB as_name lexically matches candidate domain, no search needed): ~150+ ISPs / hosters / cable TV operators where the as_name itself is the second corroborating source — major national/regional telcos (BTC Botswana, Uganda Telecom, ONE Albania, Tanzania Telecommunications, Kyrgyztelecom, Uzbektelekom, Telecom Algeria, MTN Rwanda, Vodacom Tanzania, Celcom Axiata, Triple T Broadcasting/Jasmine Thailand, MyRepublic Indonesia, Northwestel Canada, JT Jersey, Liberty Networks Colombia, ARLINK Argentina, Cable & Wireless Dominica, SETAR Aruba, AR Telecom Portugal), regional fiber providers (Trooli, Allied Telecom, OEC Fiber, Conexon Connect, Ben Lomand, Great Plains, BrightNet Oklahoma, All West, SDN, Tularosa, Blackfoot, Greeneville Energy, Avanti Broadband, Net at Once, Avanti, Aura Fiber, Stichting Breedband Delft), regional cable TV operators across Japan/Korea/Taiwan (Miyazaki Cable, Toyohashi Cable, Nagasaki Cable, Cable TV Toyama, Kurashiki Cable, Himeji Cable, Keumgang Cable Network), data center operators (eStruxture, PureVoltage, Hyonix, NovoServe, Voxility, Webzilla, Worldstream, Atman Poland, EO Data Center). - **Education** (TLD-restricted .edu / .ac.* / .edu.* — restriction is itself a corroborating source): 200+ universities and research institutions across US, Canada, Europe, Asia, and Australia, including Notre Dame, Washington State, U Texas Rio Grande Valley / Arlington / El Paso / San Antonio / Medical Branch, McMaster, U Ottawa, U Calgary, U Waterloo, Memorial U Newfoundland, U Auckland, U Otago, TU Munich, U Cologne, Goethe Frankfurt, Ruhr-Bochum, U Warwick, Chalmers, Lund, Gothenburg, Luleå, Osaka, Yonsei, Kasetsart, Pusan, Kuwait U, Aristotle Thessaloniki, Ł Tech U, Vienna U Economics, several Cancer Research Centers (MSKCC, Fred Hutchinson, MD Anderson, Cold Spring Harbor), national research institutes (KEK, IAEA, ITRI Taiwan, ETRI, IPM Iran, Smithsonian, ucar, Jefferson Lab, CSHL, mbari, Lam Research, Andritz Hydropower, sri.com, GSI Germany, Max Delbrück, jhuapl). - **Government** (.gov / .gov.* TLD-restricted, or as_name unambiguously names a government entity): NIST, NSF, NATO, DARPA, ITU, FAO, IAEA, IMF, US Centers for Disease Control, Federal Reserve, Library of Congress, Idaho/Chicago/King County/Pierce County/State of New York, Indianapolis, Tacoma, Fairfax County, Sweden's Vägverket and Forsakringskassan, Hessen GWDG, ANSTO Australia, South Florida Water Management District, Communications Research Centre Canada, Dataport Germany, Cenitex Victoria, EUROCONTROL. Skipped: Cox Enterprises (multi-product parent, no clean type fit), Tucows already added, sknt.ru already added, etc. Full triage shows 1 duplicate-skip from the apply pass. Sortlists.py runs cleanly. All 516 type values validate against base_reverse_dns_types.txt. No collisions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
d6d50a45e5 |
Add Tier 0 to the verification triage: globally-known brand at primary domain (#734)
In the previous ASN-domain coverage sweep, the agent ran web searches for entries like `bestbuy.com → Best Buy`, `ups.com → United Parcel Service`, `usps.gov → US Postal Service`, `marriott.com → Marriott`, `henkel.cn → Henkel`, `experian.com → Experian`, `jd.com → JD.com`, `ing.com → ING`, `verisign.com → Verisign`. For each of these the domain ↔ brand pairing is encyclopedic — same outcome a few seconds slower. The two-corroborating-sources rule (rule 8) was being applied mechanically: "MMDB as_name alone is one source, must fetch a second." But for globally-known brands at their primary domain, the brand identity itself is the second source. Searching for confirmation that Best Buy owns bestbuy.com is the kind of busywork the tier system exists to avoid. Adds Tier 0 with explicit guardrails — must be globally known (multinational or top-tier-national, decades-old, single canonical entity), must be the entity's primary marketing/corporate domain (not a tracking subdomain or regional ccTLD where ownership is non-obvious), and no recent acquisition/rebrand status in question. Cross-references the existing parent-too-generic sub-rule and warns against stretching to mid-size brands the agent happens to recognize. When in doubt: drop to Tier 3 and search. Also generalizes the section's lead from "redirect-target candidates" to cover MMDB coverage-gap and PSL private-domain candidates — the tier logic transfers cleanly across all three workflows. Updates the Tier 1 description with an explicit MMDB-coverage-gap analog. Refreshes the held-back-review split stat to 0 / 109 / 2 / 34 / 35 (Tier 0 didn't apply to that batch because every candidate was a redirect target that needed to inherit the *source row's* existing canonical name, not its own brand identity). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |