mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-03 06:02:17 +00:00
72bece4686e040e4cd16ba6946e071503a5c2002
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
7d87ba18be |
Ingest uncategorized-sources exports and add MMDB coverage scan with anti-poisoning guards (#829)
* Accept plain-text uncategorized-sources lists in find_unknown_base_reverse_dns.py Dashboard exports of uncategorized email sources are plain-text lists of one source name per line — a mix of raw MMDB as_name strings (when the source IP had no PTR and resolved via the IPinfo Lite MMDB) and base reverse-DNS domains. The script already translates as_names to their as_domain and subtracts mapped/known-unknown entries, but only read a hardcoded source_name-headed CSV. Add -i/--input and -o/--output flags (defaults preserve current behavior) and auto-detect the input format from the first line: a source_name CSV header selects the existing DictReader path, anything else is read as plain text with each line taken verbatim (never comma-split, since as_names contain commas) and deduped case-insensitively. Fix the missing-input error message, which reported the map path instead of the input path. Document the new entry point in the maps README and AGENTS.md, and make explicit in the brand-quality triage rule that map display names must be human-friendly operator names — never raw as_name strings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add MMDB coverage scan script with anti-poisoning guards find_unmapped_as_domains.py turns the manual "Checking ASN-domain coverage of the MMDB" recipe into a maintainer script: walk every IPv4 record in the bundled IPinfo Lite MMDB, aggregate routed footprint per as_domain, subtract mapped/known-unknown keys, apply PSL folding and the full-IP privacy filter, and emit domain,ipv4_count,as_name sorted by footprint for the collector -> classifier pipeline. Because ASN registration data is self-declared to the RIRs and as_domain derives from registrant-controlled WHOIS, bulk-categorizing the MMDB needs poisoning defenses: - An IPv4-footprint floor (--min-ips, default 4096, a /20) keeps tiny self-described ASNs out of the auto-classification queue; dropped counts are always printed. - A brand-collision guard in classify_unknown_domains.py loads the existing map (--map) and demotes any single-category candidate whose proposed display name matches an existing map name without a lexical relationship to that operator's keys into the ambiguous bucket (marked name-collision-with-existing-map-entry) for human review. HAND overrides bypass the guard. The guard protects the PTR-side flow as well as the MMDB-coverage flow. Verified: scan yields 132 candidates at the default floor (1512 dropped); collector accepts the output directly; a fixture titled as Comcast under an unrelated domain lands in ambiguous while a comcast-rooted sibling auto-promotes. Also fix the maps README links that still pointed at the root AGENTS.md for the classification workflow after its extraction to maps/AGENTS.md, and correct the classify_tsv docstring's return signature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Enhance planning guidance in CLAUDE.md by specifying auto mode activation after user approval * Codify triage flagging for identified operators with no fitting type An operator confidently identified from two corroborating sources but matching none of the README's type values should be flagged during triage with a proposed new type for the reviewer, not force-fitted and not silently recorded as known-unknown — KU means "we couldn't identify this", which would bury completed research. Extends workflow rule 7 and the LLM low-confidence list in the maps AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
18b38c991d |
Add verify skill and document model roles in CLAUDE.md (#817)
* Raise test coverage: utils, elastic, and opensearch to 100%
Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).
New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.
Bugs found by the new tests, fixed in the same PR per the testing
standards:
- parse_email() crashed with KeyError: 'Headers' on messages whose
From header is present but unparseable (e.g. a bare "From:" line):
the fallback read parsed_email["Headers"], but the parsed headers
are stored under lowercase "headers" (assigned a few lines up in
the same function), so the key never exists. At the CLI surface
this made any failure report whose embedded sample had an empty
From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
when the probe could not reach the API, contradicting its own
docstring ("other errors are logged and the token is still
accepted"): _ipinfo_api_lookup() returns None on network errors
instead of raising, so the probe's exception handler was
unreachable. The probe now checks the lookup result and warns on
failure; 401/403 still raises InvalidIPinfoAPIKey.
Dead code deleted rather than padded with tests:
- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
(the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
input branch assigns file_object or raises first (confirmed by
pyright narrowing with the guard removed).
Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use assertGreater for the reverse-DNS-map fallback size check
Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add verification claude skill for parsedmarc CLI usage and sample inputs
* Document model roles for feature work in CLAUDE.md
Codifies the plan-with-Fable / implement-with-Sonnet / review-with-Fable
split (Opus as fallback) for feature work and PR reviews.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
ae1e5adb66 |
Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659)
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @
|
||
 Sean Whalen
|
9551c8b467 | Add AGENTS.md for AI agent guidance and link from CLAUDE.md |