* Ignore Claude Code agent worktrees under .claude/worktrees/
Untracked repo snapshots from agent sessions were making repo-root
ruff check . fail on stale code and cluttering git status. ruff
respects .gitignore, so ignoring the directory fixes both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add [general] archive_directory to archive processed local files (#570)
Move successfully processed report files given as local path arguments
into <archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/,
dated from the parsed report's own metadata (aggregate begin_date,
failure arrival_date_utc, SMTP TLS begin_date) rather than the
filename. Files that fail to parse as a report (ParserError) go to
<archive_directory>/Invalid/; other failures (e.g. transient I/O
errors) leave the file in place so a later run can retry it.
Existing destination files are never overwritten: each candidate name
is claimed atomically (O_CREAT | O_EXCL) and collisions get a numeric
suffix before the extension. Files already inside the archive
directory are excluded from processing (compared via realpath so
symlinked spellings still match), so the archive can safely live
inside an input directory, as the issue requests. mbox files and
mailbox modes are unaffected; mailbox modes keep [mailbox]
archive_folder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review feedback on docstrings and the placeholder-cleanup except
- _move_file_to_archive's docstring no longer claims the copy2 fallback
replaces the placeholder atomically; only the same-filesystem os.rename
path is atomic. The fallback is a plain copy-and-overwrite, which is
still collision-safe because the placeholder already claimed the name.
- The empty except OSError in the placeholder cleanup now carries a
comment explaining that it is deliberate best-effort cleanup and the
re-raised move failure is the actionable error.
- test_general_archive_directory_unset_leaves_attribute_absent's
docstring now describes what the assertion actually tests (the
attribute staying absent from a bare Namespace) and moves the
real-CLI None-default behavior to a parenthetical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Cover the two defensive exception branches Codecov flagged
Codecov's patch report flagged four uncovered lines, all in the two
platform-dependent exception branches of the archive helpers:
- _exclude_archived_paths's except ValueError branch. Per the Python
docs for os.path.commonpath, ValueError is raised when paths "are on
the different drives" (Windows) or mix absolute and relative
pathnames; both inputs are realpath()-resolved so only the
different-drives case remains, which Linux CI can't produce
naturally. The new test simulates the raise and asserts the
non-comparable path is kept for parsing rather than excluded.
- _move_file_to_archive's except OSError placeholder-cleanup branch.
The new test fails the move with a non-OSError type and the cleanup
with OSError, then asserts the move error is what propagates (the
cleanup error is swallowed, not allowed to mask it) and that the
zero-byte placeholder survives its failed cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Tag arrival_date_utc as UTC when parsing the archive date
human_timestamp_to_datetime's docstring names arrival_date_utc as
exactly the kind of known-UTC naive string that should be parsed with
assume_utc=True; _archive_subdir_for_result was parsing it naive.
The flag is scoped to the failure branch because the shared call also
handles the other two report types: aggregate begin_date is a
local-time wall-clock string (timestamp_to_human uses
datetime.fromtimestamp), so tagging it UTC would be wrong, and SMTP TLS
begin_date carries an RFC 3339 offset, making assume_utc a no-op.
No observable behavior change: only the wall-clock year/month fields
are read and assume_utc never shifts wall-clock time, so no new test
can honestly distinguish the two versions — this aligns the call with
its dependency's documented contract and hardens against a future edit
adding a real timezone conversion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Create the archive placeholder with mode 0o600
os.open's mode parameter defaults to 0o777 (masked by the umask), so
the O_CREAT|O_EXCL placeholder in _move_file_to_archive was created
executable and group-accessible on typical umasks (0o775 under umask
002). Normally it's replaced immediately, but a placeholder that
outlives a failed move+cleanup persisted with those permissions.
Pass 0o600 explicitly. The mode never reaches the real archived file:
os.rename replaces the placeholder's inode outright, and the copy2
fallback's copystat overwrites the mode with the source file's. The
leftover-placeholder regression test now also asserts the surviving
placeholder has no owner-exec or group/other bits (umask-independent,
since the umask only clears bits); the assertion fails against the
unfixed default-mode call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add dashboard screenshot harness to the dev tooling
dashboard-dev-screenshots.py drives headless Chromium (Playwright)
against the dashboard dev stack and captures how Kibana, OpenSearch
Dashboards, Grafana, and Splunk actually render the current sample
data — for end-to-end verification of dashboard changes and PR
evidence. It encodes the platform quirks that otherwise cost time to
rediscover: fixed render waits instead of networkidle (Kibana/OSD
dashboards poll forever), pinning OSD to the global tenant (a stale
private-tenant copy silently screenshots old dashboards), and driving
the Grafana and Splunk login forms rather than HTTP basic auth. The
output directory is gitignored; usage is documented in
dashboards/README.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review findings on the screenshot harness
- Only require OPENSEARCH_INITIAL_ADMIN_PASSWORD / SPLUNK_PASSWORD when
the osd / splunk targets are selected, and fail fast before launching
Playwright with a clear message naming the missing variable(s).
- Honor GRAFANA_USER (defaulting to admin) to match the dev bootstrap
script instead of hard-coding the Grafana username.
- Print a full traceback to stderr when a target fails, instead of just
the exception message, while still continuing to the next target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Always close the browser when a screenshot target fails
Copilot round 2: each target only closed its Chromium instance on the
happy path, so a mid-run Playwright failure leaked a headless browser
while the script continued to the next target. Wrap each target body in
try/finally; closing the browser also closes its contexts and pages, so
the single b.close() covers the OSD context too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Exit with an install hint when playwright is missing
Copilot round 3: playwright is deliberately not a project dependency, so
a bare import failure produced a raw ModuleNotFoundError traceback.
Catch ImportError and exit with the same one-time install command the
module docstring documents.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Batch 1 — MMDB coverage-gap scan (find_unmapped_as_domains.py, default
4,096-IP floor): 132 candidates collected and classified. Manually
audited every auto-classified row rather than trusting the classifier
output directly, since it derives as_name straight from the MMDB and
will auto-classify off that single string even with no reachable
homepage — which fails the two-corroborating-sources bar. Demoted 10
rows to known-unknown on that basis (e.g. pellera.com's ASN registrant
is "Converge Technology Solutions", aerloop.in's is "Aerpace" — no
shared brand token, single source). Corrected several brand-quality
issues before they hit the map: sdtv.com.tw was about to ship as
"SDTV Webmail" (a login-page title) instead of "San Da Cable TV";
liyang.gov.cn was about to ship the hosting network's name instead of
the actual government operator (confirmed via Chinese-language WHOIS).
Also fixed a pre-existing bad entry noticed in passing: backwaves.net
was mapped under a garbled Chinese tagline instead of "Back Waves".
Result: 52 map rows, 82 known-unknown entries.
Batch 2 — plain-text uncategorized-sources export (842 source names via
find_unknown_base_reverse_dns.py -i, mix of raw MMDB as_name strings
and reverse-DNS domains) -> 377 unknown domains -> collected and
classified. This batch surfaced more data-quality traps: a parked
domain-registrar welcome page (serveur-vps.net) and a "Site under
maintenance" title (gou.go.ug, resolved via TLD instead) auto-classified
as if they were live operators; a WHOIS privacy-proxy service name
("GKG.NET Domain Proxy Service Administrator") slipped past the
privacy-org filter for atlasok.com; grohe.org.ru turned out to be a
Russian third-party reseller storefront, not the real Grohe corporation,
so it was held back from the map to avoid misattributing a reseller's
mail to the multinational. Refined the two-source test to distinguish
domain-WHOIS (ties directly to the domain; counts even without a
literal name match, e.g. portotelecom.net.br's WHOIS "Rondotech" backs
its self-described "Porto Telecom" brand) from MMDB as_name
(infrastructure-level signal, needs an actual lexical link, same
caution as IP-WHOIS). Used existing map entries as legitimate second
sources for redirects/aliases (vsys.host -> v-sys.org, metronet.net ->
metronet.com, mentari.net.id -> megahub.id) and cleaned up
megahub.id's own tagline-derived name to "Megahub" while there.
Result: 24 map rows, 353 known-unknown entries.
Also ignore the scratch/ working directory used to stage the
uncategorized-sources export for this pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Accept plain-text uncategorized-sources lists in find_unknown_base_reverse_dns.py
Dashboard exports of uncategorized email sources are plain-text lists of
one source name per line — a mix of raw MMDB as_name strings (when the
source IP had no PTR and resolved via the IPinfo Lite MMDB) and base
reverse-DNS domains. The script already translates as_names to their
as_domain and subtracts mapped/known-unknown entries, but only read a
hardcoded source_name-headed CSV.
Add -i/--input and -o/--output flags (defaults preserve current
behavior) and auto-detect the input format from the first line: a
source_name CSV header selects the existing DictReader path, anything
else is read as plain text with each line taken verbatim (never
comma-split, since as_names contain commas) and deduped
case-insensitively. Fix the missing-input error message, which
reported the map path instead of the input path.
Document the new entry point in the maps README and AGENTS.md, and make
explicit in the brand-quality triage rule that map display names must be
human-friendly operator names — never raw as_name strings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add MMDB coverage scan script with anti-poisoning guards
find_unmapped_as_domains.py turns the manual "Checking ASN-domain
coverage of the MMDB" recipe into a maintainer script: walk every IPv4
record in the bundled IPinfo Lite MMDB, aggregate routed footprint per
as_domain, subtract mapped/known-unknown keys, apply PSL folding and
the full-IP privacy filter, and emit domain,ipv4_count,as_name sorted
by footprint for the collector -> classifier pipeline.
Because ASN registration data is self-declared to the RIRs and
as_domain derives from registrant-controlled WHOIS, bulk-categorizing
the MMDB needs poisoning defenses:
- An IPv4-footprint floor (--min-ips, default 4096, a /20) keeps tiny
self-described ASNs out of the auto-classification queue; dropped
counts are always printed.
- A brand-collision guard in classify_unknown_domains.py loads the
existing map (--map) and demotes any single-category candidate whose
proposed display name matches an existing map name without a lexical
relationship to that operator's keys into the ambiguous bucket
(marked name-collision-with-existing-map-entry) for human review.
HAND overrides bypass the guard. The guard protects the PTR-side
flow as well as the MMDB-coverage flow.
Verified: scan yields 132 candidates at the default floor (1512
dropped); collector accepts the output directly; a fixture titled as
Comcast under an unrelated domain lands in ambiguous while a
comcast-rooted sibling auto-promotes. Also fix the maps README links
that still pointed at the root AGENTS.md for the classification
workflow after its extraction to maps/AGENTS.md, and correct the
classify_tsv docstring's return signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Enhance planning guidance in CLAUDE.md by specifying auto mode activation after user approval
* Codify triage flagging for identified operators with no fitting type
An operator confidently identified from two corroborating sources but
matching none of the README's type values should be flagged during
triage with a proposed new type for the reviewer, not force-fitted and
not silently recorded as known-unknown — KU means "we couldn't
identify this", which would bury completed research. Extends workflow
rule 7 and the LLM low-confidence list in the maps AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Split tests.py into per-module tests/test_<module>.py
The 5174-line tests.py monolith is split into per-module files under
tests/, mirroring the checkdmarc layout:
tests/test_init.py parsedmarc/__init__.py parsing surface
tests/test_cli.py parsedmarc/cli.py + config / env-vars / SIGHUP
tests/test_utils.py parsedmarc/utils.py (DNS, IP info, PSL, etc.)
tests/test_webhook.py parsedmarc/webhook.py
tests/test_kafkaclient.py parsedmarc/kafkaclient.py
tests/test_splunk.py parsedmarc/splunk.py
tests/test_syslog.py parsedmarc/syslog.py
tests/test_loganalytics.py parsedmarc/loganalytics.py
tests/test_gelf.py parsedmarc/gelf.py
tests/test_s3.py parsedmarc/s3.py
tests/test_maps.py parsedmarc/resources/maps/ maintainer scripts
The split is purely a redistribution — no test bodies changed, no tests
added or removed. All 276 existing tests pass under the new layout.
The current tests.py contains two kitchen-sink classes (`Test` at line 54
and `TestEnvVarConfig` at line 2360) holding tests that span many
modules. Their methods are routed to the correct per-module file by name
prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx,
TestSighupReload, etc.) move whole. Each target file gets its own
`class Test(unittest.TestCase)` for the redistributed kitchen-sink
methods, plus the thematic classes verbatim.
Wiring updates:
- `.github/workflows/python-tests.yml`: `pytest ... tests.py` →
`python -m pytest ... tests/` (also switches to `python -m pytest` per
the checkdmarc convention so cwd lands on the project root).
- `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]`
and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for
`parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only
batch tooling that ships out of the wheel; excluding them from
coverage makes the headline number reflect only installed library
code. Runtime coverage on the new layout is 59% (was 45% with maps
counted), and PR-B will push it to 90%+.
- `AGENTS.md`: documents the new layout and how to run individual files
/ tests; tells future contributors not to reintroduce a monolithic
tests.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore 66.9% coverage baseline (count tests/ + parsedmarc)
Master's headline 66.9% number on Codecov includes the tests.py file
itself (99.35% covered) being measured alongside parsedmarc/*. The
original tests.py had no `[tool.coverage.run]` block, so coverage's
default — "measure every file imported during the run" — counted the
test code as if it were product code.
The split commit added `source = ["parsedmarc"]` which suppressed
measurement of the test files (correct in principle, since test files
aren't shipped code), and that alone made the headline number drop by
~8 percentage points without any actual loss of testing. This commit
swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]`
so both halves are measured the way they were on master. Verified:
276 tests, 66.96% line coverage (effectively unchanged from master's
66.90%).
If you want the shipped-code-only number (was the headline that this
commit overrides), run `pytest --cov=parsedmarc tests/`. That number
is currently 59% and is the focus of the upcoming coverage-expansion PR.
Also adds junit.xml to .gitignore so the CI artefact doesn't get
accidentally committed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restrict coverage to shipped code (`source = ["parsedmarc"]`)
Reverts the prior commit's `include = ["tests/*"]`. Counting the test
files toward coverage was wrong — it conflates "shipped code exercised
by tests" with "test code that pytest auto-runs", inflates the headline
number, and rewards writing more tests rather than tests that verify
more code. Master's apparent 66.9% was an artefact of the old
monolithic tests.py having no [tool.coverage.run] block at all; coverage's
default behaviour measured every imported file, including the test file
itself at ~99% "covered", which added ~8 percentage points to the
displayed number without any real testing signal.
Restricting to `source = ["parsedmarc"]` plus the existing maps omit
gives a meaningful baseline: 59% of shipped code is exercised by the
test suite today. That's the number the next PR is targeting to lift
to 90%+ before the 10.0.0 release; the Codecov "drop" here is a
measurement correction, not a regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Auto-download psl_overrides.txt at startup (and whenever the reverse DNS
map is reloaded) via load_psl_overrides(); add local_psl_overrides_path
and psl_overrides_url config options
- Add collect_domain_info.py and detect_psl_overrides.py for bulk WHOIS/HTTP
enrichment and automatic cluster-based PSL override detection
- Block full-IPv4 reverse-DNS entries from ever entering
base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, or
unknown_base_reverse_dns.csv, and sweep pre-existing IP entries
- Add Religion and Utilities to the allowed service_type values
- Document the full map-maintenance workflow in AGENTS.md
- Substantial expansion of base_reverse_dns_map.csv (net ~+1,000 entries)
- Add 26 tests covering the new loader, IP filter, PSL fold logic, and
cluster detection
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
- Add source name and type information based on static mapping of the reverse DNS base domain
- See [this documentation](https://github.com/domainaware/parsedmarc/tree/master/parsedmarc/resources/maps) for more information, and to learn how to help!
- Replace `multiprocessing.Pool` with `Pipe` + `Process` (PR #491 closes issue #489)
- Remove unused parallel arguments (PR #492 closes issue #490)
- A static copy of the DBIP database is now included for use when a copy of the MaxMind GeoLite2 Country database is not installed (Closes#275)
- Add `ip_db_path` to as a parameter and `general` setting for a custom IP geolocation database location (Closes#184)
- Search default Homebrew path when searching for a copy of the MaxMind GeoLite2 Country database (Closes#272)
- Fix log messages written to root logger (PR #276)
- Fix `--offline` option in CLI not being passed as a boolean (PR #265)
- Set Elasticsearch shard replication to `0` (PR #274)
- Add support for syslog output (PR #263closes#227)
- Do not print TQDDM progress bar when running in a no-interactive TTY (PR #264)
Sean WhalenandClaude Fable 5
Sean Whalen
Sean Whalen