4 Commits

Include renames

Author	SHA1	Message	Date
5 people	7fed72798a	Stop system GeoIP files from shadowing the bundled IPinfo database (#813) ... * Stop system GeoIP files from shadowing the bundled IPinfo database _get_ip_database_path() searched well-known system paths (including /usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before the database parsedmarc manages, so on any host with a distro GeoIP package installed every lookup silently used a country-only — and often years-old — database instead of the bundled IPinfo Lite one. That disabled ASN enrichment entirely (asn/as_name/as_domain were None for every IP) and with it the ASN-fallback path into the reverse-DNS map, with no signal beyond a generic "IP database is more than a month old" warning. Verified live on a Fedora host whose distro GeoLite2-Country.mmdb dated to December 2019. New precedence: explicit ip_db_path -> _IP_DB_PATH selected by load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system paths as a true last resort (only consulted when the bundled data file is missing). The selected file is logged at debug level so a --debug run shows which database answered. The automatic system-path pickup was documented behavior, so installation.md now tells MaxMind GeoLite2 users to set ip_db_path explicitly, with a migration note. Both new regression tests reproduce the shadowing portably via a decoy CWD GeoLite2-Country.mmdb (the fallback list includes relative names), and fail on the unfixed code (verified by stashing the source change). Fixes https://github.com/domainaware/parsedmarc/issues/810. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: accurate fallback comment, dedup selection log, cover fallback tiers - Correct the nothing-found-anywhere comment: the os.stat() age check raises FileNotFoundError before the caller's open_database() would. - Log "Using IP database at ..." only when the selected path changes instead of on every uncached IP lookup, so --debug runs over large batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in the test fixture for order-independence. - Cover the previously untested branches of _get_ip_database_path: system-path fallback when the bundled database is missing, the FileNotFoundError when nothing exists anywhere, the stale-database warning, and the log-once-per-path behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>	2026-07-09 20:07:00 -04:00
5 people	cdda5dae62	Fix failure-report timestamp skew on non-UTC hosts in ES/OpenSearch/Splunk outputs (#812) ... * Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks arrival_date_utc is a UTC wall-clock string (generated in parse_failure_report via an aware-UTC strftime), but elastic.py, opensearch.py, and splunk.py parsed it back into a naive datetime and called .timestamp(), which per the Python docs interprets naive values as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp). On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date field, used in the failure-report dedup match query, and sent as the Splunk HEC event time was therefore off by the host's UTC offset (verified -3600 s under TZ=Europe/Warsaw in January). Add an assume_utc keyword to human_timestamp_to_datetime() / human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive parses, and use it at the three arrival_date_utc call sites. Aware inputs (explicit offsets) are unaffected; all other callers keep the existing local-time semantics, whose round-trip with timestamp_to_human is self-consistent on a single host (the broader local-time output question is tracked separately in issue #811 bug 2). The three new sink regression tests fail on the unfixed code (verified by stashing the source changes) and force TZ=Europe/Warsaw via time.tzset() so they catch the skew even on UTC CI runners. Fixes half of https://github.com/domainaware/parsedmarc/issues/811. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from the four timestamp regression tests into a shared tests/tzutil.py force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s docstring, which said the return type was float while the function returns int. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>	2026-07-09 19:45:34 -04:00
3 people	e104f1118c	Land 10.0.3 changes on master (#785) ... PR #784 was stacked on the #783 branch and its base was never retargeted to master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master. master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes. This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the >=2.2.1 floor is already on master from 10.0.2). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 13:54:40 -04:00
3 people	5b08627eaa	Split tests.py into per-module tests/test_<module>.py (#774) ... * Split tests.py into per-module tests/test_<module>.py The 5174-line tests.py monolith is split into per-module files under tests/, mirroring the checkdmarc layout: tests/test_init.py parsedmarc/__init__.py parsing surface tests/test_cli.py parsedmarc/cli.py + config / env-vars / SIGHUP tests/test_utils.py parsedmarc/utils.py (DNS, IP info, PSL, etc.) tests/test_webhook.py parsedmarc/webhook.py tests/test_kafkaclient.py parsedmarc/kafkaclient.py tests/test_splunk.py parsedmarc/splunk.py tests/test_syslog.py parsedmarc/syslog.py tests/test_loganalytics.py parsedmarc/loganalytics.py tests/test_gelf.py parsedmarc/gelf.py tests/test_s3.py parsedmarc/s3.py tests/test_maps.py parsedmarc/resources/maps/ maintainer scripts The split is purely a redistribution — no test bodies changed, no tests added or removed. All 276 existing tests pass under the new layout. The current tests.py contains two kitchen-sink classes (`Test` at line 54 and `TestEnvVarConfig` at line 2360) holding tests that span many modules. Their methods are routed to the correct per-module file by name prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx, TestSighupReload, etc.) move whole. Each target file gets its own `class Test(unittest.TestCase)` for the redistributed kitchen-sink methods, plus the thematic classes verbatim. Wiring updates: - `.github/workflows/python-tests.yml`: `pytest ... tests.py` → `python -m pytest ... tests/` (also switches to `python -m pytest` per the checkdmarc convention so cwd lands on the project root). - `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]` and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for `parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only batch tooling that ships out of the wheel; excluding them from coverage makes the headline number reflect only installed library code. Runtime coverage on the new layout is 59% (was 45% with maps counted), and PR-B will push it to 90%+. - `AGENTS.md`: documents the new layout and how to run individual files / tests; tells future contributors not to reintroduce a monolithic tests.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restore 66.9% coverage baseline (count tests/ + parsedmarc) Master's headline 66.9% number on Codecov includes the tests.py file itself (99.35% covered) being measured alongside parsedmarc/*. The original tests.py had no `[tool.coverage.run]` block, so coverage's default — "measure every file imported during the run" — counted the test code as if it were product code. The split commit added `source = ["parsedmarc"]` which suppressed measurement of the test files (correct in principle, since test files aren't shipped code), and that alone made the headline number drop by ~8 percentage points without any actual loss of testing. This commit swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]` so both halves are measured the way they were on master. Verified: 276 tests, 66.96% line coverage (effectively unchanged from master's 66.90%). If you want the shipped-code-only number (was the headline that this commit overrides), run `pytest --cov=parsedmarc tests/`. That number is currently 59% and is the focus of the upcoming coverage-expansion PR. Also adds junit.xml to .gitignore so the CI artefact doesn't get accidentally committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restrict coverage to shipped code (`source = ["parsedmarc"]`) Reverts the prior commit's `include = ["tests/*"]`. Counting the test files toward coverage was wrong — it conflates "shipped code exercised by tests" with "test code that pytest auto-runs", inflates the headline number, and rewards writing more tests rather than tests that verify more code. Master's apparent 66.9% was an artefact of the old monolithic tests.py having no [tool.coverage.run] block at all; coverage's default behaviour measured every imported file, including the test file itself at ~99% "covered", which added ~8 percentage points to the displayed number without any real testing signal. Restricting to `source = ["parsedmarc"]` plus the existing maps omit gives a meaningful baseline: 59% of shipped code is exercised by the test suite today. That's the number the next PR is targeting to lift to 90%+ before the 10.0.0 release; the Codecov "drop" here is a measurement correction, not a regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 19:29:09 -04:00