* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks
arrival_date_utc is a UTC wall-clock string (generated in
parse_failure_report via an aware-UTC strftime), but elastic.py,
opensearch.py, and splunk.py parsed it back into a naive datetime and
called .timestamp(), which per the Python docs interprets naive values
as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp).
On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date
field, used in the failure-report dedup match query, and sent as the
Splunk HEC event time was therefore off by the host's UTC offset
(verified -3600 s under TZ=Europe/Warsaw in January).
Add an assume_utc keyword to human_timestamp_to_datetime() /
human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive
parses, and use it at the three arrival_date_utc call sites. Aware
inputs (explicit offsets) are unaffected; all other callers keep the
existing local-time semantics, whose round-trip with timestamp_to_human
is self-consistent on a single host (the broader local-time output
question is tracked separately in issue #811 bug 2).
The three new sink regression tests fail on the unfixed code
(verified by stashing the source changes) and force TZ=Europe/Warsaw
via time.tzset() so they catch the skew even on UTC CI runners.
Fixes half of https://github.com/domainaware/parsedmarc/issues/811.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring
Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from
the four timestamp regression tests into a shared tests/tzutil.py
force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s
docstring, which said the return type was float while the function
returns int.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
The 10.0.3 Reply-To header flattening (elastic.py / opensearch.py line 711)
has two branches: display-name present ("Name <addr>") and absent (bare
address). The existing test only exercised the former, leaving the
empty-display-name branch uncovered — the two lines Codecov flagged on the
10.0.3 patch. Add a failure report whose Reply-To has no display name and
assert sample.headers["reply-to"] flattens to the bare address.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.
This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Expand honest test coverage from 59% to 83%; fix two latent bugs
271 new tests across the output modules, ES/OS clients, CLI config
parsing, and the top-level parsing surface. Coverage measured against
shipped code only (see [tool.coverage.run] source = ["parsedmarc"]
omit = ["*/parsedmarc/resources/maps/*.py"] in pyproject.toml).
Per-module results:
s3.py 38% → 100% (also fixes SMTP-TLS-to-S3 bug below)
gelf.py 40% → 100%
syslog.py 46% → 100%
kafkaclient.py 34% → 100%
splunk.py 24% → 100%
loganalytics.py 56% → 100%
webhook.py 78% → 100% (also removes redundant try/except)
elastic.py 36% → 99%
opensearch.py 40% → 99%
cli.py 52% → 69%
__init__.py 74% → 76% (also fixes append_json bug below)
utils.py 84% (unchanged in this PR)
TOTAL 59% → 83%
The remaining 17% is honest. The biggest unreached blocks are
_main() in cli.py and the watch-mode mailbox iteration in __init__.py,
both of which would require either standing up live subsystems (real
Elasticsearch, real IMAP) or mocking deep enough that the test would
verify the mock rather than the code. The PR-A AGENTS.md guidance —
"if 90% requires faking it, ship 85% honestly" — applies here.
Bugs fixed while writing tests:
1. parsedmarc/s3.py — SMTP-TLS-to-S3 was completely broken.
save_report_to_s3 unconditionally read report["report_metadata"]
when building S3 object metadata, but RFC 8460 §4.3 SMTP TLS
reports are flat (no report_metadata sub-object). The CLI's
surrounding try/except silently swallowed the KeyError, so every
SMTP-TLS report quietly failed to upload. Also fixes a related
issue: parse_smtp_tls_report_json stores begin_date as the raw
ISO-8601 string from the report (per the SMTPTLSReport TypedDict
and RFC 8460 §4.3), but the S3 code path assumed a datetime
with .year / .month / .day attributes. Both fixed; the broken
metadata-extraction branch now uses the flat-report fields, and
the date branch normalizes via human_timestamp_to_datetime.
2. parsedmarc/__init__.py — append_json corrupted JSON output files
on the second write. The original implementation opened files in
"a+" mode, then seek()ed backwards to overwrite the trailing "]"
with ",\n" before appending more elements. Python's docs are
explicit (https://docs.python.org/3/library/functions.html#open):
on POSIX, writes in "a"/"a+" mode always go to EOF regardless of
seek() position. The result was that the second call produced
[...]\n],\n[...] -style corrupted output instead of a single
merged array. Replaced with a read-merge-write pattern: load the
existing array (if any), append the new elements, rewrite the
whole file. The CSV cousin append_csv was not affected — it
doesn't seek backwards.
3. parsedmarc/webhook.py — removed redundant try/except blocks in
save_aggregate_report_to_webhook / save_failure_report_to_webhook
/ save_smtp_tls_report_to_webhook. _send_to_webhook already
catches every Exception itself, so the outer except blocks were
unreachable dead code (covered nothing, defended against nothing,
and inflated the source-line count without testing value).
Testing approach: mocks at SDK boundaries (boto3 resource, kafka
producer, requests session, opensearch/elasticsearch Document/Search,
azure LogsIngestionClient). Tests verify the parsedmarc-side
transformation logic — document/event construction, index/topic
naming, dedup queries, error wrapping — rather than asserting on
mock invocations as a proxy for behaviour. Where a branch is
defensive against a caller that doesn't exist in the codebase, the
test is omitted (commented in code rather than hidden behind a
pragma).
547 tests total (was 276), all passing. ruff check + format clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Document the two bug fixes from this PR in the 10.0.0 changelog
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Document testing standards in AGENTS.md
Adds a "Testing standards" section covering the principles applied in
PR-A (split) and PR-B (coverage expansion):
- Coverage measures shipped code only — don't reintroduce tests/* to
the scope, don't expand omit, don't use # pragma: no cover.
- Honest tests assert on observable behaviour, not "the mock was called".
Mock at SDK boundaries; parse the payload that gets sent.
- "If 90% requires faking it, ship 85% honestly" — coverage is a tool,
not a goal. PR-B's deliberate stops at cli.py 69% and __init__.py 76%
are the documented precedent for when to halt.
- Verify bug claims against the relevant RFC, internal types, installed
SDK source, or upstream docs before changing code. Cite the source in
the commit message and test docstring (RFC 8460 §4.3 and the Python
open() docs for #775's two bug fixes are the pattern to follow).
- Bugs found while writing tests are fixed in the same PR; the test
doubles as the regression guard.
- File layout (tests/test_<module>.py) is non-negotiable; module-level
test loggers need fresh-handler setup so test ordering doesn't break
assertLogs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Cover the corrupt-file fallback in append_json
Codecov flagged 2 missing patch-coverage lines on PR #775: the
except (json.JSONDecodeError, OSError) branch in append_json, which
falls back to overwriting when the existing file isn't a parseable
JSON array. Two new tests in tests/test_init.py:TestAppendJson
exercise both paths:
- test_corrupt_existing_file_is_overwritten_cleanly: existing file
contains invalid JSON; append_json overwrites with the new array.
- test_existing_file_with_non_list_root_is_overwritten: existing
file parses as {"foo": ...} (dict, not list); the isinstance guard
rejects it and we overwrite cleanly.
Patch coverage now 100% on the bug fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DVBandGitHub