Fix aggregate-report timestamp skew on non-UTC hosts (#819) (#821)

record["interval_begin"]/["interval_end"] are UTC wall-clock strings
(produced in __init__.py by strftime() on a datetime already converted
via human_timestamp_to_datetime(..., to_utc=True)), but the
Elasticsearch and OpenSearch per-record save loops and the Splunk HEC
aggregate-report event builder re-parsed them without assume_utc=True,
so on non-UTC hosts they were misinterpreted as local time and shifted
by the host's UTC offset -- shifting the stored date_begin/date_end,
the daily/monthly index date, and the Splunk event time. Verified by
parsing a real sample report under TZ=Europe/Warsaw vs TZ=UTC and by
reproducing the exact skew via human_timestamp_to_unix_timestamp().

This is the same class of bug fixed for arrival_date_utc in #811/#812
(commit cdda5da); the assume_utc keyword already exists on
human_timestamp_to_datetime()/human_timestamp_to_unix_timestamp() and
is reused here rather than reimplemented.

The issue also proposed changes to the report-level begin_date/end_date
parses in elastic.py/opensearch.py (~line 455) and to postgres.py.
Both were investigated and left unchanged: the report-level strings
are genuinely host-local time (from timestamp_to_human() ->
datetime.fromtimestamp()), so their existing no-assume_utc round-trip
is already correct on a single host -- postgres.py already gets this
right via two distinct helpers (_naive_local_to_timestamptz vs
_ensure_utc_suffix). Adding assume_utc to the report-level parses
would introduce a skew rather than fix one.

Fixes https://github.com/domainaware/parsedmarc/issues/819

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Sean WhalenandClaude Sonnet 5 authored and GitHub committed 2026-07-13 09:59:35 -04:00
1 parent f33951c0bb
commit 01b85ab7de
7 files changed
+79 -5

No files matched your search

+15
View File
@@ -294,6 +294,21 @@ class TestSaveAggregateReportsToSplunk(unittest.TestCase):
client.save_aggregate_reports_to_splunk(_aggregate_report())
self.assertIn("network", str(ctx.exception))
@unittest.skipUnless(hasattr(time, "tzset"), "requires POSIX time.tzset()")
def test_event_time_treats_interval_begin_as_utc(self):
"""interval_begin is a UTC wall-clock string; the HEC event
`time` must be its true UTC epoch regardless of the host
timezone. Regression test for
https://github.com/domainaware/parsedmarc/issues/819: the naive
parse used to shift the epoch by the host's UTC offset."""
force_tz(self)
client = _client()
client.session = MagicMock()
client.session.post.return_value = _ok_response()
client.save_aggregate_reports_to_splunk(_aggregate_report())
event_wrapper = json.loads(client.session.post.call_args.kwargs["data"].strip())
self.assertEqual(event_wrapper["time"], 1704067200)
class TestSaveFailureReportsToSplunk(unittest.TestCase):
def test_sends_one_event_per_report(self):