Reject reports dated in the future and bound what parse_report_file() reads (#954)

* Reject reports whose index-naming date is more than two days in the future (#947)

The date that names a report's dated Elasticsearch/OpenSearch index (an
aggregate report's begin date, a failure report's Arrival-Date, an SMTP
TLS report's start-datetime) was never compared with the present, so a
report dated 2100 named an index for 2100. Each parser now rejects such a
report with its own Invalid*Report; past dates stay unbounded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Refuse an input larger than the report size cap before parse_report_file() reads it (#949)

The 32 MiB cap was applied only after the whole input was read. A path is
now checked with os.path.getsize(), bytes-like input with len(), and a
file object is read at most one byte past the cap. The error is the
ParserError with the message the content check already gives. Three
existing tests that fed over-cap content now see the earlier rejection.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Measure a memoryview input by its byte size

len() of a memoryview counts items, which are not bytes for every format,
so the size cap in parse_report_file() compares nbytes instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Bound the aggregate end date and read file objects to their end

Address the Copilot review of #954. A normalized report's per-day records
name their indexes from their own interval starts, so the report's end date
is bounded like its begin date. A raw stream may return fewer bytes than
asked for before its end, so parse_report_file() now reads in a loop until
end of file or one byte past the cap.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Accumulate bounded reads in one buffer

Address the second Copilot review of #954. Keeping every chunk in a list
cost one bytes object per read, about 85 bytes per byte for a stream that
returns a byte at a time (22.7 MiB for 256 KiB of input). One bytearray
keeps memory proportional to the bytes read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Sean WhalenandClaude Fable 5.2 authored and GitHub committed 2026-11-10 17:09:23 -05:00
1 parent fa905e41fd
commit d9193eafea
6 files changed
+339 -12

No files matched your search

+4
View File
@@ -420,6 +420,10 @@ index with their own dates, and a warning is logged. Dated index names still com
from each report's own dates, so reports with many different dates
still create one index per date; data streams (`use_data_streams`) are
affected by neither.
A report whose begin or end date (aggregate), begin date (SMTP TLS) or
`Arrival-Date` (failure) is more than two days in the future is rejected
as invalid, because those dates name the dated indexes; past dates are
not bounded.
Set `use_data_streams = True` in the `[elasticsearch]` section to write
to [data streams] instead. Each report type then goes to one data stream
+4
View File
@@ -21,6 +21,10 @@ index with their own dates, and a warning is logged. Dated index names still com
from each report's own dates, so reports with many different dates
still create one index per date; data streams (`use_data_streams`) are
affected by neither.
A report whose begin or end date (aggregate), begin date (SMTP TLS) or
`Arrival-Date` (failure) is more than two days in the future is rejected
as invalid, because those dates name the dated indexes; past dates are
not bounded.
Set `use_data_streams = True` in the `[opensearch]` section to write to
[data streams] instead. Each report type then goes to one data stream