* Add opt-in Elasticsearch/OpenSearch data streams (#842)
New use_data_streams option in [elasticsearch] and [opensearch] saves
each report type to a data stream named like today's index without the
date suffix, instead of daily or monthly indexes. parsedmarc puts an
exact-name index template (priority 501, @timestamp mapped as date,
<stream>@custom component template on Elasticsearch) and creates the
stream; retention is left to operator-managed ILM/ISM policies, with
documented examples. Dated indexes are untouched and still searched for
duplicates and matched by the dashboards. Every document now carries
@timestamp and is saved with op_type=create in both modes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix data stream prose found by cross-check review (layer 1)
Aggregate @timestamp is each record's begin date, not the report's; the
cluster only rolls backing indices over under an attached policy; move
the Elasticsearch 8.7 note after the template it refers to; clarify the
name-collision paragraph, OpenSearch-side comments, the migrate_indexes
docstring, and the test-module headers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Tighten names and wiring tests found by cross-check review (layer 2)
Split the use_data_streams CLI passthrough test per backend so a call
site reading the other section's flag fails; name the dated-mode tests
for both changes they assert; compute index_date only in dated mode in
every save function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Align data stream docs with Elastic/OpenSearch references (layer 3)
Say the stream must exist before reindexing into it (ES reindex API:
auto-creation needs a data-stream template); an existing stream's write
index needs the ILM policy attached directly since nothing rolls it over
otherwise; target the stream name in the _settings example; note
OpenSearch's leading-dot rule for stream names and Fleet's template
priorities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Cover the Loki client-initialization error wrapper
_parse_config accepts a [loki] token alongside username/password; only
the LokiClient constructor rejects the combination. Exercise that path
through _init_output_clients with the real constructor so the
RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sean WhalenandClaude Opus 5.5
Szasza Palmer