Add opt-in Elasticsearch/OpenSearch data streams (#937)

* Add opt-in Elasticsearch/OpenSearch data streams (#842)

New use_data_streams option in [elasticsearch] and [opensearch] saves
each report type to a data stream named like today's index without the
date suffix, instead of daily or monthly indexes. parsedmarc puts an
exact-name index template (priority 501, @timestamp mapped as date,
<stream>@custom component template on Elasticsearch) and creates the
stream; retention is left to operator-managed ILM/ISM policies, with
documented examples. Dated indexes are untouched and still searched for
duplicates and matched by the dashboards. Every document now carries
@timestamp and is saved with op_type=create in both modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix data stream prose found by cross-check review (layer 1)

Aggregate @timestamp is each record's begin date, not the report's; the
cluster only rolls backing indices over under an attached policy; move
the Elasticsearch 8.7 note after the template it refers to; clarify the
name-collision paragraph, OpenSearch-side comments, the migrate_indexes
docstring, and the test-module headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tighten names and wiring tests found by cross-check review (layer 2)

Split the use_data_streams CLI passthrough test per backend so a call
site reading the other section's flag fails; name the dated-mode tests
for both changes they assert; compute index_date only in dated mode in
every save function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Align data stream docs with Elastic/OpenSearch references (layer 3)

Say the stream must exist before reindexing into it (ES reindex API:
auto-creation needs a data-stream template); an existing stream's write
index needs the ILM policy attached directly since nothing rolls it over
otherwise; target the stream name in the _settings example; note
OpenSearch's leading-dot rule for stream names and Fleet's template
priorities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cover the Loki client-initialization error wrapper

_parse_config accepts a [loki] token alongside username/password; only
the LokiClient constructor rejects the combination. Exercise that path
through _init_output_clients with the real constructor so the
RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Sean WhalenandClaude Opus 5.5 authored and GitHub committed 2026-10-31 15:37:37 -04:00
1 parent 70f96d0734
commit 56e5a09923
10 files changed
+1765 -80

No files matched your search

+160 -3
View File
@@ -405,13 +405,170 @@ curl -X POST "http://localhost:9200/smtp_tls*/_update_by_query?conflicts=proceed
It works identically on OpenSearch; just adjust the URL and credentials, same
as the aggregate command above.
## Records retention
## Data streams and records retention
Starting in version 5.0.0, `parsedmarc` stores data in a separate
index for each day to make it easy to comply with records
retention regulations such as GDPR. For more information,
index for each day (or each month, with `monthly_indexes`), named after
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
easy to comply with records retention regulations such as GDPR: delete
the indexes for the dates you no longer keep. For more information,
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
Set `use_data_streams = True` in the `[elasticsearch]` section to write
to [data streams] instead. Each report type then goes to one data stream
named like the indexes without the date: `dmarc_aggregate`,
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
the `index_prefix_domain_map` prefix applied as usual. The cluster
creates the stream's backing indices (`.ds-dmarc_aggregate-…`), and an
ILM policy you attach rolls them over and deletes old ones, so there is
one stable name per report type and far fewer shards.
Before a parsedmarc process first saves to a stream, it puts an index
template named after the stream and creates the stream if it does not
exist yet. The template:
- matches exactly the stream's name, with no wildcard, so it never applies
to the dated indexes;
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
from the configuration, for each new backing index;
- maps `@timestamp` as a date. Every document parsedmarc saves carries
`@timestamp`: each record's begin date (`date_begin`) for aggregate
reports, which is the report's begin date unless parsedmarc split a
report spanning more than a day into daily records; the arrival date
for failure reports; and the report's begin date for SMTP TLS reports.
The other fields are mapped dynamically, as in the dated indexes;
- includes a component template named `<stream>@custom`, such as
`dmarc_aggregate@custom`, if one exists.
Data streams need Elasticsearch 8.7 or later, because the template uses
`ignore_missing_component_templates` to make the `@custom` component
template optional.
parsedmarc puts the template again in every process, so changes made to
it directly do not last. Put your own settings, such as the ILM policy,
in the `@custom` component template instead.
parsedmarc does not create or change any ILM policy. Here is an example
that rolls each stream over every 30 days, or sooner when a primary
shard reaches 50 GB, and deletes a backing index 180 days after its
rollover:
```text
PUT _ilm/policy/parsedmarc
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "50gb"
}
}
},
"delete": {
"min_age": "180d",
"actions": {
"delete": {}
}
}
}
}
}
```
Attach it to each stream through its `@custom` component template:
```text
PUT _component_template/dmarc_aggregate@custom
{
"template": {
"settings": {
"index.lifecycle.name": "parsedmarc"
}
}
}
```
Repeat that for `dmarc_failure@custom` and `smtp_tls@custom`, and for the
prefixed or suffixed names if you use them. The component template
applies only to backing indices created after it. A stream that already
exists has a write index with no policy, so nothing would ever roll it
over; attach the policy to its existing backing indices as well:
```text
PUT dmarc_aggregate/_settings
{
"index.lifecycle.name": "parsedmarc"
}
```
The delete phase's `min_age` counts from the rollover, not from the
report dates, so a backing index is deleted 180 days after it stopped
receiving documents, whatever dates the reports in it cover. If you need
retention by report date, keep using dated indexes and delete them by
name.
On Elastic Cloud Serverless, which has no ILM, use a
[data stream lifecycle] instead:
```text
PUT _data_stream/dmarc_aggregate/_lifecycle
{
"data_retention": "180d"
}
```
### Switching an existing installation to data streams
Dated indexes that already exist are left alone. They keep working next
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
check for duplicate reports searches both, so a report that is already
in a dated index is not saved again into a stream. You can delete the
dated indexes when they age out of your retention period, or move their
documents into the streams. Documents saved by earlier releases have no
`@timestamp`, which a data stream requires, so add it while reindexing.
Run parsedmarc once with `use_data_streams = True` first, so the stream
and its template exist; a reindex into a name that is not a data stream
yet creates an ordinary index of that name instead:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate-*"},
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
"script": {
"lang": "painless",
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
}
}
```
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
`dmarc_failure-*`. Delete the dated indexes once the reindex has
finished; until then, the dashboards count every document twice.
If saving fails with `cannot create the data stream dmarc_aggregate: an
index or alias with that name already exists`, an index or alias already
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
write into such an index rather than mixing new documents into it. Move
its documents to a dated name, which the
dashboards still match, and delete it:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate"},
"dest": {"index": "dmarc_aggregate-legacy"}
}
DELETE dmarc_aggregate
```
[data stream lifecycle]: https://www.elastic.co/docs/manage-data/lifecycle/data-stream
[data streams]: https://www.elastic.co/docs/manage-data/data-store/data-streams
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
+150 -3
View File
@@ -7,8 +7,155 @@ To set up visual dashboards of DMARC data, install OpenSearch and Grafana.
OpenSearch: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/
Grafana: https://grafana.com/docs/grafana/latest/setup-grafana/installation/
## Records retention
## Data streams and records retention
Starting in version 5.0.0, `parsedmarc` stores data in a separate
index for each day to make it easy to comply with records
retention regulations such as GDPR.
index for each day (or each month, with `monthly_indexes`), named after
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
easy to comply with records retention regulations such as GDPR: delete
the indexes for the dates you no longer keep.
Set `use_data_streams = True` in the `[opensearch]` section to write to
[data streams] instead. Each report type then goes to one data stream
named like the indexes without the date: `dmarc_aggregate`,
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
the `index_prefix_domain_map` prefix applied as usual. The cluster
creates the stream's backing indices (`.ds-dmarc_aggregate-000001`, …),
and an [ISM] policy you attach rolls them over and deletes old data, so
there is one stable name per report type and far fewer shards. Amazon
OpenSearch Serverless (`aws_service = aoss`) has no data streams or
ISM, so leave `use_data_streams` off there.
Before a parsedmarc process first saves to a stream, it puts an index
template named after the stream and creates the stream if it does not
exist yet. The template:
- matches exactly the stream's name, with no wildcard, so it never applies
to the dated indexes;
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
from the configuration, for each new backing index;
- maps `@timestamp` as a date. Every document parsedmarc saves carries
`@timestamp`: each record's begin date (`date_begin`) for aggregate
reports, which is the report's begin date unless parsedmarc split a
report spanning more than a day into daily records; the arrival date
for failure reports; and the report's begin date for SMTP TLS reports.
The other fields are mapped dynamically, as in the dated indexes.
parsedmarc puts the template again in every process, so changes made to
it directly do not last.
parsedmarc does not create or change any ISM policy. Here is an example
that rolls each stream over every 30 days, or sooner when a primary
shard reaches 50 GB, and deletes a backing index 180 days after it was
created. Its `ism_template` attaches it to every backing index the
listed streams create from now on. List the stream names themselves,
including any prefix or suffix: a `.ds-dmarc_aggregate-*` pattern does
not match the backing indices.
```text
PUT _plugins/_ism/policies/parsedmarc
{
"policy": {
"description": "parsedmarc data stream retention",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_index_age": "30d",
"min_primary_shard_size": "50gb"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {"min_index_age": "180d"}
}
]
},
{
"name": "delete",
"actions": [{"delete": {}}],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": ["dmarc_aggregate", "dmarc_failure", "smtp_tls"],
"priority": 100
}
]
}
}
```
The `ism_template` only applies to backing indices created after the
policy. To attach the policy to the backing indices of a stream that
already exists:
```text
POST _plugins/_ism/add/dmarc_aggregate
{
"policy_id": "parsedmarc"
}
```
`min_index_age` counts from when the backing index was created, not
from the report dates, so a backing index is deleted 180 days after it
was created, whatever dates the reports in it cover. If you need
retention by report date, keep using dated indexes and delete them by
name.
### Switching an existing installation to data streams
Dated indexes that already exist are left alone. They keep working next
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
check for duplicate reports searches both, so a report that is already
in a dated index is not saved again into a stream. You can delete the
dated indexes when they age out of your retention period, or move their
documents into the streams. Documents saved by earlier releases have no
`@timestamp`, which a data stream requires, so add it while reindexing.
Run parsedmarc once with `use_data_streams = True` first, so the stream
and its template exist; a reindex into a name that is not a data stream
yet creates an ordinary index of that name instead:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate-*"},
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
"script": {
"lang": "painless",
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
}
}
```
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
`dmarc_failure-*`. Delete the dated indexes once the reindex has
finished; until then, the dashboards count every document twice.
If saving fails with `cannot create the data stream dmarc_aggregate: an
index or alias with that name already exists`, an index or alias already
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
write into such an index rather than mixing new documents into it. Move
its documents to a dated name, which the
dashboards still match, and delete it:
```text
POST _reindex
{
"source": {"index": "dmarc_aggregate"},
"dest": {"index": "dmarc_aggregate-legacy"}
}
DELETE dmarc_aggregate
```
[data streams]: https://docs.opensearch.org/latest/im-plugin/data-streams/
[ISM]: https://docs.opensearch.org/latest/im-plugin/ism/index/
+28 -6
View File
@@ -591,16 +591,27 @@ The full set of configuration options is:
- `index_suffix` - str: A suffix to apply to the index names
- `index_prefix` - str: A prefix to apply to the index names
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
(ignored, with a warning, when `use_data_streams` is `True`)
- `number_of_shards` - int: The number of shards to use when
creating the index (Default: `1`)
creating the index, or each backing index of a data stream
(Default: `1`)
- `number_of_replicas` - int: The number of replicas to use when
creating the index (Default: `0`)
creating the index, or each backing index of a data stream
(Default: `0`)
- `use_data_streams` - bool: Save each report type to a data stream
named like the index without the date suffix (e.g. `dmarc_aggregate`)
instead of to daily or monthly indexes. parsedmarc creates the stream
and its index template; retention is left to an ILM policy you attach. Requires Elasticsearch
8.7 or later.
See [Data streams and records retention](elasticsearch.md#data-streams-and-records-retention)
(Default: `False`)
- `serverless` - bool: Set to `True` when targeting an Elastic Cloud
Serverless project. Serverless manages sharding and replication itself
and rejects the `number_of_shards` / `number_of_replicas` index settings
with HTTP 400. With this flag set, parsedmarc strips those keys from the
settings sent at index creation; any other settings (e.g.
`refresh_interval`) are passed through unchanged (Default: `False`)
settings sent at index creation (or in a data stream's index template);
any other settings (e.g. `refresh_interval`) are passed through
unchanged (Default: `False`)
- `opensearch`
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
@@ -629,10 +640,21 @@ The full set of configuration options is:
- `index_suffix` - str: A suffix to apply to the index names
- `index_prefix` - str: A prefix to apply to the index names
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
(ignored, with a warning, when `use_data_streams` is `True`)
- `number_of_shards` - int: The number of shards to use when
creating the index (Default: `1`)
creating the index, or each backing index of a data stream
(Default: `1`)
- `number_of_replicas` - int: The number of replicas to use when
creating the index (Default: `0`)
creating the index, or each backing index of a data stream
(Default: `0`)
- `use_data_streams` - bool: Save each report type to a data stream
named like the index without the date suffix (e.g. `dmarc_aggregate`)
instead of to daily or monthly indexes. parsedmarc creates the stream
and its index template; retention is left to an ISM policy you attach.
OpenSearch rejects data stream names that start with `.`, so an
`index_prefix` starting with `.` cannot be used with this option.
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
(Default: `False`)
- `splunk_hec`
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
- `token` - str: The HEC token