mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-21 22:42:39 +00:00
Add opt-in Elasticsearch/OpenSearch data streams (#937)
* Add opt-in Elasticsearch/OpenSearch data streams (#842) New use_data_streams option in [elasticsearch] and [opensearch] saves each report type to a data stream named like today's index without the date suffix, instead of daily or monthly indexes. parsedmarc puts an exact-name index template (priority 501, @timestamp mapped as date, <stream>@custom component template on Elasticsearch) and creates the stream; retention is left to operator-managed ILM/ISM policies, with documented examples. Dated indexes are untouched and still searched for duplicates and matched by the dashboards. Every document now carries @timestamp and is saved with op_type=create in both modes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix data stream prose found by cross-check review (layer 1) Aggregate @timestamp is each record's begin date, not the report's; the cluster only rolls backing indices over under an attached policy; move the Elasticsearch 8.7 note after the template it refers to; clarify the name-collision paragraph, OpenSearch-side comments, the migrate_indexes docstring, and the test-module headers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tighten names and wiring tests found by cross-check review (layer 2) Split the use_data_streams CLI passthrough test per backend so a call site reading the other section's flag fails; name the dated-mode tests for both changes they assert; compute index_date only in dated mode in every save function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Align data stream docs with Elastic/OpenSearch references (layer 3) Say the stream must exist before reindexing into it (ES reindex API: auto-creation needs a data-stream template); an existing stream's write index needs the ILM policy attached directly since nothing rolls it over otherwise; target the stream name in the _settings example; note OpenSearch's leading-dot rule for stream names and Fleet's template priorities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cover the Loki client-initialization error wrapper _parse_config accepts a [loki] token alongside username/password; only the LokiClient constructor rejects the combination. Exercise that path through _init_output_clients with the real constructor so the RuntimeError("Loki: ...") wrapper Codecov flagged as uncovered is tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
70f96d0734
commit
56e5a09923
10 files changed
+1765
-80
No files matched your search
@@ -405,13 +405,170 @@ curl -X POST "http://localhost:9200/smtp_tls*/_update_by_query?conflicts=proceed
|
||||
It works identically on OpenSearch; just adjust the URL and credentials, same
|
||||
as the aggregate command above.
|
||||
|
||||
## Records retention
|
||||
## Data streams and records retention
|
||||
|
||||
Starting in version 5.0.0, `parsedmarc` stores data in a separate
|
||||
index for each day to make it easy to comply with records
|
||||
retention regulations such as GDPR. For more information,
|
||||
index for each day (or each month, with `monthly_indexes`), named after
|
||||
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
|
||||
easy to comply with records retention regulations such as GDPR: delete
|
||||
the indexes for the dates you no longer keep. For more information,
|
||||
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
|
||||
|
||||
Set `use_data_streams = True` in the `[elasticsearch]` section to write
|
||||
to [data streams] instead. Each report type then goes to one data stream
|
||||
named like the indexes without the date: `dmarc_aggregate`,
|
||||
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
|
||||
the `index_prefix_domain_map` prefix applied as usual. The cluster
|
||||
creates the stream's backing indices (`.ds-dmarc_aggregate-…`), and an
|
||||
ILM policy you attach rolls them over and deletes old ones, so there is
|
||||
one stable name per report type and far fewer shards.
|
||||
|
||||
Before a parsedmarc process first saves to a stream, it puts an index
|
||||
template named after the stream and creates the stream if it does not
|
||||
exist yet. The template:
|
||||
|
||||
- matches exactly the stream's name, with no wildcard, so it never applies
|
||||
to the dated indexes;
|
||||
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
|
||||
from the configuration, for each new backing index;
|
||||
- maps `@timestamp` as a date. Every document parsedmarc saves carries
|
||||
`@timestamp`: each record's begin date (`date_begin`) for aggregate
|
||||
reports, which is the report's begin date unless parsedmarc split a
|
||||
report spanning more than a day into daily records; the arrival date
|
||||
for failure reports; and the report's begin date for SMTP TLS reports.
|
||||
The other fields are mapped dynamically, as in the dated indexes;
|
||||
- includes a component template named `<stream>@custom`, such as
|
||||
`dmarc_aggregate@custom`, if one exists.
|
||||
|
||||
Data streams need Elasticsearch 8.7 or later, because the template uses
|
||||
`ignore_missing_component_templates` to make the `@custom` component
|
||||
template optional.
|
||||
|
||||
parsedmarc puts the template again in every process, so changes made to
|
||||
it directly do not last. Put your own settings, such as the ILM policy,
|
||||
in the `@custom` component template instead.
|
||||
|
||||
parsedmarc does not create or change any ILM policy. Here is an example
|
||||
that rolls each stream over every 30 days, or sooner when a primary
|
||||
shard reaches 50 GB, and deletes a backing index 180 days after its
|
||||
rollover:
|
||||
|
||||
```text
|
||||
PUT _ilm/policy/parsedmarc
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover": {
|
||||
"max_age": "30d",
|
||||
"max_primary_shard_size": "50gb"
|
||||
}
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "180d",
|
||||
"actions": {
|
||||
"delete": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Attach it to each stream through its `@custom` component template:
|
||||
|
||||
```text
|
||||
PUT _component_template/dmarc_aggregate@custom
|
||||
{
|
||||
"template": {
|
||||
"settings": {
|
||||
"index.lifecycle.name": "parsedmarc"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Repeat that for `dmarc_failure@custom` and `smtp_tls@custom`, and for the
|
||||
prefixed or suffixed names if you use them. The component template
|
||||
applies only to backing indices created after it. A stream that already
|
||||
exists has a write index with no policy, so nothing would ever roll it
|
||||
over; attach the policy to its existing backing indices as well:
|
||||
|
||||
```text
|
||||
PUT dmarc_aggregate/_settings
|
||||
{
|
||||
"index.lifecycle.name": "parsedmarc"
|
||||
}
|
||||
```
|
||||
|
||||
The delete phase's `min_age` counts from the rollover, not from the
|
||||
report dates, so a backing index is deleted 180 days after it stopped
|
||||
receiving documents, whatever dates the reports in it cover. If you need
|
||||
retention by report date, keep using dated indexes and delete them by
|
||||
name.
|
||||
|
||||
On Elastic Cloud Serverless, which has no ILM, use a
|
||||
[data stream lifecycle] instead:
|
||||
|
||||
```text
|
||||
PUT _data_stream/dmarc_aggregate/_lifecycle
|
||||
{
|
||||
"data_retention": "180d"
|
||||
}
|
||||
```
|
||||
|
||||
### Switching an existing installation to data streams
|
||||
|
||||
Dated indexes that already exist are left alone. They keep working next
|
||||
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
|
||||
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
|
||||
check for duplicate reports searches both, so a report that is already
|
||||
in a dated index is not saved again into a stream. You can delete the
|
||||
dated indexes when they age out of your retention period, or move their
|
||||
documents into the streams. Documents saved by earlier releases have no
|
||||
`@timestamp`, which a data stream requires, so add it while reindexing.
|
||||
Run parsedmarc once with `use_data_streams = True` first, so the stream
|
||||
and its template exist; a reindex into a name that is not a data stream
|
||||
yet creates an ordinary index of that name instead:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate-*"},
|
||||
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
|
||||
`dmarc_failure-*`. Delete the dated indexes once the reindex has
|
||||
finished; until then, the dashboards count every document twice.
|
||||
|
||||
If saving fails with `cannot create the data stream dmarc_aggregate: an
|
||||
index or alias with that name already exists`, an index or alias already
|
||||
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
|
||||
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
|
||||
write into such an index rather than mixing new documents into it. Move
|
||||
its documents to a dated name, which the
|
||||
dashboards still match, and delete it:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate"},
|
||||
"dest": {"index": "dmarc_aggregate-legacy"}
|
||||
}
|
||||
|
||||
DELETE dmarc_aggregate
|
||||
```
|
||||
|
||||
[data stream lifecycle]: https://www.elastic.co/docs/manage-data/lifecycle/data-stream
|
||||
[data streams]: https://www.elastic.co/docs/manage-data/data-store/data-streams
|
||||
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
|
||||
[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
|
||||
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
|
||||
|
||||
+150
-3
@@ -7,8 +7,155 @@ To set up visual dashboards of DMARC data, install OpenSearch and Grafana.
|
||||
OpenSearch: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/
|
||||
Grafana: https://grafana.com/docs/grafana/latest/setup-grafana/installation/
|
||||
|
||||
## Records retention
|
||||
## Data streams and records retention
|
||||
|
||||
Starting in version 5.0.0, `parsedmarc` stores data in a separate
|
||||
index for each day to make it easy to comply with records
|
||||
retention regulations such as GDPR.
|
||||
index for each day (or each month, with `monthly_indexes`), named after
|
||||
the report's date, such as `dmarc_aggregate-2024-03-31`. That makes it
|
||||
easy to comply with records retention regulations such as GDPR: delete
|
||||
the indexes for the dates you no longer keep.
|
||||
|
||||
Set `use_data_streams = True` in the `[opensearch]` section to write to
|
||||
[data streams] instead. Each report type then goes to one data stream
|
||||
named like the indexes without the date: `dmarc_aggregate`,
|
||||
`dmarc_failure`, and `smtp_tls`, with `index_prefix`, `index_suffix`, or
|
||||
the `index_prefix_domain_map` prefix applied as usual. The cluster
|
||||
creates the stream's backing indices (`.ds-dmarc_aggregate-000001`, …),
|
||||
and an [ISM] policy you attach rolls them over and deletes old data, so
|
||||
there is one stable name per report type and far fewer shards. Amazon
|
||||
OpenSearch Serverless (`aws_service = aoss`) has no data streams or
|
||||
ISM, so leave `use_data_streams` off there.
|
||||
|
||||
Before a parsedmarc process first saves to a stream, it puts an index
|
||||
template named after the stream and creates the stream if it does not
|
||||
exist yet. The template:
|
||||
|
||||
- matches exactly the stream's name, with no wildcard, so it never applies
|
||||
to the dated indexes;
|
||||
- has priority 501 and sets `number_of_shards` and `number_of_replicas`
|
||||
from the configuration, for each new backing index;
|
||||
- maps `@timestamp` as a date. Every document parsedmarc saves carries
|
||||
`@timestamp`: each record's begin date (`date_begin`) for aggregate
|
||||
reports, which is the report's begin date unless parsedmarc split a
|
||||
report spanning more than a day into daily records; the arrival date
|
||||
for failure reports; and the report's begin date for SMTP TLS reports.
|
||||
The other fields are mapped dynamically, as in the dated indexes.
|
||||
|
||||
parsedmarc puts the template again in every process, so changes made to
|
||||
it directly do not last.
|
||||
|
||||
parsedmarc does not create or change any ISM policy. Here is an example
|
||||
that rolls each stream over every 30 days, or sooner when a primary
|
||||
shard reaches 50 GB, and deletes a backing index 180 days after it was
|
||||
created. Its `ism_template` attaches it to every backing index the
|
||||
listed streams create from now on. List the stream names themselves,
|
||||
including any prefix or suffix: a `.ds-dmarc_aggregate-*` pattern does
|
||||
not match the backing indices.
|
||||
|
||||
```text
|
||||
PUT _plugins/_ism/policies/parsedmarc
|
||||
{
|
||||
"policy": {
|
||||
"description": "parsedmarc data stream retention",
|
||||
"default_state": "hot",
|
||||
"states": [
|
||||
{
|
||||
"name": "hot",
|
||||
"actions": [
|
||||
{
|
||||
"rollover": {
|
||||
"min_index_age": "30d",
|
||||
"min_primary_shard_size": "50gb"
|
||||
}
|
||||
}
|
||||
],
|
||||
"transitions": [
|
||||
{
|
||||
"state_name": "delete",
|
||||
"conditions": {"min_index_age": "180d"}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "delete",
|
||||
"actions": [{"delete": {}}],
|
||||
"transitions": []
|
||||
}
|
||||
],
|
||||
"ism_template": [
|
||||
{
|
||||
"index_patterns": ["dmarc_aggregate", "dmarc_failure", "smtp_tls"],
|
||||
"priority": 100
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `ism_template` only applies to backing indices created after the
|
||||
policy. To attach the policy to the backing indices of a stream that
|
||||
already exists:
|
||||
|
||||
```text
|
||||
POST _plugins/_ism/add/dmarc_aggregate
|
||||
{
|
||||
"policy_id": "parsedmarc"
|
||||
}
|
||||
```
|
||||
|
||||
`min_index_age` counts from when the backing index was created, not
|
||||
from the report dates, so a backing index is deleted 180 days after it
|
||||
was created, whatever dates the reports in it cover. If you need
|
||||
retention by report date, keep using dated indexes and delete them by
|
||||
name.
|
||||
|
||||
### Switching an existing installation to data streams
|
||||
|
||||
Dated indexes that already exist are left alone. They keep working next
|
||||
to the streams: the shipped dashboards use the `dmarc_aggregate*`,
|
||||
`dmarc_f*`, and `smtp_tls*` index patterns, which match both, and the
|
||||
check for duplicate reports searches both, so a report that is already
|
||||
in a dated index is not saved again into a stream. You can delete the
|
||||
dated indexes when they age out of your retention period, or move their
|
||||
documents into the streams. Documents saved by earlier releases have no
|
||||
`@timestamp`, which a data stream requires, so add it while reindexing.
|
||||
Run parsedmarc once with `use_data_streams = True` first, so the stream
|
||||
and its template exist; a reindex into a name that is not a data stream
|
||||
yet creates an ordinary index of that name instead:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate-*"},
|
||||
"dest": {"index": "dmarc_aggregate", "op_type": "create"},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "ctx._source['@timestamp'] = ctx._source.date_begin"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `date_begin` for `smtp_tls-*` too, and `arrival_date` for
|
||||
`dmarc_failure-*`. Delete the dated indexes once the reindex has
|
||||
finished; until then, the dashboards count every document twice.
|
||||
|
||||
If saving fails with `cannot create the data stream dmarc_aggregate: an
|
||||
index or alias with that name already exists`, an index or alias already
|
||||
uses the stream's name. parsedmarc releases before 5.0.0 wrote to an
|
||||
undated `dmarc_aggregate` index, for example. parsedmarc refuses to
|
||||
write into such an index rather than mixing new documents into it. Move
|
||||
its documents to a dated name, which the
|
||||
dashboards still match, and delete it:
|
||||
|
||||
```text
|
||||
POST _reindex
|
||||
{
|
||||
"source": {"index": "dmarc_aggregate"},
|
||||
"dest": {"index": "dmarc_aggregate-legacy"}
|
||||
}
|
||||
|
||||
DELETE dmarc_aggregate
|
||||
```
|
||||
|
||||
[data streams]: https://docs.opensearch.org/latest/im-plugin/data-streams/
|
||||
[ISM]: https://docs.opensearch.org/latest/im-plugin/ism/index/
|
||||
+28
-6
@@ -591,16 +591,27 @@ The full set of configuration options is:
|
||||
- `index_suffix` - str: A suffix to apply to the index names
|
||||
- `index_prefix` - str: A prefix to apply to the index names
|
||||
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
|
||||
(ignored, with a warning, when `use_data_streams` is `True`)
|
||||
- `number_of_shards` - int: The number of shards to use when
|
||||
creating the index (Default: `1`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `1`)
|
||||
- `number_of_replicas` - int: The number of replicas to use when
|
||||
creating the index (Default: `0`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `0`)
|
||||
- `use_data_streams` - bool: Save each report type to a data stream
|
||||
named like the index without the date suffix (e.g. `dmarc_aggregate`)
|
||||
instead of to daily or monthly indexes. parsedmarc creates the stream
|
||||
and its index template; retention is left to an ILM policy you attach. Requires Elasticsearch
|
||||
8.7 or later.
|
||||
See [Data streams and records retention](elasticsearch.md#data-streams-and-records-retention)
|
||||
(Default: `False`)
|
||||
- `serverless` - bool: Set to `True` when targeting an Elastic Cloud
|
||||
Serverless project. Serverless manages sharding and replication itself
|
||||
and rejects the `number_of_shards` / `number_of_replicas` index settings
|
||||
with HTTP 400. With this flag set, parsedmarc strips those keys from the
|
||||
settings sent at index creation; any other settings (e.g.
|
||||
`refresh_interval`) are passed through unchanged (Default: `False`)
|
||||
settings sent at index creation (or in a data stream's index template);
|
||||
any other settings (e.g. `refresh_interval`) are passed through
|
||||
unchanged (Default: `False`)
|
||||
- `opensearch`
|
||||
|
||||
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
|
||||
@@ -629,10 +640,21 @@ The full set of configuration options is:
|
||||
- `index_suffix` - str: A suffix to apply to the index names
|
||||
- `index_prefix` - str: A prefix to apply to the index names
|
||||
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
|
||||
(ignored, with a warning, when `use_data_streams` is `True`)
|
||||
- `number_of_shards` - int: The number of shards to use when
|
||||
creating the index (Default: `1`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `1`)
|
||||
- `number_of_replicas` - int: The number of replicas to use when
|
||||
creating the index (Default: `0`)
|
||||
creating the index, or each backing index of a data stream
|
||||
(Default: `0`)
|
||||
- `use_data_streams` - bool: Save each report type to a data stream
|
||||
named like the index without the date suffix (e.g. `dmarc_aggregate`)
|
||||
instead of to daily or monthly indexes. parsedmarc creates the stream
|
||||
and its index template; retention is left to an ISM policy you attach.
|
||||
OpenSearch rejects data stream names that start with `.`, so an
|
||||
`index_prefix` starting with `.` cannot be used with this option.
|
||||
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
|
||||
(Default: `False`)
|
||||
- `splunk_hec`
|
||||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC)
|
||||
- `token` - str: The HEC token
|
||||
|
||||
Reference in new issue
Block a user