From 4217a076de4ed4da5c3a7469c0c0c449bc4308d5 Mon Sep 17 00:00:00 2001 From: Anael Mobilia Date: Sat, 24 Dec 2022 00:15:11 +0100 Subject: [PATCH] Doc - Add info on how to update max shards (#368) Add information on how to fix "Elasticsearch error: RequestError(400, 'validation_exception', 'Validation Failed: 1: this action would add [1] shards, but this cluster currently has [1000]/[1000] maximum normal shards open;" --- docs/source/index.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/source/index.md b/docs/source/index.md index 656bf71..5313508 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -385,6 +385,32 @@ known samples you want to save to that folder (e.g. malicious samples and non-sensitive legitimate samples). ::: +:::{warning} +Elasticsearch 8 change limits policy for shards, restricting by +default to 1000. parsedmarc use a shard per analyzed day. If you +have more than ~3 years of data, you will need to update this +limit. +Check current usage (from Management -> Dev Tools -> Console): +``` +GET /_cluster/health?pretty +... + "active_primary_shards": 932, + "active_shards": 932, +... +} +``` +Update the limit to 2k per exemple: +``` +PUT _cluster/settings +{ + "persistent" : { + "cluster.max_shards_per_node" : 2000 + } +} +``` +Be warned that increasing this value increase ressources usage. +::: + ## Sample aggregate report output Here are the results from parsing the[example](https://dmarc.org/wiki/FAQ#I_need_to_implement_aggregate_reports.2C_what_do_they_look_like.3F)