Cover index_prefix_domain_map and index_suffix in index migrations (#868) (#869)

* Cover index_prefix_domain_map and index_suffix in index migrations (#868)

The startup Elasticsearch/OpenSearch index migrations built their target
index names from the [elasticsearch]/[opensearch] index_prefix and
index_suffix options alone, while the save path also honors
general.index_prefix_domain_map. A multi-tenant deployment therefore ran
the backfill guard `count` against dmarc_aggregate*/smtp_tls*, patterns
matching none of its real <tenant>_dmarc_aggregate-* indexes. The query
passes allow_no_indices=True, so a zero-match wildcard returns count 0 --
indistinguishable from "already backfilled" -- and the backfill was
skipped silently, with no log line at any level.

Resolve migration index names through a new _migration_index_names()
helper that widens both configurable axes: one name per tenant prefix in
the map plus the unprefixed name (unmapped domains are still saved
unprefixed), and, when an index_suffix is set, the unsuffixed name
alongside the suffixed one so history predating the suffix is covered. A
configured index_prefix still wins outright and suppresses the map
fan-out, matching save-time precedence. The key normalization is now
shared with get_index_prefix() via _normalize_index_prefix(), so the
names parsedmarc migrates cannot drift from the ones it writes. The
resolved lists are logged at DEBUG, and the SIGHUP reload path passes the
freshly parsed map, so a newly onboarded tenant is covered without a
restart.

Also repair the legacy published_policy.fo migration, which has been
unable to complete since mapping types were removed in Elasticsearch 7
(and never existed in OpenSearch): it read the field mapping in the
type-keyed response shape, so the check always fell through, and its
put_mapping() call passed a doc_type argument neither current client
accepts. It now reads either response shape, uses each client's current
signature, and takes its index names in a separate legacy_fo_indexes
argument -- exact names, since 5.0.0 introduced date-suffixed index names
in the same release that fixed the fo declaration, but prefixed and
suffixed where configured, since both options date back to 4.1.0. Tenant
prefixes are excluded: index_prefix_domain_map arrived in 8.19.0, and
this migration renames the index it rebuilds. The Elasticsearch copy,
removed as unreachable during the #806 client migration, is restored now
that the cause is understood.

Finally, reject an index_prefix_domain_map YAML file that is not a
mapping of string tenant names to lists of domains, instead of raising
mid-save on a non-string key or silently matching the wrong domains on a
scalar value -- `in` on a str is a substring test, so "example.co"
matches "example.com"
(https://docs.python.org/3/reference/expressions.html#membership-test-operations).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix OpenSearch capitalization and spacing in the usage docs

Copilot review of #869: the `index_prefix_domain_map` option line spelled
"OpenSearch" as "Opensearch" and had a doubled space before the type. Both
predate this branch but sit inside a hunk it rewrites. Also wrapped the line
to match the continuation-indent style of every other option in the list, and
corrected the same misspelling in the multi-tenant section a few lines above
the paragraph this branch added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Require index_prefix_domain_map domain lists to hold strings

Copilot review of #869: the shape check verified that each value was a
list but not what the list contained, so `tenant_a: [42]` passed and then
never compared equal to any domain the save path looks up -- the same
silent-misbehavior class the check exists to reject, and a contradiction
of the "any other shape is rejected at startup" claim in the docs.

Check the list's items too, and reword the error message, comment,
CHANGELOG and docs to state the rule the check now enforces: a mapping of
tenant names to lists of domain names, all strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make the legacy fo migration retry-safe, and nest its mapping body

Copilot review of #869, verified against live Elasticsearch 8.19 and
OpenSearch 3 containers rather than mocks.

Retry safety (a real defect): an attempt interrupted between creating the
-v2 index and deleting the original left debris that made create() fail
with "resource already exists" on every later startup, inside the same try
that swallows the error -- so the index was never migrated and the debris
document survived. Reproduced on a live cluster. Discard a leftover target
first; that is safe precisely because reaching this point means the
original still holds the data, since it is deleted only once the reindex
has succeeded.

Mapping body: the reviewer's concern that a dotted key under `properties`
risks a runtime failure does not hold -- both clusters accept it and
produce a byte-identical mapping, with the dot expanded into
published_policy -> properties -> fo. Switch to the nested object form
anyway, since dot expansion is conditional on the object's `subobjects`
setting and this shape never is, and derive the object/leaf names from the
dotted constant so the write cannot drift from the field the read looks up.

Both fo-migration suites now build per-name Index mocks. A single shared
mock cannot express "the original exists but its migration target does
not", which is the ordinary case and the one the retry fix turns on; the
happy path now also asserts the target index is never deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/source/elasticsearch.md

@@ -247,6 +247,25 @@ OpenSearch. Any error talking to the cluster (for example, no indexes yet
 on a fresh install) is logged as a warning and retried on the next startup,
 rather than aborting parsedmarc.
 
+Which index patterns it targets follows the ones parsedmarc writes to, and
+is logged at debug level on startup:
+
+- With `index_prefix_domain_map` configured in `[general]` and no
+  `index_prefix` set, every tenant prefix in the map gets its own index
+  pattern, alongside the unprefixed one — aggregate and failure reports for
+  a domain that is not in the map are still saved without a prefix. A
+  configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded
+  tenant is covered without a restart.
+- With an `index_prefix` set in `[elasticsearch]`/`[opensearch]`, only that
+  prefix is targeted, and the map is not consulted. That is deliberate: such
+  a deployment writes only under its own prefix, and an `_update_by_query`
+  against a pattern it does not write to could reach another deployment's
+  data on a shared cluster.
+- With an `index_suffix` set, both the suffixed and the unsuffixed pattern
+  are targeted, so documents indexed before the suffix was configured are
+  backfilled too. Note that the unsuffixed pattern also matches any *other*
+  suffix on the same cluster.
+
 If you upgrade the dashboards without pointing the new parsedmarc version
 at the cluster, or you'd rather control when the write load happens, you
 can still run the backfill manually. It is idempotent (documents that
@@ -314,7 +333,10 @@ curl -X POST "http://localhost:9200/dmarc_aggregate*/_update_by_query?conflicts=
 
 `wait_for_completion=false` returns a task ID — check progress with
 `GET _tasks/<task id>`. Adjust the index pattern if you use a custom
-`index_prefix`/`index_suffix`. After backfilling, re-import the updated
+`index_prefix`/`index_suffix`; with `index_prefix_domain_map`, run the
+command once per tenant prefix (`acme_corp_dmarc_aggregate*`) plus once for
+the unprefixed pattern, or widen it to `*dmarc_aggregate*` to cover every
+tenant in one pass. After backfilling, re-import the updated
 dashboards ndjson (the index pattern saved object changed too) per the
 import instructions above.
 

docs/source/usage.md

@@ -127,7 +127,8 @@ The full set of configuration options are:
       Elasticsearch, Splunk and/or S3
   - `save_smtp_tls` - bool: Save SMTP-STS report data to
       Elasticsearch, Splunk and/or S3
-  - `index_prefix_domain_map` -  bool: A path mapping of Opensearch/Elasticsearch index prefixes to domain names
+  - `index_prefix_domain_map` - str: Path to a YAML file mapping
+      OpenSearch/Elasticsearch index prefixes to domain names
   - `strip_attachment_payloads` - bool: Remove attachment
       payloads from results
   - `silent` - bool: Set this to `False` to output results to STDOUT
@@ -1167,12 +1168,16 @@ whalensolutions:
 
 Save it to disk where the user running ParseDMARC can read it, then set `index_prefix_domain_map` to that filepath in the `[general]` section of the ParseDMARC configuration file and do not set an `index_prefix` option in the `[elasticsearch]` or `[opensearch]` sections.
 
-When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of Opensearch or the ELK stack to only grant users access to the indexes that they need.
+When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
 
  :::{note}
  A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
 :::
 
+Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
+
+The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See [Backfilling the combined DKIM/SPF result fields](elasticsearch.md#backfilling-the-combined-dkimspf-result-fields) for the details.
+
 ## Running parsedmarc as a systemd service
 
 Use systemd to run `parsedmarc` as a service and process reports as