Finish forensic→failure rename: archive-folder migration + dashboard/doc cleanup (#776)

The forensic→failure rename (#659) left a few loose ends and one deliberate
hold-back. This closes them.

Leftover rename misses (broken paths / stale canonical names):
- CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/*
- dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml
  → dmarc_failure_dashboard.xml (the file was already renamed; the import path
  and view name were not)
- docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example
- samples/parsedmarc.ini: save_forensic → save_failure
- pyproject.toml, README.md: canonical "failure" naming
(ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.)

Archive subfolder rename + on-startup migration:
- New failure reports now archive to <archive>/Failure (was <archive>/Forensic).
- _migrate_forensic_archive_folder() runs once on startup (best-effort):
  renames Forensic→Failure when no Failure folder exists yet, merges the two
  when both exist, no-ops when there's no legacy folder, and logs-and-skips a
  mailbox it can't reorganize (warn, don't crash). This consolidates pre- and
  post-rename failure reports into one folder, replacing the previously
  documented decision to keep the folder named Forensic to avoid a split
  archive. Uses the folder-management API (folder_exists / rename_folder /
  merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0.

Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana):
- Forensic panel titles + the datasource label → Failure; the fo-column display
  label and its linked byName field-override matcher both → "Failure Policy"
  (changed together so the column-width override keeps matching).
- dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both
  pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana
  dashboards); RESEED wipe loop now also clears dmarc_failure* indices.
- Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an
  orphan export accidentally committed in #736 and referenced by nothing.

Tests (tests/test_init.py):
- TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via
  mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full
  get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds).
- TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir
  can't reproduce — a backend that raises mid-operation must warn, not crash.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-05-21 12:29:40 -04:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 327fcff2b9
commit a6778707d7
11 changed files with 181 additions and 5927 deletions
+9 -6
View File
@@ -176,9 +176,10 @@ else
echo " RESEED=1: wiping existing parsedmarc data from all backends"
# ES 8.x rejects wildcard DELETEs by default
# (action.destructive_requires_name=true). Enumerate the daily indexes
# parsedmarc rolls (dmarc_aggregate-YYYY-MM-DD, dmarc_forensic-...,
# smtp_tls-...) and DELETE each one explicitly.
for prefix in dmarc_aggregate dmarc_forensic smtp_tls; do
# parsedmarc rolls (dmarc_aggregate-YYYY-MM-DD, dmarc_failure-...,
# smtp_tls-...) and DELETE each one explicitly. dmarc_forensic-* is the
# pre-rename failure index family, kept here so RESEED clears old data.
for prefix in dmarc_aggregate dmarc_failure dmarc_forensic smtp_tls; do
for idx in $(curl -sf "http://localhost:9200/_cat/indices/${prefix}*?h=index" 2>/dev/null); do
curl -sS -X DELETE "http://localhost:9200/${idx}" >/dev/null 2>&1 || true
done
@@ -231,7 +232,7 @@ else
samples/aggregate/protection.outlook.com!example.com!1711756800!1711843200.xml
samples/aggregate/usssa.com!example.com!1538784000!1538870399.xml
samples/aggregate/veeam.com!example.com!1530133200!1530219600.xml
samples/forensic/*.eml
samples/failure/*.eml
samples/smtp_tls/*.json
samples/smtp_tls/google.com_smtp_tls_report.eml
)
@@ -268,7 +269,9 @@ log "Configuring Grafana datasources"
# Two Elasticsearch datasources, one per index family, matching the dashboard's
# template variables (dmarc-ag and dmarc-fo). Skipped when already present.
declare -a GF_DS_NAMES=("dmarc-ag" "dmarc-fo")
declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_forensic*")
# dmarc_f* matches both pre-rename dmarc_forensic* and post-rename
# dmarc_failure* indices, mirroring the OpenSearch/Kibana dashboards.
declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_f*")
declare -a GF_DS_TIME=("date_range" "arrival_date")
for i in 0 1; do
name="${GF_DS_NAMES[$i]}"
@@ -335,7 +338,7 @@ splunk_import_view() {
}
splunk_import_view dmarc_aggregate dashboards/splunk/dmarc_aggregate_dashboard.xml
splunk_import_view dmarc_forensic dashboards/splunk/dmarc_forensic_dashboard.xml
splunk_import_view dmarc_failure dashboards/splunk/dmarc_failure_dashboard.xml
splunk_import_view smtp_tls dashboards/splunk/smtp_tls_dashboard.xml
cat <<EOF