mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-30 18:57:17 +00:00
Finish forensic→failure rename: archive-folder migration + dashboard/doc cleanup (#776)
The forensic→failure rename (#659) left a few loose ends and one deliberate hold-back. This closes them. Leftover rename misses (broken paths / stale canonical names): - CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/* - dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml → dmarc_failure_dashboard.xml (the file was already renamed; the import path and view name were not) - docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example - samples/parsedmarc.ini: save_forensic → save_failure - pyproject.toml, README.md: canonical "failure" naming (ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.) Archive subfolder rename + on-startup migration: - New failure reports now archive to <archive>/Failure (was <archive>/Forensic). - _migrate_forensic_archive_folder() runs once on startup (best-effort): renames Forensic→Failure when no Failure folder exists yet, merges the two when both exist, no-ops when there's no legacy folder, and logs-and-skips a mailbox it can't reorganize (warn, don't crash). This consolidates pre- and post-rename failure reports into one folder, replacing the previously documented decision to keep the folder named Forensic to avoid a split archive. Uses the folder-management API (folder_exists / rename_folder / merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0. Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana): - Forensic panel titles + the datasource label → Failure; the fo-column display label and its linked byName field-override matcher both → "Failure Policy" (changed together so the column-width override keeps matching). - dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana dashboards); RESEED wipe loop now also clears dmarc_failure* indices. - Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an orphan export accidentally committed in #736 and referenced by nothing. Tests (tests/test_init.py): - TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds). - TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir can't reproduce — a backend that raises mid-operation must warn, not crash. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
327fcff2b9
commit
a6778707d7
@@ -176,9 +176,10 @@ else
|
||||
echo " RESEED=1: wiping existing parsedmarc data from all backends"
|
||||
# ES 8.x rejects wildcard DELETEs by default
|
||||
# (action.destructive_requires_name=true). Enumerate the daily indexes
|
||||
# parsedmarc rolls (dmarc_aggregate-YYYY-MM-DD, dmarc_forensic-...,
|
||||
# smtp_tls-...) and DELETE each one explicitly.
|
||||
for prefix in dmarc_aggregate dmarc_forensic smtp_tls; do
|
||||
# parsedmarc rolls (dmarc_aggregate-YYYY-MM-DD, dmarc_failure-...,
|
||||
# smtp_tls-...) and DELETE each one explicitly. dmarc_forensic-* is the
|
||||
# pre-rename failure index family, kept here so RESEED clears old data.
|
||||
for prefix in dmarc_aggregate dmarc_failure dmarc_forensic smtp_tls; do
|
||||
for idx in $(curl -sf "http://localhost:9200/_cat/indices/${prefix}*?h=index" 2>/dev/null); do
|
||||
curl -sS -X DELETE "http://localhost:9200/${idx}" >/dev/null 2>&1 || true
|
||||
done
|
||||
@@ -231,7 +232,7 @@ else
|
||||
samples/aggregate/protection.outlook.com!example.com!1711756800!1711843200.xml
|
||||
samples/aggregate/usssa.com!example.com!1538784000!1538870399.xml
|
||||
samples/aggregate/veeam.com!example.com!1530133200!1530219600.xml
|
||||
samples/forensic/*.eml
|
||||
samples/failure/*.eml
|
||||
samples/smtp_tls/*.json
|
||||
samples/smtp_tls/google.com_smtp_tls_report.eml
|
||||
)
|
||||
@@ -268,7 +269,9 @@ log "Configuring Grafana datasources"
|
||||
# Two Elasticsearch datasources, one per index family, matching the dashboard's
|
||||
# template variables (dmarc-ag and dmarc-fo). Skipped when already present.
|
||||
declare -a GF_DS_NAMES=("dmarc-ag" "dmarc-fo")
|
||||
declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_forensic*")
|
||||
# dmarc_f* matches both pre-rename dmarc_forensic* and post-rename
|
||||
# dmarc_failure* indices, mirroring the OpenSearch/Kibana dashboards.
|
||||
declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_f*")
|
||||
declare -a GF_DS_TIME=("date_range" "arrival_date")
|
||||
for i in 0 1; do
|
||||
name="${GF_DS_NAMES[$i]}"
|
||||
@@ -335,7 +338,7 @@ splunk_import_view() {
|
||||
}
|
||||
|
||||
splunk_import_view dmarc_aggregate dashboards/splunk/dmarc_aggregate_dashboard.xml
|
||||
splunk_import_view dmarc_forensic dashboards/splunk/dmarc_forensic_dashboard.xml
|
||||
splunk_import_view dmarc_failure dashboards/splunk/dmarc_failure_dashboard.xml
|
||||
splunk_import_view smtp_tls dashboards/splunk/smtp_tls_dashboard.xml
|
||||
|
||||
cat <<EOF
|
||||
|
||||
Reference in New Issue
Block a user