Don't archive/delete mailbox messages until output destinations confirm the save (#863)

* Don't archive/delete mailbox messages until output destinations confirm the save (#242)

get_dmarc_reports_from_mailbox() gains a keyword-only save_callback,
invoked once per fetched batch before any message is deleted or moved.
A False return (or an exception) marks the batch unsaved: its messages
stay in the reports folder for retry, and the aggregate-report dedup
keys staged for that batch are dropped so the retry reparses instead of
skipping. watch_inbox() passes its callback through as save_callback,
so watch mode gets the same protection.

To bound duplicate delivery to destinations that don't deduplicate, a
new [mailbox] max_unsaved_retries option (default 2) caps retries: a
message whose batch has failed the initial attempt plus that many
retries moves to {archive_folder}/Unsaved -- never deleted, whatever
the delete options say. Counts are process-local, so the cap applies
across watch-mode checks; one-shot runs retry indefinitely, which is
the safe direction. A raising callback (the CLI's, under
fail_on_output_error) is counted against the cap the same way before
the exception is re-raised, since mailsuite's IMAP and Maildir watch
loops swallow exceptions and keep checking.

CLI: process_reports() now returns its output-error list and the
mailbox_save_callback adapter feeds that verdict to the library;
save_output() failures (the one uncaught destination) are recorded like
every other sink's; file/mbox-derived reports are saved in a separate
pass from mailbox batches so nothing is saved twice; the combined
results fed to email_results() are filtered by index_prefix_domain_map
explicitly, restoring the SMTP TLS filtering the emailed summary lost
when saving moved into per-batch callbacks.

Credit to @mkilijanek for the original approach in #823.

Fixes #242

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address automated review feedback on #863

- Keep a message's failure counter until its move to Unsaved actually
  succeeds. The counter was popped when the message was classified
  over-cap, before the move was attempted, so a failed move handed the
  still-in-place message a fresh set of under-cap retries and duplicate
  deliveries; now the next failed save classifies it over-cap again and
  re-attempts the move. Regression-tested at cap 1, where the reset
  would observably leave the message in the INBOX instead of moving it.
- Use sys.exit(1) instead of the site-dependent exit() built-in in the
  two new ParserError handlers.
- Close the sample files opened by the new Maildir tests via context
  managers.
- Keep the docs' section-link text on one line and fix the adjacent
  pre-existing "a IMAP" typo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot re-review feedback on #863

- Document mailbox_save_callback's raise path: with fail_on_output_error
  it raises ParserError via process_reports() instead of returning
  False, and the library counts that as an unsaved batch too.
- Use str(error_) instead of error_.__str__() in the new File output
  handler.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Reject negative max_unsaved_retries with a ValueError

The option is user-configurable (INI/env/kwarg); a negative value
silently behaved like 0. Both get_dmarc_reports_from_mailbox() and
watch_inbox() now validate it at the door alongside the existing
test/delete guard. watch_inbox() validates before entering the watch
loop, because a ValueError raised inside a check would be swallowed
and endlessly retried by the IMAP and Maildir backends' per-check
exception handling instead of surfacing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix two pre-existing bugs surfaced by review of #863

- MAGIC_JSON was b"\7b" -- the octal escape \7 (BEL) plus a literal
  "b", not 0x7B, the "{" every RFC 8259 JSON object begins with -- so
  extract_report() rejected plain uncompressed JSON. Every in-tree
  caller pre-guarded with its own zip/gzip or "{" check, masking the
  dead branch; the practical impact was extract_report() as a public
  API and application/tlsrpt+gzip attachments whose payload is really
  uncompressed JSON. Now b"\x7b".

- get_index_prefix() unconditionally indexed policies[0], while
  parse_smtp_tls_report_json() accepts a report whose policies list is
  empty -- an IndexError crash whenever index_prefix_domain_map was
  configured. An empty-policies report has no domain to map, so it is
  now treated as unmappable and excluded from prefix-mapped output.
  This code was moved into filter_smtp_tls_reports_for_index_prefix()
  by this PR, which widened its exposure to the email_results() path.

Both regression tests were verified to fail against the unfixed code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Rewrap the max_unsaved_retries docstring paragraph

The ValueError sentence was inserted without reflowing, leaving a
112-character line in a docstring wrapped at ~76; ruff formats code,
not docstring prose, so it slipped through lint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

docs/source/usage.md

@@ -186,6 +186,14 @@ The full set of configuration options are:
   - `fail_on_output_error` - bool: Exit with a non-zero status code if
       any configured output destination fails while saving/publishing
       reports (Default: `False`)
+
+    :::{note}
+    This option only controls the process exit code. Retaining mailbox
+    messages whose reports could not be saved is automatic and happens
+    either way — see
+    [Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved).
+    :::
+
   - `log_file` - str: Write log messages to a file at this path
   - `n_procs` - int: Number of processes to run in parallel when
       parsing report files passed directly as CLI arguments, messages
@@ -242,9 +250,20 @@ The full set of configuration options are:
   - `test` - bool: Do not move or delete messages
   - `batch_size` - int: Number of messages to read and process
       before saving. Default `10`. Use `0` for no limit.
-  - `check_timeout` - int: Number of seconds to wait for a IMAP
+  - `check_timeout` - int: Number of seconds to wait for an IMAP
       IDLE response or the number of seconds until the next
       mail check (Default: `30`)
+  - `max_unsaved_retries` - int: How many times a batch of messages
+      whose reports could not be saved is retried before its messages
+      are moved to the `Unsaved` archive subfolder instead of being
+      retried again (Default: `2`, i.e. the initial attempt plus two
+      retries). Use `0` to move messages on the first failed save;
+      negative values are rejected.
+      Failures are counted in memory, so the cap applies across watch-mode
+      checks within one long-running process, not across separate one-shot
+      runs. See
+      [Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
+      below.
   - `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
       Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
       Defaults to `1d` if incorrect value is provided.
@@ -850,6 +869,78 @@ PUT _cluster/settings
 Increasing this value increases resource usage.
 :::
 
+### Mailbox messages are only archived once the reports are saved
+
+parsedmarc processes a mailbox in batches of `batch_size` messages. Each
+batch is written to every configured output destination *before* any of
+that batch's messages are archived or deleted. If any destination reports a
+failure — an Elasticsearch outage, an expired Splunk HEC token, an
+unreachable Kafka broker, a full `--output` disk — the whole batch is left
+in the reports folder and retried on the next run or watch-mode check, so a
+report is never removed from the mailbox while it exists nowhere else
+([issue #242](https://github.com/domainaware/parsedmarc/issues/242)).
+
+This is all-or-nothing per batch: archiving a batch because most
+destinations accepted it would still permanently lose the data for the one
+that didn't. It also applies regardless of `fail_on_output_error`, which
+only controls the process exit code. Messages that could not be parsed at
+all carry no report data, so they are filed under `Invalid` (or deleted per
+`delete_invalid`) as usual.
+
+A destination that is broken rather than briefly unavailable would
+otherwise be retried forever, so retries are capped. Once a message's batch
+has failed `max_unsaved_retries + 1` times (three times by default), that
+message is moved to `<archive_folder>/Unsaved` and stops being retried. **A
+message is never deleted on this path, whatever the `delete` options say.**
+To recover after fixing the output destination, either move the messages
+from `Archive/Unsaved` back into the reports folder, or run parsedmarc once
+with `reports_folder = Archive/Unsaved`.
+
+:::{note}
+The failure counts live in memory, so they are counted per parsedmarc
+process. In watch mode — a long-running process that checks the mailbox
+repeatedly — the cap works as described across checks. A one-shot run
+(`cron`, `systemd` timers) attempts each message exactly once and then
+exits, so its counts start over every time and the default cap is never
+reached: messages simply keep being retried on every run, which is the
+safe direction. Set `max_unsaved_retries = 0` if you want one-shot runs to
+move unsavable messages to `Unsaved` immediately instead.
+:::
+
+:::{warning}
+Retrying a batch means re-sending it. Output destinations that
+deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize
+an already-saved report — are unaffected, and S3 is idempotent because
+each report is written to an object key built from its type, date, and
+report ID, so a retry overwrites the same object. Kafka,
+Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the
+`--output` JSON/CSV files all append unconditionally, so each retry adds
+another copy of every report in the batch. That is why the default retry
+cap is deliberately low: at most three deliveries per report before its
+message is set aside in `Unsaved`. The summary email covers everything
+parsed in a run, including reports whose batch failed to save, so a report
+retried across runs can also appear in more than one summary email.
+:::
+
+:::{note}
+Not every destination can report a failed delivery. The webhook output
+deliberately logs and swallows its own HTTP and network errors, and the
+syslog and GELF outputs send through Python logging handlers, which
+swallow delivery errors by design — so an unreachable webhook, syslog, or
+GELF endpoint is *not* treated as a failed save and does not hold a
+batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
+Kafka, S3, PostgreSQL, Azure Log Analytics, and the `--output` files are
+all detected and do.
+:::
+
+:::{note}
+`since` interacts with retries: a message that ages out of the configured
+`since` window stops being fetched, and therefore stops being retried
+automatically. It is never deleted or moved — it simply stays in the
+reports folder until it is processed by a run with a wider (or no)
+`since` window.
+:::
+
 ## Environment variable configuration
 
 Any configuration option can be set via environment variables using the