Merge master into google-secops-parser

Conflict resolutions:
- CHANGELOG.md: keep this branch's Unreleased section above the
  10.4.1-10.4.3 release sections added on master.
- pyproject.toml: keep both dependency additions (google-auth from this
  branch, httpx from master).

Follow-up to the merge: convert gsecops.py's str.format() calls to
f-strings, required by the ruff 0.16.0 UP030/UP032 enforcement adopted
on master after this branch's base.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/source/api.md

@@ -7,6 +7,13 @@
    :members:
 ```
 
+## parsedmarc.config
+
+```{eval-rst}
+.. automodule:: parsedmarc.config
+   :members:
+```
+
 ## parsedmarc.elastic
 
 ```{eval-rst}

docs/source/elasticsearch.md

@@ -3,7 +3,10 @@
 To set up visual dashboards of DMARC data, install Elasticsearch and Kibana.
 
 :::{note}
-Elasticsearch and Kibana 6 or later are required
+Elasticsearch and Kibana 8 or later are required (parsedmarc's 8.x Python
+client also supports Elasticsearch 9). OpenSearch users must use the
+`[opensearch]` configuration section instead — the Elasticsearch 8.x client
+refuses to connect to non-Elasticsearch clusters.
 :::
 
 ## Installation
@@ -223,6 +226,180 @@ Kibana index patterns with versions that match the upgraded indexes:
 7. Import `export.ndjson` by clicking Import from the Kibana
    Saved Objects page
 
+## Backfilling the combined DKIM/SPF result fields
+
+As of the version fixing [#169](https://github.com/domainaware/parsedmarc/issues/169),
+aggregate documents include `dkim_results_combined` and `spf_results_combined` —
+scalar string arrays that keep each auth result's selector/scope, domain, and
+result paired, which the dashboards' alignment-detail tables aggregate on.
+Reports saved by older versions lack these fields and will not appear in
+those tables.
+
+parsedmarc now backfills this automatically. On startup, it runs a cheap
+count query against each configured aggregate index pattern to check for
+documents that have DKIM or SPF results but are missing the corresponding
+combined field. If any are found, it submits the backfill as a background
+`_update_by_query` task (`wait_for_completion=false`), so startup is never
+blocked on it; progress is logged, including the task ID. The check itself
+is idempotent — once an index is fully backfilled, later startups see a
+count of 0 and log nothing further — and it works the same way on
+OpenSearch. Any error talking to the cluster (for example, no indexes yet
+on a fresh install) is logged as a warning and retried on the next startup,
+rather than aborting parsedmarc.
+
+Which index patterns it targets follows the ones parsedmarc writes to, and
+is logged at debug level on startup:
+
+- With `index_prefix_domain_map` configured in `[general]` and no
+  `index_prefix` set, every tenant prefix in the map gets its own index
+  pattern, alongside the unprefixed one — aggregate and failure reports for
+  a domain that is not in the map are still saved without a prefix. A
+  configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded
+  tenant is covered without a restart.
+- With an `index_prefix` set in `[elasticsearch]`/`[opensearch]`, only that
+  prefix is targeted, and the map is not consulted. That is deliberate: such
+  a deployment writes only under its own prefix, and an `_update_by_query`
+  against a pattern it does not write to could reach another deployment's
+  data on a shared cluster.
+- With an `index_suffix` set, both the suffixed and the unsuffixed pattern
+  are targeted, so documents indexed before the suffix was configured are
+  backfilled too. Note that the unsuffixed pattern also matches any *other*
+  suffix on the same cluster.
+
+If you upgrade the dashboards without pointing the new parsedmarc version
+at the cluster, or you'd rather control when the write load happens, you
+can still run the backfill manually. It is idempotent (documents that
+already have the fields are skipped), so it is safe to re-run. It works
+identically on OpenSearch; just adjust the URL and credentials. The query
+matches only documents that have at least one DKIM or SPF auth result and
+lack the corresponding combined field; documents with no auth results are
+skipped, because an `exists` query cannot see an empty array, and for
+search purposes an empty `dkim_results_combined` is identical to an
+absent one. Each result is matched on either its `domain` or its `result`
+subfield as defense in depth: an empty string indexes no text tokens and
+is invisible to `exists`, and the storage shape of every historical
+parsedmarc version can't be audited, so matching either subfield ensures
+no backfillable document is skipped.
+
+```bash
+curl -X POST "http://localhost:9200/dmarc_aggregate*/_update_by_query?conflicts=proceed&wait_for_completion=false" \
+  -H "Content-Type: application/json" -d '
+{
+  "query": {
+    "bool": {
+      "minimum_should_match": 1,
+      "should": [
+        {
+          "bool": {
+            "must": [
+              {
+                "bool": {
+                  "minimum_should_match": 1,
+                  "should": [
+                    {"exists": {"field": "dkim_results.domain"}},
+                    {"exists": {"field": "dkim_results.result"}}
+                  ]
+                }
+              }
+            ],
+            "must_not": [{"exists": {"field": "dkim_results_combined"}}]
+          }
+        },
+        {
+          "bool": {
+            "must": [
+              {
+                "bool": {
+                  "minimum_should_match": 1,
+                  "should": [
+                    {"exists": {"field": "spf_results.domain"}},
+                    {"exists": {"field": "spf_results.result"}}
+                  ]
+                }
+              }
+            ],
+            "must_not": [{"exists": {"field": "spf_results_combined"}}]
+          }
+        }
+      ]
+    }
+  },
+  "script": {
+    "lang": "painless",
+    "source": "List dk = new ArrayList(); def dr = ctx._source.dkim_results; if (dr != null) { if (!(dr instanceof List)) { dr = [dr]; } for (e in dr) { if (e == null) { continue; } def sel = e.selector != null ? e.selector : \"none\"; def dom = e.domain != null ? e.domain : \"none\"; def res = e.result != null ? e.result : \"none\"; dk.add(sel + \" / \" + dom + \" / \" + res); } } ctx._source.dkim_results_combined = dk; List sp = new ArrayList(); def sr = ctx._source.spf_results; if (sr != null) { if (!(sr instanceof List)) { sr = [sr]; } for (e in sr) { if (e == null) { continue; } def sc = e.scope != null ? e.scope : \"mfrom\"; def dom = e.domain != null ? e.domain : \"none\"; def res = e.result != null ? e.result : (e.results != null ? e.results : \"none\"); sp.add(sc + \" / \" + dom + \" / \" + res); } } ctx._source.spf_results_combined = sp;"
+  }
+}'
+```
+
+`wait_for_completion=false` returns a task ID — check progress with
+`GET _tasks/<task id>`. Adjust the index pattern if you use a custom
+`index_prefix`/`index_suffix`; with `index_prefix_domain_map`, run the
+command once per tenant prefix (`acme_corp_dmarc_aggregate*`) plus once for
+the unprefixed pattern, or widen it to `*dmarc_aggregate*` to cover every
+tenant in one pass. After backfilling, re-import the updated
+dashboards ndjson (the index pattern saved object changed too) per the
+import instructions above.
+
+SMTP TLS documents have the same class of defect one level deeper:
+`policies` is an object array, and each policy's `failure_details` is an
+object array inside it. SMTP TLS documents now also carry
+`policies_combined` and `failure_details_combined`, backfilled
+automatically at startup the same way, and the equivalent manual command
+is:
+
+```bash
+curl -X POST "http://localhost:9200/smtp_tls*/_update_by_query?conflicts=proceed&wait_for_completion=false" \
+  -H "Content-Type: application/json" -d '
+{
+  "query": {
+    "bool": {
+      "minimum_should_match": 1,
+      "should": [
+        {
+          "bool": {
+            "must": [
+              {
+                "bool": {
+                  "minimum_should_match": 1,
+                  "should": [
+                    {"exists": {"field": "policies.policy_domain"}},
+                    {"exists": {"field": "policies.policy_type"}}
+                  ]
+                }
+              }
+            ],
+            "must_not": [{"exists": {"field": "policies_combined"}}]
+          }
+        },
+        {
+          "bool": {
+            "must": [
+              {
+                "bool": {
+                  "minimum_should_match": 1,
+                  "should": [
+                    {"exists": {"field": "policies.failure_details.result_type"}},
+                    {"exists": {"field": "policies.failure_details.sending_mta_ip"}}
+                  ]
+                }
+              }
+            ],
+            "must_not": [{"exists": {"field": "failure_details_combined"}}]
+          }
+        }
+      ]
+    }
+  },
+  "script": {
+    "lang": "painless",
+    "source": "List pols = new ArrayList(); List dets = new ArrayList(); def ps = ctx._source.policies; if (ps != null) { if (!(ps instanceof List)) { ps = [ps]; } for (p in ps) { if (p == null) { continue; } def dom = p.policy_domain != null ? p.policy_domain : \"none\"; def typ = p.policy_type != null ? p.policy_type : \"none\"; pols.add(dom + \" / \" + typ); def fds = p.failure_details; if (fds != null) { if (!(fds instanceof List)) { fds = [fds]; } for (f in fds) { if (f == null) { continue; } def rt = f.result_type != null ? f.result_type : \"none\"; def smi = f.sending_mta_ip != null ? f.sending_mta_ip : \"none\"; def ri = f.receiving_ip != null ? f.receiving_ip : \"none\"; def rmh = f.receiving_mx_hostname != null ? f.receiving_mx_hostname : \"none\"; dets.add(dom + \" / \" + typ + \" / \" + rt + \" / \" + smi + \" / \" + ri + \" / \" + rmh); } } } } ctx._source.policies_combined = pols; ctx._source.failure_details_combined = dets;"
+  }
+}'
+```
+
+It works identically on OpenSearch; just adjust the URL and credentials, same
+as the aggregate command above.
+
 ## Records retention
 
 Starting in version 5.0.0, `parsedmarc` stores data in a separate

docs/source/kibana.md

@@ -38,24 +38,35 @@ valid when a message is forwarded without changing the from address, which is
 often caused by a mailbox forwarding rule. This is because DKIM signatures are
 part of the message headers, whereas SPF relies on SMTP session headers.
 
-Underneath the pie charts. you can see graphs of DMARC passage and message
+Underneath the pie charts, you can see graphs of DMARC compliance and message
 disposition over time.
 
 Under the graphs you will find the most useful data tables on the dashboard. On
 the left, there is a list of organizations that are sending you DMARC reports.
 In the center, there is a list of sending servers grouped by the base domain
-in their reverse DNS. On the right, there is a list of email from domains,
-sorted by message volume.
+in their reverse DNS. On the right, there is the "Message volume and DMARC
+compliance by from domain" table, which lists email from domains with their
+message volume and a percentage of those messages that passed DMARC.
 
 By hovering your mouse over a data table value and using the magnifying glass
-icons, you can filter on our filter out different values. Start by looking at
+icons, you can filter on or filter out different values. Start by looking at
 the Message Sources by Reverse DNS table. Find a sender that you recognize,
 such as an email marketing service, hover over it, and click on the plus (+)
 magnifying glass icon, to add a filter that only shows results for that sender.
-Now, look at the Message From Header table to the right. That shows you the
-domains that a sender is sending as, which might tell you which brand/business
-is using a particular service. With that information, you can contact them and
-have them set up DKIM.
+Now, look at the Message volume and DMARC compliance by from domain table to
+the right. That shows you the domains that a sender is sending as, and what
+share of that traffic is passing DMARC, which might tell you which
+brand/business is using a particular service. With that information, you can
+contact them and have them set up DKIM.
+
+:::{note}
+The "Message volume and DMARC compliance by from domain" table is a TSVB
+visualization, used because per-domain compliance percentages require a
+Filter Ratio metric that agg-based data tables can't compute. It renders
+correctly on Kibana 8.x as imported, but *editing* it requires first enabling
+the `metrics:allowStringIndices` advanced setting, since it references the
+`dmarc_aggregate*` index as a string pattern, which Elastic has deprecated.
+:::
 
 :::{note}
 If you have a lot of B2C customers, you may see a high volume of emails as
@@ -71,7 +82,23 @@ Further down the dashboard, you can filter by source country or source IP
 address.
 
 Tables showing SPF and DKIM alignment details are located under the IP address
-table.
+table. Each row of the DKIM details table is one real DKIM signature, shown
+as a combined `selector / domain / result` value; the SPF details table
+shows `scope / domain / result` the same way. Combining the values into one
+column keeps each signature's selector, domain, and result paired together,
+rather than aggregating them as separate columns. Because a message that
+carries multiple DKIM signatures appears once per signature, summing the
+messages column across rows can exceed the total number of messages.
+
+The "Auth result filters" panel above the details tables
+provides dropdowns for the individual auth-result components — DKIM
+selector, DKIM domain, DKIM result, SPF scope, SPF domain, and SPF
+result — and filters the whole dashboard by them. Because components from
+different signatures of the same message are indexed together, combining
+two of these component filters matches documents where any signature
+satisfies each condition individually, not necessarily the same signature;
+the combined `selector / domain / result` (`scope / domain / result`)
+column remains the per-signature source of truth.
 
 :::{note}
 The alignment tables (SPF details, DKIM details) and the per-IP source
@@ -101,3 +128,15 @@ reporting organizations, the policy domains they report on, and the
 specific failure types — certificate expiry, STARTTLS not supported,
 STS policy fetch errors, validation failures, and similar — together with
 the sending and receiving MTA addresses involved.
+
+Like the DKIM and SPF details tables above, the "SMTP TLS domains" and
+"SMTP TLS failure details" tables show one row per policy and one row per
+failure detail, respectively, using combined `policy (domain / type)` and
+`failure detail (domain / type / result / sending mta / receiving ip / mx)`
+columns so that each policy's or failure detail's fields stay paired
+together, rather than aggregating them as separate columns. The
+`successful_sessions` and `failed_sessions` columns are summed per report
+document, though, not per policy: when a single report carries multiple
+policies, a row's session sums include the sibling policies from that
+report as well as its own. Fully attributing session counts to a single
+policy would require restructuring the stored documents.

docs/source/splunk.md

@@ -18,5 +18,5 @@ The Splunk dashboards display the same content and layout as the
 Kibana dashboards, although the Kibana dashboards have slightly
 easier and more flexible filtering options.
 
-[xml files]: https://github.com/domainaware/parsedmarc/tree/master/splunk
+[xml files]: https://github.com/domainaware/parsedmarc/tree/master/dashboards/splunk
 [http event collector (hec)]: http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC

docs/source/usage.md

@@ -3,23 +3,25 @@
 ## CLI help
 
 ```text
-usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads] [-o OUTPUT]
+usage: parsedmarc [-h] [-c CONFIG_FILE] [-r] [--strip-attachment-payloads] [-o OUTPUT]
                   [--aggregate-json-filename AGGREGATE_JSON_FILENAME] [--failure-json-filename FAILURE_JSON_FILENAME]
                   [--smtp-tls-json-filename SMTP_TLS_JSON_FILENAME] [--aggregate-csv-filename AGGREGATE_CSV_FILENAME]
                   [--failure-csv-filename FAILURE_CSV_FILENAME] [--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME]
-                  [-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--offline] [-s] [-w] [--verbose] [--debug]
-                  [--log-file LOG_FILE] [--no-prettify-json] [-v]
+                  [-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--dns-retries DNS_RETRIES] [--offline] [-s]
+                  [-w] [--verbose] [--debug] [--log-file LOG_FILE] [--no-prettify-json] [-v]
                   [file_path ...]
 
 Parses DMARC reports
 
 positional arguments:
-  file_path             one or more paths to aggregate or failure report files, emails, or mbox files'
+  file_path             one or more paths to aggregate or failure report files, emails, mbox files, or directories
+                        containing them
 
 options:
   -h, --help            show this help message and exit
   -c CONFIG_FILE, --config-file CONFIG_FILE
                         a path to a configuration file (--silent implied)
+  -r, --recursive       search directories given as file_path recursively, and enable '**' recursion in glob patterns
   --strip-attachment-payloads
                         remove attachment payloads from failure report output
   -o OUTPUT, --output OUTPUT
@@ -40,6 +42,8 @@ options:
                         nameservers to query
   -t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
                         number of seconds to wait for an answer from DNS (default: 2.0)
+  --dns-retries DNS_RETRIES
+                        number of times to retry DNS queries on timeout or other transient errors (default: 0)
   --offline             do not make online queries for geolocation or DNS
   -s, --silent          only print errors
   -w, --warnings        print warnings in addition to errors
@@ -123,11 +127,30 @@ The full set of configuration options are:
       Elasticsearch, Splunk and/or S3
   - `save_smtp_tls` - bool: Save SMTP-STS report data to
       Elasticsearch, Splunk and/or S3
-  - `index_prefix_domain_map` -  bool: A path mapping of Opensearch/Elasticsearch index prefixes to domain names
+  - `index_prefix_domain_map` - str: Path to a YAML file mapping
+      OpenSearch/Elasticsearch index prefixes to domain names
   - `strip_attachment_payloads` - bool: Remove attachment
       payloads from results
   - `silent` - bool: Set this to `False` to output results to STDOUT
   - `output` - str: Directory to place JSON and CSV files in.  This is required if you set either of the JSON output file options.
+  - `archive_directory` - str: Optional. When set, successfully
+      processed report files given as local file/directory path
+      arguments are moved into
+      `<archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/`
+      (year and month come from the report's own begin/arrival date, with
+      the month zero-padded). A successfully parsed report whose archive
+      date can't be determined is left in place with a logged warning.
+      Files that fail to parse as a report are moved to
+      `<archive_directory>/Invalid/`; files that fail for other reasons,
+      such as transient I/O errors, are left in place so a later run can
+      retry them. An existing destination file is never overwritten; a
+      numeric suffix is appended before the extension (e.g.
+      `report-1.xml`). This applies only to direct local file input —
+      reports fetched from mailboxes (IMAP, Microsoft Graph, Gmail API,
+      Maildir) use `[mailbox] archive_folder` instead, and mbox files are
+      never moved. Files already inside `archive_directory` are excluded
+      from processing, so the archive may safely live inside an input
+      directory. A failed move is logged and does not stop the run.
   - `aggregate_json_filename` - str: filename for the aggregate
       JSON output file
   - `failure_json_filename` - str: filename for the failure
@@ -164,13 +187,32 @@ The full set of configuration options are:
   - `fail_on_output_error` - bool: Exit with a non-zero status code if
       any configured output destination fails while saving/publishing
       reports (Default: `False`)
+
+    :::{note}
+    This option only controls the process exit code. Retaining mailbox
+    messages whose reports could not be saved is automatic and happens
+    either way — see
+    [Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved).
+    :::
+
   - `log_file` - str: Write log messages to a file at this path
-  - `n_procs` - int: Number of process to run in parallel when
-      parsing in CLI mode (Default: `1`)
+  - `n_procs` - int: Number of processes to run in parallel when
+      parsing report files passed directly as CLI arguments, messages
+      in mbox files, and messages from mailbox connections (IMAP,
+      Microsoft Graph, Gmail API, Maildir), including watch mode
+      (Default: `1`)
 
     :::{note}
     Setting this to a number larger than one can improve
-    performance when processing thousands of files
+    performance when processing thousands of files or messages
+    :::
+
+    :::{note}
+    Only parsing is parallelized across worker processes. Fetching
+    messages, deduplicating reports, archiving/deleting mailbox
+    messages, and saving/publishing to outputs all stay sequential
+    in the main process. Each worker process keeps its own DNS/GeoIP
+    cache.
     :::
 
 - `mailbox`
@@ -183,12 +225,46 @@ The full set of configuration options are:
       messages as they arrive or poll MS Graph for new messages
   - `delete` - bool: Delete messages after processing them,
       instead of archiving them
+  - `delete_aggregate` - bool: Delete aggregate report messages
+      after processing them, instead of archiving them
+      (Default: the value of `delete`)
+  - `delete_failure` - bool: Delete failure report messages
+      after processing them, instead of archiving them
+      (Default: the value of `delete`)
+  - `delete_smtp_tls` - bool: Delete SMTP TLS report messages
+      after processing them, instead of archiving them
+      (Default: the value of `delete`)
+  - `delete_invalid` - bool: Delete messages that could not be
+      parsed, instead of archiving them in the `Invalid`
+      subfolder, where they can be inspected for debugging
+      (Default: the value of `delete`)
+
+    :::{note}
+    Each of these four options overrides `delete` for one kind of
+    message only, and the other three keep inheriting `delete`. So
+    `delete = True` combined with `delete_failure = False` archives
+    failure report messages while deleting processed aggregate and
+    SMTP TLS report messages — and unparseable ones, unless
+    `delete_invalid = False` is set as well.
+    :::
+
   - `test` - bool: Do not move or delete messages
   - `batch_size` - int: Number of messages to read and process
       before saving. Default `10`. Use `0` for no limit.
-  - `check_timeout` - int: Number of seconds to wait for a IMAP
+  - `check_timeout` - int: Number of seconds to wait for an IMAP
       IDLE response or the number of seconds until the next
       mail check (Default: `30`)
+  - `max_unsaved_retries` - int: How many times a batch of messages
+      whose reports could not be saved is retried before its messages
+      are moved to the `Unsaved` archive subfolder instead of being
+      retried again (Default: `2`, i.e. the initial attempt plus two
+      retries). Use `0` to move messages on the first failed save;
+      negative values are rejected.
+      Failures are counted in memory, so the cap applies across watch-mode
+      checks within one long-running process, not across separate one-shot
+      runs. See
+      [Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
+      below.
   - `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
       Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
       Defaults to `1d` if incorrect value is provided.
@@ -249,12 +325,44 @@ The full set of configuration options are:
       could be a shared mailbox if the user has access to the mailbox
   - `graph_url` - str: Microsoft Graph URL.  Allows for use of National Clouds (ex Azure Gov)
       (Default: https://graph.microsoft.com)
+
+    :::{warning}
+    Setting `graph_url` alone is **not** sufficient for a national/sovereign
+    cloud tenant. It only changes the Microsoft Graph API root; the
+    Microsoft Entra ID (Azure AD) token endpoint used for authentication is
+    not currently configurable in parsedmarc or `mailsuite`, and always
+    defaults to the global `https://login.microsoftonline.com`. A true
+    national-cloud deployment also needs its own Entra ID endpoint, so
+    `graph_url` by itself only helps if your tenant is registered in the
+    global cloud but you specifically need to reach one of these Graph API
+    roots (per [Microsoft's national cloud deployment docs][ms-graph-clouds]):
+
+    | National cloud | Microsoft Graph URL | Entra ID endpoint (not configurable here) |
+    |---|---|---|
+    | Global (default) | `https://graph.microsoft.com` | `https://login.microsoftonline.com` |
+    | US Government L4 (GCC High) | `https://graph.microsoft.us` | `https://login.microsoftonline.us` |
+    | US Government L5 (DoD) | `https://dod-graph.microsoft.us` | `https://login.microsoftonline.us` |
+    | China, operated by 21Vianet | `https://microsoftgraph.chinacloudapi.cn` | `https://login.chinacloudapi.cn` |
+
+    [ms-graph-clouds]: https://learn.microsoft.com/en-us/graph/deployments
+    :::
   - `token_file` - str: Path to save the token file
       (Default: `.token`)
   - `allow_unencrypted_storage` - bool: Allows the Azure Identity
       module to fall back to unencrypted token cache (Default: `False`).
       Even if enabled, the cache will always try encrypted storage first.
 
+    :::{note}
+    `token_file` stores the serialized authentication record; the
+    underlying MSAL persistent token cache is separately named
+    `parsedmarc`, not `mailsuite`'s own default cache name. This is
+    deliberate: the Graph mailbox backend used to live directly in
+    parsedmarc, and moved into the `mailsuite` dependency in parsedmarc
+    9.11.0. Explicitly keeping the `parsedmarc` cache name means tokens
+    cached before that move keep working after upgrading — there is
+    nothing to migrate and no action needed on your part.
+    :::
+
     :::{note}
     You must create an app registration in Azure AD and have an
     admin grant the Microsoft Graph `Mail.ReadWrite`
@@ -263,6 +371,11 @@ The full set of configuration options are:
     username, you must grant the app `Mail.ReadWrite.Shared`.
     :::
 
+    | Auth method | Reading (own mailbox) | Reading (shared mailbox) |
+    |---|---|---|
+    | `UsernamePassword` / `DeviceCode` (delegated) | `Mail.ReadWrite` | `Mail.ReadWrite.Shared` |
+    | `ClientSecret` / `Certificate` / `ClientAssertion` (application) | `Mail.ReadWrite` (application), scoped via `New-ApplicationAccessPolicy` | same as own mailbox — app-only access is scoped by policy, not by permission name |
+
     :::{tip}
     **Troubleshooting connections.** Run with `--verbose` to log a
     redacted connection summary (auth method, tenant, client ID,
@@ -273,6 +386,11 @@ The full set of configuration options are:
     an Exchange Online-side one), Microsoft Graph SDK requests, and
     `httpx` HTTP request lines. Secret values (passwords, client
     secrets, certificate passwords) are never written to logs.
+    Connection, mailbox fetch, message send, and `--watch` failures
+    each log a single ERROR line naming the mailbox, tenant, auth
+    method, and the Graph `request-id`/`client-request-id` when
+    available — worth quoting verbatim when contacting Microsoft
+    support.
     :::
 
     :::{warning}
@@ -295,6 +413,120 @@ The full set of configuration options are:
     applies to the `Certificate` and `ClientAssertion` auth methods.
 
     :::
+
+    **Sending the summary email via Microsoft Graph.**
+    When `[msgraph]` is configured and `[smtp]` has a `to` value but no
+    `host`, the periodic summary email is sent through the same
+    already-authenticated Graph mailbox connection used for reading
+    (`/users/{mailbox}/sendMail`), and a copy is saved to Sent Items.
+    When `[smtp] host` is set, SMTP is used regardless of whether
+    `[msgraph]` is also configured — SMTP is always preferred, with no
+    automatic fallback to Graph on SMTP failure.
+
+    Example config combining `[msgraph]` with a `[smtp]` section that
+    only sets `to`/`subject` (no `host`):
+
+    ```ini
+    [msgraph]
+    auth_method = Certificate
+    client_id = ...
+    tenant_id = ...
+    mailbox = dmarc-reports@example.com
+    certificate_path = /path/to/cert.pem
+
+    [smtp]
+    to = admin@example.com
+    subject = DMARC Summary
+    ```
+
+    Required Microsoft Graph permissions, in addition to the
+    reading-related permissions documented above:
+
+    | Auth method | Reading | Sending (own mailbox) | Sending (shared mailbox) |
+    |---|---|---|---|
+    | `UsernamePassword` / `DeviceCode` (delegated) | `Mail.ReadWrite` (+`.Shared` for shared) | `Mail.Send` | `Mail.Send.Shared` |
+    | `ClientSecret` / `Certificate` / `ClientAssertion` (application) | `Mail.ReadWrite` (application) | `Mail.Send` (application) | `Mail.Send` (application), scoped via `New-ApplicationAccessPolicy` |
+
+    :::{warning}
+    Graph-based sending is only confirmed to work with the app-only
+    auth methods (`ClientSecret`, `Certificate`, `ClientAssertion`).
+    The delegated auth methods (`UsernamePassword`, `DeviceCode`)
+    currently request only the `Mail.ReadWrite`(`.Shared`) scope when
+    authenticating, not `Mail.Send` — so a delegated connection's
+    access token will not carry `Mail.Send` even if an administrator
+    has granted it, and `/sendMail` calls are expected to fail with an
+    access-denied error regardless of what's granted in Azure AD. Use
+    an app-only auth method if you need Graph-based sending.
+    :::
+
+    **Minimal example configs.** Each auth method needs a different
+    minimum set of keys. These read-only examples omit `[smtp]`; see
+    above for adding Graph-based sending on top of any of them.
+
+    `UsernamePassword` (delegated, own mailbox):
+    ```ini
+    [msgraph]
+    auth_method = UsernamePassword
+    client_id = ...
+    client_secret = ...
+    user = dmarc-reports@example.com
+    password = ...
+    ```
+
+    `DeviceCode` (delegated, interactive sign-in on first run — `user`
+    is the account that signs in; `mailbox` is the shared mailbox it
+    reads, and only needs to differ from `user` to request
+    `Mail.ReadWrite.Shared` instead of plain `Mail.ReadWrite`):
+    ```ini
+    [msgraph]
+    auth_method = DeviceCode
+    client_id = ...
+    tenant_id = ...
+    user = signing-in-user@example.com
+    mailbox = dmarc-reports@example.com
+    ```
+
+    `ClientSecret` (app-only):
+    ```ini
+    [msgraph]
+    auth_method = ClientSecret
+    client_id = ...
+    tenant_id = ...
+    client_secret = ...
+    mailbox = dmarc-reports@example.com
+    ```
+
+    `Certificate` (app-only):
+    ```ini
+    [msgraph]
+    auth_method = Certificate
+    client_id = ...
+    tenant_id = ...
+    certificate_path = /path/to/cert.pem
+    mailbox = dmarc-reports@example.com
+    ```
+
+    `ClientAssertion` (app-only, short-lived JWT — see the note above
+    about its unsuitability for `watch` mode):
+    ```ini
+    [msgraph]
+    auth_method = ClientAssertion
+    client_id = ...
+    tenant_id = ...
+    client_assertion = ...
+    mailbox = dmarc-reports@example.com
+    ```
+
+    :::{tip}
+    **Troubleshooting.**
+
+    | Error | Cause | Fix |
+    |---|---|---|
+    | *"...needs permission to access resources in your organization that only an admin can grant"* / "Admin consent required" | A delegated auth method (`UsernamePassword`, `DeviceCode`) is authenticating with a scope (`Mail.ReadWrite` or `Mail.ReadWrite.Shared`) the tenant admin hasn't consented to yet. | Have an Entra ID admin grant consent: Azure Portal → **Enterprise Applications** → *your app* → **Permissions** → **Grant admin consent**, or `az ad app permission admin-consent --id <CLIENT_ID>`. This is separate from the `New-ApplicationAccessPolicy` step above, which only applies to app-only auth. |
+    | `ErrorItemNotFound: ... Default folder Root not found` | `mailsuite` can resolve the well-known folders (`Inbox`, `Archive`, `Drafts`, `Sent Items`, `Deleted Items`, `Junk Email`) even when a mailbox's folder hierarchy hasn't fully provisioned, but a **custom, non-well-known** `reports_folder` name still fails to resolve on such a mailbox. | Point `reports_folder` at (or under) one of the six well-known folder names above, or sign into the (shared) mailbox once via Outlook/OWA to force Exchange to provision it, then retry. |
+    | `RuntimeError: Event loop is closed` | Historical bug, fixed in `mailsuite` 2.0.2. Not reachable with the `mailsuite>=2.2.2` this project requires. | Confirm your installed `mailsuite` version is current (`pip show mailsuite`); upgrade if it's somehow pinned below 2.0.2. |
+    | Invalid/rejected timestamp in the `since`/`receivedDateTime` filter | Historical bug (parsedmarc [#706](https://github.com/domainaware/parsedmarc/pull/706)/[#708](https://github.com/domainaware/parsedmarc/pull/708)): older versions appended a spurious `Z` to an already-UTC-offset ISO timestamp. Fixed since parsedmarc 9.5.1/9.5.5. | Upgrade parsedmarc if you're on a version older than 9.5.5. |
+    :::
 - `elasticsearch`
   - `hosts` - str: A comma separated list of hostnames and ports
       or URLs (e.g. `127.0.0.1:9200` or
@@ -371,20 +603,33 @@ The full set of configuration options are:
   - `aggregate_topic` - str: The Kafka topic for aggregate reports
   - `failure_topic` - str: The Kafka topic for failure reports
 - `smtp`
-  - `host` - str: The SMTP hostname
+
+  The results email is only sent when at least one aggregate, failure,
+  or SMTP TLS report was parsed during the run; an empty run (e.g. an
+  empty inbox) skips the email instead of sending headers-only CSVs.
+
+  - `host` - str: The SMTP hostname. Required unless `[msgraph]` is
+    configured, in which case omitting it sends the summary via
+    Microsoft Graph instead — see "Sending the summary email via
+    Microsoft Graph" above.
   - `port` - int: The SMTP port (Default: `25`)
   - `ssl` - bool: Require SSL/TLS instead of using STARTTLS
   - `skip_certificate_verification` - bool: Skip certificate
     verification (not recommended)
-  - `user` - str: the SMTP username
-  - `password` - str: the SMTP password
-  - `from` - str: The From header to use in the email
+  - `user` - str: the SMTP username. SMTP-only; not used when sending
+    via Microsoft Graph.
+  - `password` - str: the SMTP password. SMTP-only; not used when
+    sending via Microsoft Graph.
+  - `from` - str: The From header to use in the email. SMTP-only.
+    When sent via Microsoft Graph, the message's `From` is always the
+    `[msgraph]` mailbox — `[smtp] from` has no effect.
   - `to` - list: A list of email addresses to send to
   - `subject` - str: The Subject header to use in the email
     (Default: `parsedmarc report`)
   - `attachment` - str: The ZIP attachment filenames
+    (Default: `DMARC-<YYYY-MM-DD>.zip`)
   - `message` - str: The email message
-    (Default: `Please see the attached parsedmarc report.`)
+    (Default: `Please see the attached DMARC results.`)
 
     :::{note}
     `%` characters must be escaped with another `%` character,
@@ -653,6 +898,78 @@ PUT _cluster/settings
 Increasing this value increases resource usage.
 :::
 
+### Mailbox messages are only archived once the reports are saved
+
+parsedmarc processes a mailbox in batches of `batch_size` messages. Each
+batch is written to every configured output destination *before* any of
+that batch's messages are archived or deleted. If any destination reports a
+failure — an Elasticsearch outage, an expired Splunk HEC token, an
+unreachable Kafka broker, a full `--output` disk — the whole batch is left
+in the reports folder and retried on the next run or watch-mode check, so a
+report is never removed from the mailbox while it exists nowhere else
+([issue #242](https://github.com/domainaware/parsedmarc/issues/242)).
+
+This is all-or-nothing per batch: archiving a batch because most
+destinations accepted it would still permanently lose the data for the one
+that didn't. It also applies regardless of `fail_on_output_error`, which
+only controls the process exit code. Messages that could not be parsed at
+all carry no report data, so they are filed under `Invalid` (or deleted per
+`delete_invalid`) as usual.
+
+A destination that is broken rather than briefly unavailable would
+otherwise be retried forever, so retries are capped. Once a message's batch
+has failed `max_unsaved_retries + 1` times (three times by default), that
+message is moved to `<archive_folder>/Unsaved` and stops being retried. **A
+message is never deleted on this path, whatever the `delete` options say.**
+To recover after fixing the output destination, either move the messages
+from `Archive/Unsaved` back into the reports folder, or run parsedmarc once
+with `reports_folder = Archive/Unsaved`.
+
+:::{note}
+The failure counts live in memory, so they are counted per parsedmarc
+process. In watch mode — a long-running process that checks the mailbox
+repeatedly — the cap works as described across checks. A one-shot run
+(`cron`, `systemd` timers) attempts each message exactly once and then
+exits, so its counts start over every time and the default cap is never
+reached: messages simply keep being retried on every run, which is the
+safe direction. Set `max_unsaved_retries = 0` if you want one-shot runs to
+move unsavable messages to `Unsaved` immediately instead.
+:::
+
+:::{warning}
+Retrying a batch means re-sending it. Output destinations that
+deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize
+an already-saved report — are unaffected, and S3 is idempotent because
+each report is written to an object key built from its type, date, and
+report ID, so a retry overwrites the same object. Kafka,
+Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the
+`--output` JSON/CSV files all append unconditionally, so each retry adds
+another copy of every report in the batch. That is why the default retry
+cap is deliberately low: at most three deliveries per report before its
+message is set aside in `Unsaved`. The summary email covers everything
+parsed in a run, including reports whose batch failed to save, so a report
+retried across runs can also appear in more than one summary email.
+:::
+
+:::{note}
+Not every destination can report a failed delivery. The webhook output
+deliberately logs and swallows its own HTTP and network errors, and the
+syslog and GELF outputs send through Python logging handlers, which
+swallow delivery errors by design — so an unreachable webhook, syslog, or
+GELF endpoint is *not* treated as a failed save and does not hold a
+batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
+Kafka, S3, PostgreSQL, Azure Log Analytics, and the `--output` files are
+all detected and do.
+:::
+
+:::{note}
+`since` interacts with retries: a message that ages out of the configured
+`since` window stops being fetched, and therefore stops being retried
+automatically. It is never deleted or moved — it simply stays in the
+reports folder until it is processed by a run with a wider (or no)
+`since` window.
+:::
+
 ## Environment variable configuration
 
 Any configuration option can be set via environment variables using the
@@ -789,6 +1106,50 @@ For sections with underscores in the name, the full section name is used:
 | `webhook` | `PARSEDMARC_WEBHOOK_` |
 | `gsecops` | `PARSEDMARC_GSECOPS_` |
 
+## Using parsedmarc as a library
+
+`parsedmarc` is also importable as a regular Python package, not just a CLI
+tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`,
+`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`,
+`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and
+`watch_inbox()` — are all importable
+directly from the `parsedmarc` package. See the [API reference](api.md) for
+the full set of modules and members.
+
+Each of these functions accepts either individual option keyword arguments
+(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword
+argument carrying a `ParserConfig` instance:
+
+```python
+from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
+from parsedmarc.mail import IMAPConnection
+
+config = ParserConfig(
+    offline=False,
+    nameservers=["1.1.1.1", "1.0.0.1"],
+    dns_timeout=5.0,
+)
+
+report = parse_report_file("aggregate_report.xml.gz", config=config)
+
+connection = IMAPConnection(
+    host="imap.example.com", user="dmarc@example.com", password="..."
+)
+results = get_dmarc_reports_from_mailbox(connection, config=config)
+```
+
+A few things to keep in mind:
+
+- When `config=` is passed, the individual option keyword arguments are
+  ignored in favor of the values carried on the `ParserConfig` instance.
+- Each explicitly constructed `ParserConfig` owns its own isolated caches
+  (IP address info, seen aggregate report IDs, and the reverse DNS map).
+  Omitting `config=` falls back to the process-wide caches shared by every
+  call that doesn't pass one.
+- `keep_alive` and `n_procs` are not part of `ParserConfig` — they control
+  process/worker orchestration rather than parsing or enrichment behavior,
+  so they are always passed as separate keyword arguments.
+
 ## Performance tuning
 
 For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
@@ -799,8 +1160,12 @@ imports more predictable:
 - Reduce `mailbox.batch_size` to smaller values such as `100-500` instead of
   processing a very large message set at once. Smaller batches trade throughput
   for lower peak memory use and less sink pressure.
-- Keep `n_procs` low for mailbox-heavy runs. In practice, `1-2` workers is often
-  a safer starting point for large backfills than aggressive parallelism.
+- `n_procs` now parallelizes parsing for mailbox and mbox runs too, not just
+  report files passed directly as CLI arguments. It pays off most when a run
+  is bound by DNS/GeoIP enrichment rather than fetching or output. The
+  trade-off is memory and DNS load: at most roughly `2 * n_procs` messages
+  are held in flight at once, each worker process keeps its own DNS/GeoIP
+  cache, and DNS query volume can multiply by up to `n_procs`.
 - Use `mailbox.since` to process reports in smaller time windows such as `1d`,
   `7d`, or another interval that fits the backlog. This makes it easier to catch
   up incrementally instead of loading an entire mailbox history in one run.
@@ -832,12 +1197,16 @@ whalensolutions:
 
 Save it to disk where the user running ParseDMARC can read it, then set `index_prefix_domain_map` to that filepath in the `[general]` section of the ParseDMARC configuration file and do not set an `index_prefix` option in the `[elasticsearch]` or `[opensearch]` sections.
 
-When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of Opensearch or the ELK stack to only grant users access to the indexes that they need.
+When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
 
  :::{note}
  A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
 :::
 
+Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
+
+The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See [Backfilling the combined DKIM/SPF result fields](elasticsearch.md#backfilling-the-combined-dkimspf-result-fields) for the details.
+
 ## Running parsedmarc as a systemd service
 
 Use systemd to run `parsedmarc` as a service and process reports as