Fix mbox-run progress display and document n_procs scope (#147) (#846)

* Fix mbox-run progress display and document n_procs scope (#147)

An mbox-only run showed a misleading, permanently-stuck `0it` progress
bar: the CLI's tqdm bar only tracks report files passed directly as
arguments (mbox paths are split out first), and per-message mbox
progress is only logged at INFO, which --silent / config-file runs
hide. The empty bar is no longer created when there are no direct file
arguments, and get_dmarc_reports_from_mbox() now wraps its message loop
in a tqdm bar that auto-disables on non-TTY output (tqdm's
disable=None), so interactive mbox imports show real per-message
progress while tests, cron jobs, and piped runs stay clean.

Also documents that n_procs parallel parsing applies only to report
files passed directly on the command line — messages from mbox files
and mailbox connections (IMAP, Microsoft Graph, Gmail API, Maildir)
are always processed sequentially — and fixes the pre-existing
"Number of process" typo in that entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Gate the CLI progress bar on stderr, where tqdm renders it

Addresses Copilot review feedback on #846: tqdm writes to stderr by
default, but the guard checked sys.stdout.isatty(), so a run with
stderr redirected to a log file would write bar escapes into the log,
and a run with stdout redirected (e.g. piping the JSON output) hid the
bar even though stderr could display it. This also matches the mbox
bar's disable=None auto-check, which keys off tqdm's own output stream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -18,6 +18,7 @@
 - **SMTP TLS visualizations had the same cross-product defect as the DKIM/SPF alignment tables** ([#169](https://github.com/domainaware/parsedmarc/issues/169)): `policies` is an object array, and each policy's `failure_details` is itself an object array inside it, so stacking terms aggregations on their subfields cross-products the same way. SMTP TLS documents now also carry `policies_combined` — one `"policy_domain / policy_type"` string per policy — and `failure_details_combined` — one `"policy_domain / policy_type / result_type / sending_mta_ip / receiving_ip / receiving_mx_hostname"` string per failure detail — and documents saved by older versions are backfilled automatically at startup the same non-blocking, idempotent way as the DKIM/SPF backfill; the equivalent manual `_update_by_query` command is documented on the Elasticsearch docs page. Also fixed two adjacent dead fields found while making this change: `_SMTPTLSFailureDetailsDoc` declared `additional_information_uri`, but `add_failure_details` passed it to the constructor as `additional_information`, so it was never actually populated on the declared field; and `receiving_mx_hostname`, which `add_failure_details` has always stored, had no field declaration at all. Both are now correctly wired in the Elasticsearch and OpenSearch outputs.
 - **The Grafana "Map of Message Source Countries" panel's markers now scale with message volume and use a higher-contrast style.** Previously the markers were fixed-size 5 px dots at 50% opacity in dark green, which were nearly invisible on the dark basemap.
 - **Corrected the dead `_SPFResult.results` (plural) field declaration to `result`**, matching what was always written to it.
+- **An mbox-only run no longer shows a misleading, permanently-stuck `0it` progress bar, and mbox parsing now shows a real per-message progress bar on interactive terminals** ([#147](https://github.com/domainaware/parsedmarc/issues/147)). The CLI's progress bar only tracks report files passed directly as arguments — `n_procs` parallel parsing also only applies to those — so runs whose only input was an mbox file displayed an empty bar while messages were parsed sequentially with no visible progress (per-message progress is logged at INFO, hidden in `--silent`/config-file runs). The empty bar is no longer created, `get_dmarc_reports_from_mbox()` now wraps its message loop in a tqdm bar that auto-disables on non-TTY output, and the `n_procs` documentation now states the direct-file-arguments-only scope explicitly.
 
 ## 10.2.4
 

docs/source/usage.md

@@ -169,14 +169,22 @@ The full set of configuration options are:
       any configured output destination fails while saving/publishing
       reports (Default: `False`)
   - `log_file` - str: Write log messages to a file at this path
-  - `n_procs` - int: Number of process to run in parallel when
-      parsing in CLI mode (Default: `1`)
+  - `n_procs` - int: Number of processes to run in parallel when
+      parsing report files passed directly as CLI arguments
+      (Default: `1`)
 
     :::{note}
     Setting this to a number larger than one can improve
     performance when processing thousands of files
     :::
 
+    :::{note}
+    `n_procs` only applies to report files passed directly on the
+    command line. Messages from mbox files and from mailbox
+    connections (IMAP, Microsoft Graph, Gmail API, Maildir) are
+    always processed sequentially.
+    :::
+
 - `mailbox`
   - `reports_folder` - str: The mailbox folder (or label for
       Gmail) where the incoming reports can be found

parsedmarc/__init__.py

@@ -34,6 +34,7 @@ import mailparser
 import xmltodict
 from expiringdict import ExpiringDict
 from mailsuite.smtp import send_email
+from tqdm import tqdm
 
 from parsedmarc.constants import (
     DEFAULT_DNS_MAX_RETRIES,
@@ -2126,7 +2127,7 @@ def get_dmarc_reports_from_mbox(
         message_keys = mbox.keys()
         total_messages = len(message_keys)
         logger.debug("Found {0} messages in {1}".format(total_messages, input_))
-        for i in range(len(message_keys)):
+        for i in tqdm(range(total_messages), disable=None):
             message_key = message_keys[i]
             logger.info("Processing message {0} of {1}".format(i + 1, total_messages))
             msg_content = mbox.get_string(message_key)

parsedmarc/cli.py

@@ -2376,7 +2376,7 @@ def _main():
     results = []
 
     pbar = None
-    if sys.stdout.isatty():
+    if sys.stderr.isatty() and len(file_paths) > 0:
         pbar = tqdm(total=len(file_paths))
 
     n_procs = int(opts.n_procs or 1)