diff --git a/CHANGELOG.md b/CHANGELOG.md index b4774fa7..4e42e2b3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ - **SMTP TLS visualizations had the same cross-product defect as the DKIM/SPF alignment tables** ([#169](https://github.com/domainaware/parsedmarc/issues/169)): `policies` is an object array, and each policy's `failure_details` is itself an object array inside it, so stacking terms aggregations on their subfields cross-products the same way. SMTP TLS documents now also carry `policies_combined` — one `"policy_domain / policy_type"` string per policy — and `failure_details_combined` — one `"policy_domain / policy_type / result_type / sending_mta_ip / receiving_ip / receiving_mx_hostname"` string per failure detail — and documents saved by older versions are backfilled automatically at startup the same non-blocking, idempotent way as the DKIM/SPF backfill; the equivalent manual `_update_by_query` command is documented on the Elasticsearch docs page. Also fixed two adjacent dead fields found while making this change: `_SMTPTLSFailureDetailsDoc` declared `additional_information_uri`, but `add_failure_details` passed it to the constructor as `additional_information`, so it was never actually populated on the declared field; and `receiving_mx_hostname`, which `add_failure_details` has always stored, had no field declaration at all. Both are now correctly wired in the Elasticsearch and OpenSearch outputs. - **The Grafana "Map of Message Source Countries" panel's markers now scale with message volume and use a higher-contrast style.** Previously the markers were fixed-size 5 px dots at 50% opacity in dark green, which were nearly invisible on the dark basemap. - **Corrected the dead `_SPFResult.results` (plural) field declaration to `result`**, matching what was always written to it. +- **An mbox-only run no longer shows a misleading, permanently-stuck `0it` progress bar, and mbox parsing now shows a real per-message progress bar on interactive terminals** ([#147](https://github.com/domainaware/parsedmarc/issues/147)). The CLI's progress bar only tracks report files passed directly as arguments — `n_procs` parallel parsing also only applies to those — so runs whose only input was an mbox file displayed an empty bar while messages were parsed sequentially with no visible progress (per-message progress is logged at INFO, hidden in `--silent`/config-file runs). The empty bar is no longer created, `get_dmarc_reports_from_mbox()` now wraps its message loop in a tqdm bar that auto-disables on non-TTY output, and the `n_procs` documentation now states the direct-file-arguments-only scope explicitly. ## 10.2.4 diff --git a/docs/source/usage.md b/docs/source/usage.md index 9c0017cb..c6a93ea2 100644 --- a/docs/source/usage.md +++ b/docs/source/usage.md @@ -169,14 +169,22 @@ The full set of configuration options are: any configured output destination fails while saving/publishing reports (Default: `False`) - `log_file` - str: Write log messages to a file at this path - - `n_procs` - int: Number of process to run in parallel when - parsing in CLI mode (Default: `1`) + - `n_procs` - int: Number of processes to run in parallel when + parsing report files passed directly as CLI arguments + (Default: `1`) :::{note} Setting this to a number larger than one can improve performance when processing thousands of files ::: + :::{note} + `n_procs` only applies to report files passed directly on the + command line. Messages from mbox files and from mailbox + connections (IMAP, Microsoft Graph, Gmail API, Maildir) are + always processed sequentially. + ::: + - `mailbox` - `reports_folder` - str: The mailbox folder (or label for Gmail) where the incoming reports can be found diff --git a/parsedmarc/__init__.py b/parsedmarc/__init__.py index b02ea901..2e4cfe5d 100644 --- a/parsedmarc/__init__.py +++ b/parsedmarc/__init__.py @@ -34,6 +34,7 @@ import mailparser import xmltodict from expiringdict import ExpiringDict from mailsuite.smtp import send_email +from tqdm import tqdm from parsedmarc.constants import ( DEFAULT_DNS_MAX_RETRIES, @@ -2126,7 +2127,7 @@ def get_dmarc_reports_from_mbox( message_keys = mbox.keys() total_messages = len(message_keys) logger.debug("Found {0} messages in {1}".format(total_messages, input_)) - for i in range(len(message_keys)): + for i in tqdm(range(total_messages), disable=None): message_key = message_keys[i] logger.info("Processing message {0} of {1}".format(i + 1, total_messages)) msg_content = mbox.get_string(message_key) diff --git a/parsedmarc/cli.py b/parsedmarc/cli.py index 4d3521bd..2d4ab30c 100644 --- a/parsedmarc/cli.py +++ b/parsedmarc/cli.py @@ -2376,7 +2376,7 @@ def _main(): results = [] pbar = None - if sys.stdout.isatty(): + if sys.stderr.isatty() and len(file_paths) > 0: pbar = tqdm(total=len(file_paths)) n_procs = int(opts.n_procs or 1)