amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-09-12 00:48:00 +00:00
Code Issues Projects Releases Wiki Activity

Merge master into google-secops-parser

Browse Source 
Conflict resolutions:
- CHANGELOG.md: keep this branch's Unreleased section above the
  10.4.1-10.4.3 release sections added on master.
- pyproject.toml: keep both dependency additions (google-auth from this
  branch, httpx from master).

Follow-up to the merge: convert gsecops.py's str.format() calls to
f-strings, required by the ruff 0.16.0 UP030/UP032 enforcement adopted
on master after this branch's base.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-08-18 17:56:45 -04:00
co-authored by Claude Fable 5
parent ae5a5043ab e48521bb47
commit f15d4a309a
65 changed files with 13781 additions and 2078 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/usage.md
+386 -17
View File
@@ -3,23 +3,25 @@
## CLI help
```text
usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads] [-o OUTPUT]
usage: parsedmarc [-h] [-c CONFIG_FILE] [-r] [--strip-attachment-payloads] [-o OUTPUT]
[--aggregate-json-filename AGGREGATE_JSON_FILENAME] [--failure-json-filename FAILURE_JSON_FILENAME]
[--smtp-tls-json-filename SMTP_TLS_JSON_FILENAME] [--aggregate-csv-filename AGGREGATE_CSV_FILENAME]
[--failure-csv-filename FAILURE_CSV_FILENAME] [--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME]
[-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--offline] [-s] [-w] [--verbose] [--debug]
[--log-file LOG_FILE] [--no-prettify-json] [-v]
[-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--dns-retries DNS_RETRIES] [--offline] [-s]
[-w] [--verbose] [--debug] [--log-file LOG_FILE] [--no-prettify-json] [-v]
[file_path ...]
Parses DMARC reports
positional arguments:
file_path one or more paths to aggregate or failure report files, emails, or mbox files'
file_path one or more paths to aggregate or failure report files, emails, mbox files, or directories
containing them
options:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
a path to a configuration file (--silent implied)
-r, --recursive search directories given as file_path recursively, and enable '**' recursion in glob patterns
--strip-attachment-payloads
remove attachment payloads from failure report output
-o OUTPUT, --output OUTPUT
@@ -40,6 +42,8 @@ options:
nameservers to query
-t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
number of seconds to wait for an answer from DNS (default: 2.0)
--dns-retries DNS_RETRIES
number of times to retry DNS queries on timeout or other transient errors (default: 0)
--offline do not make online queries for geolocation or DNS
-s, --silent only print errors
-w, --warnings print warnings in addition to errors
@@ -123,11 +127,30 @@ The full set of configuration options are:
Elasticsearch, Splunk and/or S3
- `save_smtp_tls` - bool: Save SMTP-STS report data to
Elasticsearch, Splunk and/or S3
- `index_prefix_domain_map` - bool: A path mapping of Opensearch/Elasticsearch index prefixes to domain names
- `index_prefix_domain_map` - str: Path to a YAML file mapping
OpenSearch/Elasticsearch index prefixes to domain names
- `strip_attachment_payloads` - bool: Remove attachment
payloads from results
- `silent` - bool: Set this to `False` to output results to STDOUT
- `output` - str: Directory to place JSON and CSV files in. This is required if you set either of the JSON output file options.
- `archive_directory` - str: Optional. When set, successfully
processed report files given as local file/directory path
arguments are moved into
`<archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/`
(year and month come from the report's own begin/arrival date, with
the month zero-padded). A successfully parsed report whose archive
date can't be determined is left in place with a logged warning.
Files that fail to parse as a report are moved to
`<archive_directory>/Invalid/`; files that fail for other reasons,
such as transient I/O errors, are left in place so a later run can
retry them. An existing destination file is never overwritten; a
numeric suffix is appended before the extension (e.g.
`report-1.xml`). This applies only to direct local file input —
reports fetched from mailboxes (IMAP, Microsoft Graph, Gmail API,
Maildir) use `[mailbox] archive_folder` instead, and mbox files are
never moved. Files already inside `archive_directory` are excluded
from processing, so the archive may safely live inside an input
directory. A failed move is logged and does not stop the run.
- `aggregate_json_filename` - str: filename for the aggregate
JSON output file
- `failure_json_filename` - str: filename for the failure
@@ -164,13 +187,32 @@ The full set of configuration options are:
- `fail_on_output_error` - bool: Exit with a non-zero status code if
any configured output destination fails while saving/publishing
reports (Default: `False`)
:::{note}
This option only controls the process exit code. Retaining mailbox
messages whose reports could not be saved is automatic and happens
either way — see
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved).
:::
- `log_file` - str: Write log messages to a file at this path
- `n_procs` - int: Number of process to run in parallel when
parsing in CLI mode (Default: `1`)
- `n_procs` - int: Number of processes to run in parallel when
parsing report files passed directly as CLI arguments, messages
in mbox files, and messages from mailbox connections (IMAP,
Microsoft Graph, Gmail API, Maildir), including watch mode
(Default: `1`)
:::{note}
Setting this to a number larger than one can improve
performance when processing thousands of files
performance when processing thousands of files or messages
:::
:::{note}
Only parsing is parallelized across worker processes. Fetching
messages, deduplicating reports, archiving/deleting mailbox
messages, and saving/publishing to outputs all stay sequential
in the main process. Each worker process keeps its own DNS/GeoIP
cache.
:::
- `mailbox`
@@ -183,12 +225,46 @@ The full set of configuration options are:
messages as they arrive or poll MS Graph for new messages
- `delete` - bool: Delete messages after processing them,
instead of archiving them
- `delete_aggregate` - bool: Delete aggregate report messages
after processing them, instead of archiving them
(Default: the value of `delete`)
- `delete_failure` - bool: Delete failure report messages
after processing them, instead of archiving them
(Default: the value of `delete`)
- `delete_smtp_tls` - bool: Delete SMTP TLS report messages
after processing them, instead of archiving them
(Default: the value of `delete`)
- `delete_invalid` - bool: Delete messages that could not be
parsed, instead of archiving them in the `Invalid`
subfolder, where they can be inspected for debugging
(Default: the value of `delete`)
:::{note}
Each of these four options overrides `delete` for one kind of
message only, and the other three keep inheriting `delete`. So
`delete = True` combined with `delete_failure = False` archives
failure report messages while deleting processed aggregate and
SMTP TLS report messages — and unparseable ones, unless
`delete_invalid = False` is set as well.
:::
- `test` - bool: Do not move or delete messages
- `batch_size` - int: Number of messages to read and process
before saving. Default `10`. Use `0` for no limit.
- `check_timeout` - int: Number of seconds to wait for a IMAP
- `check_timeout` - int: Number of seconds to wait for an IMAP
IDLE response or the number of seconds until the next
mail check (Default: `30`)
- `max_unsaved_retries` - int: How many times a batch of messages
whose reports could not be saved is retried before its messages
are moved to the `Unsaved` archive subfolder instead of being
retried again (Default: `2`, i.e. the initial attempt plus two
retries). Use `0` to move messages on the first failed save;
negative values are rejected.
Failures are counted in memory, so the cap applies across watch-mode
checks within one long-running process, not across separate one-shot
runs. See
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
below.
- `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
Defaults to `1d` if incorrect value is provided.
@@ -249,12 +325,44 @@ The full set of configuration options are:
could be a shared mailbox if the user has access to the mailbox
- `graph_url` - str: Microsoft Graph URL. Allows for use of National Clouds (ex Azure Gov)
(Default: https://graph.microsoft.com)
:::{warning}
Setting `graph_url` alone is **not** sufficient for a national/sovereign
cloud tenant. It only changes the Microsoft Graph API root; the
Microsoft Entra ID (Azure AD) token endpoint used for authentication is
not currently configurable in parsedmarc or `mailsuite`, and always
defaults to the global `https://login.microsoftonline.com`. A true
national-cloud deployment also needs its own Entra ID endpoint, so
`graph_url` by itself only helps if your tenant is registered in the
global cloud but you specifically need to reach one of these Graph API
roots (per [Microsoft's national cloud deployment docs][ms-graph-clouds]):
| National cloud | Microsoft Graph URL | Entra ID endpoint (not configurable here) |
|---|---|---|
| Global (default) | `https://graph.microsoft.com` | `https://login.microsoftonline.com` |
| US Government L4 (GCC High) | `https://graph.microsoft.us` | `https://login.microsoftonline.us` |
| US Government L5 (DoD) | `https://dod-graph.microsoft.us` | `https://login.microsoftonline.us` |
| China, operated by 21Vianet | `https://microsoftgraph.chinacloudapi.cn` | `https://login.chinacloudapi.cn` |
[ms-graph-clouds]: https://learn.microsoft.com/en-us/graph/deployments
:::
- `token_file` - str: Path to save the token file
(Default: `.token`)
- `allow_unencrypted_storage` - bool: Allows the Azure Identity
module to fall back to unencrypted token cache (Default: `False`).
Even if enabled, the cache will always try encrypted storage first.
:::{note}
`token_file` stores the serialized authentication record; the
underlying MSAL persistent token cache is separately named
`parsedmarc`, not `mailsuite`'s own default cache name. This is
deliberate: the Graph mailbox backend used to live directly in
parsedmarc, and moved into the `mailsuite` dependency in parsedmarc
9.11.0. Explicitly keeping the `parsedmarc` cache name means tokens
cached before that move keep working after upgrading — there is
nothing to migrate and no action needed on your part.
:::
:::{note}
You must create an app registration in Azure AD and have an
admin grant the Microsoft Graph `Mail.ReadWrite`
@@ -263,6 +371,11 @@ The full set of configuration options are:
username, you must grant the app `Mail.ReadWrite.Shared`.
:::
| Auth method | Reading (own mailbox) | Reading (shared mailbox) |
|---|---|---|
| `UsernamePassword` / `DeviceCode` (delegated) | `Mail.ReadWrite` | `Mail.ReadWrite.Shared` |
| `ClientSecret` / `Certificate` / `ClientAssertion` (application) | `Mail.ReadWrite` (application), scoped via `New-ApplicationAccessPolicy` | same as own mailbox — app-only access is scoped by policy, not by permission name |
:::{tip}
**Troubleshooting connections.** Run with `--verbose` to log a
redacted connection summary (auth method, tenant, client ID,
@@ -273,6 +386,11 @@ The full set of configuration options are:
an Exchange Online-side one), Microsoft Graph SDK requests, and
`httpx` HTTP request lines. Secret values (passwords, client
secrets, certificate passwords) are never written to logs.
Connection, mailbox fetch, message send, and `--watch` failures
each log a single ERROR line naming the mailbox, tenant, auth
method, and the Graph `request-id`/`client-request-id` when
available — worth quoting verbatim when contacting Microsoft
support.
:::
:::{warning}
@@ -295,6 +413,120 @@ The full set of configuration options are:
applies to the `Certificate` and `ClientAssertion` auth methods.
:::
**Sending the summary email via Microsoft Graph.**
When `[msgraph]` is configured and `[smtp]` has a `to` value but no
`host`, the periodic summary email is sent through the same
already-authenticated Graph mailbox connection used for reading
(`/users/{mailbox}/sendMail`), and a copy is saved to Sent Items.
When `[smtp] host` is set, SMTP is used regardless of whether
`[msgraph]` is also configured — SMTP is always preferred, with no
automatic fallback to Graph on SMTP failure.
Example config combining `[msgraph]` with a `[smtp]` section that
only sets `to`/`subject` (no `host`):
```ini
[msgraph]
auth_method = Certificate
client_id = ...
tenant_id = ...
mailbox = dmarc-reports@example.com
certificate_path = /path/to/cert.pem
[smtp]
to = admin@example.com
subject = DMARC Summary
```
Required Microsoft Graph permissions, in addition to the
reading-related permissions documented above:
| Auth method | Reading | Sending (own mailbox) | Sending (shared mailbox) |
|---|---|---|---|
| `UsernamePassword` / `DeviceCode` (delegated) | `Mail.ReadWrite` (+`.Shared` for shared) | `Mail.Send` | `Mail.Send.Shared` |
| `ClientSecret` / `Certificate` / `ClientAssertion` (application) | `Mail.ReadWrite` (application) | `Mail.Send` (application) | `Mail.Send` (application), scoped via `New-ApplicationAccessPolicy` |
:::{warning}
Graph-based sending is only confirmed to work with the app-only
auth methods (`ClientSecret`, `Certificate`, `ClientAssertion`).
The delegated auth methods (`UsernamePassword`, `DeviceCode`)
currently request only the `Mail.ReadWrite`(`.Shared`) scope when
authenticating, not `Mail.Send` — so a delegated connection's
access token will not carry `Mail.Send` even if an administrator
has granted it, and `/sendMail` calls are expected to fail with an
access-denied error regardless of what's granted in Azure AD. Use
an app-only auth method if you need Graph-based sending.
:::
**Minimal example configs.** Each auth method needs a different
minimum set of keys. These read-only examples omit `[smtp]`; see
above for adding Graph-based sending on top of any of them.
`UsernamePassword` (delegated, own mailbox):
```ini
[msgraph]
auth_method = UsernamePassword
client_id = ...
client_secret = ...
user = dmarc-reports@example.com
password = ...
```
`DeviceCode` (delegated, interactive sign-in on first run — `user`
is the account that signs in; `mailbox` is the shared mailbox it
reads, and only needs to differ from `user` to request
`Mail.ReadWrite.Shared` instead of plain `Mail.ReadWrite`):
```ini
[msgraph]
auth_method = DeviceCode
client_id = ...
tenant_id = ...
user = signing-in-user@example.com
mailbox = dmarc-reports@example.com
```
`ClientSecret` (app-only):
```ini
[msgraph]
auth_method = ClientSecret
client_id = ...
tenant_id = ...
client_secret = ...
mailbox = dmarc-reports@example.com
```
`Certificate` (app-only):
```ini
[msgraph]
auth_method = Certificate
client_id = ...
tenant_id = ...
certificate_path = /path/to/cert.pem
mailbox = dmarc-reports@example.com
```
`ClientAssertion` (app-only, short-lived JWT — see the note above
about its unsuitability for `watch` mode):
```ini
[msgraph]
auth_method = ClientAssertion
client_id = ...
tenant_id = ...
client_assertion = ...
mailbox = dmarc-reports@example.com
```
:::{tip}
**Troubleshooting.**
| Error | Cause | Fix |
|---|---|---|
| *"...needs permission to access resources in your organization that only an admin can grant"* / "Admin consent required" | A delegated auth method (`UsernamePassword`, `DeviceCode`) is authenticating with a scope (`Mail.ReadWrite` or `Mail.ReadWrite.Shared`) the tenant admin hasn't consented to yet. | Have an Entra ID admin grant consent: Azure Portal → **Enterprise Applications** → *your app* → **Permissions** → **Grant admin consent**, or `az ad app permission admin-consent --id <CLIENT_ID>`. This is separate from the `New-ApplicationAccessPolicy` step above, which only applies to app-only auth. |
| `ErrorItemNotFound: ... Default folder Root not found` | `mailsuite` can resolve the well-known folders (`Inbox`, `Archive`, `Drafts`, `Sent Items`, `Deleted Items`, `Junk Email`) even when a mailbox's folder hierarchy hasn't fully provisioned, but a **custom, non-well-known** `reports_folder` name still fails to resolve on such a mailbox. | Point `reports_folder` at (or under) one of the six well-known folder names above, or sign into the (shared) mailbox once via Outlook/OWA to force Exchange to provision it, then retry. |
| `RuntimeError: Event loop is closed` | Historical bug, fixed in `mailsuite` 2.0.2. Not reachable with the `mailsuite>=2.2.2` this project requires. | Confirm your installed `mailsuite` version is current (`pip show mailsuite`); upgrade if it's somehow pinned below 2.0.2. |
| Invalid/rejected timestamp in the `since`/`receivedDateTime` filter | Historical bug (parsedmarc [#706](https://github.com/domainaware/parsedmarc/pull/706)/[#708](https://github.com/domainaware/parsedmarc/pull/708)): older versions appended a spurious `Z` to an already-UTC-offset ISO timestamp. Fixed since parsedmarc 9.5.1/9.5.5. | Upgrade parsedmarc if you're on a version older than 9.5.5. |
:::
- `elasticsearch`
- `hosts` - str: A comma separated list of hostnames and ports
or URLs (e.g. `127.0.0.1:9200` or
@@ -371,20 +603,33 @@ The full set of configuration options are:
- `aggregate_topic` - str: The Kafka topic for aggregate reports
- `failure_topic` - str: The Kafka topic for failure reports
- `smtp`
- `host` - str: The SMTP hostname
The results email is only sent when at least one aggregate, failure,
or SMTP TLS report was parsed during the run; an empty run (e.g. an
empty inbox) skips the email instead of sending headers-only CSVs.
- `host` - str: The SMTP hostname. Required unless `[msgraph]` is
configured, in which case omitting it sends the summary via
Microsoft Graph instead — see "Sending the summary email via
Microsoft Graph" above.
- `port` - int: The SMTP port (Default: `25`)
- `ssl` - bool: Require SSL/TLS instead of using STARTTLS
- `skip_certificate_verification` - bool: Skip certificate
verification (not recommended)
- `user` - str: the SMTP username
- `password` - str: the SMTP password
- `from` - str: The From header to use in the email
- `user` - str: the SMTP username. SMTP-only; not used when sending
via Microsoft Graph.
- `password` - str: the SMTP password. SMTP-only; not used when
sending via Microsoft Graph.
- `from` - str: The From header to use in the email. SMTP-only.
When sent via Microsoft Graph, the message's `From` is always the
`[msgraph]` mailbox — `[smtp] from` has no effect.
- `to` - list: A list of email addresses to send to
- `subject` - str: The Subject header to use in the email
(Default: `parsedmarc report`)
- `attachment` - str: The ZIP attachment filenames
(Default: `DMARC-<YYYY-MM-DD>.zip`)
- `message` - str: The email message
(Default: `Please see the attached parsedmarc report.`)
(Default: `Please see the attached DMARC results.`)
:::{note}
`%` characters must be escaped with another `%` character,
@@ -653,6 +898,78 @@ PUT _cluster/settings
Increasing this value increases resource usage.
:::
### Mailbox messages are only archived once the reports are saved
parsedmarc processes a mailbox in batches of `batch_size` messages. Each
batch is written to every configured output destination *before* any of
that batch's messages are archived or deleted. If any destination reports a
failure — an Elasticsearch outage, an expired Splunk HEC token, an
unreachable Kafka broker, a full `--output` disk — the whole batch is left
in the reports folder and retried on the next run or watch-mode check, so a
report is never removed from the mailbox while it exists nowhere else
([issue #242](https://github.com/domainaware/parsedmarc/issues/242)).
This is all-or-nothing per batch: archiving a batch because most
destinations accepted it would still permanently lose the data for the one
that didn't. It also applies regardless of `fail_on_output_error`, which
only controls the process exit code. Messages that could not be parsed at
all carry no report data, so they are filed under `Invalid` (or deleted per
`delete_invalid`) as usual.
A destination that is broken rather than briefly unavailable would
otherwise be retried forever, so retries are capped. Once a message's batch
has failed `max_unsaved_retries + 1` times (three times by default), that
message is moved to `<archive_folder>/Unsaved` and stops being retried. **A
message is never deleted on this path, whatever the `delete` options say.**
To recover after fixing the output destination, either move the messages
from `Archive/Unsaved` back into the reports folder, or run parsedmarc once
with `reports_folder = Archive/Unsaved`.
:::{note}
The failure counts live in memory, so they are counted per parsedmarc
process. In watch mode — a long-running process that checks the mailbox
repeatedly — the cap works as described across checks. A one-shot run
(`cron`, `systemd` timers) attempts each message exactly once and then
exits, so its counts start over every time and the default cap is never
reached: messages simply keep being retried on every run, which is the
safe direction. Set `max_unsaved_retries = 0` if you want one-shot runs to
move unsavable messages to `Unsaved` immediately instead.
:::
:::{warning}
Retrying a batch means re-sending it. Output destinations that
deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize
an already-saved report — are unaffected, and S3 is idempotent because
each report is written to an object key built from its type, date, and
report ID, so a retry overwrites the same object. Kafka,
Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the
`--output` JSON/CSV files all append unconditionally, so each retry adds
another copy of every report in the batch. That is why the default retry
cap is deliberately low: at most three deliveries per report before its
message is set aside in `Unsaved`. The summary email covers everything
parsed in a run, including reports whose batch failed to save, so a report
retried across runs can also appear in more than one summary email.
:::
:::{note}
Not every destination can report a failed delivery. The webhook output
deliberately logs and swallows its own HTTP and network errors, and the
syslog and GELF outputs send through Python logging handlers, which
swallow delivery errors by design — so an unreachable webhook, syslog, or
GELF endpoint is *not* treated as a failed save and does not hold a
batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
Kafka, S3, PostgreSQL, Azure Log Analytics, and the `--output` files are
all detected and do.
:::
:::{note}
`since` interacts with retries: a message that ages out of the configured
`since` window stops being fetched, and therefore stops being retried
automatically. It is never deleted or moved — it simply stays in the
reports folder until it is processed by a run with a wider (or no)
`since` window.
:::
## Environment variable configuration
Any configuration option can be set via environment variables using the
@@ -789,6 +1106,50 @@ For sections with underscores in the name, the full section name is used:
| `webhook` | `PARSEDMARC_WEBHOOK_` |
| `gsecops` | `PARSEDMARC_GSECOPS_` |
## Using parsedmarc as a library
`parsedmarc` is also importable as a regular Python package, not just a CLI
tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`,
`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`,
`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and
`watch_inbox()` — are all importable
directly from the `parsedmarc` package. See the [API reference](api.md) for
the full set of modules and members.
Each of these functions accepts either individual option keyword arguments
(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword
argument carrying a `ParserConfig` instance:
```python
from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
from parsedmarc.mail import IMAPConnection
config = ParserConfig(
offline=False,
nameservers=["1.1.1.1", "1.0.0.1"],
dns_timeout=5.0,
)
report = parse_report_file("aggregate_report.xml.gz", config=config)
connection = IMAPConnection(
host="imap.example.com", user="dmarc@example.com", password="..."
)
results = get_dmarc_reports_from_mailbox(connection, config=config)
```
A few things to keep in mind:
- When `config=` is passed, the individual option keyword arguments are
ignored in favor of the values carried on the `ParserConfig` instance.
- Each explicitly constructed `ParserConfig` owns its own isolated caches
(IP address info, seen aggregate report IDs, and the reverse DNS map).
Omitting `config=` falls back to the process-wide caches shared by every
call that doesn't pass one.
- `keep_alive` and `n_procs` are not part of `ParserConfig` — they control
process/worker orchestration rather than parsing or enrichment behavior,
so they are always passed as separate keyword arguments.
## Performance tuning
For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
@@ -799,8 +1160,12 @@ imports more predictable:
- Reduce `mailbox.batch_size` to smaller values such as `100-500` instead of
processing a very large message set at once. Smaller batches trade throughput
for lower peak memory use and less sink pressure.
- Keep `n_procs` low for mailbox-heavy runs. In practice, `1-2` workers is often
a safer starting point for large backfills than aggressive parallelism.
- `n_procs` now parallelizes parsing for mailbox and mbox runs too, not just
report files passed directly as CLI arguments. It pays off most when a run
is bound by DNS/GeoIP enrichment rather than fetching or output. The
trade-off is memory and DNS load: at most roughly `2 * n_procs` messages
are held in flight at once, each worker process keeps its own DNS/GeoIP
cache, and DNS query volume can multiply by up to `n_procs`.
- Use `mailbox.since` to process reports in smaller time windows such as `1d`,
`7d`, or another interval that fits the backlog. This makes it easier to catch
up incrementally instead of loading an entire mailbox history in one run.
@@ -832,12 +1197,16 @@ whalensolutions:
Save it to disk where the user running ParseDMARC can read it, then set `index_prefix_domain_map` to that filepath in the `[general]` section of the ParseDMARC configuration file and do not set an `index_prefix` option in the `[elasticsearch]` or `[opensearch]` sections.
When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of Opensearch or the ELK stack to only grant users access to the indexes that they need.
When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
:::{note}
A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
:::
Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See [Backfilling the combined DKIM/SPF result fields](elasticsearch.md#backfilling-the-combined-dkimspf-result-fields) for the details.
## Running parsedmarc as a systemd service
Use systemd to run `parsedmarc` as a service and process reports as