mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-10 19:11:00 +00:00
Sean WhalenandClaude Fable 5.1
e8292bef70
* Wrap the failure-report duplicate search in the output error The aggregate and SMTP TLS saves turn a failed duplicate-check search into ElasticsearchError or OpenSearchError; the failure-report saves let the search library's exception escape instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Refuse date strings too long to be dates human_timestamp_to_datetime() had no length limit; a digit run of a million characters took 12 s in dateutil. Add a 128 character limit (ValueError) and map it to the matching Invalid*Report at the failure Arrival-Date, the message Date header and the failure-report sample Date in the Elasticsearch and OpenSearch saves. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Skip failure sample headers with an empty name The Elasticsearch and OpenSearch failure saves normalise each header name; a name that is empty after that is not a valid field name, so leave the header out and save the rest. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add a persisted store of unfinished parse attempts A message whose parse kills the process (the out-of-memory killer, a container memory cap) is never moved out of the reports folder, so every later run parses it and dies again. parsedmarc.attempts keeps a small JSON file counting, per message, the parse attempts that started but never finished, so the mailbox loop can set such a message aside after MAX_PARSE_ATTEMPTS. The key hashes the first 64 KiB of the message because mailsuite's IMAP backend does not expose UIDVALIDITY, so a reused UID must not inherit an old count. Writes are atomic and owner only, and a failed write turns the store off instead of stopping a run. Refs GHSA-qcc2-4x3v-v4g7, GHSA-q7gm-8cj9-94hr, GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Set aside mailbox messages that keep killing the parse A message whose parse ends the process stays in the reports folder, so every one-shot run or service restart parsed it again and died before reaching the reports behind it. The new [mailbox] attempts_file option records each message in an AttemptStore before it is parsed and clears the entry once the parse returns a report or raises ParserError, so an entry left behind marks a parse that never finished; a message with MAX_PARSE_ATTEMPTS of them is disposed of as invalid without parsing. The entry is cleared before the save, so a failed save is never counted as a crash, and only after a successful move, so a failed move is retried. With n_procs > 1 a dead run had a whole submission window in flight, so counting every message submitted beside the culprit would set innocent reports aside on the third run. A message with one unfinished attempt is therefore parsed alone in the calling process after the rest of the batch, and a second death counts against it only. PARSEDMARC_MAILBOX_ATTEMPTS_FILE names the file directly, so the key joins _DIRECT_FILE_KEYS; without it the variable was read as a Docker-secret reference. Refs GHSA-qcc2-4x3v-v4g7, GHSA-q7gm-8cj9-94hr, GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stop a parallel_map job that runs past a timeout The parse limits added for GHSA-qcc2-4x3v-v4g7 bound every input known to parse slowly, but nothing bounds a parse they do not cover: with n_procs = 1 the rest of the mailbox waits behind it. parallel_map() now takes parse_timeout and timeout_result. A job that runs that long gets timeout_result(job, parse_timeout) in its place; since a ProcessPoolExecutor cannot stop one job, every worker is killed, a new pool is started and the other unfinished jobs are resubmitted. A job's clock starts when its worker reports that it began, through a queue the pool initializer hands each worker. Future.running() is not enough: the executor marks one job more than it has workers as running, so a job queued behind a slow neighbour would be charged for the wait. ParseTimeoutError subclasses InvalidDMARCReport so the mbox and mailbox loops treat a timed-out message as invalid without changes. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add parse_timeout to the mbox, mailbox and watch functions get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox() and watch_inbox() take parse_timeout and pass it to parallel_map() with the email job's timeout value. A timeout set with n_procs = 1 also takes the parallel_map() branch, since only a parse in a worker process can be stopped. The mailbox function and watch_inbox() reject an invalid value before fetching or watching, as they do for max_unsaved_retries: raised inside a watch check, the error would be retried on every check. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the [general] parse_timeout option The CLI reads parse_timeout as seconds, rejects a negative or non-numeric value as a ConfigurationError, defaults it to 0 (no limit), and passes it to the report-file, mbox, mailbox and watch calls. The watch call reads it from opts, so a SIGHUP reload takes a new value. The default stays 0 because the parse limits already bound the known inputs and a timeout routes every n_procs = 1 run through a worker process; the documentation suggests 300 seconds. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Document the attempts_file option Describe [mailbox] attempts_file in the option list and in a new usage section on messages that crash parsedmarc, say in the Unsaved note that the failed-save counts are the in-memory ones, keep the file in a state directory in the systemd example and in a named volume in the Docker Compose example, and add the Security changelog entry for the retry-loop backstop the three advisories deferred. Refs GHSA-qcc2-4x3v-v4g7, GHSA-q7gm-8cj9-94hr, GHSA-4272-4r8g-6c5h Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * List parse_timeout and attempts_file as orchestration options ParserConfig's docstring and the usage page's ParserConfig notes name the keyword arguments that stay outside the dataclass. The two new options are such arguments, so a library caller looking for them on ParserConfig is told where they go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound a suspect message's parse by parse_timeout With an attempts file, a message that was being parsed when an earlier run died is parsed alone after the rest of its batch, so that a second death is charged to it and not to the messages parsed beside it. That parse ran in the main process, where parse_timeout cannot stop it, so the one message most likely to be slow or fatal was the one the timeout did not cover. With a timeout set, each suspect now gets a worker pool of its own, one message per pool so a crash is still attributed to it alone, and its result is taken the same way as the batch's. Without a timeout the suspect is still parsed in this process. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore default signal handlers in parsing workers Under the fork start method a worker inherits the parent's signal handlers, and the CLI installs SIGTERM and SIGINT handlers that only set a flag. When a worker dies, ProcessPoolExecutor terminates the others with SIGTERM and joins them; a worker running the inherited handler ignores that signal, so the teardown waited forever and the run hung instead of failing. The pool initializer now resets both signals to their defaults. The logging tests that call the initializer in the test process put the process's own handlers back afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Order the Unreleased changelog and tie its entries together The three branches each added to the Unreleased entry. The GHSA-q7gm bullet still promised a follow-up for a persisted failed-attempt marker, which the attempts_file option now is, so it names the option. The attempts_file bullet now says that a suspect message is parsed in a worker of its own when parse_timeout is set, matching the code. The #942 fix moves up beside the other fix already on master, and the worker signal-handler fix gets its own line. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ignore SIGINT in workers and end the batch when one dies Resetting SIGINT to its default in the workers made a Ctrl-C, which the terminal sends to the whole process group, kill every worker, so the run ended in BrokenProcessPool where the CLI's handler meant to let the batch finish. Workers now ignore SIGINT and keep the default SIGTERM action, which ProcessPoolExecutor needs to tear down a broken pool. A worker that dies for another reason (the out-of-memory killer, a crash) raised BrokenProcessPool out of parallel_map, which no caller handled: the file and mbox paths ended in a traceback. parallel_map now kills the remaining workers, closes the start queue and raises ParserError from it, and the CLI file and mbox paths log it and exit 1 as the mailbox paths already did. The in-flight messages keep their attempts-file entries, so the next run parses each alone. The systemd example gains KillMode=mixed so stopping the service signals only the main process and the batch in progress finishes. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * State the parse_timeout bounds the code actually gives A slow job running beside another that expires loses its partial run and starts again, so P slow jobs cost one more round of parse_timeout than the docstring said, plus the bounded joins of the killed workers, during which no heartbeat is sent. The clocks are also checked only while the parent waits for a result, so "at most one second past the timeout" was not true while the caller is busy between results; the docstring, usage.md and the changelog now say a parse is stopped within one wait slice of the parent next waiting. ParseTimeoutError's docstring says it is the value for every timed-out input, SMTP TLS and unrecognised ones included. A new test closes the results iterator mid-run and checks the workers are killed and the start queue closed; the zero-timeout test now observes which pool ran the job instead of patching _DeadlinePool. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct the attempts-file wording and test the parallel test mode Suspect isolation happens whenever the mailbox takes the parallel path, which parse_timeout selects even with n_procs = 1; the docstring, comment, usage.md and changelog now say so. The set-aside log said a message "has crashed parsedmarc {count-1} time(s)", which overstates the crashes once a failed move has added attempts; it now reports the unfinished attempts against the cap. usage.md lists attempts_file among the _file keys read directly from the environment and among the settings a SIGHUP reloads, and says that a failed write clearing an entry after a good parse can lead to a healthy message being set aside after two such failures. The test-mode set-aside test now runs the parallel path too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Measure a date's length after its comments are removed The length guard in human_timestamp_to_datetime() ran before the RFC 5322 comments were stripped. RFC 5322 section 3.2.2 puts no limit on a comment's length, so a valid Date header with a long comment was refused, and through the sample Date path the whole report email with it. The check now runs on the stripped value, which is linear to produce; a long digit run survives stripping, so it is still refused before dateutil sees it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Save parsed reports before exiting on a dead worker A ParserError from parallel_map (a worker process that died) or from an mbox made the CLI exit at once, discarding every report parsed so far; report files already moved under archive_directory were then saved nowhere. The CLI now stops reading input, saves what it gathered, and exits with status 1. Workers ignore SIGINT, so the plain pool path's executor shutdown on an early exit (KeyboardInterrupt, closing the generator) waited for the whole submission window. Any early exit now kills the workers, as the timed path already did. The Ctrl-C test ran under fork only, where a worker inherits the test's own harmless handler, so it passed without the SIGINT line. It now runs under every available start method and signals each worker once it has started a job. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stop counting a message that is already set aside A message at the attempt limit that stayed in the folder (a failed move, test mode) was recorded again on every later run, so the logged count grew without bound. It is now set aside without being recorded, and the log names the stored count and the limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe worker death per mode and the systemd stop window A dead worker ends a one-shot run with status 1 after saving, but in watch mode the IMAP and Maildir watchers log it and check again, while Graph and Gmail exit. KillMode=mixed lets the batch finish only within TimeoutStopSec. Also say that queued messages keep attempt entries too, and re-wrap a parse_timeout note line. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Clear the seen report IDs around the worker-death CLI tests The new save-before-exit test parses a sample in the parent, which records it in the shared dedup set, so a later test of the same sample saw it as a duplicate and sent no results email. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Save file reports when a mailbox fetch ends in a ParserError A worker that died during the one-shot mailbox fetch reached the ParserError handler, which exited before the reports from report files and mbox inputs were saved; files already moved under archive_directory were then kept nowhere. The handler now saves them first. A second test covers the input-error path: no mailbox is opened and the parsed file stays under archive_directory. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say what a mailbox run keeps when a worker dies Earlier batches are saved and the batch in progress stays in the mailbox. Also say that a set-aside message is not recorded again, so its entry ages out after 30 days, and rewrap two paragraphs. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which set-aside entries age out of the attempts file The 30-day drop applies only to an entry that stays: a failed move, or a message left in place in test mode. The sentence read as if every set-aside message kept an entry, contradicting the line before it. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Carry the test attempts path in the message, not the environment test_parallel_path_records_each_attempt_before_submitting_it passed the attempts file's path to its worker job through os.environ. A forkserver worker forks from a server started before the patch, so it never saw the variable and the job raised KeyError (Python 3.14's Linux default). The path now travels in a header line of the message itself, which reaches the worker under every start method. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test the worker-side helpers and the save-failure exits Codecov listed added lines no test reached: the timed pool's initializer and job wrapper, which run only in workers; _kill_workers on an executor without _processes; the plain path's heartbeat; the CLI's progress bar; the mbox loop's stop after a file worker dies; and a failed save before exiting, on the file/mbox path and the mailbox path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read start tokens every round, not only while waiting With parse_timeout set, the parent read the workers' start tokens only inside its wait loop. When results were already done at harvest time it never waited, so each finished job stayed in the pool's token map with its whole input, and the map grew with the run past the submission window. The tokens are now read at the top of every round, and close() clears the map. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Kill the parsing workers on a second Ctrl-C The second Ctrl-C ends parsedmarc with os._exit(130), which skips the process pool's cleanup. Workers now ignore SIGINT, so under every start method a running worker survived as an orphan and kept parsing, without end when parse_timeout is off. The handler now kills the active child processes first; Python runs it in the main thread between bytecodes, so active_children() and Process.kill() are safe there. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hand timed results back through files, not the pool's pipe With parse_timeout set, a timeout kills every worker, possibly one that is sending a large result. The executor's management thread then waits for the rest of that message forever; the bounded join returns with the thread alive, and because it is not a daemon the interpreter hangs at exit. Reproduced under fork, forkserver and spawn with a 256-400 MiB result. Workers of the timed pool now pickle each outcome, error values included, to a file in a private directory the pool owns and return None; every message left on a pipe is far below PIPE_BUF, which POSIX writes atomically. The pool without a timeout never kills a running worker and is unchanged. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Report an unwritable temporary directory as a ParserError With parse_timeout set the results travel through a temporary directory. If it could not be created, or a worker could not write its result file, a raw OSError escaped parallel_map, which the CLI does not handle. The pool now raises ParserError naming the cause; a worker that cannot write its file returns the error text through the executor, still a small message, and the parent raises the same ParserError. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep every hand-back failure a short ParserError A result the worker could not pickle (PicklingError, TypeError, RecursionError, MemoryError) escaped through the executor with a traceback of any size, breaking the short-message guarantee the timed pool relies on, and a result file that did not unpickle raised a raw exception in the parent. The worker now catches any failure to write its outcome and returns a short marker, its text cut to 200 characters so the message stays below the 512-byte minimum PIPE_BUF; the parent turns it, and any failure to load a result file, into a ParserError. Also corrects the PIPE_BUF figure in the module docstring, names the temporary directory as TMPDIR, and avoids a second 128 MiB object in a test. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cap the hand-back error text in bytes and survive a failing __str__ The pipe's atomic-write bound is in bytes, so a 200-character text of multibyte characters could exceed it; the cap now applies to the UTF-8 encoding. An exception whose __str__ raises is named by its type alone, so nothing in the worker's error path can raise through the executor. A failed write's .tmp file is removed with the pool's result directory; the docstring says so. Refs GHSA-qcc2-4x3v-v4g7 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Match the pickler's message across Python versions in a test The unpicklable-result test asserted "Can't get local object", which is the wording of Python 3.12 and 3.13 only; 3.10, 3.11 and 3.14 say "Can't pickle local object". Assert the part they share. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Cover the worker-side hand-back failures in the test process The three failure branches of _run_timed_job and _take_outcome run in worker processes in the pool tests, where coverage cannot see them; call them directly here with an unwritable directory, an unpicklable result and an unreadable result file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Assert the unreadable-result message by its text, and format The temporary-directory error carries the OSError's text, not its class name, so the test asserts "Is a directory"; one long test line is wrapped as ruff format wants. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Narrow the marker before unpacking it in two tests pyright cannot narrow Optional through assertIsNotNone; a plain assert does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1749 lines
79 KiB
Markdown
1749 lines
79 KiB
Markdown
# Using parsedmarc
|
||
|
||
## CLI help
|
||
|
||
```text
|
||
usage: parsedmarc [-h] [-c CONFIG_FILE] [-r] [--strip-attachment-payloads] [-o OUTPUT]
|
||
[--aggregate-json-filename AGGREGATE_JSON_FILENAME] [--failure-json-filename FAILURE_JSON_FILENAME]
|
||
[--smtp-tls-json-filename SMTP_TLS_JSON_FILENAME] [--aggregate-csv-filename AGGREGATE_CSV_FILENAME]
|
||
[--failure-csv-filename FAILURE_CSV_FILENAME] [--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME]
|
||
[-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--dns-retries DNS_RETRIES] [--offline] [-s]
|
||
[-w] [--verbose] [--debug] [--log-file LOG_FILE] [--no-prettify-json] [-v]
|
||
[file_path ...]
|
||
|
||
Parses DMARC and SMTP TLS reports
|
||
|
||
positional arguments:
|
||
file_path one or more paths to aggregate, failure, or SMTP TLS report files, emails, mbox files, or
|
||
directories containing them
|
||
|
||
options:
|
||
-h, --help show this help message and exit
|
||
-c CONFIG_FILE, --config-file CONFIG_FILE
|
||
a path to a configuration file (--silent implied)
|
||
-r, --recursive search directories given as file_path recursively, and enable '**' recursion in glob patterns
|
||
--strip-attachment-payloads
|
||
remove attachment payloads from failure report output
|
||
-o OUTPUT, --output OUTPUT
|
||
write output files to the given directory
|
||
--aggregate-json-filename AGGREGATE_JSON_FILENAME
|
||
filename for the aggregate JSON output file
|
||
--failure-json-filename FAILURE_JSON_FILENAME
|
||
filename for the failure JSON output file
|
||
--smtp-tls-json-filename SMTP_TLS_JSON_FILENAME
|
||
filename for the SMTP TLS JSON output file
|
||
--aggregate-csv-filename AGGREGATE_CSV_FILENAME
|
||
filename for the aggregate CSV output file
|
||
--failure-csv-filename FAILURE_CSV_FILENAME
|
||
filename for the failure CSV output file
|
||
--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME
|
||
filename for the SMTP TLS CSV output file
|
||
-n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
|
||
nameservers to query: IP addresses, https:// URLs (DNS over HTTPS), and/or
|
||
tls://ip[:port][#hostname] (DNS over TLS)
|
||
-t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT, --dns-timeout DNS_TIMEOUT
|
||
number of seconds to wait for an answer from DNS (default: 2.0)
|
||
--dns-retries DNS_RETRIES
|
||
number of times to retry DNS queries on timeout or other transient errors (default: 0)
|
||
--offline do not make online queries for geolocation or DNS
|
||
-s, --silent only print errors
|
||
-w, --warnings print warnings in addition to errors
|
||
--verbose more verbose output
|
||
--debug print debugging information
|
||
--log-file LOG_FILE output logging to a file
|
||
--no-prettify-json output JSON in a single line without indentation
|
||
-v, --version show program's version number and exit
|
||
```
|
||
|
||
:::{note}
|
||
Starting in `parsedmarc` 6.0.0, most CLI options were moved to a
|
||
configuration file, described below.
|
||
:::
|
||
|
||
## Configuration file
|
||
|
||
`parsedmarc` can be configured by supplying the path to an INI file
|
||
|
||
```bash
|
||
parsedmarc -c /etc/parsedmarc.ini
|
||
```
|
||
|
||
For example
|
||
|
||
```ini
|
||
# This is an example comment
|
||
|
||
[general]
|
||
save_aggregate = True
|
||
save_failure = True
|
||
|
||
[imap]
|
||
host = imap.example.com
|
||
user = dmarcresports@example.com
|
||
password = $uperSecure
|
||
|
||
[mailbox]
|
||
watch = True
|
||
delete = False
|
||
|
||
[elasticsearch]
|
||
hosts = 127.0.0.1:9200
|
||
ssl = False
|
||
|
||
[opensearch]
|
||
hosts = https://127.0.0.1:9200
|
||
user = admin
|
||
password = PasswordGoesHere
|
||
ssl = True
|
||
|
||
[splunk_hec]
|
||
url = https://splunkhec.example.com
|
||
token = HECTokenGoesHere
|
||
index = email
|
||
|
||
[s3]
|
||
bucket = my-bucket
|
||
path = parsedmarc
|
||
|
||
[syslog]
|
||
server = localhost
|
||
port = 514
|
||
|
||
[gelf]
|
||
host = logger
|
||
port = 12201
|
||
mode = tcp
|
||
|
||
[webhook]
|
||
aggregate_url = https://aggregate_url.example.com
|
||
failure_url = https://failure_url.example.com
|
||
smtp_tls_url = https://smtp_tls_url.example.com
|
||
timeout = 60
|
||
|
||
[loki]
|
||
url = http://loki.example.com:3100
|
||
labels = host=mail.example.com
|
||
```
|
||
|
||
The full set of configuration options is:
|
||
|
||
- `general`
|
||
- `save_aggregate` - bool: Save aggregate report data to
|
||
Elasticsearch, Splunk and/or S3
|
||
- `save_failure` - bool: Save failure report data to
|
||
Elasticsearch, Splunk and/or S3
|
||
- `save_smtp_tls` - bool: Save SMTP TLS report data to
|
||
Elasticsearch, Splunk and/or S3
|
||
- `index_prefix_domain_map` - str: Path to a YAML file mapping
|
||
OpenSearch/Elasticsearch index prefixes to domain names
|
||
- `strip_attachment_payloads` - bool: Remove attachment
|
||
payloads from results
|
||
- `silent` - bool: Only print errors; set this to `False` to output
|
||
results to STDOUT (Default: `True`)
|
||
- `output` - str: Directory to place JSON and CSV files in. The JSON
|
||
and CSV filename options below only take effect when this is set.
|
||
CSV cells that begin with `=`, `+`, `-`, `@`, a tab, or a
|
||
carriage return are written with a leading apostrophe (`'`), so
|
||
spreadsheet applications treat them as text rather than formulas
|
||
([CSV injection](https://owasp.org/www-community/attacks/CSV_Injection)).
|
||
- `archive_directory` - str: Optional. When set, successfully
|
||
processed report files given as local file/directory path
|
||
arguments are moved into
|
||
`<archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/`
|
||
(year and month come from the report's own begin/arrival date, with
|
||
the month zero-padded). A successfully parsed report whose archive
|
||
date can't be determined is left in place with a logged warning.
|
||
Files that fail to parse as a report are moved to
|
||
`<archive_directory>/Invalid/`; files that fail for other reasons,
|
||
such as transient I/O errors, are left in place so a later run can
|
||
retry them. An existing destination file is never overwritten; a
|
||
numeric suffix is appended before the extension (e.g.
|
||
`report-1.xml`). This applies only to direct local file input —
|
||
reports fetched from mailboxes (IMAP, Microsoft Graph, Gmail API,
|
||
Maildir) use `[mailbox] archive_folder` instead, and mbox files are
|
||
never moved. Files already inside `archive_directory` are excluded
|
||
from processing, so the archive may safely live inside an input
|
||
directory. A failed move is logged and does not stop the run.
|
||
- `aggregate_json_filename` - str: filename for the aggregate
|
||
JSON output file
|
||
- `failure_json_filename` - str: filename for the failure
|
||
JSON output file
|
||
- `smtp_tls_json_filename` - str: filename for the SMTP TLS
|
||
JSON output file
|
||
- `aggregate_csv_filename` - str: filename for the aggregate
|
||
CSV output file
|
||
- `failure_csv_filename` - str: filename for the failure
|
||
CSV output file
|
||
- `smtp_tls_csv_filename` - str: filename for the SMTP TLS
|
||
CSV output file
|
||
- `prettify_json` - bool: Set this to `False` to output JSON in a
|
||
single line without indentation (Default: `True`)
|
||
- `normalize_timespan_threshold_hours` - float: Aggregate reports
|
||
covering a longer time span than this many hours have their
|
||
records normalized into per-day records (Default: `24`)
|
||
- `ip_db_path` - str: An optional custom path to a MMDB file
|
||
from IPinfo, MaxMind, or DBIP
|
||
- `ipinfo_url` - str: Overrides the default download URL for the
|
||
bundled IPinfo Lite MMDB (env var:
|
||
`PARSEDMARC_GENERAL_IPINFO_URL`). The pre-9.10 name `ip_db_url` is
|
||
still accepted as a deprecated alias and logs a warning.
|
||
- `ipinfo_api_token` - str: Optional [IPinfo Lite REST API] token. When
|
||
set, IP lookups hit the API first for the freshest country/ASN data
|
||
and fall back to the local MMDB on rate limit, quota exhaustion, or
|
||
network errors. An invalid token exits the process with a fatal error.
|
||
Ignored when `offline` is set. The Lite tier is free and has no
|
||
documented monthly request cap; see the IPinfo Lite docs for current
|
||
limits. (env var: `PARSEDMARC_GENERAL_IPINFO_API_TOKEN`)
|
||
- `offline` - bool: Do not use online queries for geolocation
|
||
or DNS. Also disables automatic downloading of the IP-to-country
|
||
database and reverse DNS map.
|
||
- `always_use_local_files` - bool: Disables the download of the
|
||
IP-to-country database and reverse DNS map
|
||
- `local_reverse_dns_map_path` - str: Overrides the default local file path to use for the reverse DNS map
|
||
- `reverse_dns_map_url` - str: Overrides the default download URL for the reverse DNS map
|
||
- `local_psl_overrides_path` - str: Overrides the default local file path to use for the PSL overrides list
|
||
- `psl_overrides_url` - str: Overrides the default download URL for the PSL overrides list
|
||
- `nameservers` - str: A comma separated list of
|
||
DNS resolvers (Default: `[Cloudflare's public resolvers]`). Each entry
|
||
is an IP address (DNS over UDP/TCP port 53), an `https://` URL
|
||
(DNS over HTTPS), or `tls://ip[:port][#hostname]` (DNS over TLS) —
|
||
see [Encrypted DNS](#encrypted-dns)
|
||
- `dns_test_address` - str: A dummy address used for the DNS pre-flight
|
||
check that runs when `nameservers` is set (Default: `1.1.1.1`)
|
||
- `dns_timeout` - float: DNS timeout period in seconds (Default: `2.0`)
|
||
- `dns_retries` - int: Number of times to retry a DNS query after a
|
||
timeout or other transient error (Default: `0`)
|
||
- `debug` - bool: Print debugging messages
|
||
- `verbose` - bool: More verbose output
|
||
- `warnings` - bool: Print warnings in addition to errors
|
||
- `fail_on_output_error` - bool: Exit with a non-zero status code if
|
||
any configured output destination fails while saving/publishing
|
||
reports (Default: `False`)
|
||
|
||
:::{note}
|
||
This option only controls the process exit code. Retaining mailbox
|
||
messages whose reports could not be saved is automatic and happens
|
||
either way — see
|
||
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved).
|
||
:::
|
||
|
||
- `log_file` - str: Write log messages to a file at this path
|
||
- `n_procs` - int: Number of processes to run in parallel when
|
||
parsing report files passed directly as CLI arguments, messages
|
||
in mbox files, and messages from mailbox connections (IMAP,
|
||
Microsoft Graph, Gmail API, Maildir), including watch mode
|
||
(Default: `1`)
|
||
|
||
:::{note}
|
||
Setting this to a number larger than one can improve
|
||
performance when processing thousands of files or messages
|
||
:::
|
||
|
||
:::{note}
|
||
Only parsing is parallelized across worker processes. Fetching
|
||
messages, deduplicating reports, archiving/deleting mailbox
|
||
messages, and saving/publishing to outputs all stay sequential
|
||
in the main process. Each worker process keeps its own DNS/GeoIP
|
||
cache.
|
||
:::
|
||
|
||
- `parse_timeout` - float: Stop parsing a report file or message
|
||
after this many seconds, and treat it as invalid: a report file
|
||
is logged as failed (and moved to `Invalid` under
|
||
`archive_directory`), an mbox message is skipped with a warning,
|
||
and a mailbox message is moved to the `Invalid` archive folder
|
||
(or deleted with `delete_invalid`). `0` means no limit
|
||
(Default: `0`, suggested: `300`)
|
||
|
||
:::{note}
|
||
With a timeout set, parsing always runs in worker processes, even
|
||
with `n_procs = 1`, because a parse can only be stopped by ending
|
||
the process that runs it. When a parse times out, every worker
|
||
process is stopped and new ones are started, and the other
|
||
messages or files that were still being parsed start again from
|
||
the beginning. parsedmarc checks the time only while it waits for
|
||
a parse result, so a parse is stopped within one second of
|
||
parsedmarc next waiting for a result after its time is up; while
|
||
parsedmarc is busy between results (moving messages, for example)
|
||
a parse can run on past the timeout. Choose a value well above the
|
||
time your largest legitimate reports take, including DNS lookups,
|
||
each of which can take up to `dns_timeout` seconds per try. With a
|
||
timeout set, the worker processes hand each parsed report back
|
||
through a file in a directory readable only by the parsedmarc user,
|
||
inside the temporary directory (`TMPDIR`), which must therefore be
|
||
writable; the files are removed as they are read, and their
|
||
directory once the batch has been parsed or the workers stopped.
|
||
:::
|
||
|
||
- `mailbox`
|
||
- `reports_folder` - str: The mailbox folder (or label for
|
||
Gmail) where the incoming reports can be found
|
||
(Default: `INBOX`)
|
||
- `archive_folder` - str: The mailbox folder (or label for
|
||
Gmail) to sort processed emails into (Default: `Archive`)
|
||
- `watch` - bool: Process new messages as they arrive, via the
|
||
IMAP `IDLE` command or by polling the other mailbox types
|
||
(Microsoft Graph, Gmail API, Maildir)
|
||
- `delete` - bool: Delete messages after processing them,
|
||
instead of archiving them
|
||
- `delete_aggregate` - bool: Delete aggregate report messages
|
||
after processing them, instead of archiving them
|
||
(Default: the value of `delete`)
|
||
- `delete_failure` - bool: Delete failure report messages
|
||
after processing them, instead of archiving them
|
||
(Default: the value of `delete`)
|
||
- `delete_smtp_tls` - bool: Delete SMTP TLS report messages
|
||
after processing them, instead of archiving them
|
||
(Default: the value of `delete`)
|
||
- `delete_invalid` - bool: Delete messages that could not be
|
||
parsed, instead of archiving them in the `Invalid`
|
||
subfolder, where they can be inspected for debugging
|
||
(Default: the value of `delete`)
|
||
|
||
:::{note}
|
||
Each of these four options overrides `delete` for one kind of
|
||
message only, and the other three keep inheriting `delete`. So
|
||
`delete = True` combined with `delete_failure = False` archives
|
||
failure report messages while deleting processed aggregate and
|
||
SMTP TLS report messages — and unparseable ones, unless
|
||
`delete_invalid = False` is set as well.
|
||
:::
|
||
|
||
- `test` - bool: Do not move or delete messages
|
||
- `batch_size` - int: Number of messages to read and process
|
||
before saving. Default `10`. Use `0` for no limit.
|
||
- `check_timeout` - int: Number of seconds to wait for an IMAP
|
||
IDLE response or the number of seconds until the next
|
||
mail check (Default: `30`)
|
||
- `max_unsaved_retries` - int: How many times a batch of messages
|
||
whose reports could not be saved is retried before its messages
|
||
are moved to the `Unsaved` archive subfolder instead of being
|
||
retried again (Default: `2`, i.e. the initial attempt plus two
|
||
retries). Use `0` to move messages on the first failed save;
|
||
negative values are rejected.
|
||
Failures are counted in memory, so the cap applies across watch-mode
|
||
checks within one long-running process, not across separate one-shot
|
||
runs. See
|
||
[Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
|
||
below.
|
||
- `attempts_file` - str: Path to a JSON file in which parsedmarc
|
||
counts the messages whose parsing never finished because the
|
||
process died, so that a message that keeps killing parsedmarc is
|
||
moved to the `Invalid` archive subfolder (or deleted, per
|
||
`delete_invalid`) after two such attempts instead of being parsed
|
||
on every run. The directory must be writable by the user running
|
||
parsedmarc; the file is created on first use. Unset by default
|
||
(no record is kept). See
|
||
[Messages that crash parsedmarc are not retried forever](#messages-that-crash-parsedmarc-are-not-retried-forever)
|
||
below.
|
||
- `since` - str: Search for messages since a certain time. (Examples: `5m|3h|2d|1w`)
|
||
Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
|
||
Defaults to `1d` if an incorrect value is provided.
|
||
- `imap`
|
||
- `host` - str: The IMAP server hostname or IP address
|
||
- `port` - int: The IMAP server port (Default: `993`)
|
||
|
||
:::{note}
|
||
`%` characters must be escaped with another `%` character,
|
||
so use `%%` wherever a `%` character is used.
|
||
:::
|
||
|
||
:::{note}
|
||
Starting in version 8.0.0, most options from the `imap`
|
||
section have been moved to the `mailbox` section.
|
||
:::
|
||
|
||
:::{note}
|
||
If your host recommends another port, still try 993
|
||
:::
|
||
|
||
- `ssl` - bool: Use an encrypted SSL/TLS connection from the start
|
||
(implicit TLS, usually port 993) (Default: `True`). When `False`,
|
||
the connection starts in plain text and must be upgraded with
|
||
STARTTLS before logging in (usually port 143); see `starttls`.
|
||
- `starttls` - bool: Whether to upgrade a plain-text (`ssl = False`)
|
||
connection with STARTTLS. When unset (the default), the connection
|
||
is upgraded if the server advertises STARTTLS, and parsedmarc
|
||
refuses to log in if it does not, so the credentials are never sent
|
||
in plain text. `true` always requires the upgrade. `false` never
|
||
upgrades, even when the server advertises STARTTLS, so the
|
||
credentials and messages are sent in plain text; use it only for a
|
||
server on the same host, such as [DavMail](davmail.md) on the
|
||
loopback address. Ignored when `ssl = True`.
|
||
- `skip_certificate_verification` - bool: Skip certificate
|
||
verification (not recommended)
|
||
- `timeout` - int: Number of seconds to wait for an IMAP operation
|
||
(Default: `30`)
|
||
- `max_retries` - int: Maximum number of retries after an IMAP
|
||
timeout (Default: `4`)
|
||
- `user` - str: The IMAP user
|
||
- `password` - str: The IMAP password
|
||
- `msgraph`
|
||
|
||
Requires the `msgraph` extra: `pip install "parsedmarc[msgraph]"`
|
||
|
||
- `auth_method` - str: Authentication method, valid types are
|
||
`UsernamePassword`, `DeviceCode`, `ClientSecret`, `Certificate`, or
|
||
`ClientAssertion` (Default: `UsernamePassword`).
|
||
- `user` - str: The M365 user, required when the auth method is
|
||
UsernamePassword
|
||
- `password` - str: The user password, required when the auth
|
||
method is UsernamePassword
|
||
- `client_id` - str: The app registration's client ID
|
||
- `client_secret` - str: The app registration's secret. Required when
|
||
the auth method is `UsernamePassword` or `ClientSecret`
|
||
- `certificate_path` - str: Path to a PEM or PKCS12 certificate
|
||
including the private key. Required when the auth method is
|
||
`Certificate`
|
||
- `certificate_password` - str: Optional password for the
|
||
certificate file when using `Certificate` auth
|
||
- `client_assertion` - str: A signed JWT client assertion, required
|
||
when the auth method is `ClientAssertion`
|
||
|
||
:::{note}
|
||
`client_assertion` is a static value, so it is only usable for as
|
||
long as the JWT it contains remains unexpired (assertions are
|
||
typically short-lived, on the order of minutes). It is best suited
|
||
to short one-shot runs; for long-running `watch` mode, prefer
|
||
`Certificate` or `ClientSecret` auth, which can refresh
|
||
indefinitely.
|
||
:::
|
||
- `tenant_id` - str: The Azure AD tenant ID. This is required
|
||
for all auth methods except UsernamePassword.
|
||
- `mailbox` - str: The mailbox name. This defaults to the
|
||
current user if using the UsernamePassword auth method, but
|
||
could be a shared mailbox if the user has access to the mailbox
|
||
- `graph_url` - str: Microsoft Graph URL. Allows for use of national
|
||
clouds (e.g. Azure Gov)
|
||
(Default: `https://graph.microsoft.com`)
|
||
|
||
:::{warning}
|
||
Setting `graph_url` alone is **not** sufficient for a national/sovereign
|
||
cloud tenant. It only changes the Microsoft Graph API root; the
|
||
Microsoft Entra ID (Azure AD) token endpoint used for authentication is
|
||
not currently configurable in parsedmarc or `mailsuite`, and always
|
||
defaults to the global `https://login.microsoftonline.com`. A true
|
||
national-cloud deployment also needs its own Entra ID endpoint, so
|
||
`graph_url` by itself only helps if your tenant is registered in the
|
||
global cloud but you specifically need to reach one of these Graph API
|
||
roots (per [Microsoft's national cloud deployment docs][ms-graph-clouds]):
|
||
|
||
| National cloud | Microsoft Graph URL | Entra ID endpoint (not configurable here) |
|
||
|---|---|---|
|
||
| Global (default) | `https://graph.microsoft.com` | `https://login.microsoftonline.com` |
|
||
| US Government L4 (GCC High) | `https://graph.microsoft.us` | `https://login.microsoftonline.us` |
|
||
| US Government L5 (DoD) | `https://dod-graph.microsoft.us` | `https://login.microsoftonline.us` |
|
||
| China, operated by 21Vianet | `https://microsoftgraph.chinacloudapi.cn` | `https://login.chinacloudapi.cn` |
|
||
|
||
[ms-graph-clouds]: https://learn.microsoft.com/en-us/graph/deployments
|
||
:::
|
||
- `token_file` - str: Path to save the token file
|
||
(Default: `.token`)
|
||
- `allow_unencrypted_storage` - bool: Allows the Azure Identity
|
||
module to fall back to unencrypted token cache (Default: `False`).
|
||
Even if enabled, the cache will always try encrypted storage first.
|
||
|
||
:::{note}
|
||
`token_file` stores the serialized authentication record; the
|
||
underlying MSAL persistent token cache is separately named
|
||
`parsedmarc`, not `mailsuite`'s own default cache name. This is
|
||
deliberate: the Graph mailbox backend used to live directly in
|
||
parsedmarc, and moved into the `mailsuite` dependency in parsedmarc
|
||
9.11.0. Explicitly keeping the `parsedmarc` cache name means tokens
|
||
cached before that move keep working after upgrading — there is
|
||
nothing to migrate and no action needed on your part.
|
||
:::
|
||
|
||
:::{note}
|
||
You must create an app registration in Azure AD and have an
|
||
admin grant the Microsoft Graph `Mail.ReadWrite`
|
||
(delegated) permission to the app. If you are using
|
||
`UsernamePassword` auth and the mailbox is different from the
|
||
username, you must grant the app `Mail.ReadWrite.Shared`.
|
||
:::
|
||
|
||
| Auth method | Reading (own mailbox) | Reading (shared mailbox) |
|
||
|---|---|---|
|
||
| `UsernamePassword` / `DeviceCode` (delegated) | `Mail.ReadWrite` | `Mail.ReadWrite.Shared` |
|
||
| `ClientSecret` / `Certificate` / `ClientAssertion` (application) | `Mail.ReadWrite` (application), scoped via `New-ApplicationAccessPolicy` | same as own mailbox — app-only access is scoped by policy, not by permission name |
|
||
|
||
:::{tip}
|
||
**Troubleshooting connections.** Run with `--verbose` to log a
|
||
redacted connection summary (auth method, tenant, client ID,
|
||
mailbox, Graph URL) before the connection attempt, and with
|
||
`--debug` to additionally surface the underlying library activity —
|
||
`azure.identity` token acquisition (including `AADSTS` error codes
|
||
from Entra ID, which distinguish a local configuration problem from
|
||
an Exchange Online-side one), Microsoft Graph SDK requests, and
|
||
`httpx` HTTP request lines. Secret values (passwords, client
|
||
secrets, certificate passwords) are never written to logs.
|
||
Connection, mailbox fetch, message send, and `--watch` failures
|
||
each log a single ERROR line naming the mailbox, tenant, auth
|
||
method, and the Graph `request-id`/`client-request-id` when
|
||
available — worth quoting verbatim when contacting Microsoft
|
||
support.
|
||
:::
|
||
|
||
:::{warning}
|
||
If you are using the `ClientSecret` auth method, you need to
|
||
grant the `Mail.ReadWrite` (application) permission to the
|
||
app. You must also restrict the application's access to a
|
||
specific mailbox since it allows all mailboxes by default.
|
||
Use the `New-ApplicationAccessPolicy` command in the
|
||
Exchange PowerShell module. If you need to scope the policy to
|
||
shared mailboxes, you can add them to a mail enabled security
|
||
group and use that as the group id.
|
||
|
||
```powershell
|
||
New-ApplicationAccessPolicy -AccessRight RestrictAccess
|
||
-AppId "<CLIENT_ID>" -PolicyScopeGroupId "<MAILBOX>"
|
||
-Description "Restrict access to dmarc reports mailbox."
|
||
```
|
||
|
||
The same application permission and mailbox scoping guidance
|
||
applies to the `Certificate` and `ClientAssertion` auth methods.
|
||
|
||
:::
|
||
|
||
**Sending the summary email via Microsoft Graph.**
|
||
When `[msgraph]` is configured and `[smtp]` has a `to` value but no
|
||
`host`, the periodic summary email is sent through the same
|
||
already-authenticated Graph mailbox connection used for reading
|
||
(`/users/{mailbox}/sendMail`), and a copy is saved to Sent Items.
|
||
When `[smtp] host` is set, SMTP is used regardless of whether
|
||
`[msgraph]` is also configured — SMTP is always preferred, with no
|
||
automatic fallback to Graph on SMTP failure.
|
||
|
||
Example config combining `[msgraph]` with a `[smtp]` section that
|
||
only sets `to`/`subject` (no `host`):
|
||
|
||
```ini
|
||
[msgraph]
|
||
auth_method = Certificate
|
||
client_id = ...
|
||
tenant_id = ...
|
||
mailbox = dmarc-reports@example.com
|
||
certificate_path = /path/to/cert.pem
|
||
|
||
[smtp]
|
||
to = admin@example.com
|
||
subject = DMARC Summary
|
||
```
|
||
|
||
Required Microsoft Graph permissions, in addition to the
|
||
reading-related permissions documented above:
|
||
|
||
| Auth method | Reading | Sending (own mailbox) | Sending (shared mailbox) |
|
||
|---|---|---|---|
|
||
| `UsernamePassword` / `DeviceCode` (delegated) | `Mail.ReadWrite` (+`.Shared` for shared) | `Mail.Send` | `Mail.Send.Shared` |
|
||
| `ClientSecret` / `Certificate` / `ClientAssertion` (application) | `Mail.ReadWrite` (application) | `Mail.Send` (application) | `Mail.Send` (application), scoped via `New-ApplicationAccessPolicy` |
|
||
|
||
:::{warning}
|
||
Graph-based sending is only confirmed to work with the app-only
|
||
auth methods (`ClientSecret`, `Certificate`, `ClientAssertion`).
|
||
The delegated auth methods (`UsernamePassword`, `DeviceCode`)
|
||
currently request only the `Mail.ReadWrite`(`.Shared`) scope when
|
||
authenticating, not `Mail.Send` — so a delegated connection's
|
||
access token will not carry `Mail.Send` even if an administrator
|
||
has granted it, and `/sendMail` calls are expected to fail with an
|
||
access-denied error regardless of what's granted in Azure AD. Use
|
||
an app-only auth method if you need Graph-based sending.
|
||
:::
|
||
|
||
**Minimal example configs.** Each auth method needs a different
|
||
minimum set of keys. These read-only examples omit `[smtp]`; see
|
||
above for adding Graph-based sending on top of any of them.
|
||
|
||
`UsernamePassword` (delegated, own mailbox):
|
||
```ini
|
||
[msgraph]
|
||
auth_method = UsernamePassword
|
||
client_id = ...
|
||
client_secret = ...
|
||
user = dmarc-reports@example.com
|
||
password = ...
|
||
```
|
||
|
||
`DeviceCode` (delegated, interactive sign-in on first run — `user`
|
||
is the account that signs in; `mailbox` is the shared mailbox it
|
||
reads, and only needs to differ from `user` to request
|
||
`Mail.ReadWrite.Shared` instead of plain `Mail.ReadWrite`):
|
||
```ini
|
||
[msgraph]
|
||
auth_method = DeviceCode
|
||
client_id = ...
|
||
tenant_id = ...
|
||
user = signing-in-user@example.com
|
||
mailbox = dmarc-reports@example.com
|
||
```
|
||
|
||
`ClientSecret` (app-only):
|
||
```ini
|
||
[msgraph]
|
||
auth_method = ClientSecret
|
||
client_id = ...
|
||
tenant_id = ...
|
||
client_secret = ...
|
||
mailbox = dmarc-reports@example.com
|
||
```
|
||
|
||
`Certificate` (app-only):
|
||
```ini
|
||
[msgraph]
|
||
auth_method = Certificate
|
||
client_id = ...
|
||
tenant_id = ...
|
||
certificate_path = /path/to/cert.pem
|
||
mailbox = dmarc-reports@example.com
|
||
```
|
||
|
||
`ClientAssertion` (app-only, short-lived JWT — see the note above
|
||
about its unsuitability for `watch` mode):
|
||
```ini
|
||
[msgraph]
|
||
auth_method = ClientAssertion
|
||
client_id = ...
|
||
tenant_id = ...
|
||
client_assertion = ...
|
||
mailbox = dmarc-reports@example.com
|
||
```
|
||
|
||
:::{tip}
|
||
**Troubleshooting.**
|
||
|
||
| Error | Cause | Fix |
|
||
|---|---|---|
|
||
| *"...needs permission to access resources in your organization that only an admin can grant"* / "Admin consent required" | A delegated auth method (`UsernamePassword`, `DeviceCode`) is authenticating with a scope (`Mail.ReadWrite` or `Mail.ReadWrite.Shared`) the tenant admin hasn't consented to yet. | Have an Entra ID admin grant consent: Azure Portal → **Enterprise Applications** → *your app* → **Permissions** → **Grant admin consent**, or `az ad app permission admin-consent --id <CLIENT_ID>`. This is separate from the `New-ApplicationAccessPolicy` step above, which only applies to app-only auth. |
|
||
| `ErrorItemNotFound: ... Default folder Root not found` | `mailsuite` can resolve the well-known folders (`Inbox`, `Archive`, `Drafts`, `Sent Items`, `Deleted Items`, `Junk Email`) even when a mailbox's folder hierarchy hasn't fully provisioned, but a **custom, non-well-known** `reports_folder` name still fails to resolve on such a mailbox. | Point `reports_folder` at (or under) one of the six well-known folder names above, or sign into the (shared) mailbox once via Outlook/OWA to force Exchange to provision it, then retry. |
|
||
| `RuntimeError: Event loop is closed` | Historical bug, fixed in `mailsuite` 2.0.2. Not reachable with the `mailsuite>=2.2.2` this project requires. | Confirm your installed `mailsuite` version is current (`pip show mailsuite`); upgrade if it's somehow pinned below 2.0.2. |
|
||
| Invalid/rejected timestamp in the `since`/`receivedDateTime` filter | Historical bug (parsedmarc [#706](https://github.com/domainaware/parsedmarc/pull/706)/[#708](https://github.com/domainaware/parsedmarc/pull/708)): older versions appended a spurious `Z` to an already-UTC-offset ISO timestamp. Fixed since parsedmarc 9.5.1/9.5.5. | Upgrade parsedmarc if you're on a version older than 9.5.5. |
|
||
:::
|
||
- `elasticsearch`
|
||
|
||
Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
|
||
|
||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||
Do not put credentials in the URL; use `user` and `password`
|
||
(or `api_key`) instead.
|
||
- `user` - str: Basic auth username
|
||
- `password` - str: Basic auth password
|
||
- `api_key` - str: API key
|
||
- `ssl` - bool: Use an encrypted SSL/TLS connection
|
||
(Default: `True`)
|
||
- `timeout` - float: Timeout in seconds (Default: 60)
|
||
- `cert_path` - str: Path to a trusted CA certificates file
|
||
- `skip_certificate_verification` - bool: Skip certificate
|
||
verification (not recommended)
|
||
- `index_suffix` - str: A suffix to apply to the index names
|
||
- `index_prefix` - str: A prefix to apply to the index names
|
||
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
|
||
(ignored, with a warning, when `use_data_streams` is `True`)
|
||
- `number_of_shards` - int: The number of shards to use when
|
||
creating the index, or each backing index of a data stream
|
||
(Default: `1`)
|
||
- `number_of_replicas` - int: The number of replicas to use when
|
||
creating the index, or each backing index of a data stream
|
||
(Default: `0`)
|
||
- `use_data_streams` - bool: Save each report type to a data stream
|
||
named like the index without the date suffix (e.g. `dmarc_aggregate`)
|
||
instead of to daily or monthly indexes. parsedmarc creates the stream
|
||
and its index template; retention is left to an ILM policy you attach. Requires Elasticsearch
|
||
8.7 or later.
|
||
See [Data streams and records retention](elasticsearch.md#data-streams-and-records-retention)
|
||
(Default: `False`)
|
||
- `serverless` - bool: Set to `True` when targeting an Elastic Cloud
|
||
Serverless project. Serverless manages sharding and replication itself
|
||
and rejects the `number_of_shards` / `number_of_replicas` index settings
|
||
with HTTP 400. With this flag set, parsedmarc strips those keys from the
|
||
settings sent at index creation (or in a data stream's index template);
|
||
any other settings (e.g. `refresh_interval`) are passed through
|
||
unchanged (Default: `False`)
|
||
- `opensearch`
|
||
|
||
Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
|
||
|
||
- `hosts` - str: A comma-separated list of hostnames and ports
|
||
or URLs (e.g. `127.0.0.1:9200` or `https://localhost:9200`).
|
||
Do not put credentials in the URL; use `user` and `password`
|
||
(or `api_key`) instead.
|
||
- `user` - str: Basic auth username
|
||
- `password` - str: Basic auth password
|
||
- `api_key` - str: API key
|
||
- `auth_type` - str: Authentication type: `basic` (default) or `awssigv4` (the key `authentication_type` is accepted as an alias for this option)
|
||
- `aws_region` - str: AWS region for SigV4 authentication
|
||
(required when `auth_type = awssigv4`)
|
||
- `aws_service` - str: AWS service for SigV4 signing (Default: `es`)
|
||
- `ssl` - bool: Use an encrypted SSL/TLS connection
|
||
(Default: `True`)
|
||
- `timeout` - float: Timeout in seconds (Default: 60)
|
||
- `cert_path` - str: Path to a trusted CA certificates file
|
||
- `skip_certificate_verification` - bool: Skip certificate
|
||
verification (not recommended)
|
||
- `index_suffix` - str: A suffix to apply to the index names
|
||
- `index_prefix` - str: A prefix to apply to the index names
|
||
- `monthly_indexes` - bool: Use monthly indexes instead of daily indexes
|
||
(ignored, with a warning, when `use_data_streams` is `True`)
|
||
- `number_of_shards` - int: The number of shards to use when
|
||
creating the index, or each backing index of a data stream
|
||
(Default: `1`)
|
||
- `number_of_replicas` - int: The number of replicas to use when
|
||
creating the index, or each backing index of a data stream
|
||
(Default: `0`)
|
||
- `use_data_streams` - bool: Save each report type to a data stream
|
||
named like the index without the date suffix (e.g. `dmarc_aggregate`)
|
||
instead of to daily or monthly indexes. parsedmarc creates the stream
|
||
and its index template; retention is left to an ISM policy you attach.
|
||
OpenSearch rejects data stream names that start with `.`, so an
|
||
`index_prefix` starting with `.` cannot be used with this option.
|
||
See [Data streams and records retention](opensearch.md#data-streams-and-records-retention)
|
||
(Default: `False`)
|
||
- `splunk_hec`
|
||
- `url` - str: The URL of the Splunk HTTP Events Collector (HEC).
|
||
Do not put credentials in the URL; use `token` instead.
|
||
- `token` - str: The HEC token
|
||
- `index` - str: The Splunk index to use
|
||
- `skip_certificate_verification` - bool: Skip certificate
|
||
verification (not recommended)
|
||
- `kafka`
|
||
|
||
Requires the `kafka` extra: `pip install "parsedmarc[kafka]"`
|
||
|
||
- `hosts` - str: A comma separated list of Kafka hosts
|
||
- `user` - str: The Kafka user
|
||
- `password` - str: The Kafka password
|
||
- `ssl` - bool: Use an encrypted SSL/TLS connection (Default: `True`)
|
||
- `skip_certificate_verification` - bool: Skip certificate
|
||
verification (not recommended)
|
||
- `aggregate_topic` - str: The Kafka topic for aggregate reports
|
||
- `failure_topic` - str: The Kafka topic for failure reports
|
||
- `smtp_tls_topic` - str: The Kafka topic for SMTP TLS reports
|
||
- `smtp`
|
||
|
||
The results email is only sent when at least one aggregate, failure,
|
||
or SMTP TLS report was parsed during the run; an empty run (e.g. an
|
||
empty inbox) skips the email instead of sending headers-only CSVs.
|
||
|
||
- `host` - str: The SMTP hostname. Required unless `[msgraph]` is
|
||
configured, in which case omitting it sends the summary via
|
||
Microsoft Graph instead — see "Sending the summary email via
|
||
Microsoft Graph" above.
|
||
- `port` - int: The SMTP port (Default: `25`). Usually `587` for
|
||
STARTTLS, or `465` with `ssl = True`.
|
||
- `ssl` - bool: Use an encrypted SSL/TLS connection from the start
|
||
(implicit TLS, usually port 465) (Default: `False`). When `False`,
|
||
the connection starts in plain text and must be upgraded with
|
||
STARTTLS before authenticating; see `starttls`.
|
||
- `starttls` - bool: Whether to upgrade a plain-text (`ssl = False`)
|
||
connection with STARTTLS. When unset (the default), the connection
|
||
is upgraded if the server advertises STARTTLS, and parsedmarc
|
||
refuses to authenticate if it does not, so the credentials are never
|
||
sent in plain text ("Failed to email results" is logged). `true`
|
||
always requires the upgrade. `false` never upgrades, even when the
|
||
server advertises STARTTLS, so the credentials and the results
|
||
email are sent in plain text; use it only for a server on the same
|
||
host. Ignored when `ssl = True`.
|
||
- `skip_certificate_verification` - bool: Skip certificate
|
||
verification (not recommended)
|
||
- `user` - str: the SMTP username. SMTP-only; not used when sending
|
||
via Microsoft Graph.
|
||
- `password` - str: the SMTP password. SMTP-only; not used when
|
||
sending via Microsoft Graph.
|
||
- `from` - str: The From header to use in the email. SMTP-only.
|
||
When sent via Microsoft Graph, the message's `From` is always the
|
||
`[msgraph]` mailbox — `[smtp] from` has no effect.
|
||
- `to` - list: A list of email addresses to send to
|
||
- `subject` - str: The Subject header to use in the email
|
||
(Default: `parsedmarc report`)
|
||
- `attachment` - str: The ZIP attachment filename
|
||
(Default: `DMARC-<YYYY-MM-DD>.zip`)
|
||
- `message` - str: The email message
|
||
(Default: `Please see the attached DMARC results.`)
|
||
|
||
:::{note}
|
||
`%` characters must be escaped with another `%` character,
|
||
so use `%%` wherever a `%` character is used.
|
||
:::
|
||
- `postgresql`
|
||
- `host` - str: The PostgreSQL server hostname or IP address.
|
||
Required unless `connection_string` is provided.
|
||
- `port` - int: The PostgreSQL server port (Default: `5432`)
|
||
- `user` - str: The database user name (Optional)
|
||
- `password` - str: The database user password (Optional)
|
||
- `database` - str: The database name (Optional)
|
||
- `connection_string` - str: A full libpq connection string or URI
|
||
(e.g. `postgresql://user@host/dbname`). When provided,
|
||
all individual parameters above are ignored. Prefer the separate
|
||
`host`, `port`, `user`, `password`, and `database` settings and do
|
||
not put the password in the connection string: a malformed
|
||
connection string can make the driver echo it back in an error
|
||
message. parsedmarc replaces such errors with a fixed message in
|
||
its log, but cannot stop the driver from producing them.
|
||
|
||
The PostgreSQL backend is an optional extra. Install it with
|
||
`pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
|
||
prebuilt binary wheels are not available for every platform, which is
|
||
why it is the one extra that `parsedmarc[all]` deliberately leaves out
|
||
— combine the two with `pip install "parsedmarc[all,postgresql]"`. The
|
||
prebuilt Docker image (`ghcr.io/domainaware/parsedmarc`) already
|
||
bundles `[all,postgresql]`, so the PostgreSQL backend works out of the
|
||
box in the container — `psycopg` ships `amd64` and `arm64` binary
|
||
wheels, both of which the image supports.
|
||
|
||
Tables are created automatically on first run using
|
||
`CREATE TABLE IF NOT EXISTS`, so no manual schema migration is needed
|
||
for fresh installations.
|
||
|
||
**Example configuration:**
|
||
|
||
```ini
|
||
[postgresql]
|
||
host = localhost
|
||
port = 5432
|
||
user = parsedmarc
|
||
password = secret
|
||
database = parsedmarc
|
||
```
|
||
|
||
Or using a DSN/URI, with the password supplied separately (for
|
||
example through the `PGPASSWORD` environment variable or a
|
||
`~/.pgpass` file, both of which libpq reads):
|
||
|
||
```ini
|
||
[postgresql]
|
||
connection_string = postgresql://parsedmarc@localhost/parsedmarc
|
||
```
|
||
|
||
Saving parsed data to PostgreSQL is controlled by the `[general]`
|
||
options `save_aggregate`, `save_failure`, and `save_smtp_tls`
|
||
(`save_forensic` is still accepted as a deprecated alias for
|
||
`save_failure`). These flags must be set to `True` for the
|
||
corresponding report types (aggregate DMARC, failure DMARC, and
|
||
SMTP TLS reports) or no data will be written to PostgreSQL, even if
|
||
this section is configured.
|
||
|
||
- `s3`
|
||
|
||
Requires the `s3` extra: `pip install "parsedmarc[s3]"`
|
||
|
||
- `bucket` - str: The S3 bucket name
|
||
- `path` - str: The path to upload reports to (Default: `/`)
|
||
- `region_name` - str: The region name (Optional)
|
||
- `endpoint_url` - str: The endpoint URL (Optional)
|
||
- `access_key_id` - str: The access key id (Optional)
|
||
- `secret_access_key` - str: The secret access key (Optional)
|
||
|
||
Each report is saved as one JSON object, with this key:
|
||
`<path>/<report type>/year=YYYY/month=MM/day=DD/<report ID>_<hash>.json`.
|
||
The date is the report's begin date. Failure reports have no report ID
|
||
or begin date, so they use the reported domain and the arrival date
|
||
instead. Aggregate report dates are in the host's local time (SMTP TLS
|
||
dates are as the report gives them, and failure arrival dates are in
|
||
UTC), so for aggregate reports the key's day and the hash depend on the
|
||
host's time zone.
|
||
|
||
The report ID comes from the sender, so it is percent-encoded before it
|
||
is used in the key. `/`, the characters AWS lists as "characters to
|
||
avoid" (`` \ { } ^ [ ] " < > ~ # | ` `` and `%`), control characters,
|
||
and non-ASCII characters are encoded (for example, `/` becomes `%2F`),
|
||
so the ID is always a single key segment, and `%` appears in it only as
|
||
the escape character. If the key would be longer than the 1,024-byte S3
|
||
limit, the report ID is shortened.
|
||
|
||
The hash is the first 16 hex characters of a SHA-256 hash of the report
|
||
type and the fields that identify the report:
|
||
|
||
- Aggregate: the organization name, report ID, policy domain, begin
|
||
date, and end date
|
||
- SMTP TLS: the organization name, report ID, begin date, end date, and
|
||
the domain of each policy
|
||
- Failure: the arrival date, reported domain, original mail from,
|
||
original recipient, original envelope ID, source IP address, and the
|
||
message sample
|
||
|
||
Only fields from the report itself are hashed, not the IP enrichment
|
||
(reverse DNS, country, and so on), so a report that is saved again
|
||
overwrites its own object. Two reports get the same key only when all of
|
||
these fields match, so a report that reuses another report's ID, but
|
||
differs in any of these fields, does not replace it. These fields are
|
||
not secret, though: a forged report whose reporter name, report ID,
|
||
policy domain, and dates all match a real one still replaces it. Turn
|
||
on S3 Versioning on the bucket to keep both.
|
||
|
||
Each object also gets metadata from the report: `org_name`, `org_email`,
|
||
`report_id`, `begin_date`, and `end_date` for aggregate reports;
|
||
`org_name`, `report_id`, `begin_date`, and `end_date` for SMTP TLS
|
||
reports; and `reported_domain` and `arrival_date` for failure reports.
|
||
AWS recommends US-ASCII metadata, and botocore rejects anything else, so
|
||
in each value `%`, control characters, and non-ASCII characters are
|
||
percent-encoded, and the value is cut to at most 256 characters.
|
||
- `syslog`
|
||
- `server` - str: The Syslog server name or IP address
|
||
- `port` - int: The port to use (Default: `514`)
|
||
- `protocol` - str: The protocol to use: `udp`, `tcp`, or `tls` (Default: `udp`)
|
||
- `cafile_path` - str: Path to CA certificate file for TLS server verification (Optional)
|
||
- `certfile_path` - str: Path to client certificate file for TLS authentication (Optional)
|
||
- `keyfile_path` - str: Path to client private key file for TLS authentication (Optional)
|
||
- `timeout` - float: Connection timeout in seconds for TCP/TLS (Default: `5.0`)
|
||
- `retry_attempts` - int: Number of retry attempts for failed connections (Default: `3`)
|
||
- `retry_delay` - int: Delay in seconds between retry attempts (Default: `5`)
|
||
|
||
**Example UDP configuration (default):**
|
||
|
||
```ini
|
||
[syslog]
|
||
server = syslog.example.com
|
||
port = 514
|
||
```
|
||
|
||
**Example TCP configuration:**
|
||
|
||
```ini
|
||
[syslog]
|
||
server = syslog.example.com
|
||
port = 6514
|
||
protocol = tcp
|
||
timeout = 10.0
|
||
retry_attempts = 5
|
||
```
|
||
|
||
**Example TLS configuration with server verification:**
|
||
|
||
```ini
|
||
[syslog]
|
||
server = syslog.example.com
|
||
port = 6514
|
||
protocol = tls
|
||
cafile_path = /path/to/ca-cert.pem
|
||
timeout = 10.0
|
||
```
|
||
|
||
**Example TLS configuration with mutual authentication:**
|
||
|
||
```ini
|
||
[syslog]
|
||
server = syslog.example.com
|
||
port = 6514
|
||
protocol = tls
|
||
cafile_path = /path/to/ca-cert.pem
|
||
certfile_path = /path/to/client-cert.pem
|
||
keyfile_path = /path/to/client-key.pem
|
||
timeout = 10.0
|
||
retry_attempts = 3
|
||
retry_delay = 5
|
||
```
|
||
|
||
- `gmail_api`
|
||
|
||
Requires the `gmail` extra: `pip install "parsedmarc[gmail]"`
|
||
|
||
- `credentials_file` - str: Path to file containing the
|
||
credentials, None to disable (Default: `None`)
|
||
- `token_file` - str: Path to save the token file
|
||
(Default: `.token`)
|
||
- `auth_mode` - str: Authentication mode, `installed_app` (default)
|
||
or `service_account`
|
||
- `service_account_user` - str: Delegated mailbox user for Gmail
|
||
service account auth (required for domain-wide delegation). Also
|
||
accepted as `delegated_user` for backward compatibility.
|
||
|
||
:::{note}
|
||
`credentials_file` and `token_file` can be obtained by following the
|
||
Gmail API [quickstart](https://developers.google.com/gmail/api/quickstart/python).
|
||
Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
|
||
:::
|
||
:::{note}
|
||
When `auth_mode = service_account`, `credentials_file` must point to a
|
||
Google service account key JSON file, and `token_file` is not used.
|
||
:::
|
||
- `include_spam_trash` - bool: Include messages in Spam and
|
||
Trash when searching reports (Default: `False`)
|
||
- `scopes` - str: Comma separated list of scopes to use when
|
||
acquiring credentials
|
||
(Default: `https://www.googleapis.com/auth/gmail.modify`)
|
||
- `oauth2_port` - int: The TCP port for the local server to
|
||
listen on for the OAuth2 response (Default: `8080`)
|
||
- `paginate_messages` - bool: When `True`, fetch all applicable Gmail messages.
|
||
When `False`, only fetch up to 100 new messages per run (Default: `True`)
|
||
- `log_analytics`
|
||
|
||
Requires the `loganalytics` extra: `pip install "parsedmarc[loganalytics]"`
|
||
|
||
- `client_id` - str: The app registration's client ID
|
||
- `client_secret` - str: The app registration's client secret
|
||
- `tenant_id` - str: The tenant id where the app registration resides
|
||
- `dce` - str: The Data Collection Endpoint (DCE). Example: `https://{DCE-NAME}.{REGION}.ingest.monitor.azure.com`.
|
||
- `dcr_immutable_id` - str: The immutable ID of the Data Collection Rule (DCR)
|
||
- `dcr_aggregate_stream` - str: The stream name for aggregate reports in the DCR
|
||
- `dcr_failure_stream` - str: The stream name for the failure reports in the DCR
|
||
- `dcr_smtp_tls_stream` - str: The stream name for the SMTP TLS reports in the DCR
|
||
|
||
:::{note}
|
||
Information regarding the setup of the Data Collection Rule can be found [in the Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal).
|
||
:::
|
||
- `gelf`
|
||
|
||
Requires the `gelf` extra: `pip install "parsedmarc[gelf]"`
|
||
|
||
- `host` - str: The GELF server name or IP address
|
||
- `port` - int: The port to use
|
||
- `mode` - str: The GELF transport type to use. Valid modes: `tcp`, `udp`, `tls`
|
||
|
||
- `maildir`
|
||
- `maildir_path` - str: Full path to the maildir location (the key
|
||
`path` is accepted as an alias). Required to read from a maildir.
|
||
- `maildir_create` - bool: Create the maildir if not present
|
||
(Default: `False`; the key `create` is accepted as an alias)
|
||
|
||
- `webhook` - Post the individual reports to a webhook URL with the report as the JSON body
|
||
- `aggregate_url` - str: URL of the webhook which should receive the aggregate reports
|
||
- `failure_url` - str: URL of the webhook which should receive the failure reports
|
||
- `smtp_tls_url` - str: URL of the webhook which should receive the SMTP TLS reports
|
||
- `timeout` - int: Timeout in seconds for webhook requests (Default: `60`)
|
||
|
||
- `loki` - Push every report record as one JSON log line to the
|
||
[Grafana Loki push API](https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs).
|
||
The lines are the same flat records the syslog and GELF outputs send,
|
||
plus `report_type` (`aggregate`, `failure`, or `smtp_tls`) and `domain`
|
||
(the reported domain: `domain`, `reported_domain`, or `policy_domain`),
|
||
so LogQL's `| json` parser can filter and sum them, e.g. messages failing
|
||
DMARC per domain:
|
||
`sum by (domain) (sum_over_time({job="parsedmarc", report_type="aggregate"} | json dmarc_aligned, count | dmarc_aligned="false" | unwrap count [1d]))`.
|
||
Name the fields to extract: a bare `| json` turns every field into a
|
||
label, and the many distinct values quickly exceed Loki's per-query
|
||
series limit (`max_query_series`, 500 by default).
|
||
A ready-made Grafana dashboard for these lines ships in
|
||
[dashboards/grafana/Grafana-DMARC_Reports-Loki.json](https://github.com/domainaware/parsedmarc/blob/master/dashboards/grafana/Grafana-DMARC_Reports-Loki.json);
|
||
import it and pick your Loki datasource for its `DS_LOKI` input.
|
||
It expects the default `job=parsedmarc` label and `label_fields`.
|
||
- `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
|
||
URL has no path. A URL with a path is used as is.
|
||
Do not put credentials in the URL; use `username` and `password`
|
||
(or `token`) instead.
|
||
- `labels` - str: Comma-separated `name=value` stream labels, merged over
|
||
the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
|
||
(in a container, the FQDN is the container's hostname)
|
||
- `label_fields` - str: Comma-separated record fields used as stream
|
||
labels (Default: `report_type, domain`; empty for none). Keep these to
|
||
fields with few distinct values — every combination is its own Loki
|
||
stream, so fields like `source_ip_address` belong in the line only.
|
||
Records without a value for a field get no such label.
|
||
- `timestamp` - str: `ingest` (the time of the push) or `report_begin`
|
||
(the begin of the reported period, or the arrival date of a failure
|
||
report) (Default: `ingest`). Loki rejects entries older than its
|
||
`reject_old_samples_max_age` (one week by default) and, within a
|
||
stream, entries far older than the newest one, so `report_begin`
|
||
loses late or out-of-order reports unless Loki is configured for them.
|
||
The reported period is in the line either way (`begin_date`,
|
||
`end_date`).
|
||
- `username` - str: Basic authentication user
|
||
- `password` - str: Basic authentication password
|
||
- `token` - str: Bearer token, instead of basic authentication
|
||
- `tenant_id` - str: Tenant ID, sent as the `X-Scope-OrgID` header
|
||
- `skip_certificate_verification` - bool: Skip TLS certificate
|
||
verification (not recommended)
|
||
- `timeout` - float: Timeout in seconds for push requests (Default: `60`)
|
||
- `batch_size` - int: Maximum log lines per push request
|
||
(Default: `1000`)
|
||
|
||
:::{warning}
|
||
It is **strongly recommended** to **not** use the `nameservers`
|
||
setting. By default, `parsedmarc` uses
|
||
[Cloudflare's public resolvers], which are much faster and more
|
||
reliable than Google, Cisco OpenDNS, or even most local resolvers.
|
||
|
||
The `nameservers` option should only be used if your network
|
||
blocks DNS requests to outside resolvers, or blocks plain DNS
|
||
entirely — see [Encrypted DNS](#encrypted-dns).
|
||
:::
|
||
|
||
:::{note}
|
||
`save_aggregate` and `save_failure` are separate options
|
||
because you may not want to save failure reports
|
||
(formerly known as forensic reports) to your Elasticsearch instance,
|
||
particularly if you are in a highly-regulated industry that
|
||
handles sensitive data, such as healthcare or finance. If your
|
||
legitimate outgoing email fails DMARC, it is possible
|
||
that email may appear later in a failure report.
|
||
|
||
Failure reports contain the original headers of an email that
|
||
failed a DMARC check, and sometimes may also include the
|
||
full message body, depending on the policy of the reporting
|
||
organization.
|
||
|
||
Most reporting organizations do not send failure reports of any
|
||
kind for privacy reasons. While aggregate DMARC reports are sent
|
||
at least daily, it is normal to receive very few failure reports.
|
||
|
||
An alternative approach is to still collect failure/ruf
|
||
reports in your DMARC inbox, but run `parsedmarc` with
|
||
```save_failure = True``` manually on a separate IMAP folder (using
|
||
the ```reports_folder``` option), after you have manually moved
|
||
known samples you want to save to that folder
|
||
(e.g. malicious samples and non-sensitive legitimate samples).
|
||
:::
|
||
|
||
:::{warning}
|
||
Elasticsearch 8 changed the limits policy for shards, restricting them
|
||
by default to 1000. parsedmarc uses a shard per analyzed day. If you
|
||
have more than ~3 years of data, you will need to update this
|
||
limit.
|
||
Check current usage (from Management -> Dev Tools -> Console):
|
||
|
||
```text
|
||
GET /_cluster/health?pretty
|
||
{
|
||
...
|
||
"active_primary_shards": 932,
|
||
"active_shards": 932,
|
||
...
|
||
}
|
||
```
|
||
|
||
For example, update the limit to 2000:
|
||
|
||
```text
|
||
PUT _cluster/settings
|
||
{
|
||
"persistent" : {
|
||
"cluster.max_shards_per_node" : 2000
|
||
}
|
||
}
|
||
```
|
||
|
||
Increasing this value increases resource usage.
|
||
:::
|
||
|
||
### Encrypted DNS
|
||
|
||
Every entry in the `nameservers` list picks its own transport:
|
||
|
||
- An IP address — plain DNS over UDP and TCP port 53, the default and the
|
||
behavior of every earlier release.
|
||
- An `https://` URL — DNS over HTTPS (DoH).
|
||
- `tls://ip[:port][#hostname]` — DNS over TLS (DoT). The port defaults to
|
||
853, and the optional `#hostname` names the TLS certificate identity of
|
||
the server (SNI), matching systemd-resolved's syntax. The host itself must
|
||
be an IP address, and an IPv6 address must be wrapped in brackets, so that
|
||
its colons cannot be mistaken for the port separator —
|
||
`tls://[2620:fe::fe]#dns.quad9.net`.
|
||
|
||
Forms can be mixed, and are tried in the order given:
|
||
|
||
```ini
|
||
[general]
|
||
nameservers = https://cloudflare-dns.com/dns-query, tls://9.9.9.9#dns.quad9.net
|
||
```
|
||
|
||
On a network where outbound DNS is blocked but an HTTP proxy is available,
|
||
configure DoH nameservers and set the standard proxy environment variables
|
||
([issue #880](https://github.com/domainaware/parsedmarc/issues/880)):
|
||
|
||
```bash
|
||
export HTTPS_PROXY=http://proxy.example.net:3128
|
||
export NO_PROXY=elasticsearch.example.net,127.0.0.1
|
||
```
|
||
|
||
parsedmarc's DoH queries honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`,
|
||
and the proxy resolves the DoH server's own hostname on parsedmarc's behalf,
|
||
so such a deployment needs no access to UDP port 53 at all.
|
||
|
||
If the proxy inspects TLS, point the standard `SSL_CERT_FILE` environment
|
||
variable at your organization's CA bundle so its certificate is trusted:
|
||
|
||
```bash
|
||
export SSL_CERT_FILE=/etc/ssl/certs/corporate-ca.pem
|
||
```
|
||
|
||
:::{note}
|
||
parsedmarc's DoT connections are made directly to TCP port 853 and do not
|
||
use a proxy. Use DoH on a proxy-only network.
|
||
:::
|
||
|
||
### Mailbox messages are only archived once the reports are saved
|
||
|
||
parsedmarc processes a mailbox in batches of `batch_size` messages. Each
|
||
batch is written to every configured output destination *before* any of
|
||
that batch's messages are archived or deleted. If any destination reports a
|
||
failure — an Elasticsearch outage, an expired Splunk HEC token, an
|
||
unreachable Kafka broker, a full `--output` disk — the whole batch is left
|
||
in the reports folder and retried on the next run or watch-mode check, so a
|
||
report is never removed from the mailbox while it exists nowhere else
|
||
([issue #242](https://github.com/domainaware/parsedmarc/issues/242)).
|
||
|
||
This is all-or-nothing per batch: archiving a batch because most
|
||
destinations accepted it would still permanently lose the data for the one
|
||
that didn't. It also applies regardless of `fail_on_output_error`, which
|
||
only controls the process exit code. Messages that could not be parsed at
|
||
all carry no report data, so they are filed under `Invalid` (or deleted per
|
||
`delete_invalid`) as usual.
|
||
|
||
A destination that is broken rather than briefly unavailable would
|
||
otherwise be retried forever, so retries are capped. Once a message's batch
|
||
has failed `max_unsaved_retries + 1` times (three times by default), that
|
||
message is moved to `<archive_folder>/Unsaved` and stops being retried. **A
|
||
message is never deleted on this path, whatever the `delete` options say.**
|
||
To recover after fixing the output destination, either move the messages
|
||
from `Archive/Unsaved` back into the reports folder, or run parsedmarc once
|
||
with `reports_folder = Archive/Unsaved`.
|
||
|
||
:::{note}
|
||
The failed-save counts live in memory, so they are counted per
|
||
parsedmarc process. In watch mode — a long-running process that checks
|
||
the mailbox repeatedly — the cap works as described across checks. A
|
||
one-shot run (`cron`, `systemd` timers) attempts each message exactly once
|
||
and then exits, so its counts start over every time and the default cap is
|
||
never reached: messages simply keep being retried on every run, which is
|
||
the safe direction. Set `max_unsaved_retries = 0` if you want one-shot
|
||
runs to move unsavable messages to `Unsaved` immediately instead. A
|
||
message that crashes parsedmarc while it is being parsed is a different
|
||
case, counted on disk by `attempts_file`; see
|
||
[Messages that crash parsedmarc are not retried forever](#messages-that-crash-parsedmarc-are-not-retried-forever).
|
||
:::
|
||
|
||
:::{warning}
|
||
Retrying a batch means re-sending it. Output destinations that
|
||
deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize
|
||
an already-saved report — are unaffected, and S3 is idempotent because
|
||
each report is written to an object key built from its type, date, report
|
||
ID, and a hash of the fields that identify the report, so a retry
|
||
overwrites the same object. Kafka,
|
||
Splunk HEC, syslog, GELF, webhooks, Loki, Azure Log Analytics, and the
|
||
`--output` JSON/CSV files all append unconditionally, so each retry adds
|
||
another copy of every report in the batch. That is why the default retry
|
||
cap is deliberately low: at most three deliveries per report before its
|
||
message is set aside in `Unsaved`. The summary email covers everything
|
||
parsed in a run, including reports whose batch failed to save, so a report
|
||
retried across runs can also appear in more than one summary email.
|
||
:::
|
||
|
||
:::{note}
|
||
Not every destination can report a failed delivery. The webhook output
|
||
deliberately logs and swallows its own HTTP and network errors, and the
|
||
syslog and GELF outputs send through Python logging handlers, which
|
||
swallow delivery errors by design — so an unreachable webhook, syslog, or
|
||
GELF endpoint is *not* treated as a failed save and does not hold a
|
||
batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
|
||
Kafka, S3, PostgreSQL, Loki, Azure Log Analytics, and the `--output` files
|
||
are all detected and do.
|
||
:::
|
||
|
||
:::{note}
|
||
`since` interacts with retries: a message that ages out of the configured
|
||
`since` window stops being fetched, and therefore stops being retried
|
||
automatically. It is never deleted or moved — it simply stays in the
|
||
reports folder until it is processed by a run with a wider (or no)
|
||
`since` window.
|
||
:::
|
||
|
||
### Messages that crash parsedmarc are not retried forever
|
||
|
||
A message is only moved out of the reports folder after it has been
|
||
parsed. If parsing a message kills the parsedmarc process — the
|
||
out-of-memory killer, a container memory limit, a crash in a library —
|
||
the message stays where it is, and every later run, or every restart of
|
||
the service, fetches it again and dies on it again, so the reports behind
|
||
it in the mailbox are never processed. parsedmarc rejects the message shapes known to
|
||
do this, but cannot rule out new ones.
|
||
|
||
Set `[mailbox]` `attempts_file` to a path to guard against that loop:
|
||
|
||
```ini
|
||
[mailbox]
|
||
attempts_file = /var/lib/parsedmarc/attempts.json
|
||
```
|
||
|
||
parsedmarc then writes an entry to that file before it parses each
|
||
message, and removes the entry as soon as the parse returns, whether it
|
||
found a report or rejected the message as invalid. An entry that is still
|
||
there on a later run belongs to a message whose parse never finished. Once
|
||
a message has two such unfinished attempts, the next run logs an error
|
||
naming the folder, the message and the count, and moves the message to
|
||
`<archive_folder>/Invalid` without parsing it (or deletes it, if
|
||
`delete_invalid` is set). Its entry is removed once the message has been
|
||
moved, so moving it back into the reports folder tries it again from the
|
||
start; if the move fails, the entry stays and the next run tries the move
|
||
again. A set-aside message whose move failed, or one left in place in
|
||
`test` mode, is not recorded again, so its entry is not refreshed and is
|
||
dropped after 30 days, after which the message is parsed up to twice more
|
||
before it is set aside again. Two attempts are allowed
|
||
so that one crash for an unrelated reason (a reboot, a deployment) never
|
||
sets a message aside. With `n_procs` above 1, or with `parse_timeout` set
|
||
(which parses in worker processes even with `n_procs = 1`), several
|
||
messages are parsed at once, so every message that was being parsed or
|
||
waiting for a worker when a run died has an entry; on the next run each of
|
||
them is parsed on its own after the rest of its batch, so a second crash
|
||
counts against the one message that caused it. It is parsed in the main
|
||
process, or, when `parse_timeout` is set, in a worker process of its own,
|
||
so the timeout still applies to it. If a worker process dies while it
|
||
parses (the out-of-memory killer can end a worker without ending
|
||
parsedmarc), the batch ends with a logged error. A one-shot run saves the
|
||
reports parsed from report files and mbox files before it and exits with
|
||
status 1; for a mailbox, earlier batches are saved and the batch in
|
||
progress stays in the mailbox. In watch mode the IMAP and Maildir watchers
|
||
log it as a warning and check the mailbox again, so the process keeps
|
||
running, while the Microsoft Graph and Gmail watchers exit with status 1.
|
||
Either way the entries of the messages that were being parsed or waiting
|
||
for a worker stay in the file, so the next run or check parses each of
|
||
them on its own in the same way.
|
||
|
||
This count is separate from the in-memory failed-save counts of
|
||
`max_unsaved_retries` above: an entry is removed when the parse
|
||
returns, before the reports are saved, so a message whose batch could not
|
||
be saved is retried and moved to `Unsaved` as described above, never to
|
||
`Invalid`. A message is identified by the reports folder, its id and a
|
||
hash of its beginning, so a new message that reuses an old id starts with
|
||
no attempts.
|
||
|
||
Use one file per configuration (per mailbox and reports folder): two
|
||
parsedmarc processes writing one file overwrite each other's entries. The
|
||
file is written atomically, readable by its owner only, and entries older
|
||
than 30 days are dropped. A file that is not a valid attempts file is
|
||
logged as a warning and replaced. If the file cannot be written, an error
|
||
is logged and the protection is off for the rest of that batch; reports are
|
||
still processed. If the write that removes an entry after a successful
|
||
parse is the one that fails, the entry stays (and the protection is off
|
||
for the rest of that batch), so a healthy message can be set aside as
|
||
invalid after two such failures; the write error is logged each time. In
|
||
`test` mode messages are recorded and set aside as usual but never moved.
|
||
|
||
## Environment variable configuration
|
||
|
||
Any configuration option can be set via environment variables using the
|
||
naming convention `PARSEDMARC_{SECTION}_{KEY}` (uppercase). This is
|
||
especially useful for Docker deployments where file permissions make it
|
||
difficult to use config files for secrets.
|
||
|
||
**Priority order:** CLI arguments > environment variables > config file > defaults
|
||
|
||
### Examples
|
||
|
||
```bash
|
||
# Set IMAP credentials via env vars
|
||
export PARSEDMARC_IMAP_HOST=imap.example.com
|
||
export PARSEDMARC_IMAP_USER=dmarc@example.com
|
||
export PARSEDMARC_IMAP_PASSWORD=secret
|
||
|
||
# Elasticsearch
|
||
export PARSEDMARC_ELASTICSEARCH_HOSTS=http://localhost:9200
|
||
export PARSEDMARC_ELASTICSEARCH_SSL=false
|
||
|
||
# Splunk HEC (note: section name splunk_hec becomes SPLUNK_HEC)
|
||
export PARSEDMARC_SPLUNK_HEC_URL=https://splunk.example.com
|
||
export PARSEDMARC_SPLUNK_HEC_TOKEN=my-hec-token
|
||
export PARSEDMARC_SPLUNK_HEC_INDEX=email
|
||
|
||
# General settings
|
||
export PARSEDMARC_GENERAL_SAVE_AGGREGATE=true
|
||
export PARSEDMARC_GENERAL_DEBUG=true
|
||
```
|
||
|
||
### Specifying the config file via environment variable
|
||
|
||
```bash
|
||
export PARSEDMARC_CONFIG_FILE=/etc/parsedmarc.ini
|
||
parsedmarc
|
||
```
|
||
|
||
### Running without a config file (env-only mode)
|
||
|
||
When no config file is given (neither `-c` flag nor `PARSEDMARC_CONFIG_FILE`),
|
||
parsedmarc will still pick up any `PARSEDMARC_*` environment variables. This
|
||
enables fully file-less deployments:
|
||
|
||
```bash
|
||
export PARSEDMARC_GENERAL_SAVE_AGGREGATE=true
|
||
export PARSEDMARC_GENERAL_OFFLINE=true
|
||
export PARSEDMARC_ELASTICSEARCH_HOSTS=http://elasticsearch:9200
|
||
parsedmarc /path/to/reports/*
|
||
```
|
||
|
||
### Docker Compose example
|
||
|
||
```yaml
|
||
services:
|
||
parsedmarc:
|
||
image: parsedmarc:latest
|
||
environment:
|
||
PARSEDMARC_IMAP_HOST: imap.example.com
|
||
PARSEDMARC_IMAP_USER: dmarc@example.com
|
||
PARSEDMARC_IMAP_PASSWORD: ${IMAP_PASSWORD}
|
||
PARSEDMARC_MAILBOX_WATCH: "true"
|
||
PARSEDMARC_MAILBOX_ATTEMPTS_FILE: /home/parsedmarc/attempts.json
|
||
PARSEDMARC_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
|
||
PARSEDMARC_GENERAL_SAVE_AGGREGATE: "true"
|
||
PARSEDMARC_GENERAL_SAVE_FAILURE: "true"
|
||
volumes:
|
||
- parsedmarc-state:/home/parsedmarc
|
||
|
||
volumes:
|
||
parsedmarc-state:
|
||
```
|
||
|
||
The `parsedmarc-state` volume keeps the
|
||
[attempts file](#messages-that-crash-parsedmarc-are-not-retried-forever)
|
||
when the container is recreated, as it is on an image upgrade. It is
|
||
mounted on the home directory of the image's `parsedmarc` user because
|
||
Docker gives a new named volume the owner of the directory it is mounted
|
||
on, and that directory is one the user can write.
|
||
|
||
### Docker secrets (`_FILE` suffix)
|
||
|
||
Any `PARSEDMARC_{SECTION}_{KEY}` environment variable can also be supplied
|
||
via a file by appending `_FILE` to its name. The file's contents (with any
|
||
trailing CR/LF characters stripped) are used as the value. This is the
|
||
same convention used by the official Postgres, MariaDB, and Redis container
|
||
images, and is designed to plug straight into Docker / Docker Compose /
|
||
Kubernetes secrets so credentials never appear in plain `environment:`
|
||
blocks (where they would be readable via `docker inspect`, container logs,
|
||
and `/proc/<pid>/environ`).
|
||
|
||
The bare `DEBUG` / `PARSEDMARC_DEBUG` aliases and `PARSEDMARC_CONFIG_FILE`
|
||
do not have a `_FILE` form; only `PARSEDMARC_{SECTION}_{KEY}` vars resolved
|
||
to a known config section are eligible.
|
||
|
||
If both the direct env var and the `_FILE` variant are set, the `_FILE`
|
||
variant wins. If the file does not exist or is unreadable, parsedmarc
|
||
exits with a configuration error rather than silently falling back to an
|
||
empty value.
|
||
|
||
```yaml
|
||
secrets:
|
||
imap_password:
|
||
file: ./secrets/imap_password.txt
|
||
|
||
services:
|
||
parsedmarc:
|
||
image: parsedmarc:latest
|
||
secrets:
|
||
- imap_password
|
||
environment:
|
||
PARSEDMARC_IMAP_HOST: imap.example.com
|
||
PARSEDMARC_IMAP_USER: dmarc@example.com
|
||
PARSEDMARC_IMAP_PASSWORD_FILE: /run/secrets/imap_password
|
||
```
|
||
|
||
Note that a small set of config keys whose own names already end in
|
||
`_file` (`[general] log_file`, `[msgraph] token_file`,
|
||
`[gmail_api] credentials_file`, `[gmail_api] token_file`,
|
||
`[mailbox] attempts_file`) keep their
|
||
pre-existing meaning when set via `PARSEDMARC_..._FILE` — that env var is
|
||
the path itself, not a wrapper around a file containing the path. To pass
|
||
*those* paths via a Docker secret, double up the suffix
|
||
(`PARSEDMARC_GMAIL_API_CREDENTIALS_FILE_FILE`); the inner contents are
|
||
then read and stored as the `credentials_file` value.
|
||
|
||
### Section name mapping
|
||
|
||
For sections with underscores in the name, the full section name is used:
|
||
|
||
| Section | Env var prefix |
|
||
| --- | --- |
|
||
| `general` | `PARSEDMARC_GENERAL_` |
|
||
| `mailbox` | `PARSEDMARC_MAILBOX_` |
|
||
| `imap` | `PARSEDMARC_IMAP_` |
|
||
| `msgraph` | `PARSEDMARC_MSGRAPH_` |
|
||
| `elasticsearch` | `PARSEDMARC_ELASTICSEARCH_` |
|
||
| `opensearch` | `PARSEDMARC_OPENSEARCH_` |
|
||
| `splunk_hec` | `PARSEDMARC_SPLUNK_HEC_` |
|
||
| `kafka` | `PARSEDMARC_KAFKA_` |
|
||
| `smtp` | `PARSEDMARC_SMTP_` |
|
||
| `s3` | `PARSEDMARC_S3_` |
|
||
| `postgresql` | `PARSEDMARC_POSTGRESQL_` |
|
||
| `syslog` | `PARSEDMARC_SYSLOG_` |
|
||
| `gmail_api` | `PARSEDMARC_GMAIL_API_` |
|
||
| `maildir` | `PARSEDMARC_MAILDIR_` |
|
||
| `log_analytics` | `PARSEDMARC_LOG_ANALYTICS_` |
|
||
| `gelf` | `PARSEDMARC_GELF_` |
|
||
| `webhook` | `PARSEDMARC_WEBHOOK_` |
|
||
| `loki` | `PARSEDMARC_LOKI_` |
|
||
|
||
## Using parsedmarc as a library
|
||
|
||
`parsedmarc` is also importable as a regular Python package, not just a CLI
|
||
tool. The main entry points — `parse_report_file()`, `parse_aggregate_report_xml()`,
|
||
`parse_aggregate_report_file()`, `parse_failure_report()`, `parse_report_email()`,
|
||
`get_dmarc_reports_from_mbox()`, `get_dmarc_reports_from_mailbox()`, and
|
||
`watch_inbox()` — are all importable
|
||
directly from the `parsedmarc` package. See the [API reference](api.md) for
|
||
the full set of modules and members.
|
||
|
||
Each of these functions accepts either individual option keyword arguments
|
||
(`offline`, `nameservers`, `dns_timeout`, etc.) or a single `config=` keyword
|
||
argument carrying a `ParserConfig` instance:
|
||
|
||
```python
|
||
from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
|
||
from parsedmarc.mail import IMAPConnection
|
||
|
||
config = ParserConfig(
|
||
offline=False,
|
||
nameservers=["1.1.1.1", "1.0.0.1"],
|
||
dns_timeout=5.0,
|
||
)
|
||
|
||
report = parse_report_file("aggregate_report.xml.gz", config=config)
|
||
|
||
connection = IMAPConnection(
|
||
host="imap.example.com", user="dmarc@example.com", password="..."
|
||
)
|
||
results = get_dmarc_reports_from_mailbox(connection, config=config)
|
||
```
|
||
|
||
A few things to keep in mind:
|
||
|
||
- When `config=` is passed, the individual option keyword arguments are
|
||
ignored in favor of the values carried on the `ParserConfig` instance.
|
||
- Each explicitly constructed `ParserConfig` owns its own isolated caches
|
||
(IP address info, seen aggregate report IDs, and the reverse DNS map).
|
||
Omitting `config=` falls back to the process-wide caches shared by every
|
||
call that doesn't pass one.
|
||
- `keep_alive`, `n_procs`, `parse_timeout` and `attempts_file` are not part
|
||
of `ParserConfig` — they control process/worker or mailbox orchestration
|
||
rather than parsing or enrichment behavior, so they are always passed as
|
||
separate keyword arguments.
|
||
|
||
## Performance tuning
|
||
|
||
For large mailbox imports or backfills, parsedmarc can consume a noticeable amount
|
||
of memory, especially when it runs on the same host as Elasticsearch or
|
||
OpenSearch. The following settings can reduce peak memory usage and make long
|
||
imports more predictable:
|
||
|
||
- Reduce `mailbox.batch_size` to smaller values such as `100-500` instead of
|
||
processing a very large message set at once. Smaller batches trade throughput
|
||
for lower peak memory use and less sink pressure.
|
||
- `n_procs` now parallelizes parsing for mailbox and mbox runs too, not just
|
||
report files passed directly as CLI arguments. It pays off most when a run
|
||
is bound by DNS/GeoIP enrichment rather than fetching or output. The
|
||
trade-off is memory and DNS load: at most roughly `2 * n_procs` messages
|
||
are held in flight at once, each worker process keeps its own DNS/GeoIP
|
||
cache, and DNS query volume can multiply by up to `n_procs`.
|
||
- Set `parse_timeout` (for example to `300`) so that one message or file that
|
||
takes far too long to parse is set aside as invalid instead of holding up
|
||
the rest of the run.
|
||
- Use `mailbox.since` to process reports in smaller time windows such as `1d`,
|
||
`7d`, or another interval that fits the backlog. This makes it easier to catch
|
||
up incrementally instead of loading an entire mailbox history in one run.
|
||
- Set `strip_attachment_payloads = True` when failure reports contain large
|
||
attachments and you do not need to retain the raw payloads in the parsed
|
||
output.
|
||
- Prefer running parsedmarc separately from Elasticsearch or OpenSearch, or
|
||
reserve enough RAM for both services if they must share a host.
|
||
- For very large imports, prefer incremental supervised runs, such as a
|
||
scheduler or systemd service, over infrequent massive backfills.
|
||
|
||
These are operational tuning recommendations rather than hard requirements, but
|
||
they are often enough to avoid memory pressure and reduce failures during
|
||
high-volume mailbox processing.
|
||
|
||
## Multi-tenant support
|
||
|
||
Starting in `8.19.0`, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:
|
||
|
||
```yaml
|
||
example:
|
||
- example.com
|
||
- example.net
|
||
- example.org
|
||
|
||
whalensolutions:
|
||
- whalensolutions.com
|
||
```
|
||
|
||
Save it to disk where the user running ParseDMARC can read it, then set `index_prefix_domain_map` to that filepath in the `[general]` section of the ParseDMARC configuration file and do not set an `index_prefix` option in the `[elasticsearch]` or `[opensearch]` sections.
|
||
|
||
When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
|
||
|
||
:::{note}
|
||
A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
|
||
:::
|
||
|
||
Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
|
||
|
||
The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (`SIGHUP`) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See [Backfilling the combined DKIM/SPF result fields](elasticsearch.md#backfilling-the-combined-dkimspf-result-fields) for the details.
|
||
|
||
## Running parsedmarc as a systemd service
|
||
|
||
Use systemd to run `parsedmarc` as a service and process reports as
|
||
they arrive.
|
||
|
||
This assumes `parsedmarc` has been installed into
|
||
`/opt/parsedmarc/venv` under a `parsedmarc` system user, as described
|
||
in [Installing parsedmarc](installation.md#installing-parsedmarc).
|
||
|
||
Protect the `parsedmarc` configuration file from prying eyes
|
||
|
||
```bash
|
||
sudo chown root:parsedmarc /etc/parsedmarc.ini
|
||
sudo chmod u=rw,g=r,o= /etc/parsedmarc.ini
|
||
```
|
||
|
||
Create the service configuration file
|
||
|
||
```bash
|
||
sudo nano /etc/systemd/system/parsedmarc.service
|
||
```
|
||
|
||
```ini
|
||
[Unit]
|
||
Description=parsedmarc mailbox watcher
|
||
Documentation=https://domainaware.github.io/parsedmarc/
|
||
Wants=network-online.target
|
||
After=network.target network-online.target elasticsearch.service
|
||
|
||
[Service]
|
||
ExecStart=/opt/parsedmarc/venv/bin/parsedmarc -c /etc/parsedmarc.ini
|
||
ExecReload=/bin/kill -HUP $MAINPID
|
||
KillSignal=SIGTERM
|
||
KillMode=mixed
|
||
TimeoutStopSec=60
|
||
User=parsedmarc
|
||
Group=parsedmarc
|
||
StateDirectory=parsedmarc
|
||
Restart=always
|
||
RestartSec=5m
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
```
|
||
|
||
`KillMode=mixed` sends the stop signal to the main parsedmarc process
|
||
only, so the batch in progress can finish before it exits, within
|
||
`TimeoutStopSec`, after which systemd kills the whole group; the default
|
||
control-group mode would also stop its parsing worker processes at once
|
||
and abandon the batch.
|
||
|
||
`StateDirectory=parsedmarc` makes systemd create `/var/lib/parsedmarc`,
|
||
owned by the `parsedmarc` user, before the service starts. Point the
|
||
[attempts file](#messages-that-crash-parsedmarc-are-not-retried-forever)
|
||
there in `/etc/parsedmarc.ini`, so a message that kills the service is
|
||
not parsed again on every restart:
|
||
|
||
```ini
|
||
[mailbox]
|
||
attempts_file = /var/lib/parsedmarc/attempts.json
|
||
```
|
||
|
||
Then, enable the service
|
||
|
||
```bash
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable parsedmarc.service
|
||
sudo service parsedmarc restart
|
||
```
|
||
|
||
:::{note}
|
||
You must also run the above commands whenever you edit
|
||
`parsedmarc.service`.
|
||
:::
|
||
|
||
:::{warning}
|
||
Always restart the service every time you upgrade to a new version of
|
||
`parsedmarc`:
|
||
|
||
```bash
|
||
sudo service parsedmarc restart
|
||
```
|
||
|
||
:::
|
||
|
||
:::{note}
|
||
On `systemctl stop`/`restart` (or Ctrl-C) `parsedmarc` finishes the
|
||
current batch, flushes its outputs, and exits cleanly. Shutdown is
|
||
observed at batch boundaries, so the worst-case delay is roughly
|
||
`[mailbox] check_timeout` (default 30s) plus the batch's processing and
|
||
flush time. Keep `TimeoutStopSec` comfortably above
|
||
`check_timeout` (≈2×, and raise both together) or systemd will
|
||
`SIGKILL` mid-batch. In the foreground, a second Ctrl-C force-quits
|
||
immediately, skipping the output flush.
|
||
:::
|
||
|
||
### Reloading configuration without restarting
|
||
|
||
When running in watch mode, `parsedmarc` supports reloading its
|
||
configuration file without restarting the service or interrupting
|
||
report processing that is already in progress. Send a `SIGHUP` signal
|
||
to the process, or use `systemctl reload` if the unit file includes
|
||
the `ExecReload` line shown above:
|
||
|
||
```bash
|
||
sudo systemctl reload parsedmarc
|
||
```
|
||
|
||
The reload takes effect after the current batch of reports finishes
|
||
processing and all output operations (Elasticsearch, Kafka, S3, etc.)
|
||
for that batch have completed. The following settings are reloaded:
|
||
|
||
- All output destinations (Elasticsearch, OpenSearch, Kafka, S3,
|
||
Splunk, syslog, GELF, webhooks, Log Analytics)
|
||
- Multi-tenant index prefix domain map (`index_prefix_domain_map` —
|
||
the referenced YAML file is re-read on reload)
|
||
- DNS and GeoIP settings (`nameservers`, `dns_timeout`, `ip_db_path`,
|
||
`ipinfo_url`, `offline`, etc.)
|
||
- Processing flags (`strip_attachment_payloads`, `batch_size`,
|
||
`check_timeout`, `parse_timeout`, `attempts_file`, etc.)
|
||
- Log level (`debug`, `verbose`, `warnings`, `silent`)
|
||
|
||
Mailbox connection settings (IMAP host/credentials, Microsoft Graph,
|
||
Gmail API, Maildir path) are **not** reloaded — changing those still
|
||
requires a full restart.
|
||
|
||
On a **successful** reload, the output clients for the updated
|
||
configuration are created first, and the connections they replace are
|
||
closed once the new configuration is live. The service then resumes
|
||
watching with the new settings.
|
||
|
||
If the new configuration file contains errors (missing required
|
||
settings, an output client that cannot be built — a PostgreSQL server,
|
||
Kafka broker, or TCP/TLS syslog server that cannot be reached, or an
|
||
`[opensearch]` `auth_type` that is not supported — an unreadable reverse
|
||
DNS map or PSL overrides file, a `log_file` that cannot be opened for
|
||
writing, a non-numeric `batch_size`, etc.), the **entire reload is
|
||
aborted** — any output clients built for the new configuration are
|
||
closed again, and the previous configuration remains fully active: the
|
||
old output clients stay open and connected, and the reverse DNS map, the
|
||
PSL overrides, the IP database selection, and every other setting keep
|
||
the values they had before the `SIGHUP`. This means a typo in one
|
||
section will not take down an otherwise working setup. Unlike startup,
|
||
where an unwritable `log_file` is only a warning, on reload it is one of
|
||
the errors that abort the reload, so the previous log file keeps
|
||
receiving logs — including the one explaining why. A `log_file` that
|
||
could not be opened when parsedmarc started (a warning at startup) is
|
||
retried by the next reload even if the setting is unchanged. (One thing
|
||
an aborted reload does not put back: if it got as far as downloading a
|
||
new IP database from a changed `ipinfo_url`, that file stays in the
|
||
shared cache directory. Which database parsedmarc *uses* is unchanged,
|
||
and the cached file is the same one the next restart would download.)
|
||
Elasticsearch and OpenSearch are the exception to "cannot be reached":
|
||
their clients connect lazily and the index migration only warns when the
|
||
cluster is unreachable, so a reload that points at an unreachable
|
||
cluster still commits, and the failure shows up on the first save
|
||
instead. Check the logs for details:
|
||
|
||
```bash
|
||
journalctl -u parsedmarc.service -r
|
||
```
|
||
|
||
To check the status of the service, run:
|
||
|
||
```bash
|
||
service parsedmarc status
|
||
```
|
||
|
||
:::{note}
|
||
In the event of a crash, systemd will restart the service after 5
|
||
minutes (per the `RestartSec` setting above), but the
|
||
`service parsedmarc status` command will only show
|
||
the logs for the current process. To view the logs for previous runs
|
||
as well as the current process (newest to oldest), run:
|
||
|
||
```bash
|
||
journalctl -u parsedmarc.service -r
|
||
```
|
||
|
||
:::
|
||
|
||
[cloudflare's public resolvers]: https://1.1.1.1/
|
||
[ipinfo lite rest api]: https://ipinfo.io/developers/lite-api
|