parsedmarc/docs/source/usage.md

Using parsedmarc

CLI help

usage: parsedmarc [-h] [-c CONFIG_FILE] [-r] [--strip-attachment-payloads] [-o OUTPUT]
                  [--aggregate-json-filename AGGREGATE_JSON_FILENAME] [--failure-json-filename FAILURE_JSON_FILENAME]
                  [--smtp-tls-json-filename SMTP_TLS_JSON_FILENAME] [--aggregate-csv-filename AGGREGATE_CSV_FILENAME]
                  [--failure-csv-filename FAILURE_CSV_FILENAME] [--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME]
                  [-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--dns-retries DNS_RETRIES] [--offline] [-s]
                  [-w] [--verbose] [--debug] [--log-file LOG_FILE] [--no-prettify-json] [-v]
                  [file_path ...]

Parses DMARC and SMTP TLS reports

positional arguments:
  file_path             one or more paths to aggregate, failure, or SMTP TLS report files, emails, mbox files, or
                        directories containing them

options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        a path to a configuration file (--silent implied)
  -r, --recursive       search directories given as file_path recursively, and enable '**' recursion in glob patterns
  --strip-attachment-payloads
                        remove attachment payloads from failure report output
  -o OUTPUT, --output OUTPUT
                        write output files to the given directory
  --aggregate-json-filename AGGREGATE_JSON_FILENAME
                        filename for the aggregate JSON output file
  --failure-json-filename FAILURE_JSON_FILENAME
                        filename for the failure JSON output file
  --smtp-tls-json-filename SMTP_TLS_JSON_FILENAME
                        filename for the SMTP TLS JSON output file
  --aggregate-csv-filename AGGREGATE_CSV_FILENAME
                        filename for the aggregate CSV output file
  --failure-csv-filename FAILURE_CSV_FILENAME
                        filename for the failure CSV output file
  --smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME
                        filename for the SMTP TLS CSV output file
  -n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
                        nameservers to query: IP addresses, https:// URLs (DNS over HTTPS), and/or
                        tls://ip[:port][#hostname] (DNS over TLS)
  -t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT, --dns-timeout DNS_TIMEOUT
                        number of seconds to wait for an answer from DNS (default: 2.0)
  --dns-retries DNS_RETRIES
                        number of times to retry DNS queries on timeout or other transient errors (default: 0)
  --offline             do not make online queries for geolocation or DNS
  -s, --silent          only print errors
  -w, --warnings        print warnings in addition to errors
  --verbose             more verbose output
  --debug               print debugging information
  --log-file LOG_FILE   output logging to a file
  --no-prettify-json    output JSON in a single line without indentation
  -v, --version         show program's version number and exit

:::{note} Starting in parsedmarc 6.0.0, most CLI options were moved to a configuration file, described below. :::

Configuration file

parsedmarc can be configured by supplying the path to an INI file

parsedmarc -c /etc/parsedmarc.ini

For example

# This is an example comment

[general]
save_aggregate = True
save_failure = True

[imap]
host = imap.example.com
user = dmarcresports@example.com
password = $uperSecure

[mailbox]
watch = True
delete = False

[elasticsearch]
hosts = 127.0.0.1:9200
ssl = False

[opensearch]
hosts = https://127.0.0.1:9200
user = admin
password = PasswordGoesHere
ssl = True

[splunk_hec]
url = https://splunkhec.example.com
token = HECTokenGoesHere
index = email

[s3]
bucket = my-bucket
path = parsedmarc

[syslog]
server = localhost
port = 514

[gelf]
host = logger
port = 12201
mode = tcp

[webhook]
aggregate_url = https://aggregate_url.example.com
failure_url = https://failure_url.example.com
smtp_tls_url = https://smtp_tls_url.example.com
timeout = 60

[loki]
url = http://loki.example.com:3100
labels = host=mail.example.com

The full set of configuration options is:

• general

  • save_aggregate - bool: Save aggregate report data to Elasticsearch, Splunk and/or S3

  • save_failure - bool: Save failure report data to Elasticsearch, Splunk and/or S3

  • save_smtp_tls - bool: Save SMTP TLS report data to Elasticsearch, Splunk and/or S3

  • index_prefix_domain_map - str: Path to a YAML file mapping OpenSearch/Elasticsearch index prefixes to domain names

  • strip_attachment_payloads - bool: Remove attachment payloads from results

  • silent - bool: Only print errors; set this to False to output results to STDOUT (Default: True)

  • output - str: Directory to place JSON and CSV files in. The JSON and CSV filename options below only take effect when this is set. CSV cells that begin with =, +, -, @, a tab, or a carriage return are written with a leading apostrophe ('), so spreadsheet applications treat them as text rather than formulas (CSV injection).

  • archive_directory - str: Optional. When set, successfully processed report files given as local file/directory path arguments are moved into <archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/ (year and month come from the report's own begin/arrival date, with the month zero-padded). A successfully parsed report whose archive date can't be determined is left in place with a logged warning. Files that fail to parse as a report are moved to <archive_directory>/Invalid/; files that fail for other reasons, such as transient I/O errors, are left in place so a later run can retry them. An existing destination file is never overwritten; a numeric suffix is appended before the extension (e.g. report-1.xml). This applies only to direct local file input — reports fetched from mailboxes (IMAP, Microsoft Graph, Gmail API, Maildir) use [mailbox] archive_folder instead, and mbox files are never moved. Files already inside archive_directory are excluded from processing, so the archive may safely live inside an input directory. A failed move is logged and does not stop the run.

  • aggregate_json_filename - str: filename for the aggregate JSON output file

  • failure_json_filename - str: filename for the failure JSON output file

  • smtp_tls_json_filename - str: filename for the SMTP TLS JSON output file

  • aggregate_csv_filename - str: filename for the aggregate CSV output file

  • failure_csv_filename - str: filename for the failure CSV output file

  • smtp_tls_csv_filename - str: filename for the SMTP TLS CSV output file

  • prettify_json - bool: Set this to False to output JSON in a single line without indentation (Default: True)

  • normalize_timespan_threshold_hours - float: Aggregate reports covering a longer time span than this many hours have their records normalized into per-day records (Default: 24)

  • ip_db_path - str: An optional custom path to a MMDB file from IPinfo, MaxMind, or DBIP

  • ipinfo_url - str: Overrides the default download URL for the bundled IPinfo Lite MMDB (env var: PARSEDMARC_GENERAL_IPINFO_URL). The pre-9.10 name ip_db_url is still accepted as a deprecated alias and logs a warning.

  • ipinfo_api_token - str: Optional IPinfo Lite REST API token. When set, IP lookups hit the API first for the freshest country/ASN data and fall back to the local MMDB on rate limit, quota exhaustion, or network errors. An invalid token exits the process with a fatal error. Ignored when offline is set. The Lite tier is free and has no documented monthly request cap; see the IPinfo Lite docs for current limits. (env var: PARSEDMARC_GENERAL_IPINFO_API_TOKEN)

  • offline - bool: Do not use online queries for geolocation or DNS. Also disables automatic downloading of the IP-to-country database and reverse DNS map.

  • always_use_local_files - bool: Disables the download of the IP-to-country database and reverse DNS map

  • local_reverse_dns_map_path - str: Overrides the default local file path to use for the reverse DNS map

  • reverse_dns_map_url - str: Overrides the default download URL for the reverse DNS map

  • local_psl_overrides_path - str: Overrides the default local file path to use for the PSL overrides list

  • psl_overrides_url - str: Overrides the default download URL for the PSL overrides list

  • nameservers - str: A comma separated list of DNS resolvers (Default: [Cloudflare's public resolvers]). Each entry is an IP address (DNS over UDP/TCP port 53), an https:// URL (DNS over HTTPS), or tls://ip[:port][#hostname] (DNS over TLS) — see Encrypted DNS

  • dns_test_address - str: A dummy address used for the DNS pre-flight check that runs when nameservers is set (Default: 1.1.1.1)

  • dns_timeout - float: DNS timeout period in seconds (Default: 2.0)

  • dns_retries - int: Number of times to retry a DNS query after a timeout or other transient error (Default: 0)

  • debug - bool: Print debugging messages

  • verbose - bool: More verbose output

  • warnings - bool: Print warnings in addition to errors

  • fail_on_output_error - bool: Exit with a non-zero status code if any configured output destination fails while saving/publishing reports (Default: False)

    :::{note} This option only controls the process exit code. Retaining mailbox messages whose reports could not be saved is automatic and happens either way — see Mailbox messages are only archived once the reports are saved. :::

  • log_file - str: Write log messages to a file at this path

  • n_procs - int: Number of processes to run in parallel when parsing report files passed directly as CLI arguments, messages in mbox files, and messages from mailbox connections (IMAP, Microsoft Graph, Gmail API, Maildir), including watch mode (Default: 1)

    :::{note} Setting this to a number larger than one can improve performance when processing thousands of files or messages :::

    :::{note} Only parsing is parallelized across worker processes. Fetching messages, deduplicating reports, archiving/deleting mailbox messages, and saving/publishing to outputs all stay sequential in the main process. Each worker process keeps its own DNS/GeoIP cache. :::

  • parse_timeout - float: Stop parsing a report file or message after this many seconds, and treat it as invalid: a report file is logged as failed (and moved to Invalid under archive_directory), an mbox message is skipped with a warning, and a mailbox message is moved to the Invalid archive folder (or deleted with delete_invalid). 0 means no limit (Default: 0, suggested: 300)

    :::{note} With a timeout set, parsing always runs in worker processes, even with n_procs = 1, because a parse can only be stopped by ending the process that runs it. When a parse times out, every worker process is stopped and new ones are started, and the other messages or files that were still being parsed start again from the beginning. parsedmarc checks the time only while it waits for a parse result, so a parse is stopped within one second of parsedmarc next waiting for a result after its time is up; while parsedmarc is busy between results (moving messages, for example) a parse can run on past the timeout. Choose a value well above the time your largest legitimate reports take, including DNS lookups, each of which can take up to dns_timeout seconds per try. With a timeout set, the worker processes hand each parsed report back through a file in a directory readable only by the parsedmarc user, inside the temporary directory (TMPDIR), which must therefore be writable; the files are removed as they are read, and their directory once the batch has been parsed or the workers stopped. :::

• mailbox

  • reports_folder - str: The mailbox folder (or label for Gmail) where the incoming reports can be found (Default: INBOX)

  • archive_folder - str: The mailbox folder (or label for Gmail) to sort processed emails into (Default: Archive)

  • watch - bool: Process new messages as they arrive, via the IMAP IDLE command or by polling the other mailbox types (Microsoft Graph, Gmail API, Maildir)

  • delete - bool: Delete messages after processing them, instead of archiving them

  • delete_aggregate - bool: Delete aggregate report messages after processing them, instead of archiving them (Default: the value of delete)

  • delete_failure - bool: Delete failure report messages after processing them, instead of archiving them (Default: the value of delete)

  • delete_smtp_tls - bool: Delete SMTP TLS report messages after processing them, instead of archiving them (Default: the value of delete)

  • delete_invalid - bool: Delete messages that could not be parsed, instead of archiving them in the Invalid subfolder, where they can be inspected for debugging (Default: the value of delete)

    :::{note} Each of these four options overrides delete for one kind of message only, and the other three keep inheriting delete. So delete = True combined with delete_failure = False archives failure report messages while deleting processed aggregate and SMTP TLS report messages — and unparseable ones, unless delete_invalid = False is set as well. :::

  • test - bool: Do not move or delete messages

  • batch_size - int: Number of messages to read and process before saving. Default 10. Use 0 for no limit.

  • check_timeout - int: Number of seconds to wait for an IMAP IDLE response or the number of seconds until the next mail check (Default: 30)

  • max_unsaved_retries - int: How many times a batch of messages whose reports could not be saved is retried before its messages are moved to the Unsaved archive subfolder instead of being retried again (Default: 2, i.e. the initial attempt plus two retries). Use 0 to move messages on the first failed save; negative values are rejected. Failures are counted in memory, so the cap applies across watch-mode checks within one long-running process, not across separate one-shot runs. See Mailbox messages are only archived once the reports are saved below.

  • attempts_file - str: Path to a JSON file in which parsedmarc counts the messages whose parsing never finished because the process died, so that a message that keeps killing parsedmarc is moved to the Invalid archive subfolder (or deleted, per delete_invalid) after two such attempts instead of being parsed on every run. The directory must be writable by the user running parsedmarc; the file is created on first use. Unset by default (no record is kept). See Messages that crash parsedmarc are not retried forever below.

  • since - str: Search for messages since a certain time. (Examples: 5m|3h|2d|1w) Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}. Defaults to 1d if an incorrect value is provided.

• imap

  • host - str: The IMAP server hostname or IP address

  • port - int: The IMAP server port (Default: 993)

    :::{note} % characters must be escaped with another % character, so use %% wherever a % character is used. :::

    :::{note} Starting in version 8.0.0, most options from the imap section have been moved to the mailbox section. :::

    :::{note} If your host recommends another port, still try 993 :::

  • ssl - bool: Use an encrypted SSL/TLS connection from the start (implicit TLS, usually port 993) (Default: True). When False, the connection starts in plain text and must be upgraded with STARTTLS before logging in (usually port 143); see starttls.

  • starttls - bool: Whether to upgrade a plain-text (ssl = False) connection with STARTTLS. When unset (the default), the connection is upgraded if the server advertises STARTTLS, and parsedmarc refuses to log in if it does not, so the credentials are never sent in plain text. true always requires the upgrade. false never upgrades, even when the server advertises STARTTLS, so the credentials and messages are sent in plain text; use it only for a server on the same host, such as DavMail on the loopback address. Ignored when ssl = True.

  • skip_certificate_verification - bool: Skip certificate verification (not recommended)

  • timeout - int: Number of seconds to wait for an IMAP operation (Default: 30)

  • max_retries - int: Maximum number of retries after an IMAP timeout (Default: 4)

  • user - str: The IMAP user

  • password - str: The IMAP password

• msgraph

  Requires the msgraph extra: pip install "parsedmarc[msgraph]"

  • auth_method - str: Authentication method, valid types are UsernamePassword, DeviceCode, ClientSecret, Certificate, or ClientAssertion (Default: UsernamePassword).

  • user - str: The M365 user, required when the auth method is UsernamePassword

  • password - str: The user password, required when the auth method is UsernamePassword

  • client_id - str: The app registration's client ID

  • client_secret - str: The app registration's secret. Required when the auth method is UsernamePassword or ClientSecret

  • certificate_path - str: Path to a PEM or PKCS12 certificate including the private key. Required when the auth method is Certificate

  • certificate_password - str: Optional password for the certificate file when using Certificate auth

  • client_assertion - str: A signed JWT client assertion, required when the auth method is ClientAssertion

    :::{note} client_assertion is a static value, so it is only usable for as long as the JWT it contains remains unexpired (assertions are typically short-lived, on the order of minutes). It is best suited to short one-shot runs; for long-running watch mode, prefer Certificate or ClientSecret auth, which can refresh indefinitely. :::

  • tenant_id - str: The Azure AD tenant ID. This is required for all auth methods except UsernamePassword.

  • mailbox - str: The mailbox name. This defaults to the current user if using the UsernamePassword auth method, but could be a shared mailbox if the user has access to the mailbox

  • graph_url - str: Microsoft Graph URL. Allows for use of national clouds (e.g. Azure Gov) (Default: https://graph.microsoft.com)

    :::{warning} Setting graph_url alone is not sufficient for a national/sovereign cloud tenant. It only changes the Microsoft Graph API root; the Microsoft Entra ID (Azure AD) token endpoint used for authentication is not currently configurable in parsedmarc or mailsuite, and always defaults to the global https://login.microsoftonline.com. A true national-cloud deployment also needs its own Entra ID endpoint, so graph_url by itself only helps if your tenant is registered in the global cloud but you specifically need to reach one of these Graph API roots (per Microsoft's national cloud deployment docs):

    National cloud	Microsoft Graph URL	Entra ID endpoint (not configurable here)
    Global (default)	https://graph.microsoft.com	https://login.microsoftonline.com
    US Government L4 (GCC High)	https://graph.microsoft.us	https://login.microsoftonline.us
    US Government L5 (DoD)	https://dod-graph.microsoft.us	https://login.microsoftonline.us
    China, operated by 21Vianet	https://microsoftgraph.chinacloudapi.cn	https://login.chinacloudapi.cn

    :::

  • token_file - str: Path to save the token file (Default: .token)

  • allow_unencrypted_storage - bool: Allows the Azure Identity module to fall back to unencrypted token cache (Default: False). Even if enabled, the cache will always try encrypted storage first.

    :::{note} token_file stores the serialized authentication record; the underlying MSAL persistent token cache is separately named parsedmarc, not mailsuite's own default cache name. This is deliberate: the Graph mailbox backend used to live directly in parsedmarc, and moved into the mailsuite dependency in parsedmarc 9.11.0. Explicitly keeping the parsedmarc cache name means tokens cached before that move keep working after upgrading — there is nothing to migrate and no action needed on your part. :::

    :::{note} You must create an app registration in Azure AD and have an admin grant the Microsoft Graph Mail.ReadWrite (delegated) permission to the app. If you are using UsernamePassword auth and the mailbox is different from the username, you must grant the app Mail.ReadWrite.Shared. :::

    Auth method	Reading (own mailbox)	Reading (shared mailbox)
    UsernamePassword / DeviceCode (delegated)	Mail.ReadWrite	Mail.ReadWrite.Shared
    ClientSecret / Certificate / ClientAssertion (application)	Mail.ReadWrite (application), scoped via New-ApplicationAccessPolicy	same as own mailbox — app-only access is scoped by policy, not by permission name

    :::{tip} Troubleshooting connections. Run with --verbose to log a redacted connection summary (auth method, tenant, client ID, mailbox, Graph URL) before the connection attempt, and with --debug to additionally surface the underlying library activity — azure.identity token acquisition (including AADSTS error codes from Entra ID, which distinguish a local configuration problem from an Exchange Online-side one), Microsoft Graph SDK requests, and httpx HTTP request lines. Secret values (passwords, client secrets, certificate passwords) are never written to logs. Connection, mailbox fetch, message send, and --watch failures each log a single ERROR line naming the mailbox, tenant, auth method, and the Graph request-id/client-request-id when available — worth quoting verbatim when contacting Microsoft support. :::

    :::{warning} If you are using the ClientSecret auth method, you need to grant the Mail.ReadWrite (application) permission to the app. You must also restrict the application's access to a specific mailbox since it allows all mailboxes by default. Use the New-ApplicationAccessPolicy command in the Exchange PowerShell module. If you need to scope the policy to shared mailboxes, you can add them to a mail enabled security group and use that as the group id.

    New-ApplicationAccessPolicy -AccessRight RestrictAccess
    -AppId "<CLIENT_ID>" -PolicyScopeGroupId "<MAILBOX>"
    -Description "Restrict access to dmarc reports mailbox."
    

    The same application permission and mailbox scoping guidance applies to the Certificate and ClientAssertion auth methods.

    :::

    Sending the summary email via Microsoft Graph. When [msgraph] is configured and [smtp] has a to value but no host, the periodic summary email is sent through the same already-authenticated Graph mailbox connection used for reading (/users/{mailbox}/sendMail), and a copy is saved to Sent Items. When [smtp] host is set, SMTP is used regardless of whether [msgraph] is also configured — SMTP is always preferred, with no automatic fallback to Graph on SMTP failure.

    Example config combining [msgraph] with a [smtp] section that only sets to/subject (no host):

    [msgraph]
    auth_method = Certificate
    client_id = ...
    tenant_id = ...
    mailbox = dmarc-reports@example.com
    certificate_path = /path/to/cert.pem
    
    [smtp]
    to = admin@example.com
    subject = DMARC Summary
    

    Required Microsoft Graph permissions, in addition to the reading-related permissions documented above:

    Auth method	Reading	Sending (own mailbox)	Sending (shared mailbox)
    UsernamePassword / DeviceCode (delegated)	Mail.ReadWrite (+.Shared for shared)	Mail.Send	Mail.Send.Shared
    ClientSecret / Certificate / ClientAssertion (application)	Mail.ReadWrite (application)	Mail.Send (application)	Mail.Send (application), scoped via New-ApplicationAccessPolicy

    :::{warning} Graph-based sending is only confirmed to work with the app-only auth methods (ClientSecret, Certificate, ClientAssertion). The delegated auth methods (UsernamePassword, DeviceCode) currently request only the Mail.ReadWrite(.Shared) scope when authenticating, not Mail.Send — so a delegated connection's access token will not carry Mail.Send even if an administrator has granted it, and /sendMail calls are expected to fail with an access-denied error regardless of what's granted in Azure AD. Use an app-only auth method if you need Graph-based sending. :::

    Minimal example configs. Each auth method needs a different minimum set of keys. These read-only examples omit [smtp]; see above for adding Graph-based sending on top of any of them.

    UsernamePassword (delegated, own mailbox):

    [msgraph]
    auth_method = UsernamePassword
    client_id = ...
    client_secret = ...
    user = dmarc-reports@example.com
    password = ...
    

    DeviceCode (delegated, interactive sign-in on first run — user is the account that signs in; mailbox is the shared mailbox it reads, and only needs to differ from user to request Mail.ReadWrite.Shared instead of plain Mail.ReadWrite):

    [msgraph]
    auth_method = DeviceCode
    client_id = ...
    tenant_id = ...
    user = signing-in-user@example.com
    mailbox = dmarc-reports@example.com
    

    ClientSecret (app-only):

    [msgraph]
    auth_method = ClientSecret
    client_id = ...
    tenant_id = ...
    client_secret = ...
    mailbox = dmarc-reports@example.com
    

    Certificate (app-only):

    [msgraph]
    auth_method = Certificate
    client_id = ...
    tenant_id = ...
    certificate_path = /path/to/cert.pem
    mailbox = dmarc-reports@example.com
    

    ClientAssertion (app-only, short-lived JWT — see the note above about its unsuitability for watch mode):

    [msgraph]
    auth_method = ClientAssertion
    client_id = ...
    tenant_id = ...
    client_assertion = ...
    mailbox = dmarc-reports@example.com
    

    :::{tip} Troubleshooting.

    Error	Cause	Fix
    "...needs permission to access resources in your organization that only an admin can grant" / "Admin consent required"	A delegated auth method (UsernamePassword, DeviceCode) is authenticating with a scope (Mail.ReadWrite or Mail.ReadWrite.Shared) the tenant admin hasn't consented to yet.	Have an Entra ID admin grant consent: Azure Portal → Enterprise Applications → your app → Permissions → Grant admin consent, or az ad app permission admin-consent --id <CLIENT_ID>. This is separate from the New-ApplicationAccessPolicy step above, which only applies to app-only auth.
    ErrorItemNotFound: ... Default folder Root not found	mailsuite can resolve the well-known folders (Inbox, Archive, Drafts, Sent Items, Deleted Items, Junk Email) even when a mailbox's folder hierarchy hasn't fully provisioned, but a custom, non-well-known reports_folder name still fails to resolve on such a mailbox.	Point reports_folder at (or under) one of the six well-known folder names above, or sign into the (shared) mailbox once via Outlook/OWA to force Exchange to provision it, then retry.
    RuntimeError: Event loop is closed	Historical bug, fixed in mailsuite 2.0.2. Not reachable with the mailsuite>=2.2.2 this project requires.	Confirm your installed mailsuite version is current (pip show mailsuite); upgrade if it's somehow pinned below 2.0.2.
    Invalid/rejected timestamp in the since/receivedDateTime filter	Historical bug (parsedmarc #706/#708): older versions appended a spurious Z to an already-UTC-offset ISO timestamp. Fixed since parsedmarc 9.5.1/9.5.5.	Upgrade parsedmarc if you're on a version older than 9.5.5.
    :::

• elasticsearch

  Requires the elastic extra: pip install "parsedmarc[elastic]"

  • hosts - str: A comma-separated list of hostnames and ports or URLs (e.g. 127.0.0.1:9200 or https://localhost:9200). Do not put credentials in the URL; use user and password (or api_key) instead.
  • user - str: Basic auth username
  • password - str: Basic auth password
  • api_key - str: API key
  • ssl - bool: Use an encrypted SSL/TLS connection (Default: True)
  • timeout - float: Timeout in seconds (Default: 60)
  • cert_path - str: Path to a trusted CA certificates file
  • skip_certificate_verification - bool: Skip certificate verification (not recommended)
  • index_suffix - str: A suffix to apply to the index names
  • index_prefix - str: A prefix to apply to the index names
  • monthly_indexes - bool: Use monthly indexes instead of daily indexes (ignored, with a warning, when use_data_streams is True)
  • number_of_shards - int: The number of shards to use when creating the index, or each backing index of a data stream (Default: 1)
  • number_of_replicas - int: The number of replicas to use when creating the index, or each backing index of a data stream (Default: 0)
  • use_data_streams - bool: Save each report type to a data stream named like the index without the date suffix (e.g. dmarc_aggregate) instead of to daily or monthly indexes. parsedmarc creates the stream and its index template; retention is left to an ILM policy you attach. Requires Elasticsearch 8.7 or later. See Data streams and records retention (Default: False)
  • serverless - bool: Set to True when targeting an Elastic Cloud Serverless project. Serverless manages sharding and replication itself and rejects the number_of_shards / number_of_replicas index settings with HTTP 400. With this flag set, parsedmarc strips those keys from the settings sent at index creation (or in a data stream's index template); any other settings (e.g. refresh_interval) are passed through unchanged (Default: False)

• opensearch

  Requires the opensearch extra: pip install "parsedmarc[opensearch]"

  • hosts - str: A comma-separated list of hostnames and ports or URLs (e.g. 127.0.0.1:9200 or https://localhost:9200). Do not put credentials in the URL; use user and password (or api_key) instead.
  • user - str: Basic auth username
  • password - str: Basic auth password
  • api_key - str: API key
  • auth_type - str: Authentication type: basic (default) or awssigv4 (the key authentication_type is accepted as an alias for this option)
  • aws_region - str: AWS region for SigV4 authentication (required when auth_type = awssigv4)
  • aws_service - str: AWS service for SigV4 signing (Default: es)
  • ssl - bool: Use an encrypted SSL/TLS connection (Default: True)
  • timeout - float: Timeout in seconds (Default: 60)
  • cert_path - str: Path to a trusted CA certificates file
  • skip_certificate_verification - bool: Skip certificate verification (not recommended)
  • index_suffix - str: A suffix to apply to the index names
  • index_prefix - str: A prefix to apply to the index names
  • monthly_indexes - bool: Use monthly indexes instead of daily indexes (ignored, with a warning, when use_data_streams is True)
  • number_of_shards - int: The number of shards to use when creating the index, or each backing index of a data stream (Default: 1)
  • number_of_replicas - int: The number of replicas to use when creating the index, or each backing index of a data stream (Default: 0)
  • use_data_streams - bool: Save each report type to a data stream named like the index without the date suffix (e.g. dmarc_aggregate) instead of to daily or monthly indexes. parsedmarc creates the stream and its index template; retention is left to an ISM policy you attach. OpenSearch rejects data stream names that start with ., so an index_prefix starting with . cannot be used with this option. See Data streams and records retention (Default: False)

• splunk_hec

  • url - str: The URL of the Splunk HTTP Events Collector (HEC). Do not put credentials in the URL; use token instead.
  • token - str: The HEC token
  • index - str: The Splunk index to use
  • skip_certificate_verification - bool: Skip certificate verification (not recommended)

• kafka

  Requires the kafka extra: pip install "parsedmarc[kafka]"

  • hosts - str: A comma separated list of Kafka hosts
  • user - str: The Kafka user
  • password - str: The Kafka password
  • ssl - bool: Use an encrypted SSL/TLS connection (Default: True)
  • skip_certificate_verification - bool: Skip certificate verification (not recommended)
  • aggregate_topic - str: The Kafka topic for aggregate reports
  • failure_topic - str: The Kafka topic for failure reports
  • smtp_tls_topic - str: The Kafka topic for SMTP TLS reports

• smtp

  The results email is only sent when at least one aggregate, failure, or SMTP TLS report was parsed during the run; an empty run (e.g. an empty inbox) skips the email instead of sending headers-only CSVs.

  • host - str: The SMTP hostname. Required unless [msgraph] is configured, in which case omitting it sends the summary via Microsoft Graph instead — see "Sending the summary email via Microsoft Graph" above.

  • port - int: The SMTP port (Default: 25). Usually 587 for STARTTLS, or 465 with ssl = True.

  • ssl - bool: Use an encrypted SSL/TLS connection from the start (implicit TLS, usually port 465) (Default: False). When False, the connection starts in plain text and must be upgraded with STARTTLS before authenticating; see starttls.

  • starttls - bool: Whether to upgrade a plain-text (ssl = False) connection with STARTTLS. When unset (the default), the connection is upgraded if the server advertises STARTTLS, and parsedmarc refuses to authenticate if it does not, so the credentials are never sent in plain text ("Failed to email results" is logged). true always requires the upgrade. false never upgrades, even when the server advertises STARTTLS, so the credentials and the results email are sent in plain text; use it only for a server on the same host. Ignored when ssl = True.

  • skip_certificate_verification - bool: Skip certificate verification (not recommended)

  • user - str: the SMTP username. SMTP-only; not used when sending via Microsoft Graph.

  • password - str: the SMTP password. SMTP-only; not used when sending via Microsoft Graph.

  • from - str: The From header to use in the email. SMTP-only. When sent via Microsoft Graph, the message's From is always the [msgraph] mailbox — [smtp] from has no effect.

  • to - list: A list of email addresses to send to

  • subject - str: The Subject header to use in the email (Default: parsedmarc report)

  • attachment - str: The ZIP attachment filename (Default: DMARC-<YYYY-MM-DD>.zip)

  • message - str: The email message (Default: Please see the attached DMARC results.)

    :::{note} % characters must be escaped with another % character, so use %% wherever a % character is used. :::

• postgresql

  • host - str: The PostgreSQL server hostname or IP address. Required unless connection_string is provided.
  • port - int: The PostgreSQL server port (Default: 5432)
  • user - str: The database user name (Optional)
  • password - str: The database user password (Optional)
  • database - str: The database name (Optional)
  • connection_string - str: A full libpq connection string or URI (e.g. postgresql://user@host/dbname). When provided, all individual parameters above are ignored. Prefer the separate host, port, user, password, and database settings and do not put the password in the connection string: a malformed connection string can make the driver echo it back in an error message. parsedmarc replaces such errors with a fixed message in its log, but cannot stop the driver from producing them.

  The PostgreSQL backend is an optional extra. Install it with pip install "parsedmarc[postgresql]" (it pulls in psycopg); the prebuilt binary wheels are not available for every platform, which is why it is the one extra that parsedmarc[all] deliberately leaves out — combine the two with pip install "parsedmarc[all,postgresql]". The prebuilt Docker image (ghcr.io/domainaware/parsedmarc) already bundles [all,postgresql], so the PostgreSQL backend works out of the box in the container — psycopg ships amd64 and arm64 binary wheels, both of which the image supports.

  Tables are created automatically on first run using CREATE TABLE IF NOT EXISTS, so no manual schema migration is needed for fresh installations.

  Example configuration:

  [postgresql]
  host = localhost
  port = 5432
  user = parsedmarc
  password = secret
  database = parsedmarc
  

  Or using a DSN/URI, with the password supplied separately (for example through the PGPASSWORD environment variable or a ~/.pgpass file, both of which libpq reads):

  [postgresql]
  connection_string = postgresql://parsedmarc@localhost/parsedmarc
  

  Saving parsed data to PostgreSQL is controlled by the [general] options save_aggregate, save_failure, and save_smtp_tls (save_forensic is still accepted as a deprecated alias for save_failure). These flags must be set to True for the corresponding report types (aggregate DMARC, failure DMARC, and SMTP TLS reports) or no data will be written to PostgreSQL, even if this section is configured.

• s3

  Requires the s3 extra: pip install "parsedmarc[s3]"

  • bucket - str: The S3 bucket name
  • path - str: The path to upload reports to (Default: /)
  • region_name - str: The region name (Optional)
  • endpoint_url - str: The endpoint URL (Optional)
  • access_key_id - str: The access key id (Optional)
  • secret_access_key - str: The secret access key (Optional)

  Each report is saved as one JSON object, with this key: <path>/<report type>/year=YYYY/month=MM/day=DD/<report ID>_<hash>.json. The date is the report's begin date. Failure reports have no report ID or begin date, so they use the reported domain and the arrival date instead. Aggregate report dates are in the host's local time (SMTP TLS dates are as the report gives them, and failure arrival dates are in UTC), so for aggregate reports the key's day and the hash depend on the host's time zone.

  The report ID comes from the sender, so it is percent-encoded before it is used in the key. /, the characters AWS lists as "characters to avoid" (\ { } ^ [ ] " < > ~ # | ` and %), control characters, and non-ASCII characters are encoded (for example, / becomes %2F), so the ID is always a single key segment, and % appears in it only as the escape character. If the key would be longer than the 1,024-byte S3 limit, the report ID is shortened.

  The hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report:

  • Aggregate: the organization name, report ID, policy domain, begin date, and end date
  • SMTP TLS: the organization name, report ID, begin date, end date, and the domain of each policy
  • Failure: the arrival date, reported domain, original mail from, original recipient, original envelope ID, source IP address, and the message sample

  Only fields from the report itself are hashed, not the IP enrichment (reverse DNS, country, and so on), so a report that is saved again overwrites its own object. Two reports get the same key only when all of these fields match, so a report that reuses another report's ID, but differs in any of these fields, does not replace it. These fields are not secret, though: a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turn on S3 Versioning on the bucket to keep both.

  Each object also gets metadata from the report: org_name, org_email, report_id, begin_date, and end_date for aggregate reports; org_name, report_id, begin_date, and end_date for SMTP TLS reports; and reported_domain and arrival_date for failure reports. AWS recommends US-ASCII metadata, and botocore rejects anything else, so in each value %, control characters, and non-ASCII characters are percent-encoded, and the value is cut to at most 256 characters.

• syslog

  • server - str: The Syslog server name or IP address
  • port - int: The port to use (Default: 514)
  • protocol - str: The protocol to use: udp, tcp, or tls (Default: udp)
  • cafile_path - str: Path to CA certificate file for TLS server verification (Optional)
  • certfile_path - str: Path to client certificate file for TLS authentication (Optional)
  • keyfile_path - str: Path to client private key file for TLS authentication (Optional)
  • timeout - float: Connection timeout in seconds for TCP/TLS (Default: 5.0)
  • retry_attempts - int: Number of retry attempts for failed connections (Default: 3)
  • retry_delay - int: Delay in seconds between retry attempts (Default: 5)

  Example UDP configuration (default):

  [syslog]
  server = syslog.example.com
  port = 514
  

  Example TCP configuration:

  [syslog]
  server = syslog.example.com
  port = 6514
  protocol = tcp
  timeout = 10.0
  retry_attempts = 5
  

  Example TLS configuration with server verification:

  [syslog]
  server = syslog.example.com
  port = 6514
  protocol = tls
  cafile_path = /path/to/ca-cert.pem
  timeout = 10.0
  

  Example TLS configuration with mutual authentication:

  [syslog]
  server = syslog.example.com
  port = 6514
  protocol = tls
  cafile_path = /path/to/ca-cert.pem
  certfile_path = /path/to/client-cert.pem
  keyfile_path = /path/to/client-key.pem
  timeout = 10.0
  retry_attempts = 3
  retry_delay = 5
  

• gmail_api

  Requires the gmail extra: pip install "parsedmarc[gmail]"

  • credentials_file - str: Path to file containing the credentials, None to disable (Default: None)

  • token_file - str: Path to save the token file (Default: .token)

  • auth_mode - str: Authentication mode, installed_app (default) or service_account

  • service_account_user - str: Delegated mailbox user for Gmail service account auth (required for domain-wide delegation). Also accepted as delegated_user for backward compatibility.

    :::{note} credentials_file and token_file can be obtained by following the Gmail API quickstart. Please change the scope to https://www.googleapis.com/auth/gmail.modify. ::: :::{note} When auth_mode = service_account, credentials_file must point to a Google service account key JSON file, and token_file is not used. :::

  • include_spam_trash - bool: Include messages in Spam and Trash when searching reports (Default: False)

  • scopes - str: Comma separated list of scopes to use when acquiring credentials (Default: https://www.googleapis.com/auth/gmail.modify)

  • oauth2_port - int: The TCP port for the local server to listen on for the OAuth2 response (Default: 8080)

  • paginate_messages - bool: When True, fetch all applicable Gmail messages. When False, only fetch up to 100 new messages per run (Default: True)

• log_analytics

  Requires the loganalytics extra: pip install "parsedmarc[loganalytics]"

  • client_id - str: The app registration's client ID
  • client_secret - str: The app registration's client secret
  • tenant_id - str: The tenant id where the app registration resides
  • dce - str: The Data Collection Endpoint (DCE). Example: https://{DCE-NAME}.{REGION}.ingest.monitor.azure.com.
  • dcr_immutable_id - str: The immutable ID of the Data Collection Rule (DCR)
  • dcr_aggregate_stream - str: The stream name for aggregate reports in the DCR
  • dcr_failure_stream - str: The stream name for the failure reports in the DCR
  • dcr_smtp_tls_stream - str: The stream name for the SMTP TLS reports in the DCR

  :::{note} Information regarding the setup of the Data Collection Rule can be found in the Azure documentation. :::

• gelf

  Requires the gelf extra: pip install "parsedmarc[gelf]"

  • host - str: The GELF server name or IP address
  • port - int: The port to use
  • mode - str: The GELF transport type to use. Valid modes: tcp, udp, tls

• maildir

  • maildir_path - str: Full path to the maildir location (the key path is accepted as an alias). Required to read from a maildir.
  • maildir_create - bool: Create the maildir if not present (Default: False; the key create is accepted as an alias)

• webhook - Post the individual reports to a webhook URL with the report as the JSON body

  • aggregate_url - str: URL of the webhook which should receive the aggregate reports
  • failure_url - str: URL of the webhook which should receive the failure reports
  • smtp_tls_url - str: URL of the webhook which should receive the SMTP TLS reports
  • timeout - int: Timeout in seconds for webhook requests (Default: 60)

• loki - Push every report record as one JSON log line to the Grafana Loki push API. The lines are the same flat records the syslog and GELF outputs send, plus report_type (aggregate, failure, or smtp_tls) and domain (the reported domain: domain, reported_domain, or policy_domain), so LogQL's | json parser can filter and sum them, e.g. messages failing DMARC per domain: sum by (domain) (sum_over_time({job="parsedmarc", report_type="aggregate"} | json dmarc_aligned, count | dmarc_aligned="false" | unwrap count [1d])). Name the fields to extract: a bare | json turns every field into a label, and the many distinct values quickly exceed Loki's per-query series limit (max_query_series, 500 by default). A ready-made Grafana dashboard for these lines ships in dashboards/grafana/Grafana-DMARC_Reports-Loki.json; import it and pick your Loki datasource for its DS_LOKI input. It expects the default job=parsedmarc label and label_fields.

  • url - str: The Loki base URL; /loki/api/v1/push is appended when the URL has no path. A URL with a path is used as is. Do not put credentials in the URL; use username and password (or token) instead.
  • labels - str: Comma-separated name=value stream labels, merged over the defaults job=parsedmarc and host=<FQDN of the parsedmarc host> (in a container, the FQDN is the container's hostname)
  • label_fields - str: Comma-separated record fields used as stream labels (Default: report_type, domain; empty for none). Keep these to fields with few distinct values — every combination is its own Loki stream, so fields like source_ip_address belong in the line only. Records without a value for a field get no such label.
  • timestamp - str: ingest (the time of the push) or report_begin (the begin of the reported period, or the arrival date of a failure report) (Default: ingest). Loki rejects entries older than its reject_old_samples_max_age (one week by default) and, within a stream, entries far older than the newest one, so report_begin loses late or out-of-order reports unless Loki is configured for them. The reported period is in the line either way (begin_date, end_date).
  • username - str: Basic authentication user
  • password - str: Basic authentication password
  • token - str: Bearer token, instead of basic authentication
  • tenant_id - str: Tenant ID, sent as the X-Scope-OrgID header
  • skip_certificate_verification - bool: Skip TLS certificate verification (not recommended)
  • timeout - float: Timeout in seconds for push requests (Default: 60)
  • batch_size - int: Maximum log lines per push request (Default: 1000)

:::{warning} It is strongly recommended to not use the nameservers setting. By default, parsedmarc uses Cloudflare's public resolvers, which are much faster and more reliable than Google, Cisco OpenDNS, or even most local resolvers.

The nameservers option should only be used if your network blocks DNS requests to outside resolvers, or blocks plain DNS entirely — see Encrypted DNS. :::

:::{note} save_aggregate and save_failure are separate options because you may not want to save failure reports (formerly known as forensic reports) to your Elasticsearch instance, particularly if you are in a highly-regulated industry that handles sensitive data, such as healthcare or finance. If your legitimate outgoing email fails DMARC, it is possible that email may appear later in a failure report.

Failure reports contain the original headers of an email that failed a DMARC check, and sometimes may also include the full message body, depending on the policy of the reporting organization.

Most reporting organizations do not send failure reports of any kind for privacy reasons. While aggregate DMARC reports are sent at least daily, it is normal to receive very few failure reports.

An alternative approach is to still collect failure/ruf reports in your DMARC inbox, but run parsedmarc with save_failure = True manually on a separate IMAP folder (using the reports_folder option), after you have manually moved known samples you want to save to that folder (e.g. malicious samples and non-sensitive legitimate samples). :::

:::{warning} Elasticsearch 8 changed the limits policy for shards, restricting them by default to 1000. parsedmarc uses a shard per analyzed day. If you have more than ~3 years of data, you will need to update this limit. Check current usage (from Management -> Dev Tools -> Console):

GET /_cluster/health?pretty
{
...
  "active_primary_shards": 932,
  "active_shards": 932,
...
}

For example, update the limit to 2000:

PUT _cluster/settings
{
  "persistent" : {
    "cluster.max_shards_per_node" : 2000
  }
}

Increasing this value increases resource usage. :::

Encrypted DNS

Every entry in the nameservers list picks its own transport:

• An IP address — plain DNS over UDP and TCP port 53, the default and the behavior of every earlier release.
• An https:// URL — DNS over HTTPS (DoH).
• tls://ip[:port][#hostname] — DNS over TLS (DoT). The port defaults to 853, and the optional #hostname names the TLS certificate identity of the server (SNI), matching systemd-resolved's syntax. The host itself must be an IP address, and an IPv6 address must be wrapped in brackets, so that its colons cannot be mistaken for the port separator — tls://[2620:fe::fe]#dns.quad9.net.

Forms can be mixed, and are tried in the order given:

[general]
nameservers = https://cloudflare-dns.com/dns-query, tls://9.9.9.9#dns.quad9.net

On a network where outbound DNS is blocked but an HTTP proxy is available, configure DoH nameservers and set the standard proxy environment variables (issue #880):

export HTTPS_PROXY=http://proxy.example.net:3128
export NO_PROXY=elasticsearch.example.net,127.0.0.1

parsedmarc's DoH queries honor HTTP_PROXY, HTTPS_PROXY, and NO_PROXY, and the proxy resolves the DoH server's own hostname on parsedmarc's behalf, so such a deployment needs no access to UDP port 53 at all.

If the proxy inspects TLS, point the standard SSL_CERT_FILE environment variable at your organization's CA bundle so its certificate is trusted:

export SSL_CERT_FILE=/etc/ssl/certs/corporate-ca.pem

:::{note} parsedmarc's DoT connections are made directly to TCP port 853 and do not use a proxy. Use DoH on a proxy-only network. :::

Mailbox messages are only archived once the reports are saved

parsedmarc processes a mailbox in batches of batch_size messages. Each batch is written to every configured output destination before any of that batch's messages are archived or deleted. If any destination reports a failure — an Elasticsearch outage, an expired Splunk HEC token, an unreachable Kafka broker, a full --output disk — the whole batch is left in the reports folder and retried on the next run or watch-mode check, so a report is never removed from the mailbox while it exists nowhere else (issue #242).

This is all-or-nothing per batch: archiving a batch because most destinations accepted it would still permanently lose the data for the one that didn't. It also applies regardless of fail_on_output_error, which only controls the process exit code. Messages that could not be parsed at all carry no report data, so they are filed under Invalid (or deleted per delete_invalid) as usual.

A destination that is broken rather than briefly unavailable would otherwise be retried forever, so retries are capped. Once a message's batch has failed max_unsaved_retries + 1 times (three times by default), that message is moved to <archive_folder>/Unsaved and stops being retried. A message is never deleted on this path, whatever the delete options say. To recover after fixing the output destination, either move the messages from Archive/Unsaved back into the reports folder, or run parsedmarc once with reports_folder = Archive/Unsaved.

:::{note} The failed-save counts live in memory, so they are counted per parsedmarc process. In watch mode — a long-running process that checks the mailbox repeatedly — the cap works as described across checks. A one-shot run (cron, systemd timers) attempts each message exactly once and then exits, so its counts start over every time and the default cap is never reached: messages simply keep being retried on every run, which is the safe direction. Set max_unsaved_retries = 0 if you want one-shot runs to move unsavable messages to Unsaved immediately instead. A message that crashes parsedmarc while it is being parsed is a different case, counted on disk by attempts_file; see Messages that crash parsedmarc are not retried forever. :::

:::{warning} Retrying a batch means re-sending it. Output destinations that deduplicate — Elasticsearch, OpenSearch, and PostgreSQL, which recognize an already-saved report — are unaffected, and S3 is idempotent because each report is written to an object key built from its type, date, report ID, and a hash of the fields that identify the report, so a retry overwrites the same object. Kafka, Splunk HEC, syslog, GELF, webhooks, Loki, Azure Log Analytics, and the --output JSON/CSV files all append unconditionally, so each retry adds another copy of every report in the batch. That is why the default retry cap is deliberately low: at most three deliveries per report before its message is set aside in Unsaved. The summary email covers everything parsed in a run, including reports whose batch failed to save, so a report retried across runs can also appear in more than one summary email. :::

:::{note} Not every destination can report a failed delivery. The webhook output deliberately logs and swallows its own HTTP and network errors, and the syslog and GELF outputs send through Python logging handlers, which swallow delivery errors by design — so an unreachable webhook, syslog, or GELF endpoint is not treated as a failed save and does not hold a batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC, Kafka, S3, PostgreSQL, Loki, Azure Log Analytics, and the --output files are all detected and do. :::

:::{note} since interacts with retries: a message that ages out of the configured since window stops being fetched, and therefore stops being retried automatically. It is never deleted or moved — it simply stays in the reports folder until it is processed by a run with a wider (or no) since window. :::

Messages that crash parsedmarc are not retried forever

A message is only moved out of the reports folder after it has been parsed. If parsing a message kills the parsedmarc process — the out-of-memory killer, a container memory limit, a crash in a library — the message stays where it is, and every later run, or every restart of the service, fetches it again and dies on it again, so the reports behind it in the mailbox are never processed. parsedmarc rejects the message shapes known to do this, but cannot rule out new ones.

Set [mailbox] attempts_file to a path to guard against that loop:

[mailbox]
attempts_file = /var/lib/parsedmarc/attempts.json

parsedmarc then writes an entry to that file before it parses each message, and removes the entry as soon as the parse returns, whether it found a report or rejected the message as invalid. An entry that is still there on a later run belongs to a message whose parse never finished. Once a message has two such unfinished attempts, the next run logs an error naming the folder, the message and the count, and moves the message to <archive_folder>/Invalid without parsing it (or deletes it, if delete_invalid is set). Its entry is removed once the message has been moved, so moving it back into the reports folder tries it again from the start; if the move fails, the entry stays and the next run tries the move again. A set-aside message whose move failed, or one left in place in test mode, is not recorded again, so its entry is not refreshed and is dropped after 30 days, after which the message is parsed up to twice more before it is set aside again. Two attempts are allowed so that one crash for an unrelated reason (a reboot, a deployment) never sets a message aside. With n_procs above 1, or with parse_timeout set (which parses in worker processes even with n_procs = 1), several messages are parsed at once, so every message that was being parsed or waiting for a worker when a run died has an entry; on the next run each of them is parsed on its own after the rest of its batch, so a second crash counts against the one message that caused it. It is parsed in the main process, or, when parse_timeout is set, in a worker process of its own, so the timeout still applies to it. If a worker process dies while it parses (the out-of-memory killer can end a worker without ending parsedmarc), the batch ends with a logged error. A one-shot run saves the reports parsed from report files and mbox files before it and exits with status 1; for a mailbox, earlier batches are saved and the batch in progress stays in the mailbox. In watch mode the IMAP and Maildir watchers log it as a warning and check the mailbox again, so the process keeps running, while the Microsoft Graph and Gmail watchers exit with status 1. Either way the entries of the messages that were being parsed or waiting for a worker stay in the file, so the next run or check parses each of them on its own in the same way.

This count is separate from the in-memory failed-save counts of max_unsaved_retries above: an entry is removed when the parse returns, before the reports are saved, so a message whose batch could not be saved is retried and moved to Unsaved as described above, never to Invalid. A message is identified by the reports folder, its id and a hash of its beginning, so a new message that reuses an old id starts with no attempts.

Use one file per configuration (per mailbox and reports folder): two parsedmarc processes writing one file overwrite each other's entries. The file is written atomically, readable by its owner only, and entries older than 30 days are dropped. A file that is not a valid attempts file is logged as a warning and replaced. If the file cannot be written, an error is logged and the protection is off for the rest of that batch; reports are still processed. If the write that removes an entry after a successful parse is the one that fails, the entry stays (and the protection is off for the rest of that batch), so a healthy message can be set aside as invalid after two such failures; the write error is logged each time. In test mode messages are recorded and set aside as usual but never moved.

Environment variable configuration

Any configuration option can be set via environment variables using the naming convention PARSEDMARC_{SECTION}_{KEY} (uppercase). This is especially useful for Docker deployments where file permissions make it difficult to use config files for secrets.

Priority order: CLI arguments > environment variables > config file > defaults

Examples

# Set IMAP credentials via env vars
export PARSEDMARC_IMAP_HOST=imap.example.com
export PARSEDMARC_IMAP_USER=dmarc@example.com
export PARSEDMARC_IMAP_PASSWORD=secret

# Elasticsearch
export PARSEDMARC_ELASTICSEARCH_HOSTS=http://localhost:9200
export PARSEDMARC_ELASTICSEARCH_SSL=false

# Splunk HEC (note: section name splunk_hec becomes SPLUNK_HEC)
export PARSEDMARC_SPLUNK_HEC_URL=https://splunk.example.com
export PARSEDMARC_SPLUNK_HEC_TOKEN=my-hec-token
export PARSEDMARC_SPLUNK_HEC_INDEX=email

# General settings
export PARSEDMARC_GENERAL_SAVE_AGGREGATE=true
export PARSEDMARC_GENERAL_DEBUG=true

Specifying the config file via environment variable

export PARSEDMARC_CONFIG_FILE=/etc/parsedmarc.ini
parsedmarc

Running without a config file (env-only mode)

When no config file is given (neither -c flag nor PARSEDMARC_CONFIG_FILE), parsedmarc will still pick up any PARSEDMARC_* environment variables. This enables fully file-less deployments:

export PARSEDMARC_GENERAL_SAVE_AGGREGATE=true
export PARSEDMARC_GENERAL_OFFLINE=true
export PARSEDMARC_ELASTICSEARCH_HOSTS=http://elasticsearch:9200
parsedmarc /path/to/reports/*

Docker Compose example

services:
  parsedmarc:
    image: parsedmarc:latest
    environment:
      PARSEDMARC_IMAP_HOST: imap.example.com
      PARSEDMARC_IMAP_USER: dmarc@example.com
      PARSEDMARC_IMAP_PASSWORD: ${IMAP_PASSWORD}
      PARSEDMARC_MAILBOX_WATCH: "true"
      PARSEDMARC_MAILBOX_ATTEMPTS_FILE: /home/parsedmarc/attempts.json
      PARSEDMARC_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      PARSEDMARC_GENERAL_SAVE_AGGREGATE: "true"
      PARSEDMARC_GENERAL_SAVE_FAILURE: "true"
    volumes:
      - parsedmarc-state:/home/parsedmarc

volumes:
  parsedmarc-state:

The parsedmarc-state volume keeps the attempts file when the container is recreated, as it is on an image upgrade. It is mounted on the home directory of the image's parsedmarc user because Docker gives a new named volume the owner of the directory it is mounted on, and that directory is one the user can write.

Docker secrets (_FILE suffix)

Any PARSEDMARC_{SECTION}_{KEY} environment variable can also be supplied via a file by appending _FILE to its name. The file's contents (with any trailing CR/LF characters stripped) are used as the value. This is the same convention used by the official Postgres, MariaDB, and Redis container images, and is designed to plug straight into Docker / Docker Compose / Kubernetes secrets so credentials never appear in plain environment: blocks (where they would be readable via docker inspect, container logs, and /proc/<pid>/environ).

The bare DEBUG / PARSEDMARC_DEBUG aliases and PARSEDMARC_CONFIG_FILE do not have a _FILE form; only PARSEDMARC_{SECTION}_{KEY} vars resolved to a known config section are eligible.

If both the direct env var and the _FILE variant are set, the _FILE variant wins. If the file does not exist or is unreadable, parsedmarc exits with a configuration error rather than silently falling back to an empty value.

secrets:
  imap_password:
    file: ./secrets/imap_password.txt

services:
  parsedmarc:
    image: parsedmarc:latest
    secrets:
      - imap_password
    environment:
      PARSEDMARC_IMAP_HOST: imap.example.com
      PARSEDMARC_IMAP_USER: dmarc@example.com
      PARSEDMARC_IMAP_PASSWORD_FILE: /run/secrets/imap_password

Note that a small set of config keys whose own names already end in _file ([general] log_file, [msgraph] token_file, [gmail_api] credentials_file, [gmail_api] token_file, [mailbox] attempts_file) keep their pre-existing meaning when set via PARSEDMARC_..._FILE — that env var is the path itself, not a wrapper around a file containing the path. To pass those paths via a Docker secret, double up the suffix (PARSEDMARC_GMAIL_API_CREDENTIALS_FILE_FILE); the inner contents are then read and stored as the credentials_file value.

Section name mapping

For sections with underscores in the name, the full section name is used:

Section	Env var prefix
general	PARSEDMARC_GENERAL_
mailbox	PARSEDMARC_MAILBOX_
imap	PARSEDMARC_IMAP_
msgraph	PARSEDMARC_MSGRAPH_
elasticsearch	PARSEDMARC_ELASTICSEARCH_
opensearch	PARSEDMARC_OPENSEARCH_
splunk_hec	PARSEDMARC_SPLUNK_HEC_
kafka	PARSEDMARC_KAFKA_
smtp	PARSEDMARC_SMTP_
s3	PARSEDMARC_S3_
postgresql	PARSEDMARC_POSTGRESQL_
syslog	PARSEDMARC_SYSLOG_
gmail_api	PARSEDMARC_GMAIL_API_
maildir	PARSEDMARC_MAILDIR_
log_analytics	PARSEDMARC_LOG_ANALYTICS_
gelf	PARSEDMARC_GELF_
webhook	PARSEDMARC_WEBHOOK_
loki	PARSEDMARC_LOKI_

Using parsedmarc as a library

parsedmarc is also importable as a regular Python package, not just a CLI tool. The main entry points — parse_report_file(), parse_aggregate_report_xml(), parse_aggregate_report_file(), parse_failure_report(), parse_report_email(), get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox(), and watch_inbox() — are all importable directly from the parsedmarc package. See the API reference for the full set of modules and members.

Each of these functions accepts either individual option keyword arguments (offline, nameservers, dns_timeout, etc.) or a single config= keyword argument carrying a ParserConfig instance:

from parsedmarc import ParserConfig, parse_report_file, get_dmarc_reports_from_mailbox
from parsedmarc.mail import IMAPConnection

config = ParserConfig(
    offline=False,
    nameservers=["1.1.1.1", "1.0.0.1"],
    dns_timeout=5.0,
)

report = parse_report_file("aggregate_report.xml.gz", config=config)

connection = IMAPConnection(
    host="imap.example.com", user="dmarc@example.com", password="..."
)
results = get_dmarc_reports_from_mailbox(connection, config=config)

A few things to keep in mind:

• When config= is passed, the individual option keyword arguments are ignored in favor of the values carried on the ParserConfig instance.
• Each explicitly constructed ParserConfig owns its own isolated caches (IP address info, seen aggregate report IDs, and the reverse DNS map). Omitting config= falls back to the process-wide caches shared by every call that doesn't pass one.
• keep_alive, n_procs, parse_timeout and attempts_file are not part of ParserConfig — they control process/worker or mailbox orchestration rather than parsing or enrichment behavior, so they are always passed as separate keyword arguments.

Performance tuning

For large mailbox imports or backfills, parsedmarc can consume a noticeable amount of memory, especially when it runs on the same host as Elasticsearch or OpenSearch. The following settings can reduce peak memory usage and make long imports more predictable:

• Reduce mailbox.batch_size to smaller values such as 100-500 instead of processing a very large message set at once. Smaller batches trade throughput for lower peak memory use and less sink pressure.
• n_procs now parallelizes parsing for mailbox and mbox runs too, not just report files passed directly as CLI arguments. It pays off most when a run is bound by DNS/GeoIP enrichment rather than fetching or output. The trade-off is memory and DNS load: at most roughly 2 * n_procs messages are held in flight at once, each worker process keeps its own DNS/GeoIP cache, and DNS query volume can multiply by up to n_procs.
• Set parse_timeout (for example to 300) so that one message or file that takes far too long to parse is set aside as invalid instead of holding up the rest of the run.
• Use mailbox.since to process reports in smaller time windows such as 1d, 7d, or another interval that fits the backlog. This makes it easier to catch up incrementally instead of loading an entire mailbox history in one run.
• Set strip_attachment_payloads = True when failure reports contain large attachments and you do not need to retain the raw payloads in the parsed output.
• Prefer running parsedmarc separately from Elasticsearch or OpenSearch, or reserve enough RAM for both services if they must share a host.
• For very large imports, prefer incremental supervised runs, such as a scheduler or systemd service, over infrequent massive backfills.

These are operational tuning recommendations rather than hard requirements, but they are often enough to avoid memory pressure and reduce failures during high-volume mailbox processing.

Multi-tenant support

Starting in 8.19.0, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:

example:
  - example.com
  - example.net
  - example.org

whalensolutions:
  - whalensolutions.com

Save it to disk where the user running ParseDMARC can read it, then set index_prefix_domain_map to that filepath in the [general] section of the ParseDMARC configuration file and do not set an index_prefix option in the [elasticsearch] or [opensearch] sections.

When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.

:::{note} A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used. :::

Each key must be a tenant name and each value a list of domain names, all strings; a file of any other shape is rejected at startup.

The index migrations and backfills that run at startup cover every tenant prefix in the map, in addition to the unprefixed indexes that hold reports for domains the map does not list. A configuration reload (SIGHUP) re-reads the map, so a newly onboarded tenant is covered without restarting parsedmarc. See Backfilling the combined DKIM/SPF result fields for the details.

Running parsedmarc as a systemd service

Use systemd to run parsedmarc as a service and process reports as they arrive.

This assumes parsedmarc has been installed into /opt/parsedmarc/venv under a parsedmarc system user, as described in Installing parsedmarc.

Protect the parsedmarc configuration file from prying eyes

sudo chown root:parsedmarc /etc/parsedmarc.ini
sudo chmod u=rw,g=r,o= /etc/parsedmarc.ini

Create the service configuration file

sudo nano /etc/systemd/system/parsedmarc.service

[Unit]
Description=parsedmarc mailbox watcher
Documentation=https://domainaware.github.io/parsedmarc/
Wants=network-online.target
After=network.target network-online.target elasticsearch.service

[Service]
ExecStart=/opt/parsedmarc/venv/bin/parsedmarc -c /etc/parsedmarc.ini
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=60
User=parsedmarc
Group=parsedmarc
StateDirectory=parsedmarc
Restart=always
RestartSec=5m

[Install]
WantedBy=multi-user.target

KillMode=mixed sends the stop signal to the main parsedmarc process only, so the batch in progress can finish before it exits, within TimeoutStopSec, after which systemd kills the whole group; the default control-group mode would also stop its parsing worker processes at once and abandon the batch.

StateDirectory=parsedmarc makes systemd create /var/lib/parsedmarc, owned by the parsedmarc user, before the service starts. Point the attempts file there in /etc/parsedmarc.ini, so a message that kills the service is not parsed again on every restart:

[mailbox]
attempts_file = /var/lib/parsedmarc/attempts.json

Then, enable the service

sudo systemctl daemon-reload
sudo systemctl enable parsedmarc.service
sudo service parsedmarc restart

:::{note} You must also run the above commands whenever you edit parsedmarc.service. :::

:::{warning} Always restart the service every time you upgrade to a new version of parsedmarc:

sudo service parsedmarc restart

:::

:::{note} On systemctl stop/restart (or Ctrl-C) parsedmarc finishes the current batch, flushes its outputs, and exits cleanly. Shutdown is observed at batch boundaries, so the worst-case delay is roughly [mailbox] check_timeout (default 30s) plus the batch's processing and flush time. Keep TimeoutStopSec comfortably above check_timeout (≈2×, and raise both together) or systemd will SIGKILL mid-batch. In the foreground, a second Ctrl-C force-quits immediately, skipping the output flush. :::

Reloading configuration without restarting

When running in watch mode, parsedmarc supports reloading its configuration file without restarting the service or interrupting report processing that is already in progress. Send a SIGHUP signal to the process, or use systemctl reload if the unit file includes the ExecReload line shown above:

sudo systemctl reload parsedmarc

The reload takes effect after the current batch of reports finishes processing and all output operations (Elasticsearch, Kafka, S3, etc.) for that batch have completed. The following settings are reloaded:

• All output destinations (Elasticsearch, OpenSearch, Kafka, S3, Splunk, syslog, GELF, webhooks, Log Analytics)
• Multi-tenant index prefix domain map (index_prefix_domain_map — the referenced YAML file is re-read on reload)
• DNS and GeoIP settings (nameservers, dns_timeout, ip_db_path, ipinfo_url, offline, etc.)
• Processing flags (strip_attachment_payloads, batch_size, check_timeout, parse_timeout, attempts_file, etc.)
• Log level (debug, verbose, warnings, silent)

Mailbox connection settings (IMAP host/credentials, Microsoft Graph, Gmail API, Maildir path) are not reloaded — changing those still requires a full restart.

On a successful reload, the output clients for the updated configuration are created first, and the connections they replace are closed once the new configuration is live. The service then resumes watching with the new settings.

If the new configuration file contains errors (missing required settings, an output client that cannot be built — a PostgreSQL server, Kafka broker, or TCP/TLS syslog server that cannot be reached, or an [opensearch] auth_type that is not supported — an unreadable reverse DNS map or PSL overrides file, a log_file that cannot be opened for writing, a non-numeric batch_size, etc.), the entire reload is aborted — any output clients built for the new configuration are closed again, and the previous configuration remains fully active: the old output clients stay open and connected, and the reverse DNS map, the PSL overrides, the IP database selection, and every other setting keep the values they had before the SIGHUP. This means a typo in one section will not take down an otherwise working setup. Unlike startup, where an unwritable log_file is only a warning, on reload it is one of the errors that abort the reload, so the previous log file keeps receiving logs — including the one explaining why. A log_file that could not be opened when parsedmarc started (a warning at startup) is retried by the next reload even if the setting is unchanged. (One thing an aborted reload does not put back: if it got as far as downloading a new IP database from a changed ipinfo_url, that file stays in the shared cache directory. Which database parsedmarc uses is unchanged, and the cached file is the same one the next restart would download.) Elasticsearch and OpenSearch are the exception to "cannot be reached": their clients connect lazily and the index migration only warns when the cluster is unreachable, so a reload that points at an unreachable cluster still commits, and the failure shows up on the first save instead. Check the logs for details:

journalctl -u parsedmarc.service -r

To check the status of the service, run:

service parsedmarc status

:::{note} In the event of a crash, systemd will restart the service after 5 minutes (per the RestartSec setting above), but the service parsedmarc status command will only show the logs for the current process. To view the logs for previous runs as well as the current process (newest to oldest), run:

journalctl -u parsedmarc.service -r

:::