* Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
6.7 KiB
What about mailing lists?
When you deploy DMARC on your domain, you might find that messages relayed by mailing lists are failing DMARC, most likely because the mailing list is spoofing your from address, and modifying the subject, footer, or other part of the message, thereby breaking the DKIM signature.
Mailing list best practices
Ideally, a mailing list should forward messages without altering the headers or body content at all. Joe Nelson does a fantastic job of explaining exactly what mailing lists should and shouldn't do to be fully DMARC compliant. Rather than repeat his fine work, here's a summary:
Do
-
Retain headers from the original message
-
Add RFC 2369 List-Unsubscribe headers to outgoing messages, instead of adding unsubscribe links to the body
List-Unsubscribe: https://list.example.com/unsubscribe-link
-
Add RFC 2919 List-Id headers instead of modifying the subject
List-Id: Example Mailing List <list.example.com>
Modern mail clients and webmail services generate unsubscribe buttons based on these headers.
Do not
- Remove or modify any existing headers from the original message, including From, Date, Subject, etc.
- Add to or remove content from the message body, including traditional disclaimers and unsubscribe footers
In addition to complying with DMARC, this configuration ensures that Reply and Reply All actions work like they would with any email message. Reply replies to the message sender, and Reply All replies to the sender and the list.
Even without a subject prefix or body footer, mailing list users can still tell that a message came from the mailing list, because the message was sent to the mailing list post address, and not their email address.
Configuration steps for common mailing list platforms are listed below.
Mailman 2
Navigate to General Settings, and configure the settings below
============================ ==========
**Setting** **Value**
**subject_prefix**
**from_is_list** No
**first_strip_reply_to** No
**reply_goes_to_list** Poster
**include_rfc2369_headers** Yes
**include_list_post_header** Yes
**include_sender_header** No
============================ ==========
Navigate to Non-digest options, and configure the settings below
=================== ==========
**Setting** **Value**
**msg_header**
**msg_footer**
**scrub_nondigest** No
=================== ==========
Navigate to Privacy Options> Sending Filters, and configure the settings below
====================================== ==========
**Setting** **Value**
**dmarc_moderation_action** Accept
**dmarc_quarantine_moderation_action** Yes
**dmarc_none_moderation_action** Yes
====================================== ==========
Mailman 3
Navigate to Settings> List Identity
Make Subject prefix blank.
Navigate to Settings> Alter Messages
Configure the settings below
====================================== ==========
**Setting** **Value**
**Convert html to plaintext** No
**Include RFC2369 headers** Yes
**Include the list post header** Yes
**Explicit reply-to address**
**First strip replyto** No
**Reply goes to list** No munging
====================================== ==========
Navigate to Settings> DMARC Mitigation
Configure the settings below
================================== ===============================
**Setting** **Value**
**DMARC mitigation action** No DMARC mitigations
**DMARC mitigate unconditionally** No
================================== ===============================
Create a blank footer template for your mailing list to remove the message footer. Unfortunately, the Postorius mailing list admin UI will not allow you to create an empty template, so you'll have to create one using the system's command line instead, for example:
touch var/templates/lists/list.example.com/en/list:member:regular:footer
Where list.example.com is the list ID, and en is the language.
Then restart mailman core.
LISTSERV
LISTSERV 16.0-2017a and higher will rewrite the From header for domains that enforce with a DMARC quarantine or reject policy.
Some additional steps are needed for Linux hosts.
Workarounds
If a mailing list must go against best practices and modify the message (e.g. to add a required legal footer), the mailing list administrator must configure the list to replace the From address of the message (also known as munging) with the address of the mailing list, so they no longer spoof email addresses with domains protected by DMARC.
Configuration steps for common mailing list platforms are listed below.
Mailman 2
Navigate to Privacy Options> Sending Filters, and configure the settings below
====================================== ==========
**Setting** **Value**
**dmarc_moderation_action** Munge From
**dmarc_quarantine_moderation_action** Yes
**dmarc_none_moderation_action** Yes
====================================== ==========
:::{note} Message wrapping could be used as the DMARC mitigation action instead. In that case, the original message is added as an attachment to the mailing list message, but that could interfere with inbox searching, or mobile clients.
On the other hand, replacing the From address might cause users to accidentally reply to the entire list, when they only intended to reply to the original sender.
Choose the option that best fits your community. :::
Mailman 3
In the DMARC Mitigations tab of the Settings page, configure the settings below
================================== ===============================
**Setting** **Value**
**DMARC mitigation action** Replace From: with list address
**DMARC mitigate unconditionally** No
================================== ===============================
:::{note} Message wrapping could be used as the DMARC mitigation action instead. In that case, the original message is added as an attachment to the mailing list message, but that could interfere with inbox searching, or mobile clients.
On the other hand, replacing the From address might cause users to accidentally reply to the entire list, when they only intended to reply to the original sender. :::