mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-09-06 14:07:59 +00:00
* Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
207 lines
6.7 KiB
Markdown
207 lines
6.7 KiB
Markdown
## What about mailing lists?
|
|
|
|
When you deploy DMARC on your domain, you might find that messages
|
|
relayed by mailing lists are failing DMARC, most likely because the mailing
|
|
list is spoofing your from address, and modifying the subject,
|
|
footer, or other part of the message, thereby breaking the
|
|
DKIM signature.
|
|
|
|
### Mailing list best practices
|
|
|
|
Ideally, a mailing list should forward messages without altering the
|
|
headers or body content at all. [Joe Nelson] does a fantastic job of
|
|
explaining exactly what mailing lists should and shouldn't do to be
|
|
fully DMARC compliant. Rather than repeat his fine work, here's a
|
|
summary:
|
|
|
|
#### Do
|
|
|
|
- Retain headers from the original message
|
|
|
|
- Add [RFC 2369] List-Unsubscribe headers to outgoing messages, instead of
|
|
adding unsubscribe links to the body
|
|
|
|
> List-Unsubscribe: <https://list.example.com/unsubscribe-link>
|
|
|
|
- Add [RFC 2919] List-Id headers instead of modifying the subject
|
|
|
|
> List-Id: Example Mailing List <list.example.com>
|
|
|
|
Modern mail clients and webmail services generate unsubscribe buttons based on
|
|
these headers.
|
|
|
|
#### Do not
|
|
|
|
- Remove or modify any existing headers from the original message, including
|
|
From, Date, Subject, etc.
|
|
- Add to or remove content from the message body, **including traditional
|
|
disclaimers and unsubscribe footers**
|
|
|
|
In addition to complying with DMARC, this configuration ensures that Reply
|
|
and Reply All actions work like they would with any email message. Reply
|
|
replies to the message sender, and Reply All replies to the sender and the
|
|
list.
|
|
|
|
Even without a subject prefix or body footer, mailing list users can still
|
|
tell that a message came from the mailing list, because the message was sent
|
|
to the mailing list post address, and not their email address.
|
|
|
|
Configuration steps for common mailing list platforms are listed below.
|
|
|
|
#### Mailman 2
|
|
|
|
Navigate to General Settings, and configure the settings below
|
|
|
|
```{eval-rst}
|
|
============================ ==========
|
|
**Setting** **Value**
|
|
**subject_prefix**
|
|
**from_is_list** No
|
|
**first_strip_reply_to** No
|
|
**reply_goes_to_list** Poster
|
|
**include_rfc2369_headers** Yes
|
|
**include_list_post_header** Yes
|
|
**include_sender_header** No
|
|
============================ ==========
|
|
```
|
|
|
|
Navigate to Non-digest options, and configure the settings below
|
|
|
|
```{eval-rst}
|
|
=================== ==========
|
|
**Setting** **Value**
|
|
**msg_header**
|
|
**msg_footer**
|
|
**scrub_nondigest** No
|
|
=================== ==========
|
|
```
|
|
|
|
Navigate to Privacy Options> Sending Filters, and configure the settings below
|
|
|
|
```{eval-rst}
|
|
====================================== ==========
|
|
**Setting** **Value**
|
|
**dmarc_moderation_action** Accept
|
|
**dmarc_quarantine_moderation_action** Yes
|
|
**dmarc_none_moderation_action** Yes
|
|
====================================== ==========
|
|
```
|
|
|
|
#### Mailman 3
|
|
|
|
Navigate to Settings> List Identity
|
|
|
|
Make Subject prefix blank.
|
|
|
|
Navigate to Settings> Alter Messages
|
|
|
|
Configure the settings below
|
|
|
|
```{eval-rst}
|
|
====================================== ==========
|
|
**Setting** **Value**
|
|
**Convert html to plaintext** No
|
|
**Include RFC2369 headers** Yes
|
|
**Include the list post header** Yes
|
|
**Explicit reply-to address**
|
|
**First strip replyto** No
|
|
**Reply goes to list** No munging
|
|
====================================== ==========
|
|
```
|
|
|
|
Navigate to Settings> DMARC Mitigation
|
|
|
|
Configure the settings below
|
|
|
|
```{eval-rst}
|
|
================================== ===============================
|
|
**Setting** **Value**
|
|
**DMARC mitigation action** No DMARC mitigations
|
|
**DMARC mitigate unconditionally** No
|
|
================================== ===============================
|
|
```
|
|
|
|
Create a blank footer template for your mailing list to remove the message
|
|
footer. Unfortunately, the Postorius mailing list admin UI will not allow you
|
|
to create an empty template, so you'll have to create one using the system's
|
|
command line instead, for example:
|
|
|
|
```bash
|
|
touch var/templates/lists/list.example.com/en/list:member:regular:footer
|
|
```
|
|
|
|
Where `list.example.com` is the list ID, and `en` is the language.
|
|
|
|
Then restart mailman core.
|
|
|
|
#### LISTSERV
|
|
|
|
[LISTSERV 16.0-2017a] and higher will rewrite the From header for domains
|
|
that enforce with a DMARC quarantine or reject policy.
|
|
|
|
Some additional steps are needed for Linux hosts.
|
|
|
|
#### Workarounds
|
|
|
|
If a mailing list must go **against** best practices and
|
|
modify the message (e.g. to add a required legal footer), the mailing
|
|
list administrator must configure the list to replace the From address of the
|
|
message (also known as munging) with the address of the mailing list, so they
|
|
no longer spoof email addresses with domains protected by DMARC.
|
|
|
|
Configuration steps for common mailing list platforms are listed below.
|
|
|
|
##### Mailman 2
|
|
|
|
Navigate to Privacy Options> Sending Filters, and configure the settings below
|
|
|
|
```{eval-rst}
|
|
====================================== ==========
|
|
**Setting** **Value**
|
|
**dmarc_moderation_action** Munge From
|
|
**dmarc_quarantine_moderation_action** Yes
|
|
**dmarc_none_moderation_action** Yes
|
|
====================================== ==========
|
|
```
|
|
|
|
:::{note}
|
|
Message wrapping could be used as the DMARC mitigation action instead. In
|
|
that case, the original message is added as an attachment to the mailing
|
|
list message, but that could interfere with inbox searching, or mobile
|
|
clients.
|
|
|
|
On the other hand, replacing the From address might cause users to
|
|
accidentally reply to the entire list, when they only intended to reply to
|
|
the original sender.
|
|
|
|
Choose the option that best fits your community.
|
|
:::
|
|
|
|
##### Mailman 3
|
|
|
|
In the DMARC Mitigations tab of the Settings page, configure the settings below
|
|
|
|
```{eval-rst}
|
|
================================== ===============================
|
|
**Setting** **Value**
|
|
**DMARC mitigation action** Replace From: with list address
|
|
**DMARC mitigate unconditionally** No
|
|
================================== ===============================
|
|
```
|
|
|
|
:::{note}
|
|
Message wrapping could be used as the DMARC mitigation action instead. In
|
|
that case, the original message is added as an attachment to the mailing
|
|
list message, but that could interfere with inbox searching, or mobile
|
|
clients.
|
|
|
|
On the other hand, replacing the From address might cause users to
|
|
accidentally reply to the entire list, when they only intended to reply to
|
|
the original sender.
|
|
:::
|
|
|
|
[joe nelson]: https://begriffs.com/posts/2018-09-18-dmarc-mailing-list.html
|
|
[listserv 16.0-2017a]: https://www.lsoft.com/news/dmarc-issue1-2018.asp
|
|
[rfc 2369]: https://tools.ietf.org/html/rfc2369
|
|
[rfc 2919]: https://tools.ietf.org/html/rfc2919
|