Files
parsedmarc/docs/source/mailing-lists.md
T
07bca1ad28 Make the output and mailbox integrations optional extras (#888)
* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-08-28 17:33:18 -04:00

207 lines
6.7 KiB
Markdown

## What about mailing lists?
When you deploy DMARC on your domain, you might find that messages
relayed by mailing lists are failing DMARC, most likely because the mailing
list is spoofing your from address, and modifying the subject,
footer, or other part of the message, thereby breaking the
DKIM signature.
### Mailing list best practices
Ideally, a mailing list should forward messages without altering the
headers or body content at all. [Joe Nelson] does a fantastic job of
explaining exactly what mailing lists should and shouldn't do to be
fully DMARC compliant. Rather than repeat his fine work, here's a
summary:
#### Do
- Retain headers from the original message
- Add [RFC 2369] List-Unsubscribe headers to outgoing messages, instead of
adding unsubscribe links to the body
> List-Unsubscribe: <https://list.example.com/unsubscribe-link>
- Add [RFC 2919] List-Id headers instead of modifying the subject
> List-Id: Example Mailing List <list.example.com>
Modern mail clients and webmail services generate unsubscribe buttons based on
these headers.
#### Do not
- Remove or modify any existing headers from the original message, including
From, Date, Subject, etc.
- Add to or remove content from the message body, **including traditional
disclaimers and unsubscribe footers**
In addition to complying with DMARC, this configuration ensures that Reply
and Reply All actions work like they would with any email message. Reply
replies to the message sender, and Reply All replies to the sender and the
list.
Even without a subject prefix or body footer, mailing list users can still
tell that a message came from the mailing list, because the message was sent
to the mailing list post address, and not their email address.
Configuration steps for common mailing list platforms are listed below.
#### Mailman 2
Navigate to General Settings, and configure the settings below
```{eval-rst}
============================ ==========
**Setting** **Value**
**subject_prefix**
**from_is_list** No
**first_strip_reply_to** No
**reply_goes_to_list** Poster
**include_rfc2369_headers** Yes
**include_list_post_header** Yes
**include_sender_header** No
============================ ==========
```
Navigate to Non-digest options, and configure the settings below
```{eval-rst}
=================== ==========
**Setting** **Value**
**msg_header**
**msg_footer**
**scrub_nondigest** No
=================== ==========
```
Navigate to Privacy Options> Sending Filters, and configure the settings below
```{eval-rst}
====================================== ==========
**Setting** **Value**
**dmarc_moderation_action** Accept
**dmarc_quarantine_moderation_action** Yes
**dmarc_none_moderation_action** Yes
====================================== ==========
```
#### Mailman 3
Navigate to Settings> List Identity
Make Subject prefix blank.
Navigate to Settings> Alter Messages
Configure the settings below
```{eval-rst}
====================================== ==========
**Setting** **Value**
**Convert html to plaintext** No
**Include RFC2369 headers** Yes
**Include the list post header** Yes
**Explicit reply-to address**
**First strip replyto** No
**Reply goes to list** No munging
====================================== ==========
```
Navigate to Settings> DMARC Mitigation
Configure the settings below
```{eval-rst}
================================== ===============================
**Setting** **Value**
**DMARC mitigation action** No DMARC mitigations
**DMARC mitigate unconditionally** No
================================== ===============================
```
Create a blank footer template for your mailing list to remove the message
footer. Unfortunately, the Postorius mailing list admin UI will not allow you
to create an empty template, so you'll have to create one using the system's
command line instead, for example:
```bash
touch var/templates/lists/list.example.com/en/list:member:regular:footer
```
Where `list.example.com` is the list ID, and `en` is the language.
Then restart mailman core.
#### LISTSERV
[LISTSERV 16.0-2017a] and higher will rewrite the From header for domains
that enforce with a DMARC quarantine or reject policy.
Some additional steps are needed for Linux hosts.
#### Workarounds
If a mailing list must go **against** best practices and
modify the message (e.g. to add a required legal footer), the mailing
list administrator must configure the list to replace the From address of the
message (also known as munging) with the address of the mailing list, so they
no longer spoof email addresses with domains protected by DMARC.
Configuration steps for common mailing list platforms are listed below.
##### Mailman 2
Navigate to Privacy Options> Sending Filters, and configure the settings below
```{eval-rst}
====================================== ==========
**Setting** **Value**
**dmarc_moderation_action** Munge From
**dmarc_quarantine_moderation_action** Yes
**dmarc_none_moderation_action** Yes
====================================== ==========
```
:::{note}
Message wrapping could be used as the DMARC mitigation action instead. In
that case, the original message is added as an attachment to the mailing
list message, but that could interfere with inbox searching, or mobile
clients.
On the other hand, replacing the From address might cause users to
accidentally reply to the entire list, when they only intended to reply to
the original sender.
Choose the option that best fits your community.
:::
##### Mailman 3
In the DMARC Mitigations tab of the Settings page, configure the settings below
```{eval-rst}
================================== ===============================
**Setting** **Value**
**DMARC mitigation action** Replace From: with list address
**DMARC mitigate unconditionally** No
================================== ===============================
```
:::{note}
Message wrapping could be used as the DMARC mitigation action instead. In
that case, the original message is added as an attachment to the mailing
list message, but that could interfere with inbox searching, or mobile
clients.
On the other hand, replacing the From address might cause users to
accidentally reply to the entire list, when they only intended to reply to
the original sender.
:::
[joe nelson]: https://begriffs.com/posts/2018-09-18-dmarc-mailing-list.html
[listserv 16.0-2017a]: https://www.lsoft.com/news/dmarc-issue1-2018.asp
[rfc 2369]: https://tools.ietf.org/html/rfc2369
[rfc 2919]: https://tools.ietf.org/html/rfc2919