Make the output and mailbox integrations optional extras (#888)

* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

.github/workflows/docs.yml

@@ -38,7 +38,10 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install .[build]
+          # all extra included so docs/source/api.md can autodoc
+          # parsedmarc.elastic and parsedmarc.opensearch, which import
+          # their SDKs at module level
+          pip install .[build,all]
 
       - name: Build docs
         run: make -C docs html

.github/workflows/python-tests.yml

@@ -23,9 +23,12 @@ jobs:
     - name: Install Python dependencies
       run: |
         python -m pip install --upgrade pip
-        # postgresql extra included so pyright can resolve the optional
-        # psycopg import in parsedmarc/postgres.py
-        pip install .[build,postgresql]
+        # all and postgresql extras included so pyright can resolve every
+        # optional import (parsedmarc/cli.py's guarded output modules, the
+        # psycopg import in parsedmarc/postgres.py) and so the docs build
+        # step below can autodoc parsedmarc.elastic and
+        # parsedmarc.opensearch, which import their SDKs at module level
+        pip install .[build,all,postgresql]
     - name: Check code style
       run: |
         ruff check .
@@ -77,13 +80,17 @@ jobs:
     - name: Install Python dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install .[build]
+        # all extra included because the output-module tests import their
+        # SDKs at module level. postgresql is deliberately left out so
+        # tests/test_postgres.py's absent-psycopg arm stays exercised.
+        pip install .[build,all]
     - name: Run unit tests
       run: |
         python -m pytest --cov --cov-report=xml --junitxml=junit.xml -o junit_family=legacy tests/
     - name: Test sample DMARC reports
       run: |
-        pip install -e .
+        # [all] because ci.ini configures an [elasticsearch] output
+        pip install -e .[all]
         parsedmarc --debug -c ci.ini samples/aggregate/*
         parsedmarc --debug -c ci.ini samples/failure/*
     - name: Test building packages

AGENTS.md

@@ -9,8 +9,8 @@ parsedmarc is a Python module and CLI utility for parsing DMARC aggregate (RUA),
 ## Common Commands
 
 ```bash
-# Install with dev/build dependencies
-pip install .[build]
+# Install with dev/build dependencies and every optional integration
+pip install ".[build,all,postgresql]"
 
 # Run all tests with coverage
 pytest --cov --cov-report=xml tests/
@@ -26,8 +26,8 @@ ruff check .
 ruff format .
 
 # Type check (config in pyproject.toml [tool.pyright]; CI enforces zero
-# errors/warnings; needs the [postgresql] extra installed so the optional
-# psycopg import resolves)
+# errors/warnings; needs the [all] and [postgresql] extras installed so the
+# optional integration imports resolve)
 pyright
 
 # Test CLI with sample reports

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## Unreleased
+
+### Changes
+
+- **Breaking: the output and mailbox integrations are now optional extras, so `pip install parsedmarc` installs the parsing core and a working core CLI instead of every SDK** ([#883](https://github.com/domainaware/parsedmarc/issues/883)). A 10.x install pulled in the Elasticsearch, OpenSearch, Kafka, AWS (boto3), Azure, Gmail, and Microsoft Graph client libraries whether or not a deployment used any of them — roughly 1 GB of site-packages against roughly 300 MB without — which is a lot to ask of the motivating case, running parsedmarc as a parsing library on a small mail host. The base install now covers the parsing library plus the CLI reading from files, IMAP, Maildir, and mbox, and writing CSV/JSON, Splunk HEC, webhook, and syslog output; those outputs need only `httpx` and the standard library, so they deliberately have no extra of their own. Everything else moves behind an extra: `elastic` (Elasticsearch), `opensearch` (OpenSearch, including the boto3 SigV4 signer), `kafka`, `s3`, `gelf`, `loganalytics` (Azure Monitor), `msgraph` (Microsoft 365 mailboxes), and `gmail` (Gmail API mailboxes), joining the `postgresql` extra that already existed. **CLI users upgrading from 10.x should switch their upgrade command to `pip install -U "parsedmarc[all]"`** — the new umbrella extra — to keep every integration available; add `postgresql` to the list (`parsedmarc[all,postgresql]`) if the PostgreSQL backend is in use. `all` deliberately excludes `postgresql` because `psycopg`'s prebuilt binary wheels do not exist for every platform, and `pip install parsedmarc[all]` must not fail on a platform they do not cover. Users of the prebuilt Docker image (`ghcr.io/domainaware/parsedmarc`) see no change at all: the image now installs `[all,postgresql]`, so it still bundles every integration. A configuration section whose extra is missing no longer fails with an `ImportError` traceback at startup; it fails fast with a `ConfigurationError` naming both the section and the command that fixes it, e.g. `The [elasticsearch] configuration section requires the elastic extra: pip install parsedmarc[elastic]`. Finally, the never-imported `dateparser` dependency is dropped in favor of declaring `python-dateutil`, which `parsedmarc.utils` actually imports and which used to arrive only transitively through `dateparser`.
+
 ## 10.5.0
 
 ### New features

CONTRIBUTING.md

@@ -10,16 +10,22 @@ Use a virtual environment for local development.
 python3 -m venv .venv
 . .venv/bin/activate
 python -m pip install --upgrade pip
-pip install .[build]
+pip install ".[build,all,postgresql]"
 ```
 
+The `all` and `postgresql` extras are what the CI lint job installs: they
+pull in every optional integration, so `pyright`, the docs build, and the
+full test suite can all resolve the optional imports.
+
 ## Before opening a pull request
 
 Run the checks that match your change:
 
 ```bash
 ruff check .
-pytest --cov --cov-report=xml tests.py
+ruff format --check .
+pyright
+pytest --cov --cov-report=xml tests/
 ```
 
 If you changed documentation:

Dockerfile

@@ -27,14 +27,18 @@ COPY --from=build /app/dist/*.whl /tmp/dist/
 RUN set -ex; \
     groupadd --gid ${USER_GID} ${USERNAME}; \
     useradd --uid ${USER_UID} --gid ${USER_GID} -m ${USERNAME}; \
-    # Install the wheel with the [postgresql] extra so the prebuilt image
-    # ships the PostgreSQL output backend (psycopg). Resolve the globbed wheel
-    # path into a variable first: `*.whl[postgresql]` would otherwise be parsed
-    # as a shell bracket glob rather than a pip extras spec. psycopg[binary]
-    # has prebuilt manylinux wheels for both amd64 and arm64, so this adds no
-    # source-build step on either platform.
+    # Install the wheel with the [all] and [postgresql] extras so the prebuilt
+    # image ships every output and mailbox integration, including the
+    # PostgreSQL backend (psycopg). The image deliberately bundles everything:
+    # container users see no change across the packaging split that made
+    # `pip install parsedmarc` the parsing core plus the base CLI.
+    # Resolve the globbed wheel path into a variable first:
+    # `*.whl[all,postgresql]` would otherwise be parsed as a shell bracket
+    # glob rather than a pip extras spec. psycopg[binary] has prebuilt
+    # manylinux wheels for both amd64 and arm64, so this adds no source-build
+    # step on either platform.
     whl="$(ls /tmp/dist/*.whl)"; \
-    pip install "${whl}[postgresql]"; \
+    pip install "${whl}[all,postgresql]"; \
     rm -rf /tmp/dist
 
 USER $USERNAME

README.md

@@ -13,10 +13,11 @@ Package](https://img.shields.io/pypi/v/parsedmarc.svg)](https://pypi.org/project
 </p>
 
 `parsedmarc` is a Python module and CLI utility for parsing DMARC
-reports. When used with Elasticsearch and Kibana (or Splunk), it works
-as a self-hosted open-source alternative to commercial DMARC report
-processing services such as Agari Brand Protection, Dmarcian, OnDMARC,
-ProofPoint Email Fraud Defense, and Valimail.
+reports. When used with Elasticsearch and Kibana (or Splunk), or with
+OpenSearch and Grafana, it works as a self-hosted open-source
+alternative to commercial DMARC report processing services such as
+Agari Brand Protection, Dmarcian, OnDMARC, ProofPoint Email Fraud
+Defense, and Valimail.
 
 > [!NOTE]
 > __Domain-based Message Authentication, Reporting, and Conformance__ (DMARC) is an email authentication protocol.

dashboard-dev-bootstrap.sh

@@ -220,13 +220,15 @@ else
     fi
 
     # Resolve a Python environment for the seed and make sure parsedmarc plus
-    # the PostgreSQL extra (psycopg) are installed in it, so the same run can
-    # populate Postgres. Precedence:
+    # every output integration ([all]) and the PostgreSQL extra (psycopg) are
+    # installed in it, so the same run can populate Elasticsearch, OpenSearch,
+    # and Postgres. Precedence:
     #   1. An explicit PARSEDMARC_BIN — used as-is, nothing installed.
     #   2. An already-activated virtualenv ($VIRTUAL_ENV).
     #   3. An existing repo venv/ or .venv/.
     #   4. Otherwise a freshly created $REPO_ROOT/venv.
-    # Cases 2-4 run `pip install -e .[postgresql]` only when the CLI or psycopg
+    # Cases 2-4 run `pip install -e ".[all,postgresql]"` only when the CLI,
+    # psycopg, or an [all] integration (elasticsearch stands in for the group)
     # is missing, so it's a no-op once the environment is set up.
     if [ -n "${PARSEDMARC_BIN:-}" ]; then
         if [ ! -x "$PARSEDMARC_BIN" ]; then
@@ -251,9 +253,9 @@ else
         fi
         PARSEDMARC_BIN="$seed_venv/bin/parsedmarc"
         if [ ! -x "$PARSEDMARC_BIN" ] ||
-            ! "$seed_venv/bin/python" -c 'import psycopg' >/dev/null 2>&1; then
-            echo "  installing parsedmarc[postgresql] into $seed_venv"
-            "$seed_venv/bin/python" -m pip install -q -e "${REPO_ROOT}[postgresql]"
+            ! "$seed_venv/bin/python" -c 'import psycopg, elasticsearch' >/dev/null 2>&1; then
+            echo "  installing parsedmarc[all,postgresql] into $seed_venv"
+            "$seed_venv/bin/python" -m pip install -q -e "${REPO_ROOT}[all,postgresql]"
         fi
     fi
     if [ ! -x "$PARSEDMARC_BIN" ]; then
@@ -309,7 +311,16 @@ else
         # Reached only for an explicit PARSEDMARC_BIN whose env lacks psycopg
         # (the auto-resolved venv path installs the extra above).
         echo "  NOTE: 'psycopg' is not available to ${PARSEDMARC_BIN} — skipping the"
-        echo "        PostgreSQL seed. Enable it with: pip install -e '.[postgresql]'"
+        echo "        PostgreSQL seed. Enable it with: pip install -e '.[all,postgresql]'"
+    fi
+    # Like psycopg above, this is reachable only for an explicit
+    # PARSEDMARC_BIN: without the [all] integrations the seed run below
+    # exits 1 on its ConfigurationError (swallowed by the `|| true`),
+    # leaving every dashboard empty with nothing on the console to say why.
+    if [ -x "$seed_python" ] && ! "$seed_python" -c 'import elasticsearch' >/dev/null 2>&1; then
+        echo "  WARNING: the Elasticsearch/OpenSearch clients are not available to"
+        echo "           ${PARSEDMARC_BIN} — the seed run will fail to initialize its"
+        echo "           outputs. Install them with: pip install -e '.[all,postgresql]'"
     fi
     env "${pg_seed_env[@]}" \
         "$PARSEDMARC_BIN" -t 2.0 --dns-retries 1 -c parsedmarc-dev.ini "${SAMPLE_FILES[@]}" || true

dashboards/README.md

@@ -11,7 +11,7 @@ Edits to any of these files should be exported from a running instance after aut
 
 ## The dev stack
 
-[docker-compose.dashboard-dev.yml](../docker-compose.dashboard-dev.yml) brings up every viz target at once so a single dashboard change can be authored and re-exported across all four UIs in one session. It `include:`s [docker-compose.yml](../docker-compose.yml) for the Elasticsearch and OpenSearch backends, then layers on Kibana, OpenSearch Dashboards, Grafana, and Splunk.
+[docker-compose.dashboard-dev.yml](../docker-compose.dashboard-dev.yml) brings up every viz target at once so a single dashboard change can be authored and re-exported across all four UIs in one session. It `include:`s [docker-compose.yml](../docker-compose.yml) for the Elasticsearch and OpenSearch backends, then layers on Kibana, OpenSearch Dashboards, Grafana, Splunk, and a PostgreSQL backend.
 
 | Service               | URL                                              | Credentials                                            |
 | --------------------- | ------------------------------------------------ | ------------------------------------------------------ |
@@ -20,6 +20,7 @@ Edits to any of these files should be exported from a running instance after aut
 | Kibana                | http://localhost:5601                            | (security disabled)                                    |
 | OpenSearch Dashboards | http://localhost:5602                            | `admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD`         |
 | Grafana               | http://localhost:3000                            | `admin` / `$GRAFANA_PASSWORD`                          |
+| PostgreSQL            | localhost:5432                                   | `parsedmarc` / `parsedmarc` (override: `POSTGRESQL_*`) |
 | Splunk Web / HEC      | http://localhost:8000 / https://localhost:8088   | `admin` / `$SPLUNK_PASSWORD`, HEC token `$SPLUNK_HEC_TOKEN` |
 
 All ports bind to `127.0.0.1` only.
@@ -37,7 +38,7 @@ All ports bind to `127.0.0.1` only.
    ```
 
    Pick any values you like — these are local-only dev secrets. Both `.env` and `parsedmarc*.ini` are gitignored. The matching values must also appear in [parsedmarc-dev.ini](../parsedmarc-dev.ini), which the bootstrap script feeds to the parsedmarc CLI for sample-data ingestion.
-3. The parsedmarc CLI on `PATH` (or in `./venv/bin/`) — `pip install -e .[build]` from the repo root works. Override the lookup with `PARSEDMARC_BIN=/path/to/parsedmarc` if needed.
+3. The parsedmarc CLI on `PATH` (or in `./venv/bin/`) — `pip install -e ".[all,postgresql]"` from the repo root works. The `all` and `postgresql` extras are what the bootstrap script installs, and the seed needs them: the Elasticsearch, OpenSearch, and PostgreSQL outputs each live behind an extra. Override the lookup with `PARSEDMARC_BIN=/path/to/parsedmarc` if needed.
 
 ## One-shot bootstrap
 
@@ -51,7 +52,7 @@ It does, in order:
 
 1. `docker compose -f docker-compose.dashboard-dev.yml up -d` and waits for every service's health endpoint.
 2. Provisions Splunk: creates the `email` index, creates the `DMARC` app, configures the auto-created HEC token to allow the `email` index, and scopes the search-app's "scheduled export" announcement view away from `global` so it stops appearing in the DMARC app's dashboard list.
-3. Seeds Elasticsearch, OpenSearch, and Splunk with parsedmarc-parsed sample reports (from [samples/](../samples/)) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass `RESEED=1` to wipe and re-seed all three backends.
+3. Seeds Elasticsearch, OpenSearch, Splunk, and PostgreSQL with parsedmarc-parsed sample reports (from [samples/](../samples/)) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass `RESEED=1` to wipe and re-seed all four backends.
 4. Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is **edit in the UI → export → save into this directory → re-run the bootstrap script** to verify the file imports cleanly into a fresh service.
 
 VS Code users can run this via the **Dev Dashboard: Bootstrap** task in [.vscode/tasks.json](../.vscode/tasks.json). **Dev Dashboard: Up** brings the stack up without importing or seeding.
@@ -109,7 +110,7 @@ forms driven rather than HTTP basic auth.
 RESEED=1 ./dashboard-dev-bootstrap.sh
 ```
 
-Wipes every `dmarc_aggregate*` / `dmarc_failure*` / `dmarc_forensic*` / `smtp_tls*` index from ES and OS, drops and recreates the Splunk `email` index, then re-runs the parsedmarc CLI against the curated sample list. Use this after changing parsedmarc's enrichment or output schemas.
+Wipes every `dmarc_aggregate*` / `dmarc_failure*` / `dmarc_forensic*` / `smtp_tls*` index from ES and OS, drops and recreates the Splunk `email` index and the PostgreSQL schema, then re-runs the parsedmarc CLI against the curated sample list. Use this after changing parsedmarc's enrichment or output schemas.
 
 ## Tearing the stack down
 

docs/source/davmail.md

@@ -2,12 +2,12 @@
 
 :::{note}
 Starting in 8.0.0, parsedmarc supports accessing Microsoft/Office 365
-inboxes via the Microsoft Graph API, which is preferred over Davmail.
+inboxes via the Microsoft Graph API, which is preferred over DavMail.
 :::
 
 Some organizations do not allow IMAP or the Microsoft Graph API,
 and only support Exchange Web Services (EWS)/Outlook Web Access (OWA).
-In that case, Davmail will need to be set up
+In that case, DavMail will need to be set up
 as a local EWS/OWA IMAP gateway. It can even work where
 [Modern Auth/multi-factor authentication] is required.
 
@@ -22,7 +22,7 @@ Install Java:
 sudo apt-get install default-jre-headless
 ```
 
-Configure Davmail by creating a `davmail.properties` file
+Configure DavMail by creating a `davmail.properties` file
 
 ```properties
 # DavMail settings, see http://davmail.sourceforge.net/ for documentation
@@ -135,7 +135,7 @@ Then, enable the service
 
 ```bash
 sudo systemctl daemon-reload
-sudo systemctl enable parsedmarc.service
+sudo systemctl enable davmail.service
 sudo service davmail restart
 ```
 
@@ -163,7 +163,7 @@ service davmail status
 :::{note}
 In the event of a crash, systemd will restart the service after 5
 minutes, but the `service davmail status` command will only show the
-logs for the current process. To vew the logs for previous runs as
+logs for the current process. To view the logs for previous runs as
 well as the current process (newest to oldest), run:
 
 ```bash
@@ -174,8 +174,8 @@ journalctl -u davmail.service -r
 
 ## Configuring parsedmarc for DavMail
 
-Because you are interacting with DavMail server over the loopback
-(i.e. `127.0.0.1`), add the following options to `parsedmarc.ini`
+Because you are interacting with the DavMail server over the loopback
+(i.e. `127.0.0.1`), add the following options to the `parsedmarc.ini`
 config file:
 
 ```ini

docs/source/dmarc.md

@@ -16,12 +16,12 @@ check out the sister project,
 
 ### Lookalike domains
 
-DMARC protects against domain spoofing, not lookalike domains. for open source
+DMARC protects against domain spoofing, not lookalike domains. For open source
 lookalike domain monitoring, check out [DomainAware](https://github.com/seanthegeek/domainaware).
 
 ## DMARC Alignment Guide
 
-DMARC ensures that SPF and DKM authentication mechanisms actually authenticate
+DMARC ensures that SPF and DKIM authentication mechanisms actually authenticate
 against the same domain that the end user sees.
 
 A message passes a DMARC check by passing DKIM or SPF, **as long as the related

docs/source/elasticsearch.md

@@ -55,13 +55,17 @@ sudo systemctl start elasticsearch.service
 sudo systemctl start kibana.service
 ```
 
-As of Elasticsearch 8.7, activate secure mode (xpack.security.*.ssl)
+Since Elasticsearch 8.0, security is enabled and auto-configured on
+first startup: TLS certificates are generated, the `xpack.security.*`
+settings below are written to `elasticsearch.yml`, and a password is
+generated for the `elastic` user. Verify the settings are present —
+and add them only if your install skipped auto-configuration:
 
 ```bash
 sudo vim /etc/elasticsearch/elasticsearch.yml
 ```
 
-Add the following configuration
+The security configuration looks like this:
 
 ```text
 # Enable security features
@@ -92,11 +96,12 @@ openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout kibana.key -out kiba
 Or, to create a Certificate Signing Request (CSR) for a CA, run:
 
 ```bash
-openssl req -newkey rsa:4096-nodes -keyout kibana.key -out kibana.csr
+openssl req -newkey rsa:4096 -nodes -keyout kibana.key -out kibana.csr
 ```
 
 Fill in the prompts. Watch out for Common Name (e.g. server FQDN or YOUR
-domain name), which is the IP address or domain name that you will use to access Kibana. it is the most important field.
+domain name), which is the IP address or domain name that you will use to
+access Kibana. It is the most important field.
 
 If you generated a CSR, remove the CSR after you have your certs
 
@@ -129,7 +134,7 @@ server.ssl.key: /etc/kibana/kibana.key
 
 :::{note}
 For more security, you can configure Kibana to use a local network connection
-to elasticsearch :
+to Elasticsearch:
 ```text
 elasticsearch.hosts: ['https://SERVER_IP:9200']
 ```
@@ -149,14 +154,14 @@ Enroll Kibana in Elasticsearch
 sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
 ```
 
-Then access to your web server at `https://SERVER_IP:5601`, accept the self-signed
-certificate and paste the token in the "Enrollment token" field.
+Then access your web server at `https://SERVER_IP:5601`, accept the self-signed
+certificate, and paste the token in the "Enrollment token" field.
 
 ```bash
 sudo /usr/share/kibana/bin/kibana-verification-code
 ```
 
-Then put the verification code to your web browser.
+Then enter the verification code in your web browser.
 
 End Kibana configuration
 
@@ -182,12 +187,12 @@ sudo systemctl restart elasticsearch
 Now that Elasticsearch is up and running, use `parsedmarc` to send data to
 it.
 
-Download (right-click the link and click save as) [export.ndjson].
+Download (right-click the link and click save as) [opensearch_dashboards.ndjson].
 
-Connect to kibana using the "elastic" user and the password you previously provide
-on the console ("End Kibana configuration" part).
+Connect to Kibana using the "elastic" user and the password you previously
+provided on the console ("End Kibana configuration" part).
 
-Import `export.ndjson` the Saved Objects tab of the Stack management
+Import `opensearch_dashboards.ndjson` in the Saved Objects tab of the Stack Management
 page of Kibana. (Hamburger menu -> "Management" -> "Stack Management" ->
 "Kibana" -> "Saved Objects")
 
@@ -198,20 +203,20 @@ the commercial [X-Pack].
 
 ```{image} _static/screenshots/saved-objects.png
 :align: center
-:alt: A screenshot of setting the Saved Objects Stack management UI in Kibana
+:alt: A screenshot of the Saved Objects Stack Management UI in Kibana
 :target: _static/screenshots/saved-objects.png
 ```
 
 ```{image} _static/screenshots/confirm-overwrite.png
 :align: center
-:alt: A screenshot of the overwrite conformation prompt
+:alt: A screenshot of the overwrite confirmation prompt
 :target: _static/screenshots/confirm-overwrite.png
 ```
 
 ## Upgrading Kibana index patterns
 
 `parsedmarc` 5.0.0 makes some changes to the way data is indexed in
-Elasticsearch. if you are upgrading from a previous release of
+Elasticsearch. If you are upgrading from a previous release of
 `parsedmarc`, you need to complete the following steps to replace the
 Kibana index patterns with versions that match the upgraded indexes:
 
@@ -220,10 +225,10 @@ Kibana index patterns with versions that match the upgraded indexes:
 3. Check the checkboxes for the `dmarc_aggregate` and `dmarc_failure`
    index patterns
 4. Click Delete
-5. Click Delete on the conformation message
+5. Click Delete on the confirmation message
 6. Download (right-click the link and click save as)
-   the latest version of [export.ndjson]
-7. Import `export.ndjson` by clicking Import from the Kibana
+   the latest version of [opensearch_dashboards.ndjson]
+7. Import `opensearch_dashboards.ndjson` by clicking Import from the Kibana
    Saved Objects page
 
 ## Backfilling the combined DKIM/SPF result fields
@@ -408,6 +413,6 @@ retention regulations such as GDPR. For more information,
 check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
 
 [elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
-[export.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
+[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
 [kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
 [x-pack]: https://www.elastic.co/products/x-pack

docs/source/installation.md

@@ -61,7 +61,7 @@ On Debian or Ubuntu systems, run:
 sudo apt-get install -y python3-pip python3-venv python3-dev libxml2-dev libxslt-dev
 ```
 
-On CentOS, RHEL, oR Rocky Linux systems, run:
+On CentOS, RHEL, or Rocky Linux systems, run:
 
 ```bash
 sudo dnf install -y python3 python3-pip python3-devel libxml2-devel libxslt-devel
@@ -89,13 +89,64 @@ any files created later are also owned by `parsedmarc`
 ```bash
 sudo -u parsedmarc python3 -m venv /opt/parsedmarc/venv
 sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install --upgrade pip
-sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install --upgrade parsedmarc
+sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip install --upgrade "parsedmarc[all]"
 ```
 
 To upgrade `parsedmarc` later, re-run the last command above and then
 restart the service.
 
-## Optional dependencies
+### Choosing what to install
+
+Starting with the next major release, the integrations that talk to
+external systems live in optional extras, so an install only carries the
+dependencies it actually uses.
+
+`pip install parsedmarc` — the base install — provides:
+
+- the parsing library (aggregate, failure, and SMTP TLS reports),
+- the CLI, reading reports from files, an IMAP mailbox, a Maildir, or an
+  mbox file,
+- and CSV/JSON, Splunk HEC, webhook, and syslog output.
+
+Everything else needs an extra:
+
+| Extra | Enables |
+|---|---|
+| `elastic` | The `[elasticsearch]` output |
+| `opensearch` | The `[opensearch]` output |
+| `kafka` | The `[kafka]` output |
+| `s3` | The `[s3]` output |
+| `gelf` | The `[gelf]` output |
+| `loganalytics` | The `[log_analytics]` (Azure Monitor) output |
+| `msgraph` | The `[msgraph]` mailbox input (Microsoft 365) |
+| `gmail` | The `[gmail_api]` mailbox input |
+| `postgresql` | The `[postgresql]` output |
+
+Extras can be combined: `pip install "parsedmarc[elastic,msgraph]"`.
+
+`pip install "parsedmarc[all]"` installs every extra in the table
+**except** `postgresql`, which stays separate because `psycopg`'s
+prebuilt binary wheels are not available for every platform — folding it
+into `all` would make `parsedmarc[all]` fail to install there. Add it
+explicitly when you need it: `pip install "parsedmarc[all,postgresql]"`.
+
+If a configuration file names a section whose extra is not installed,
+`parsedmarc` exits at startup with an error naming the exact
+`pip install` command to run.
+
+:::{note}
+**Upgrading from 10.x:** every 10.x install carried the Elasticsearch,
+OpenSearch, Kafka, AWS, Azure, Gmail, and Microsoft Graph packages,
+whether or not it used them. They are no longer installed by
+`pip install parsedmarc`, so switch your install command to
+`pip install --upgrade "parsedmarc[all]"` (adding `postgresql` if you
+use it) to keep every integration available. An in-place upgrade does
+not uninstall packages you already have, but a rebuilt virtualenv — or
+any fresh install — gets only what the extras name. Users of the
+prebuilt Docker image are unaffected: it bundles `[all,postgresql]`.
+:::
+
+## Optional system dependencies
 
 If you would like to be able to parse emails saved from Microsoft
 Outlook (i.e. OLE .msg files), install `msgconvert`:

docs/source/kibana.md

@@ -28,7 +28,7 @@ will filter for that value.
 
 :::{note}
 Messages should not be considered malicious just because they failed to pass
-DMARC; especially if you have just started collecting data. It may be a
+DMARC, especially if you have just started collecting data. It may be a
 legitimate service that needs SPF and DKIM configured correctly.
 :::
 
@@ -36,7 +36,7 @@ Start by filtering the results to only show failed DKIM alignment. While DMARC
 passes if a message passes SPF or DKIM alignment, only DKIM alignment remains
 valid when a message is forwarded without changing the from address, which is
 often caused by a mailbox forwarding rule. This is because DKIM signatures are
-part of the message headers, whereas SPF relies on SMTP session headers.
+part of the message headers, whereas SPF relies on the SMTP envelope.
 
 Underneath the pie charts, you can see graphs of DMARC compliance and message
 disposition over time.
@@ -74,7 +74,7 @@ your domains coming from consumer email services, such as Google/Gmail and
 Yahoo! This occurs when customers have mailbox rules in place that forward
 emails from an old account to a new account, which is why DKIM
 authentication is so important, as mentioned earlier. Similar patterns may
-be observed with businesses who send from reverse DNS addressees of
+be observed with businesses who send from reverse DNS addresses of
 parent, subsidiary, and outdated brands.
 :::
 

docs/source/mailing-lists.md

@@ -130,7 +130,7 @@ command line instead, for example:
 touch var/templates/lists/list.example.com/en/list:member:regular:footer
 ```
 
-Where `list.example.com` the list ID, and `en` is the language.
+Where `list.example.com` is the list ID, and `en` is the language.
 
 Then restart mailman core.
 

docs/source/output.md

@@ -5,7 +5,7 @@
 Here are the results from parsing the [example](https://dmarc.org/wiki/FAQ#I_need_to_implement_aggregate_reports.2C_what_do_they_look_like.3F)
 report from the dmarc.org wiki. It's actually an older draft of
 the 1.0 report schema standardized in
-[RFC 7480 Appendix C](https://tools.ietf.org/html/rfc7489#appendix-C).
+[RFC 7489 Appendix C](https://tools.ietf.org/html/rfc7489#appendix-C).
 This draft schema is still in wide use.
 
 `parsedmarc` produces consistent, normalized output, regardless

docs/source/splunk.md

@@ -1,7 +1,7 @@
 # Splunk
 
 Starting in version 4.3.0 `parsedmarc` supports sending aggregate and/or
-failure DMARC data to a Splunk [HTTP Event collector (HEC)].
+failure DMARC data to a Splunk [HTTP Event Collector (HEC)].
 
 The project repository contains [XML files] for premade Splunk
 dashboards for aggregate and failure DMARC reports.
@@ -19,4 +19,4 @@ Kibana dashboards, although the Kibana dashboards have slightly
 easier and more flexible filtering options.
 
 [xml files]: https://github.com/domainaware/parsedmarc/tree/master/dashboards/splunk
-[http event collector (hec)]: http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC
+[http event collector (hec)]: https://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC

docs/source/usage.md

@@ -14,8 +14,8 @@ usage: parsedmarc [-h] [-c CONFIG_FILE] [-r] [--strip-attachment-payloads] [-o O
 Parses DMARC reports
 
 positional arguments:
-  file_path             one or more paths to aggregate or failure report files, emails, mbox files, or directories
-                        containing them
+  file_path             one or more paths to aggregate, failure, or SMTP TLS report files, emails, mbox files, or
+                        directories containing them
 
 options:
   -h, --help            show this help message and exit
@@ -119,21 +119,23 @@ smtp_tls_url = https://smtp_tls_url.example.com
 timeout = 60
 ```
 
-The full set of configuration options are:
+The full set of configuration options is:
 
 - `general`
   - `save_aggregate` - bool: Save aggregate report data to
       Elasticsearch, Splunk and/or S3
   - `save_failure` - bool: Save failure report data to
       Elasticsearch, Splunk and/or S3
-  - `save_smtp_tls` - bool: Save SMTP-STS report data to
+  - `save_smtp_tls` - bool: Save SMTP TLS report data to
       Elasticsearch, Splunk and/or S3
   - `index_prefix_domain_map` - str: Path to a YAML file mapping
       OpenSearch/Elasticsearch index prefixes to domain names
   - `strip_attachment_payloads` - bool: Remove attachment
       payloads from results
-  - `silent` - bool: Set this to `False` to output results to STDOUT
-  - `output` - str: Directory to place JSON and CSV files in.  This is required if you set either of the JSON output file options.
+  - `silent` - bool: Only print errors; set this to `False` to output
+      results to STDOUT (Default: `True`)
+  - `output` - str: Directory to place JSON and CSV files in. The JSON
+      and CSV filename options below only take effect when this is set.
   - `archive_directory` - str: Optional. When set, successfully
       processed report files given as local file/directory path
       arguments are moved into
@@ -156,6 +158,19 @@ The full set of configuration options are:
       JSON output file
   - `failure_json_filename` - str: filename for the failure
       JSON output file
+  - `smtp_tls_json_filename` - str: filename for the SMTP TLS
+      JSON output file
+  - `aggregate_csv_filename` - str: filename for the aggregate
+      CSV output file
+  - `failure_csv_filename` - str: filename for the failure
+      CSV output file
+  - `smtp_tls_csv_filename` - str: filename for the SMTP TLS
+      CSV output file
+  - `prettify_json` - bool: Set this to `False` to output JSON in a
+      single line without indentation (Default: `True`)
+  - `normalize_timespan_threshold_hours` - float: Aggregate reports
+      covering a longer time span than this many hours have their
+      records normalized into per-day records (Default: `24`)
   - `ip_db_path` - str: An optional custom path to a MMDB file
       from IPinfo, MaxMind, or DBIP
   - `ipinfo_url` - str: Overrides the default download URL for the
@@ -172,24 +187,25 @@ The full set of configuration options are:
   - `offline` - bool: Do not use online queries for geolocation
       or DNS. Also disables automatic downloading of the IP-to-country
       database and reverse DNS map.
-  - `always_use_local_files` - Disables the download of the
+  - `always_use_local_files` - bool: Disables the download of the
       IP-to-country database and reverse DNS map
-  - `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
-  - `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map
-  - `local_psl_overrides_path` - Overrides the default local file path to use for the PSL overrides list
-  - `psl_overrides_url` - Overrides the default download URL for the PSL overrides list
+  - `local_reverse_dns_map_path` - str: Overrides the default local file path to use for the reverse DNS map
+  - `reverse_dns_map_url` - str: Overrides the default download URL for the reverse DNS map
+  - `local_psl_overrides_path` - str: Overrides the default local file path to use for the PSL overrides list
+  - `psl_overrides_url` - str: Overrides the default download URL for the PSL overrides list
   - `nameservers` - str: A comma separated list of
       DNS resolvers (Default: `[Cloudflare's public resolvers]`). Each entry
       is an IP address (DNS over UDP/TCP port 53), an `https://` URL
       (DNS over HTTPS), or `tls://ip[:port][#hostname]` (DNS over TLS) —
       see [Encrypted DNS](#encrypted-dns)
-  - `dns_test_address` - str: a dummy address used for DNS pre-flight checks
-      (Default: 1.1.1.1)
-  - `dns_timeout` - float: DNS timeout period
+  - `dns_test_address` - str: A dummy address used for the DNS pre-flight
+      check that runs when `nameservers` is set (Default: `1.1.1.1`)
+  - `dns_timeout` - float: DNS timeout period in seconds (Default: `2.0`)
   - `dns_retries` - int: Number of times to retry a DNS query after a
-      timeout or other transient error (Default: 0)
+      timeout or other transient error (Default: `0`)
   - `debug` - bool: Print debugging messages
-  - `silent` - bool: Only print errors (Default: `True`)
+  - `verbose` - bool: More verbose output
+  - `warnings` - bool: Print warnings in addition to errors
   - `fail_on_output_error` - bool: Exit with a non-zero status code if
       any configured output destination fails while saving/publishing
       reports (Default: `False`)
@@ -227,8 +243,9 @@ The full set of configuration options are:
       (Default: `INBOX`)
   - `archive_folder` - str: The mailbox folder (or label for
       Gmail) to sort processed emails into (Default: `Archive`)
-  - `watch` - bool: Use the IMAP `IDLE` command to process
-      messages as they arrive or poll MS Graph for new messages
+  - `watch` - bool: Process new messages as they arrive, via the
+      IMAP `IDLE` command or by polling the other mailbox types
+      (Microsoft Graph, Gmail API, Maildir)
   - `delete` - bool: Delete messages after processing them,
       instead of archiving them
   - `delete_aggregate` - bool: Delete aggregate report messages
@@ -271,9 +288,9 @@ The full set of configuration options are:
       runs. See
       [Mailbox messages are only archived once the reports are saved](#mailbox-messages-are-only-archived-once-the-reports-are-saved)
       below.
-  - `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
+  - `since` - str: Search for messages since a certain time. (Examples: `5m|3h|2d|1w`)
       Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
-      Defaults to `1d` if incorrect value is provided.
+      Defaults to `1d` if an incorrect value is provided.
 - `imap`
   - `host` - str: The IMAP server hostname or IP address
   - `port` - int: The IMAP server port (Default: `993`)
@@ -296,9 +313,16 @@ The full set of configuration options are:
       (Default: `True`)
   - `skip_certificate_verification` - bool: Skip certificate
       verification (not recommended)
+  - `timeout` - int: Number of seconds to wait for an IMAP operation
+      (Default: `30`)
+  - `max_retries` - int: Maximum number of retries after an IMAP
+      timeout (Default: `4`)
   - `user` - str: The IMAP user
   - `password` - str: The IMAP password
 - `msgraph`
+
+  Requires the `msgraph` extra: `pip install "parsedmarc[msgraph]"`
+
   - `auth_method` - str: Authentication method, valid types are
       `UsernamePassword`, `DeviceCode`, `ClientSecret`, `Certificate`, or
       `ClientAssertion` (Default: `UsernamePassword`).
@@ -307,7 +331,8 @@ The full set of configuration options are:
   - `password` - str: The user password, required when the auth
       method is UsernamePassword
   - `client_id` - str: The app registration's client ID
-  - `client_secret` - str: The app registration's secret
+  - `client_secret` - str: The app registration's secret. Required when
+      the auth method is `UsernamePassword` or `ClientSecret`
   - `certificate_path` - str: Path to a PEM or PKCS12 certificate
       including the private key. Required when the auth method is
       `Certificate`
@@ -329,8 +354,9 @@ The full set of configuration options are:
   - `mailbox` - str: The mailbox name. This defaults to the
       current user if using the UsernamePassword auth method, but
       could be a shared mailbox if the user has access to the mailbox
-  - `graph_url` - str: Microsoft Graph URL.  Allows for use of National Clouds (ex Azure Gov)
-      (Default: https://graph.microsoft.com)
+  - `graph_url` - str: Microsoft Graph URL. Allows for use of national
+      clouds (e.g. Azure Gov)
+      (Default: `https://graph.microsoft.com`)
 
     :::{warning}
     Setting `graph_url` alone is **not** sufficient for a national/sovereign
@@ -534,6 +560,9 @@ The full set of configuration options are:
     | Invalid/rejected timestamp in the `since`/`receivedDateTime` filter | Historical bug (parsedmarc [#706](https://github.com/domainaware/parsedmarc/pull/706)/[#708](https://github.com/domainaware/parsedmarc/pull/708)): older versions appended a spurious `Z` to an already-UTC-offset ISO timestamp. Fixed since parsedmarc 9.5.1/9.5.5. | Upgrade parsedmarc if you're on a version older than 9.5.5. |
     :::
 - `elasticsearch`
+
+  Requires the `elastic` extra: `pip install "parsedmarc[elastic]"`
+
   - `hosts` - str: A comma separated list of hostnames and ports
       or URLs (e.g. `127.0.0.1:9200` or
       `https://user:secret@localhost`)
@@ -548,7 +577,7 @@ The full set of configuration options are:
   - `ssl` - bool: Use an encrypted SSL/TLS connection
     (Default: `True`)
   - `timeout` - float: Timeout in seconds (Default: 60)
-  - `cert_path` - str: Path to a trusted certificates
+  - `cert_path` - str: Path to a trusted CA certificates file
   - `skip_certificate_verification` - bool: Skip certificate
     verification (not recommended)
   - `index_suffix` - str: A suffix to apply to the index names
@@ -565,6 +594,9 @@ The full set of configuration options are:
     settings sent at index creation; any other settings (e.g.
     `refresh_interval`) are passed through unchanged (Default: `False`)
 - `opensearch`
+
+  Requires the `opensearch` extra: `pip install "parsedmarc[opensearch]"`
+
   - `hosts` - str: A comma separated list of hostnames and ports
     or URLs (e.g. `127.0.0.1:9200` or
     `https://user:secret@localhost`)
@@ -583,7 +615,7 @@ The full set of configuration options are:
   - `ssl` - bool: Use an encrypted SSL/TLS connection
     (Default: `True`)
   - `timeout` - float: Timeout in seconds (Default: 60)
-  - `cert_path` - str: Path to a trusted certificates
+  - `cert_path` - str: Path to a trusted CA certificates file
   - `skip_certificate_verification` - bool: Skip certificate
     verification (not recommended)
   - `index_suffix` - str: A suffix to apply to the index names
@@ -600,14 +632,18 @@ The full set of configuration options are:
   - `skip_certificate_verification` - bool: Skip certificate
     verification (not recommended)
 - `kafka`
+
+  Requires the `kafka` extra: `pip install "parsedmarc[kafka]"`
+
   - `hosts` - str: A comma separated list of Kafka hosts
   - `user` - str: The Kafka user
-  - `passsword` - str: The Kafka password
+  - `password` - str: The Kafka password
   - `ssl` - bool: Use an encrypted SSL/TLS connection (Default: `True`)
   - `skip_certificate_verification` - bool: Skip certificate
     verification (not recommended)
   - `aggregate_topic` - str: The Kafka topic for aggregate reports
   - `failure_topic` - str: The Kafka topic for failure reports
+  - `smtp_tls_topic` - str: The Kafka topic for SMTP TLS reports
 - `smtp`
 
   The results email is only sent when at least one aggregate, failure,
@@ -632,7 +668,7 @@ The full set of configuration options are:
   - `to` - list: A list of email addresses to send to
   - `subject` - str: The Subject header to use in the email
     (Default: `parsedmarc report`)
-  - `attachment` - str: The ZIP attachment filenames
+  - `attachment` - str: The ZIP attachment filename
     (Default: `DMARC-<YYYY-MM-DD>.zip`)
   - `message` - str: The email message
     (Default: `Please see the attached DMARC results.`)
@@ -653,13 +689,14 @@ The full set of configuration options are:
     all individual parameters above are ignored.
 
   The PostgreSQL backend is an optional extra. Install it with
-  `pip install parsedmarc[postgresql]` (it pulls in `psycopg`); the
+  `pip install "parsedmarc[postgresql]"` (it pulls in `psycopg`); the
   prebuilt binary wheels are not available for every platform, which is
-  why it is not a mandatory dependency. The prebuilt Docker image
-  (`ghcr.io/domainaware/parsedmarc`) already bundles this extra, so the
-  PostgreSQL backend works out of the box in the container — `psycopg`
-  ships `amd64` and `arm64` binary wheels, both of which the image
-  supports.
+  why it is the one extra that `parsedmarc[all]` deliberately leaves out
+  — combine the two with `pip install "parsedmarc[all,postgresql]"`. The
+  prebuilt Docker image (`ghcr.io/domainaware/parsedmarc`) already
+  bundles `[all,postgresql]`, so the PostgreSQL backend works out of the
+  box in the container — `psycopg` ships `amd64` and `arm64` binary
+  wheels, both of which the image supports.
 
   Tables are created automatically on first run using
   `CREATE TABLE IF NOT EXISTS`, so no manual schema migration is needed
@@ -692,6 +729,9 @@ The full set of configuration options are:
   this section is configured.
 
 - `s3`
+
+  Requires the `s3` extra: `pip install "parsedmarc[s3]"`
+
   - `bucket` - str: The S3 bucket name
   - `path` - str: The path to upload reports to (Default: `/`)
   - `region_name` - str: The region name (Optional)
@@ -755,6 +795,9 @@ The full set of configuration options are:
   ```
 
 - `gmail_api`
+
+  Requires the `gmail` extra: `pip install "parsedmarc[gmail]"`
+
   - `credentials_file` - str: Path to file containing the
       credentials, None to disable (Default: `None`)
   - `token_file` - str: Path to save the token file
@@ -766,7 +809,9 @@ The full set of configuration options are:
       accepted as `delegated_user` for backward compatibility.
 
     :::{note}
-    credentials_file and token_file can be got with [quickstart](https://developers.google.com/gmail/api/quickstart/python).Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
+    `credentials_file` and `token_file` can be obtained by following the
+    Gmail API [quickstart](https://developers.google.com/gmail/api/quickstart/python).
+    Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
     :::
     :::{note}
     When `auth_mode = service_account`, `credentials_file` must point to a
@@ -782,6 +827,9 @@ The full set of configuration options are:
   - `paginate_messages` - bool: When `True`, fetch all applicable Gmail messages.
       When `False`, only fetch up to 100 new messages per run (Default: `True`)
 - `log_analytics`
+
+  Requires the `loganalytics` extra: `pip install "parsedmarc[loganalytics]"`
+
   - `client_id` - str: The app registration's client ID
   - `client_secret` - str: The app registration's client secret
   - `tenant_id` - str: The tenant id where the app registration resides
@@ -792,22 +840,27 @@ The full set of configuration options are:
   - `dcr_smtp_tls_stream` - str: The stream name for the SMTP TLS reports in the DCR
 
   :::{note}
-    Information regarding the setup of the Data Collection Rule can be found [in the Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal).
-    :::
+  Information regarding the setup of the Data Collection Rule can be found [in the Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal).
+  :::
 - `gelf`
+
+  Requires the `gelf` extra: `pip install "parsedmarc[gelf]"`
+
   - `host` - str: The GELF server name or IP address
   - `port` - int: The port to use
   - `mode` - str: The GELF transport type to use. Valid modes: `tcp`, `udp`, `tls`
 
 - `maildir`
-  - `maildir_path` - str: Full path for mailbox maildir location (Default: `INBOX`)
-  - `maildir_create` - bool: Create maildir if not present (Default: False)
+  - `maildir_path` - str: Full path to the maildir location (the key
+    `path` is accepted as an alias). Required to read from a maildir.
+  - `maildir_create` - bool: Create the maildir if not present
+    (Default: `False`; the key `create` is accepted as an alias)
 
-- `webhook` - Post the individual reports to a webhook url with the report as the JSON body
+- `webhook` - Post the individual reports to a webhook URL with the report as the JSON body
   - `aggregate_url` - str: URL of the webhook which should receive the aggregate reports
   - `failure_url` - str: URL of the webhook which should receive the failure reports
-  - `smtp_tls_url` - str: URL of the webhook which should receive the smtp_tls reports
-  - `timeout` - int: Interval in which the webhook call should timeout
+  - `smtp_tls_url` - str: URL of the webhook which should receive the SMTP TLS reports
+  - `timeout` - int: Timeout in seconds for webhook requests (Default: `60`)
 
 :::{warning}
 It is **strongly recommended** to **not** use the `nameservers`
@@ -847,8 +900,8 @@ known samples you want to save to that folder
 :::
 
 :::{warning}
-Elasticsearch 8 change limits policy for shards, restricting by
-default to 1000. parsedmarc use a shard per analyzed day. If you
+Elasticsearch 8 changed the limits policy for shards, restricting them
+by default to 1000. parsedmarc uses a shard per analyzed day. If you
 have more than ~3 years of data, you will need to update this
 limit.
 Check current usage (from Management -> Dev Tools -> Console):
@@ -863,7 +916,7 @@ GET /_cluster/health?pretty
 }
 ```
 
-Update the limit to 2k per example:
+For example, update the limit to 2000:
 
 ```text
 PUT _cluster/settings
@@ -1123,6 +1176,7 @@ For sections with underscores in the name, the full section name is used:
 | `kafka` | `PARSEDMARC_KAFKA_` |
 | `smtp` | `PARSEDMARC_SMTP_` |
 | `s3` | `PARSEDMARC_S3_` |
+| `postgresql` | `PARSEDMARC_POSTGRESQL_` |
 | `syslog` | `PARSEDMARC_SYSLOG_` |
 | `gmail_api` | `PARSEDMARC_GMAIL_API_` |
 | `maildir` | `PARSEDMARC_MAILDIR_` |
@@ -1207,7 +1261,7 @@ high-volume mailbox processing.
 
 ## Multi-tenant support
 
-Starting in `8.19.0`, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file that is formatted where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:
+Starting in `8.19.0`, ParseDMARC provides multi-tenant support by placing data into separate OpenSearch or Elasticsearch index prefixes. To set this up, create a YAML file where each key is a tenant name, and the value is a list of domains related to that tenant, not including subdomains, like this:
 
 ```yaml
 example:
@@ -1223,8 +1277,8 @@ Save it to disk where the user running ParseDMARC can read it, then set `index_p
 
 When configured correctly, if ParseDMARC finds that a report is related to a domain in the mapping, the report will be saved in an index name that has the tenant name prefixed to it with a trailing underscore. Then, you can use the security features of OpenSearch or the ELK stack to only grant users access to the indexes that they need.
 
- :::{note}
- A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
+:::{note}
+A domain cannot be used in multiple tenant lists. Only the first prefix list that contains the matching domain is used.
 :::
 
 Each key must be a tenant name and each value a *list* of domain names, all strings; a file of any other shape is rejected at startup.
@@ -1301,9 +1355,9 @@ sudo service parsedmarc restart
 On `systemctl stop`/`restart` (or Ctrl-C) `parsedmarc` finishes the
 current batch, flushes its outputs, and exits cleanly. Shutdown is
 observed at batch boundaries, so the worst-case delay is roughly
-`mailbox_check_timeout` (default 30s) plus the batch's processing and
+`[mailbox] check_timeout` (default 30s) plus the batch's processing and
 flush time. Keep `TimeoutStopSec` comfortably above
-`mailbox_check_timeout` (≈2×, and raise both together) or systemd will
+`check_timeout` (≈2×, and raise both together) or systemd will
 `SIGKILL` mid-batch. In the foreground, a second Ctrl-C force-quits
 immediately, skipping the output flush.
 :::
@@ -1360,8 +1414,9 @@ service parsedmarc status
 ```
 
 :::{note}
-In the event of a crash, systemd will restart the service after 10
-minutes, but the `service parsedmarc status` command will only show
+In the event of a crash, systemd will restart the service after 5
+minutes (per the `RestartSec` setting above), but the
+`service parsedmarc status` command will only show
 the logs for the current process. To view the logs for previous runs
 as well as the current process (newest to oldest), run:
 

parsedmarc/__init__.py

@@ -767,8 +767,8 @@ def parse_smtp_tls_report_json(report: str | bytes) -> SMTPTLSReport:
 def parsed_smtp_tls_reports_to_csv_rows(
     reports: SMTPTLSReport | list[SMTPTLSReport],
 ) -> list[dict[str, Any]]:
-    """Converts one oor more parsed SMTP TLS reports into a list of single
-    layer dict objects suitable for use in a CSV"""
+    """Converts one or more parsed SMTP TLS reports into a list of
+    single-layer dict objects suitable for use in a CSV"""
     if isinstance(reports, dict):
         reports = [reports]
 
@@ -812,10 +812,10 @@ def parsed_smtp_tls_reports_to_csv(
     headers
 
     Args:
-        reports: A parsed aggregate report or list of parsed aggregate reports
+        reports: A parsed SMTP TLS report or list of parsed SMTP TLS reports
 
     Returns:
-        str: Parsed aggregate report data in flat CSV format, including headers
+        str: Parsed SMTP TLS report data in flat CSV format, including headers
     """
 
     fields = [
@@ -1154,16 +1154,16 @@ def parse_aggregate_report_xml(
 
 def extract_report(content: bytes | str | BinaryIO) -> str:
     """
-    Extracts text from a zip or gzip file, as a base64-encoded string,
-    file-like object, or bytes.
+    Extracts report text from zip- or gzip-compressed content, and returns
+    plain XML or JSON content decoded as-is.
 
     Args:
-        content: report file as a base64-encoded string, file-like object or
-        bytes.
+        content: The report as a base64-encoded string, file-like object,
+            or bytes. A string that is not valid base64 is returned
+            unchanged.
 
     Returns:
         str: The extracted text
-
     """
     file_object: BinaryIO | None = None
     header: bytes
@@ -1264,11 +1264,11 @@ def parse_aggregate_report_file(
     normalize_timespan_threshold_hours: float = 24.0,
     config: ParserConfig | None = None,
 ) -> AggregateReport:
-    """Parses a file at the given path, a file-like object. or bytes as an
+    """Parses a file at the given path, a file-like object, or bytes as an
     aggregate DMARC report
 
     Args:
-        _input (str | bytes | IO): A path to a file, a file like object, or bytes
+        _input (str | bytes | IO): A path to a file, a file-like object, or bytes
         offline (bool): Do not query online for geolocation or DNS
         always_use_local_files (bool): Do not download files
         reverse_dns_map_path (str): Path to a reverse DNS map file
@@ -1546,12 +1546,13 @@ def parse_failure_report(
     Args:
         feedback_report (str): A message's feedback report as a string
         sample (str): The RFC 822 headers or RFC 822 message sample
+        msg_date (datetime): The message's date, used as the arrival date
+            when the feedback report has no ``Arrival-Date`` field
         ip_db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP
         always_use_local_files (bool): Do not download files
         reverse_dns_map_path (str): Path to a reverse DNS map file
         reverse_dns_map_url (str): URL to a reverse DNS map file
         offline (bool): Do not query online for geolocation or DNS
-        msg_date (str): The message's date header
         nameservers (list): A list of one or more nameservers to use
             (Cloudflare's public DNS resolvers by default)
         dns_timeout (float): Sets the DNS timeout in seconds
@@ -1943,7 +1944,7 @@ def parse_report_email(
     config: ParserConfig | None = None,
 ) -> ParsedReport:
     """
-    Parses a DMARC report from an email
+    Parses a DMARC or SMTP TLS report from an email
 
     Args:
         input_: An emailed DMARC report in RFC 822 format, as bytes or a string
@@ -1951,7 +1952,7 @@ def parse_report_email(
         always_use_local_files (bool): Do not download files
         reverse_dns_map_path (str): Path to a reverse DNS map
         reverse_dns_map_url (str): URL to a reverse DNS map
-        offline (bool): Do not query online for geolocation on DNS
+        offline (bool): Do not query online for geolocation or DNS
         nameservers (list): A list of one or more nameservers to use
         dns_timeout (float): Sets the DNS timeout in seconds
         dns_retries (int): Number of times to retry DNS queries on timeout
@@ -1968,7 +1969,7 @@ def parse_report_email(
 
     Returns:
         dict:
-        * ``report_type``: ``aggregate`` or ``failure``
+        * ``report_type``: ``aggregate``, ``failure``, or ``smtp_tls``
         * ``report``: The parsed report
     """
     cfg = _resolve_config(
@@ -2202,8 +2203,8 @@ def parse_report_file(
     normalize_timespan_threshold_hours: float = 24.0,
     config: ParserConfig | None = None,
 ) -> ParsedReport:
-    """Parses a DMARC aggregate or failure file at the given path, a
-    file-like object. or bytes
+    """Parses a DMARC aggregate report, DMARC failure report, or SMTP TLS
+    report from a file at the given path, a file-like object, or bytes
 
     Args:
         input_ (str | os.PathLike | bytes | BinaryIO): A path to a file,
@@ -2429,7 +2430,7 @@ def get_dmarc_reports_from_mbox(
     config: ParserConfig | None = None,
 ) -> ParsingResults:
     """Parses a mailbox in mbox format containing e-mails with attached
-    DMARC reports
+    DMARC and SMTP TLS reports
 
     Args:
         input_ (str): A path to a mbox file
@@ -2625,7 +2626,7 @@ def get_dmarc_reports_from_mailbox(
     config: ParserConfig | None = None,
 ) -> ParsingResults:
     """
-    Fetches and parses DMARC reports from a mailbox
+    Fetches and parses DMARC and SMTP TLS reports from a mailbox
 
     Args:
         connection: A Mailbox connection object
@@ -2663,7 +2664,7 @@ def get_dmarc_reports_from_mailbox(
         results (dict): Results from the previous run
         batch_size (int): Number of messages to read and process before saving
             (use 0 for no limit)
-        since: Search for messages since certain time
+        since: Search for messages since a certain time
             (units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"})
         create_folders (bool): Whether to create the destination folders
             (not used in watch)
@@ -3184,8 +3185,8 @@ def watch_inbox(
     config: ParserConfig | None = None,
 ):
     """
-    Watches the mailbox for new messages and
-      sends the results to a callback function
+    Watches the mailbox for new messages and sends the results to a
+    callback function
 
     Args:
         mailbox_connection: The mailbox connection object
@@ -3204,7 +3205,7 @@ def watch_inbox(
             then is backend-specific: the Microsoft Graph and Gmail backends
             let it propagate and end the watch, while mailsuite's IMAP and
             Maildir watch loops log it and keep checking.
-        reports_folder (str): The IMAP folder where reports can be found
+        reports_folder (str): The folder where reports can be found
         archive_folder (str): The folder to move processed mail to
         delete (bool): Delete messages after processing them
         delete_aggregate (bool | None): Delete aggregate report messages
@@ -3224,8 +3225,8 @@ def watch_inbox(
             can be inspected for debugging; ``None`` (the default) inherits
             the value of ``delete``
         test (bool): Do not move or delete messages after processing them
-        check_timeout (int): Number of seconds to wait for a IMAP IDLE response
-            or the number of seconds until the next mail check
+        check_timeout (int): Number of seconds to wait for an IMAP IDLE
+            response or the number of seconds until the next mail check
         ip_db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP
         always_use_local_files (bool): Do not download files
         reverse_dns_map_path (str): Path to a reverse DNS map file
@@ -3236,10 +3237,10 @@ def watch_inbox(
         dns_timeout (float): Set the DNS query timeout
         dns_retries (int): Number of times to retry DNS queries on timeout
             or other transient errors
-        strip_attachment_payloads (bool): Replace attachment payloads in
-            failure report samples with None
+        strip_attachment_payloads (bool): Remove attachment payloads from
+            failure report results
         batch_size (int): Number of messages to read and process before saving
-        since: Search for messages since certain time
+        since: Search for messages since a certain time
         normalize_timespan_threshold_hours (float): Normalize timespans beyond this
         config_reloading: Optional callable that returns True when a config
             reload (or shutdown) has been requested (e.g. via SIGHUP/SIGTERM).
@@ -3558,7 +3559,7 @@ def email_results(
         mail_from: The value of the message from header
         mail_to (list): A list of addresses to mail to
         mail_cc (list): A list of addresses to CC
-        mail_bcc (list): A list addresses to BCC
+        mail_bcc (list): A list of addresses to BCC
         port (int): Port to use
         require_encryption (bool): Require a secure connection from the start
         verify (bool): verify the SSL/TLS certificate
@@ -3617,7 +3618,7 @@ def email_results_via_msgraph(
             Graph mailbox connection
         mail_to (list): A list of addresses to mail to
         mail_cc (list): A list of addresses to CC
-        mail_bcc (list): A list addresses to BCC
+        mail_bcc (list): A list of addresses to BCC
         subject (str): Overrides the default message subject
         attachment_filename (str): Override the default attachment filename
         message (str): Override the default plain text body

parsedmarc/cli.py

@@ -17,11 +17,10 @@ from argparse import ArgumentParser, Namespace
 from configparser import ConfigParser
 from glob import escape as glob_escape, glob
 from ssl import CERT_NONE, create_default_context
+from typing import TYPE_CHECKING
 
 import httpx
 import yaml
-from azure.core.exceptions import ClientAuthenticationError
-from kiota_abstractions.api_error import APIError
 from tqdm import tqdm
 
 from parsedmarc import (
@@ -32,25 +31,87 @@ from parsedmarc import (
     ParserConfig,
     ParserError,
     __version__,
-    elastic,
     email_results,
     email_results_via_msgraph,
-    gelf,
     get_dmarc_reports_from_mailbox,
     get_dmarc_reports_from_mbox,
-    kafkaclient,
-    loganalytics,
-    opensearch,
     postgres,
-    s3,
     save_output,
     splunk,
     syslog,
     watch_inbox,
     webhook,
 )
+
+# Output integrations that need an optional extra (issue #883). Each one
+# imports a third-party SDK at module level -- elastic.py and
+# opensearch.py define DSL Document classes there, so they cannot guard
+# the import internally the way postgres.py does -- and so the guard
+# lives here, at the import site. A missing module becomes ``None``; the
+# config sections that would use it fail fast in _init_output_clients()
+# with a pip-install hint. splunk, syslog, webhook, and postgres are
+# imported eagerly above: they need only httpx, the standard library, or
+# their own internal guard.
+if TYPE_CHECKING:
+    from parsedmarc import elastic, gelf, kafkaclient, loganalytics, opensearch, s3
+else:
+    try:
+        from parsedmarc import elastic
+    except ModuleNotFoundError:
+        elastic = None
+
+    try:
+        from parsedmarc import gelf
+    except ModuleNotFoundError:
+        gelf = None
+
+    try:
+        from parsedmarc import kafkaclient
+    except ModuleNotFoundError:
+        kafkaclient = None
+
+    try:
+        from parsedmarc import loganalytics
+    except ModuleNotFoundError:
+        loganalytics = None
+
+    try:
+        from parsedmarc import opensearch
+    except ModuleNotFoundError:
+        opensearch = None
+
+    try:
+        from parsedmarc import s3
+    except ModuleNotFoundError:
+        s3 = None
+
+# Microsoft Graph error types, used only in ``except`` clauses and one
+# ``isinstance`` check around Graph mailbox calls. Without the msgraph
+# extra a Graph connection cannot be constructed at all (its
+# parsedmarc.mail placeholder raises the extra's ImportError), so those
+# handlers are unreachable and the placeholders below are never matched.
+if TYPE_CHECKING:
+    from azure.core.exceptions import ClientAuthenticationError
+    from kiota_abstractions.api_error import APIError
+else:
+    try:
+        from azure.core.exceptions import ClientAuthenticationError
+    except ModuleNotFoundError:
+
+        class ClientAuthenticationError(Exception):
+            """Never-raised placeholder for the absent msgraph extra."""
+
+    try:
+        from kiota_abstractions.api_error import APIError
+    except ModuleNotFoundError:
+
+        class APIError(Exception):
+            """Never-raised placeholder for the absent msgraph extra."""
+
+
 from parsedmarc.constants import DEFAULT_DNS_MAX_RETRIES, DEFAULT_DNS_TIMEOUT
 from parsedmarc.log import logger
+import parsedmarc.mail
 from parsedmarc.mail import (
     AuthMethod,
     GmailConnection,
@@ -91,6 +152,24 @@ class ConfigurationError(Exception):
     pass
 
 
+def _missing_extra_hint(section: str, extra: str) -> str:
+    """Return the error message for a config section whose extra is missing.
+
+    Args:
+        section (str): The INI section name, without brackets.
+        extra (str): The name of the extra that provides the section's
+            integration.
+
+    Returns:
+        str: A message naming the section, the extra, and the exact pip
+        command that installs it.
+    """
+    return (
+        f"The [{section}] configuration section requires the {extra} extra: "
+        f"pip install parsedmarc[{extra}]"
+    )
+
+
 def _normalize_graph_auth_method(value: str) -> str:
     """Return the canonical :class:`AuthMethod` member name for *value*.
 
@@ -112,7 +191,7 @@ def _normalize_graph_auth_method(value: str) -> str:
 
 
 def _str_to_list(s):
-    """Converts a comma separated string to a list"""
+    """Converts a comma-separated string to a list"""
     _list = s.split(",")
     return list(map(lambda i: i.lstrip(), _list))
 
@@ -621,7 +700,9 @@ def _load_config(config_file: str | None = None) -> ConfigParser:
         ``PARSEDMARC_*`` environment variables.
 
     Raises:
-        ConfigurationError: If *config_file* is given but does not exist.
+        ConfigurationError: If *config_file* is given but does not exist or
+            is not readable, or if a ``PARSEDMARC_..._FILE`` secret file
+            cannot be read.
     """
     config = ConfigParser(interpolation=None)
     if config_file is not None:
@@ -920,6 +1001,17 @@ def _parse_config(config: ConfigParser, opts):
             )
 
     if "msgraph" in config.sections():
+        # Without the msgraph extra, parsedmarc.mail binds a placeholder
+        # class (outside the MailboxConnection hierarchy) whose
+        # construction raises mailsuite's ImportError — fail fast here
+        # with parsedmarc's own install hint instead. Checked on the
+        # parsedmarc.mail module, the authoritative source of the
+        # placeholder state, not this module's rebound name (which tests
+        # replace with mocks).
+        if not issubclass(
+            parsedmarc.mail.MSGraphConnection, parsedmarc.mail.MailboxConnection
+        ):
+            raise ConfigurationError(_missing_extra_hint("msgraph", "msgraph"))
         graph_config = config["msgraph"]
         opts.graph_token_file = _expand_path(graph_config.get("token_file", ".token"))
 
@@ -1315,6 +1407,11 @@ def _parse_config(config: ConfigParser, opts):
             opts.syslog_retry_delay = 5
 
     if "gmail_api" in config.sections():
+        # Same placeholder detection as the msgraph section above.
+        if not issubclass(
+            parsedmarc.mail.GmailConnection, parsedmarc.mail.MailboxConnection
+        ):
+            raise ConfigurationError(_missing_extra_hint("gmail_api", "gmail"))
         gmail_api_config = config["gmail_api"]
         gmail_creds = gmail_api_config.get("credentials_file")
         opts.gmail_api_credentials_file = (
@@ -1564,6 +1661,12 @@ def _init_output_clients(opts, index_prefix_domain_map=None):
     """
     clients = {}
 
+    # Each check below is deliberately outside the try/except that wraps
+    # its constructor: those handlers re-raise everything as RuntimeError,
+    # which would bury the install hint.
+    if opts.s3_bucket and s3 is None:
+        raise ConfigurationError(_missing_extra_hint("s3", "s3"))
+
     try:
         if opts.s3_bucket:
             logger.debug("Initializing S3 client: bucket=%s", opts.s3_bucket)
@@ -1578,6 +1681,15 @@ def _init_output_clients(opts, index_prefix_domain_map=None):
     except Exception as e:
         raise RuntimeError(f"S3: {e}") from e
 
+    # postgres.py guards its own psycopg import, so the module is always
+    # importable; check the SDK here so a missing extra is a fail-fast
+    # ConfigurationError rather than a PostgreSQLError that the startup
+    # retry loop would retry for over a minute before exiting.
+    if (
+        opts.postgresql_host or opts.postgresql_connection_string
+    ) and postgres.psycopg is None:
+        raise ConfigurationError(_missing_extra_hint("postgresql", "postgresql"))
+
     try:
         if opts.postgresql_host or opts.postgresql_connection_string:
             logger.debug("Initializing PostgreSQL client")
@@ -1635,6 +1747,9 @@ def _init_output_clients(opts, index_prefix_domain_map=None):
         except Exception as e:
             raise RuntimeError(f"Splunk HEC: {e}") from e
 
+    if opts.kafka_hosts and kafkaclient is None:
+        raise ConfigurationError(_missing_extra_hint("kafka", "kafka"))
+
     try:
         if opts.kafka_hosts:
             logger.debug("Initializing Kafka client: hosts=%s", opts.kafka_hosts)
@@ -1653,6 +1768,9 @@ def _init_output_clients(opts, index_prefix_domain_map=None):
     except Exception as e:
         raise RuntimeError(f"Kafka: {e}") from e
 
+    if opts.gelf_host and gelf is None:
+        raise ConfigurationError(_missing_extra_hint("gelf", "gelf"))
+
     try:
         if opts.gelf_host:
             logger.debug(
@@ -1684,12 +1802,27 @@ def _init_output_clients(opts, index_prefix_domain_map=None):
     except Exception as e:
         raise RuntimeError(f"Webhook: {e}") from e
 
+    # The Log Analytics client is built per batch in process_reports(),
+    # under the same opts.la_dce guard. Checking it here means a missing
+    # extra is reported at startup -- and again on a SIGHUP reload --
+    # rather than once reports are already in hand.
+    if opts.la_dce and loganalytics is None:
+        raise ConfigurationError(_missing_extra_hint("log_analytics", "loganalytics"))
+
     # Elasticsearch and OpenSearch mutate module-level global state via
     # connections.create_connection(), which cannot be rolled back if a later
     # step fails.  Initialise them last so that all other clients are created
     # successfully first; this minimizes the window for partial-init problems
     # during config reload.
     if opts.save_aggregate or opts.save_failure or opts.save_smtp_tls:
+        # Scoped to the same condition as the constructors below, which is
+        # also the condition under which process_reports() dereferences
+        # these modules to save reports.
+        if opts.elasticsearch_hosts and elastic is None:
+            raise ConfigurationError(_missing_extra_hint("elasticsearch", "elastic"))
+        if opts.opensearch_hosts and opensearch is None:
+            raise ConfigurationError(_missing_extra_hint("opensearch", "opensearch"))
+
         try:
             if opts.elasticsearch_hosts:
                 logger.debug(
@@ -1951,7 +2084,8 @@ def _main():
         Returns the list of human-readable output-error messages recorded
         along the way -- empty when every destination accepted the reports.
         Callers use that as the "was this batch saved?" signal; see
-        ``mailbox_save_callback()``.
+        ``mailbox_save_callback()``. With ``fail_on_output_error`` enabled,
+        a non-empty list is raised as ``ParserError`` instead of returned.
         """
         output_errors = []
 
@@ -2324,8 +2458,8 @@ def _main():
     arg_parser.add_argument(
         "file_path",
         nargs="*",
-        help="one or more paths to aggregate or failure report files, "
-        "emails, mbox files, or directories containing them",
+        help="one or more paths to aggregate, failure, or SMTP TLS report "
+        "files, emails, mbox files, or directories containing them",
     )
     arg_parser.add_argument(
         "-r",
@@ -2396,7 +2530,7 @@ def _main():
     arg_parser.add_argument(
         "--offline",
         action="store_true",
-        help="do not make online queries for geolocation  or  DNS",
+        help="do not make online queries for geolocation or DNS",
     )
     arg_parser.add_argument(
         "-s", "--silent", action="store_true", help="only print errors"

parsedmarc/elastic.py

@@ -930,6 +930,7 @@ def save_aggregate_report_to_elasticsearch(
 
     Raises:
             AlreadySaved
+            ElasticsearchError
     """
     logger.info("Saving aggregate report to Elasticsearch")
     aggregate_report = aggregate_report.copy()
@@ -1112,6 +1113,8 @@ def save_failure_report_to_elasticsearch(
 
     Raises:
         AlreadySaved
+        ElasticsearchError
+        InvalidFailureReport
 
     """
     logger.info("Saving failure report to Elasticsearch")
@@ -1297,6 +1300,7 @@ def save_smtp_tls_report_to_elasticsearch(
 
     Raises:
             AlreadySaved
+            ElasticsearchError
     """
     logger.info("Saving smtp tls report to Elasticsearch")
     org_name = report["organization_name"]

parsedmarc/kafkaclient.py

@@ -50,7 +50,7 @@ class KafkaClient(object):
             ssl_context (SSLContext): SSL context options
 
         Notes:
-            ``use_ssl=True`` is implied when a username or password are
+            ``ssl=True`` is implied when a username or password is
             supplied.
 
             When using Azure Event Hubs, the username is literally
@@ -81,7 +81,7 @@ class KafkaClient(object):
     def strip_metadata(report: dict[str, Any]):
         """
         Duplicates org_name, org_email and report_id into JSON root
-        and removes report_metadata key to bring it more inline
+        and removes report_metadata key to bring it more in line
         with Elastic output.
         """
         report["org_name"] = report["report_metadata"]["org_name"]
@@ -94,7 +94,7 @@ class KafkaClient(object):
     @staticmethod
     def generate_date_range(report: dict[str, Any]):
         """
-        Creates a date_range timestamp with format YYYY-MM-DD-T-HH:MM:SS
+        Creates a date_range timestamp with format YYYY-MM-DDTHH:MM:SS
         based on begin and end dates for easier parsing in Kibana.
 
         Move to utils to avoid duplication w/ elastic?
@@ -160,9 +160,7 @@ class KafkaClient(object):
         failure_topic: str,
     ):
         """
-        Saves failure DMARC reports to Kafka, sends individual
-        records (slices) since Kafka requires messages to be <= 1MB
-        by default.
+        Saves failure DMARC reports to Kafka as a single message
 
         Args:
             failure_reports (list):  A list of failure report dicts
@@ -197,9 +195,7 @@ class KafkaClient(object):
         smtp_tls_topic: str,
     ):
         """
-        Saves SMTP TLS reports to Kafka, sends individual
-        records (slices) since Kafka requires messages to be <= 1MB
-        by default.
+        Saves SMTP TLS reports to Kafka as a single message
 
         Args:
             smtp_tls_reports (list):  A list of SMTP TLS report dicts

parsedmarc/loganalytics.py

@@ -12,7 +12,7 @@ from parsedmarc.log import logger
 
 
 class LogAnalyticsException(Exception):
-    """Raised when an Elasticsearch error occurs"""
+    """Raised when a Log Analytics error occurs"""
 
 
 class LogAnalyticsConfig:
@@ -22,12 +22,12 @@ class LogAnalyticsConfig:
 
     Properties:
         client_id (str):
-            The client ID of the service principle.
+            The client ID of the service principal.
         client_secret (str):
-            The client secret of the service principle.
+            The client secret of the service principal.
         tenant_id (str):
             The tenant ID where
-            the service principle resides.
+            the service principal resides.
         dce (str):
             The Data Collection Endpoint (DCE)
             used by the Data Collection Rule (DCR).
@@ -115,16 +115,16 @@ class LogAnalyticsClient(object):
         dcr_stream: str,
     ):
         """
-        Background function to publish given
-        DMARC report to specific Data Collection Rule.
+        Publishes the given reports to the specified
+        Data Collection Rule stream.
 
         Args:
             results (list):
-                The results generated by parsedmarc.
+                The parsed reports generated by parsedmarc.
             logs_client (LogsIngestionClient):
-                The client used to send the DMARC reports.
+                The client used to send the reports.
             dcr_stream (str):
-                The stream name where the DMARC reports needs to be pushed.
+                The stream name where the reports need to be pushed.
         """
         try:
             logs_client.upload(self.conf.dcr_immutable_id, dcr_stream, results)
@@ -141,18 +141,18 @@ class LogAnalyticsClient(object):
         """
         Function to publish DMARC and/or SMTP TLS reports to Log Analytics
         via Data Collection Rules (DCR).
-        Look below for docs:
+        See:
         https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview
 
         Args:
-            results (list):
-                The DMARC reports (Aggregate & Failure)
+            results (dict):
+                The parsing results (aggregate, failure, and SMTP TLS reports)
             save_aggregate (bool):
-                Whether Aggregate reports can be saved into Log Analytics
+                Whether aggregate reports can be saved into Log Analytics
             save_failure (bool):
-                Whether Failure reports can be saved into Log Analytics
+                Whether failure reports can be saved into Log Analytics
             save_smtp_tls (bool):
-                Whether Failure reports can be saved into Log Analytics
+                Whether SMTP TLS reports can be saved into Log Analytics
         """
         conf = self.conf
         credential = ClientSecretCredential(

parsedmarc/opensearch.py

@@ -861,6 +861,7 @@ def save_aggregate_report_to_opensearch(
 
     Raises:
             AlreadySaved
+            OpenSearchError
     """
     logger.info("Saving aggregate report to OpenSearch")
     aggregate_report = aggregate_report.copy()
@@ -1040,6 +1041,8 @@ def save_failure_report_to_opensearch(
 
     Raises:
         AlreadySaved
+        OpenSearchError
+        InvalidFailureReport
 
     """
     logger.info("Saving failure report to OpenSearch")
@@ -1223,6 +1226,7 @@ def save_smtp_tls_report_to_opensearch(
 
     Raises:
             AlreadySaved
+            OpenSearchError
     """
     logger.info("Saving SMTP TLS report to OpenSearch")
     org_name = report["organization_name"]

parsedmarc/postgres.py

@@ -443,8 +443,8 @@ class PostgreSQLClient:
         """Saves a parsed aggregate DMARC report to PostgreSQL.
 
         Args:
-            report: A parsed aggregate report dictionary as returned by
-                :func:`parsedmarc.parse_report_file`.
+            report: A parsed aggregate report dictionary (the ``report``
+                value of a :func:`parsedmarc.parse_report_file` result).
 
         Raises:
             AlreadySaved: If an identical report is already present.
@@ -620,8 +620,8 @@ class PostgreSQLClient:
         """Saves a parsed failure (RUF) DMARC report to PostgreSQL.
 
         Args:
-            report: A parsed failure report dictionary as returned by
-                :func:`parsedmarc.parse_report_file`.
+            report: A parsed failure report dictionary (the ``report``
+                value of a :func:`parsedmarc.parse_report_file` result).
 
         Raises:
             AlreadySaved: If a matching failure report is already present.
@@ -759,8 +759,8 @@ class PostgreSQLClient:
         """Saves a parsed SMTP TLS report to PostgreSQL.
 
         Args:
-            report: A parsed SMTP TLS report dictionary as returned by
-                :func:`parsedmarc.parse_report_file`.
+            report: A parsed SMTP TLS report dictionary (the ``report``
+                value of a :func:`parsedmarc.parse_report_file` result).
 
         Raises:
             AlreadySaved: If an identical report is already present.

parsedmarc/resources/maps/AGENTS.md

@@ -124,7 +124,7 @@ When `unknown_base_reverse_dns.csv` has new entries, follow this order rather th
   **Link-following: when the search snippet is just a hostname pointer.** DDG sometimes returns titles like `Link to fcs.health.gov.il` (literal placeholder for a subdomain it indexed but never snapshotted) or just `yangon.mfa.gov.il` (bare hostname, no other words). Those snippets carry no classifier signal — there's no description of the operator, no industry vocabulary, just the host name. The collector recognizes both patterns (`Link to <hostname>` prefix and bare-hostname-only titles) and follows the pointer: it fetches the target hostname directly with `_fetch_homepage`, and if the fetch returns real (non-bot-blocked) content, replaces the row's title and description with that content. The link target is recorded in a `link_target_domain` column. `title_source` is set to `search→<target>` to make the path auditable.
 
   When `link_target_domain` is set on a row that classifies, `classify_unknown_domains.py` emits **two** map rows under the same `(name, type)` — the original input *and* the target — so both keys can be looked up. The original input is the "og" domain; the target is what the search engine led us to. Both belong in the map: the same operator may show up in DMARC reports under either base.
-- `classify_unknown_domains.py` — regex-based multilingual classifier that consumes a `collect_domain_info.py` TSV and emits map / ambiguous / known-unknown additions. Useful for both lookup paths into `base_reverse_dns_map.csv`: the original PTR-side flow (classifying reverse-DNS base domains discovered from DMARC report source IPs) and the MMDB-coverage flow (classifying ASN domains lifted from the bundled IPinfo Lite MMDB). Detectors cover all 44 industry types in the README, and every detector aims for **concept parity across the same broad language pool** — see the concept-parity rule below. The classifier is the regex baseline of step 4 of the unknown-domain workflow (see "Workflow for classifying unknown domains" above) — it catches the obvious cases at scale and leaves the genuinely ambiguous to manual / LLM review.
+- `classify_unknown_domains.py` — regex-based multilingual classifier that consumes a `collect_domain_info.py` TSV and emits map / ambiguous / known-unknown additions. Useful for both lookup paths into `base_reverse_dns_map.csv`: the original PTR-side flow (classifying reverse-DNS base domains discovered from DMARC report source IPs) and the MMDB-coverage flow (classifying ASN domains lifted from the bundled IPinfo Lite MMDB). Detectors cover all 46 industry types in the README, and every detector aims for **concept parity across the same broad language pool** — see the concept-parity rule below. The classifier is the regex baseline of step 4 of the unknown-domain workflow (see "Workflow for classifying unknown domains" above) — it catches the obvious cases at scale and leaves the genuinely ambiguous to manual / LLM review.
 
   **Three output buckets**. Per-row, the classifier returns one of three states:
   1. `--map-out` (CSV `domain,name,type`) — exactly one detector category fired. Auto-promote: append to `base_reverse_dns_map.csv`.

parsedmarc/resources/maps/README.md

@@ -169,7 +169,7 @@ python classify_unknown_domains.py \
     --ku-out /tmp/ku_additions.txt
 ```
 
-Detectors cover all 44 industry types listed in [base_reverse_dns_map.csv](#base_reverse_dns_mapcsv) above. Multilingual coverage is broadest for the high-volume detectors — Healthcare, Travel, Government, Retail, Finance, ISP, Web Host, Manufacturing, Logistics, Real Estate, Automotive, Legal, Agriculture have concept-translation parity across ~30 languages with multiple synonyms per language. Smaller detectors (Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media, Defense, IaaS, PaaS, SaaS, Beauty, Print, Publishing, Religion, Science, Event Planning, Staffing, Email Security, Email Provider, Marketing, Construction, Industrial, Utilities, Energy, Government Media, Physical Security, News, Nonprofit, Entertainment, Technology, Consulting) have ~10–20 languages with 1–3 keywords each. Each successive batch is expected to refine multilingual coverage as new patterns surface in the unclassified pool.
+Detectors cover all 46 industry types listed in [base_reverse_dns_map.csv](#base_reverse_dns_mapcsv) above. Multilingual coverage is broadest for the high-volume detectors — Healthcare, Travel, Government, Retail, Finance, ISP, Web Host, Manufacturing, Logistics, Real Estate, Automotive, Legal, Agriculture have concept-translation parity across ~30 languages with multiple synonyms per language. Smaller detectors (Photography, Sports, MSSP, Conglomerate, Search Engine, Social Media, Defense, IaaS, PaaS, SaaS, Beauty, Print, Publishing, Religion, Science, Event Planning, Staffing, Email Security, Email Provider, Marketing, Construction, Industrial, Utilities, Energy, Government Media, Physical Security, News, Nonprofit, Entertainment, Technology, Consulting) have ~10–20 languages with 1–3 keywords each. Each successive batch is expected to refine multilingual coverage as new patterns surface in the unclassified pool.
 
 Brand-name selection prefers (in order): the MMDB `as_name` for the domain; the page title's first segment; non-redacted WHOIS registrant org; domain-derived fallback. A `clean_brand` step strips common legal-form suffixes (LLC / GmbH / Ltda / EIRELI / sp. z o.o. / etc.) and prefixes (PT, OOO). When the title has multiple segments separated by `|` / `-` / `—` etc., the segment whose simplified form contains the domain root is preferred — so e.g. accessmontana.com whose `as_name` is "MONTANA WEST, L.L.C." but whose title is "Internet, Phone & TV Bundles | Access Montana" maps to "Access Montana", not "Montana West".
 

parsedmarc/resources/maps/classify_unknown_domains.py

@@ -27,7 +27,7 @@ classifier (or skim it by hand)" — this script is the regex baseline that
 catches obvious cases at scale and leaves only the genuinely ambiguous to
 manual / LLM review.
 
-Detectors cover all 44 industry types listed in `README.md` (every type
+Detectors cover all 46 industry types listed in `README.md` (every type
 defined for `base_reverse_dns_map.csv`'s `type` column). Every detector
 aims for concept-translation parity across the same broad language pool
 (typically 25–35 languages including major Romance, Germanic, Slavic,
@@ -62,6 +62,11 @@ Outputs:
     --map-out: three-column CSV (domain, name, type) — append to
         base_reverse_dns_map.csv
     --ku-out: one domain per line — append to known_unknown_base_reverse_dns.txt
+    --ambiguous-out: TSV (domain, name, primary_type, alternatives, title) of
+        rows where two or more distinct detector categories fired — not
+        auto-promoted; a human must adjudicate each row
+    --dropped-out: one domain per line — domains silently dropped per the
+        AGENTS.md content rule; remove them from any tracked list files
 
 The HAND dict at the top of the file is an extension point for explicit
 overrides (e.g. acquisition aliases, brand-name corrections). It is empty
@@ -9808,6 +9813,8 @@ def classify_tsv(
       classifier won't auto-promote these — a human must pick one of the
       candidates (or a different category, or reject the row to KU).
     - ``ku`` — domains where no detector fired.
+    - ``dropped`` — domains silently dropped per the AGENTS.md content rule;
+      the caller removes these from any tracked list files.
     - ``stats`` — counters.
 
     ``map_names`` is ``{normalized display name: {existing map keys}}`` as

parsedmarc/resources/maps/collect_domain_info.py

@@ -7,7 +7,8 @@ useful for classifying an unknown sender:
 
     domain, whois_org, whois_country, registrar, title, description,
     rebrand_signal, external_links, final_url, http_status, ips,
-    ip_whois_org, ip_whois_netname, ip_whois_country, error
+    ip_whois_org, ip_whois_netname, ip_whois_country, error,
+    title_source, link_target_domain
 
 `rebrand_signal` flags rows whose page text matches a phrase like "now X" or
 "formerly known as X" — useful both for classifying an unknown sender ("we

parsedmarc/resources/maps/sortlists.py

@@ -114,8 +114,10 @@ def sort_csv(
 
     - filepath: Path to the CSV to sort.
     - field: The field name to sort by.
+    - sort_field_value_must_be_unique: Require each row's sort-field value to
+      be unique across the file.
     - fields_to_lowercase: Permanently lowercases these field(s) in the data.
-    - strip_whitespace: Remove all whitespace at the beginning and of field values.
+    - strip_whitespace: Remove whitespace at the beginning and end of field values.
     - case_insensitive_sort: Ignore case when sorting without changing values.
     - required_fields: A list of fields that must have data in all rows.
     - allowed_values: A mapping of allowed values for fields.
@@ -205,9 +207,11 @@ def sort_list_file(
     """Read a list from a file, sort it, optionally strip and deduplicate the values,
     then write that list back to the file.
 
-    - Filepath: The path to the file.
+    - filepath: The path to the file.
     - lowercase: Lowercase all values prior to sorting.
-    - remove_blank_lines: Remove any plank lines.
+    - strip: Strip leading and trailing whitespace from each value.
+    - deduplicate: Remove duplicate values.
+    - remove_blank_lines: Remove any blank lines.
     - ending_newline: End the file with a newline, even if remove_blank_lines is true.
     - newline: The newline character to use.
     """

parsedmarc/splunk.py

@@ -188,7 +188,7 @@ class HECClient(object):
         self, reports: list[dict[str, Any]] | dict[str, Any]
     ):
         """
-        Saves aggregate DMARC reports to Splunk
+        Saves SMTP TLS reports to Splunk
 
         Args:
             reports: A list of SMTP TLS report dictionaries

parsedmarc/utils.py

@@ -89,7 +89,7 @@ def load_psl_overrides(
         always_use_local_file (bool): Always use a local overrides file
         local_file_path (str): Path to a local overrides file
         url (str): URL to a PSL overrides file
-        offline (bool): Use the built-in copy of the overrides
+        offline (bool): Do not make online requests
 
     Returns:
         list[str]: the module-level ``psl_overrides`` list
@@ -195,7 +195,8 @@ def get_base_domain(domain: str) -> str | None:
         domain (str): A domain or subdomain
 
     Returns:
-        str: The base domain of the given domain
+        str: The base domain of the given domain, or ``None`` if one
+        cannot be determined
 
     """
     domain = domain.lower()
@@ -570,7 +571,7 @@ def human_timestamp_to_unix_timestamp(
     Converts a human-readable timestamp into a UNIX timestamp
 
     Args:
-        human_timestamp (str): A timestamp in `YYYY-MM-DD HH:MM:SS`` format
+        human_timestamp (str): A timestamp in ``YYYY-MM-DD HH:MM:SS`` format
         assume_utc (bool): Treat a timestamp that carries no UTC offset as
             UTC wall-clock time instead of local time
 
@@ -599,7 +600,9 @@ def load_ip_db(
 ) -> None:
     """
     Downloads the IP-to-country MMDB database from a URL and caches it
-    locally. Falls back to the bundled copy on failure or when offline.
+    locally. An existing ``local_file_path`` is used as-is, with no
+    download. On download failure (or when offline), a previously cached
+    download is used if available, falling back to the bundled copy.
 
     Args:
         always_use_local_file: Always use a local/bundled database file
@@ -689,9 +692,9 @@ def configure_ipinfo_api(
     """Configure the IPinfo Lite REST API as the primary source for IP lookups.
 
     When a token is configured, ``get_ip_address_db_record()`` hits the API
-    first for every lookup and falls back to the MMDB on network errors. An
-    invalid token raises ``InvalidIPinfoAPIKey`` — the CLI catches that and
-    exits fatally.
+    first for every lookup and falls back to the MMDB on network errors or
+    non-2xx responses. An invalid token raises ``InvalidIPinfoAPIKey`` — the
+    CLI catches that and exits fatally.
 
     Args:
         token: IPinfo API token. ``None`` or empty disables the API.
@@ -718,8 +721,9 @@ def configure_ipinfo_api(
 def _ipinfo_api_lookup(ip_address: str) -> _IPDatabaseRecord | None:
     """Look up an IP via the IPinfo Lite REST API.
 
-    Returns the normalized record on success, or ``None`` on network error or
-    any non-2xx response (other than 401/403). 401/403 raises
+    Returns the normalized record on success, or ``None`` when no token is
+    configured, on network error, on a malformed response body, or on any
+    non-2xx response other than 401/403. 401/403 raises
     ``InvalidIPinfoAPIKey``.
     """
     if not _IPINFO_API_TOKEN:
@@ -881,13 +885,14 @@ def get_ip_address_db_record(
     """Look up an IP and return country + ASN fields.
 
     If the IPinfo Lite API is configured via ``configure_ipinfo_api()``, the
-    API is queried first; any non-fatal failure (rate limit, quota, network)
-    falls through to the MMDB. An invalid API token raises
-    ``InvalidIPinfoAPIKey`` and is not caught here.
+    API is queried first; any non-fatal failure (a network error, or a
+    non-2xx response other than 401/403) falls through to the MMDB. An
+    invalid API token raises ``InvalidIPinfoAPIKey`` and is not caught here.
 
-    IPinfo Lite carries ``country_code``, ``as_name``, and ``as_domain`` on
-    every record. MaxMind/DBIP country-only databases carry only country, so
-    ``as_name`` / ``as_domain`` come back None for those users.
+    IPinfo Lite carries ``country_code``, ``asn``, ``as_name``, and
+    ``as_domain`` on every record. MaxMind/DBIP country-only databases carry
+    only country, so ``asn`` / ``as_name`` / ``as_domain`` come back None
+    for those users.
     """
     api_record = _ipinfo_api_lookup(ip_address)
     if api_record is not None:
@@ -918,7 +923,8 @@ def get_ip_address_country(
         db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP
 
     Returns:
-        str: And ISO country code associated with the given IP address
+        str: An ISO country code associated with the given IP address,
+        or ``None`` if the country is unknown
     """
     return get_ip_address_db_record(ip_address, db_path=db_path)["country"]
 
@@ -936,9 +942,9 @@ def load_reverse_dns_map(
     """
     Loads the reverse DNS map from a URL or local file.
 
-    Clears and repopulates the given map dict in place. If the map is
-    fetched from a URL, that is tried first; on failure (or if offline/local
-    mode is selected) the bundled CSV is used as a fallback.
+    Clears and repopulates the given map dict in place. The URL is tried
+    first; on failure (or when ``offline``/``always_use_local_file`` is set)
+    the local path is used, defaulting to the bundled CSV.
 
     ``psl_overrides.txt`` is reloaded at the same time using the same
     ``offline`` / ``always_use_local_file`` flags (with separate path/URL
@@ -950,7 +956,7 @@ def load_reverse_dns_map(
         always_use_local_file (bool): Always use a local map file
         local_file_path (str): Path to a local map file
         url (str): URL to a reverse DNS map
-        offline (bool): Use the built-in copy of the reverse DNS map
+        offline (bool): Do not make online requests
         psl_overrides_path (str): Path to a local PSL overrides file
         psl_overrides_url (str): URL to a PSL overrides file
     """
@@ -1032,14 +1038,15 @@ def get_service_from_reverse_dns_base_domain(
         always_use_local_file (bool): Always use a local map file
         local_file_path (str): Path to a local map file
         url (str): URL to a reverse DNS map
-        offline (bool): Use the built-in copy of the reverse DNS map
+        offline (bool): Do not make online requests
         reverse_dns_map (dict): A reverse DNS map
         psl_overrides_path (str): Path to a local PSL overrides file
         psl_overrides_url (str): URL to a PSL overrides file
+
     Returns:
         dict: A dictionary containing name and type.
         If the service is unknown, the name will be
-        the supplied reverse_dns_base_domain and the type will be None
+        the supplied ``base_domain`` and the type will be None
     """
 
     base_domain = base_domain.lower().strip()
@@ -1086,14 +1093,15 @@ def get_ip_address_info(
     psl_overrides_url: str | None = None,
 ) -> IPAddressInfo:
     """
-    Returns reverse DNS and country information for the given IP address
+    Returns reverse DNS, country, ASN, and service information for the
+    given IP address
 
     Args:
         ip_address (str): The IP address to check
-        ip_db_path (str): path to a MMDB file from MaxMind or DBIP
+        ip_db_path (str): Path to a MMDB file from IPinfo, MaxMind, or DBIP
         reverse_dns_map_path (str): Path to a reverse DNS map file
-        reverse_dns_map_url (str): URL to the reverse DNS map file
         always_use_local_files (bool): Do not download files
+        reverse_dns_map_url (str): URL to the reverse DNS map file
         cache (ExpiringDict): Cache storage
         reverse_dns_map (dict): A reverse DNS map
         offline (bool): Do not make online queries for geolocation or DNS
@@ -1106,7 +1114,8 @@ def get_ip_address_info(
         psl_overrides_url (str): URL to a PSL overrides file
 
     Returns:
-        dict: ``ip_address``, ``reverse_dns``, ``country``
+        dict: ``ip_address``, ``reverse_dns``, ``country``, ``base_domain``,
+        ``name``, ``type``, ``asn``, ``as_name``, ``as_domain``
 
     """
     ip_address = ip_address.lower()
@@ -1256,13 +1265,13 @@ def get_filename_safe_string(string: str) -> str:
 
 def is_mbox(path: str) -> bool:
     """
-    Checks if the given content is an MBOX mailbox file
+    Checks if the file at the given path is an mbox mailbox file
 
     Args:
-        path: Content to check
+        path (str): Path to the file to check
 
     Returns:
-        bool: A flag that indicates if the file is an MBOX mailbox file
+        bool: A flag that indicates if the file is an mbox mailbox file
     """
     _is_mbox = False
     try:
@@ -1292,14 +1301,18 @@ def is_outlook_msg(content) -> bool:
 
 def convert_outlook_msg(msg_bytes: bytes) -> bytes:
     """
-    Uses the ``msgconvert`` Perl utility to convert an Outlook MS file to
+    Uses the ``msgconvert`` Perl utility to convert an Outlook MSG file to
     standard RFC 822 format
 
     Args:
         msg_bytes (bytes): the content of the .msg file
 
     Returns:
-        A RFC 822 bytes payload
+        An RFC 822 bytes payload
+
+    Raises:
+        ValueError: The supplied bytes are not an Outlook MSG file
+        EmailParserError: The ``msgconvert`` utility is not installed
     """
     if not is_outlook_msg(msg_bytes):
         raise ValueError("The supplied bytes are not an Outlook MSG file")

pyproject.toml

@@ -36,37 +36,90 @@ classifiers = [
     "Programming Language :: Python :: 3.14",
 ]
 requires-python = ">=3.10"
+# The base install is the parsing core plus a working core CLI: file,
+# IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and
+# syslog output. Every other output and mailbox integration lives in an extra
+# below, so a library or small-mail-host install does not have to carry
+# the Elasticsearch, OpenSearch, Kafka, AWS, Azure, Gmail, and Microsoft
+# Graph SDKs (see issue #883).
 dependencies = [
-    "azure-identity>=1.8.0",
-    "azure-monitor-ingestion>=1.0.0",
-    "boto3>=1.16.63",
-    "dateparser>=1.1.1",
     # The [doh] extra supplies the httpx/h2 floors DNS over HTTPS needs;
     # 2.7.0 is the floor verified against the dns.nameserver and
     # dns.query.https(session=...) APIs utils.py builds on.
     "dnspython[doh]>=2.7.0",
-    "elasticsearch>=8.18,<9",
     "expiringdict>=1.1.4",
     # The runtime HTTP library (utils.py fetches, webhook and Splunk HEC
     # clients, Graph error handling in cli.py). The floor matches
     # microsoft-kiota-http's own requirement.
     "httpx>=0.25",
-    "kafka-python>=2.3.2",
     "lxml>=4.4.0",
-    "mailsuite[gmail,msgraph]>=2.3.1",
+    # No extras: the base mailsuite supplies IMAP and Maildir connections
+    # plus mail-parser. Gmail and Microsoft Graph come from the gmail and
+    # msgraph extras below.
+    "mailsuite>=2.3.1",
     "maxminddb>=2.0.0",
-    # Imported directly in cli.py for Graph error handling; otherwise
-    # only a transitive dep of mailsuite[msgraph] -> msgraph-sdk.
-    "microsoft-kiota-abstractions>=1.8.0",
-    "opensearch-py>=2.4.2,<=4.0.0",
     "publicsuffixlist>=0.10.0",
-    "pygelf>=0.4.2",
+    # Imported directly at utils.py:39 (dateutil.parser). It used to
+    # arrive transitively via dateparser, which nothing ever imported.
+    "python-dateutil>=2.8.0",
     "tqdm>=4.31.1",
     "xmltodict>=0.12.0",
     "PyYAML>=6.0.3"
 ]
 
+# Splunk HEC, webhook, and syslog outputs deliberately have no extra:
+# they need only httpx (a base dependency) and the standard library, so
+# do not add empty marker extras for them.
 [project.optional-dependencies]
+elastic = [
+    "elasticsearch>=8.18,<9",
+]
+opensearch = [
+    "opensearch-py>=2.4.2,<=4.0.0",
+    # boto3 supplies the SigV4 signer opensearch.py imports for AWS auth.
+    "boto3>=1.16.63",
+]
+kafka = [
+    "kafka-python>=2.3.2",
+]
+s3 = [
+    "boto3>=1.16.63",
+]
+gelf = [
+    "pygelf>=0.4.2",
+]
+loganalytics = [
+    "azure-identity>=1.8.0",
+    "azure-monitor-ingestion>=1.0.0",
+]
+msgraph = [
+    "mailsuite[msgraph]>=2.3.1",
+    # Imported directly in cli.py for Graph error handling; otherwise
+    # only a transitive dep of mailsuite[msgraph] -> msgraph-sdk.
+    "microsoft-kiota-abstractions>=1.8.0",
+]
+gmail = [
+    "mailsuite[gmail]>=2.3.1",
+]
+# `all` must stay the union of every extra above — and must keep
+# excluding `postgresql`: psycopg's prebuilt binary wheels do not exist
+# for every platform/arch, so folding it in here would make
+# `pip install parsedmarc[all]` fail on platforms it does not cover.
+# Spelled out as an explicit package list rather than a self-referential
+# `parsedmarc[elastic,...]` spec, which would make the project a
+# dependency of itself; keep this list in sync when an extra above
+# gains, drops, or re-floors a package.
+all = [
+    "azure-identity>=1.8.0",
+    "azure-monitor-ingestion>=1.0.0",
+    "boto3>=1.16.63",
+    "elasticsearch>=8.18,<9",
+    "kafka-python>=2.3.2",
+    "mailsuite[gmail,msgraph]>=2.3.1",
+    "microsoft-kiota-abstractions>=1.8.0",
+    "opensearch-py>=2.4.2,<=4.0.0",
+    "pygelf>=0.4.2",
+]
 postgresql = [
     # Optional output backend. psycopg ships prebuilt binary wheels via the
     # [binary] extra, but those wheels don't exist for every platform/arch,
@@ -89,6 +142,10 @@ build = [
     "pyright==1.1.411",
     "pytest",
     "pytest-cov",
+    # Test-only: tests/test_init.py imports pytz for a fixed-offset
+    # timezone. It used to arrive transitively via dateparser, which the
+    # base dependencies no longer declare.
+    "pytz",
     # Used only by the out-of-wheel maintainer script
     # parsedmarc/resources/maps/collect_domain_info.py, which deliberately
     # stays on requests because its permissive-TLS fallback is built on
@@ -163,9 +220,10 @@ select = [
 [tool.pyright]
 # The whole codebase passes pyright with zero errors and warnings; CI
 # enforces this (see .github/workflows/python-tests.yml). Run locally with
-# `pyright` from the repo root. Requires the [postgresql] extra to be
-# installed so the optional psycopg import in parsedmarc/postgres.py
-# resolves.
+# `pyright` from the repo root. Requires the [all] and [postgresql] extras
+# to be installed so the optional integration imports (parsedmarc/cli.py's
+# guarded output modules, the psycopg import in parsedmarc/postgres.py)
+# resolve.
 include = ["parsedmarc", "tests", "docs"]
 typeCheckingMode = "standard"
 

tests/test_cli.py

@@ -1,6 +1,9 @@
 """Tests for parsedmarc.cli — CLI entry point, config parsing,
 env-var overrides, mailbox watch wiring, and SIGHUP reload."""
 
+import importlib
+import importlib.abc
+import importlib.machinery
 import io
 import json
 import logging
@@ -11,14 +14,17 @@ import sys
 import tempfile
 import unittest
 import zipfile
+from collections.abc import Iterator, Sequence
 from configparser import ConfigParser
+from contextlib import contextmanager
 from tempfile import NamedTemporaryFile
-from types import SimpleNamespace
+from types import ModuleType, SimpleNamespace
 from typing import cast
 from unittest.mock import MagicMock, patch
 
 import httpx
 from azure.core.exceptions import ClientAuthenticationError
+from kiota_abstractions.api_error import APIError
 from msgraph.generated.models.o_data_errors.inner_error import InnerError
 from msgraph.generated.models.o_data_errors.main_error import MainError
 from msgraph.generated.models.o_data_errors.o_data_error import ODataError
@@ -26,6 +32,8 @@ from msgraph.generated.models.o_data_errors.o_data_error import ODataError
 import parsedmarc
 import parsedmarc.cli
 import parsedmarc.elastic
+import parsedmarc.log
+import parsedmarc.mail
 import parsedmarc.opensearch as opensearch_module
 from parsedmarc.types import AggregateReport, ParsedReport
 
@@ -6499,6 +6507,18 @@ class TestPostgreSQLCliWiring(unittest.TestCase):
     _init_output_clients wiring can't drift apart.
     """
 
+    def setUp(self):
+        # These tests simulate "[postgresql] configured and the SDK
+        # available", mocking PostgreSQLClient at the SDK boundary. CI's
+        # unit-test job deliberately omits the postgresql extra, so the
+        # module-level psycopg handle must be patched present too --
+        # otherwise _init_output_clients' presence check correctly
+        # reports the missing extra and _main exits 1 before any of the
+        # wiring under test runs.
+        patcher = patch.object(parsedmarc.cli.postgres, "psycopg", MagicMock())
+        patcher.start()
+        self.addCleanup(patcher.stop)
+
     def test_postgresql_config_constructs_client_and_creates_tables(self):
         config = """[general]
 save_aggregate = true
@@ -6983,5 +7003,455 @@ class TestConfigureLogging(unittest.TestCase):
         self.assertTrue(any("Unable to write to log file" in m for m in cm.output))
 
 
+# Top-level module roots pulled in by parsedmarc's optional output and
+# mailbox extras. Blocking all of them makes an install with no extras
+# at all -- the new default (#883) -- look the way it does to the import
+# system.
+OPTIONAL_SDK_ROOTS = (
+    "azure",
+    "boto3",
+    "botocore",
+    "elastic_transport",
+    "elasticsearch",
+    "google",
+    "google_auth_oauthlib",
+    "googleapiclient",
+    "kafka",
+    "kiota_abstractions",
+    "kiota_authentication_azure",
+    "msgraph",
+    "msgraph_core",
+    "opensearchpy",
+    "pygelf",
+)
+
+# The parsedmarc submodules cli.py imports behind the extras guard, in
+# the order their guarded import blocks appear in cli.py.
+GUARDED_CLI_MODULES = (
+    "elastic",
+    "gelf",
+    "kafkaclient",
+    "loganalytics",
+    "opensearch",
+    "s3",
+)
+
+
+class _ImportBlocker(importlib.abc.MetaPathFinder):
+    """A meta path finder that makes chosen module roots unimportable.
+
+    Installed at the head of ``sys.meta_path``, it raises
+    ``ModuleNotFoundError`` for any module whose top-level root is
+    blocked — the exact exception a genuinely absent package produces,
+    and the only one cli.py's import guards catch, so a broken-but-
+    present SDK still fails loudly instead of masquerading as a missing
+    extra.
+    """
+
+    def __init__(self, roots: frozenset[str]) -> None:
+        self.roots = roots
+
+    def find_spec(
+        self,
+        fullname: str,
+        path: Sequence[str] | None = None,
+        target: ModuleType | None = None,
+    ) -> importlib.machinery.ModuleSpec | None:
+        if fullname.split(".")[0] in self.roots:
+            raise ModuleNotFoundError(f"blocked for test: {fullname}")
+        return None
+
+
+def _is_affected(name: str) -> bool:
+    """Is *name* a module that has to be reloaded while the SDKs are blocked?
+
+    The blocked roots themselves, the whole ``mailsuite`` tree (its
+    ``mailbox`` package caches the ``gmail``/``graph`` submodules as
+    attributes once loaded, so a stale entry would satisfy
+    ``parsedmarc.mail``'s guarded imports without ever reaching a blocked
+    root), and the ``parsedmarc`` submodules that import the blocked SDKs.
+    """
+    root = name.split(".")[0]
+    if root in OPTIONAL_SDK_ROOTS or root == "mailsuite":
+        return True
+    return name in {f"parsedmarc.{m}" for m in (*GUARDED_CLI_MODULES, "mail")}
+
+
+@contextmanager
+def _cli_without_optional_sdks() -> Iterator[ModuleType]:
+    """Reload :mod:`parsedmarc.cli` with every optional SDK unimportable.
+
+    Yields the reloaded module -- the same object as before, since
+    ``importlib.reload`` re-executes in place. On exit, including when
+    the body raises, the finder is removed, anything imported while
+    blocked is discarded, the snapshots are put back, and
+    :mod:`parsedmarc.cli` is reloaded once more so later tests see the
+    real modules again. ``parsedmarc``'s own submodule attributes are
+    snapshotted and deleted too: ``from parsedmarc import elastic``
+    resolves against the package's attributes first, so leaving them in
+    place would satisfy the import without consulting the blocker.
+    """
+    blocker = _ImportBlocker(frozenset(OPTIONAL_SDK_ROOTS))
+    removed_modules: dict[str, ModuleType] = {}
+    for name in list(sys.modules):
+        if _is_affected(name):
+            removed_modules[name] = sys.modules.pop(name)
+    removed_attrs: dict[str, ModuleType] = {}
+    for attr in (*GUARDED_CLI_MODULES, "mail"):
+        if attr in vars(parsedmarc):
+            removed_attrs[attr] = getattr(parsedmarc, attr)
+            delattr(parsedmarc, attr)
+    # Every reload re-runs cli.py's module level, which adds a stream
+    # handler to the parsedmarc logger; restore the original list so the
+    # handlers do not accumulate across tests.
+    handlers = list(parsedmarc.log.logger.handlers)
+    sys.meta_path.insert(0, blocker)
+    try:
+        importlib.invalidate_caches()
+        importlib.reload(parsedmarc.cli)
+        yield parsedmarc.cli
+    finally:
+        sys.meta_path.remove(blocker)
+        for name in list(sys.modules):
+            if _is_affected(name):
+                del sys.modules[name]
+        sys.modules.update(removed_modules)
+        for attr, value in removed_attrs.items():
+            setattr(parsedmarc, attr, value)
+        importlib.invalidate_caches()
+        importlib.reload(parsedmarc.cli)
+        parsedmarc.log.logger.handlers[:] = handlers
+
+
+def _output_client_opts(**overrides) -> SimpleNamespace:
+    """Build the opts namespace ``_init_output_clients`` reads.
+
+    Every option defaults to ``None`` -- falsy, so each output's ``if``
+    guard is skipped -- and *overrides* turn on exactly one output.
+    """
+    names = (
+        "elasticsearch_api_key",
+        "elasticsearch_hosts",
+        "elasticsearch_index_prefix",
+        "elasticsearch_index_suffix",
+        "elasticsearch_password",
+        "elasticsearch_serverless",
+        "elasticsearch_skip_certificate_verification",
+        "elasticsearch_ssl",
+        "elasticsearch_ssl_cert_path",
+        "elasticsearch_timeout",
+        "elasticsearch_username",
+        "gelf_host",
+        "gelf_mode",
+        "gelf_port",
+        "hec",
+        "hec_index",
+        "hec_skip_certificate_verification",
+        "hec_token",
+        "kafka_hosts",
+        "kafka_password",
+        "kafka_skip_certificate_verification",
+        "kafka_username",
+        "la_dce",
+        "opensearch_api_key",
+        "opensearch_auth_type",
+        "opensearch_aws_region",
+        "opensearch_aws_service",
+        "opensearch_hosts",
+        "opensearch_index_prefix",
+        "opensearch_index_suffix",
+        "opensearch_password",
+        "opensearch_skip_certificate_verification",
+        "opensearch_ssl",
+        "opensearch_ssl_cert_path",
+        "opensearch_timeout",
+        "opensearch_username",
+        "postgresql_connection_string",
+        "postgresql_database",
+        "postgresql_host",
+        "postgresql_password",
+        "postgresql_port",
+        "postgresql_user",
+        "s3_access_key_id",
+        "s3_bucket",
+        "s3_endpoint_url",
+        "s3_path",
+        "s3_region_name",
+        "s3_secret_access_key",
+        "save_aggregate",
+        "save_failure",
+        "save_smtp_tls",
+        "syslog_cafile_path",
+        "syslog_certfile_path",
+        "syslog_keyfile_path",
+        "syslog_port",
+        "syslog_protocol",
+        "syslog_retry_attempts",
+        "syslog_retry_delay",
+        "syslog_server",
+        "syslog_timeout",
+        "webhook_aggregate_url",
+        "webhook_failure_url",
+        "webhook_smtp_tls_url",
+        "webhook_timeout",
+    )
+    unknown = set(overrides) - set(names)
+    assert not unknown, f"unknown opts: {sorted(unknown)}"
+    values: dict[str, object] = {name: None for name in names}
+    values.update(overrides)
+    return SimpleNamespace(**values)
+
+
+class TestOptionalIntegrationExtras(unittest.TestCase):
+    """The optional-extras packaging split (#883).
+
+    ``pip install parsedmarc`` no longer ships the Elasticsearch,
+    OpenSearch, Kafka, AWS, Azure, Gmail, or Microsoft Graph SDKs, so
+    ``parsedmarc.cli`` must import without any of them, and a config
+    section whose extra is missing must fail with an actionable
+    ``ConfigurationError`` rather than an ``ImportError`` traceback or an
+    ``AttributeError`` on ``None``.
+    """
+
+    # (cli.py module global, INI section name, extra name, opts that
+    # select that output, the key _init_output_clients() adds for it --
+    # None for Log Analytics, whose client is built per batch in
+    # process_reports() instead)
+    GUARDED_OUTPUTS = (
+        (
+            "elastic",
+            "elasticsearch",
+            "elastic",
+            {"elasticsearch_hosts": ["host:9200"]},
+            "elasticsearch",
+        ),
+        (
+            "opensearch",
+            "opensearch",
+            "opensearch",
+            {"opensearch_hosts": ["host:9200"]},
+            "opensearch",
+        ),
+        (
+            "kafkaclient",
+            "kafka",
+            "kafka",
+            {"kafka_hosts": ["host:9092"]},
+            "kafka_client",
+        ),
+        ("s3", "s3", "s3", {"s3_bucket": "reports"}, "s3_client"),
+        (
+            "gelf",
+            "gelf",
+            "gelf",
+            {"gelf_host": "logger", "gelf_port": 12201},
+            "gelf_client",
+        ),
+        (
+            "loganalytics",
+            "log_analytics",
+            "loganalytics",
+            {"la_dce": "https://dce.example.com"},
+            None,
+        ),
+    )
+
+    def test_cli_imports_with_no_optional_sdk_installed(self):
+        """parsedmarc.cli imports with every optional SDK absent.
+
+        This is the base-install guarantee: the core CLI (file, IMAP,
+        Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and
+        syslog output) works with none of the extras installed. Each
+        guarded module becomes ``None``; the eagerly imported outputs,
+        which need only httpx or the standard library, stay real.
+        """
+        with _cli_without_optional_sdks() as blocked_cli:
+            self.assertIs(blocked_cli, sys.modules["parsedmarc.cli"])
+            for name in GUARDED_CLI_MODULES:
+                with self.subTest(module=name):
+                    self.assertIsNone(getattr(blocked_cli, name))
+            for name in ("postgres", "splunk", "syslog", "webhook"):
+                with self.subTest(module=name):
+                    self.assertIsInstance(getattr(blocked_cli, name), ModuleType)
+
+        # The other half: once the SDKs are back, so are the modules --
+        # a guard that left them None would break every later test.
+        for name in GUARDED_CLI_MODULES:
+            with self.subTest(module=name):
+                self.assertIsInstance(getattr(parsedmarc.cli, name), ModuleType)
+
+    def test_graph_error_types_fall_back_to_sentinel_classes(self):
+        """The Graph error handling still works with the msgraph extra absent.
+
+        ``ClientAuthenticationError`` and ``APIError`` appear only in
+        ``except`` tuples and one ``isinstance`` check. Without the extra
+        a Graph connection cannot even be constructed -- its
+        ``parsedmarc.mail`` placeholder raises the extra's ImportError --
+        so those handlers are unreachable, and the placeholders exist
+        purely to keep the module importable. They must be exception
+        classes distinct from the real ones, must never match an
+        unrelated exception, and must leave ``_log_msgraph_failure``'s
+        non-APIError branch intact.
+        """
+        with _cli_without_optional_sdks() as blocked_cli:
+            for name in ("ClientAuthenticationError", "APIError"):
+                with self.subTest(name=name):
+                    sentinel = getattr(blocked_cli, name)
+                    self.assertTrue(issubclass(sentinel, Exception))
+                    self.assertIsNot(sentinel, globals()[name])
+
+            # A real error raised where Graph errors are handled is not
+            # swallowed by the sentinels.
+            with self.assertRaises(ValueError):
+                try:
+                    raise ValueError("boom")
+                except (
+                    blocked_cli.ClientAuthenticationError,
+                    blocked_cli.APIError,
+                    httpx.HTTPError,
+                ):
+                    self.fail("a sentinel caught an unrelated exception")
+
+            with self.assertLogs("parsedmarc.log", level="ERROR") as cm:
+                blocked_cli._log_msgraph_failure(
+                    ValueError("no  token"),
+                    stage="connection",
+                    mailbox="reports@example.com",
+                    tenant_id="tenant",
+                    auth_method="ClientSecret",
+                )
+            self.assertIn("ValueError: no token", cm.output[0])
+
+        self.assertIs(
+            parsedmarc.cli.ClientAuthenticationError, ClientAuthenticationError
+        )
+        self.assertIs(parsedmarc.cli.APIError, APIError)
+
+    def test_missing_extra_raises_configuration_error_with_install_hint(self):
+        """A configured output whose module is None names its extra.
+
+        ``None`` is exactly what the import guard leaves behind when the
+        extra is not installed, so patching the module global reproduces
+        that state without reloading. The message must name the INI
+        section and the pip command, and it must be a
+        ``ConfigurationError``: _main() reports those to the user
+        directly, while the ``RuntimeError`` wrappers around each
+        constructor would bury the hint in a traceback.
+        """
+        for module, section, extra, options, _key in self.GUARDED_OUTPUTS:
+            with self.subTest(extra=extra):
+                opts = _output_client_opts(save_aggregate=True, **options)
+                with patch.object(parsedmarc.cli, module, None):
+                    with self.assertRaises(
+                        parsedmarc.cli.ConfigurationError
+                    ) as context:
+                        parsedmarc.cli._init_output_clients(opts)
+                self.assertEqual(
+                    str(context.exception),
+                    f"The [{section}] configuration section requires the {extra} "
+                    f"extra: pip install parsedmarc[{extra}]",
+                )
+
+    def test_present_extra_initializes_without_the_hint(self):
+        """The same config is accepted when the module is importable.
+
+        The negative half of the test above: the check must key on the
+        module being absent, not on the section being configured. The
+        module is replaced by a mock so no client opens a real
+        connection, and the client the section asks for is the only one
+        built.
+        """
+        for module, _section, extra, options, key in self.GUARDED_OUTPUTS:
+            with self.subTest(extra=extra):
+                opts = _output_client_opts(save_aggregate=True, **options)
+                with patch.object(parsedmarc.cli, module, MagicMock()):
+                    clients = parsedmarc.cli._init_output_clients(opts)
+                self.assertEqual(sorted(clients), [key] if key else [])
+
+    def test_log_analytics_extra_is_checked_before_reports_are_parsed(self):
+        """The Log Analytics client is built per batch, checked at startup.
+
+        ``process_reports()`` constructs ``LogAnalyticsClient`` for every
+        batch under ``opts.la_dce``, long after reports have been fetched
+        and parsed. The presence check therefore lives in
+        ``_init_output_clients()``, which runs before any mailbox is
+        opened and again on a SIGHUP reload, so a missing extra is
+        reported immediately instead of once there is data to lose. With
+        no ``la_dce`` configured, a missing module is not an error at all.
+        """
+        with patch.object(parsedmarc.cli, "loganalytics", None):
+            self.assertEqual(
+                parsedmarc.cli._init_output_clients(_output_client_opts()), {}
+            )
+
+    def test_missing_postgresql_extra_raises_configuration_error(self):
+        """A configured [postgresql] section without psycopg fails fast.
+
+        postgres.py guards its own psycopg import, so the module is
+        always importable and ``postgres.psycopg is None`` is exactly the
+        missing-extra state. Without the presence check the constructor's
+        PostgreSQLError became a RuntimeError that the startup retry loop
+        retried for over a minute before exiting; a ConfigurationError
+        exits immediately with the install hint. The negative half:
+        with psycopg present, the same opts build the client.
+        """
+        opts = _output_client_opts(postgresql_host="db.example.com")
+        with patch.object(parsedmarc.cli.postgres, "psycopg", None):
+            with self.assertRaises(parsedmarc.cli.ConfigurationError) as context:
+                parsedmarc.cli._init_output_clients(opts)
+        self.assertEqual(
+            str(context.exception),
+            "The [postgresql] configuration section requires the postgresql "
+            "extra: pip install parsedmarc[postgresql]",
+        )
+        with (
+            patch.object(parsedmarc.cli.postgres, "psycopg", MagicMock()),
+            patch.object(parsedmarc.cli.postgres, "PostgreSQLClient") as mock_client,
+        ):
+            clients = parsedmarc.cli._init_output_clients(opts)
+        self.assertEqual(sorted(clients), ["postgresql_client"])
+        self.assertIs(clients["postgresql_client"], mock_client.return_value)
+
+    def test_missing_mailbox_extras_fail_fast_at_config_time(self):
+        """[msgraph]/[gmail_api] sections without their extra fail in
+        _parse_config with parsedmarc's own install hint.
+
+        Without the extra, parsedmarc.mail binds placeholder connection
+        classes outside the MailboxConnection hierarchy whose
+        construction raises mailsuite's ImportError — which named
+        mailsuite's extra, not parsedmarc's, and surfaced as a logged
+        traceback long after config parsing. The subclass check turns
+        that into an immediate ConfigurationError naming the parsedmarc
+        extra. The check reads parsedmarc.mail's own attributes — the
+        authoritative placeholder state — not cli's rebound names, which
+        other tests replace with MagicMock instances that issubclass()
+        would reject. The negative half — real classes pass the check —
+        is covered by every existing msgraph/gmail_api _parse_config
+        test, which runs with the real mailsuite classes imported.
+        """
+        from parsedmarc.cli import _parse_config
+
+        class _Placeholder:
+            """Stand-in for the parsedmarc.mail missing-extra placeholder."""
+
+        cases = (
+            ("MSGraphConnection", "msgraph", "msgraph"),
+            ("GmailConnection", "gmail_api", "gmail"),
+        )
+        for class_name, section, extra in cases:
+            with self.subTest(section=section):
+                cp = _config_with(section, {})
+                with patch.object(parsedmarc.mail, class_name, _Placeholder):
+                    with self.assertRaises(
+                        parsedmarc.cli.ConfigurationError
+                    ) as context:
+                        _parse_config(cp, _opts())
+                self.assertEqual(
+                    str(context.exception),
+                    f"The [{section}] configuration section requires the {extra} "
+                    f"extra: pip install parsedmarc[{extra}]",
+                )
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)

tests/test_elastic.py

@@ -1213,9 +1213,10 @@ class TestSaveFailureReport(unittest.TestCase):
 
     def test_sample_address_lists_indexed_for_reply_to_cc_bcc_attachments(self):
         """A failure report sample can carry reply_to / cc / bcc /
-        attachments. Each populates a nested InnerDoc on the sample —
-        if the add_* helpers regress, those nested docs would be
-        silently empty in Elasticsearch."""
+        attachments. Each populates a nested InnerDoc on the sample;
+        this drives all four add_* helper paths (nested-doc contents
+        are asserted separately in
+        test_reply_to_header_flattened_and_indexed)."""
         report = _failure_report()
         report["parsed_sample"]["reply_to"] = [
             {"display_name": "RT", "address": "rt@example.com"}

tests/test_gelf.py

@@ -93,8 +93,8 @@ def _sample_aggregate_report() -> AggregateReport:
 
 
 class _Handler(logging.Handler):
-    """Capture the (record, extra) of every log emit, so tests can
-    assert on what GelfClient actually pushed."""
+    """Capture the (message, parsedmarc payload) of every log emit, so
+    tests can assert on what GelfClient actually pushed."""
 
     def __init__(self):
         super().__init__()
@@ -212,8 +212,8 @@ class TestGelfClientSaveAggregate(unittest.TestCase):
 
 class TestGelfClientSaveFailure(unittest.TestCase):
     """save_failure_report_to_gelf operates on already-parsed failure
-    reports. Build one through the CSV-row helper to verify GelfClient
-    surfaces the right fields."""
+    reports, flattening each through the CSV-row helper; verify
+    GelfClient surfaces the right fields."""
 
     def _sample_failure_report(self) -> FailureReport:
         report = {

tests/test_init.py

@@ -303,7 +303,7 @@ class Test(unittest.TestCase):
         self.assertEqual(record["policy_evaluated"]["disposition"], "none")
 
     def testEmptySample(self):
-        """Test empty/unparasable report"""
+        """Test empty/unparseable report"""
         with self.assertRaises(parsedmarc.ParserError):
             parsedmarc.parse_report_file("samples/empty.xml", offline=OFFLINE_MODE)
 

tests/test_kafkaclient.py

@@ -187,8 +187,10 @@ class TestSaveFailureReportsToKafka(unittest.TestCase):
             return KafkaClient(kafka_hosts=["b:9092"])
 
     def test_sends_full_list_in_one_message(self):
-        """Failure reports go in a single Kafka message — the comment
-        in source code documents the 1MB-per-message default."""
+        """Failure reports are sent as one Kafka message carrying the
+        whole list — unlike aggregate records, which are sent as
+        individual slices to stay under Kafka's default 1MB message
+        cap."""
         client = self._client()
         reports = [{"id": "f1"}, {"id": "f2"}]
         client.save_failure_reports_to_kafka(reports, "dmarc-failure")

tests/test_loganalytics.py

@@ -98,9 +98,9 @@ class TestPublishJson(unittest.TestCase):
 class TestPublishResults(unittest.TestCase):
     """publish_results gates each report type behind both a config flag
     (save_aggregate / save_failure / save_smtp_tls) and a configured
-    stream name. Both gates need to work — a missing stream alone is a
-    config bug that should be silent, but an explicit save_*=False
-    means the operator opted out."""
+    stream name. Both gates need to work — a missing stream is skipped
+    silently (a partially configured client is normal), while an
+    explicit save_*=False means the operator opted out."""
 
     def _publish_with(self, results, **flags):
         flags.setdefault("save_aggregate", True)

tests/test_opensearch.py

@@ -797,7 +797,7 @@ class TestSaveAggregateReport(unittest.TestCase):
     def test_already_saved_raises_when_search_returns_hit(self):
         """The dedup query is the only thing preventing
         double-indexing on re-run. A regression would silently
-        re-save reports, inflating Kibana counts."""
+        re-save reports, inflating dashboard counts."""
         with (
             patch("parsedmarc.opensearch.Search", return_value=_populated_search()),
             patch("parsedmarc.opensearch.Index"),
@@ -859,8 +859,8 @@ class TestSaveAggregateReport(unittest.TestCase):
         self.assertIn("dmarc_aggregate-2024-01", index_calls)
 
     def test_index_name_honours_suffix_and_prefix(self):
-        """Prefix/suffix support multi-tenant setups where one ES
-        cluster serves several DMARC owners."""
+        """Prefix/suffix support multi-tenant setups where one
+        OpenSearch cluster serves several DMARC owners."""
         with (
             patch("parsedmarc.opensearch.Search", return_value=_empty_search()),
             patch("parsedmarc.opensearch.Index") as mock_index_cls,
@@ -1001,9 +1001,9 @@ class TestAggregateDocCombinedResults(unittest.TestCase):
     def test_add_dkim_result_appends_combined_string(self):
         """Regression guard for issue #169: dkim_results/spf_results are
         arrays of objects that the engine dynamic-maps as plain ``object``
-        (not ``nested``) and flattens, so Kibana/Grafana tables cannot
-        terms-aggregate their subfields without producing a cross-product
-        of selector/domain/result values. The composed
+        (not ``nested``) and flattens, so OpenSearch Dashboards/Grafana
+        tables cannot terms-aggregate their subfields without producing a
+        cross-product of selector/domain/result values. The composed
         "selector / domain / result" string preserves the per-signature
         pairing that the flattened array loses."""
         doc = opensearch_module._AggregateReportDoc()
@@ -1206,9 +1206,10 @@ class TestSaveFailureReport(unittest.TestCase):
 
     def test_sample_address_lists_indexed_for_reply_to_cc_bcc_attachments(self):
         """A failure report sample can carry reply_to / cc / bcc /
-        attachments. Each populates a nested InnerDoc on the sample —
-        if the add_* helpers regress, those nested docs would be
-        silently empty in OpenSearch."""
+        attachments. Each populates a nested InnerDoc on the sample;
+        this drives all four add_* helper paths (nested-doc contents
+        are asserted separately in
+        test_reply_to_header_flattened_and_indexed)."""
         report = _failure_report()
         report["parsed_sample"]["reply_to"] = [
             {"display_name": "RT", "address": "rt@example.com"}

tests/test_parallel.py

@@ -241,8 +241,9 @@ class TestWorkerLogging(_ParallelTestCase):
 
 class TestInitWorkerLogging(unittest.TestCase):
     """_init_worker_logging must be usable directly as a pool initializer:
-    with no log files it just sets the level (adds a console handler),
-    and with log files it attaches a FileHandler per path."""
+    with no log files it sets the level (configure_logging still ensures
+    a console handler), and with log files it attaches a FileHandler per
+    path."""
 
     def setUp(self):
         from parsedmarc.log import logger as plog

tests/test_postgres.py

@@ -28,8 +28,9 @@ from parsedmarc.postgres import (
 
 OFFLINE_MODE = os.environ.get("GITHUB_ACTIONS", "false").lower() == "true"
 
-# psycopg is an optional dependency and is not installed in CI (which installs
-# only the [build] extra). The save methods mock the connection, but the
+# psycopg is an optional dependency and is not installed in CI (whose
+# unit-test job installs the [build,all] extras, deliberately without
+# postgresql). The save methods mock the connection, but the
 # failure path also references ``psycopg_json.Jsonb`` at module scope, so
 # mock that SDK boundary for the whole module when psycopg is absent.
 _types_patcher = None

tests/test_utils.py

@@ -78,9 +78,9 @@ class Test(unittest.TestCase):
         """When neither PTR nor an ASN-map entry resolves, the raw AS name
         is used as source_name with type left null — better than leaving
         the row unattributed."""
-        # 204.79.197.100 is in an ASN whose as_domain is not in the map at
-        # the time of this test (msn.com); this exercises the as_name
-        # fallback branch without depending on a specific map state.
+        # A mocked DB record whose as_domain is not in the reverse-DNS map
+        # exercises the as_name fallback branch without depending on a
+        # specific map state.
         from unittest.mock import patch
 
         with patch(

tests/test_webhook.py

@@ -70,8 +70,8 @@ class TestWebhookClientSaveMethods(unittest.TestCase):
 
 
 class TestWebhookClientDictPayload(unittest.TestCase):
-    """``_send_to_webhook`` accepts ``bytes | str | dict``. httpx only
-    form-encodes a dict via ``data=``; string/bytes payloads must use
+    """``_send_to_webhook`` accepts ``bytes | str | dict``. A dict is
+    form-encoded via httpx's ``data=``; string/bytes payloads must use
     ``content=`` since httpx's ``data=`` is form-encoding only."""
 
     def test_dict_payload_uses_data_kwarg(self):