The module-level try/except import is environment-dependent: with
psycopg installed the ImportError fallback never runs, and without it
(CI's test job) the successful-import arm never completes — so Codecov
flags one side or the other no matter where coverage is measured (it
flagged the import line on master right after #798 merged).
Exercise both arms explicitly: execute the module's source into a
fresh, throwaway module object (importlib.util.module_from_spec +
exec_module) under a patched sys.modules — a None entry forces
ImportError, fake module entries force the success path — and assert
on the psycopg / psycopg_json bindings each arm produces. The
throwaway-module approach (rather than importlib.reload) leaves the
canonical parsedmarc.postgres untouched, so the identity of
PostgreSQLError / AlreadySaved held by the rest of the test module is
preserved.
Verified covered in both environments: with the venv's real psycopg,
and with psycopg hidden via a PYTHONPATH shim to simulate CI; the
import block reports no missing lines either way.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Make the whole codebase pass pyright cleanly and enforce it in CI
Fix all 102 pyright (1.1.410, standard mode) errors across the library,
tests, and maps scripts, then pin and enforce the zero-errors bar:
- postgres.py: make the optional psycopg import TYPE_CHECKING-aware so
the module is properly typed while keeping the runtime install-hint
fallback; import psycopg.types.json explicitly as psycopg_json (the
old psycopg_types.json attribute access only worked because psycopg
imports the submodule eagerly); have _connect()/_ensure_connected()
return the live connection so save methods use a non-Optional local;
type the DDL list as list[LiteralString] to match psycopg's execute()
overloads.
- kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error
fallback statically via TYPE_CHECKING (kafka-python 3.0 removed
NoBrokersAvailable), which also fixes _BootstrapError's import
resolution in tests.
- syslog.py: go through getattr/setattr for SysLogHandler.socket
(absent from typeshed); type the save_* methods with the report
TypedDicts (single or list, matching cli.py call sites — gelf.py gets
the same signatures); raise ValueError when retry_attempts < 1
instead of falling through and registering a None handler (bug fix,
with a regression test and a CHANGELOG entry).
- elastic.py / opensearch.py: human_result params are Optional[str].
- maps scripts: sort_csv declared a return type but never returned
(now -> None); seen_sort_field_values was possibly unbound;
convert_to_utf8's src_encoding is Optional[str].
- tests: cast sample-report dict helpers to their TypedDicts; mark
deliberate wrong-type calls with targeted pyright ignores; add
narrowing asserts for Optional results; access the mocked
KafkaProducer through a cast helper; match the mailsuite
fetch_message base signature (**kwargs); patch the renamed
parsedmarc.postgres.psycopg_json in test_postgres's setUpModule.
Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc,
tests, docs; standard mode), pyright==1.1.410 pinned in the [build]
extra (pinned exactly so a new pyright release can't break CI without a
code change), and a "Check types" step in the lint CI job — which now
also runs ruff format --check and installs the [postgresql] extra so
the optional psycopg import resolves. Documented in AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Set session headers via update() instead of replacing the dict
requests 2.34 ships inline type annotations, and Session.headers is a
CaseInsensitiveDict[str] — assigning a plain dict fails pyright there
(the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid
it). headers.update() is correctly typed against both versions, and is
the documented requests idiom: it overrides User-Agent and the
client-specific headers while keeping the session's defaults
(Accept-Encoding, Connection) instead of wiping them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
PR #784 was stacked on the #783 branch and its base was never retargeted to
master, so it merged into fix/mailsuite-2.2.1-empty-address instead of master.
master therefore has 10.0.2 (#783's squash) but is missing the 10.0.3 changes.
This re-lands exactly that delta — the Reply-To/Delivered-To parser fix, the
ES/OS Reply-To header flattening, and the Splunk/OpenSearch/Grafana failure
dashboard fixes, with the version bumped to 10.0.3. No mailsuite re-bump (the
>=2.2.1 floor is already on master from 10.0.2).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a PostgreSQL output backend as a lighter-weight alternative to
Elasticsearch/OpenSearch, configured via a [postgresql] section
(host/port/user/password/database or a libpq connection_string). Tables
are created automatically on first run; a Grafana dashboard is included.
- psycopg is an optional extra (pip install parsedmarc[postgresql]); the
import is guarded so `import parsedmarc` works without it, and
PostgreSQLClient raises a clear install hint when constructed without
the driver. Binary wheels aren't available for every platform.
- Schema captures the RFC 9990 / DMARCbis aggregate fields: np, testing,
discovery_method, generator, xml_namespace, and per-result human_result
on the DKIM/SPF auth-result tables.
- forensic -> failure naming throughout (table dmarc_failure_report,
save_failure_report_to_postgresql, dashboard, docs) to match #659.
- Failure-report de-duplication mirrors the Elasticsearch backend exactly:
arrival date + From + To + Subject (NULL-safe via IS NOT DISTINCT FROM;
semantic JSONB equality). Aggregate and SMTP-TLS use ON CONFLICT.
- PostgreSQLClient.close() for clean CLI shutdown; comment documents why
the two timestamp helpers must stay distinct (report dates are local,
record/SMTP-TLS dates are UTC).
- CLI: config parse raises ConfigurationError on missing
host/connection_string; wired into _init_output_clients + save loops.
- Tests in tests/test_postgres.py (helpers, mocked-DB save assertions,
create_tables, connect/error wrapping, dedup, real-sample round trip)
and tests/test_cli.py (config parse + end-to-end save wiring incl.
AlreadySaved/PostgreSQLError handling). postgres.py at 99% line
coverage; only _main's output-client-init retry path is left.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sean Whalen
Fabio Scaccabarozzi