Files
parsedmarc/parsedmarc/postgres.py
T
02fcfd6e82 Merge commit from fork
* Evict expired dedup-cache entries on insert and hash the key (GHSA-9fw6-mwqr-p43m)

The seen-aggregate-report cache is an expiringdict.ExpiringDict, which
removes an expired entry only when that same key is read again, or drops
its oldest entry once it holds max_len (100 million) entries. A report
that is never received again is never read again, so in watch mode every
distinct aggregate report stayed in memory for the life of the process.
The key was the JSON of the sender-controlled reporter, Report-ID and
policy domain, so ten 5.6 KB attachments with a 5 MiB report_id held
64 MiB indefinitely.

SeenReportCache (parsedmarc/config.py) subclasses ExpiringDict and removes
expired entries from the head of its insertion order on every insert;
the cache's max_len is now 100,000. _aggregate_report_dedup_key() now
returns the SHA-256 hex digest of that JSON, so a key is 64 characters
whatever the report contains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep a re-added dedup-cache key at the tail so the expiry sweep is not blocked

OrderedDict refreshes a re-set key's time in place, so a key added again while still stored kept its old position at the head and stopped the insert-time sweep before the expired entries behind it. SeenReportCache now moves such a key to the tail before storing it, keeping insertion order equal to age order.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Bound failed reverse-DNS lookups to 120 seconds per aggregate report (GHSA-rwfh-ww9c-jm9m)

Every record of an aggregate report made one blocking PTR lookup of a sender-chosen source IP, with no bound on count or time; an unanswering reverse zone costs the full 8 s DNS lifetime with the default nameservers, and a weak-fallback result is never cached, so one IP repeated in every record was looked up every time. A 3,500-record report could block processing for hours.

Lookups that return no PTR now share a 120-second budget per report; once it is spent, the remaining records are enriched from the local IP database only, their results are not cached, and one warning is logged. Each distinct source IP is looked up at most once per report, and lookups that find a PTR are not charged, so large legitimate reports keep their enrichment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep values PostgreSQL rejects out of its output, and let one rejected report not hold the batch (GHSA-fxr7-7h9q-v949)

An aggregate count above 2**31 - 1, an SMTP TLS sending-mta-ip or
receiving-ip that is not an IP address, and a NUL in a failure report's
sample each made the PostgreSQL save raise PostgreSQLError, which marked
the whole mailbox batch unsaved, so legitimate reports from other senders
were never archived. The parser now skips a record whose count is outside
0..MAX_MESSAGE_COUNT and rejects such an SMTP TLS session count (closing
the same rejection in Elasticsearch/OpenSearch), drops an invalid TLS IP
with a warning, and the PostgreSQL output replaces NUL with U+FFFD in
text, text array and JSONB values. Any other psycopg.DataError raises the
new PostgreSQLRejectedReport, which the CLI logs and skips for that one
report unless fail_on_output_error is set.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Check an aggregate record's count before its reverse-DNS lookup

A record whose count is outside 0..MAX_MESSAGE_COUNT is dropped, so its
source IP is no longer looked up first, which could cost up to 8 seconds
per record. A regression test mocks dnspython and asserts that the dropped
record's address is never queried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Require TLS for the PostgreSQL output by default and add [postgresql] sslmode and sslrootcert (GHSA-xvp7-7762-gx9g)

PostgreSQLClient connected without an sslmode, so libpq used its default,
prefer: when the server, or an on-path attacker, declined TLS, it continued
over plain TCP and answered a clear-text password request. The client now
passes sslmode=require unless the new sslmode argument, an sslmode in the
connection string, or PGSSLMODE chooses a mode, and sslrootcert names the
CA certificate for verify-ca and verify-full. A warning is logged at
startup when a TCP host other than localhost or a loopback address is used
with a mode that does not verify the server certificate. A malformed
connection string is now rejected by a parse before connecting, with the
same withheld-details error. A server without TLS fails with libpq's
"server does not support SSL, but SSL was required" until sslmode is set
to disable or prefer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Withhold the hosts URL from Elasticsearch and OpenSearch client-build errors (GHSA-74xv-rmpx-2pjm)

A password with an unencoded "/", "?" or "#" in a [elasticsearch] or
[opensearch] hosts URL makes elastic_transport raise "Could not parse URL
'<url>'" and opensearch-py (via urllib.parse) raise "Port could not be cast
to integer value as '<password prefix>'". The startup wrapper repeated that
text at ERROR without --debug and chained the original, which the SIGHUP
reload's logger.exception printed again.

set_hosts() now runs through _set_search_hosts(), which replaces any
ValueError with a fixed message raised outside the except block. Other
errors in the block keep their text with URL credentials masked and are
raised from None. This completes the GHSA-hjr3-rx4c-24cv fix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Cache the downloaded IP database in a private per-user directory and validate it (GHSA-5f56-8ph8-v9q5)

load_ip_db() cached the IPinfo Lite download at the fixed, shared
<tmp>/parsedmarc/ipinfo_lite.mmdb, where another local user could plant
a symlink to have the download overwrite a file the parsedmarc user can
write, or plant a database that was used whenever the download failed.

The cache now lives in $XDG_CACHE_HOME/parsedmarc or ~/.cache/parsedmarc
(%LOCALAPPDATA%\parsedmarc\cache on Windows), created 0700 and used only
if it is a real directory the user owns and can write that is not
group- or world-writable; otherwise a per-run mkdtemp() directory is
used. The download is written with mkstemp(), opened with maxminddb to
validate it, and moved into place with os.replace(). A cached copy is
used only if it is a regular file owned by the user.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Skip reports over the Logs Ingestion API size limit and publish the Log Analytics streams independently (GHSA-rg4h-58fm-8pc9)

The Azure Monitor ingestion SDK packs whole reports into chunks of up to
1 MiB and never splits one, so a report whose JSON is larger is sent as a
call of its own, and the Logs Ingestion API rejects any call over 1 MB
(https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits).
The rejected upload raised before the failure and SMTP TLS streams were
sent, and the CLI held the whole batch for retry and finally Unsaved.

publish_json() now measures each report as the SDK does and leaves out
one over 1,000,000 bytes with a warning naming it, and publish_results()
publishes each stream in its own try and raises one LogAnalyticsException
naming every stream that failed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Address review: redact the PostgreSQL warning host, drop the connect error chain, and fix Log Analytics and docs wording

libpq ends a URI's userinfo at the first "@", so the rest of a password
containing "@" becomes part of the host. The PostgreSQL startup warning
now credential-masks every host-like value it logs (host list entries,
hostaddr, PGHOST, PGHOSTADDR), checks hostaddr before host as libpq
does, and the general connect error is raised from None so the driver's
unredacted text no longer reaches a logged traceback. The CLI's
output-client Raises docs now say which outputs do not chain.

publish_json() returns the number of reports uploaded, and
publish_results() logs "Successfully pushed" only for a stream that
uploaded something.

The [postgresql] URI example uses the Unix socket and explains that TCP
to localhost needs TLS on the server or sslmode = disable, the
[log_analytics] docs note the SDK's 1 MiB call packing, the
reverse-DNS-budget wording says "local IP database and reverse DNS map"
everywhere, and the GHSA-fxr7 entry no longer claims the parser rejects
every unreadable start-datetime.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Require mailsuite 2.3.7

mailsuite 2.3.7 fixes parse_email() keeping a CRLF message's carriage
returns in address header values (seanthegeek/mailsuite#74). The floor
is raised in the base dependency and the msgraph, gmail and all extras,
and the usage docs' stated requirement is updated to match.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Drop Python 3.10 and add Python 3.15 to the supported versions and CI (#953)

Python 3.10 reaches end of life in October 2026. requires-python is now
>=3.11, the unused requires_python key under [build-system] (not read by
hatchling, and its <3.15 bound read as excluding 3.15) is removed, and
Python 3.15 joins the CI matrix, classifiers, and support tables. Docs,
AGENTS.md, and comments that named 3.10 as the floor are updated.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

* Keep aggregate identifiers and reasons within their schema (#950)

* Keep aggregate identifiers and reasons within their schema

AI assistance: OpenAI Codex.

* Expand the #948 CHANGELOG entry and tighten its tests

RFC 9990 Appendix A and RFC 7489 Appendix C define only header_from,
envelope_from and envelope_to under identifiers, and only type and
comment in a reason. Cite them in the test, check that envelope_from
and envelope_to survive next to a dropped nested unknown element, and
cover an unknown key on the legacy identities path. The CHANGELOG
entry now names the affected outputs and the compatibility change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Reject reports dated in the future and bound what parse_report_file() reads (#954)

* Reject reports whose index-naming date is more than two days in the future (#947)

The date that names a report's dated Elasticsearch/OpenSearch index (an
aggregate report's begin date, a failure report's Arrival-Date, an SMTP
TLS report's start-datetime) was never compared with the present, so a
report dated 2100 named an index for 2100. Each parser now rejects such a
report with its own Invalid*Report; past dates stay unbounded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Refuse an input larger than the report size cap before parse_report_file() reads it (#949)

The 32 MiB cap was applied only after the whole input was read. A path is
now checked with os.path.getsize(), bytes-like input with len(), and a
file object is read at most one byte past the cap. The error is the
ParserError with the message the content check already gives. Three
existing tests that fed over-cap content now see the earlier rejection.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Measure a memoryview input by its byte size

len() of a memoryview counts items, which are not bytes for every format,
so the size cap in parse_report_file() compares nbytes instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Bound the aggregate end date and read file objects to their end

Address the Copilot review of #954. A normalized report's per-day records
name their indexes from their own interval starts, so the report's end date
is bounded like its begin date. A raw stream may return fewer bytes than
asked for before its end, so parse_report_file() now reads in a loop until
end of file or one byte past the cap.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Accumulate bounded reads in one buffer

Address the second Copilot review of #954. Keeping every chunk in a list
cost one bytes object per read, about 85 bytes per byte for a stream that
returns a byte at a time (22.7 MiB for 256 KiB of input). One bytearray
keeps memory proportional to the bytes read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

* Skip a message with an invalid report attachment instead of aborting the mbox (#955)

* Skip a message with an invalid report attachment instead of aborting the mbox

parse_report_email() raised a bare ParserError when a report attachment was
invalid, which get_dmarc_reports_from_mbox() re-raises, so one such message
ended the whole file. Raise InvalidDMARCReport (a ParserError subclass), which
the mbox loop skips with a warning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Skip an invalid SMTP TLS attachment in an mbox too

Address the Copilot review of #955. An invalid SMTP TLS attachment raised
InvalidSMTPTLSReport, which is not an InvalidDMARCReport, so it still
stopped the mbox. parse_report_email() now names the message's subject in
that error, and both mbox loops skip it with a warning. A message that
cannot be parsed at all still raises a plain ParserError.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

* Address the second review: refuse an unencoded @ in a PostgreSQL URI, charge slow reverse-DNS lookups, and smaller fixes

- PostgreSQL: a connection string whose parsed host, hostaddr, port or
  dbname holds an "@" (the rest of a URI password with an unencoded "@",
  which libpq splits at the first "@") is refused with a fixed error
  before any warning is logged or any connection is attempted; _connect
  parses the string on every path. A leading "@" in a host entry (Linux
  abstract socket) is still allowed. _log_safe_host stays as defence in
  depth for the host argument and PGHOST/PGHOSTADDR.
- Log Analytics: publish_json catches AzureError, so a ServiceRequestError
  or ServiceResponseError no longer stops the remaining streams.
- MAX_MESSAGE_COUNT comment and CHANGELOG: the bound matches PostgreSQL's
  INTEGER columns and the Elasticsearch/OpenSearch integer mapping of the
  SMTP TLS session counts; the aggregate message_count is dynamically
  mapped as long.
- Reverse DNS budget: a lookup taking at least SLOW_REVERSE_DNS_SECONDS
  (1.0) is charged whether or not it returned a name; the warning now says
  "failed or were slow".
- An IPv6 zone ID in an aggregate record's source IP skips the record
  before any DNS lookup, and in a failure report's Source-IP raises
  InvalidFailureReport; one helper serves these and the SMTP TLS path.
- IP database cache: a relative candidate directory (HOME unset, no
  passwd entry) is rejected; offline / always_use_local_file creates no
  directory and no temporary directory and logs no "downloaded again".
- No startup warning for sslmode=require with an existing sslrootcert
  file, which libpq verifies as verify-ca.
- PostgreSQLRejectedReport truncates the driver text to 200 characters.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Address the third review: refuse a URI with more than one @, bound total reverse-DNS time, strip IPv6 zone IDs

- postgres: a URI connection string (postgresql:// or postgres://) with
  more than one literal "@" is refused on the raw string, before any log
  line or connect, closing the abstract-socket host-list and query
  parameter bypasses of the parsed-value check. libpq decodes %40 in a
  password, host, database name and query value, so a URI never needs a
  second "@". The parsed-value check remains for keyword/value strings.
- Reverse DNS: every online lookup is charged to a new 600 s per-report
  total budget (MAX_REVERSE_DNS_SECONDS_PER_REPORT); failed or slow ones
  still also use the 120 s budget. The warning names the limit reached.
- An IPv6 zone ID in an aggregate source_ip or a failure report's
  Source-IP is now stripped with a warning instead of rejecting the
  record or report; SMTP TLS still drops its optional IP fields.
- _rejection_detail truncates the driver text before escaping it.
- Docs and CHANGELOG entries updated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Withhold everything derived from a PostgreSQL connection string from logs and errors

Fourth review of GHSA-xvp7-7762-gx9g: libpq ends a URI's userinfo at the
first "@" or "/", so postgresql://u:SECRETX/x@db.invalid/d parses to host
"u", port "SECRETX" and dbname "x@db.invalid/d". It has one "@", so the
raw-count refusal let it through, and connecting raised 'invalid integer
value "SECRETX" for connection option "port"', a password fragment in the
error and its traceback. A variant put the fragment in sslmode, which the
startup warning printed. Three rounds of shape checks each missed a shape.

A. With a connection_string, nothing parsed from it is logged or raised:
   - A connection failure raises a fixed "PostgreSQL connection failed
     (details withheld ...)" from None, with " [SQLSTATE xxxxx]" appended
     only when the driver error carries a well-formed SQLSTATE
     (^[0-9A-Z]{5}$). A string libpq cannot parse keeps its own fixed
     message, and any error from the pre-parse is replaced by it.
   - The startup warning names no host and no value from the string; the
     mode is printed only when it is one of SSL_MODES.
   With the separate host/port/user/password/database arguments, the
   driver's message (credential-masked) and the host in the warning are
   kept.

B. The early refusal stays as a convenience that names the common
   mistake, mirroring libpq's scan: a URI is refused when the text after
   "://" holds more than one "@", or an "@" with a "/" before it, and the
   parsed-value check (_has_misplaced_at) runs on URIs again as well as
   keyword/value strings. Docstrings, usage.md and the CHANGELOG now say
   the refusal is a convenience and A is what keeps the password out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* Wrap every PostgreSQL connect failure and narrow the connection-string refusal

- postgres._connect(): everything from parsing the settings to holding a
  configured connection (pre-parse, ssl kwargs, connect, autocommit,
  dumper registration) is protected. A non-driver error (psycopg's
  UnicodeEncodeError carrying the whole built connection string for a
  password with a lone surrogate; on Python 3.15 the unwrapped
  UnicodeEncodeError('idna', '<host>') for an over-long host label) is
  replaced with a fixed message from None: the withheld connection
  failure message with a connection string, a new fixed "a connection
  setting could not be passed to the driver" message with the separate
  settings. KeyboardInterrupt/SystemExit still propagate.
- cli._init_output_clients(): the PostgreSQL wrapper no longer chains
  (from None), and a non-PostgreSQLError gets a fixed message.
- _has_unencoded_at() now mirrors libpq's URI scan (userinfo up to the
  first "@" or "/") instead of counting "@", so a percent-encoded
  database name (d%40x) and an "@" in a query value after a "/" (a
  certificate path) are accepted, while every password-spill shape in
  the table is still refused except two that read like legitimate
  strings.
- _has_misplaced_at() is removed: a keyword/value password is its own
  keyword, so an "@" in a kv host, hostaddr or dbname is not a password
  fragment, and libpq does not percent-decode kv values, so the encoding
  advice was wrong. The refusal message now names the database name.
- Docstrings, usage.md and the GHSA-xvp7 CHANGELOG entry updated; tests
  for real psycopg (surrogate password, long host label), fake-psycopg
  non-driver errors at connect/autocommit/dumpers for both modes, a
  KeyboardInterrupt check, the rule table (also checked against the real
  parser), legitimate strings that must connect, and the CLI wrapper.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: jwalith4418-Mj <jwalith4418@gmail.com>
2026-11-10 18:44:59 -05:00

1381 lines
61 KiB
Python

# -*- coding: utf-8 -*-
from __future__ import annotations
import ipaddress
import os
import re
from datetime import datetime
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
# LiteralString requires Python >= 3.11, so only import it for type checking
from typing import LiteralString
import psycopg
from psycopg import conninfo as psycopg_conninfo
from psycopg.abc import Buffer
from psycopg.types import json as psycopg_json
from psycopg.types import string as psycopg_string
else:
try:
import psycopg
from psycopg import conninfo as psycopg_conninfo
from psycopg.types import json as psycopg_json
from psycopg.types import string as psycopg_string
except ImportError:
psycopg = None
psycopg_conninfo = None
psycopg_json = None
psycopg_string = None
from parsedmarc.log import logger
from parsedmarc.utils import (
human_timestamp_to_datetime,
redact_url_credentials,
sanitize_for_log,
)
# psycopg is an optional dependency (the PostgreSQL backend is opt-in). The
# pure helper functions below work without it; only PostgreSQLClient needs a
# live driver, so the import error surfaces at client construction with a
# pip-install hint rather than breaking ``import parsedmarc`` for everyone.
_PSYCOPG_INSTALL_HINT = (
"The PostgreSQL backend requires the 'psycopg' package. "
"Install it with: pip install parsedmarc[postgresql]"
)
# libpq's sslmode values, weakest first. libpq's own default is "prefer",
# which falls back to plain TCP when the server (or an on-path attacker)
# declines TLS, so parsedmarc defaults to "require" (GHSA-xvp7-7762-gx9g).
SSL_MODES = ("disable", "allow", "prefer", "require", "verify-ca", "verify-full")
DEFAULT_SSL_MODE = "require"
_VERIFYING_SSL_MODES = ("verify-ca", "verify-full")
_WITHHELD_CONNECTION_STRING_ERROR = (
"Invalid PostgreSQL connection_string (details withheld "
"because they may include the password)"
)
_UNENCODED_AT_CONNECTION_STRING_ERROR = (
'PostgreSQL connection_string appears to have an unencoded "@" or "/" '
"in its user name, password or database name (details withheld because "
'they may include the password); percent-encode "@" as %40 and "/" as '
"%2F, or use the separate host, port, user, password and database "
"settings"
)
# With a connection string, a connection failure is reported with this fixed
# text and not libpq's: libpq's message can quote any value parsed from the
# string (a port, a host, an sslmode), and when the string's user name or
# password holds an unencoded "@" or "/", such a value is a fragment of the
# password. See PostgreSQLClient._connect().
_WITHHELD_CONNECTION_FAILURE_ERROR = (
"PostgreSQL connection failed (details withheld because the "
"connection string may include the password; use the separate host, "
"port, user, password and database settings to see the driver's "
"message)"
)
# With the separate settings, an error that is not the driver's own (a
# password the driver cannot encode, say) is reported with this fixed text:
# such an error quotes the value it could not handle, or the whole
# connection string psycopg built from the settings, password included.
_WITHHELD_CONNECTION_SETTING_ERROR = (
"PostgreSQL connection failed: a connection setting could not be passed "
"to the driver (details withheld because they may include the password)"
)
# A SQLSTATE is five digits or upper-case letters (PostgreSQL docs, Appendix
# A, "PostgreSQL Error Codes"); anything else is not included.
_SQLSTATE_PATTERN = re.compile(r"[0-9A-Z]{5}")
def _sqlstate_suffix(exc: BaseException) -> str:
"""Return `` [SQLSTATE xxxxx]`` for a driver error that carries a
well-formed SQLSTATE, else an empty string.
psycopg sets ``sqlstate`` when the server answered with an error (for
example ``28P01``, a failed password). It is a fixed code chosen by the
server, not text taken from the connection string, so it is safe to
show even when the driver's message is withheld.
"""
sqlstate = getattr(exc, "sqlstate", None)
if isinstance(sqlstate, str) and _SQLSTATE_PATTERN.fullmatch(sqlstate):
return f" [SQLSTATE {sqlstate}]"
return ""
def _is_local_host(host: str) -> bool:
"""Return whether one libpq ``host`` entry is a Unix socket or loopback.
An empty entry means libpq's default Unix socket, and a value starting
with ``/`` (a socket directory) or ``@`` (a Linux abstract socket) is a
socket as well.
"""
if not host or host.startswith(("/", "@")) or host.lower() == "localhost":
return True
try:
return ipaddress.ip_address(host).is_loopback
except ValueError:
return False
def _log_safe_host(host: str) -> str:
"""Return a libpq ``host`` (or ``hostaddr``) value safe to log.
Each entry of a comma-separated list is credential-masked with
``redact_url_credentials()`` (``pw@db.example`` becomes
``***@db.example``), and the result is escaped for the log.
"""
entries = [str(redact_url_credentials(entry)) for entry in host.split(",")]
return sanitize_for_log(",".join(entries))
_URI_PREFIXES = ("postgresql://", "postgres://")
def _is_uri(connection_string: str) -> bool:
"""Return whether a connection string is in libpq's URI form.
libpq itself recognizes only the exact lowercase prefixes
``postgresql://`` and ``postgres://`` at the very start of the string,
and parses anything else as keyword/value pairs. The check here also
ignores case and leading whitespace: such a string fails libpq's
keyword/value parse anyway, so treating it as a URI only refuses it
with a different fixed message.
"""
return connection_string.lstrip().lower().startswith(_URI_PREFIXES)
def _port_is_numeric(entry: str) -> bool:
"""Return whether one ``host[:port]`` entry of a URI authority has no
port or an all-digit one. A bracketed IPv6 address is skipped first, so
its colons are not taken for a port separator."""
entry = entry.strip()
if entry.startswith("["):
entry = entry.partition("]")[2]
if ":" not in entry:
return True
port = entry.rpartition(":")[2]
return port.isascii() and port.isdigit()
def _has_unencoded_at(connection_string: str) -> bool:
"""Return whether a URI connection string looks like its user name or
password holds an unencoded ``@`` or ``/``.
The scan mirrors libpq's URI parser. libpq looks for the userinfo by
scanning the text after ``://`` up to the first ``@`` or ``/``; when it
meets ``@`` first, everything before it is the user name and password,
the host list runs to the next ``/`` or ``?``, and the database name
from that ``/`` to the next ``?``. So the rest of a password containing
an unencoded ``@`` lands in the host
(``postgresql://u:s3c@pw@db.example/d`` has host ``pw@db.example``),
the port, the database name or a query parameter, and a password
containing an unencoded ``/`` ends the authority early
(``postgresql://u:SECRET/x@db.example/d`` has no userinfo, host ``u``,
port ``SECRET`` and dbname ``x@db.example/d``).
The string is refused when, with userinfo, the text from it to the next
``/`` holds another ``@`` (the host list, or a query parameter written
before any ``/``), or the database name holds an ``@``; and, without
userinfo, when the database name holds an ``@``, or an ``@`` follows an
authority whose port is not a number (``u:pa?sslmode=SECRET/x@db/d``
has port ``pa`` and sslmode ``SECRET/x@db/d``). A URI never needs an
unencoded ``@`` in those places: libpq decodes ``%40`` and ``%2F`` in
every component. An ``@`` in a query value after a ``/``
(``?sslrootcert=/home/a@b/ca.pem``) is not refused, because nothing
tells it apart from a legitimate one. For the same reason two shapes
are not recognized: an all-digit password before a ``/``, followed by
an ``@`` only in a query value (``u:12345/x?sslrootcert=a@b``), and a
password with an ``@``, then a ``?``, then a ``/``
(``u:s3c@pw?sslmode=SECRET/x@db``).
This refusal is a convenience that names the common mistake; it does
not keep the password out of the logs. That is done by withholding
every value parsed from a connection string from the startup warning
and the connection error (``PostgreSQLClient._connect()``), whatever
shape the string has.
"""
rest = connection_string.partition("://")[2]
first = next((i for i, char in enumerate(rest) if char in "@/"), -1)
if first == -1:
return False
if rest[first] == "@":
segment, slash, tail = rest[first + 1 :].partition("/")
if "@" in segment:
return True
# A "?" before the "/" puts that "/" in a query value.
return bool(slash) and "?" not in segment and "@" in tail.partition("?")[0]
authority, question, _ = rest[:first].partition("?")
tail = rest[first + 1 :]
if not question and "@" in tail.partition("?")[0]:
return True
return "@" in tail and not all(
_port_is_numeric(entry) for entry in authority.split(",")
)
# Two timestamp conventions coexist in parsed reports, so two helpers are
# needed — do not collapse them into one. Aggregate *report* begin/end dates
# come from ``timestamp_to_human()`` → ``datetime.fromtimestamp()``, which is
# **local** naive time, so they go through ``_naive_local_to_timestamptz``.
# Aggregate *record* interval_begin/end and SMTP-TLS begin/end are already
# **UTC** naive strings, so they only need a ``+00`` suffix via
# ``_ensure_utc_suffix``. Using the wrong helper silently shifts timestamps.
def _ensure_utc_suffix(value: str | None) -> str | None:
"""Append ``+00`` to a timestamp string if it lacks timezone info.
Several parsers produce ``YYYY-MM-DD HH:MM:SS`` format strings that
are known to be UTC but lack an explicit offset. PostgreSQL
``TIMESTAMPTZ`` columns need the offset to avoid interpreting the
value in the session timezone. A value that is not a string is
returned unchanged, for PostgreSQL to accept or reject.
"""
if (
isinstance(value, str)
and value
and "+" not in value
and "-" not in value[10:]
and "Z" not in value
):
return value + "+00"
return value
def _naive_local_to_timestamptz(value: str | None) -> str | None:
"""Convert a naive local-time string to an ISO 8601 string with offset.
``timestamp_to_human()`` produces ``YYYY-MM-DD HH:MM:SS`` in
**local** time (via ``datetime.fromtimestamp()``). Inserting such
a string into a ``TIMESTAMPTZ`` column would cause PostgreSQL to
interpret it using the *session* timezone, which may differ from
the machine's local timezone.
This helper re-parses the string, attaches the local timezone
offset, and returns an ISO 8601 representation that PostgreSQL
will interpret unambiguously.
"""
if not value:
return value
naive = datetime.strptime(value, "%Y-%m-%d %H:%M:%S")
aware = naive.astimezone() # attaches the local system timezone
return aware.isoformat()
def _normalize_arrival_date(value: str | None) -> str | None:
"""Normalize a failure-report ``arrival_date`` for safe TIMESTAMPTZ insert.
The arrival date may be an RFC 2822 string (e.g.
``Fri, 28 Oct 2022 00:34:24 +0800``) or an ISO 8601 string.
``human_timestamp_to_datetime`` (backed by *dateutil*) can parse
both. We convert to UTC and return an ISO 8601 string with offset
so PostgreSQL interprets it unambiguously.
"""
if not value:
return value
try:
dt = human_timestamp_to_datetime(value, to_utc=True)
return dt.strftime("%Y-%m-%d %H:%M:%S") + "+00"
except Exception:
# If parsing fails, return as-is and let PostgreSQL try.
return value
def _contact_info_to_text(
value: str | list | None,
) -> str | None:
"""Ensure ``contact_info`` is a plain string.
The TLS-RPT ``contact-info`` field is normally a single string, but
the TypedDict allows ``str | list[str]``. If a list is
encountered, join the entries so they fit into a ``TEXT`` column.
"""
if value is None:
return None
if isinstance(value, list):
return ", ".join(str(v) for v in value)
return str(value)
# PostgreSQL cannot store a NUL character in text or jsonb, and a sender can
# plant one in a failure report's sample through an RFC 2047 encoded-word
# (=?utf-8?q?a=00b?=). It is replaced with U+FFFD, the replacement character,
# which keeps the text searchable and marks the spot (GHSA-fxr7-7h9q-v949).
_NUL_REPLACEMENT = "\ufffd"
def _strip_nul(value: Any) -> Any:
"""Return ``value`` with every NUL in its strings replaced by U+FFFD.
Dict keys and values and list items are replaced recursively; any
other value is returned unchanged. Used for the values wrapped in
``Jsonb``, which the server rejects when they contain ``\\u0000``.
"""
if isinstance(value, str):
return value.replace("\x00", _NUL_REPLACEMENT)
if isinstance(value, dict):
return {_strip_nul(k): _strip_nul(v) for k, v in value.items()}
if isinstance(value, list):
return [_strip_nul(item) for item in value]
return value
def _nul_safe_dumper_classes() -> tuple[type, type]:
"""Return the text and binary ``str`` dumpers that replace NUL.
psycopg dumps a ``str`` parameter with ``StrDumperUnknown`` (text
format) or ``StrBinaryDumper`` (binary format); the first raises
``DataError`` on a NUL and the server rejects the second. ``list[str]``
parameters (``TEXT[]`` columns) dump each item with the same dumpers.
The subclasses replace NUL with U+FFFD first.
"""
class _NulSafeStrDumper(psycopg_string.StrDumperUnknown):
def dump(self, obj: str) -> Buffer | None:
return super().dump(obj.replace("\x00", _NUL_REPLACEMENT))
class _NulSafeStrBinaryDumper(psycopg_string.StrBinaryDumper):
def dump(self, obj: str) -> Buffer | None:
return super().dump(obj.replace("\x00", _NUL_REPLACEMENT))
return _NulSafeStrDumper, _NulSafeStrBinaryDumper
def _register_nul_safe_dumpers(conn: psycopg.Connection) -> None:
"""Make ``conn`` dump every ``str`` parameter with NUL replaced.
psycopg uses the dumper registered last for a type as its default, so
the binary dumper is registered first and the text one last, the order
psycopg's own ``register_default_adapters`` uses.
"""
text_dumper, binary_dumper = _nul_safe_dumper_classes()
conn.adapters.register_dumper(str, binary_dumper)
conn.adapters.register_dumper(str, text_dumper)
# Most characters of a driver message that a PostgreSQLRejectedReport quotes:
# a DataError can quote the rejected value, which the sender chose.
_MAX_REJECTION_DETAIL_CHARS = 200
def _rejection_detail(exc: BaseException) -> str:
"""Return the driver's message for a rejected value, cut to
``_MAX_REJECTION_DETAIL_CHARS`` characters with an ellipsis and then
escaped for the log (cutting first means the cut cannot split an
escape sequence)."""
detail = str(exc)
suffix = ""
if len(detail) > _MAX_REJECTION_DETAIL_CHARS:
detail = detail[:_MAX_REJECTION_DETAIL_CHARS]
suffix = "..."
return sanitize_for_log(detail) + suffix
class PostgreSQLError(RuntimeError):
"""Raised when a PostgreSQL-level error occurs"""
class PostgreSQLRejectedReport(PostgreSQLError):
"""Raised when PostgreSQL rejected a value of one report.
psycopg raised ``DataError`` (a value the column type cannot hold) and
that report's transaction was rolled back. The connection and the
schema are fine, so the next report will save.
"""
class AlreadySaved(ValueError):
"""Raised when an identical report already exists in the database"""
class PostgreSQLClient:
"""A client for saving DMARC reports to a PostgreSQL database.
Accepts either a full libpq connection string/DSN via
*connection_string* or individual connection parameters. When both
are supplied *connection_string* takes precedence.
"""
def __init__(
self,
connection_string: str | None = None,
host: str | None = None,
port: int = 5432,
user: str | None = None,
password: str | None = None,
database: str | None = None,
sslmode: str | None = None,
sslrootcert: str | None = None,
) -> None:
"""
Initializes the PostgreSQLClient and opens a database connection.
Args:
connection_string: A libpq connection string or URI
(e.g. ``postgresql://user:pass@host/dbname``). When
present, *host*, *port*, *user*, *password* and *database*
are ignored; *sslmode* and *sslrootcert* still apply and
override the string's own values.
host: Database server hostname or IP address.
port: Database server port (default: 5432).
user: Database user name.
password: Database user password.
database: Database name to connect to.
sslmode (str | None): A libpq sslmode, one of ``SSL_MODES``.
``None`` keeps an ``sslmode`` set in *connection_string* or
the ``PGSSLMODE`` environment variable, and otherwise uses
``require``.
sslrootcert: Path to the CA certificate file that
``verify-ca`` and ``verify-full`` check the server's
certificate against.
Raises:
PostgreSQLError: If psycopg is not installed, the connection
string is malformed, or the connection attempt fails.
"""
if psycopg is None:
raise PostgreSQLError(_PSYCOPG_INSTALL_HINT)
# Store parameters so we can reconnect later if needed.
self._connection_string = connection_string
self._host = host
self._port = port
self._user = user
self._password = password
self._database = database
self._sslmode = sslmode
self._sslrootcert = sslrootcert
self._conn: psycopg.Connection | None = None
self._warn_if_server_unverified()
self._connect()
def _connection_string_params(self) -> dict[str, Any]:
"""Return the parameters set in the connection string, if any.
Raises:
PostgreSQLError: If the connection string is malformed. libpq
words its parse errors with the string or a fragment of it,
which can contain the password, so the detail is withheld
(any error from the parse is replaced). Also, as a
convenience that names the common mistake, if a URI looks
like its user name or password holds an unencoded ``@`` or
``/`` (``_has_unencoded_at()``). That check runs before
anything is logged, but it recognizes shapes and is not
what keeps the password out of the logs; the startup
warning and the connection error withhold every value
parsed from the string instead. A keyword/value string is
not checked: its password is a keyword of its own, which
an ``@`` or ``/`` does not end.
"""
if not self._connection_string:
return {}
if _is_uri(self._connection_string) and _has_unencoded_at(
self._connection_string
):
raise PostgreSQLError(_UNENCODED_AT_CONNECTION_STRING_ERROR) from None
try:
params = psycopg_conninfo.conninfo_to_dict(self._connection_string)
except Exception:
raise PostgreSQLError(_WITHHELD_CONNECTION_STRING_ERROR) from None
return params
def _ssl_connect_kwargs(self) -> dict[str, Any]:
"""Return the TLS keyword arguments for ``psycopg.connect()``.
An explicit *sslmode* wins. Without one, an ``sslmode`` in the
connection string or the ``PGSSLMODE`` environment variable, both
of which libpq reads itself, is left in effect, and otherwise
``require`` replaces libpq's ``prefer`` default.
"""
kwargs: dict[str, Any] = {}
if self._sslmode:
kwargs["sslmode"] = self._sslmode
elif not (
self._connection_string_params().get("sslmode")
or os.environ.get("PGSSLMODE")
):
kwargs["sslmode"] = DEFAULT_SSL_MODE
if self._sslrootcert:
kwargs["sslrootcert"] = self._sslrootcert
return kwargs
def _warn_if_server_unverified(self) -> None:
"""Log a warning when a remote server's certificate is not checked.
Below ``verify-ca``, anyone who can impersonate a TCP host can
receive the password. A Unix socket, ``localhost`` or a loopback
address is not warned about, and neither is an unset host (libpq's
default socket).
The host is the one libpq connects to: the connection string's
``hostaddr`` (else the ``PGHOSTADDR`` environment variable) when
set, since libpq connects to it and uses ``host`` only to name the
server, and otherwise the *host* argument or the connection
string's ``host``, else the ``PGHOST`` environment variable. Every
entry of a comma-separated host list is considered.
With a connection string, the warning names no host and no other
value from the string, whatever its shape: when the string's user
name or password holds an unencoded ``@`` or ``/``, libpq parses a
fragment of the password as the host, the port or a query
parameter such as ``sslmode``. The mode is printed only when it is
one of ``SSL_MODES``. With the separate *host* argument (or
``PGHOST``/``PGHOSTADDR``, which the operator sets), the warning
names the host, credential-masked and escaped, which covers a host
with an ``@``.
With ``sslmode=require`` and a root certificate file that exists,
libpq verifies the server as ``verify-ca`` does ("if a root CA file
exists, the behavior of sslmode=require will be the same as that of
verify-ca", PostgreSQL docs, libpq SSL support), so that is not
warned about either.
"""
params = self._connection_string_params()
if self._connection_string:
host = params.get("host")
hostaddr = params.get("hostaddr")
else:
host = self._host
hostaddr = None
host = (
hostaddr or os.environ.get("PGHOSTADDR") or host or os.environ.get("PGHOST")
)
if not host or all(_is_local_host(h.strip()) for h in host.split(",")):
return
mode = (
self._ssl_connect_kwargs().get("sslmode")
or params.get("sslmode")
or os.environ.get("PGSSLMODE")
)
if mode in _VERIFYING_SSL_MODES:
return
rootcert = (
self._sslrootcert
or params.get("sslrootcert")
or os.environ.get("PGSSLROOTCERT")
)
if mode == "require" and rootcert and os.path.isfile(rootcert):
return
if self._connection_string:
shown_mode = mode if mode in SSL_MODES else "not a recognized value"
logger.warning(
"The PostgreSQL connection_string does not verify the server "
f"certificate (sslmode is {shown_mode}); set [postgresql] "
"sslmode = verify-full and sslrootcert to protect the password "
"from an impersonated server"
)
return
logger.warning(
f"PostgreSQL connection to {_log_safe_host(host)} uses "
f"sslmode={sanitize_for_log(str(mode))}, which does not verify the "
"server certificate; set [postgresql] sslmode = verify-full and "
"sslrootcert to protect the password from an impersonated server"
)
def _connect(self) -> psycopg.Connection:
"""Open a new database connection using stored parameters.
Raises:
PostgreSQLError: If the connection string is malformed or the
connection attempt fails. With a connection string, the
driver's message is withheld (only a well-formed SQLSTATE
is kept); with the separate arguments, it is passed on with
URL credentials masked. Any other error raised between
reading the settings and holding a configured connection
(a password the driver cannot encode, a host name the
resolver rejects) is replaced with a fixed message, since
its text can quote the password or the whole connection
string. Either way the original error is not chained.
"""
logger.debug("Connecting to PostgreSQL")
try:
# Parse the connection string first, on every path (the sslmode
# lookup below skips it when sslmode is set), so a malformed
# string or one the early refusal recognizes never reaches the
# driver.
self._connection_string_params()
ssl_kwargs = self._ssl_connect_kwargs()
if self._connection_string:
conn = psycopg.connect(self._connection_string, **ssl_kwargs)
else:
conn = psycopg.connect(
host=self._host,
port=self._port,
user=self._user,
password=self._password,
dbname=self._database,
**ssl_kwargs,
)
conn.autocommit = False
_register_nul_safe_dumpers(conn)
except PostgreSQLError:
# Already a fixed, secret-free message (_connection_string_params).
raise
except psycopg.Error as exc:
# Chaining would print the driver's own text, unredacted,
# wherever the error is logged with its traceback (the SIGHUP
# reload path does that), so every branch raises from None.
if self._connection_string:
# libpq reports a malformed connection string by echoing the
# whole string, or the fragment it could not parse. psycopg
# raises ProgrammingError for those parse failures. The
# string was already parsed once (_connection_string_params),
# so this guards the same failure if it surfaces here.
if isinstance(exc, psycopg.ProgrammingError):
raise PostgreSQLError(_WITHHELD_CONNECTION_STRING_ERROR) from None
# Any other failure can quote a value parsed from the string
# (libpq: 'invalid integer value "..." for connection option
# "port"'), and with an unencoded "@" or "/" in the user
# name or password, that value is part of the password. No
# check of the string's shape can rule that out, so the text
# is withheld whatever the string looks like.
raise PostgreSQLError(
_WITHHELD_CONNECTION_FAILURE_ERROR + _sqlstate_suffix(exc)
) from None
# With the separate arguments, the text names the host, and a
# host from the argument or PGHOST can hold anything, so it is
# credential-masked.
raise PostgreSQLError(redact_url_credentials(str(exc))) from None
except Exception:
# Not the driver's own error: psycopg encodes the connection
# string it builds from the separate arguments, so a password
# holding a lone surrogate (an environment variable with invalid
# UTF-8) raises UnicodeEncodeError carrying that whole string,
# and on Python 3.15 a host label over 63 characters raises
# UnicodeEncodeError('idna', '<host>', ...) from name resolution,
# which psycopg does not wrap. Such text is never passed on.
# KeyboardInterrupt and SystemExit are not Exceptions and pass.
if self._connection_string:
raise PostgreSQLError(_WITHHELD_CONNECTION_FAILURE_ERROR) from None
raise PostgreSQLError(_WITHHELD_CONNECTION_SETTING_ERROR) from None
self._conn = conn
return conn
def close(self) -> None:
"""Close the database connection if it is open.
Called by the CLI's output-client cleanup on shutdown / config
reload. Safe to call multiple times.
"""
if self._conn is not None and not self._conn.closed:
self._conn.close()
def _ensure_connected(self) -> psycopg.Connection:
"""Check the connection health and reconnect if necessary.
When *parsedmarc* runs in watch mode the process can stay alive
for days or weeks. PostgreSQL may drop idle connections (e.g.
server restart, ``idle_in_transaction_session_timeout``, TCP
keep-alive expiry). This method detects a closed connection
and transparently re-establishes it so that subsequent
``save_*`` calls succeed without manual intervention.
"""
conn = self._conn
if conn is None or conn.closed:
logger.warning("PostgreSQL connection lost — attempting to reconnect")
conn = self._connect()
return conn
def create_tables(self) -> None:
"""Creates all required tables if they do not already exist.
This method is idempotent and safe to call on every startup.
Raises:
PostgreSQLError: If table creation fails.
"""
conn = self._ensure_connected()
ddl_statements: list[LiteralString] = [
# ----------------------------------------------------------------
# Aggregate reports
# ----------------------------------------------------------------
"""
CREATE TABLE IF NOT EXISTS dmarc_aggregate_report (
id BIGSERIAL PRIMARY KEY,
xml_schema TEXT,
xml_namespace TEXT,
org_name TEXT NOT NULL,
org_email TEXT,
org_extra_contact_info TEXT,
generator TEXT,
report_id TEXT NOT NULL,
begin_date TIMESTAMPTZ NOT NULL,
end_date TIMESTAMPTZ NOT NULL,
errors TEXT[],
domain TEXT NOT NULL,
adkim TEXT,
aspf TEXT,
policy TEXT,
subdomain_policy TEXT,
pct TEXT,
fo TEXT,
np TEXT,
testing TEXT,
discovery_method TEXT,
UNIQUE (org_name, report_id, domain, begin_date, end_date)
)
""",
"""
CREATE TABLE IF NOT EXISTS dmarc_aggregate_record (
id BIGSERIAL PRIMARY KEY,
report_id BIGINT NOT NULL
REFERENCES dmarc_aggregate_report(id)
ON DELETE CASCADE,
interval_begin TIMESTAMPTZ,
interval_end TIMESTAMPTZ,
source_ip_address INET,
source_country TEXT,
source_reverse_dns TEXT,
source_base_domain TEXT,
source_name TEXT,
source_type TEXT,
message_count INTEGER NOT NULL,
spf_aligned BOOLEAN,
dkim_aligned BOOLEAN,
dmarc_passed BOOLEAN,
disposition TEXT,
policy_dkim TEXT,
policy_spf TEXT,
header_from TEXT,
envelope_from TEXT,
envelope_to TEXT
)
""",
"""
CREATE TABLE IF NOT EXISTS dmarc_aggregate_record_dkim (
id BIGSERIAL PRIMARY KEY,
record_id BIGINT NOT NULL
REFERENCES dmarc_aggregate_record(id)
ON DELETE CASCADE,
domain TEXT,
selector TEXT,
result TEXT,
human_result TEXT
)
""",
"""
CREATE TABLE IF NOT EXISTS dmarc_aggregate_record_spf (
id BIGSERIAL PRIMARY KEY,
record_id BIGINT NOT NULL
REFERENCES dmarc_aggregate_record(id)
ON DELETE CASCADE,
domain TEXT,
scope TEXT,
result TEXT,
human_result TEXT
)
""",
"""
CREATE TABLE IF NOT EXISTS dmarc_aggregate_record_policy_override (
id BIGSERIAL PRIMARY KEY,
record_id BIGINT NOT NULL
REFERENCES dmarc_aggregate_record(id)
ON DELETE CASCADE,
override_type TEXT,
comment TEXT
)
""",
# ----------------------------------------------------------------
# Failure reports
# ----------------------------------------------------------------
"""
CREATE TABLE IF NOT EXISTS dmarc_failure_report (
id BIGSERIAL PRIMARY KEY,
feedback_type TEXT,
user_agent TEXT,
version TEXT,
original_envelope_id TEXT,
original_mail_from TEXT,
original_rcpt_to TEXT,
arrival_date TIMESTAMPTZ,
arrival_date_utc TIMESTAMPTZ,
authentication_results TEXT,
delivery_result TEXT,
auth_failure TEXT[],
authentication_mechanisms TEXT[],
dkim_domain TEXT,
reported_domain TEXT,
sample_headers_only BOOLEAN,
source_ip_address INET,
source_country TEXT,
source_reverse_dns TEXT,
source_base_domain TEXT,
source_name TEXT,
source_type TEXT,
sample TEXT,
sample_date TEXT,
sample_subject TEXT,
sample_body TEXT,
sample_has_defects BOOLEAN,
sample_headers JSONB,
sample_from JSONB,
sample_to JSONB
)
""",
"""
CREATE TABLE IF NOT EXISTS dmarc_failure_sample_address (
id BIGSERIAL PRIMARY KEY,
report_id BIGINT NOT NULL
REFERENCES dmarc_failure_report(id)
ON DELETE CASCADE,
address_type TEXT,
display_name TEXT,
address TEXT
)
""",
# ----------------------------------------------------------------
# SMTP TLS reports
# ----------------------------------------------------------------
"""
CREATE TABLE IF NOT EXISTS smtp_tls_report (
id BIGSERIAL PRIMARY KEY,
organization_name TEXT NOT NULL,
begin_date TIMESTAMPTZ NOT NULL,
end_date TIMESTAMPTZ NOT NULL,
contact_info TEXT,
report_id TEXT NOT NULL,
UNIQUE (organization_name, report_id, begin_date, end_date)
)
""",
"""
CREATE TABLE IF NOT EXISTS smtp_tls_policy (
id BIGSERIAL PRIMARY KEY,
report_id BIGINT NOT NULL
REFERENCES smtp_tls_report(id)
ON DELETE CASCADE,
policy_domain TEXT,
policy_type TEXT,
policy_strings TEXT[],
mx_host_patterns TEXT[],
successful_session_count INTEGER,
failed_session_count INTEGER
)
""",
"""
CREATE TABLE IF NOT EXISTS smtp_tls_failure_detail (
id BIGSERIAL PRIMARY KEY,
policy_id BIGINT NOT NULL
REFERENCES smtp_tls_policy(id)
ON DELETE CASCADE,
result_type TEXT,
failed_session_count INTEGER,
sending_mta_ip INET,
receiving_ip INET,
receiving_mx_hostname TEXT,
receiving_mx_helo TEXT,
additional_info_uri TEXT,
failure_reason_code TEXT
)
""",
# ----- indexes for Grafana dashboard query performance -----
"""
CREATE INDEX IF NOT EXISTS idx_agg_report_begin_date
ON dmarc_aggregate_report (begin_date)
""",
"""
CREATE INDEX IF NOT EXISTS idx_agg_record_report_id
ON dmarc_aggregate_record (report_id)
""",
"""
CREATE INDEX IF NOT EXISTS idx_agg_record_header_from
ON dmarc_aggregate_record (header_from)
""",
"""
CREATE INDEX IF NOT EXISTS idx_failure_report_arrival_date
ON dmarc_failure_report (arrival_date_utc)
""",
"""
CREATE INDEX IF NOT EXISTS idx_smtp_tls_report_begin_date
ON smtp_tls_report (begin_date)
""",
"""
CREATE INDEX IF NOT EXISTS idx_smtp_tls_policy_report_id
ON smtp_tls_policy (report_id)
""",
# ----- remaining foreign-key columns (PostgreSQL does not index
# the referencing side) -----
"""
CREATE INDEX IF NOT EXISTS idx_agg_record_dkim_record_id
ON dmarc_aggregate_record_dkim (record_id)
""",
"""
CREATE INDEX IF NOT EXISTS idx_agg_record_spf_record_id
ON dmarc_aggregate_record_spf (record_id)
""",
"""
CREATE INDEX IF NOT EXISTS idx_agg_record_override_record_id
ON dmarc_aggregate_record_policy_override (record_id)
""",
"""
CREATE INDEX IF NOT EXISTS idx_failure_sample_address_report_id
ON dmarc_failure_sample_address (report_id)
""",
"""
CREATE INDEX IF NOT EXISTS idx_smtp_tls_failure_detail_policy_id
ON smtp_tls_failure_detail (policy_id)
""",
]
try:
with conn.transaction():
with conn.cursor() as cur:
for stmt in ddl_statements:
cur.execute(stmt)
logger.debug("PostgreSQL tables verified / created")
except psycopg.Error as exc:
raise PostgreSQLError(str(exc)) from exc
def save_aggregate_report_to_postgresql(self, report: dict) -> None:
"""Saves a parsed aggregate DMARC report to PostgreSQL.
Args:
report: A parsed aggregate report dictionary (the ``report``
value of a :func:`parsedmarc.parse_report_file` result).
Raises:
AlreadySaved: If an identical report is already present.
PostgreSQLRejectedReport: If PostgreSQL rejected a value of the
report.
PostgreSQLError: If another database error occurs.
"""
conn = self._ensure_connected()
meta = report.get("report_metadata", {})
pub = report.get("policy_published", {})
try:
with conn.transaction():
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO dmarc_aggregate_report (
xml_schema, xml_namespace, org_name, org_email,
org_extra_contact_info, generator, report_id,
begin_date, end_date, errors,
domain, adkim, aspf, policy,
subdomain_policy, pct, fo,
np, testing, discovery_method
) VALUES (
%s, %s, %s, %s,
%s, %s, %s,
%s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s,
%s, %s, %s
)
ON CONFLICT (org_name, report_id, domain,
begin_date, end_date)
DO NOTHING
RETURNING id
""",
(
report.get("xml_schema"),
report.get("xml_namespace"),
meta.get("org_name"),
meta.get("org_email"),
meta.get("org_extra_contact_info"),
meta.get("generator"),
meta.get("report_id"),
_naive_local_to_timestamptz(meta.get("begin_date")),
_naive_local_to_timestamptz(meta.get("end_date")),
meta.get("errors") or [],
pub.get("domain"),
pub.get("adkim"),
pub.get("aspf"),
pub.get("p"),
pub.get("sp"),
pub.get("pct"),
pub.get("fo"),
pub.get("np"),
pub.get("testing"),
pub.get("discovery_method"),
),
)
row = cur.fetchone()
if row is None:
raise AlreadySaved(
"Aggregate report {report_id} from {org} "
"has already been saved".format(
report_id=meta.get("report_id"),
org=meta.get("org_name"),
)
)
report_db_id: int = row[0]
for record in report.get("records", []):
src = record.get("source", {})
pol = record.get("policy_evaluated", {})
idens = record.get("identifiers", {})
cur.execute(
"""
INSERT INTO dmarc_aggregate_record (
report_id, interval_begin, interval_end,
source_ip_address, source_country,
source_reverse_dns, source_base_domain,
source_name, source_type,
message_count,
spf_aligned, dkim_aligned, dmarc_passed,
disposition, policy_dkim, policy_spf,
header_from, envelope_from, envelope_to
) VALUES (
%s, %s, %s,
%s, %s, %s, %s, %s, %s,
%s,
%s, %s, %s,
%s, %s, %s,
%s, %s, %s
)
RETURNING id
""",
(
report_db_id,
_ensure_utc_suffix(record.get("interval_begin")),
_ensure_utc_suffix(record.get("interval_end")),
src.get("ip_address"),
src.get("country"),
src.get("reverse_dns"),
src.get("base_domain"),
src.get("name"),
src.get("type"),
record.get("count"),
record.get("alignment", {}).get("spf"),
record.get("alignment", {}).get("dkim"),
record.get("alignment", {}).get("dmarc"),
pol.get("disposition"),
pol.get("dkim"),
pol.get("spf"),
idens.get("header_from"),
idens.get("envelope_from"),
idens.get("envelope_to"),
),
)
# INSERT ... RETURNING always yields one row
record_db_id: int = cur.fetchone()[0] # pyright: ignore[reportOptionalSubscript]
for dkim in record.get("auth_results", {}).get("dkim", []):
cur.execute(
"""
INSERT INTO dmarc_aggregate_record_dkim
(record_id, domain, selector, result,
human_result)
VALUES (%s, %s, %s, %s, %s)
""",
(
record_db_id,
dkim.get("domain"),
dkim.get("selector"),
dkim.get("result"),
dkim.get("human_result"),
),
)
for spf in record.get("auth_results", {}).get("spf", []):
cur.execute(
"""
INSERT INTO dmarc_aggregate_record_spf
(record_id, domain, scope, result,
human_result)
VALUES (%s, %s, %s, %s, %s)
""",
(
record_db_id,
spf.get("domain"),
spf.get("scope"),
spf.get("result"),
spf.get("human_result"),
),
)
for override in pol.get("policy_override_reasons", []):
cur.execute(
"""
INSERT INTO dmarc_aggregate_record_policy_override
(record_id, override_type, comment)
VALUES (%s, %s, %s)
""",
(
record_db_id,
override.get("type"),
override.get("comment"),
),
)
except AlreadySaved:
raise
except psycopg.DataError as exc:
raise PostgreSQLRejectedReport(
"PostgreSQL rejected a value of the aggregate report from "
f"{sanitize_for_log(str(meta.get('org_name')))} with ID "
f"{sanitize_for_log(str(meta.get('report_id')))}: "
f"{_rejection_detail(exc)}"
) from exc
except psycopg.Error as exc:
raise PostgreSQLError(str(exc)) from exc
def save_failure_report_to_postgresql(self, report: dict) -> None:
"""Saves a parsed failure (RUF) DMARC report to PostgreSQL.
Args:
report: A parsed failure report dictionary (the ``report``
value of a :func:`parsedmarc.parse_report_file` result).
Raises:
AlreadySaved: If a matching failure report is already present.
PostgreSQLRejectedReport: If PostgreSQL rejected a value of the
report.
PostgreSQLError: If another database error occurs.
"""
conn = self._ensure_connected()
sample = report.get("parsed_sample", {}) or {}
src = report.get("source", {}) or {}
arrival_date_utc = _ensure_utc_suffix(report.get("arrival_date_utc"))
sample_subject = sample.get("subject")
# JSONB values are reused by both the dedup check and the INSERT.
# jsonb cannot hold \u0000, so NUL is replaced before wrapping. The
# str dumpers registered in _connect() cover the text columns.
sample_headers = (
psycopg_json.Jsonb(_strip_nul(sample["headers"]))
if sample.get("headers")
else None
)
sample_from = (
psycopg_json.Jsonb(_strip_nul(sample["from"]))
if sample.get("from")
else None
)
sample_to = (
psycopg_json.Jsonb(_strip_nul(sample["to"])) if sample.get("to") else None
)
try:
with conn.transaction():
with conn.cursor() as cur:
# Failure reports have no natural primary key, so mirror the
# Elasticsearch backend's query-then-insert dedup on the same
# dimensions it uses: arrival date + From + To + Subject.
# IS NOT DISTINCT FROM is NULL-safe (no PG15 NULLS NOT
# DISTINCT dependency); JSONB equality is semantic, so key
# order within the From/To objects doesn't matter.
cur.execute(
"""
SELECT 1 FROM dmarc_failure_report
WHERE arrival_date_utc IS NOT DISTINCT FROM %s
AND sample_subject IS NOT DISTINCT FROM %s
AND sample_from IS NOT DISTINCT FROM %s
AND sample_to IS NOT DISTINCT FROM %s
LIMIT 1
""",
(arrival_date_utc, sample_subject, sample_from, sample_to),
)
if cur.fetchone() is not None:
raise AlreadySaved(
f"A failure report with subject {sample_subject!r} arriving "
f"at {arrival_date_utc} has already been saved"
)
cur.execute(
"""
INSERT INTO dmarc_failure_report (
feedback_type, user_agent, version,
original_envelope_id, original_mail_from,
original_rcpt_to, arrival_date, arrival_date_utc,
authentication_results, delivery_result,
auth_failure, authentication_mechanisms,
dkim_domain, reported_domain, sample_headers_only,
source_ip_address, source_country,
source_reverse_dns, source_base_domain,
source_name, source_type,
sample, sample_date, sample_subject,
sample_body, sample_has_defects,
sample_headers, sample_from, sample_to
) VALUES (
%s, %s, %s,
%s, %s,
%s, %s, %s,
%s, %s,
%s, %s,
%s, %s, %s,
%s, %s,
%s, %s,
%s, %s,
%s, %s, %s,
%s, %s,
%s, %s, %s
)
RETURNING id
""",
(
report.get("feedback_type"),
report.get("user_agent"),
report.get("version"),
report.get("original_envelope_id"),
report.get("original_mail_from"),
report.get("original_rcpt_to"),
_normalize_arrival_date(report.get("arrival_date")),
arrival_date_utc,
report.get("authentication_results"),
report.get("delivery_result"),
report.get("auth_failure") or [],
report.get("authentication_mechanisms") or [],
report.get("dkim_domain"),
report.get("reported_domain"),
report.get("sample_headers_only"),
src.get("ip_address"),
src.get("country"),
src.get("reverse_dns"),
src.get("base_domain"),
src.get("name"),
src.get("type"),
report.get("sample"),
sample.get("date"),
sample_subject,
sample.get("body"),
sample.get("has_defects"),
sample_headers,
sample_from,
sample_to,
),
)
# INSERT ... RETURNING always yields one row
report_db_id: int = cur.fetchone()[0] # pyright: ignore[reportOptionalSubscript]
for addr_type in ("to", "cc", "bcc", "reply_to"):
entries = sample.get(addr_type) or []
if isinstance(entries, dict):
entries = [entries]
for entry in entries:
cur.execute(
"""
INSERT INTO dmarc_failure_sample_address
(report_id, address_type,
display_name, address)
VALUES (%s, %s, %s, %s)
""",
(
report_db_id,
addr_type,
entry.get("display_name"),
entry.get("address"),
),
)
except AlreadySaved:
raise
except psycopg.DataError as exc:
raise PostgreSQLRejectedReport(
"PostgreSQL rejected a value of the failure report for "
f"{sanitize_for_log(str(report.get('reported_domain')))} "
f"arriving at {sanitize_for_log(str(arrival_date_utc))}: "
f"{_rejection_detail(exc)}"
) from exc
except psycopg.Error as exc:
raise PostgreSQLError(str(exc)) from exc
def save_smtp_tls_report_to_postgresql(self, report: dict) -> None:
"""Saves a parsed SMTP TLS report to PostgreSQL.
Args:
report: A parsed SMTP TLS report dictionary (the ``report``
value of a :func:`parsedmarc.parse_report_file` result).
Raises:
AlreadySaved: If an identical report is already present.
PostgreSQLRejectedReport: If PostgreSQL rejected a value of the
report.
PostgreSQLError: If another database error occurs.
"""
conn = self._ensure_connected()
try:
with conn.transaction():
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO smtp_tls_report (
organization_name, begin_date, end_date,
contact_info, report_id
) VALUES (%s, %s, %s, %s, %s)
ON CONFLICT (organization_name, report_id,
begin_date, end_date)
DO NOTHING
RETURNING id
""",
(
report.get("organization_name"),
_ensure_utc_suffix(report.get("begin_date")),
_ensure_utc_suffix(report.get("end_date")),
_contact_info_to_text(report.get("contact_info")),
report.get("report_id"),
),
)
row = cur.fetchone()
if row is None:
raise AlreadySaved(
"SMTP TLS report {report_id} from {org} "
"has already been saved".format(
report_id=report.get("report_id"),
org=report.get("organization_name"),
)
)
report_db_id: int = row[0]
for policy in report.get("policies", []):
cur.execute(
"""
INSERT INTO smtp_tls_policy (
report_id, policy_domain, policy_type,
policy_strings, mx_host_patterns,
successful_session_count, failed_session_count
) VALUES (%s, %s, %s, %s, %s, %s, %s)
RETURNING id
""",
(
report_db_id,
policy.get("policy_domain"),
policy.get("policy_type"),
policy.get("policy_strings") or [],
policy.get("mx_host_patterns") or [],
policy.get("successful_session_count"),
policy.get("failed_session_count"),
),
)
# INSERT ... RETURNING always yields one row
policy_db_id: int = cur.fetchone()[0] # pyright: ignore[reportOptionalSubscript]
for detail in policy.get("failure_details", []):
cur.execute(
"""
INSERT INTO smtp_tls_failure_detail (
policy_id, result_type,
failed_session_count,
sending_mta_ip, receiving_ip,
receiving_mx_hostname, receiving_mx_helo,
additional_info_uri, failure_reason_code
) VALUES (
%s, %s, %s, %s, %s, %s, %s, %s, %s
)
""",
(
policy_db_id,
detail.get("result_type"),
detail.get("failed_session_count"),
detail.get("sending_mta_ip"),
detail.get("receiving_ip"),
detail.get("receiving_mx_hostname"),
detail.get("receiving_mx_helo"),
detail.get("additional_info_uri"),
detail.get("failure_reason_code"),
),
)
except AlreadySaved:
raise
except psycopg.DataError as exc:
raise PostgreSQLRejectedReport(
"PostgreSQL rejected a value of the SMTP TLS report from "
f"{sanitize_for_log(str(report.get('organization_name')))} with ID "
f"{sanitize_for_log(str(report.get('report_id')))}: "
f"{_rejection_detail(exc)}"
) from exc
except psycopg.Error as exc:
raise PostgreSQLError(str(exc)) from exc