From 771c6df0bdf5649633a97815c1fef34137317589 Mon Sep 17 00:00:00 2001 From: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:15:01 -0500 Subject: [PATCH] PostgreSQL connection follow-ups and the message_count mapping (#959) - Accept a URI whose host has an empty port (postgresql://db:/d? sslrootcert=/a@b): libpq parses it as no port, but _port_is_numeric() treated it as non-numeric and the early refusal rejected the string. - Close the connection when setting autocommit or registering the NUL-safe dumpers fails after psycopg.connect() succeeded, including on KeyboardInterrupt; an error from close() is dropped. - With a connection string, report a create_tables() failure with a fixed message (plus a well-formed SQLSTATE), since the server's text can quote a session setting from the string's options; with the separate settings keep the server's text, URL credentials masked. Both raise from None. The CLI now closes a PostgreSQL client whose create_tables() raised, as it is not yet in the clients to roll back. - Declare the Elasticsearch and OpenSearch aggregate documents' message_count and original_timespan_seconds as Long() instead of the bare Integer class, matching the long type dynamic mapping has always given them (the document classes' mappings are never installed). Correct the MAX_MESSAGE_COUNT comment, the _smtp_tls_count docstring and the GHSA-fxr7 changelog entry, which described an integer Elasticsearch/OpenSearch mapping of the counts that is not installed. Co-authored-by: Claude Fable 5.2 --- CHANGELOG.md | 5 +- docs/source/usage.md | 14 ++-- parsedmarc/__init__.py | 5 +- parsedmarc/cli.py | 11 ++- parsedmarc/constants.py | 11 +-- parsedmarc/elastic.py | 5 +- parsedmarc/opensearch.py | 5 +- parsedmarc/postgres.py | 56 ++++++++++++-- tests/test_cli.py | 46 ++++++++++++ tests/test_elastic.py | 46 ++++++++++++ tests/test_opensearch.py | 46 ++++++++++++ tests/test_postgres.py | 158 ++++++++++++++++++++++++++++++++++++++- 12 files changed, 375 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index add774ac..9fee84a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,8 @@ - **The seen-aggregate-report cache no longer keeps every distinct report in memory for the life of the process** ([GHSA-9fw6-mwqr-p43m](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-9fw6-mwqr-p43m), affects 11.2.2 and earlier). The cache that recognizes a duplicate aggregate report is meant to forget a report after one hour, but the `expiringdict` package it is built on removes an expired entry only when that same key is looked up again, and otherwise drops its oldest entry only once it holds `max_len` entries, which was 100 million. A report that is never received again is never looked up again, so in watch mode every distinct aggregate report stayed in memory until the process exited. The cache key was the reporter's name, the Report-ID and the policy domain as JSON, all three chosen by the sender with no length limit below the 32 MiB decompression cap: ten emails each carrying a 5.6 KB gzip attachment with a 5 MiB `` left 64 MiB held by the cache, and none of it was released two hours after the one-hour expiry. Expired entries are now removed each time a report is added to the cache, the cache holds at most 100,000 entries, and the key is a SHA-256 digest of the three fields, so its size no longer depends on the report. **Compatibility:** the cache is process-local and never saved, so nothing persisted changes; a report is still recognized as a duplicate for one hour. Reported by helixnebulatech. - **Reverse DNS lookups are limited to 120 seconds per aggregate report when they fail or are slow, and to 600 seconds in total** ([GHSA-rwfh-ww9c-jm9m](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-rwfh-ww9c-jm9m), affects 11.2.2 and earlier). Each `` of an aggregate report triggered one blocking reverse DNS (PTR) lookup of its source IP, one record after another, with no bound on the number of lookups or the time they took, and the sender chooses those IPs. An IP whose reverse zone never answers costs the whole DNS lifetime, `dns_timeout` times the number of nameservers: 2 s × 4 default nameservers = 8 s. Repeating one such IP did not help, because a result with no PTR whose AS domain is not in the reverse DNS map is deliberately kept out of the IP address cache, so the same IP was looked up again in every record. Measured against a nameserver that never answers, parsing took 2.2 s per record with one nameserver and 8.2 s per record with four, so a 1 MB report of about 3,500 records held up processing for hours, and no other message was processed meanwhile. Lookups that return no PTR, and lookups that take at least one second whether or not they return a name (a sender who controls the reverse zone could otherwise answer every query with a name just before the timeout), now share a budget of 120 seconds per report. Fast lookups that find a PTR are not counted toward it, so a large legitimate report keeps its enrichment, but every lookup counts toward a second budget of 600 seconds per report for all reverse DNS lookups, since a sender who controls the reverse zone could otherwise answer every query just under one second (5,000 distinct source IPs resolving in 100 ms each use 500 seconds). Once either budget is spent, the report's remaining records are enriched from the local IP database and reverse DNS map only, and one warning is logged: "Reverse DNS lookups for this report failed or were slow for 120 seconds; the remaining records are enriched from the local IP database and reverse DNS map only", or "Reverse DNS lookups for this report took more than 600 seconds; ..." with the same ending. Results produced after the limit are not cached, so those IPs are looked up normally in the next report. Each distinct source IP is also looked up at most once per report. Deployments with `offline = True` make no DNS queries and were never affected. Reported by helixnebulatech. -- **A report value that PostgreSQL rejects no longer holds back the whole batch** ([GHSA-fxr7-7h9q-v949](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-fxr7-7h9q-v949), affects the PostgreSQL output in 11.2.2 and earlier). The PostgreSQL output inserted report values into typed columns without checking them, and three values a sender controls were rejected by the server or the driver: an aggregate `` above 2,147,483,647 (`integer out of range` in the `INTEGER` `message_count` column), an SMTP TLS `sending-mta-ip` or `receiving-ip` that is not an IP address (`invalid input syntax for type inet`), and a NUL character in a failure report's sample, for example a `Subject:` of `=?utf-8?q?a=00b?=` (PostgreSQL cannot store NUL in `text` or `jsonb`). A fourth trigger, an SMTP TLS `start-datetime` PostgreSQL cannot read, has mostly been rejected by the parser since 11.2.2; the remainder (partial dates such as `2016`, which the parser still accepts and PostgreSQL may reject) now raise `PostgreSQLRejectedReport` like any other rejected value and no longer hold back the batch. Each rejection was recorded as an output error, which marks the whole batch as not saved, so none of the batch's messages was archived, including legitimate reports from other senders. In watch mode the batch was retried `max_unsaved_retries` times and then moved to `Archive/Unsaved`; in one-shot mode (cron) the retry counter restarts with every run, so the batch stayed in the inbox and its legitimate reports were sent again to every output that does not deduplicate on every run. An aggregate record whose count is outside 0 to 2,147,483,647 is now skipped with a "Could not parse record" warning (a report with only that one record becomes invalid, as with any other bad record), and an SMTP TLS session count outside that range makes the report invalid; the bound matches PostgreSQL's `INTEGER` count columns and the Elasticsearch and OpenSearch `integer` mapping of the SMTP TLS session counts, so it also closes the same rejection of an oversized session count in those outputs (the aggregate `message_count` is not in their mapping and is mapped dynamically as `long`). An SMTP TLS `sending-mta-ip` or `receiving-ip` that is not an IP address (or carries an IPv6 zone ID) is dropped from the failure details with a warning, which also keeps it out of the Elasticsearch and OpenSearch `ip` fields. An IPv6 zone ID (`fe80::1%eth0`), which the `inet` type cannot hold, is removed from an aggregate record's `source_ip` and a failure report's `Source-IP` (everything from the first `%`) with a warning naming the field, and the record or report is kept with the bare address. The PostgreSQL output replaces NUL with U+FFFD, the replacement character, in every text, text array and JSONB value it writes. Any other value PostgreSQL rejects (a `psycopg.DataError`) now raises the new `PostgreSQLRejectedReport`, a subclass of `PostgreSQLError` whose message names the report; the CLI logs it as an error and skips that report in the PostgreSQL output only, and the batch is saved and archived as usual. With `fail_on_output_error` set it is an output error that stops the run, as before. The in-memory `max_unsaved_retries` counter in one-shot mode is unchanged: the batches it still holds back are the ones an output could not save at all, such as an unreachable server, which the documentation describes as the intended safe direction. Reported by helixnebulatech. -- **The PostgreSQL output no longer sends the database password in plain text when an attacker declines TLS** ([GHSA-xvp7-7762-gx9g](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-xvp7-7762-gx9g), affects the PostgreSQL output in 11.2.2 and earlier). `PostgreSQLClient` connected without an `sslmode`, so libpq used its default, `prefer`: it asks the server for TLS and, when the answer is no, continues over plain TCP and answers whatever authentication request comes next. That answer arrives before any encryption, so an on-path attacker who declined TLS and then requested a clear-text password received the database password (CWE-319), and with it every stored report. The output now connects with `sslmode=require` by default, so a connection without TLS fails before the user name or password is sent. The new `[postgresql]` `sslmode` option (and the `sslmode` argument of `PostgreSQLClient`) sets any libpq mode, and `sslrootcert` names the CA certificate that `verify-ca` and `verify-full` check the server against. When `sslmode` is not set, an `sslmode` in `connection_string` or the `PGSSLMODE` environment variable is respected. A warning is logged at startup when a TCP host other than `localhost` or a loopback address is used with a mode that does not verify the server certificate (`require` with an existing `sslrootcert` file is not warned about, because libpq then verifies the certificate as `verify-ca` does). With a `connection_string`, nothing parsed from the string reaches the log or an error message: a connection failure is reported with a fixed message (plus the server's SQLSTATE code, when it sent one) instead of the driver's, and the startup warning names neither the server nor any other value from the string. libpq ends a URI's user name and password at the first `@` or `/`, so the rest of a password containing either becomes the host, port, database name, or a query parameter, which the warning and the driver's errors would otherwise repeat (`postgresql://u:SECRET/x@db/d` failed with `invalid integer value "SECRET" for connection option "port"`). The separate `host`, `port`, `user`, `password`, and `database` settings keep the driver's message, with URL credentials masked. As a convenience, a URI that libpq would parse that way (an unencoded `@` in its host list or database name, or in a query value before the first `/`, or an `@` after a host whose port is not a number) is refused with a fixed error before anything is logged or a connection is attempted; this recognizes the common mistake but is not what keeps the password out of the logs. Percent-encode `@` and `/` in a URI's user name or password (`%40`, `%2F`). A keyword/value connection string is not checked, because its password is a keyword of its own. Any other error while connecting, such as the `UnicodeEncodeError` psycopg raises with the whole connection string it built when the password holds a character it cannot encode, is replaced with a fixed message as well, and the CLI no longer chains the PostgreSQL client's error to the `RuntimeError` it raises. **Compatibility:** a server that offers no TLS at all now fails with libpq's "server does not support SSL, but SSL was required". Set `[postgresql] sslmode = disable` (or `prefer`) to allow it again, recommended only for a Unix socket or the loopback address. `require` still accepts any certificate, so a remote server should use `sslmode = verify-full` with `sslrootcert`. Reported by helixnebulatech. +- **A report value that PostgreSQL rejects no longer holds back the whole batch** ([GHSA-fxr7-7h9q-v949](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-fxr7-7h9q-v949), affects the PostgreSQL output in 11.2.2 and earlier). The PostgreSQL output inserted report values into typed columns without checking them, and three values a sender controls were rejected by the server or the driver: an aggregate `` above 2,147,483,647 (`integer out of range` in the `INTEGER` `message_count` column), an SMTP TLS `sending-mta-ip` or `receiving-ip` that is not an IP address (`invalid input syntax for type inet`), and a NUL character in a failure report's sample, for example a `Subject:` of `=?utf-8?q?a=00b?=` (PostgreSQL cannot store NUL in `text` or `jsonb`). A fourth trigger, an SMTP TLS `start-datetime` PostgreSQL cannot read, has mostly been rejected by the parser since 11.2.2; the remainder (partial dates such as `2016`, which the parser still accepts and PostgreSQL may reject) now raise `PostgreSQLRejectedReport` like any other rejected value and no longer hold back the batch. Each rejection was recorded as an output error, which marks the whole batch as not saved, so none of the batch's messages was archived, including legitimate reports from other senders. In watch mode the batch was retried `max_unsaved_retries` times and then moved to `Archive/Unsaved`; in one-shot mode (cron) the retry counter restarts with every run, so the batch stayed in the inbox and its legitimate reports were sent again to every output that does not deduplicate on every run. An aggregate record whose count is outside 0 to 2,147,483,647 is now skipped with a "Could not parse record" warning (a report with only that one record becomes invalid, as with any other bad record), and an SMTP TLS session count outside that range makes the report invalid; the bound matches PostgreSQL's `INTEGER` count columns (the Elasticsearch and OpenSearch outputs install no mapping for any count, so those clusters map each one dynamically as `long`, a range the bound also keeps every count within). An SMTP TLS `sending-mta-ip` or `receiving-ip` that is not an IP address (or carries an IPv6 zone ID) is dropped from the failure details with a warning, which also keeps it out of the Elasticsearch and OpenSearch `ip` fields. An IPv6 zone ID (`fe80::1%eth0`), which the `inet` type cannot hold, is removed from an aggregate record's `source_ip` and a failure report's `Source-IP` (everything from the first `%`) with a warning naming the field, and the record or report is kept with the bare address. The PostgreSQL output replaces NUL with U+FFFD, the replacement character, in every text, text array and JSONB value it writes. Any other value PostgreSQL rejects (a `psycopg.DataError`) now raises the new `PostgreSQLRejectedReport`, a subclass of `PostgreSQLError` whose message names the report; the CLI logs it as an error and skips that report in the PostgreSQL output only, and the batch is saved and archived as usual. With `fail_on_output_error` set it is an output error that stops the run, as before. The in-memory `max_unsaved_retries` counter in one-shot mode is unchanged: the batches it still holds back are the ones an output could not save at all, such as an unreachable server, which the documentation describes as the intended safe direction. Reported by helixnebulatech. +- **The PostgreSQL output no longer sends the database password in plain text when an attacker declines TLS** ([GHSA-xvp7-7762-gx9g](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-xvp7-7762-gx9g), affects the PostgreSQL output in 11.2.2 and earlier). `PostgreSQLClient` connected without an `sslmode`, so libpq used its default, `prefer`: it asks the server for TLS and, when the answer is no, continues over plain TCP and answers whatever authentication request comes next. That answer arrives before any encryption, so an on-path attacker who declined TLS and then requested a clear-text password received the database password (CWE-319), and with it every stored report. The output now connects with `sslmode=require` by default, so a connection without TLS fails before the user name or password is sent. The new `[postgresql]` `sslmode` option (and the `sslmode` argument of `PostgreSQLClient`) sets any libpq mode, and `sslrootcert` names the CA certificate that `verify-ca` and `verify-full` check the server against. When `sslmode` is not set, an `sslmode` in `connection_string` or the `PGSSLMODE` environment variable is respected. A warning is logged at startup when a TCP host other than `localhost` or a loopback address is used with a mode that does not verify the server certificate (`require` with an existing `sslrootcert` file is not warned about, because libpq then verifies the certificate as `verify-ca` does). With a `connection_string`, nothing parsed from the string reaches the log or an error message: a connection failure is reported with a fixed message (plus the server's SQLSTATE code, when it sent one) instead of the driver's, and the startup warning names neither the server nor any other value from the string. libpq ends a URI's user name and password at the first `@` or `/`, so the rest of a password containing either becomes the host, port, database name, or a query parameter, which the warning and the driver's errors would otherwise repeat (`postgresql://u:SECRET/x@db/d` failed with `invalid integer value "SECRET" for connection option "port"`). The separate `host`, `port`, `user`, `password`, and `database` settings keep the driver's message, with URL credentials masked. As a convenience, a URI that libpq would parse that way (an unencoded `@` in its host list or database name, or in a query value before the first `/`, or an `@` after a host whose port is neither empty nor a number) is refused with a fixed error before anything is logged or a connection is attempted; this recognizes the common mistake but is not what keeps the password out of the logs. Percent-encode `@` and `/` in a URI's user name or password (`%40`, `%2F`). A keyword/value connection string is not checked, because its password is a keyword of its own. Any other error while connecting, such as the `UnicodeEncodeError` psycopg raises with the whole connection string it built when the password holds a character it cannot encode, is replaced with a fixed message as well, and the CLI no longer chains the PostgreSQL client's error to the `RuntimeError` it raises. A table creation failure is likewise reported with a fixed message (plus the SQLSTATE) instead of the server's, which can quote a setting from the string's `options` (`tablespace "X" does not exist` for `-c default_tablespace=X`), and a connection whose setup or table creation fails is closed instead of left open. **Compatibility:** a server that offers no TLS at all now fails with libpq's "server does not support SSL, but SSL was required". Set `[postgresql] sslmode = disable` (or `prefer`) to allow it again, recommended only for a Unix socket or the loopback address. `require` still accepts any certificate, so a remote server should use `sslmode = verify-full` with `sslrootcert`. Reported by helixnebulatech. - **An Elasticsearch or OpenSearch password containing `/`, `?` or `#` no longer appears in the startup error** ([GHSA-74xv-rmpx-2pjm](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-74xv-rmpx-2pjm), affects 11.2.2 and earlier; an incomplete fix for [GHSA-hjr3-rx4c-24cv](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-hjr3-rx4c-24cv)). 11.2.0 masked the credentials in `hosts` URLs logged with `--debug`, but not in the error raised when the client cannot be built. When the password in a `hosts` URL contains an unencoded `/`, `?` or `#`, the URL cannot be parsed, and the client library's error quotes the whole URL (Elasticsearch) or the start of the password (OpenSearch). parsedmarc repeated that text at ERROR level, without `--debug`, on the console, in `log_file` and in any log pipeline: `Output client error: Elasticsearch: Could not parse URL 'https://dmarc:@es.example:9200'` and `Output client error: OpenSearch: Port could not be cast to integer value as ''`. The error was also chained to the library's, so a failed SIGHUP reload, which logs the full traceback, printed it a second time. A `hosts` value that cannot be parsed now fails with a fixed message that names no part of it: "Elasticsearch: could not build a client from the hosts setting (details withheld because they may include credentials; URL-encode special characters in the password, or use the separate user and password settings)", and likewise for OpenSearch. Any other Elasticsearch or OpenSearch startup error keeps its text with URL credentials masked, and neither is chained to the original error any more. Reported by helixnebulatech. - **The downloaded IP database is cached in a private per-user directory and checked before it is used** ([GHSA-5f56-8ph8-v9q5](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-5f56-8ph8-v9q5), affects 9.6.0 through 11.2.2). `load_ip_db()` cached the IPinfo Lite database it downloads at a fixed path in the shared system temporary directory, `/tmp/parsedmarc/ipinfo_lite.mmdb` on Linux. Another local user could create `/tmp/parsedmarc` first; parsedmarc accepted the existing directory, wrote the download through a symlink planted at `ipinfo_lite.mmdb.tmp`, and used whatever file was at `ipinfo_lite.mmdb` whenever the download was skipped or failed, which that user could force, with no check of its owner or content. That let them overwrite any file the parsedmarc user can write (its configuration, a Gmail `token.json`, an output file) with the download, or supply the MMDB file that decides every source IP's country and ASN and that `maxminddb` then parses. The database is now cached in `$XDG_CACHE_HOME/parsedmarc` or `~/.cache/parsedmarc` (`%LOCALAPPDATA%\parsedmarc\cache` on Windows), created with mode `0700` and used only if it is a real directory, not a symlink, that the user can write to and, on POSIX, owns and is not writable by its group or by others. Otherwise parsedmarc logs a warning and uses a new private temporary directory for that run, downloading the database again on every run, and never changes the permissions of a directory it did not create. The download is written to a new file created exclusively with `mkstemp()`, opened with `maxminddb` to check that it is a valid MMDB, and only then moved into place with an atomic `os.replace()`, which replaces a symlink rather than following it. A cached copy is used only if it is a regular file and, on POSIX, owned by the user; otherwise the bundled copy is used. The old `/tmp/parsedmarc/ipinfo_lite.mmdb` is no longer read, and can be removed by hand. Only multi-user hosts whose temporary directory is shared were exposed; the Docker image and systemd units with `PrivateTmp=` were not. Reported by helixnebulatech. - **One aggregate report too large for the Logs Ingestion API no longer stops the Log Analytics output from publishing the batch** ([GHSA-rg4h-58fm-8pc9](https://github.com/domainaware/parsedmarc/security/advisories/GHSA-rg4h-58fm-8pc9), affects the Log Analytics output in 11.2.2 and earlier). The Azure Monitor ingestion SDK packs whole reports into upload calls of up to 1 MiB and never splits a report, so a report whose JSON is larger than that is sent as a call of its own, and the Logs Ingestion API rejects any call over 1 MB. An aggregate report of a few thousand records is that large, while its gzipped XML is only a few kilobytes. The rejected upload raised an error before the failure and SMTP TLS streams were sent, and parsedmarc treated it as an output error: the whole batch, legitimate reports from other senders included, was retried and finally moved to the `Unsaved` folder. A report whose JSON is over 1,000,000 bytes is now left out with a warning that names it, and the other reports are uploaded. Each of the three streams is now published even when another fails; a failed upload is still reported as an output error, naming every stream that failed. Reported by helixnebulatech. @@ -27,6 +27,7 @@ - **One invalid report attachment no longer stops the rest of an mbox file from being parsed.** `parse_report_email()` raised a bare `ParserError` for a message whose aggregate XML or SMTP TLS JSON attachment was invalid (for example an aggregate report whose date range spans more than 366 days), and the mbox loop only skipped `InvalidDMARCReport`, so that one message aborted the whole file. It now raises `InvalidDMARCReport`, or `InvalidSMTPTLSReport` for an SMTP TLS attachment (both still `ParserError` subclasses), and the mbox loop skips both with a warning and parses the rest. A message that cannot be parsed at all still stops the file with a `ParserError`. Mailbox connections were not affected. Reported by mjatin2007. - **Unknown child elements under an aggregate record's `` and inside a `` are no longer copied into the parsed report** ([#948](https://github.com/domainaware/parsedmarc/issues/948)). `_parse_report_record()` copied both mappings whole, so an element that RFC 9990 Appendix A and RFC 7489 Appendix C do not define reached the JSON, Kafka, S3 and webhook outputs with whatever structure the report gave it. Only `header_from`, `envelope_from` and `envelope_to`, and a reason's `type` and `comment`, are kept. **Compatibility:** a consumer of those outputs that read such an extra key no longer receives it. +- **The Elasticsearch and OpenSearch aggregate document classes declare `message_count` and `original_timespan_seconds` as `long`.** Both were written `= Integer` without parentheses, which assigns the field class itself and declares no field, so reading either attribute from an `_AggregateReportDoc` returned the class instead of the stored value. They are now `Long()`, not `Integer()`, because that is the type the clusters already hold: parsedmarc does not install the document classes' mappings (the indexes and templates it creates declare only the date fields), so Elasticsearch and OpenSearch have always mapped both fields dynamically as `long`. The documents sent are unchanged, and existing and new indexes are unaffected. ## 11.2.2 diff --git a/docs/source/usage.md b/docs/source/usage.md index 7dab32e3..4738c62a 100644 --- a/docs/source/usage.md +++ b/docs/source/usage.md @@ -807,10 +807,12 @@ The full set of configuration options is: not put the password in the connection string: a malformed connection string can make the driver echo it back in an error message. With a connection string, parsedmarc withholds the - driver's message for a connection failure (keeping only the - server's SQLSTATE code, when there is one) and does not name the - server in its startup warning, because either could repeat part of - a password; the separate settings give full diagnostics. An `@` or + driver's message for a connection failure and the server's message + for a table creation failure (keeping only the server's SQLSTATE + code, when there is one) and does not name the server in its + startup warning, because any of them could repeat part of a + password or a setting from the string; the separate settings give + full diagnostics. An `@` or `/` in a URI's user name or password must be percent-encoded (`@` as `%40`, `/` as `%2F`): libpq ends the user name and password at the first `@` or `/`, so the rest of the password would be parsed @@ -818,8 +820,8 @@ The full set of configuration options is: convenience, a URI that libpq would parse that way is refused with a fixed error before anything is logged: one with an unencoded `@` in its host list or database name, or in a query value written before - the first `/`, or with an `@` after a host whose port is not a - number. An `@` in such a place is written `%40` (an abstract socket + the first `/`, or with an `@` after a host whose port is neither + empty nor a number. An `@` in such a place is written `%40` (an abstract socket host as `%40sock`, a database name `d@x` as `d%40x`); an `@` in a query value after a `/`, such as a certificate path, is accepted as written. A keyword/value connection string is not checked: its diff --git a/parsedmarc/__init__.py b/parsedmarc/__init__.py index 0e5b07ba..4e8722df 100644 --- a/parsedmarc/__init__.py +++ b/parsedmarc/__init__.py @@ -1168,9 +1168,8 @@ def _smtp_tls_count(value: Any, field: str) -> int | None: JSON number". ``null`` is tolerated as an unreported count. A JSON ``true``/``false`` is rejected even though ``bool`` subclasses ``int``. A count outside ``0..MAX_MESSAGE_COUNT`` is rejected because the - PostgreSQL, Elasticsearch and OpenSearch outputs store counts as 32-bit - integers (GHSA-fxr7-7h9q-v949). Raises ``InvalidSMTPTLSReport`` - otherwise. + PostgreSQL output stores counts as 32-bit integers + (GHSA-fxr7-7h9q-v949). Raises ``InvalidSMTPTLSReport`` otherwise. """ if value is None: return value diff --git a/parsedmarc/cli.py b/parsedmarc/cli.py index 2836646c..9b5ac1d0 100644 --- a/parsedmarc/cli.py +++ b/parsedmarc/cli.py @@ -4,6 +4,7 @@ """A CLI for parsing DMARC and SMTP TLS reports""" import atexit +import contextlib import functools import http.client import json @@ -2227,7 +2228,15 @@ def _build_output_clients(opts, clients, index_prefix_domain_map=None): sslmode=opts.postgresql_sslmode, sslrootcert=opts.postgresql_sslrootcert, ) - pg_client.create_tables() + try: + pg_client.create_tables() + except BaseException: + # Not yet in clients, so the rollback in _init_output_clients + # would not close its connection. An error from close() is + # dropped so the create_tables() failure is the one raised. + with contextlib.suppress(Exception): + pg_client.close() + raise clients["postgresql_client"] = pg_client # A PostgreSQLError's text is built by postgres.py: a fixed message, the # driver's text with URL credentials masked, or, with a connection diff --git a/parsedmarc/constants.py b/parsedmarc/constants.py index c5635c1b..3a699a4c 100644 --- a/parsedmarc/constants.py +++ b/parsedmarc/constants.py @@ -79,9 +79,10 @@ SLOW_REVERSE_DNS_SECONDS = 1.0 MAX_REVERSE_DNS_SECONDS_PER_REPORT = 600.0 # Largest message or session count a report may give (GHSA-fxr7-7h9q-v949). -# It matches the PostgreSQL output's INTEGER count columns and the -# Elasticsearch/OpenSearch integer mapping of the SMTP TLS session counts, -# all signed 32-bit, so a larger count was rejected by those outputs. (The -# aggregate message_count is not in the Elasticsearch/OpenSearch mapping, -# so it is mapped dynamically as long.) No real reporter approaches it. +# It matches the PostgreSQL output's INTEGER count columns, signed 32-bit, +# so a larger count was rejected by that output. (The Elasticsearch and +# OpenSearch outputs do not install their document classes' mappings, and +# the mappings they do install declare no counts, so the clusters map every +# count dynamically as long; the aggregate document class declares +# message_count as Long() to match.) No real reporter approaches it. MAX_MESSAGE_COUNT = 2**31 - 1 diff --git a/parsedmarc/elastic.py b/parsedmarc/elastic.py index e182c44c..e20f65c3 100644 --- a/parsedmarc/elastic.py +++ b/parsedmarc/elastic.py @@ -14,6 +14,7 @@ from elasticsearch.dsl import ( Integer, Ip, Keyword, + Long, Nested, Object, Q, @@ -413,7 +414,7 @@ class _AggregateReportDoc(Document): date_begin = Date() date_end = Date() normalized_timespan = Boolean() - original_timespan_seconds = Integer + original_timespan_seconds = Long() errors = Text() published_policy = Object(_PublishedPolicy) source_ip_address = Ip() @@ -425,7 +426,7 @@ class _AggregateReportDoc(Document): source_asn = Integer() source_as_name = Text() source_as_domain = Text() - message_count = Integer + message_count = Long() disposition = Text() dkim_aligned = Boolean() spf_aligned = Boolean() diff --git a/parsedmarc/opensearch.py b/parsedmarc/opensearch.py index 7c5905cb..2c9539b6 100644 --- a/parsedmarc/opensearch.py +++ b/parsedmarc/opensearch.py @@ -15,6 +15,7 @@ from opensearchpy import ( Integer, Ip, Keyword, + Long, Nested, NotFoundError, Object, @@ -376,7 +377,7 @@ class _AggregateReportDoc(Document): date_begin = Date() date_end = Date() normalized_timespan = Boolean() - original_timespan_seconds = Integer + original_timespan_seconds = Long() errors = Text() published_policy = Object(_PublishedPolicy) source_ip_address = Ip() @@ -388,7 +389,7 @@ class _AggregateReportDoc(Document): source_asn = Integer() source_as_name = Text() source_as_domain = Text() - message_count = Integer + message_count = Long() disposition = Text() dkim_aligned = Boolean() spf_aligned = Boolean() diff --git a/parsedmarc/postgres.py b/parsedmarc/postgres.py index 23a9480c..7f5dd778 100644 --- a/parsedmarc/postgres.py +++ b/parsedmarc/postgres.py @@ -2,6 +2,7 @@ from __future__ import annotations +import contextlib import ipaddress import os import re @@ -77,6 +78,17 @@ _WITHHELD_CONNECTION_FAILURE_ERROR = ( "message)" ) +# With a connection string, a table creation failure is reported with this +# fixed text and not the server's: the server's message can quote a session +# setting taken from the string's options (for "-c default_tablespace=X", +# 'tablespace "X" does not exist'). See PostgreSQLClient.create_tables(). +_WITHHELD_TABLE_CREATION_ERROR = ( + "PostgreSQL table creation failed (details withheld because the " + "connection string may include the password; use the separate host, " + "port, user, password and database settings to see the server's " + "message)" +) + # With the separate settings, an error that is not the driver's own (a # password the driver cannot encode, say) is reported with this fixed text: # such an error quotes the value it could not handle, or the whole @@ -150,15 +162,16 @@ def _is_uri(connection_string: str) -> bool: def _port_is_numeric(entry: str) -> bool: """Return whether one ``host[:port]`` entry of a URI authority has no - port or an all-digit one. A bracketed IPv6 address is skipped first, so - its colons are not taken for a port separator.""" + port, an empty one (``db:``, which libpq parses as no port) or an + all-digit one. A bracketed IPv6 address is skipped first, so its colons + are not taken for a port separator.""" entry = entry.strip() if entry.startswith("["): entry = entry.partition("]")[2] if ":" not in entry: return True port = entry.rpartition(":")[2] - return port.isascii() and port.isdigit() + return not port or (port.isascii() and port.isdigit()) def _has_unencoded_at(connection_string: str) -> bool: @@ -187,8 +200,9 @@ def _has_unencoded_at(connection_string: str) -> bool: every component. An ``@`` in a query value after a ``/`` (``?sslrootcert=/home/a@b/ca.pem``) is not refused, because nothing tells it apart from a legitimate one. For the same reason two shapes - are not recognized: an all-digit password before a ``/``, followed by - an ``@`` only in a query value (``u:12345/x?sslrootcert=a@b``), and a + are not recognized: an empty or all-digit password before a ``/``, + followed by an ``@`` only in a query value + (``u:12345/x?sslrootcert=a@b``, ``u:/x?sslrootcert=a@b``), and a password with an ``@``, then a ``?``, then a ``/`` (``u:s3c@pw?sslmode=SECRET/x@db``). @@ -587,9 +601,14 @@ class PostgreSQLClient: (a password the driver cannot encode, a host name the resolver rejects) is replaced with a fixed message, since its text can quote the password or the whole connection - string. Either way the original error is not chained. + string. Either way the original error is not chained. A + connection opened before its setup failed is closed. """ logger.debug("Connecting to PostgreSQL") + # A connection that fails its setup (autocommit, the dumpers) is + # closed before the error propagates; nothing else holds it. + opened: psycopg.Connection | None = None + configured = False try: # Parse the connection string first, on every path (the sslmode # lookup below skips it when sslmode is set), so a malformed @@ -608,8 +627,10 @@ class PostgreSQLClient: dbname=self._database, **ssl_kwargs, ) + opened = conn conn.autocommit = False _register_nul_safe_dumpers(conn) + configured = True except PostgreSQLError: # Already a fixed, secret-free message (_connection_string_params). raise @@ -650,6 +671,12 @@ class PostgreSQLClient: if self._connection_string: raise PostgreSQLError(_WITHHELD_CONNECTION_FAILURE_ERROR) from None raise PostgreSQLError(_WITHHELD_CONNECTION_SETTING_ERROR) from None + finally: + # Also on KeyboardInterrupt and SystemExit. An error from + # close() is dropped so the original failure is the one raised. + if opened is not None and not configured: + with contextlib.suppress(Exception): + opened.close() self._conn = conn return conn @@ -684,7 +711,13 @@ class PostgreSQLClient: This method is idempotent and safe to call on every startup. Raises: - PostgreSQLError: If table creation fails. + PostgreSQLError: If table creation fails. With a connection + string, the server's message is withheld (only a + well-formed SQLSTATE is kept): it can quote a session + setting from the string's ``options``, such as a + ``default_tablespace``. With the separate arguments, it is + passed on with URL credentials masked. Either way the + original error is not chained. """ conn = self._ensure_connected() ddl_statements: list[LiteralString] = [ @@ -925,7 +958,14 @@ class PostgreSQLClient: cur.execute(stmt) logger.debug("PostgreSQL tables verified / created") except psycopg.Error as exc: - raise PostgreSQLError(str(exc)) from exc + # Raised from None, as in _connect(): a chain would print the + # server's own text wherever the error is logged with its + # traceback. + if self._connection_string: + raise PostgreSQLError( + _WITHHELD_TABLE_CREATION_ERROR + _sqlstate_suffix(exc) + ) from None + raise PostgreSQLError(redact_url_credentials(str(exc))) from None def save_aggregate_report_to_postgresql(self, report: dict) -> None: """Saves a parsed aggregate DMARC report to PostgreSQL. diff --git a/tests/test_cli.py b/tests/test_cli.py index 5dc564b1..4018866b 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -6062,6 +6062,52 @@ class TestPostgreSQLClientBuildErrorsDoNotLeakCredentials(unittest.TestCase): ) self._assert_clean(exc, trace, output) + def test_client_is_closed_when_create_tables_fails(self): + """Follow-up to GHSA-xvp7-7762-gx9g: a client whose create_tables() + raised was not yet in ``clients``, so the rollback did not close + it and its connection stayed open. It is now closed once (an + error from close() does not replace the original), and the + RuntimeError carries no chain.""" + for close_error in (None, OSError("close failed SECRETX")): + with self.subTest(close_error=close_error): + opts = _output_client_opts(postgresql_host="db.invalid") + client = MagicMock() + client.create_tables.side_effect = ( + parsedmarc.cli.postgres.PostgreSQLError( + "PostgreSQL table creation failed" + ) + ) + client.close.side_effect = close_error + with ( + patch.object(parsedmarc.cli.postgres, "psycopg", MagicMock()), + patch.object( + parsedmarc.cli.postgres, + "PostgreSQLClient", + return_value=client, + ), + ): + exc, trace, output = self._init_error(opts) + client.close.assert_called_once_with() + self.assertEqual( + str(exc), "PostgreSQL: PostgreSQL table creation failed" + ) + self._assert_clean(exc, trace, output) + + def test_client_is_kept_open_when_create_tables_succeeds(self): + """A client whose tables were created is returned, not closed.""" + opts = _output_client_opts(postgresql_host="db.invalid") + client = MagicMock() + with ( + patch.object(parsedmarc.cli.postgres, "psycopg", MagicMock()), + patch.object( + parsedmarc.cli.postgres, "PostgreSQLClient", return_value=client + ), + ): + clients = parsedmarc.cli._init_output_clients(opts) + self.assertIs(clients["postgresql_client"], client) + client.create_tables.assert_called_once_with() + client.close.assert_not_called() + class TestHttpxLogRedaction(unittest.TestCase): """GHSA-hjr3-rx4c-24cv: httpx logs the full request URL at INFO.""" diff --git a/tests/test_elastic.py b/tests/test_elastic.py index 49dc3be9..588f1705 100644 --- a/tests/test_elastic.py +++ b/tests/test_elastic.py @@ -1078,6 +1078,52 @@ class TestAggregateDocCombinedResults(unittest.TestCase): # --------------------------------------------------------------------------- +class TestAggregateDocCountFields(unittest.TestCase): + """``message_count`` and ``original_timespan_seconds`` were declared + ``= Integer`` without parentheses, which binds the field class and + declares no field: they were missing from the document mapping, and + reading either attribute returned the class instead of the value. They + are ``Long()``, the type dynamic mapping has always given them in the + cluster (parsedmarc does not install the document classes' mappings, + and the mappings it does install declare no counts).""" + + def test_count_fields_are_mapped_as_long(self): + properties = elastic_module._AggregateReportDoc._doc_type.mapping.to_dict()[ + "properties" + ] + for name in ("message_count", "original_timespan_seconds"): + with self.subTest(field=name): + self.assertEqual(properties[name], {"type": "long"}) + + def test_count_fields_read_back_as_values(self): + doc = elastic_module._AggregateReportDoc( + message_count=3, original_timespan_seconds=86400 + ) + self.assertEqual(doc.message_count, 3) + self.assertEqual(doc.original_timespan_seconds, 86400) + body = doc.to_dict() + self.assertEqual(body["message_count"], 3) + self.assertEqual(body["original_timespan_seconds"], 86400) + + def test_no_document_class_binds_a_bare_field_class(self): + """Every field on every document class is a field instance; a + class attribute that is a elasticsearch.dsl Field subclass itself is a + declaration missing its parentheses.""" + from elasticsearch.dsl import Field + + checked = 0 + for owner in vars(elastic_module).values(): + if not (isinstance(owner, type) and hasattr(owner, "_doc_type")): + continue + checked += 1 + for name, value in vars(owner).items(): + with self.subTest(cls=owner.__name__, field=name): + self.assertFalse( + isinstance(value, type) and issubclass(value, Field) + ) + self.assertGreater(checked, 5) + + class TestSaveFailureReport(unittest.TestCase): def test_save_emits_one_document(self): with ( diff --git a/tests/test_opensearch.py b/tests/test_opensearch.py index f92637ab..9e611fbd 100644 --- a/tests/test_opensearch.py +++ b/tests/test_opensearch.py @@ -1108,6 +1108,52 @@ class TestAggregateDocCombinedResults(unittest.TestCase): # --------------------------------------------------------------------------- +class TestAggregateDocCountFields(unittest.TestCase): + """``message_count`` and ``original_timespan_seconds`` were declared + ``= Integer`` without parentheses, which binds the field class and + declares no field: they were missing from the document mapping, and + reading either attribute returned the class instead of the value. They + are ``Long()``, the type dynamic mapping has always given them in the + cluster (parsedmarc does not install the document classes' mappings, + and the mappings it does install declare no counts).""" + + def test_count_fields_are_mapped_as_long(self): + properties = opensearch_module._AggregateReportDoc._doc_type.mapping.to_dict()[ + "properties" + ] + for name in ("message_count", "original_timespan_seconds"): + with self.subTest(field=name): + self.assertEqual(properties[name], {"type": "long"}) + + def test_count_fields_read_back_as_values(self): + doc = opensearch_module._AggregateReportDoc( + message_count=3, original_timespan_seconds=86400 + ) + self.assertEqual(doc.message_count, 3) + self.assertEqual(doc.original_timespan_seconds, 86400) + body = doc.to_dict() + self.assertEqual(body["message_count"], 3) + self.assertEqual(body["original_timespan_seconds"], 86400) + + def test_no_document_class_binds_a_bare_field_class(self): + """Every field on every document class is a field instance; a + class attribute that is a opensearchpy Field subclass itself is a + declaration missing its parentheses.""" + from opensearchpy.helpers.field import Field + + checked = 0 + for owner in vars(opensearch_module).values(): + if not (isinstance(owner, type) and hasattr(owner, "_doc_type")): + continue + checked += 1 + for name, value in vars(owner).items(): + with self.subTest(cls=owner.__name__, field=name): + self.assertFalse( + isinstance(value, type) and issubclass(value, Field) + ) + self.assertGreater(checked, 5) + + class TestSaveFailureReport(unittest.TestCase): def test_save_emits_one_document(self): with ( diff --git a/tests/test_postgres.py b/tests/test_postgres.py index 4378f492..6d9bc3cb 100644 --- a/tests/test_postgres.py +++ b/tests/test_postgres.py @@ -1531,11 +1531,13 @@ _AT_RULE_CASES = ( ("u/x:SECRETX@db.invalid/d", True, "SECRETX"), ("u:/SECRETX@db.invalid/d", True, "SECRETX"), ("u:12345/x@db.invalid/d", True, "12345"), - # Not recognized, because each reads like a legitimate string: an - # all-digit password before a "/", with the "@" only in a query value - # (host "u", port 12345), and an "@", "?" and "/" in a password (host - # "pw" and an sslmode after a "?", like a certificate path after one). + # Not recognized, because each reads like a legitimate string: an empty + # or all-digit password before a "/", with the "@" only in a query + # value (host "u", port 12345 or none), and an "@", "?" and "/" in a + # password (host "pw" and an sslmode after a "?", like a certificate + # path after one). ("u:12345/x?sslrootcert=a@b", False, "12345"), + ("u:/x?sslrootcert=SECRET@b", False, "SECRET"), ("u:s3c@pw?sslmode=SECRET/x@db", False, "SECRET"), ("u:p%40w@db/d", False, None), ("u@db/d", False, None), @@ -1549,6 +1551,10 @@ _AT_RULE_CASES = ( ("u:p@db.invalid/d%40x", False, None), ("[::1]:5432/d", False, None), ("[::1]:5432/d?sslrootcert=/a@b/c", False, None), + # An empty port is no port to libpq (host "db", sslrootcert "/a@b"). + ("db:/d?sslrootcert=/a@b", False, None), + ("h1:,h2:5433/d?sslrootcert=/a@b", False, None), + ("[::1]:/d?sslrootcert=/a@b", False, None), ("u:p@[::1]:5432/d", False, None), ("u:pa?x@db/d", False, None), ("u:p@h1:5432,h2:5433/d", False, None), @@ -1803,6 +1809,150 @@ class TestConnectionStringDetailsWithheld(unittest.TestCase): with self.assertRaises(KeyboardInterrupt): PostgreSQLClient(host="db.invalid", sslmode="verify-full") + def _setup_failure(self, failure, stage, close_error=None): + """Construct a client whose connect succeeds and whose *stage* + (``autocommit`` or ``dumpers``) raises *failure*; return (the + exception raised, the connection mock). For an error made by + ``_driver_error()``, ``psycopg.Error`` is its base class, so it + takes the driver branch; any other error is not the driver's.""" + with contextlib.ExitStack() as stack: + mock_psycopg = stack.enter_context(patch("parsedmarc.postgres.psycopg")) + stack.enter_context(_without_libpq_env()) + if hasattr(type(failure), "sqlstate"): + mock_psycopg.Error = type(failure).__mro__[1] + else: + mock_psycopg.Error = type("Error", (Exception,), {}) + mock_psycopg.ProgrammingError = type( + "ProgrammingError", (mock_psycopg.Error,), {} + ) + conn = MagicMock() + if close_error is not None: + conn.close.side_effect = close_error + if stage == "autocommit": + type(conn).autocommit = property( + lambda self: False, + lambda self, value: (_ for _ in ()).throw(failure), + ) + else: + conn.adapters.register_dumper.side_effect = failure + mock_psycopg.connect.return_value = conn + with self.assertRaises(BaseException) as ctx: + PostgreSQLClient( + connection_string="host=db.invalid password=SECRETX", + sslmode="verify-full", + ) + return ctx.exception, conn + + def test_connection_is_closed_when_its_setup_fails(self): + """Follow-up to GHSA-xvp7-7762-gx9g: when psycopg.connect succeeded + but setting autocommit or registering the dumpers raised, the open + connection was dropped without close(), so the server kept the + session until the process exited. It is now closed once, and the + raised error is still the fixed message with no chain.""" + withheld = ( + "PostgreSQL connection failed (details withheld because the " + "connection string may include the password; use the " + "separate host, port, user, password and database settings " + "to see the driver's message)" + ) + for stage in ("autocommit", "dumpers"): + for failure, message in ( + (ValueError("SECRETX"), withheld), + ( + self._driver_error("SECRETX", "25001"), + withheld + " [SQLSTATE 25001]", + ), + ): + with self.subTest(stage=stage, failure=type(failure).__name__): + exc, conn = self._setup_failure(failure, stage) + conn.close.assert_called_once_with() + self.assertIsInstance(exc, PostgreSQLError) + self.assertEqual(str(exc), message) + self._assert_no_secret_anywhere(exc, []) + + def test_close_error_after_a_setup_failure_is_dropped(self): + """An error from close() does not replace the setup failure.""" + exc, conn = self._setup_failure( + ValueError("SECRETX"), "dumpers", close_error=OSError("close SECRETX") + ) + conn.close.assert_called_once_with() + self.assertIsInstance(exc, PostgreSQLError) + self._assert_no_secret_anywhere(exc, []) + + def test_connection_is_closed_on_keyboard_interrupt_during_setup(self): + """A Ctrl-C during the setup propagates as KeyboardInterrupt and the + connection is still closed.""" + exc, conn = self._setup_failure(KeyboardInterrupt(), "autocommit") + self.assertIsInstance(exc, KeyboardInterrupt) + conn.close.assert_called_once_with() + + def test_connection_is_not_closed_when_setup_succeeds(self): + """A connection whose setup succeeded is kept open as the client's.""" + with patch("parsedmarc.postgres.psycopg") as mock_psycopg, _without_libpq_env(): + mock_psycopg.Error = type("Error", (Exception,), {}) + conn = MagicMock() + mock_psycopg.connect.return_value = conn + client = PostgreSQLClient( + connection_string="host=db.invalid password=p", sslmode="verify-full" + ) + conn.close.assert_not_called() + self.assertIs(client._conn, conn) + + def _create_tables_error(self, client_kwargs, driver_error): + """Construct a client, make its DDL raise *driver_error*, and return + (the PostgreSQLError create_tables() raised, log lines).""" + with contextlib.ExitStack() as stack: + mock_psycopg = stack.enter_context(patch("parsedmarc.postgres.psycopg")) + stack.enter_context(_without_libpq_env()) + logs = stack.enter_context(_captured_log_text()) + mock_psycopg.Error = type(driver_error).__mro__[1] + conn = MagicMock() + conn.closed = False + mock_psycopg.connect.return_value = conn + client = PostgreSQLClient(**client_kwargs, sslmode="verify-full") + cur = _mock_cursor(conn, []) + cur.execute.side_effect = driver_error + with self.assertRaises(PostgreSQLError) as ctx: + client.create_tables() + return ctx.exception, logs + + def test_create_tables_error_withheld_with_a_connection_string(self): + """Follow-up to GHSA-xvp7-7762-gx9g: create_tables() raised + PostgreSQLError(str(exc)) from exc, and a server error can quote a + session setting from the string's options (``tablespace "X" does + not exist`` for ``-c default_tablespace=X``). With a connection + string the message is now fixed, with a well-formed SQLSTATE and + no chain.""" + dsn = "host=db.invalid password=pw options='-c default_tablespace=SECRETX'" + for sqlstate, suffix in (("42704", " [SQLSTATE 42704]"), ("bad", "")): + with self.subTest(sqlstate=sqlstate): + exc, logs = self._create_tables_error( + {"connection_string": dsn}, + self._driver_error('tablespace "SECRETX" does not exist', sqlstate), + ) + self.assertEqual( + str(exc), + "PostgreSQL table creation failed (details withheld because " + "the connection string may include the password; use the " + "separate host, port, user, password and database settings " + "to see the server's message)" + suffix, + ) + self._assert_no_secret_anywhere(exc, logs) + + def test_create_tables_error_kept_with_the_separate_settings(self): + """With the separate settings the server's message is passed on, + URL credentials masked, and still not chained.""" + exc, _ = self._create_tables_error( + {"host": "db.invalid", "user": "u", "password": "pw"}, + self._driver_error( + 'tablespace "t" does not exist (postgresql://u:SECRETX@db/d)', "42704" + ), + ) + self.assertEqual( + str(exc), 'tablespace "t" does not exist (postgresql://u:***@db/d)' + ) + self._assert_no_secret_anywhere(exc, []) + @unittest.skipUnless(PSYCOPG_INSTALLED, "psycopg is not installed") def test_real_psycopg_surrogate_password_is_withheld(self): """With the real driver, a password holding a lone surrogate makes