diff --git a/CHANGELOG.md b/CHANGELOG.md index 415c780e..9a3c2a09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +### New features + +- **DNS over HTTPS (DoH) and DNS over TLS (DoT) are now supported through the existing `nameservers` option** ([#880](https://github.com/domainaware/parsedmarc/issues/880)). No new configuration option is involved: each entry in the comma-separated list now picks its own transport, and the forms can be mixed in one list. An IP address means plain DNS over UDP and TCP port 53, exactly as before; an `https://` URL means DoH; and `tls://ip[:port][#hostname]` means DoT, where the port defaults to 853 and the optional `#hostname` supplies the TLS certificate identity (SNI) to verify the server against, matching systemd-resolved's syntax. DoH queries go through a per-process `httpx` client with its environment trust left at the default, so they honor the standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` variables — the motivating case, a corporate network that blocks outbound DNS but offers an HTTP proxy, where the proxy also resolves the DoH server's own hostname, so no access to UDP port 53 is needed at all — as well as `SSL_CERT_FILE`/`SSL_CERT_DIR`, for a private CA in front of a TLS-inspecting proxy. dnspython's own DoH support cannot do this, because it builds its HTTP client with a custom transport, which is precisely the condition under which httpx ignores proxy environment variables. DoT connections are made directly to TCP port 853 and do not use a proxy. A `nameservers` list of plain IP addresses — including the default — behaves exactly as it did before, and the pre-flight DNS check at startup now exercises whichever transports are configured, so a bad DoH or DoT entry is reported before any mailbox is opened. The `dnspython` requirement is now `dnspython[doh]>=2.7.0`. + ### Bug fixes - **Failure report MIME parts are now decoded according to their `Content-Transfer-Encoding`** ([#882](https://github.com/domainaware/parsedmarc/issues/882)). `parse_report_email()` read every part's payload without asking the standard library to decode it, so any transfer encoding was left in place. Two things went wrong as a result. A quoted-printable `text/rfc822-headers` sample part kept its RFC 2045 §6.7 soft line breaks — a `=` followed by a line ending, with none of the RFC 5322 folding whitespace a header parser needs — which split long headers mid-value; the sample's `From` header became unparseable, and since the report itself carried no `Reported-Domain` field, the whole failure report was discarded with `TypeError: 'NoneType' object is not subscriptable`. Separately, a quoted-printable `message/feedback-report` part parsed "successfully" with silently corrupted field values, such as an `authentication_results` of `mx.example.com; dmarc=3Dfail`. Decoding is applied only to the `message/feedback-report` part and to the sample part; all other branches (attachments, `application/tlsrpt+*`, `text/plain`) still receive the raw payload, since they decode base64 and sniff zip/gzip magic themselves. Parts declaring no transfer encoding, or a 7bit/8bit one, are likewise left alone: there is nothing to undo, and running such a part through the standard library's decoder would round-trip its already-correct text through `raw-unicode-escape` bytes and replace every non-ASCII character with U+FFFD. Note that a composite part carrying a real transfer encoding is illegal per RFC 2045 §6.4, but reporters send them anyway, and the standard library nests the still-encoded text as a child message rather than decoding it — so those parts are decoded by hand. That nested parse needs one repair first: a soft line break splits a long field onto a continuation line with no colon and no leading whitespace, so the parser treats the remainder as a message body and re-serializing inserts a blank line the encoded text never had, which would otherwise truncate the value at its first split. Any unexpected failure in the new decoding step falls back to the previous behavior, so no report that parsed before can start failing because of it. diff --git a/docs/source/usage.md b/docs/source/usage.md index 774985b3..2223f34d 100644 --- a/docs/source/usage.md +++ b/docs/source/usage.md @@ -39,7 +39,8 @@ options: --smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME filename for the SMTP TLS CSV output file -n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...] - nameservers to query + nameservers to query: IP addresses, https:// URLs (DNS over HTTPS), and/or + tls://ip[:port][#hostname] (DNS over TLS) -t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT number of seconds to wait for an answer from DNS (default: 2.0) --dns-retries DNS_RETRIES @@ -178,10 +179,15 @@ The full set of configuration options are: - `local_psl_overrides_path` - Overrides the default local file path to use for the PSL overrides list - `psl_overrides_url` - Overrides the default download URL for the PSL overrides list - `nameservers` - str: A comma separated list of - DNS resolvers (Default: `[Cloudflare's public resolvers]`) + DNS resolvers (Default: `[Cloudflare's public resolvers]`). Each entry + is an IP address (DNS over UDP/TCP port 53), an `https://` URL + (DNS over HTTPS), or `tls://ip[:port][#hostname]` (DNS over TLS) — + see [Encrypted DNS](#encrypted-dns) - `dns_test_address` - str: a dummy address used for DNS pre-flight checks (Default: 1.1.1.1) - `dns_timeout` - float: DNS timeout period + - `dns_retries` - int: Number of times to retry a DNS query after a + timeout or other transient error (Default: 0) - `debug` - bool: Print debugging messages - `silent` - bool: Only print errors (Default: `True`) - `fail_on_output_error` - bool: Exit with a non-zero status code if @@ -810,7 +816,8 @@ setting. By default, `parsedmarc` uses reliable than Google, Cisco OpenDNS, or even most local resolvers. The `nameservers` option should only be used if your network -blocks DNS requests to outside resolvers. +blocks DNS requests to outside resolvers, or blocks plain DNS +entirely — see [Encrypted DNS](#encrypted-dns). ::: :::{note} @@ -870,6 +877,52 @@ PUT _cluster/settings Increasing this value increases resource usage. ::: +### Encrypted DNS + +Every entry in the `nameservers` list picks its own transport: + +- An IP address — plain DNS over UDP and TCP port 53, the default and the + behavior of every earlier release. +- An `https://` URL — DNS over HTTPS (DoH). +- `tls://ip[:port][#hostname]` — DNS over TLS (DoT). The port defaults to + 853, and the optional `#hostname` names the TLS certificate identity of + the server (SNI), matching systemd-resolved's syntax. The host itself must + be an IP address, and an IPv6 address must be wrapped in brackets, so that + its colons cannot be mistaken for the port separator — + `tls://[2620:fe::fe]#dns.quad9.net`. + +Forms can be mixed, and are tried in the order given: + +```ini +[general] +nameservers = https://cloudflare-dns.com/dns-query, tls://9.9.9.9#dns.quad9.net +``` + +On a network where outbound DNS is blocked but an HTTP proxy is available, +configure DoH nameservers and set the standard proxy environment variables +([issue #880](https://github.com/domainaware/parsedmarc/issues/880)): + +```bash +export HTTPS_PROXY=http://proxy.example.net:3128 +export NO_PROXY=elasticsearch.example.net,127.0.0.1 +``` + +parsedmarc's DoH queries honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`, +and the proxy resolves the DoH server's own hostname on parsedmarc's behalf, +so such a deployment needs no access to UDP port 53 at all. + +If the proxy inspects TLS, point the standard `SSL_CERT_FILE` environment +variable at your organization's CA bundle so its certificate is trusted: + +```bash +export SSL_CERT_FILE=/etc/ssl/certs/corporate-ca.pem +``` + +:::{note} +parsedmarc's DoT connections are made directly to TCP port 853 and do not +use a proxy. Use DoH on a proxy-only network. +::: + ### Mailbox messages are only archived once the reports are saved parsedmarc processes a mailbox in batches of `batch_size` messages. Each diff --git a/parsedmarc/cli.py b/parsedmarc/cli.py index 1d2d394f..329ef6a1 100644 --- a/parsedmarc/cli.py +++ b/parsedmarc/cli.py @@ -2372,7 +2372,11 @@ def _main(): default="smtp_tls.csv", ) arg_parser.add_argument( - "-n", "--nameservers", nargs="+", help="nameservers to query" + "-n", + "--nameservers", + nargs="+", + help="nameservers to query: IP addresses, https:// URLs (DNS over " + "HTTPS), and/or tls://ip[:port][#hostname] (DNS over TLS)", ) arg_parser.add_argument( "-t", diff --git a/parsedmarc/utils.py b/parsedmarc/utils.py index 1737a141..117c9507 100644 --- a/parsedmarc/utils.py +++ b/parsedmarc/utils.py @@ -18,6 +18,7 @@ import subprocess import tempfile from datetime import datetime, timedelta, timezone from typing import TypedDict, cast +from urllib.parse import urlsplit import mailparser from expiringdict import ExpiringDict @@ -26,6 +27,10 @@ from importlib.resources import files import dns.exception +import dns.inet +import dns.message +import dns.nameserver +import dns.query import dns.resolver import dns.reversename import httpx @@ -51,6 +56,11 @@ _RETRYABLE_DNS_ERRORS = ( OSError, ) +# The process-wide httpx client used for DNS over HTTPS queries, and the PID +# it was created under. See _get_doh_session(). +_DOH_SESSION: httpx.Client | None = None +_DOH_SESSION_PID: int | None = None + parenthesis_regex = re.compile(r"\s*\(.*\)\s*") null_file = subprocess.DEVNULL @@ -196,6 +206,170 @@ def get_base_domain(domain: str) -> str | None: return publicsuffix +def _get_doh_session() -> httpx.Client: + """ + Returns the shared ``httpx.Client`` used for DNS over HTTPS queries. + + The client is created on first use and reused afterwards, so DoH queries + share TLS connections instead of renegotiating one per lookup. It is + deliberately never closed: like the module's other shared state, it lives + for the life of the process. + + The client is rebuilt when the current PID differs from the one it was + created under, because a ``fork()``-based worker pool (``n_procs``) would + otherwise inherit — and concurrently use — the parent's sockets. + + ``httpx.Client`` defaults are what make this work behind a corporate + proxy: ``trust_env=True`` honors ``HTTP_PROXY``/``HTTPS_PROXY``/ + ``NO_PROXY`` and ``SSL_CERT_FILE``/``SSL_CERT_DIR``, and ``verify=True`` + keeps certificate verification on. Neither is overridden here. + + Returns: + httpx.Client: The shared DoH client for this process + """ + global _DOH_SESSION, _DOH_SESSION_PID + pid = os.getpid() + if _DOH_SESSION is None or _DOH_SESSION_PID != pid: + _DOH_SESSION = httpx.Client(http1=True, http2=True) + _DOH_SESSION_PID = pid + return _DOH_SESSION + + +class _SessionDoHNameserver(dns.nameserver.DoHNameserver): + """ + A DNS over HTTPS nameserver that queries through a shared ``httpx`` + client. + + dnspython's stock ``DoHNameserver`` calls ``dns.query.https()`` without a + ``session``, which makes that function build an ``httpx.Client`` with its + own custom transport — and httpx only reads proxy environment variables + when no transport is supplied (``allow_env_proxies = trust_env and + transport is None``). Stock DoH therefore ignores ``HTTPS_PROXY`` + entirely — and a proxy is the only way out of the networks this exists + for (https://github.com/domainaware/parsedmarc/issues/880). + + Passing our own session instead gets environment proxies, environment CA + configuration (``SSL_CERT_FILE``), and connection reuse across queries. + ``bootstrap_address`` is deliberately not passed: with a session, the DoH + server's hostname is resolved by httpx — locally through the OS resolver, + or by the proxy itself via ``CONNECT`` when one is configured — so no + UDP/53 access is required. + """ + + def query( + self, + request: dns.message.QueryMessage, + timeout: float, + source: str | None, + source_port: int, + max_size: bool = False, + one_rr_per_rrset: bool = False, + ignore_trailing: bool = False, + ) -> dns.message.Message: + return dns.query.https( + request, + self.url, + timeout=timeout, + source=source, + source_port=source_port, + one_rr_per_rrset=one_rr_per_rrset, + ignore_trailing=ignore_trailing, + verify=self.verify, + post=(not self.want_get), + http_version=self.http_version, + session=_get_doh_session(), + ) + + +def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver: + """ + Parses a ``tls://ip[:port][#hostname]`` nameserver entry. + + The optional ``#hostname`` suffix names the TLS certificate identity to + use for SNI and verification, matching systemd-resolved's syntax. + + Args: + entry (str): A ``tls://`` nameserver entry + + Returns: + dns.nameserver.DoTNameserver: The parsed nameserver + + Raises: + ValueError: The entry has no host, an unusable port, a host that + is not a literal IP address, or extra URL components + """ + parts = urlsplit(entry) + try: + port = parts.port + except ValueError as e: + # urlsplit only validates the port when it is accessed + raise ValueError(f"Invalid DNS over TLS nameserver {entry}: {e}") from e + if parts.username or parts.path or parts.query: + # Catch tls://9.9.9.9/dns.quad9.net — a plausible slash-for-# + # typo that would otherwise "work" with no certificate identity + # and fail only at query time with an opaque TLS error + raise ValueError( + f"Invalid DNS over TLS nameserver {entry}: only " + "tls://ip[:port][#hostname] is supported — the TLS certificate " + "identity is given after #, not /" + ) + address = parts.hostname + if not address: + raise ValueError(f"Invalid DNS over TLS nameserver {entry}: missing IP address") + if not dns.inet.is_address(address): + raise ValueError( + f"Invalid DNS over TLS nameserver {entry}: {address} is not an IP " + "address. Use tls://ip[:port][#hostname], where the optional " + "#hostname is the TLS certificate identity of the server" + ) + hostname = parts.fragment or None + if port is None: + return dns.nameserver.DoTNameserver(address, hostname=hostname) + return dns.nameserver.DoTNameserver(address, port, hostname) + + +def _nameservers_to_resolver_input( + nameservers: list[str], +) -> list[str | dns.nameserver.Nameserver]: + """ + Converts configured nameserver strings into values that + ``dns.resolver.Resolver.nameservers`` accepts. + + ``https://`` entries become DNS over HTTPS nameservers that share this + process's ``httpx`` client (so proxy and CA environment variables apply), + and ``tls://ip[:port][#hostname]`` entries become DNS over TLS + nameservers. Everything else — plain IPv4/IPv6 addresses — is passed + through untouched, leaving dnspython to enrich and validate it exactly as + before. + + Args: + nameservers (list[str]): The configured nameservers + + Returns: + list: A list of strings and/or ``dns.nameserver.Nameserver`` objects, + in the configured order + + Raises: + ValueError: A ``tls://`` entry is malformed + """ + resolver_input: list[str | dns.nameserver.Nameserver] = [] + for entry in nameservers: + try: + # urlsplit lowercases the scheme, so HTTPS:// and TLS:// work too + scheme = urlsplit(entry).scheme + except ValueError: + # e.g. an unbalanced IPv6 bracket; let dnspython reject it with + # its own message about what a nameserver may be + scheme = "" + if scheme == "https": + resolver_input.append(_SessionDoHNameserver(entry)) + elif scheme == "tls": + resolver_input.append(_parse_dot_nameserver(entry)) + else: + resolver_input.append(entry) + return resolver_input + + def query_dns( domain: str, record_type: str, @@ -217,7 +391,12 @@ def query_dns( (Cloudflare's public DNS resolvers by default). Pass ``parsedmarc.constants.RECOMMENDED_DNS_NAMESERVERS`` for a cross-provider mix that fails over when one provider's path is - slow or broken. + slow or broken. Each entry is an IP address (DNS over UDP/TCP + port 53), an ``https://`` URL (DNS over HTTPS, honoring the + ``HTTP_PROXY``/``HTTPS_PROXY``/``NO_PROXY`` and ``SSL_CERT_FILE`` + environment variables), or ``tls://ip[:port][#hostname]`` (DNS + over TLS, port 853 by default, with the optional ``#hostname`` + naming the server's TLS certificate identity). timeout (float): Overall DNS lifetime budget in seconds per configured nameserver. Per-query UDP attempts are capped at ``min(1.0, timeout)`` so dnspython retries within the lifetime on @@ -250,7 +429,7 @@ def query_dns( "2606:4700:4700::1111", "2606:4700:4700::1001", ] - resolver.nameservers = nameservers + resolver.nameservers = _nameservers_to_resolver_input(nameservers) # Cap per-query UDP timeout at 1s so dnspython retries within the # lifetime window on transient packet loss — otherwise with a single # nameserver and timeout == lifetime, one dropped UDP datagram consumes diff --git a/pyproject.toml b/pyproject.toml index 42822b06..cf1ace7e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -41,7 +41,10 @@ dependencies = [ "azure-monitor-ingestion>=1.0.0", "boto3>=1.16.63", "dateparser>=1.1.1", - "dnspython>=2.0.0", + # The [doh] extra supplies the httpx/h2 floors DNS over HTTPS needs; + # 2.7.0 is the floor verified against the dns.nameserver and + # dns.query.https(session=...) APIs utils.py builds on. + "dnspython[doh]>=2.7.0", "elasticsearch>=8.18,<9", "expiringdict>=1.1.4", # The runtime HTTP library (utils.py fetches, webhook and Splunk HEC diff --git a/tests/test_cli.py b/tests/test_cli.py index e1b406c2..8d480954 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -5747,6 +5747,59 @@ class TestParseConfigGeneral(unittest.TestCase): self.assertEqual(opts.dns_retries, 2) self.assertEqual(opts.nameservers, ["1.1.1.1", "8.8.8.8"]) + def test_general_nameservers_accept_doh_and_dot_entries(self): + """A nameservers list may mix DNS over HTTPS URLs and DNS over TLS + entries with plain IP addresses (issue #880); _parse_config splits + and strips them like any other list value, and hands them to the DNS + pre-flight check (mocked here so no network is needed).""" + from parsedmarc.cli import _parse_config + + cp = _config_with( + "general", + { + "dns_test_address": "1.1.1.1", + "dns_timeout": "5.0", + "nameservers": ( + "https://cloudflare-dns.com/dns-query, tls://9.9.9.9#dns.quad9.net" + ), + }, + ) + opts = _opts() + with patch( + "parsedmarc.cli.get_reverse_dns", return_value="one.one.one.one" + ) as mock_reverse_dns: + _parse_config(cp, opts) + self.assertEqual( + opts.nameservers, + ["https://cloudflare-dns.com/dns-query", "tls://9.9.9.9#dns.quad9.net"], + ) + self.assertEqual( + mock_reverse_dns.call_args.kwargs["nameservers"], opts.nameservers + ) + + def test_general_nameservers_malformed_dot_entry_fails_pre_flight(self): + """A malformed DNS over TLS nameservers entry raises a + ConfigurationError at startup, before any mailbox work begins. No + mocking is needed: the entry mapper rejects tls://not-an-ip with a + ValueError before any query is sent (the pre-flight path passes no + cache, so nothing can short-circuit it), and _parse_config wraps + any pre-flight failure in a ConfigurationError.""" + from parsedmarc.cli import ConfigurationError, _parse_config + + cp = _config_with( + "general", + { + "dns_test_address": "1.1.1.1", + "dns_timeout": "2.0", + "nameservers": "tls://not-an-ip", + }, + ) + opts = _opts() + with self.assertRaises(ConfigurationError) as ctx: + _parse_config(cp, opts) + self.assertIn("pre-flight", str(ctx.exception)) + self.assertIn("tls://not-an-ip", str(ctx.exception)) + def test_general_normalize_timespan_threshold(self): from parsedmarc.cli import _parse_config diff --git a/tests/test_utils.py b/tests/test_utils.py index 4e0fe8aa..c0f28825 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -8,9 +8,15 @@ import unittest from datetime import datetime, timezone from importlib.resources import files from tempfile import NamedTemporaryFile +from typing import cast from unittest.mock import MagicMock, patch import dns.exception +import dns.message +import dns.nameserver +import dns.rdata +import dns.rdataclass +import dns.rdatatype import dns.resolver import httpx from expiringdict import ExpiringDict @@ -1229,6 +1235,235 @@ class TestQueryDnsRetries(unittest.TestCase): self.assertEqual(mock_resolve.call_count, 3) +class TestEncryptedDnsNameservers(unittest.TestCase): + """Tests for the DNS over HTTPS and DNS over TLS transports selected by + the form of each ``nameservers`` entry + (https://github.com/domainaware/parsedmarc/issues/880). The end-to-end + tests mock at the dnspython SDK boundary (``dns.query.https`` / + ``dns.query.tls``) and assert on the answer query_dns parses back out of + a real DNS response message, so the whole chain — entry mapping, + resolver, nameserver object, transport call — is exercised.""" + + def setUp(self): + # The DoH session is process-wide state; keep test ordering from + # leaking a client (or a PID) between tests. + self._old_session = parsedmarc.utils._DOH_SESSION + self._old_pid = parsedmarc.utils._DOH_SESSION_PID + parsedmarc.utils._DOH_SESSION = None + parsedmarc.utils._DOH_SESSION_PID = None + + def restore(): + session = parsedmarc.utils._DOH_SESSION + if session is not None and session is not self._old_session: + session.close() + parsedmarc.utils._DOH_SESSION = self._old_session + parsedmarc.utils._DOH_SESSION_PID = self._old_pid + + self.addCleanup(restore) + + @staticmethod + def _ptr_responder(captured: list, answer: str = "dns.example."): + """Builds a dns.query stand-in that answers any PTR query with + ``answer``, recording the keyword arguments it was called with.""" + + def responder(request, *args, **kwargs): + captured.append(kwargs) + response = dns.message.make_response(request) + # find_rrset(create=True) rather than appending to + # response.answer, so the message's rrset index is updated too — + # dnspython looks the answer up through that index. + rrset = response.find_rrset( + response.answer, + request.question[0].name, + dns.rdataclass.IN, + dns.rdatatype.PTR, + create=True, + ) + rrset.add(dns.rdata.from_text("IN", "PTR", answer), 300) + return response + + return responder + + def test_plain_ip_entries_are_passed_through_untouched(self): + """An IP address is handed to dnspython as the same string object, + leaving its own Do53 enrichment (and port defaulting) in charge.""" + entries = ["1.1.1.1", "2606:4700:4700::1111"] + mapped = parsedmarc.utils._nameservers_to_resolver_input(entries) + self.assertEqual(len(mapped), 2) + for original, result in zip(entries, mapped): + self.assertIs(result, original) + + def _map_doh(self, entry: str) -> parsedmarc.utils._SessionDoHNameserver: + """Maps one entry and asserts it produced a DoH nameserver.""" + (mapped,) = parsedmarc.utils._nameservers_to_resolver_input([entry]) + self.assertIsInstance(mapped, parsedmarc.utils._SessionDoHNameserver) + return cast(parsedmarc.utils._SessionDoHNameserver, mapped) + + def _map_dot(self, entry: str) -> dns.nameserver.DoTNameserver: + """Maps one entry and asserts it produced a DoT nameserver.""" + (mapped,) = parsedmarc.utils._nameservers_to_resolver_input([entry]) + self.assertIsInstance(mapped, dns.nameserver.DoTNameserver) + return cast(dns.nameserver.DoTNameserver, mapped) + + def test_https_entry_becomes_a_session_doh_nameserver(self): + """An https:// entry maps to the DoH nameserver subclass that + queries through parsedmarc's own httpx client.""" + url = "https://cloudflare-dns.com/dns-query" + self.assertEqual(self._map_doh(url).url, url) + + def test_uppercase_scheme_is_recognized(self): + """The scheme is compared as urlsplit reports it, which is + lowercased, so HTTPS:// selects DoH rather than falling through to + dnspython as an unusable string.""" + self._map_doh("HTTPS://cloudflare-dns.com/dns-query") + + def test_tls_entry_defaults_to_port_853_with_no_hostname(self): + """tls:// alone uses DoTNameserver's default port and performs + no certificate-identity substitution.""" + mapped = self._map_dot("tls://9.9.9.9") + self.assertEqual(mapped.address, "9.9.9.9") + self.assertEqual(mapped.port, 853) + self.assertIsNone(mapped.hostname) + + def test_tls_entry_port_and_hostname_are_parsed(self): + """tls://ip:port#hostname sets both the port and the TLS + certificate identity.""" + mapped = self._map_dot("tls://9.9.9.9:8853#dns.quad9.net") + self.assertEqual(mapped.address, "9.9.9.9") + self.assertEqual(mapped.port, 8853) + self.assertEqual(mapped.hostname, "dns.quad9.net") + + def test_bracketed_ipv6_tls_entry_is_parsed(self): + """An IPv6 address is bracketed so the colon before the port is + unambiguous; the brackets are not part of the address.""" + mapped = self._map_dot("tls://[2620:fe::fe]:853#dns.quad9.net") + self.assertEqual(mapped.address, "2620:fe::fe") + self.assertEqual(mapped.port, 853) + + def test_mixed_list_preserves_order_and_types(self): + """Transports can be mixed in one nameservers list, and the + configured order is the failover order dnspython will use.""" + mapped = parsedmarc.utils._nameservers_to_resolver_input( + [ + "1.1.1.1", + "https://cloudflare-dns.com/dns-query", + "tls://9.9.9.9#dns.quad9.net", + ] + ) + self.assertEqual(mapped[0], "1.1.1.1") + self.assertIsInstance(mapped[1], parsedmarc.utils._SessionDoHNameserver) + self.assertIsInstance(mapped[2], dns.nameserver.DoTNameserver) + + def test_tls_entry_without_a_host_is_rejected(self): + with self.assertRaises(ValueError) as ctx: + parsedmarc.utils._nameservers_to_resolver_input(["tls://"]) + self.assertIn("tls://", str(ctx.exception)) + + def test_tls_entry_with_a_hostname_instead_of_an_ip_is_rejected(self): + """DoTNameserver takes an IP address, not a name — it has no + resolver of its own to bootstrap with. The certificate identity is + supplied by the #hostname suffix instead.""" + with self.assertRaises(ValueError) as ctx: + parsedmarc.utils._nameservers_to_resolver_input(["tls://dns.quad9.net"]) + self.assertIn("tls://dns.quad9.net", str(ctx.exception)) + + def test_tls_entry_with_an_invalid_port_is_rejected(self): + """urlsplit only validates the port when it is read, so the + ValueError it raises there is re-raised naming the entry.""" + with self.assertRaises(ValueError) as ctx: + parsedmarc.utils._nameservers_to_resolver_input(["tls://9.9.9.9:notaport"]) + self.assertIn("tls://9.9.9.9:notaport", str(ctx.exception)) + + def test_tls_entry_with_a_path_is_rejected(self): + """tls://9.9.9.9/dns.quad9.net — a plausible slash-for-# typo — + must fail at configuration time naming the entry, not parse + "successfully" with no TLS certificate identity and then fail at + query time with an opaque certificate error. Userinfo and query + components are rejected the same way.""" + for entry in ( + "tls://9.9.9.9/dns.quad9.net", + "tls://user@9.9.9.9", + "tls://9.9.9.9?hostname=dns.quad9.net", + ): + with self.subTest(entry=entry): + with self.assertRaises(ValueError) as ctx: + parsedmarc.utils._nameservers_to_resolver_input([entry]) + self.assertIn(entry, str(ctx.exception)) + + def test_unsplittable_entry_is_left_for_dnspython_to_reject(self): + """urlsplit itself raises on some malformed input (an unbalanced + IPv6 bracket). Such an entry is passed through rather than reported + as a DoH/DoT problem, and dnspython rejects it with its own message + about what a nameserver may be.""" + entry = "https://[::1" + (mapped,) = parsedmarc.utils._nameservers_to_resolver_input([entry]) + self.assertIs(mapped, entry) + with self.assertRaises(ValueError): + dns.resolver.Resolver(configure=False).nameservers = [entry] + + def test_doh_query_returns_answers_through_a_shared_httpx_client(self): + """An https:// nameserver resolves through dns.query.https, and the + session it is given is the module's shared client, an httpx.Client + with trust_env left on — httpx's documented switch for honoring + HTTP_PROXY/HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE, which is what a + proxy-only network needs (issue #880). dnspython's stock DoH + nameserver passes no session at all, and the client dns.query.https + builds for itself has a custom transport, which disables environment + proxies. Asserting identity with the shared client also observes the + connection-reuse half of the claim: a fresh per-query client would + satisfy every other assertion here.""" + captured: list = [] + with patch("dns.query.https", side_effect=self._ptr_responder(captured)): + records = parsedmarc.utils.query_dns( + "1.0.0.1.in-addr.arpa", + "PTR", + cache=ExpiringDict(max_len=10, max_age_seconds=60), + nameservers=["https://dns.example/dns-query"], + timeout=2, + ) + self.assertEqual(records, ["dns.example"]) + self.assertEqual(len(captured), 1) + session = captured[0]["session"] + self.assertIsInstance(session, httpx.Client) + self.assertIs(session.trust_env, True) + self.assertIs(session, parsedmarc.utils._DOH_SESSION) + + def test_dot_query_returns_answers_and_passes_the_tls_hostname(self): + """A tls:// nameserver resolves through dns.query.tls, and the + #hostname suffix reaches it as server_hostname — the name the + server's certificate is checked against.""" + captured: list = [] + with patch("dns.query.tls", side_effect=self._ptr_responder(captured)): + records = parsedmarc.utils.query_dns( + "1.0.0.1.in-addr.arpa", + "PTR", + cache=ExpiringDict(max_len=10, max_age_seconds=60), + nameservers=["tls://9.9.9.9#dns.quad9.net"], + timeout=2, + ) + self.assertEqual(records, ["dns.example"]) + self.assertEqual(len(captured), 1) + self.assertEqual(captured[0]["server_hostname"], "dns.quad9.net") + self.assertEqual(captured[0]["port"], 853) + + def test_doh_session_is_reused_within_a_process(self): + """Consecutive calls hand back the same client, so DoH lookups + reuse the connection instead of renegotiating TLS per query.""" + first = parsedmarc.utils._get_doh_session() + self.assertIs(parsedmarc.utils._get_doh_session(), first) + + def test_doh_session_is_rebuilt_after_a_fork(self): + """A forked worker (n_procs) inherits the module globals, including + the parent's open sockets. Recording the creating PID makes the + child build its own client rather than share the parent's.""" + parent_session = parsedmarc.utils._get_doh_session() + parsedmarc.utils._DOH_SESSION_PID = os.getpid() + 1 + child_session = parsedmarc.utils._get_doh_session() + self.addCleanup(parent_session.close) + self.assertIsNot(child_session, parent_session) + self.assertEqual(parsedmarc.utils._DOH_SESSION_PID, os.getpid()) + + class TestLoadIpDb(unittest.TestCase): """Tests for the load_ip_db() download/cache/bundled fallback chain, mocking at the httpx SDK boundary."""