Add DNS over HTTPS and DNS over TLS support via the nameservers option (#886)

Each entry in the existing nameservers option now selects its own
transport (#880): an IP address means plain DNS on port 53 exactly as
before, an https:// URL means DoH, and tls://ip[:port][#hostname] means
DoT, with the optional #hostname naming the TLS certificate identity
(systemd-resolved syntax). Forms can be mixed in one list, and no new
configuration option is involved.

DoH queries go through a shared per-process httpx client passed to
dns.query.https as session=, which is what makes them honor HTTP_PROXY/
HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE — the motivating proxy-only
corporate network case. dnspython's stock DoH path cannot do this: it
builds its httpx client around a custom transport, and httpx only reads
proxy environment variables when no transport is supplied
(allow_env_proxies = trust_env and transport is None). The client is
rebuilt when the PID changes so fork-based worker pools never share a
parent's sockets.

The dnspython requirement becomes dnspython[doh]>=2.7.0 — the extra
supplies the httpx/h2 floors DoH needs, and 2.7.0 is the floor verified
against the dns.nameserver and dns.query.https(session=...) APIs used.

The startup DNS pre-flight check now exercises whichever transports are
configured, so a malformed DoH/DoT entry raises ConfigurationError
before any mailbox work begins. Malformed tls:// entries — including
the plausible slash-for-# typo tls://9.9.9.9/dns.quad9.net, which would
otherwise silently drop the certificate identity — are rejected at
configuration time naming the entry.

Verified live: DoH A/PTR queries against Cloudflare and DoT against
Quad9 (tls://9.9.9.9#dns.quad9.net), plus a full CLI run over a sample
report with encrypted-DNS-only nameservers.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-08-28 15:03:51 -04:00
committed by GitHub
co-authored by Claude Fable 5
parent ffdb220af9
commit cd864f4cfc
7 changed files with 538 additions and 7 deletions
+4
View File
@@ -2,6 +2,10 @@
## Unreleased
### New features
- **DNS over HTTPS (DoH) and DNS over TLS (DoT) are now supported through the existing `nameservers` option** ([#880](https://github.com/domainaware/parsedmarc/issues/880)). No new configuration option is involved: each entry in the comma-separated list now picks its own transport, and the forms can be mixed in one list. An IP address means plain DNS over UDP and TCP port 53, exactly as before; an `https://` URL means DoH; and `tls://ip[:port][#hostname]` means DoT, where the port defaults to 853 and the optional `#hostname` supplies the TLS certificate identity (SNI) to verify the server against, matching systemd-resolved's syntax. DoH queries go through a per-process `httpx` client with its environment trust left at the default, so they honor the standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` variables — the motivating case, a corporate network that blocks outbound DNS but offers an HTTP proxy, where the proxy also resolves the DoH server's own hostname, so no access to UDP port 53 is needed at all — as well as `SSL_CERT_FILE`/`SSL_CERT_DIR`, for a private CA in front of a TLS-inspecting proxy. dnspython's own DoH support cannot do this, because it builds its HTTP client with a custom transport, which is precisely the condition under which httpx ignores proxy environment variables. DoT connections are made directly to TCP port 853 and do not use a proxy. A `nameservers` list of plain IP addresses — including the default — behaves exactly as it did before, and the pre-flight DNS check at startup now exercises whichever transports are configured, so a bad DoH or DoT entry is reported before any mailbox is opened. The `dnspython` requirement is now `dnspython[doh]>=2.7.0`.
### Bug fixes
- **Failure report MIME parts are now decoded according to their `Content-Transfer-Encoding`** ([#882](https://github.com/domainaware/parsedmarc/issues/882)). `parse_report_email()` read every part's payload without asking the standard library to decode it, so any transfer encoding was left in place. Two things went wrong as a result. A quoted-printable `text/rfc822-headers` sample part kept its RFC 2045 §6.7 soft line breaks — a `=` followed by a line ending, with none of the RFC 5322 folding whitespace a header parser needs — which split long headers mid-value; the sample's `From` header became unparseable, and since the report itself carried no `Reported-Domain` field, the whole failure report was discarded with `TypeError: 'NoneType' object is not subscriptable`. Separately, a quoted-printable `message/feedback-report` part parsed "successfully" with silently corrupted field values, such as an `authentication_results` of `mx.example.com; dmarc=3Dfail`. Decoding is applied only to the `message/feedback-report` part and to the sample part; all other branches (attachments, `application/tlsrpt+*`, `text/plain`) still receive the raw payload, since they decode base64 and sniff zip/gzip magic themselves. Parts declaring no transfer encoding, or a 7bit/8bit one, are likewise left alone: there is nothing to undo, and running such a part through the standard library's decoder would round-trip its already-correct text through `raw-unicode-escape` bytes and replace every non-ASCII character with U+FFFD. Note that a composite part carrying a real transfer encoding is illegal per RFC 2045 §6.4, but reporters send them anyway, and the standard library nests the still-encoded text as a child message rather than decoding it — so those parts are decoded by hand. That nested parse needs one repair first: a soft line break splits a long field onto a continuation line with no colon and no leading whitespace, so the parser treats the remainder as a message body and re-serializing inserts a blank line the encoded text never had, which would otherwise truncate the value at its first split. Any unexpected failure in the new decoding step falls back to the previous behavior, so no report that parsed before can start failing because of it.
+56 -3
View File
@@ -39,7 +39,8 @@ options:
--smtp-tls-csv-filename SMTP_TLS_CSV_FILENAME
filename for the SMTP TLS CSV output file
-n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
nameservers to query
nameservers to query: IP addresses, https:// URLs (DNS over HTTPS), and/or
tls://ip[:port][#hostname] (DNS over TLS)
-t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
number of seconds to wait for an answer from DNS (default: 2.0)
--dns-retries DNS_RETRIES
@@ -178,10 +179,15 @@ The full set of configuration options are:
- `local_psl_overrides_path` - Overrides the default local file path to use for the PSL overrides list
- `psl_overrides_url` - Overrides the default download URL for the PSL overrides list
- `nameservers` - str: A comma separated list of
DNS resolvers (Default: `[Cloudflare's public resolvers]`)
DNS resolvers (Default: `[Cloudflare's public resolvers]`). Each entry
is an IP address (DNS over UDP/TCP port 53), an `https://` URL
(DNS over HTTPS), or `tls://ip[:port][#hostname]` (DNS over TLS) —
see [Encrypted DNS](#encrypted-dns)
- `dns_test_address` - str: a dummy address used for DNS pre-flight checks
(Default: 1.1.1.1)
- `dns_timeout` - float: DNS timeout period
- `dns_retries` - int: Number of times to retry a DNS query after a
timeout or other transient error (Default: 0)
- `debug` - bool: Print debugging messages
- `silent` - bool: Only print errors (Default: `True`)
- `fail_on_output_error` - bool: Exit with a non-zero status code if
@@ -810,7 +816,8 @@ setting. By default, `parsedmarc` uses
reliable than Google, Cisco OpenDNS, or even most local resolvers.
The `nameservers` option should only be used if your network
blocks DNS requests to outside resolvers.
blocks DNS requests to outside resolvers, or blocks plain DNS
entirely — see [Encrypted DNS](#encrypted-dns).
:::
:::{note}
@@ -870,6 +877,52 @@ PUT _cluster/settings
Increasing this value increases resource usage.
:::
### Encrypted DNS
Every entry in the `nameservers` list picks its own transport:
- An IP address — plain DNS over UDP and TCP port 53, the default and the
behavior of every earlier release.
- An `https://` URL — DNS over HTTPS (DoH).
- `tls://ip[:port][#hostname]` — DNS over TLS (DoT). The port defaults to
853, and the optional `#hostname` names the TLS certificate identity of
the server (SNI), matching systemd-resolved's syntax. The host itself must
be an IP address, and an IPv6 address must be wrapped in brackets, so that
its colons cannot be mistaken for the port separator —
`tls://[2620:fe::fe]#dns.quad9.net`.
Forms can be mixed, and are tried in the order given:
```ini
[general]
nameservers = https://cloudflare-dns.com/dns-query, tls://9.9.9.9#dns.quad9.net
```
On a network where outbound DNS is blocked but an HTTP proxy is available,
configure DoH nameservers and set the standard proxy environment variables
([issue #880](https://github.com/domainaware/parsedmarc/issues/880)):
```bash
export HTTPS_PROXY=http://proxy.example.net:3128
export NO_PROXY=elasticsearch.example.net,127.0.0.1
```
parsedmarc's DoH queries honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`,
and the proxy resolves the DoH server's own hostname on parsedmarc's behalf,
so such a deployment needs no access to UDP port 53 at all.
If the proxy inspects TLS, point the standard `SSL_CERT_FILE` environment
variable at your organization's CA bundle so its certificate is trusted:
```bash
export SSL_CERT_FILE=/etc/ssl/certs/corporate-ca.pem
```
:::{note}
parsedmarc's DoT connections are made directly to TCP port 853 and do not
use a proxy. Use DoH on a proxy-only network.
:::
### Mailbox messages are only archived once the reports are saved
parsedmarc processes a mailbox in batches of `batch_size` messages. Each
+5 -1
View File
@@ -2372,7 +2372,11 @@ def _main():
default="smtp_tls.csv",
)
arg_parser.add_argument(
"-n", "--nameservers", nargs="+", help="nameservers to query"
"-n",
"--nameservers",
nargs="+",
help="nameservers to query: IP addresses, https:// URLs (DNS over "
"HTTPS), and/or tls://ip[:port][#hostname] (DNS over TLS)",
)
arg_parser.add_argument(
"-t",
+181 -2
View File
@@ -18,6 +18,7 @@ import subprocess
import tempfile
from datetime import datetime, timedelta, timezone
from typing import TypedDict, cast
from urllib.parse import urlsplit
import mailparser
from expiringdict import ExpiringDict
@@ -26,6 +27,10 @@ from importlib.resources import files
import dns.exception
import dns.inet
import dns.message
import dns.nameserver
import dns.query
import dns.resolver
import dns.reversename
import httpx
@@ -51,6 +56,11 @@ _RETRYABLE_DNS_ERRORS = (
OSError,
)
# The process-wide httpx client used for DNS over HTTPS queries, and the PID
# it was created under. See _get_doh_session().
_DOH_SESSION: httpx.Client | None = None
_DOH_SESSION_PID: int | None = None
parenthesis_regex = re.compile(r"\s*\(.*\)\s*")
null_file = subprocess.DEVNULL
@@ -196,6 +206,170 @@ def get_base_domain(domain: str) -> str | None:
return publicsuffix
def _get_doh_session() -> httpx.Client:
"""
Returns the shared ``httpx.Client`` used for DNS over HTTPS queries.
The client is created on first use and reused afterwards, so DoH queries
share TLS connections instead of renegotiating one per lookup. It is
deliberately never closed: like the module's other shared state, it lives
for the life of the process.
The client is rebuilt when the current PID differs from the one it was
created under, because a ``fork()``-based worker pool (``n_procs``) would
otherwise inherit and concurrently use the parent's sockets.
``httpx.Client`` defaults are what make this work behind a corporate
proxy: ``trust_env=True`` honors ``HTTP_PROXY``/``HTTPS_PROXY``/
``NO_PROXY`` and ``SSL_CERT_FILE``/``SSL_CERT_DIR``, and ``verify=True``
keeps certificate verification on. Neither is overridden here.
Returns:
httpx.Client: The shared DoH client for this process
"""
global _DOH_SESSION, _DOH_SESSION_PID
pid = os.getpid()
if _DOH_SESSION is None or _DOH_SESSION_PID != pid:
_DOH_SESSION = httpx.Client(http1=True, http2=True)
_DOH_SESSION_PID = pid
return _DOH_SESSION
class _SessionDoHNameserver(dns.nameserver.DoHNameserver):
"""
A DNS over HTTPS nameserver that queries through a shared ``httpx``
client.
dnspython's stock ``DoHNameserver`` calls ``dns.query.https()`` without a
``session``, which makes that function build an ``httpx.Client`` with its
own custom transport and httpx only reads proxy environment variables
when no transport is supplied (``allow_env_proxies = trust_env and
transport is None``). Stock DoH therefore ignores ``HTTPS_PROXY``
entirely and a proxy is the only way out of the networks this exists
for (https://github.com/domainaware/parsedmarc/issues/880).
Passing our own session instead gets environment proxies, environment CA
configuration (``SSL_CERT_FILE``), and connection reuse across queries.
``bootstrap_address`` is deliberately not passed: with a session, the DoH
server's hostname is resolved by httpx — locally through the OS resolver,
or by the proxy itself via ``CONNECT`` when one is configured so no
UDP/53 access is required.
"""
def query(
self,
request: dns.message.QueryMessage,
timeout: float,
source: str | None,
source_port: int,
max_size: bool = False,
one_rr_per_rrset: bool = False,
ignore_trailing: bool = False,
) -> dns.message.Message:
return dns.query.https(
request,
self.url,
timeout=timeout,
source=source,
source_port=source_port,
one_rr_per_rrset=one_rr_per_rrset,
ignore_trailing=ignore_trailing,
verify=self.verify,
post=(not self.want_get),
http_version=self.http_version,
session=_get_doh_session(),
)
def _parse_dot_nameserver(entry: str) -> dns.nameserver.DoTNameserver:
"""
Parses a ``tls://ip[:port][#hostname]`` nameserver entry.
The optional ``#hostname`` suffix names the TLS certificate identity to
use for SNI and verification, matching systemd-resolved's syntax.
Args:
entry (str): A ``tls://`` nameserver entry
Returns:
dns.nameserver.DoTNameserver: The parsed nameserver
Raises:
ValueError: The entry has no host, an unusable port, a host that
is not a literal IP address, or extra URL components
"""
parts = urlsplit(entry)
try:
port = parts.port
except ValueError as e:
# urlsplit only validates the port when it is accessed
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: {e}") from e
if parts.username or parts.path or parts.query:
# Catch tls://9.9.9.9/dns.quad9.net — a plausible slash-for-#
# typo that would otherwise "work" with no certificate identity
# and fail only at query time with an opaque TLS error
raise ValueError(
f"Invalid DNS over TLS nameserver {entry}: only "
"tls://ip[:port][#hostname] is supported — the TLS certificate "
"identity is given after #, not /"
)
address = parts.hostname
if not address:
raise ValueError(f"Invalid DNS over TLS nameserver {entry}: missing IP address")
if not dns.inet.is_address(address):
raise ValueError(
f"Invalid DNS over TLS nameserver {entry}: {address} is not an IP "
"address. Use tls://ip[:port][#hostname], where the optional "
"#hostname is the TLS certificate identity of the server"
)
hostname = parts.fragment or None
if port is None:
return dns.nameserver.DoTNameserver(address, hostname=hostname)
return dns.nameserver.DoTNameserver(address, port, hostname)
def _nameservers_to_resolver_input(
nameservers: list[str],
) -> list[str | dns.nameserver.Nameserver]:
"""
Converts configured nameserver strings into values that
``dns.resolver.Resolver.nameservers`` accepts.
``https://`` entries become DNS over HTTPS nameservers that share this
process's ``httpx`` client (so proxy and CA environment variables apply),
and ``tls://ip[:port][#hostname]`` entries become DNS over TLS
nameservers. Everything else plain IPv4/IPv6 addresses is passed
through untouched, leaving dnspython to enrich and validate it exactly as
before.
Args:
nameservers (list[str]): The configured nameservers
Returns:
list: A list of strings and/or ``dns.nameserver.Nameserver`` objects,
in the configured order
Raises:
ValueError: A ``tls://`` entry is malformed
"""
resolver_input: list[str | dns.nameserver.Nameserver] = []
for entry in nameservers:
try:
# urlsplit lowercases the scheme, so HTTPS:// and TLS:// work too
scheme = urlsplit(entry).scheme
except ValueError:
# e.g. an unbalanced IPv6 bracket; let dnspython reject it with
# its own message about what a nameserver may be
scheme = ""
if scheme == "https":
resolver_input.append(_SessionDoHNameserver(entry))
elif scheme == "tls":
resolver_input.append(_parse_dot_nameserver(entry))
else:
resolver_input.append(entry)
return resolver_input
def query_dns(
domain: str,
record_type: str,
@@ -217,7 +391,12 @@ def query_dns(
(Cloudflare's public DNS resolvers by default). Pass
``parsedmarc.constants.RECOMMENDED_DNS_NAMESERVERS`` for a
cross-provider mix that fails over when one provider's path is
slow or broken.
slow or broken. Each entry is an IP address (DNS over UDP/TCP
port 53), an ``https://`` URL (DNS over HTTPS, honoring the
``HTTP_PROXY``/``HTTPS_PROXY``/``NO_PROXY`` and ``SSL_CERT_FILE``
environment variables), or ``tls://ip[:port][#hostname]`` (DNS
over TLS, port 853 by default, with the optional ``#hostname``
naming the server's TLS certificate identity).
timeout (float): Overall DNS lifetime budget in seconds per
configured nameserver. Per-query UDP attempts are capped at
``min(1.0, timeout)`` so dnspython retries within the lifetime on
@@ -250,7 +429,7 @@ def query_dns(
"2606:4700:4700::1111",
"2606:4700:4700::1001",
]
resolver.nameservers = nameservers
resolver.nameservers = _nameservers_to_resolver_input(nameservers)
# Cap per-query UDP timeout at 1s so dnspython retries within the
# lifetime window on transient packet loss — otherwise with a single
# nameserver and timeout == lifetime, one dropped UDP datagram consumes
+4 -1
View File
@@ -41,7 +41,10 @@ dependencies = [
"azure-monitor-ingestion>=1.0.0",
"boto3>=1.16.63",
"dateparser>=1.1.1",
"dnspython>=2.0.0",
# The [doh] extra supplies the httpx/h2 floors DNS over HTTPS needs;
# 2.7.0 is the floor verified against the dns.nameserver and
# dns.query.https(session=...) APIs utils.py builds on.
"dnspython[doh]>=2.7.0",
"elasticsearch>=8.18,<9",
"expiringdict>=1.1.4",
# The runtime HTTP library (utils.py fetches, webhook and Splunk HEC
+53
View File
@@ -5747,6 +5747,59 @@ class TestParseConfigGeneral(unittest.TestCase):
self.assertEqual(opts.dns_retries, 2)
self.assertEqual(opts.nameservers, ["1.1.1.1", "8.8.8.8"])
def test_general_nameservers_accept_doh_and_dot_entries(self):
"""A nameservers list may mix DNS over HTTPS URLs and DNS over TLS
entries with plain IP addresses (issue #880); _parse_config splits
and strips them like any other list value, and hands them to the DNS
pre-flight check (mocked here so no network is needed)."""
from parsedmarc.cli import _parse_config
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "5.0",
"nameservers": (
"https://cloudflare-dns.com/dns-query, tls://9.9.9.9#dns.quad9.net"
),
},
)
opts = _opts()
with patch(
"parsedmarc.cli.get_reverse_dns", return_value="one.one.one.one"
) as mock_reverse_dns:
_parse_config(cp, opts)
self.assertEqual(
opts.nameservers,
["https://cloudflare-dns.com/dns-query", "tls://9.9.9.9#dns.quad9.net"],
)
self.assertEqual(
mock_reverse_dns.call_args.kwargs["nameservers"], opts.nameservers
)
def test_general_nameservers_malformed_dot_entry_fails_pre_flight(self):
"""A malformed DNS over TLS nameservers entry raises a
ConfigurationError at startup, before any mailbox work begins. No
mocking is needed: the entry mapper rejects tls://not-an-ip with a
ValueError before any query is sent (the pre-flight path passes no
cache, so nothing can short-circuit it), and _parse_config wraps
any pre-flight failure in a ConfigurationError."""
from parsedmarc.cli import ConfigurationError, _parse_config
cp = _config_with(
"general",
{
"dns_test_address": "1.1.1.1",
"dns_timeout": "2.0",
"nameservers": "tls://not-an-ip",
},
)
opts = _opts()
with self.assertRaises(ConfigurationError) as ctx:
_parse_config(cp, opts)
self.assertIn("pre-flight", str(ctx.exception))
self.assertIn("tls://not-an-ip", str(ctx.exception))
def test_general_normalize_timespan_threshold(self):
from parsedmarc.cli import _parse_config
+235
View File
@@ -8,9 +8,15 @@ import unittest
from datetime import datetime, timezone
from importlib.resources import files
from tempfile import NamedTemporaryFile
from typing import cast
from unittest.mock import MagicMock, patch
import dns.exception
import dns.message
import dns.nameserver
import dns.rdata
import dns.rdataclass
import dns.rdatatype
import dns.resolver
import httpx
from expiringdict import ExpiringDict
@@ -1229,6 +1235,235 @@ class TestQueryDnsRetries(unittest.TestCase):
self.assertEqual(mock_resolve.call_count, 3)
class TestEncryptedDnsNameservers(unittest.TestCase):
"""Tests for the DNS over HTTPS and DNS over TLS transports selected by
the form of each ``nameservers`` entry
(https://github.com/domainaware/parsedmarc/issues/880). The end-to-end
tests mock at the dnspython SDK boundary (``dns.query.https`` /
``dns.query.tls``) and assert on the answer query_dns parses back out of
a real DNS response message, so the whole chain entry mapping,
resolver, nameserver object, transport call is exercised."""
def setUp(self):
# The DoH session is process-wide state; keep test ordering from
# leaking a client (or a PID) between tests.
self._old_session = parsedmarc.utils._DOH_SESSION
self._old_pid = parsedmarc.utils._DOH_SESSION_PID
parsedmarc.utils._DOH_SESSION = None
parsedmarc.utils._DOH_SESSION_PID = None
def restore():
session = parsedmarc.utils._DOH_SESSION
if session is not None and session is not self._old_session:
session.close()
parsedmarc.utils._DOH_SESSION = self._old_session
parsedmarc.utils._DOH_SESSION_PID = self._old_pid
self.addCleanup(restore)
@staticmethod
def _ptr_responder(captured: list, answer: str = "dns.example."):
"""Builds a dns.query stand-in that answers any PTR query with
``answer``, recording the keyword arguments it was called with."""
def responder(request, *args, **kwargs):
captured.append(kwargs)
response = dns.message.make_response(request)
# find_rrset(create=True) rather than appending to
# response.answer, so the message's rrset index is updated too —
# dnspython looks the answer up through that index.
rrset = response.find_rrset(
response.answer,
request.question[0].name,
dns.rdataclass.IN,
dns.rdatatype.PTR,
create=True,
)
rrset.add(dns.rdata.from_text("IN", "PTR", answer), 300)
return response
return responder
def test_plain_ip_entries_are_passed_through_untouched(self):
"""An IP address is handed to dnspython as the same string object,
leaving its own Do53 enrichment (and port defaulting) in charge."""
entries = ["1.1.1.1", "2606:4700:4700::1111"]
mapped = parsedmarc.utils._nameservers_to_resolver_input(entries)
self.assertEqual(len(mapped), 2)
for original, result in zip(entries, mapped):
self.assertIs(result, original)
def _map_doh(self, entry: str) -> parsedmarc.utils._SessionDoHNameserver:
"""Maps one entry and asserts it produced a DoH nameserver."""
(mapped,) = parsedmarc.utils._nameservers_to_resolver_input([entry])
self.assertIsInstance(mapped, parsedmarc.utils._SessionDoHNameserver)
return cast(parsedmarc.utils._SessionDoHNameserver, mapped)
def _map_dot(self, entry: str) -> dns.nameserver.DoTNameserver:
"""Maps one entry and asserts it produced a DoT nameserver."""
(mapped,) = parsedmarc.utils._nameservers_to_resolver_input([entry])
self.assertIsInstance(mapped, dns.nameserver.DoTNameserver)
return cast(dns.nameserver.DoTNameserver, mapped)
def test_https_entry_becomes_a_session_doh_nameserver(self):
"""An https:// entry maps to the DoH nameserver subclass that
queries through parsedmarc's own httpx client."""
url = "https://cloudflare-dns.com/dns-query"
self.assertEqual(self._map_doh(url).url, url)
def test_uppercase_scheme_is_recognized(self):
"""The scheme is compared as urlsplit reports it, which is
lowercased, so HTTPS:// selects DoH rather than falling through to
dnspython as an unusable string."""
self._map_doh("HTTPS://cloudflare-dns.com/dns-query")
def test_tls_entry_defaults_to_port_853_with_no_hostname(self):
"""tls://<ip> alone uses DoTNameserver's default port and performs
no certificate-identity substitution."""
mapped = self._map_dot("tls://9.9.9.9")
self.assertEqual(mapped.address, "9.9.9.9")
self.assertEqual(mapped.port, 853)
self.assertIsNone(mapped.hostname)
def test_tls_entry_port_and_hostname_are_parsed(self):
"""tls://ip:port#hostname sets both the port and the TLS
certificate identity."""
mapped = self._map_dot("tls://9.9.9.9:8853#dns.quad9.net")
self.assertEqual(mapped.address, "9.9.9.9")
self.assertEqual(mapped.port, 8853)
self.assertEqual(mapped.hostname, "dns.quad9.net")
def test_bracketed_ipv6_tls_entry_is_parsed(self):
"""An IPv6 address is bracketed so the colon before the port is
unambiguous; the brackets are not part of the address."""
mapped = self._map_dot("tls://[2620:fe::fe]:853#dns.quad9.net")
self.assertEqual(mapped.address, "2620:fe::fe")
self.assertEqual(mapped.port, 853)
def test_mixed_list_preserves_order_and_types(self):
"""Transports can be mixed in one nameservers list, and the
configured order is the failover order dnspython will use."""
mapped = parsedmarc.utils._nameservers_to_resolver_input(
[
"1.1.1.1",
"https://cloudflare-dns.com/dns-query",
"tls://9.9.9.9#dns.quad9.net",
]
)
self.assertEqual(mapped[0], "1.1.1.1")
self.assertIsInstance(mapped[1], parsedmarc.utils._SessionDoHNameserver)
self.assertIsInstance(mapped[2], dns.nameserver.DoTNameserver)
def test_tls_entry_without_a_host_is_rejected(self):
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils._nameservers_to_resolver_input(["tls://"])
self.assertIn("tls://", str(ctx.exception))
def test_tls_entry_with_a_hostname_instead_of_an_ip_is_rejected(self):
"""DoTNameserver takes an IP address, not a name — it has no
resolver of its own to bootstrap with. The certificate identity is
supplied by the #hostname suffix instead."""
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils._nameservers_to_resolver_input(["tls://dns.quad9.net"])
self.assertIn("tls://dns.quad9.net", str(ctx.exception))
def test_tls_entry_with_an_invalid_port_is_rejected(self):
"""urlsplit only validates the port when it is read, so the
ValueError it raises there is re-raised naming the entry."""
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils._nameservers_to_resolver_input(["tls://9.9.9.9:notaport"])
self.assertIn("tls://9.9.9.9:notaport", str(ctx.exception))
def test_tls_entry_with_a_path_is_rejected(self):
"""tls://9.9.9.9/dns.quad9.net — a plausible slash-for-# typo —
must fail at configuration time naming the entry, not parse
"successfully" with no TLS certificate identity and then fail at
query time with an opaque certificate error. Userinfo and query
components are rejected the same way."""
for entry in (
"tls://9.9.9.9/dns.quad9.net",
"tls://user@9.9.9.9",
"tls://9.9.9.9?hostname=dns.quad9.net",
):
with self.subTest(entry=entry):
with self.assertRaises(ValueError) as ctx:
parsedmarc.utils._nameservers_to_resolver_input([entry])
self.assertIn(entry, str(ctx.exception))
def test_unsplittable_entry_is_left_for_dnspython_to_reject(self):
"""urlsplit itself raises on some malformed input (an unbalanced
IPv6 bracket). Such an entry is passed through rather than reported
as a DoH/DoT problem, and dnspython rejects it with its own message
about what a nameserver may be."""
entry = "https://[::1"
(mapped,) = parsedmarc.utils._nameservers_to_resolver_input([entry])
self.assertIs(mapped, entry)
with self.assertRaises(ValueError):
dns.resolver.Resolver(configure=False).nameservers = [entry]
def test_doh_query_returns_answers_through_a_shared_httpx_client(self):
"""An https:// nameserver resolves through dns.query.https, and the
session it is given is the module's shared client, an httpx.Client
with trust_env left on httpx's documented switch for honoring
HTTP_PROXY/HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE, which is what a
proxy-only network needs (issue #880). dnspython's stock DoH
nameserver passes no session at all, and the client dns.query.https
builds for itself has a custom transport, which disables environment
proxies. Asserting identity with the shared client also observes the
connection-reuse half of the claim: a fresh per-query client would
satisfy every other assertion here."""
captured: list = []
with patch("dns.query.https", side_effect=self._ptr_responder(captured)):
records = parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa",
"PTR",
cache=ExpiringDict(max_len=10, max_age_seconds=60),
nameservers=["https://dns.example/dns-query"],
timeout=2,
)
self.assertEqual(records, ["dns.example"])
self.assertEqual(len(captured), 1)
session = captured[0]["session"]
self.assertIsInstance(session, httpx.Client)
self.assertIs(session.trust_env, True)
self.assertIs(session, parsedmarc.utils._DOH_SESSION)
def test_dot_query_returns_answers_and_passes_the_tls_hostname(self):
"""A tls:// nameserver resolves through dns.query.tls, and the
#hostname suffix reaches it as server_hostname — the name the
server's certificate is checked against."""
captured: list = []
with patch("dns.query.tls", side_effect=self._ptr_responder(captured)):
records = parsedmarc.utils.query_dns(
"1.0.0.1.in-addr.arpa",
"PTR",
cache=ExpiringDict(max_len=10, max_age_seconds=60),
nameservers=["tls://9.9.9.9#dns.quad9.net"],
timeout=2,
)
self.assertEqual(records, ["dns.example"])
self.assertEqual(len(captured), 1)
self.assertEqual(captured[0]["server_hostname"], "dns.quad9.net")
self.assertEqual(captured[0]["port"], 853)
def test_doh_session_is_reused_within_a_process(self):
"""Consecutive calls hand back the same client, so DoH lookups
reuse the connection instead of renegotiating TLS per query."""
first = parsedmarc.utils._get_doh_session()
self.assertIs(parsedmarc.utils._get_doh_session(), first)
def test_doh_session_is_rebuilt_after_a_fork(self):
"""A forked worker (n_procs) inherits the module globals, including
the parent's open sockets. Recording the creating PID makes the
child build its own client rather than share the parent's."""
parent_session = parsedmarc.utils._get_doh_session()
parsedmarc.utils._DOH_SESSION_PID = os.getpid() + 1
child_session = parsedmarc.utils._get_doh_session()
self.addCleanup(parent_session.close)
self.assertIsNot(child_session, parent_session)
self.assertEqual(parsedmarc.utils._DOH_SESSION_PID, os.getpid())
class TestLoadIpDb(unittest.TestCase):
"""Tests for the load_ip_db() download/cache/bundled fallback chain,
mocking at the httpx SDK boundary."""