The two connect_tcp implementations each repeated the per-attempt budget
arithmetic verbatim, and only the sync one delegated resolution and error
mapping to a helper, so a drift between the copies could silently give one
stack a different timeout policy from the other. In the IMAP client,
_connect_pinned re-asserted a fact its only caller had already established,
which reads as a runtime invariant check on a security-relevant path when it
is only a narrowing aid.
Move the budget arithmetic into one helper called from both loops, add an
async twin of the resolve helper so the two loop bodies differ only by await,
and pass the narrowed address tuple into _connect_pinned instead of asserting
it. The PinnedIMAP4 docstring now spells out that no pinning and pinning that
yielded nothing are different things, and the monotonic clock seam says why it
exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every webhook test that reaches a real socket ran with internal requests
disallowed, and the rest intercept above the transport. Wiring the
transport to always disallow internal addresses would therefore have
passed the suite while breaking webhooks to internal hosts on every
default install.
A new test sends a webhook to localhost with internal requests allowed
and checks that the payload arrives and that the guard resolved nothing.
The Host header test's docstring is also corrected: the header now comes
from the URL, not from a resolved hostname.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Outbound host checks parsed IP literals themselves and skipped the
resolver for them. That second parser is what let a host such as
8.8.8.8%2eexample pass as the public address 8.8.8.8, and even with
zone ids limited to IPv6 it remains one more place where the checked
host can be read differently from the connected one.
The separate literal parsing is removed. Every host now goes to
getaddrinfo, which answers numeric literals itself without a lookup, and
only the addresses it returns are classified. Zone ids are still dropped
from the resolver's answers, where a scoped IPv6 literal comes back as
fe80::1%1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The literal check stripped everything after the first "%" from any host,
so a name such as 8.8.8.8%2e169-254-169-254.sslip.io was accepted as the
public address 8.8.8.8 without a DNS lookup. requests percent-decodes the
host before connecting, so Remote OCR would then resolve
8.8.8.8.169-254-169-254.sslip.io and reach an internal address even with
internal endpoints disallowed.
A zone id is now stripped only when the part before "%" is an IPv6
address; any other host containing "%" is looked up as a name. URL
validation with internal addresses disallowed also rejects a host that
contains "%" at all, since the name checked there could otherwise differ
from the one an HTTP client decodes and dials.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The internal-address settings now describe that a hostname is blocked if any resolved address is non-public. Guarded requests connect directly without proxy variables, and webhook requests never follow redirects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every consumer now uses the guarded transports, so the request-rewriting
transport and its helpers are no longer needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_mailbox declared port and security as always-present types, but
MailAccount.imap_port is nullable and imap_security is stored as a plain
integer, so type checking flagged every call site as passing the wrong
type.
Widen the annotations to port: int | None and security: int, matching the
model fields; IntegerChoices members still compare equal to plain ints, so
the existing branching in get_mailbox is unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_mailbox validates the IMAP host with resolve_public_addresses and keeps
its existing error messages; the pinned client dials typed addresses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blocked webhook now raises OutboundRequestBlockedError, which the task
treats as an expected failure and does not retry. The webhook security
tests run against a real local server instead of a patched resolver.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AIClient raises LLMBlockedError when a request was refused by the outbound
connection policy, including when the openai SDK wraps the block in
APIConnectionError. ai_suggestions answers 502 instead of a 500.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The environment proxy test could not fail for any regression in the guard:
httpx ignores environment proxy variables whenever a transport is passed
explicitly, and the address the test pointed the proxy at was itself
internal, so a blocked direct connection was indistinguishable from a
blocked proxied one. It now builds the client through the production
factory, which does not pass a transport, points the proxy variables at a
second recording server, and asserts the real origin server receives the
request while the proxy server sees no connection at all.
Several tests asserting a blocked request never reached the target server
checked only a connection counter that is incremented after accept() in
the server thread, which does not rule out a guard that connects and then
fails validation afterward. Each of those tests now also asserts the dial
recorder saw no address dialled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A local HTTP server, a per-hostname resolver fake and dial spies exercise
the guarded transports end to end: address fallback, blocking before any
connection, the Host header, TLS server name, per-host connection pooling,
numeric host spellings, environment proxies and redirects to blocked hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The transports install the outbound guard on httpcore's connection pool
after checking its exact layout, and accept no proxy, uds or retries
options. httpx, httpcore and anyio become declared dependencies, pinned
narrowly where private attributes are relied on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With internal addresses disallowed, the backend resolves the origin host,
rejects it if any address is non-public, and dials the validated literals
in family-interleaved order under the caller's connect timeout. Resolver
failures surface as connect errors; unix sockets are always refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
urllib3 treats a backslash as the end of the URL authority, while urlparse
and httpx do not. For a URL such as http://127.0.0.1\@evil.example/ the
check resolved evil.example while urllib3 would connect to 127.0.0.1, so a
redirect to such a URL could reach an internal host.
When internal addresses are disallowed, validate_outbound_http_url now
rejects any URL containing a backslash, an ASCII control character or
whitespace before resolving it. URLs validated with internal addresses
allowed are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
resolve_public_addresses and its async twin return every resolved address
in resolver order, de-duplicated and zone-stripped, and reject the whole
name if any address is non-public. validate_outbound_http_url uses them and
keeps its existing messages.
validate_outbound_http_url now resolves the hostname as httpx and urllib3
encode it (IDNA 2008). It previously let getaddrinfo apply the stdlib IDNA
2003 codec, which encodes characters such as "ß" differently, so a URL
could pass the check under one DNS name and be connected to under another.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
is_public_ip now takes an ipaddress object and relies on is_global, keeping
multicast and the NAT64 well-known prefix as explicit extra exclusions.
Adds OutboundRequestBlockedError and HostResolutionError, both picklable so
Celery keeps them intact on task failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing outside the documents app imports documents.tests any more.
Ruff now bans the module everywhere except src/documents/tests, where in-app imports remain fine.
The mail message and mailbox builders, the fake libmagic and the classifier preprocessor stub lived inside test_mail.py and test_classifier.py, so other test modules imported them by importing a test module.
They now live in helpers modules beside the tests that use them.
util_call_with_backoff made every call wait 20 seconds even when the first attempt succeeded
The helper now sleeps only after a failure that has another attempt left.
Test modules in paperless, paperless_mail and documents imported filesystem assertions, the migration test base, the retry helper and the streaming-response reader out of documents/tests/utils.py, which kept each app's tests coupled to another app's test package.
They now live in paperless_testing, and the progress manager fake is renamed FakeProgressManager and now subclasses the real ProgressManager, overriding only the transport, so the payload it records is built by the production code. The twenty places that patched documents.tasks.ProgressManager by hand now use a fake_progress_manager fixture.
Two fixtures created a temporary index directory and pointed INDEX_DIR at it, and paperless_dirs did the same, so a test that requested more than one got whichever assignment ran last. The search conftest no longer defines its own index_dir fixture, the tests that took it read paperless_dirs.index_dir instead, and _search_index is now a thin wrapper that requests paperless_dirs. The fixture that yields a Document is renamed from indexed_document to searchable_document so it no longer differs by one character from the index_document factory next to it.
TantivyBackend(path=None) built an in-memory index that is not used by production ever used. The backend now requires a path, the open, write-batch and rebuild branches are gone, and the shared backend fixture and the fulltext similar-documents fixture build a real index under the per-test directory layout.
* Chore: Speed up test setup by hashing passwords with MD5 and batching index writes
Don't use Django's default PBKDF2, about 600 ms per use and 110 uses across the suite. Switches to MD5 instead.
Also fixes a test that didn't batch update the search index
* Chore: Stop the invalid webhook params test from waiting on a Celery broker
test_workflow_webhook_action_url_invalid_params_headers left send_webhook.apply_async unpatched, so it tried to actually enqueu and waited for the timeout.
Nineteen single-file fixtures in the parsers conftest had no consumers anywhere in the test tree.
Two fixtures were both called samples_dir and resolved one directory apart They are now document_samples_dir and parser_samples_dir
Enables pytest-randomly, which has sat commented out in pyproject.toml
since the Pytest 9 upgrade. Tests now run in a different order every
session, so a test cannot quietly depend on another having run first.
The unittest side of the suite built its temp directory tree with
tempfile.mkdtemp and a manually enabled override_settings, cleaned up only if
tearDown ran. That is now gone. DirectoriesMixin lives alongside the layout it
bridges and does nothing but hand the paperless_dirs fixture to TestCase
subclasses as self.dirs, so both halves of the suite get the same twelve
settings, the same directory shapes and cleanup owned by tmp_path.
The mixin moves to paperless_testing.dirs rather than staying in the documents
test utilities, because modules in paperless and paperless_mail import it
across the app boundary. The thirty-eight consuming modules change only their
import line; self.dirs.scratch_dir and its siblings keep working.
The temp directory layout used by the tests was written out four separate
times: once in the documents conftest, once in the paperless checks tests,
once in a fixture local to the NFC upload tests, and once in the helper
behind the old paperless_environment context manager. Each copy covered a
different subset of the settings, so which directories a test actually got
depended on which copy it happened to reach.
The layout now lives in paperless_testing.dirs. build_paperless_dirs owns
where things go and creates them, dirs_settings maps them onto Django
setting names and is pure, and a paperless_dirs fixture in the root conftest
applies that mapping through pytest-django's settings fixture so every app
can reach it. Tests that need a second environment part way through a test
body use the paperless_environment context manager from the same module,
which expresses the identical layout through override_settings. The three
redundant implementations and the old media settings fixture are gone, and
their consumers now take paperless_dirs.
The model factories lived in the documents test package, but three other
apps needed them. The AI, mail and testing suites all reached across an app
boundary to import from documents.tests.factories, which made a private test
package into a shared dependency.
The factories now live in the shared testing package, where cross-app use is
the intended use.
All four Django apps have test code in common, but the only place to put it
was the documents app's own tests package, so paperless, paperless_ai and
paperless_mail each reached across an app boundary to import helpers and
relied on fixtures that were only defined for the documents test path.
This adds a root src/conftest.py holding the fixtures every app needs and an
new src/paperless_testing package for shared helpers a test names
Images with an out-of-spec EXIF orientation value (e.g. 0) fail
conversion with img2pdf.ExifOrientationError, which aborts archive
generation and, during consumption with OCR disabled, fails the whole
document.
Pass rotation=img2pdf.Rotation.ifvalid so invalid orientation values
are ignored while valid values (1, 3, 6, 8) are still applied.
Co-authored-by: zhzy0077 <zhzy0077@users.noreply.github.com>
* Fix: only pass --conf to flower when flowerconfig.py exists
The service passes --conf=${PAPERLESS_SRC_DIR}/paperless/flowerconfig.py
unconditionally, but the image does not ship that file. flower 2.0.1 tolerated
this because it ignored a missing config whose basename matched its own default
name; 2.1.0 dropped that basename() call (mher/flower#1391) and now raises
FileNotFoundError, so flower crash-loops and never binds 5555.
Pass the flag only when the file is present.
stumpylogandClaude Opus 5
GitHub Actions
shamoon
dependabot[bot]andstumpylog
Zhiyuan Zhengandzhzy0077
Bitfoo