Several tests in test_network.py and test_network_integration.py had
grown into near-copies of each other, differing only in the URL or
host they fed the guard while asserting the same thing. That made the
suite noisier to read and to extend than the behaviour it covers
warrants.
Merge the invalid-host rejection tests into one parametrized test over
all seven unusual host forms, and likewise for their allow_internal
counterparts that assert nothing is rejected. Merge the
resolve_public_addresses tests that only check the host reaches the
resolver unchanged, and the ones that check a private answer blocks
regardless of how the host was spelled, keeping the stricter resolver
call assertion on both merged cases. Add localhost as a fourth case to
the numeric-host-forms test in the transport integration tests, so the
name and the non-canonical numeric spellings of loopback are covered
by one test.
No behaviour or assertion strength changes; this only reduces
duplicated test bodies while keeping every original case addressable
by its own parametrize id.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six tests in test_network_integration.py and one in test_workflows.py each
patched paperless.network.is_public_ip inline to make every resolved
address count as public, so the guard's own address policy would not get
in the way of whatever the test was actually checking. The line was
repeated verbatim at every call site and understated what it did: read
literally it makes every address public, while each docstring around it
already explained the intent as loopback being treated as public.
Add an every_address_is_public fixture to conftest.py, following the
existing pattern for the other outbound test fixtures: a thin wrapper
that imports a new allow_all_addresses helper from
paperless_testing.outbound and calls it with mocker. Apply it at each
call site with pytest.mark.usefixtures, dropping the mocker parameter
from tests that no longer need it directly. The test that patches
is_public_ip with a selective side effect to allow only one address is
left untouched, since that selectivity is load-bearing for what it
verifies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two connect_tcp implementations each repeated the per-attempt budget
arithmetic verbatim, and only the sync one delegated resolution and error
mapping to a helper, so a drift between the copies could silently give one
stack a different timeout policy from the other. In the IMAP client,
_connect_pinned re-asserted a fact its only caller had already established,
which reads as a runtime invariant check on a security-relevant path when it
is only a narrowing aid.
Move the budget arithmetic into one helper called from both loops, add an
async twin of the resolve helper so the two loop bodies differ only by await,
and pass the narrowed address tuple into _connect_pinned instead of asserting
it. The PinnedIMAP4 docstring now spells out that no pinning and pinning that
yielded nothing are different things, and the monotonic clock seam says why it
exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Outbound host checks parsed IP literals themselves and skipped the
resolver for them. That second parser is what let a host such as
8.8.8.8%2eexample pass as the public address 8.8.8.8, and even with
zone ids limited to IPv6 it remains one more place where the checked
host can be read differently from the connected one.
The separate literal parsing is removed. Every host now goes to
getaddrinfo, which answers numeric literals itself without a lookup, and
only the addresses it returns are classified. Zone ids are still dropped
from the resolver's answers, where a scoped IPv6 literal comes back as
fe80::1%1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The literal check stripped everything after the first "%" from any host,
so a name such as 8.8.8.8%2e169-254-169-254.sslip.io was accepted as the
public address 8.8.8.8 without a DNS lookup. requests percent-decodes the
host before connecting, so Remote OCR would then resolve
8.8.8.8.169-254-169-254.sslip.io and reach an internal address even with
internal endpoints disallowed.
A zone id is now stripped only when the part before "%" is an IPv6
address; any other host containing "%" is looked up as a name. URL
validation with internal addresses disallowed also rejects a host that
contains "%" at all, since the name checked there could otherwise differ
from the one an HTTP client decodes and dials.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every consumer now uses the guarded transports, so the request-rewriting
transport and its helpers are no longer needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The environment proxy test could not fail for any regression in the guard:
httpx ignores environment proxy variables whenever a transport is passed
explicitly, and the address the test pointed the proxy at was itself
internal, so a blocked direct connection was indistinguishable from a
blocked proxied one. It now builds the client through the production
factory, which does not pass a transport, points the proxy variables at a
second recording server, and asserts the real origin server receives the
request while the proxy server sees no connection at all.
Several tests asserting a blocked request never reached the target server
checked only a connection counter that is incremented after accept() in
the server thread, which does not rule out a guard that connects and then
fails validation afterward. Each of those tests now also asserts the dial
recorder saw no address dialled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A local HTTP server, a per-hostname resolver fake and dial spies exercise
the guarded transports end to end: address fallback, blocking before any
connection, the Host header, TLS server name, per-host connection pooling,
numeric host spellings, environment proxies and redirects to blocked hosts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The transports install the outbound guard on httpcore's connection pool
after checking its exact layout, and accept no proxy, uds or retries
options. httpx, httpcore and anyio become declared dependencies, pinned
narrowly where private attributes are relied on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With internal addresses disallowed, the backend resolves the origin host,
rejects it if any address is non-public, and dials the validated literals
in family-interleaved order under the caller's connect timeout. Resolver
failures surface as connect errors; unix sockets are always refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
urllib3 treats a backslash as the end of the URL authority, while urlparse
and httpx do not. For a URL such as http://127.0.0.1\@evil.example/ the
check resolved evil.example while urllib3 would connect to 127.0.0.1, so a
redirect to such a URL could reach an internal host.
When internal addresses are disallowed, validate_outbound_http_url now
rejects any URL containing a backslash, an ASCII control character or
whitespace before resolving it. URLs validated with internal addresses
allowed are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
resolve_public_addresses and its async twin return every resolved address
in resolver order, de-duplicated and zone-stripped, and reject the whole
name if any address is non-public. validate_outbound_http_url uses them and
keeps its existing messages.
validate_outbound_http_url now resolves the hostname as httpx and urllib3
encode it (IDNA 2008). It previously let getaddrinfo apply the stdlib IDNA
2003 codec, which encodes characters such as "ß" differently, so a URL
could pass the check under one DNS name and be connected to under another.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
is_public_ip now takes an ipaddress object and relies on is_global, keeping
multicast and the NAT64 well-known prefix as explicit extra exclusions.
Adds OutboundRequestBlockedError and HostResolutionError, both picklable so
Celery keeps them intact on task failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Test modules in paperless, paperless_mail and documents imported filesystem assertions, the migration test base, the retry helper and the streaming-response reader out of documents/tests/utils.py, which kept each app's tests coupled to another app's test package.
They now live in paperless_testing, and the progress manager fake is renamed FakeProgressManager and now subclasses the real ProgressManager, overriding only the transport, so the payload it records is built by the production code. The twenty places that patched documents.tasks.ProgressManager by hand now use a fake_progress_manager fixture.
Nineteen single-file fixtures in the parsers conftest had no consumers anywhere in the test tree.
Two fixtures were both called samples_dir and resolved one directory apart They are now document_samples_dir and parser_samples_dir
The unittest side of the suite built its temp directory tree with
tempfile.mkdtemp and a manually enabled override_settings, cleaned up only if
tearDown ran. That is now gone. DirectoriesMixin lives alongside the layout it
bridges and does nothing but hand the paperless_dirs fixture to TestCase
subclasses as self.dirs, so both halves of the suite get the same twelve
settings, the same directory shapes and cleanup owned by tmp_path.
The mixin moves to paperless_testing.dirs rather than staying in the documents
test utilities, because modules in paperless and paperless_mail import it
across the app boundary. The thirty-eight consuming modules change only their
import line; self.dirs.scratch_dir and its siblings keep working.
The temp directory layout used by the tests was written out four separate
times: once in the documents conftest, once in the paperless checks tests,
once in a fixture local to the NFC upload tests, and once in the helper
behind the old paperless_environment context manager. Each copy covered a
different subset of the settings, so which directories a test actually got
depended on which copy it happened to reach.
The layout now lives in paperless_testing.dirs. build_paperless_dirs owns
where things go and creates them, dirs_settings maps them onto Django
setting names and is pure, and a paperless_dirs fixture in the root conftest
applies that mapping through pytest-django's settings fixture so every app
can reach it. Tests that need a second environment part way through a test
body use the paperless_environment context manager from the same module,
which expresses the identical layout through override_settings. The three
redundant implementations and the old media settings fixture are gone, and
their consumers now take paperless_dirs.
Images with an out-of-spec EXIF orientation value (e.g. 0) fail
conversion with img2pdf.ExifOrientationError, which aborts archive
generation and, during consumption with OCR disabled, fails the whole
document.
Pass rotation=img2pdf.Rotation.ifvalid so invalid orientation values
are ignored while valid values (1, 3, 6, 8) are still applied.
Co-authored-by: zhzy0077 <zhzy0077@users.noreply.github.com>
* Preprocesses classifier content with Tantivy instead of NLTK
Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.
Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.
* Copies packages instead of hardlinking them in backend CI, some NLTK thing
* Adds a normalization to NFC to better fit what Tantivy expects
JSONField(binary=True) accepts any JSON value, so a truthy non-dict
(bool/int/list/string) silently passed validation.
Additionally, require string substitutes in barcode_tag_mapping, a non-string
substitute value made the barcode tag substitution fail inside
get_tag_ids, and the tag was silently skipped. barcode_tag_mapping values
must now be strings (empty still means "use the raw barcode").
PR #12741 sets the result backend for Celery to Redis, but forget to carryover
transport option `global_keyprefix`. This resulted keys with prefix
`celery-task-meta-` prefix to be created.
This fix unbreaks strict Redis ACLs that allow a single prefix.
* Fix: mail rule loops forever when all attachments are duplicates
When every attachment in a mail is rejected as a duplicate, the chord's
header tasks all fail. Celery's default task_allow_error_cb_on_chord_header
skips the error callback in that case, so no ProcessedMail row is ever
created, and the same mail is refetched and reprocessed on every poll for
as long as it stays in the rule's maximum_age window.
* Minor simplifications and cleanup
* Security: validate remote OCR endpoint against internal SSRF
Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.
* Validates te outbound url again right before use
* cover empty-value branch of validate_remote_ocr_endpoint because coverage
* re-validate remote OCR endpoint on every outbound request
* uses_remote_service + allow_remote to allow opt-in / out of remote OCR
* Add to parser dev docs
* remote_ocr_mode config setting
* Checks for remote_ocr_mode and fix import
* Update config.component.spec.ts
* More tests for remote_ocr_mode
* Docs for remote_ocr_mode
* Ok, wire up the remote_ocr_mode with allow_remote for consumer
* Update consumer.py
* Format remote OCR mode check tests
* Use get_choice_from_env
* Backend changes and migration for remote OCR Config
* Backend tests
* Frontend stuff, with sections
* Docs
* Update test_tesseract_parser.py
* Actually we cant use this any more, in case settings are in app config
* Dont mark entire test file for db, use a mock for empty engine settings
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks
* Updates the documentation to be correct about the respecting now
* merge conflict fixing
stumpylogandClaude Opus 5
Zhiyuan Zhengandzhzy0077
shamoon
Philipp Defner
Berk D. Demir
dependabot[bot]andTrenton Holmes
Thomas Steinbachandshamoon
Sebastian Wielandandshamoon
