Compare commits

..
Author SHA1 Message Date
stumpylogandClaude Opus 5 296ddff37e Collapse near-duplicate outbound guard tests into parametrized cases
Several tests in test_network.py and test_network_integration.py had
grown into near-copies of each other, differing only in the URL or
host they fed the guard while asserting the same thing. That made the
suite noisier to read and to extend than the behaviour it covers
warrants.

Merge the invalid-host rejection tests into one parametrized test over
all seven unusual host forms, and likewise for their allow_internal
counterparts that assert nothing is rejected. Merge the
resolve_public_addresses tests that only check the host reaches the
resolver unchanged, and the ones that check a private answer blocks
regardless of how the host was spelled, keeping the stricter resolver
call assertion on both merged cases. Add localhost as a fourth case to
the numeric-host-forms test in the transport integration tests, so the
name and the non-canonical numeric spellings of loopback are covered
by one test.

No behaviour or assertion strength changes; this only reduces
duplicated test bodies while keeping every original case addressable
by its own parametrize id.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 13:49:35 -07:00
stumpylogandClaude Opus 5 de8edc31c5 Fold the outbound guard's public-address patch into a named fixture
Six tests in test_network_integration.py and one in test_workflows.py each
patched paperless.network.is_public_ip inline to make every resolved
address count as public, so the guard's own address policy would not get
in the way of whatever the test was actually checking. The line was
repeated verbatim at every call site and understated what it did: read
literally it makes every address public, while each docstring around it
already explained the intent as loopback being treated as public.

Add an every_address_is_public fixture to conftest.py, following the
existing pattern for the other outbound test fixtures: a thin wrapper
that imports a new allow_all_addresses helper from
paperless_testing.outbound and calls it with mocker. Apply it at each
call site with pytest.mark.usefixtures, dropping the mocker parameter
from tests that no longer need it directly. The test that patches
is_public_ip with a selective side effect to allow only one address is
left untouched, since that selectivity is load-bearing for what it
verifies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:56:55 -07:00
stumpylogandClaude Opus 5 46c150eb67 Deduplicate the outbound connect guard's sync and async paths
The two connect_tcp implementations each repeated the per-attempt budget
arithmetic verbatim, and only the sync one delegated resolution and error
mapping to a helper, so a drift between the copies could silently give one
stack a different timeout policy from the other. In the IMAP client,
_connect_pinned re-asserted a fact its only caller had already established,
which reads as a runtime invariant check on a security-relevant path when it
is only a narrowing aid.

Move the budget arithmetic into one helper called from both loops, add an
async twin of the resolve helper so the two loop bodies differ only by await,
and pass the narrowed address tuple into _connect_pinned instead of asserting
it. The PinnedIMAP4 docstring now spells out that no pinning and pinning that
yielded nothing are different things, and the monotonic clock seam says why it
exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:06 -07:00
stumpylogandClaude Opus 5 58777076c5 Cover webhooks to internal hosts when they are allowed
Every webhook test that reaches a real socket ran with internal requests
disallowed, and the rest intercept above the transport. Wiring the
transport to always disallow internal addresses would therefore have
passed the suite while breaking webhooks to internal hosts on every
default install.

A new test sends a webhook to localhost with internal requests allowed
and checks that the payload arrives and that the guard resolved nothing.
The Host header test's docstring is also corrected: the header now comes
from the URL, not from a resolved hostname.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:06 -07:00
stumpylogandClaude Opus 5 d414297c05 Fix: Validate IP literals from the resolver's answer only
Outbound host checks parsed IP literals themselves and skipped the
resolver for them. That second parser is what let a host such as
8.8.8.8%2eexample pass as the public address 8.8.8.8, and even with
zone ids limited to IPv6 it remains one more place where the checked
host can be read differently from the connected one.

The separate literal parsing is removed. Every host now goes to
getaddrinfo, which answers numeric literals itself without a lookup, and
only the addresses it returns are classified. Zone ids are still dropped
from the resolver's answers, where a scoped IPv6 literal comes back as
fe80::1%1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:06 -07:00
stumpylogandClaude Opus 5 d53a930aed Fix: Stop treating a dotted quad followed by "%" as an IP literal
The literal check stripped everything after the first "%" from any host,
so a name such as 8.8.8.8%2e169-254-169-254.sslip.io was accepted as the
public address 8.8.8.8 without a DNS lookup. requests percent-decodes the
host before connecting, so Remote OCR would then resolve
8.8.8.8.169-254-169-254.sslip.io and reach an internal address even with
internal endpoints disallowed.

A zone id is now stripped only when the part before "%" is an IPv6
address; any other host containing "%" is looked up as a name. URL
validation with internal addresses disallowed also rejects a host that
contains "%" at all, since the name checked there could otherwise differ
from the one an HTTP client decodes and dials.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 78ca920771 Document outbound connection policy for internal-address settings
The internal-address settings now describe that a hostname is blocked if any resolved address is non-public. Guarded requests connect directly without proxy variables, and webhook requests never follow redirects.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 4c5255a8dd Remove the URL-rewriting pinned transport
Every consumer now uses the guarded transports, so the request-rewriting
transport and its helpers are no longer needed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 283e49b1ee Match get_mailbox annotations to what callers actually pass
get_mailbox declared port and security as always-present types, but
MailAccount.imap_port is nullable and imap_security is stored as a plain
integer, so type checking flagged every call site as passing the wrong
type.

Widen the annotations to port: int | None and security: int, matching the
model fields; IntegerChoices members still compare equal to plain ints, so
the existing branching in get_mailbox is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 3130fc3a7c Use shared outbound resolution for IMAP host pinning
get_mailbox validates the IMAP host with resolve_public_addresses and keeps
its existing error messages; the pinned client dials typed addresses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 8f56c6167b Use the guarded transport for workflow webhooks
A blocked webhook now raises OutboundRequestBlockedError, which the task
treats as an expected failure and does not retry. The webhook security
tests run against a real local server instead of a patched resolver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:48:44 -07:00
stumpylogandClaude Opus 5 eb644f3fac Report outbound policy blocks from AI requests as 502
AIClient raises LLMBlockedError when a request was refused by the outbound
connection policy, including when the openai SDK wraps the block in
APIConnectionError. ai_suggestions answers 502 instead of a 500.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:48:44 -07:00
stumpylogandClaude Opus 5 6ad00ca55a Use guarded transports for AI LLM and embedding clients
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:48:21 -07:00
stumpylogandClaude Opus 5 526adbad1a Fix two weak assertions in the outbound guard integration tests
The environment proxy test could not fail for any regression in the guard:
httpx ignores environment proxy variables whenever a transport is passed
explicitly, and the address the test pointed the proxy at was itself
internal, so a blocked direct connection was indistinguishable from a
blocked proxied one. It now builds the client through the production
factory, which does not pass a transport, points the proxy variables at a
second recording server, and asserts the real origin server receives the
request while the proxy server sees no connection at all.

Several tests asserting a blocked request never reached the target server
checked only a connection counter that is incremented after accept() in
the server thread, which does not rule out a guard that connects and then
fails validation afterward. Each of those tests now also asserts the dial
recorder saw no address dialled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:48:21 -07:00
stumpylogandClaude Opus 5 d53453fb71 Add real-socket tests for the outbound connection guard
A local HTTP server, a per-hostname resolver fake and dial spies exercise
the guarded transports end to end: address fallback, blocking before any
connection, the Host header, TLS server name, per-host connection pooling,
numeric host spellings, environment proxies and redirects to blocked hosts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:48:20 -07:00
stumpylogandClaude Opus 5 4eaf8032ad Add guarded httpx transports and client factories
The transports install the outbound guard on httpcore's connection pool
after checking its exact layout, and accept no proxy, uds or retries
options. httpx, httpcore and anyio become declared dependencies, pinned
narrowly where private attributes are relied on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:47:40 -07:00
stumpylogandClaude Opus 5 9e1f938d56 Guard outbound connections in the httpcore network backend
With internal addresses disallowed, the backend resolves the origin host,
rejects it if any address is non-public, and dials the validated literals
in family-interleaved order under the caller's connect timeout. Resolver
failures surface as connect errors; unix sockets are always refused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:47:40 -07:00
stumpylogandClaude Opus 5 0da50ad348 Reject outbound URLs that HTTP clients may split differently
urllib3 treats a backslash as the end of the URL authority, while urlparse
and httpx do not. For a URL such as http://127.0.0.1\@evil.example/ the
check resolved evil.example while urllib3 would connect to 127.0.0.1, so a
redirect to such a URL could reach an internal host.

When internal addresses are disallowed, validate_outbound_http_url now
rejects any URL containing a backslash, an ASCII control character or
whitespace before resolving it. URLs validated with internal addresses
allowed are unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:47:40 -07:00
stumpylogandClaude Opus 5 5e971bc0ce Resolve outbound hosts to validated public addresses
resolve_public_addresses and its async twin return every resolved address
in resolver order, de-duplicated and zone-stripped, and reject the whole
name if any address is non-public. validate_outbound_http_url uses them and
keeps its existing messages.

validate_outbound_http_url now resolves the hostname as httpx and urllib3
encode it (IDNA 2008). It previously let getaddrinfo apply the stdlib IDNA
2003 codec, which encodes characters such as "ß" differently, so a URL
could pass the check under one DNS name and be connected to under another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:47:40 -07:00
stumpylogandClaude Opus 5 17a91c385d Type outbound block errors and classify addresses with is_global
is_public_ip now takes an ipaddress object and relies on is_global, keeping
multicast and the NAT64 well-known prefix as explicit extra exclusions.
Adds OutboundRequestBlockedError and HostResolutionError, both picklable so
Celery keeps them intact on task failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:47:40 -07:00
Trenton H 15b73b890c Chore: Ban cross-app imports of documents.tests (#14224)
Nothing outside the documents app imports documents.tests any more.

Ruff now bans the module everywhere except src/documents/tests, where in-app imports remain fine.
2026-09-22 08:02:18 -07:00
Trenton H 02d355061f Chore: Move helpers that other test modules imported out of test files (#14223)
The mail message and mailbox builders, the fake libmagic and the classifier preprocessor stub lived inside test_mail.py and test_classifier.py, so other test modules imported them by importing a test module.

They now live in helpers modules beside the tests that use them.
2026-09-22 08:02:18 -07:00
Trenton H d8b5b4d447 Chore: Stop the live-service retry helper sleeping after a success (#14222)
util_call_with_backoff made every call wait 20 seconds even when the first attempt succeeded

The helper now sleeps only after a failure that has another attempt left.
2026-09-22 08:02:18 -07:00
Trenton H 03ac4aed7e Chore: Move the cross-app test helpers into the shared layer and add a progress fixture (#14221)
Test modules in paperless, paperless_mail and documents imported filesystem assertions, the migration test base, the retry helper and the streaming-response reader out of documents/tests/utils.py, which kept each app's tests coupled to another app's test package.

They now live in paperless_testing, and the progress manager fake is renamed FakeProgressManager and now subclasses the real ProgressManager, overriding only the transport, so the payload it records is built by the production code. The twenty places that patched documents.tasks.ProgressManager by hand now use a fake_progress_manager fixture.
2026-09-22 08:02:17 -07:00
Trenton H 90d23bad9c Chore: Give the search index directory one owner in the tests (#14217)
Two fixtures created a temporary index directory and pointed INDEX_DIR at it, and paperless_dirs did the same, so a test that requested more than one got whichever assignment ran last. The search conftest no longer defines its own index_dir fixture, the tests that took it read paperless_dirs.index_dir instead, and _search_index is now a thin wrapper that requests paperless_dirs. The fixture that yields a Document is renamed from indexed_document to searchable_document so it no longer differs by one character from the index_document factory next to it.
2026-09-22 08:02:17 -07:00
Trenton H e4367b5648 Chore: Delete the in-memory Tantivy backend and run the search tests on disk (#14216)
TantivyBackend(path=None) built an in-memory index that is not used by production ever used. The backend now requires a path, the open, write-batch and rebuild branches are gone, and the shared backend fixture and the fulltext similar-documents fixture build a real index under the per-test directory layout.
2026-09-22 08:02:16 -07:00
Trenton H cb85441c2f Chore: Decrease backend test suite time (a little) (#14213)
* Chore: Speed up test setup by hashing passwords with MD5 and batching index writes

Don't use Django's default PBKDF2, about 600 ms per use and 110 uses across the suite. Switches to MD5 instead.

Also fixes a test that didn't batch update the search index

* Chore: Stop the invalid webhook params test from waiting on a Celery broker

test_workflow_webhook_action_url_invalid_params_headers left send_webhook.apply_async unpatched, so it tried to actually enqueu and waited for the timeout.
2026-09-21 20:17:40 +00:00
Trenton H cceaa559d4 Chore: Rename sample directory fixtures and drop unused parser ones (#14215)
Nineteen single-file fixtures in the parsers conftest had no consumers anywhere in the test tree.

Two fixtures were both called samples_dir and resolved one directory apart They are now document_samples_dir and parser_samples_dir
2026-09-21 12:59:32 -07:00
GitHub Actions a748d4c64f Auto translate strings 2026-09-21 19:01:49 +00:00
shamoon 452ed005bd Fix: handle legacy bulk edit page range with missing page_count (#14212) 2026-09-21 19:00:14 +00:00
Trenton H 40058ff7d5 Chore: Give the test suite a larger regex timeout (#14211)
Maybe the random ordering sometimes causes heavier tests to run alongside timed regex ones?
2026-09-21 11:38:54 -07:00
Trenton H f502cd5e34 Chore: Add shared helpers for granting test permissions and use them (#14200) 2026-09-21 10:57:14 -07:00
Trenton H 99ce6b5db3 Chore: Share the API client fixtures and build test users cheaply (#14199) 2026-09-21 10:57:14 -07:00
Trenton H 2d955e9697 Chore: Randomize test order and seed Faker per run (#14173)
Enables pytest-randomly, which has sat commented out in pyproject.toml
since the Pytest 9 upgrade. Tests now run in a different order every
session, so a test cannot quietly depend on another having run first.
2026-09-21 10:57:13 -07:00
Trenton H f440e8d33c Chore: Move unittest directory setup onto the shared fixture (#14172)
The unittest side of the suite built its temp directory tree with
tempfile.mkdtemp and a manually enabled override_settings, cleaned up only if
tearDown ran. That is now gone. DirectoriesMixin lives alongside the layout it
bridges and does nothing but hand the paperless_dirs fixture to TestCase
subclasses as self.dirs, so both halves of the suite get the same twelve
settings, the same directory shapes and cleanup owned by tmp_path.

The mixin moves to paperless_testing.dirs rather than staying in the documents
test utilities, because modules in paperless and paperless_mail import it
across the app boundary. The thirty-eight consuming modules change only their
import line; self.dirs.scratch_dir and its siblings keep working.
2026-09-21 10:57:12 -07:00
Trenton H 4c264651e8 Chore: State the test directory layout in one place (#14171)
The temp directory layout used by the tests was written out four separate
times: once in the documents conftest, once in the paperless checks tests,
once in a fixture local to the NFC upload tests, and once in the helper
behind the old paperless_environment context manager. Each copy covered a
different subset of the settings, so which directories a test actually got
depended on which copy it happened to reach.

The layout now lives in paperless_testing.dirs. build_paperless_dirs owns
where things go and creates them, dirs_settings maps them onto Django
setting names and is pure, and a paperless_dirs fixture in the root conftest
applies that mapping through pytest-django's settings fixture so every app
can reach it. Tests that need a second environment part way through a test
body use the paperless_environment context manager from the same module,
which expresses the identical layout through override_settings. The three
redundant implementations and the old media settings fixture are gone, and
their consumers now take paperless_dirs.
2026-09-21 10:57:11 -07:00
Trenton H 48d97b78bb Chore: Move model factories to the shared test layer (#14170)
The model factories lived in the documents test package, but three other
apps needed them. The AI, mail and testing suites all reached across an app
boundary to import from documents.tests.factories, which made a private test
package into a shared dependency.

The factories now live in the shared testing package, where cross-app use is
the intended use.
2026-09-21 10:57:11 -07:00
Trenton H 659a0cb2ef Chore: Add a shared test support layer (#14168)
All four Django apps have test code in common, but the only place to put it
was the documents app's own tests package, so paperless, paperless_ai and
paperless_mail each reached across an app boundary to import helpers and
relied on fixtures that were only defined for the documents test path.

This adds a root src/conftest.py holding the fixtures every app needs and an
new src/paperless_testing package for shared helpers a test names
2026-09-21 10:57:11 -07:00
Trenton H d02d1e1711 Chore: Remove unused test helpers (#14166) 2026-09-21 10:57:10 -07:00
dependabot[bot]andstumpylog 3b41810e7b Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 2 updates (#14133)
* Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 2 updates

Bumps the pre-commit-dependencies group with 2 updates in the / directory: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit) and [https://github.com/tox-dev/pyproject-fmt](https://github.com/tox-dev/pyproject-fmt).


Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.16.5 to 0.16.7
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](https://github.com/astral-sh/ruff-pre-commit/compare/v0.16.5...v0.16.7)

Updates `https://github.com/tox-dev/pyproject-fmt` from v2.28.1 to 2.29.4
- [Release notes](https://github.com/tox-dev/pyproject-fmt/releases)
- [Commits](https://github.com/tox-dev/pyproject-fmt/compare/v2.28.1...v2.29.4)

---
updated-dependencies:
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.16.6
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/tox-dev/pyproject-fmt
  dependency-version: 2.29.4
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Runs new formatting

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
2026-09-21 07:49:12 -07:00
Zhiyuan Zhengandzhzy0077 12314fcaa8 Fix: ignore invalid EXIF orientation when generating image archives (#14203)
Images with an out-of-spec EXIF orientation value (e.g. 0) fail
conversion with img2pdf.ExifOrientationError, which aborts archive
generation and, during consumption with OCR disabled, fails the whole
document.

Pass rotation=img2pdf.Rotation.ifvalid so invalid orientation values
are ignored while valid values (1, 3, 6, 8) are still applied.

Co-authored-by: zhzy0077 <zhzy0077@users.noreply.github.com>
2026-09-21 07:21:24 -07:00
129 changed files with 5744 additions and 3147 deletions
+2
View File
@@ -15,6 +15,8 @@
# Test related
**/.pytest_cache
**/tests
src/paperless_testing
src/conftest.py
**/*.spec.ts
**/htmlcov
# Local folders
+2 -2
View File
@@ -50,12 +50,12 @@ repos:
- 'prettier-plugin-organize-imports@4.3.0'
# Python hooks
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.5
rev: v0.16.7
hooks:
- id: ruff-check
- id: ruff-format
- repo: https://github.com/tox-dev/pyproject-fmt
rev: "v2.28.1"
rev: "v2.29.4"
hooks:
- id: pyproject-fmt
additional_dependencies: [tomli]
+6 -1
View File
@@ -1576,6 +1576,9 @@ ports.
#### [`PAPERLESS_WEBHOOKS_ALLOW_INTERNAL_REQUESTS=<bool>`](#PAPERLESS_WEBHOOKS_ALLOW_INTERNAL_REQUESTS) {#PAPERLESS_WEBHOOKS_ALLOW_INTERNAL_REQUESTS}
: If set to false, webhooks cannot be sent to internal URLs (e.g., localhost).
A hostname is blocked if any of the addresses it resolves to is non-public.
Webhook requests connect directly, without using the `HTTP_PROXY` or
`HTTPS_PROXY` environment variables, and never follow redirects.
Defaults to true, which allows internal requests.
@@ -1584,7 +1587,7 @@ ports.
#### [`PAPERLESS_EMAIL_ALLOW_INTERNAL_HOSTS=<bool>`](#PAPERLESS_EMAIL_ALLOW_INTERNAL_HOSTS) {#PAPERLESS_EMAIL_ALLOW_INTERNAL_HOSTS}
: If set to false, incoming mail account connections are blocked when the
configured IMAP hostname resolves to a non-public address (for example,
configured IMAP hostname resolves to any non-public address (for example,
localhost, link-local, or RFC1918 private ranges).
Defaults to true, which allows internal hosts.
@@ -2214,6 +2217,8 @@ used with the OpenAI-compatible backend to target a custom provider or local gat
#### [`PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS=<bool>`](#PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS) {#PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS}
: If set to false, Paperless blocks AI endpoint URLs that resolve to non-public addresses (e.g., localhost, etc).
A hostname is blocked if any of the addresses it resolves to is non-public, and redirects are checked the same way.
Requests to a configured AI endpoint connect directly, without using the `HTTP_PROXY` or `HTTPS_PROXY` environment variables.
Defaults to true, which allows internal endpoints.
+1
View File
@@ -150,6 +150,7 @@ pnpm ng build --configuration production
is loaded as well. However, the tests rely on the default
configuration. This is not ideal. But for now, make sure no settings
except for DEBUG are overridden when testing.
- Tests run in a random order each session, so that one test cannot quietly depend on another having run first. The seed is printed at the top of the run; pass `--randomly-seed=<seed>` to replay that exact order, or `--randomly-seed=last` to repeat the previous run.
!!! note
+1 -3
View File
@@ -76,9 +76,7 @@ is not supported by any of the available parsers.
**A:** Not by default. As of v3, a file whose contents match an existing document is still
consumed, and the duplicate is flagged in the UI — open the document and check the
**Duplicates** tab to review documents that share the same content, or filter the document
list by **Duplicates** to find all of them (see
[Duplicate documents](usage.md#duplicate-documents)). If you prefer the old
**Duplicates** tab to review documents that share the same content. If you prefer the old
behavior of rejecting duplicates during consumption, set
[`PAPERLESS_CONSUMER_DELETE_DUPLICATES`](configuration.md#PAPERLESS_CONSUMER_DELETE_DUPLICATES)
to `true`.
+9 -8
View File
@@ -299,18 +299,19 @@ for details.
### Duplicate documents
By default, Paperless-ngx **does not reject duplicates**. If you consume a file whose
contents match an existing document (same original or archive checksum), the new copy is
still consumed and a warning is logged.
contents exactly match an existing document (same checksum), the new copy is still
consumed and a warning is logged. The task entry for the upload also flags that a
duplicate was detected and links to the existing document(s).
When a document has duplicates, a **Duplicates** tab appears on its detail page, listing
the other documents you can view that share the same content (including any in the trash).
To find all documents with duplicates, choose **Duplicates** in the document list's text
filter dropdown, or use `has_duplicates=true` in the REST API.
To review duplicates, open a document and switch to the **Duplicates** tab on the
document detail page. It lists other documents that share the same content, including any
that are in the trash (shown with a badge), and links to each so you can decide which to
keep.
If you would rather reject duplicates at consumption time (the pre-v3 behavior), set
[`PAPERLESS_CONSUMER_DELETE_DUPLICATES`](configuration.md#PAPERLESS_CONSUMER_DELETE_DUPLICATES)
to `true`. The duplicate file is then deleted instead of consumed, and the task fails with
a "Document already exists" message linking to the existing document.
a "document already exists" message.
## Document Suggestions
@@ -612,7 +613,7 @@ The following workflow action types are available:
- The request headers as key-value pairs
For security reasons, webhooks can be limited to specific ports and disallowed from connecting to local URLs. See the relevant
[configuration settings](configuration.md#workflow-webhooks) to change this behavior. If you are allowing non-admins to create workflows,
[configuration settings](configuration.md#workflow-webhooks) to change this behavior. Webhook requests connect directly (proxy environment variables are not used) and do not follow redirects. If you are allowing non-admins to create workflows,
you may want to adjust these settings to prevent abuse.
##### Move to Trash {#workflow-action-move-to-trash}
+19 -5
View File
@@ -1,7 +1,9 @@
[project]
name = "paperless-ngx"
version = "3.2.1"
description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
description = """\
A community-supported supercharged document management system: scan, index and archive all your physical documents\
"""
readme = "README.md"
requires-python = ">=3.11"
classifiers = [
@@ -10,10 +12,12 @@ classifiers = [
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.15",
]
# TODO: Move certain things to groups and then utilize that further
# This will allow testing to not install a webserver, mysql, etc
dependencies = [
"anyio>=4.12",
"azure-ai-documentintelligence>=1.0.2",
"babel>=2.17",
"bleach~=6.4.0",
@@ -44,6 +48,8 @@ dependencies = [
"filelock~=3.32.0",
"flower>=2.0.1,<2.2",
"gotenberg-client[httpx]~=1.0",
"httpcore~=1.0.9",
"httpx~=0.28.1",
"httpx-oauth~=0.17",
"ijson>=3.5.1",
"imap-tools>=1.14,<1.16",
@@ -117,7 +123,7 @@ testing = [
"pytest-env~=1.7.0",
"pytest-httpx",
"pytest-mock~=3.15.1",
# "pytest-randomly~=4.0.1",
"pytest-randomly~=5.0.0",
"pytest-rerunfailures~=16.4",
"pytest-sugar",
"pytest-xdist~=3.8.0",
@@ -244,6 +250,10 @@ per-file-ignores."docker/wait-for-redis.py" = [
per-file-ignores."src/documents/models.py" = [
"SIM115",
]
per-file-ignores."src/documents/tests/*.py" = [
"TID251",
]
flake8-tidy-imports.banned-api."documents.tests".msg = "Shared test infrastructure lives in src/paperless_testing/."
isort.force-single-line = true
[tool.codespell]
@@ -271,9 +281,9 @@ plugins = [
]
[tool.pyrefly]
baseline = ".pyrefly-baseline.json"
python-platform = "linux"
search-path = [ "src" ]
baseline = ".pyrefly-baseline.json"
[tool.django-stubs]
django_settings_module = "paperless.settings"
@@ -326,6 +336,8 @@ PAPERLESS_CACHE_BACKEND = "django.core.cache.backends.locmem.LocMemCache"
PAPERLESS_CHANNELS_BACKEND = "channels.layers.InMemoryChannelLayer"
# I don't think anything hits this, but just in case, basically infinite
PAPERLESS_TOKEN_THROTTLE_RATE = "1000/min"
# The 0.1s production default trips on a stalled CI runner, the date parsing tests then find no dates
PAPERLESS_MATCH_REGEX_TIMEOUT_SECONDS = "5"
[tool.coverage.run]
source = [
@@ -334,13 +346,15 @@ source = [
omit = [
"*/tests/*",
"manage.py",
"paperless/wsgi.py",
"paperless/auth.py",
"paperless/wsgi.py",
"src/conftest.py",
"src/paperless_testing/*",
]
[tool.coverage.report]
exclude_also = [
"if settings.AUDIT_LOG_ENABLED:",
"if AUDIT_LOG_ENABLED:",
"if settings.AUDIT_LOG_ENABLED:",
"if TYPE_CHECKING:",
]
+193
View File
@@ -0,0 +1,193 @@
"""Fixtures available to every Paperless-ngx app.
Loaded automatically for every test path. Keep module-scope imports minimal:
this file is imported for every session, so anything heavy belongs inside
the fixture body that needs it.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
if TYPE_CHECKING:
from collections.abc import Generator
from pathlib import Path
from django.contrib.auth.models import User
from pytest_django.fixtures import Settings
from pytest_mock import MockerFixture
from rest_framework.test import APIClient
from paperless_testing.dirs import PaperlessDirs
from paperless_testing.fakes.progress import FakeProgressManager
from paperless_testing.outbound import DialRecorder
from paperless_testing.outbound import FakeDNS
from paperless_testing.outbound import LocalHTTPServer
@pytest.fixture(scope="session", autouse=True)
def faker_session_locale() -> str:
"""Pin Faker's locale so generated data does not follow the host locale.
The seed itself is left to pytest-randomly, which derives one per run.
"""
return "en_US"
@pytest.fixture(autouse=True)
def _fast_password_hasher(settings: Settings) -> None:
"""Hash test passwords with MD5 instead of Django's default PBKDF2.
PBKDF2 is deliberately slow, and every ``admin_user`` or
``create_superuser`` call pays for it: about 600 ms each. No test depends
on the hash format, only on ``check_password`` and on the stored value
changing when the password does.
"""
settings.PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]
@pytest.fixture(autouse=True)
def _clear_content_type_caches() -> None:
"""Clear Django's ContentType cache and guardian's lru_cache before each test.
Tests that delete and reinsert ContentType/Permission rows (e.g. the
importer) corrupt both caches. Without this fixture a subsequent test on
the same xdist worker sees stale ContentType objects and guardian raises
MixedContentTypeError.
"""
from django.contrib.contenttypes.models import ContentType
from guardian.shortcuts import clear_ct_cache
ContentType.objects.clear_cache()
clear_ct_cache()
@pytest.fixture(autouse=True)
def _clear_django_caches() -> None:
"""Clear every configured cache before each test.
Cached values outlive the test that wrote them: the classifier keys its
vectorized content on a hash of the content itself, so a second test
generating the same fixture data takes the cache-hit path and never calls
the code it is asserting against.
"""
from django.core.cache import caches
for cache in caches.all(initialized_only=False):
cache.clear()
@pytest.fixture
def paperless_dirs(
tmp_path: Path,
settings: Settings,
) -> Generator[PaperlessDirs, None, None]:
"""The standard temp directory layout, applied to Django settings."""
from documents.search import reset_backend
from paperless_testing.dirs import build_paperless_dirs
from paperless_testing.dirs import dirs_settings
dirs = build_paperless_dirs(tmp_path)
for name, value in dirs_settings(dirs).items():
setattr(settings, name, value)
# Not directory settings, but they are needed alongside the layout by the
# sanity checker tests.
settings.IGNORABLE_FILES = {".DS_Store", "Thumbs.db", "desktop.ini"}
settings.APP_LOGO = ""
reset_backend()
yield dirs
reset_backend()
@pytest.fixture
def rest_api_client() -> APIClient:
"""The basic DRF APIClient, unauthenticated."""
from rest_framework.test import APIClient
return APIClient()
@pytest.fixture
def regular_user(db: None) -> User:
"""Unprivileged user for permission boundary tests."""
from paperless_testing.factories import UserFactory
return UserFactory(username="regular")
@pytest.fixture
def admin_client(rest_api_client: APIClient, admin_user: User) -> APIClient:
"""Admin client pre-authenticated and sending the v10 Accept header."""
rest_api_client.force_authenticate(user=admin_user)
rest_api_client.credentials(HTTP_ACCEPT="application/json; version=10")
return rest_api_client
@pytest.fixture
def v9_client(rest_api_client: APIClient, admin_user: User) -> APIClient:
"""Admin client pre-authenticated and sending the v9 Accept header."""
rest_api_client.force_authenticate(user=admin_user)
rest_api_client.credentials(HTTP_ACCEPT="application/json; version=9")
return rest_api_client
@pytest.fixture
def user_client(rest_api_client: APIClient, regular_user: User) -> APIClient:
"""Regular-user client pre-authenticated and sending the v10 Accept header."""
rest_api_client.force_authenticate(user=regular_user)
rest_api_client.credentials(HTTP_ACCEPT="application/json; version=10")
return rest_api_client
@pytest.fixture
def fake_progress_manager(
monkeypatch: pytest.MonkeyPatch,
) -> type[FakeProgressManager]:
"""Replace documents.tasks.ProgressManager with the fake, so consuming a file
in a test never tries to reach a broker."""
from paperless_testing.fakes.progress import FakeProgressManager
monkeypatch.setattr("documents.tasks.ProgressManager", FakeProgressManager)
return FakeProgressManager
@pytest.fixture
def local_http_server() -> Generator[LocalHTTPServer, None, None]:
"""A recording HTTP server on 127.0.0.1, for outbound connection tests."""
from paperless_testing.outbound import running_http_server
with running_http_server() as server:
yield server
@pytest.fixture
def fake_dns(mocker: MockerFixture) -> FakeDNS:
"""Per-hostname answers for the outbound guard's resolver hooks."""
from paperless_testing.outbound import install_fake_dns
return install_fake_dns(mocker)
@pytest.fixture
def dial_recorder(mocker: MockerFixture) -> DialRecorder:
"""Records which addresses the outbound guard actually dialled."""
from paperless_testing.outbound import install_dial_recorder
return install_dial_recorder(mocker)
@pytest.fixture
def every_address_is_public(mocker: MockerFixture) -> None:
"""Disable the outbound guard's address policy: every address passes.
For tests that are not themselves exercising which addresses the guard
accepts, so loopback and other private addresses dial just like a
public one.
"""
from paperless_testing.outbound import allow_all_addresses
allow_all_addresses(mocker)
+50 -63
View File
@@ -196,52 +196,49 @@ class WriteBatch:
return self._raw_writer
def __enter__(self) -> Self:
if self._backend._path is not None:
lock_path = self._backend._path / ".tantivy.lock"
self._lock = filelock.FileLock(str(lock_path))
for attempt in range(_LOCK_RETRY_ATTEMPTS):
try:
self._lock.acquire(timeout=self._lock_timeout)
break
except filelock.Timeout:
if attempt == _LOCK_RETRY_ATTEMPTS - 1:
raise SearchIndexLockError(
f"Could not acquire index lock after {_LOCK_RETRY_ATTEMPTS} "
f"attempts (timeout={self._lock_timeout}s each)",
)
sleep_s = random.uniform(
0,
min(_LOCK_BACKOFF_CAP, _LOCK_BACKOFF_BASE * (2**attempt)),
lock_path = self._backend._path / ".tantivy.lock"
self._lock = filelock.FileLock(str(lock_path))
for attempt in range(_LOCK_RETRY_ATTEMPTS):
try:
self._lock.acquire(timeout=self._lock_timeout)
break
except filelock.Timeout:
if attempt == _LOCK_RETRY_ATTEMPTS - 1:
raise SearchIndexLockError(
f"Could not acquire index lock after {_LOCK_RETRY_ATTEMPTS} "
f"attempts (timeout={self._lock_timeout}s each)",
)
logger.debug(
"Index lock contention; retrying in %.2fs (attempt %d/%d)",
sleep_s,
attempt + 1,
_LOCK_RETRY_ATTEMPTS,
)
time.sleep(sleep_s)
sleep_s = random.uniform(
0,
min(_LOCK_BACKOFF_CAP, _LOCK_BACKOFF_BASE * (2**attempt)),
)
logger.debug(
"Index lock contention; retrying in %.2fs (attempt %d/%d)",
sleep_s,
attempt + 1,
_LOCK_RETRY_ATTEMPTS,
)
time.sleep(sleep_s)
# Open a fresh Index (and thus a fresh Tantivy ManagedDirectory)
# for the write, rather than reusing the process-local cached
# index. ManagedDirectory loads its GC bookkeeping (.managed.json)
# once, at construction, and never re-reads it; paperless runs
# several long-lived processes (Granian workers, Celery workers)
# that take turns writing under the file lock above. A cached,
# long-lived writer index would carry a stale managed-files view
# and, on commit, overwrite .managed.json with that stale view -
# permanently losing track of segment files other processes
# registered in the meantime, so they can never be garbage
# collected. Reopening fresh here always picks up the current
# on-disk state. The long-lived self._backend._index is used for
# reads only and is reloaded (not reopened) after commit below.
write_index = tantivy.Index(
build_schema(),
path=str(self._backend._path),
)
register_tokenizers(write_index, settings.SEARCH_LANGUAGE)
self._raw_writer = write_index.writer()
else:
self._raw_writer = self._backend._index.writer()
# Open a fresh Index (and thus a fresh Tantivy ManagedDirectory)
# for the write, rather than reusing the process-local cached
# index. ManagedDirectory loads its GC bookkeeping (.managed.json)
# once, at construction, and never re-reads it; paperless runs
# several long-lived processes (Granian workers, Celery workers)
# that take turns writing under the file lock above. A cached,
# long-lived writer index would carry a stale managed-files view
# and, on commit, overwrite .managed.json with that stale view -
# permanently losing track of segment files other processes
# registered in the meantime, so they can never be garbage
# collected. Reopening fresh here always picks up the current
# on-disk state. The long-lived self._backend._index is used for
# reads only and is reloaded (not reopened) after commit below.
write_index = tantivy.Index(
build_schema(),
path=str(self._backend._path),
)
register_tokenizers(write_index, settings.SEARCH_LANGUAGE)
self._raw_writer = write_index.writer()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
@@ -372,9 +369,8 @@ class TantivyBackend:
Tantivy search backend with explicit lifecycle management.
Provides full-text search capabilities using the Tantivy search engine.
Supports in-memory indexes (for testing) and persistent on-disk indexes
(for production use). Handles document indexing, search queries, autocompletion,
and "more like this" functionality.
Keeps a persistent on-disk index. Handles document indexing, search queries,
autocompletion, and "more like this" functionality.
The backend manages its own connection lifecycle and can be reset when
the underlying index directory changes (e.g., during test isolation).
@@ -408,9 +404,7 @@ class TantivyBackend:
},
)
def __init__(self, path: Path | None = None):
# path=None → in-memory index (for tests)
# path=some_dir → on-disk index (for production)
def __init__(self, path: Path):
self._path = path
self._raw_index: tantivy.Index | None = None
self._raw_schema: tantivy.Schema | None = None
@@ -429,16 +423,13 @@ class TantivyBackend:
"""
Open or rebuild the index as needed.
For disk-based indexes, checks if rebuilding is needed due to schema
version or language changes. Registers custom tokenizers after opening.
Checks if rebuilding is needed due to schema version or language
changes. Registers custom tokenizers after opening.
Safe to call multiple times - subsequent calls are no-ops.
"""
if self._raw_index is not None:
return # pragma: no cover
if self._path is not None:
self._raw_index = open_or_rebuild_index(self._path)
else:
self._raw_index = tantivy.Index(build_schema())
self._raw_index = open_or_rebuild_index(self._path)
register_tokenizers(self._raw_index, settings.SEARCH_LANGUAGE)
self._raw_schema = self._raw_index.schema
@@ -1102,13 +1093,9 @@ class TantivyBackend:
writer's threads). Larger values buffer more docs in RAM before
flushing a segment, deferring merge work; they do not avoid it.
"""
# Create new index (on-disk or in-memory)
if self._path is not None:
wipe_index(self._path)
new_index = tantivy.Index(build_schema(), path=str(self._path))
_write_sentinels(self._path)
else:
new_index = tantivy.Index(build_schema())
wipe_index(self._path)
new_index = tantivy.Index(build_schema(), path=str(self._path))
_write_sentinels(self._path)
register_tokenizers(new_index, settings.SEARCH_LANGUAGE)
# Point instance at the new index so _build_tantivy_doc uses it
+3 -1
View File
@@ -2098,6 +2098,8 @@ class BulkEditSerializer(
if not isinstance(parameters["pages"], str):
raise serializers.ValidationError("invalid pages specified")
page_count = Document.objects.get(id=document_id).page_count
if not page_count:
raise serializers.ValidationError("document page count is unknown")
pages = []
for group in parameters["pages"].split(","):
start, is_range, end = group.partition("-")
@@ -2107,7 +2109,7 @@ class BulkEditSerializer(
except ValueError as e:
raise serializers.ValidationError("invalid pages specified") from e
# Bound the range before building it, a huge one would exhaust memory
if not 1 <= first <= last or (page_count and last > page_count):
if not 1 <= first <= last <= page_count:
raise serializers.ValidationError("invalid pages specified")
pages.append(list(range(first, last + 1)))
parameters["pages"] = pages
+18 -143
View File
@@ -1,88 +1,41 @@
import shutil
import zoneinfo
from collections.abc import Generator
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING
import filelock
import pytest
from django.contrib.auth import get_user_model
from django.contrib.contenttypes.models import ContentType
from guardian.shortcuts import clear_ct_cache
from pytest_django.fixtures import Settings
from rest_framework.test import APIClient
from documents.tests.factories import DocumentFactory
UserModelT = get_user_model()
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from documents.models import Document
@dataclass(frozen=True, slots=True)
class PaperlessDirs:
"""Standard Paperless-ngx directory layout for tests."""
media: Path
originals: Path
archive: Path
thumbnails: Path
from paperless_testing.dirs import PaperlessDirs
@pytest.fixture(scope="session")
def samples_dir() -> Path:
def document_samples_dir() -> Path:
"""Path to the shared test sample documents."""
return Path(__file__).parent / "samples" / "documents"
@pytest.fixture()
def paperless_dirs(tmp_path: Path) -> PaperlessDirs:
"""Create and return the directory structure for testing."""
media = tmp_path / "media"
dirs = PaperlessDirs(
media=media,
originals=media / "documents" / "originals",
archive=media / "documents" / "archive",
thumbnails=media / "documents" / "thumbnails",
)
for d in (dirs.originals, dirs.archive, dirs.thumbnails):
d.mkdir(parents=True)
return dirs
@pytest.fixture()
def _media_settings(paperless_dirs: PaperlessDirs, settings) -> None:
"""Configure Django settings to point at temp directories."""
settings.MEDIA_ROOT = paperless_dirs.media
settings.ORIGINALS_DIR = paperless_dirs.originals
settings.ARCHIVE_DIR = paperless_dirs.archive
settings.THUMBNAIL_DIR = paperless_dirs.thumbnails
settings.MEDIA_LOCK = paperless_dirs.media / "media.lock"
settings.IGNORABLE_FILES = {".DS_Store", "Thumbs.db", "desktop.ini"}
settings.APP_LOGO = ""
@pytest.fixture()
def sample_doc(
paperless_dirs: PaperlessDirs,
_media_settings: None,
samples_dir: Path,
paperless_dirs: "PaperlessDirs",
document_samples_dir: Path,
) -> "Document":
"""Create a document with valid files and matching checksums."""
with filelock.FileLock(paperless_dirs.media / "media.lock"):
with filelock.FileLock(paperless_dirs.media_lock):
shutil.copy(
samples_dir / "originals" / "0000001.pdf",
paperless_dirs.originals / "0000001.pdf",
document_samples_dir / "originals" / "0000001.pdf",
paperless_dirs.originals_dir / "0000001.pdf",
)
shutil.copy(
samples_dir / "archive" / "0000001.pdf",
paperless_dirs.archive / "0000001.pdf",
document_samples_dir / "archive" / "0000001.pdf",
paperless_dirs.archive_dir / "0000001.pdf",
)
shutil.copy(
samples_dir / "thumbnails" / "0000001.webp",
paperless_dirs.thumbnails / "0000001.webp",
document_samples_dir / "thumbnails" / "0000001.webp",
paperless_dirs.thumbnail_dir / "0000001.webp",
)
return DocumentFactory(
@@ -97,95 +50,17 @@ def sample_doc(
)
@pytest.fixture()
def _search_index(
tmp_path: Path,
settings: Settings,
) -> Generator[None, None, None]:
"""Create a temp index directory and point INDEX_DIR at it.
@pytest.fixture
def _search_index(paperless_dirs: "PaperlessDirs") -> None:
"""Point the search backend at a fresh, empty index directory.
Resets the backend singleton before and after so each test gets a clean
index rather than reusing a stale singleton from another test.
paperless_dirs owns INDEX_DIR and resets the backend singleton on both
sides of the test, so requesting it is all that is needed.
"""
from documents.search import reset_backend
index_dir = tmp_path / "index"
index_dir.mkdir()
settings.INDEX_DIR = index_dir
reset_backend()
yield
reset_backend()
@pytest.fixture()
def settings_timezone(settings: Settings) -> zoneinfo.ZoneInfo:
return zoneinfo.ZoneInfo(settings.TIME_ZONE)
@pytest.fixture
def rest_api_client():
"""
The basic DRF ApiClient
"""
yield APIClient()
@pytest.fixture()
def regular_user(django_user_model: type[UserModelT]) -> UserModelT:
"""Unprivileged authenticated user for permission boundary tests."""
return django_user_model.objects.create_user(username="regular", password="regular")
@pytest.fixture()
def admin_client(rest_api_client: APIClient, admin_user: UserModelT) -> APIClient:
"""Admin client pre-authenticated and sending the v10 Accept header."""
rest_api_client.force_authenticate(user=admin_user)
rest_api_client.credentials(HTTP_ACCEPT="application/json; version=10")
return rest_api_client
@pytest.fixture()
def v9_client(rest_api_client: APIClient, admin_user: UserModelT) -> APIClient:
"""Admin client pre-authenticated and sending the v9 Accept header."""
rest_api_client.force_authenticate(user=admin_user)
rest_api_client.credentials(HTTP_ACCEPT="application/json; version=9")
return rest_api_client
@pytest.fixture()
def user_client(rest_api_client: APIClient, regular_user: UserModelT) -> APIClient:
"""Regular-user client pre-authenticated and sending the v10 Accept header."""
rest_api_client.force_authenticate(user=regular_user)
rest_api_client.credentials(HTTP_ACCEPT="application/json; version=10")
return rest_api_client
@pytest.fixture(autouse=True)
def _clear_content_type_caches() -> None:
"""Clear Django's ContentType cache and guardian's lru_cache before each test.
Tests that delete and reinsert ContentType/Permission rows (e.g. the
importer) corrupt both caches. Without this fixture a subsequent test on
the same xdist worker sees stale ContentType objects and guardian raises
MixedContentTypeError.
"""
ContentType.objects.clear_cache()
clear_ct_cache()
@pytest.fixture(scope="session", autouse=True)
def faker_session_locale():
"""Set Faker locale for reproducibility."""
return "en_US"
@pytest.fixture(scope="session", autouse=True)
def faker_seed():
return 12345
@pytest.fixture
def indexed_document(_search_index: None) -> "Document":
def searchable_document(_search_index: None) -> "Document":
"""One searchable document, for tests about what the search endpoint
returns rather than about what it finds.
"""
+10
View File
@@ -0,0 +1,10 @@
import re
def dummy_preprocess(content: str) -> str:
"""
Simpler, faster pre-processing for testing purposes
"""
content = content.lower().strip()
content = re.sub(r"\s+", " ", content)
return content
@@ -15,11 +15,11 @@ from rich.console import Console
from documents.management.commands.document_sanity_checker import Command
from documents.sanity_checker import SanityCheckMessages
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from documents.models import Document
from documents.tests.conftest import PaperlessDirs
from paperless_testing.dirs import PaperlessDirs
def _render_to_string(messages: SanityCheckMessages) -> str:
@@ -71,7 +71,7 @@ class TestRenderResultsWithIssues:
assert "INFO" in output
assert "No OCR data" in output
@pytest.mark.usefixtures("_media_settings")
@pytest.mark.usefixtures("paperless_dirs")
def test_global_message(self) -> None:
msgs = SanityCheckMessages()
msgs.warning(None, "Orphaned file: /tmp/stray.pdf")
@@ -87,7 +87,7 @@ class TestRenderResultsWithIssues:
assert "Thumbnail missing" in output
assert "Checksum mismatch" in output
@pytest.mark.usefixtures("_media_settings")
@pytest.mark.usefixtures("paperless_dirs")
def test_unknown_doc_pk(self) -> None:
msgs = SanityCheckMessages()
msgs.error(99999, "Ghost document")
@@ -184,7 +184,6 @@ class TestDocumentSanityCheckerCommand:
assert "ERROR" in output
assert "Original of document does not exist" in output
@pytest.mark.usefixtures("_media_settings")
def test_checksum_mismatch(self, paperless_dirs: PaperlessDirs) -> None:
"""Lightweight document with zero-byte files triggers checksum mismatch."""
doc = DocumentFactory(
+4 -12
View File
@@ -9,29 +9,21 @@ from documents.search._backend import TantivyBackend
from documents.search._backend import reset_backend
from documents.search._schema import build_schema
from documents.search._tokenizer import register_tokenizers
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Generator
from pathlib import Path
from pytest_django.fixtures import Settings
from documents.models import Document
from paperless_testing.dirs import PaperlessDirs
@pytest.fixture
def index_dir(tmp_path: Path, settings: Settings) -> Path:
path = tmp_path / "index"
path.mkdir()
settings.INDEX_DIR = path
return path
@pytest.fixture
def backend() -> Generator[TantivyBackend, None, None]:
b = TantivyBackend() # path=None → in-memory index
def backend(paperless_dirs: PaperlessDirs) -> Generator[TantivyBackend, None, None]:
b = TantivyBackend(path=paperless_dirs.index_dir)
b.open()
try:
yield b
@@ -14,7 +14,6 @@ from typing import TYPE_CHECKING
import pytest
import time_machine
from django.contrib.auth.models import User
from documents.models import CustomField
from documents.models import CustomFieldInstance
@@ -22,7 +21,8 @@ from documents.models import DocumentType
from documents.models import Note
from documents.models import StoragePath
from documents.search._query import parse_user_query
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
if TYPE_CHECKING:
from collections.abc import Callable
@@ -148,7 +148,7 @@ class TestJsonSubpaths:
THEN:
- Only the document with alice's note matches
"""
alice = User.objects.create_user(username="alice")
alice = UserFactory(username="alice")
doc_with_note = DocumentFactory(
title="Has note",
content="x",
+17 -16
View File
@@ -3,10 +3,8 @@ from pathlib import Path
import pytest
from django.contrib.auth.models import Group
from django.contrib.auth.models import User
from django.db import connection
from django.test.utils import CaptureQueriesContext
from guardian.shortcuts import assign_perm
from pytest_mock import MockerFixture
from documents.models import CustomField
@@ -19,11 +17,12 @@ from documents.search._backend import WriteBatch
from documents.search._backend import get_backend
from documents.search._backend import reset_backend
from documents.signals.handlers import add_to_index
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from documents.tests.factories import DocumentTypeFactory
from documents.tests.factories import TagFactory
from documents.tests.factories import UserFactory
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import DocumentTypeFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_object
pytestmark = [pytest.mark.search, pytest.mark.django_db]
@@ -189,7 +188,7 @@ class TestAddOrUpdateIds:
pk=1,
owner=owner,
)
assign_perm("view_document", user, doc)
grant_object(user, doc, "view_document")
with backend.batch_update() as batch:
batch.add_or_update_ids([doc.pk])
@@ -209,7 +208,7 @@ class TestAddOrUpdateIds:
pk=1,
owner=owner,
)
assign_perm("view_document", group, doc)
grant_object(group, doc, "view_document")
with backend.batch_update() as batch:
batch.add_or_update_ids([doc.pk])
@@ -763,8 +762,8 @@ class TestSearchIds:
def test_respects_permission_filter(self, backend: TantivyBackend) -> None:
"""search_ids must respect user permission filtering."""
owner = User.objects.create_user("ids_owner")
other = User.objects.create_user("ids_other")
owner = UserFactory(username="ids_owner")
other = UserFactory(username="ids_other")
doc = Document.objects.create(
title="private doc",
content="secret keyword",
@@ -843,7 +842,7 @@ class TestRebuild:
content="group secret keyword",
owner=owner,
)
assign_perm("view_document", group, doc)
grant_object(group, doc, "view_document")
backend.rebuild(Document.objects.all())
@@ -948,7 +947,8 @@ class TestSingleton:
yield
reset_backend()
def test_returns_same_instance_on_repeated_calls(self, index_dir) -> None:
@pytest.mark.usefixtures("paperless_dirs")
def test_returns_same_instance_on_repeated_calls(self) -> None:
"""Singleton pattern: repeated calls to get_backend() must return the same instance."""
assert get_backend() is get_backend()
@@ -965,7 +965,8 @@ class TestSingleton:
assert b1 is not b2
assert b2._path == tmp_path / "b"
def test_reset_forces_new_instance(self, index_dir) -> None:
@pytest.mark.usefixtures("paperless_dirs")
def test_reset_forces_new_instance(self) -> None:
"""reset_backend() must force creation of a new backend instance on next get_backend() call."""
b1 = get_backend()
reset_backend()
@@ -1071,7 +1072,7 @@ class TestFieldHandling:
def test_notes_include_user_information(self, backend: TantivyBackend) -> None:
"""Notes must be indexed with user information when available for structured queries."""
user = User.objects.create_user("notewriter")
user = UserFactory(username="notewriter")
doc = Document.objects.create(
title="Doc with notes",
content="test",
@@ -1173,7 +1174,7 @@ class TestHighlightHits:
notes.note: prefix so the query targets notes content directly, but
the snippet is generated from notes_text which stores the same text.
"""
user = User.objects.create_user("hl_noteuser")
user = UserFactory(username="hl_noteuser")
doc = Document.objects.create(
title="Doc with matching note",
content="unrelated content",
@@ -27,7 +27,7 @@ import time_machine
from documents.models import Note
from documents.models import Tag
from documents.search._errors import InvalidDateQuery
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from collections.abc import Callable
@@ -269,7 +269,7 @@ class TestDocumentedDateForms:
yield
@pytest.fixture
def dated(self, index_document: Callable[..., Document]) -> dict[str, int]:
def dated(self, backend: TantivyBackend) -> dict[str, int]:
stamps = {
"today": datetime(2026, 6, 15, 9, 0, tzinfo=UTC),
"yesterday": datetime(2026, 6, 14, 9, 0, tzinfo=UTC),
@@ -279,14 +279,14 @@ class TestDocumentedDateForms:
"january": datetime(2026, 1, 10, 10, 0, tzinfo=UTC),
"old": datetime(2005, 3, 4, 15, 30, tzinfo=UTC),
}
return {
label: index_document(
title=label,
content="dated body",
added=stamp,
).pk
docs = {
label: DocumentFactory(title=label, content="dated body", added=stamp)
for label, stamp in stamps.items()
}
with backend.batch_update() as batch:
for doc in docs.values():
batch.add_or_update(doc)
return {label: doc.pk for label, doc in docs.items()}
@pytest.mark.parametrize(
("query", "label"),
@@ -29,7 +29,7 @@ from rest_framework import status
from documents.search._backend import SearchMode
from documents.search._query import parse_simple_text_highlight_query
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from rest_framework.test import APIClient
@@ -17,12 +17,12 @@ from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from django.contrib.auth.models import User
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Note
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
if TYPE_CHECKING:
from collections.abc import Callable
@@ -53,7 +53,7 @@ class TestBareJsonFieldPrefixes:
decoy's content match does not resurface through a demoted
text search
"""
alice = User.objects.create_user(username="alice")
alice = UserFactory(username="alice")
with_note = DocumentFactory(title="Has note", content="x")
Note.objects.create(document=with_note, user=alice, note="crocodile")
backend.add_or_update(with_note)
@@ -116,7 +116,7 @@ class TestBareJsonFieldPrefixes:
document; the default-subpath resolution for the bare
prefix does not interfere with explicit subpath addressing
"""
bob = User.objects.create_user(username="bob")
bob = UserFactory(username="bob")
doc = DocumentFactory(title="Bob note", content="x")
Note.objects.create(document=doc, user=bob, note="remark")
backend.add_or_update(doc)
@@ -20,7 +20,6 @@ from typing import TYPE_CHECKING
import pytest
import tantivy
from django.contrib.auth.models import User
from whoosh_compat import FieldKind
from documents.models import CustomField
@@ -28,6 +27,7 @@ from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import Note
from documents.search._fields import PUBLIC_FIELDS
from paperless_testing.factories import UserFactory
if TYPE_CHECKING:
from documents.search._backend import TantivyBackend
@@ -49,7 +49,7 @@ class TestJsonSubpathsAreWrittenAtIndexTime:
- Every subpath PUBLIC_FIELDS declares for notes/custom_fields
is present as a key in the document's stored JSON payload
"""
user = User.objects.create_user(username="completeness-user")
user = UserFactory(username="completeness-user")
field = CustomField.objects.create(
name="Completeness Field",
data_type=CustomField.FieldDataType.STRING,
@@ -15,7 +15,7 @@ from documents.search._backend import SearchIndexLockError
from documents.search._backend import TantivyBackend
from documents.tasks import index_document
from documents.tasks import remove_document_from_index
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from collections.abc import Generator
@@ -1,6 +1,6 @@
import pytest
from documents.tests.utils import TestMigrations
from paperless_testing.migrations import TestMigrations
pytestmark = pytest.mark.search
@@ -18,13 +18,14 @@ from typing import TYPE_CHECKING
import pytest
from django.contrib.auth.models import Group
from django.contrib.auth.models import User
from guardian.shortcuts import assign_perm
from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_object
if TYPE_CHECKING:
from documents.search._backend import TantivyBackend
@@ -34,22 +35,22 @@ pytestmark = [pytest.mark.search, pytest.mark.django_db]
@pytest.fixture
def owner() -> User:
return User.objects.create_user(username="owner")
return UserFactory(username="owner")
@pytest.fixture
def stranger() -> User:
return User.objects.create_user(username="stranger")
return UserFactory(username="stranger")
@pytest.fixture
def viewer() -> User:
return User.objects.create_user(username="viewer")
return UserFactory(username="viewer")
@pytest.fixture
def group_member() -> User:
user = User.objects.create_user(username="group_member")
user = UserFactory(username="group_member")
user.groups.add(Group.objects.create(name="accounting"))
return user
@@ -127,7 +128,7 @@ class TestPermissionFilteringOnIndexedDocuments:
checksum="perm-shared-user",
owner=owner,
)
assign_perm("view_document", viewer, doc)
grant_object(viewer, doc, "view_document")
backend.add_or_update(doc)
assert backend.search_ids("invoice", user=viewer) == [doc.pk]
@@ -157,7 +158,7 @@ class TestPermissionFilteringOnIndexedDocuments:
checksum="perm-shared-group",
owner=owner,
)
assign_perm("view_document", group_member.groups.first(), doc)
grant_object(group_member.groups.first(), doc, "view_document")
backend.add_or_update(doc)
assert backend.search_ids("invoice", user=group_member) == [doc.pk]
+22 -19
View File
@@ -13,11 +13,11 @@ from documents.search._schema import needs_rebuild
from documents.search._schema import schema_fingerprint
if TYPE_CHECKING:
from pathlib import Path
import tantivy
from pytest_django.fixtures import Settings
from paperless_testing.dirs import PaperlessDirs
pytestmark = pytest.mark.search
@@ -25,16 +25,19 @@ pytestmark = pytest.mark.search
class TestNeedsRebuild:
"""needs_rebuild covers all sentinel-file states that require a full reindex."""
def test_returns_true_when_settings_file_missing(self, index_dir: Path) -> None:
assert needs_rebuild(index_dir) is True
def test_returns_true_when_settings_file_missing(
self,
paperless_dirs: PaperlessDirs,
) -> None:
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_returns_false_when_version_and_language_match(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
settings.SEARCH_LANGUAGE = "en"
(index_dir / ".index_settings.json").write_text(
(paperless_dirs.index_dir / ".index_settings.json").write_text(
json.dumps(
{
"schema_version": SCHEMA_VERSION,
@@ -43,51 +46,51 @@ class TestNeedsRebuild:
},
),
)
assert needs_rebuild(index_dir) is False
assert needs_rebuild(paperless_dirs.index_dir) is False
def test_returns_true_on_schema_version_mismatch(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
settings.SEARCH_LANGUAGE = None
(index_dir / ".index_settings.json").write_text(
(paperless_dirs.index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": SCHEMA_VERSION - 1, "language": None}),
)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_returns_true_when_version_is_not_an_integer(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
settings.SEARCH_LANGUAGE = None
(index_dir / ".index_settings.json").write_text(
(paperless_dirs.index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": "not-a-number", "language": None}),
)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_returns_true_when_language_key_missing(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
settings.SEARCH_LANGUAGE = "en"
(index_dir / ".index_settings.json").write_text(
(paperless_dirs.index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": SCHEMA_VERSION}),
)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_returns_true_when_language_differs(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
settings.SEARCH_LANGUAGE = "de"
(index_dir / ".index_settings.json").write_text(
(paperless_dirs.index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": SCHEMA_VERSION, "language": "en"}),
)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def _schema_fields(schema: tantivy.Schema) -> dict[str, dict]:
@@ -35,6 +35,8 @@ if TYPE_CHECKING:
from pytest_django.fixtures import SettingsWrapper
from paperless_testing.dirs import PaperlessDirs
pytestmark = pytest.mark.search
# The on-disk field layout of a v2 index, pinned as data. Any edit here is an
@@ -469,7 +471,7 @@ def _fingerprint_of(descriptors: list[FieldDescriptor]) -> str:
class TestNeedsRebuildOnFingerprint:
def test_matching_fingerprint_does_not_rebuild(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: SettingsWrapper,
) -> None:
"""
@@ -482,13 +484,13 @@ class TestNeedsRebuildOnFingerprint:
- It returns False
"""
settings.SEARCH_LANGUAGE = None
_sentinels(index_dir)
_sentinels(paperless_dirs.index_dir)
assert needs_rebuild(index_dir) is False
assert needs_rebuild(paperless_dirs.index_dir) is False
def test_stale_fingerprint_rebuilds_despite_a_matching_version(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: SettingsWrapper,
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -505,7 +507,7 @@ class TestNeedsRebuildOnFingerprint:
every subsequent write would raise
"""
settings.SEARCH_LANGUAGE = None
_sentinels(index_dir)
_sentinels(paperless_dirs.index_dir)
extended = [
*field_descriptors(),
FieldDescriptor(
@@ -519,11 +521,11 @@ class TestNeedsRebuildOnFingerprint:
]
monkeypatch.setattr(_schema, "field_descriptors", lambda: extended)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_reordered_schema_rebuilds(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: SettingsWrapper,
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -538,16 +540,16 @@ class TestNeedsRebuildOnFingerprint:
- It returns True
"""
settings.SEARCH_LANGUAGE = None
_sentinels(index_dir)
_sentinels(paperless_dirs.index_dir)
reordered = field_descriptors()
reordered[1], reordered[2] = reordered[2], reordered[1]
monkeypatch.setattr(_schema, "field_descriptors", lambda: reordered)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_missing_fingerprint_rebuilds(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: SettingsWrapper,
) -> None:
"""
@@ -561,15 +563,15 @@ class TestNeedsRebuildOnFingerprint:
is rebuilt rather than trusted
"""
settings.SEARCH_LANGUAGE = None
(index_dir / ".index_settings.json").write_text(
(paperless_dirs.index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": SCHEMA_VERSION, "language": None}),
)
assert needs_rebuild(index_dir) is True
assert needs_rebuild(paperless_dirs.index_dir) is True
def test_written_sentinels_satisfy_the_check(
self,
index_dir: Path,
paperless_dirs: PaperlessDirs,
settings: SettingsWrapper,
) -> None:
"""
@@ -582,6 +584,6 @@ class TestNeedsRebuildOnFingerprint:
- It returns False
"""
settings.SEARCH_LANGUAGE = "en"
_write_sentinels(index_dir)
_write_sentinels(paperless_dirs.index_dir)
assert needs_rebuild(index_dir) is False
assert needs_rebuild(paperless_dirs.index_dir) is False
+4 -4
View File
@@ -16,11 +16,11 @@ from documents.models import Document
from documents.models import Tag
from documents.search import get_backend
from documents.search import reset_backend
from documents.tests.factories import DocumentFactory
from documents.tests.factories import TagFactory
from documents.tests.factories import UserFactory
from documents.tests.utils import DirectoriesMixin
from paperless.admin import PaperlessUserAdmin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory
@pytest.fixture
+5 -5
View File
@@ -3,7 +3,6 @@ from io import BytesIO
from pathlib import Path
from unittest.mock import patch
from django.contrib.auth.models import User
from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import override_settings
from PIL import Image
@@ -11,10 +10,11 @@ from PIL.PngImagePlugin import PngInfo
from rest_framework import status
from rest_framework.test import APITestCase
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import read_streaming_response
from paperless.models import ApplicationConfiguration
from paperless.models import ColorConvertChoices
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.http import read_streaming_response
class TestApiAppConfig(DirectoriesMixin, APITestCase):
@@ -23,7 +23,7 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
user = User.objects.create_superuser(username="temp_admin")
user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=user)
def test_api_get_config(self) -> None:
@@ -267,7 +267,7 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
THEN:
- old app_logo file is deleted
"""
admin = User.objects.create_superuser(username="admin")
admin = UserFactory(username="admin", superuser=True)
self.client.force_login(user=admin)
response = self.client.get("/logo/")
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
@@ -4,8 +4,6 @@ import json
import shutil
import zipfile
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from django.utils import timezone
from rest_framework import status
@@ -14,9 +12,11 @@ from rest_framework.test import APITestCase
from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import SampleDirMixin
from documents.tests.utils import read_streaming_response
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.http import read_streaming_response
from paperless_testing.permissions import grant_global
class TestBulkDownload(DirectoriesMixin, SampleDirMixin, APITestCase):
@@ -25,7 +25,7 @@ class TestBulkDownload(DirectoriesMixin, SampleDirMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
self.doc1 = Document.objects.create(title="unrelated", checksum="A")
@@ -326,10 +326,8 @@ class TestBulkDownload(DirectoriesMixin, SampleDirMixin, APITestCase):
)
def test_download_insufficient_permissions(self) -> None:
user = User.objects.create_user(username="temp_user")
user.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
user = UserFactory(username="temp_user")
grant_global(user, "view_document")
self.client.force_authenticate(user=user)
self.doc2.owner = self.user
+56 -34
View File
@@ -2,10 +2,8 @@ import json
from unittest import mock
from auditlog.models import LogEntry
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from guardian.shortcuts import assign_perm
from rest_framework import status
from rest_framework.test import APITestCase
@@ -15,14 +13,18 @@ from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_all_global
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestBulkEditAPI(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
user = User.objects.create_superuser(username="temp_admin")
user = UserFactory(username="temp_admin", superuser=True)
self.user = user
self.client.force_authenticate(user=user)
@@ -284,9 +286,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m,
) -> None:
self.setup_mock(m, "modify_custom_fields")
user = User.objects.create_user(username="doc-owner")
user.user_permissions.add(Permission.objects.get(codename="change_document"))
other_user = User.objects.create_user(username="other-user")
user = UserFactory(username="doc-owner")
grant_global(user, "change_document")
other_user = UserFactory(username="other-user")
source_doc = Document.objects.create(
checksum="source",
title="Source",
@@ -787,10 +789,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_bulk_edit_with_all_true_resolves_owned_duplicates(self, m) -> None:
self.setup_mock(m, "set_storage_path")
user = User.objects.create_user(username="duplicate-owner")
user.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
user = UserFactory(username="duplicate-owner")
grant_global(user, "change_document")
first_duplicate = Document.objects.create(
checksum="owned-duplicate",
title="First duplicate",
@@ -1178,7 +1178,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
user1 = User.objects.create(username="user1")
self.client.force_authenticate(user=user1)
assign_perm("view_document", user1, self.doc2)
grant_object(user1, self.doc2, "view_document")
response = self.client.post(
"/api/documents/selection_data/",
@@ -1188,9 +1188,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
user1.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(user1, "view_document")
user1 = User.objects.get(pk=user1.pk)
self.client.force_authenticate(user=user1)
response = self.client.post(
@@ -1533,7 +1531,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.owner = User.objects.get(username="temp_admin")
self.doc1.save()
user1 = User.objects.create(username="user1")
user1.user_permissions.add(*Permission.objects.all())
grant_all_global(user1)
user1.save()
self.client.force_authenticate(user=user1)
@@ -1587,8 +1585,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.owner = User.objects.get(username="temp_admin")
self.doc1.save()
user1 = User.objects.create(username="user1")
assign_perm("view_document", user1, self.doc1)
user1.user_permissions.add(*Permission.objects.all())
grant_object(user1, self.doc1, "view_document")
grant_all_global(user1)
user1.save()
self.client.force_authenticate(user=user1)
@@ -1609,7 +1607,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m.assert_not_called()
self.assertEqual(response.content, b"Insufficient permissions")
assign_perm("change_document", user1, self.doc1)
grant_object(user1, self.doc1, "change_document")
response = self.client.post(
"/api/documents/bulk_edit/",
@@ -1786,6 +1784,36 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertIn(b"invalid pages specified", response.content)
m.assert_not_called()
@mock.patch("documents.serialisers.bulk_edit.split")
def test_bulk_edit_split_rejects_unknown_page_count(self, m) -> None:
"""
GIVEN:
- A legacy split bulk edit of a document without a page count
WHEN:
- API to bulk edit is called
THEN:
- API returns HTTP 400
- split is not called
"""
self.setup_mock(m, "split")
for pages in ("1", "1-5000000"):
with self.subTest(pages=pages):
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc1.id],
"method": "split",
"parameters": {"pages": pages},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn(b"document page count is unknown", response.content)
m.assert_not_called()
@mock.patch("documents.serialisers.bulk_edit.split")
def test_bulk_edit_split_parses_pages(self, m) -> None:
"""
@@ -1819,7 +1847,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.owner = User.objects.get(username="temp_admin")
self.doc1.save()
user1 = User.objects.create(username="user1")
user1.user_permissions.add(*Permission.objects.all())
grant_all_global(user1)
user1.save()
self.client.force_authenticate(user=user1)
@@ -1880,7 +1908,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.owner = User.objects.get(username="temp_admin")
self.doc1.save()
user1 = User.objects.create(username="user1")
user1.user_permissions.add(*Permission.objects.all())
grant_all_global(user1)
user1.save()
self.client.force_authenticate(user=user1)
@@ -1919,11 +1947,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
@mock.patch("documents.views.bulk_edit.merge")
def test_merge_and_delete_requires_change_permission(self, m) -> None:
self.setup_mock(m, "merge")
user = User.objects.create_user(username="no-change")
user.user_permissions.add(
Permission.objects.get(codename="add_document"),
Permission.objects.get(codename="delete_document"),
)
user = UserFactory(username="no-change")
grant_global(user, "add_document", "delete_document")
self.client.force_authenticate(user=user)
response = self.client.post(
@@ -2310,7 +2335,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.owner = User.objects.get(username="temp_admin")
self.doc1.save()
user1 = User.objects.create(username="user1")
user1.user_permissions.add(*Permission.objects.all())
grant_all_global(user1)
user1.save()
self.client.force_authenticate(user=user1)
@@ -2345,7 +2370,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
@mock.patch("documents.views.bulk_edit.edit_pdf")
def test_edit_pdf_update_requires_change_permission(self, m) -> None:
self.setup_mock(m, "edit_pdf")
user = User.objects.create_user(username="no-change")
user = UserFactory(username="no-change")
self.client.force_authenticate(user=user)
response = self.client.post(
@@ -2372,11 +2397,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
) -> None:
self.setup_mock(edit_pdf_mock, "edit_pdf")
self.setup_mock(remove_password_mock, "remove_password")
user = User.objects.create_user(username="no-delete")
user.user_permissions.add(
Permission.objects.get(codename="add_document"),
Permission.objects.get(codename="change_document"),
)
user = UserFactory(username="no-delete")
grant_global(user, "add_document", "change_document")
self.client.force_authenticate(user=user)
cases = [
@@ -2463,7 +2485,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.owner = User.objects.get(username="temp_admin")
self.doc1.save()
user1 = User.objects.create(username="user1")
user1.user_permissions.add(*Permission.objects.all())
grant_all_global(user1)
user1.save()
self.client.force_authenticate(user=user1)
+6 -6
View File
@@ -4,20 +4,22 @@ from typing import TYPE_CHECKING
from unittest import mock
import pytest
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from rest_framework import status
from rest_framework.test import APIClient
from rest_framework.test import APITestCase
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
if TYPE_CHECKING:
from django.contrib.auth.models import User
from pytest_mock import MockerFixture
class TestChatStreamingViewInputValidation(APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
def _mock_ai_enabled(self) -> mock.MagicMock:
@@ -113,9 +115,7 @@ class TestChatStreamingViewUnrestrictedFlag:
needs to reach the view at all. Model-level only: says nothing
about which documents (if any) this user can actually see.
"""
regular_user.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
grant_global(regular_user, "view_document")
return user_client
@pytest.mark.parametrize(
+13 -16
View File
@@ -4,23 +4,24 @@ from unittest import mock
from unittest.mock import ANY
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from guardian.shortcuts import assign_perm
from rest_framework import status
from rest_framework.test import APITestCase
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestCustomFieldsAPI(DirectoriesMixin, APITestCase):
ENDPOINT = "/api/custom_fields/"
def setUp(self) -> None:
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
return super().setUp()
@@ -1174,11 +1175,9 @@ class TestCustomFieldsAPI(DirectoriesMixin, APITestCase):
def test_documentlink_patch_requires_change_permission_on_target_documents(
self,
) -> None:
source_owner = User.objects.create_user(username="source-owner")
source_owner.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
other_user = User.objects.create_user(username="other-user")
source_owner = UserFactory(username="source-owner")
grant_global(source_owner, "change_document")
other_user = UserFactory(username="other-user")
source_doc = Document.objects.create(
title="Source",
@@ -1221,11 +1220,9 @@ class TestCustomFieldsAPI(DirectoriesMixin, APITestCase):
def test_documentlink_patch_allowed_with_change_permission_on_target_documents(
self,
) -> None:
source_owner = User.objects.create_user(username="source-owner")
source_owner.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
other_user = User.objects.create_user(username="other-user")
source_owner = UserFactory(username="source-owner")
grant_global(source_owner, "change_document")
other_user = UserFactory(username="other-user")
source_doc = Document.objects.create(
title="Source",
@@ -1244,7 +1241,7 @@ class TestCustomFieldsAPI(DirectoriesMixin, APITestCase):
data_type=CustomField.FieldDataType.DOCUMENTLINK,
)
assign_perm("change_document", source_owner, target_doc)
grant_object(source_owner, target_doc, "change_document")
self.client.force_authenticate(user=source_owner)
resp = self.client.patch(
@@ -1337,7 +1334,7 @@ class TestCustomFieldsAPI(DirectoriesMixin, APITestCase):
self.assertEqual(results[0]["document_count"], 1)
# Test as user without access to the document
non_superuser = User.objects.create_user(username="non_superuser")
non_superuser = UserFactory(username="non_superuser")
non_superuser.user_permissions.add(
*Permission.objects.exclude(codename="view_global_statistics"),
)
@@ -5,8 +5,6 @@ from typing import TYPE_CHECKING
from unittest import mock
from auditlog.models import LogEntry # type: ignore[import-untyped]
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import TestCase as DjangoTestCase
@@ -18,10 +16,12 @@ from documents.data_models import DocumentSource
from documents.filters import EffectiveContentFilter
from documents.filters import TitleContentFilter
from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import read_streaming_response
from documents.versioning import annotate_effective_content
from documents.views import DocumentSelectionMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.http import read_streaming_response
from paperless_testing.permissions import grant_global
if TYPE_CHECKING:
from pathlib import Path
@@ -31,7 +31,7 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
def _make_pdf_upload(self, name: str = "version.pdf") -> SimpleUploadedFile:
@@ -89,11 +89,9 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(resp.status_code, status.HTTP_404_NOT_FOUND)
def test_root_endpoint_returns_403_when_user_lacks_permission(self) -> None:
owner = User.objects.create_user(username="owner")
viewer = User.objects.create_user(username="viewer")
viewer.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="owner")
viewer = UserFactory(username="viewer")
grant_global(viewer, "view_document")
root = Document.objects.create(
title="root",
checksum="root",
@@ -283,11 +281,9 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(mock_backend.add_or_update.call_args[0][0].id, root.id)
def test_delete_version_returns_403_without_permission(self) -> None:
owner = User.objects.create_user(username="owner")
other = User.objects.create_user(username="other")
other.user_permissions.add(
Permission.objects.get(codename="delete_document"),
)
owner = UserFactory(username="owner")
other = UserFactory(username="other")
grant_global(other, "delete_document")
root = Document.objects.create(
title="root",
checksum="root",
@@ -371,11 +367,9 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertTrue(resp.data["is_root"])
def test_update_version_label_returns_403_without_permission(self) -> None:
owner = User.objects.create_user(username="owner")
other = User.objects.create_user(username="other")
other.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
owner = UserFactory(username="owner")
other = UserFactory(username="other")
grant_global(other, "change_document")
root = Document.objects.create(
title="root",
checksum="root",
@@ -553,11 +547,9 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(resp.status_code, status.HTTP_404_NOT_FOUND)
def test_metadata_returns_403_when_user_lacks_permission(self) -> None:
owner = User.objects.create_user(username="owner")
other = User.objects.create_user(username="other")
other.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="owner")
other = UserFactory(username="other")
grant_global(other, "view_document")
doc = Document.objects.create(
title="root",
checksum="root",
@@ -653,8 +645,8 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(resp.status_code, status.HTTP_500_INTERNAL_SERVER_ERROR)
def test_update_version_returns_403_without_permission(self) -> None:
owner = User.objects.create_user(username="owner")
other = User.objects.create_user(username="other")
owner = UserFactory(username="owner")
other = UserFactory(username="other")
root = Document.objects.create(
title="root",
checksum="root",
@@ -672,8 +664,8 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(resp.status_code, status.HTTP_403_FORBIDDEN)
def test_update_version_requires_global_change_permission(self) -> None:
user = User.objects.create_user(username="add-only")
user.user_permissions.add(Permission.objects.get(codename="add_document"))
user = UserFactory(username="add-only")
grant_global(user, "add_document")
root = Document.objects.create(
title="root",
checksum="root",
@@ -978,7 +970,7 @@ class TestVersionAwareFilters(DjangoTestCase):
superseded content -- selecting documents the list view, filtered by
the same term, does not show.
"""
user = User.objects.create_superuser(username="bulk_selection")
user = UserFactory(username="bulk_selection", superuser=True)
selected = DocumentSelectionMixin()._resolve_document_ids(
user=user,
@@ -1005,7 +997,7 @@ class TestBulkSelectionExcludesVersions(DjangoTestCase):
"Select all matching" reconstructs the document list, which never
contains version documents as rows of their own.
"""
user = User.objects.create_superuser(username="bulk_versions")
user = UserFactory(username="bulk_versions", superuser=True)
root = Document.objects.create(
title="shared-title root",
checksum="bulk-root",
+87 -100
View File
@@ -23,7 +23,6 @@ from django.core.files.uploadedfile import SimpleUploadedFile
from django.db import DataError
from django.test import override_settings
from django.utils import timezone
from guardian.shortcuts import assign_perm
from rest_framework import status
from rest_framework.test import APITestCase
@@ -48,18 +47,22 @@ from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from documents.signals.handlers import run_workflows
from documents.tests.factories import DocumentFactory
from documents.tests.factories import TagFactory
from documents.tests.utils import ConsumeTaskMixin
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import read_streaming_response
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory
from paperless_testing.http import read_streaming_response
from paperless_testing.permissions import grant_all_global
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
cache.clear()
@@ -357,10 +360,10 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
with Path(filename).open("wb") as f:
f.write(content)
user1 = User.objects.create_user(username="test1")
user2 = User.objects.create_user(username="test2")
user1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user2.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user1 = UserFactory(username="test1")
user2 = UserFactory(username="test2")
grant_global(user1, "view_document")
grant_global(user2, "view_document")
self.client.force_authenticate(user2)
@@ -383,7 +386,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
response = self.client.get(f"/api/documents/{doc.pk}/thumb/")
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
assign_perm("view_document", user2, doc)
grant_object(user2, doc, "view_document")
response = self.client.get(f"/api/documents/{doc.pk}/download/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
@@ -760,8 +763,8 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
- History is returned
"""
# No auditlog permissions
user = User.objects.create_user(username="test")
user.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user = UserFactory(username="test")
grant_global(user, "view_document")
self.client.force_authenticate(user=user)
doc = Document.objects.create(
title="First title",
@@ -776,7 +779,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
# superuser
user.is_superuser = True
user.save()
user2 = User.objects.create_user(username="test2")
user2 = UserFactory(username="test2")
doc2 = Document.objects.create(
title="Second title",
checksum="456",
@@ -1073,11 +1076,9 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(response.data["duplicate_documents"], [])
def test_has_duplicates_filter_respects_document_permissions(self) -> None:
owner = User.objects.create_user(username="duplicate-owner")
requester = User.objects.create_user(username="duplicate-requester")
requester.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="duplicate-owner")
requester = UserFactory(username="duplicate-requester")
grant_global(requester, "view_document")
visible_document = Document.objects.create(
title="visible document",
checksum="permission-match",
@@ -1096,7 +1097,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
[document["id"] for document in response.data["results"]],
)
assign_perm("view_document", requester, hidden_duplicate)
grant_object(requester, hidden_duplicate, "view_document")
response = self.client.get("/api/documents/?has_duplicates=true")
self.assertIn(
visible_document.id,
@@ -1317,10 +1318,10 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Owner filters work correctly but still respect permissions
"""
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
u2.user_permissions.add(*Permission.objects.filter(codename="view_document"))
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
grant_global(u1, "view_document")
grant_global(u2, "view_document")
u1_doc1 = Document.objects.create(
title="none1",
@@ -1353,7 +1354,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
)
self.client.force_authenticate(user=u1)
assign_perm("view_document", u1, u2_doc2)
grant_object(u1, u2_doc2, "view_document")
# Will not show any u1 docs or u2_doc1 which isn't shared
response = self.client.get(f"/api/documents/?owner__id__none={u1.id}")
@@ -1400,7 +1401,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
[u1_doc1.id, u1_doc2.id, u2_doc2.id],
)
assign_perm("view_document", u2, u1_doc1)
grant_object(u2, u1_doc1, "view_document")
# Will show only documents shared by user
response = self.client.get(f"/api/documents/?shared_by__id={u1.id}")
@@ -1424,8 +1425,8 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
- The document is returned exactly once, not once per permission path
(regression test for https://github.com/paperless-ngx/paperless-ngx/issues/13331)
"""
user = User.objects.create_user("user1")
user.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user = UserFactory(username="user1")
grant_global(user, "view_document")
group = Group.objects.create(name="group1")
user.groups.add(group)
@@ -1433,7 +1434,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
tag2 = TagFactory()
doc = DocumentFactory(title="shared", owner=user)
doc.tags.add(tag1, tag2)
assign_perm("view_document", group, doc)
grant_object(group, doc, "view_document")
self.client.force_authenticate(user=user)
response = self.client.get(
@@ -1452,11 +1453,9 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- The document does not appear in their results
"""
owner = User.objects.create_user("owner1")
stranger = User.objects.create_user("stranger1")
stranger.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
owner = UserFactory(username="owner1")
stranger = UserFactory(username="stranger1")
grant_global(stranger, "view_document")
DocumentFactory(title="private", owner=owner)
@@ -1474,17 +1473,17 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Only the group member sees the document
"""
owner = User.objects.create_user("owner2")
member = User.objects.create_user("member1")
non_member = User.objects.create_user("nonmember1")
owner = UserFactory(username="owner2")
member = UserFactory(username="member1")
non_member = UserFactory(username="nonmember1")
for u in (member, non_member):
u.user_permissions.add(*Permission.objects.filter(codename="view_document"))
grant_global(u, "view_document")
group = Group.objects.create(name="group2")
member.groups.add(group)
doc = DocumentFactory(title="shared2", owner=owner)
assign_perm("view_document", group, doc)
grant_object(group, doc, "view_document")
self.client.force_authenticate(user=member)
response = self.client.get("/api/documents/")
@@ -1785,8 +1784,8 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Statistics only include inbox counts for tags accessible by the user
"""
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
inbox_tag_u1 = Tag.objects.create(name="inbox_u1", is_inbox_tag=True, owner=u1)
Tag.objects.create(name="inbox_u2", is_inbox_tag=True, owner=u2)
doc_u1 = Document.objects.create(
@@ -1816,11 +1815,9 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(response.data["documents_inbox"], 0)
def test_statistics_with_statistics_permission(self) -> None:
owner = User.objects.create_user("owner")
stats_user = User.objects.create_user("stats-user")
stats_user.user_permissions.add(
Permission.objects.get(codename="view_global_statistics"),
)
owner = UserFactory(username="owner")
stats_user = UserFactory(username="stats-user")
grant_global(stats_user, "view_global_statistics")
inbox_tag = Tag.objects.create(
name="stats_inbox",
@@ -1986,7 +1983,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_405_METHOD_NOT_ALLOWED)
def test_upload_insufficient_permissions(self) -> None:
self.client.force_authenticate(user=User.objects.create_user("testuser2"))
self.client.force_authenticate(user=UserFactory(username="testuser2"))
with (Path(__file__).parent / "samples" / "simple.pdf").open("rb") as f:
response = self.client.post(
@@ -2782,9 +2779,9 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
mock_get_date_parser.assert_not_called()
def test_saved_views(self) -> None:
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u3 = User.objects.create_user("user3")
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
u3 = UserFactory(username="user3")
view_perm = Permission.objects.get(codename="view_savedview")
change_perm = Permission.objects.get(codename="change_savedview")
@@ -2807,9 +2804,9 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
sort_field="",
)
assign_perm("view_savedview", u1, v2)
assign_perm("change_savedview", u1, v2)
assign_perm("view_savedview", u1, v3)
grant_object(u1, v2, "view_savedview")
grant_object(u1, v2, "change_savedview")
grant_object(u1, v3, "view_savedview")
self.client.force_authenticate(user=u1)
@@ -3064,7 +3061,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertListEqual(saved_view_settings["sidebar_views_visible_ids"], [v2.id])
def test_saved_view_create_update_patch(self) -> None:
User.objects.create_user("user1")
UserFactory(username="user1")
view = {
"name": "test",
@@ -3127,7 +3124,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
- Display options are updated
- Display fields are validated
"""
User.objects.create_user("user1")
UserFactory(username="user1")
view = {
"name": "test",
@@ -3568,11 +3565,11 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Notes are neither created nor deleted
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.all())
user1 = UserFactory(username="test1")
grant_all_global(user1)
user1.save()
user2 = User.objects.create_user(username="test2")
user2 = UserFactory(username="test2")
user2.save()
doc = Document.objects.create(
@@ -3592,7 +3589,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(resp.content, b"Insufficient permissions to view notes")
self.assertEqual(resp.status_code, status.HTTP_403_FORBIDDEN)
assign_perm("view_document", user1, doc)
grant_object(user1, doc, "view_document")
resp = self.client.post(
f"/api/documents/{doc.pk}/notes/",
@@ -3616,12 +3613,8 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
def test_notes_require_global_document_permissions(self) -> None:
user = User.objects.create_user(username="note_editor")
user.user_permissions.add(
*Permission.objects.filter(
codename__in=["view_note", "add_note", "delete_note"],
),
)
user = UserFactory(username="note_editor")
grant_global(user, "view_note", "add_note", "delete_note")
doc = Document.objects.create(
title="test",
mime_type="application/pdf",
@@ -3634,9 +3627,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
response = self.client.get(f"/api/documents/{doc.pk}/notes/")
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
user.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(user, "view_document")
user = User.objects.get(pk=user.pk)
self.client.force_authenticate(user)
response = self.client.get(f"/api/documents/{doc.pk}/notes/")
@@ -3648,9 +3639,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
user.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
grant_global(user, "change_document")
user = User.objects.get(pk=user.pk)
self.client.force_authenticate(user)
response = self.client.post(
@@ -3797,12 +3786,12 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
- Unique items are created
- Non-unique items are not allowed
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.filter(codename="add_tag"))
user1 = UserFactory(username="test1")
grant_global(user1, "add_tag")
user1.save()
user2 = User.objects.create_user(username="test2")
user2.user_permissions.add(*Permission.objects.filter(codename="add_tag"))
user2 = UserFactory(username="test2")
grant_global(user2, "add_tag")
user2.save()
# User 1 creates tag 1 owned by user 1 by default
@@ -3857,12 +3846,12 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
- Unique items are created
- Non-unique items are not allowed on update
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.filter(codename="change_tag"))
user1 = UserFactory(username="test1")
grant_global(user1, "change_tag")
user1.save()
user2 = User.objects.create_user(username="test2")
user2.user_permissions.add(*Permission.objects.filter(codename="change_tag"))
user2 = UserFactory(username="test2")
grant_global(user2, "change_tag")
user2.save()
# Create name tag 1 owned by user 1
@@ -3993,11 +3982,11 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Links only shown if user has permissions
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.all())
user1 = UserFactory(username="test1")
grant_all_global(user1)
user1.save()
user2 = User.objects.create_user(username="test2")
user2 = UserFactory(username="test2")
user2.save()
doc = Document.objects.create(
@@ -4017,7 +4006,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(resp.content, b"Insufficient permissions to add share link")
self.assertEqual(resp.status_code, status.HTTP_403_FORBIDDEN)
assign_perm("change_document", user1, doc)
grant_object(user1, doc, "change_document")
resp = self.client.get(
f"/api/documents/{doc.pk}/share_links/",
@@ -4034,11 +4023,11 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Share link creation is denied until view permission is granted
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.filter(codename="add_sharelink"))
user1 = UserFactory(username="test1")
grant_global(user1, "add_sharelink")
user1.save()
user2 = User.objects.create_user(username="test2")
user2 = UserFactory(username="test2")
user2.save()
doc = Document.objects.create(
@@ -4060,7 +4049,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
)
self.assertEqual(create_resp.status_code, status.HTTP_403_FORBIDDEN)
assign_perm("view_document", user1, doc)
grant_object(user1, doc, "view_document")
create_resp = self.client.post(
"/api/share_links/",
@@ -4072,9 +4061,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
)
self.assertEqual(create_resp.status_code, status.HTTP_403_FORBIDDEN)
user1.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(user1, "view_document")
user1 = User.objects.get(pk=user1.pk)
self.client.force_authenticate(user1)
create_resp = self.client.post(
@@ -4097,11 +4084,11 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- ASN +1 from user2's doc is returned for user1
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.all())
user1 = UserFactory(username="test1")
grant_all_global(user1)
user1.save()
user2 = User.objects.create_user(username="test2")
user2 = UserFactory(username="test2")
user2.save()
doc1 = Document.objects.create(
@@ -4141,8 +4128,8 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- ASN 1 is returned
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.all())
user1 = UserFactory(username="test1")
grant_all_global(user1)
user1.save()
doc1 = Document.objects.create(
@@ -4170,7 +4157,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Explicit error is returned
"""
user1 = User.objects.create_superuser(username="test1")
user1 = UserFactory(username="test1", superuser=True)
self.client.force_authenticate(user1)
@@ -4348,8 +4335,8 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
THEN:
- Error response is returned
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.all())
user1 = UserFactory(username="test1")
grant_all_global(user1)
user1.save()
doc = Document.objects.create(
@@ -4454,7 +4441,7 @@ class TestDocumentApiTagColors(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
@@ -4534,7 +4521,7 @@ class TestDocumentApiCustomFieldsSorting(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
self.doc1 = Document.objects.create(
+8 -8
View File
@@ -2,16 +2,16 @@ import json
import shutil
from unittest import mock
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.core import mail
from django.test import override_settings
from rest_framework import status
from rest_framework.test import APITestCase
from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import SampleDirMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
@@ -20,7 +20,7 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
self.doc1 = Document.objects.create(
@@ -303,8 +303,8 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
THEN:
- Forbidden response is returned
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user1 = UserFactory(username="test1")
grant_global(user1, "view_document")
doc_owned = Document.objects.create(
title="owned_doc",
@@ -338,8 +338,8 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
THEN:
- Request succeeds
"""
user1 = User.objects.create_user(username="test1")
user1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user1 = UserFactory(username="test1")
grant_global(user1, "view_document")
self.client.force_authenticate(user1)
@@ -4,7 +4,6 @@ from collections.abc import Callable
from datetime import date
from urllib.parse import quote
from django.contrib.auth.models import User
from rest_framework.test import APITestCase
from documents.models import CustomField
@@ -13,7 +12,8 @@ from documents.models import Document
from documents.models import SavedView
from documents.models import SavedViewFilterRule
from documents.serialisers import DocumentSerializer
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
class DocumentWrapper:
@@ -35,7 +35,7 @@ class TestCustomFieldsSearch(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
# Create one custom field per type. The fields are called f"{type}_field".
+35 -54
View File
@@ -3,12 +3,10 @@ import json
from unittest import mock
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.db import connection
from django.test import override_settings
from django.test.utils import CaptureQueriesContext
from guardian.shortcuts import assign_perm
from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
from rest_framework import status
@@ -21,14 +19,17 @@ from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestApiObjects(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
user = User.objects.create_superuser(username="temp_admin")
user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=user)
self.tag1 = Tag.objects.create(name="t1", is_inbox_tag=True)
@@ -153,7 +154,7 @@ class TestApiObjects(DirectoriesMixin, APITestCase):
# A newer document owned by another user must not leak through the
# aggregate for a non-superuser who cannot view it
other = User.objects.create_user(username="other")
other = UserFactory(username="other")
Document.objects.create(
mime_type="application/pdf",
correspondent=self.c1,
@@ -162,10 +163,8 @@ class TestApiObjects(DirectoriesMixin, APITestCase):
owner=other,
)
user = User.objects.create_user(username="regular")
user.user_permissions.add(
Permission.objects.get(codename="view_correspondent"),
)
user = UserFactory(username="regular")
grant_global(user, "view_correspondent")
self.client.force_authenticate(user=user)
response = self.client.get("/api/correspondents/?last_correspondence=true")
@@ -200,7 +199,7 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
user = User.objects.create_superuser(username="temp_admin")
user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=user)
self.sp1 = StoragePath.objects.create(name="sp1", path="Something/{checksum}")
@@ -455,11 +454,9 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
self.assertEqual(response.data, "folder/Something.pdf")
def test_test_storage_path_requires_document_view_permission(self) -> None:
owner = User.objects.create_user(username="owner")
unprivileged = User.objects.create_user(username="unprivileged")
unprivileged.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="owner")
unprivileged = UserFactory(username="unprivileged")
grant_global(unprivileged, "view_document")
document = Document.objects.create(
mime_type="application/pdf",
owner=owner,
@@ -481,15 +478,15 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
self.assertIn("document", response.data)
def test_test_storage_path_allows_shared_document_view_permission(self) -> None:
owner = User.objects.create_user(username="owner")
viewer = User.objects.create_user(username="viewer")
owner = UserFactory(username="owner")
viewer = UserFactory(username="viewer")
document = Document.objects.create(
mime_type="application/pdf",
owner=owner,
title="Shared",
checksum="123",
)
assign_perm("view_document", viewer, document)
grant_object(viewer, document, "view_document")
self.client.force_authenticate(user=viewer)
response = self.client.post(
@@ -504,9 +501,7 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
viewer.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(viewer, "view_document")
viewer = User.objects.get(pk=viewer.pk)
self.client.force_authenticate(user=viewer)
response = self.client.post(
@@ -545,14 +540,12 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
def test_test_storage_path_exposes_basic_document_context_but_not_sensitive_owner_data(
self,
) -> None:
owner = User.objects.create_user(
owner = UserFactory(
username="owner",
password="password",
email="owner@example.com",
)
owner.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(owner, "view_document")
document = Document.objects.create(
mime_type="application/pdf",
owner=owner,
@@ -614,8 +607,8 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
def test_test_storage_path_includes_related_objects_for_visible_document(
self,
) -> None:
owner = User.objects.create_user(username="owner")
viewer = User.objects.create_user(username="viewer")
owner = UserFactory(username="owner")
viewer = UserFactory(username="viewer")
private_correspondent = Correspondent.objects.create(
name="Private Correspondent",
owner=owner,
@@ -627,10 +620,8 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
title="Document",
checksum="123",
)
assign_perm("view_document", viewer, document)
viewer.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_object(viewer, document, "view_document")
grant_global(viewer, "view_document")
self.client.force_authenticate(user=viewer)
response = self.client.post(
@@ -662,7 +653,7 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
self.assertEqual(response.data, "Private Correspondent.pdf")
def test_test_storage_path_superuser_can_view_private_related_objects(self) -> None:
owner = User.objects.create_user(username="owner")
owner = UserFactory(username="owner")
private_correspondent = Correspondent.objects.create(
name="Private Correspondent",
owner=owner,
@@ -693,8 +684,8 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
def test_test_storage_path_includes_doc_type_storage_path_and_tags(
self,
) -> None:
owner = User.objects.create_user(username="owner")
viewer = User.objects.create_user(username="viewer")
owner = UserFactory(username="owner")
viewer = UserFactory(username="viewer")
private_document_type = DocumentType.objects.create(
name="Private Type",
owner=owner,
@@ -717,10 +708,8 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
checksum="123",
)
document.tags.add(private_tag)
assign_perm("view_document", viewer, document)
viewer.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_object(viewer, document, "view_document")
grant_global(viewer, "view_document")
self.client.force_authenticate(user=viewer)
response = self.client.post(
@@ -756,8 +745,8 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
def test_test_storage_path_includes_custom_fields_for_visible_document(
self,
) -> None:
owner = User.objects.create_user(username="owner")
viewer = User.objects.create_user(username="viewer")
owner = UserFactory(username="owner")
viewer = UserFactory(username="viewer")
document = Document.objects.create(
mime_type="application/pdf",
owner=owner,
@@ -773,10 +762,8 @@ class TestApiStoragePaths(DirectoriesMixin, APITestCase):
field=custom_field,
value_int=42,
)
assign_perm("view_document", viewer, document)
viewer.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_object(viewer, document, "view_document")
grant_global(viewer, "view_document")
self.client.force_authenticate(user=viewer)
response = self.client.post(
@@ -798,7 +785,7 @@ class TestBulkEditObjects(APITestCase):
def setUp(self) -> None:
super().setUp()
self.temp_admin = User.objects.create_superuser(username="temp_admin")
self.temp_admin = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.temp_admin)
self.t1 = Tag.objects.create(name="t1")
@@ -1030,9 +1017,7 @@ class TestBulkEditObjects(APITestCase):
THEN:
- User is able to delete objects
"""
self.user1.user_permissions.add(
*Permission.objects.filter(codename="delete_tag"),
)
grant_global(self.user1, "delete_tag")
self.user1.save()
self.client.force_authenticate(user=self.user1)
@@ -1062,9 +1047,7 @@ class TestBulkEditObjects(APITestCase):
self.t2.owner = User.objects.get(username="temp_admin")
self.t2.save()
self.user1.user_permissions.add(
*Permission.objects.filter(codename="delete_tag"),
)
grant_global(self.user1, "delete_tag")
self.user1.save()
self.client.force_authenticate(user=self.user1)
@@ -1097,9 +1080,7 @@ class TestBulkEditObjects(APITestCase):
self.t2.owner = User.objects.get(username="temp_admin")
self.t2.save()
self.user1.user_permissions.add(
*Permission.objects.filter(codename="delete_tag"),
)
grant_global(self.user1, "delete_tag")
self.user1.save()
self.client.force_authenticate(user=self.user1)
+62 -76
View File
@@ -5,9 +5,7 @@ from unittest import mock
from allauth.mfa.models import Authenticator
from allauth.mfa.totp.internal import auth as totp_auth
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from guardian.shortcuts import assign_perm
from guardian.shortcuts import get_perms
from guardian.shortcuts import get_users_with_perms
from rest_framework import status
@@ -19,7 +17,11 @@ from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_all_global
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestApiAuth(DirectoriesMixin, APITestCase):
@@ -93,14 +95,14 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
self.assertNotIn("X-Version", response)
def test_api_version_with_auth(self) -> None:
user = User.objects.create_superuser(username="test")
user = UserFactory(username="test", superuser=True)
self.client.force_authenticate(user)
response = self.client.get("/api/documents/")
self.assertIn("X-Api-Version", response)
self.assertIn("X-Version", response)
def test_api_insufficient_permissions(self) -> None:
user = User.objects.create_user(username="test")
user = UserFactory(username="test")
self.client.force_authenticate(user)
Document.objects.create(title="Test")
@@ -137,8 +139,8 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
)
def test_api_sufficient_permissions(self) -> None:
user = User.objects.create_user(username="test")
user.user_permissions.add(*Permission.objects.all())
user = UserFactory(username="test")
grant_all_global(user)
user.is_staff = True
self.client.force_authenticate(user)
@@ -166,9 +168,9 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
)
def test_api_get_object_permissions(self) -> None:
user1 = User.objects.create_user(username="test1")
user2 = User.objects.create_user(username="test2")
user1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user1 = UserFactory(username="test1")
user2 = UserFactory(username="test2")
grant_global(user1, "view_document")
self.client.force_authenticate(user1)
self.assertEqual(
@@ -205,7 +207,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
THEN:
- Object created with current user as owner
"""
user1 = User.objects.create_superuser(username="user1")
user1 = UserFactory(username="user1", superuser=True)
self.client.force_authenticate(user1)
@@ -234,7 +236,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
THEN:
- Object created with no owner
"""
user1 = User.objects.create_superuser(username="user1")
user1 = UserFactory(username="user1", superuser=True)
self.client.force_authenticate(user1)
@@ -265,7 +267,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
THEN:
- Object permissions are set appropriately
"""
user1 = User.objects.create_superuser(username="user1")
user1 = UserFactory(username="user1", superuser=True)
user2 = User.objects.create(username="user2")
group1 = Group.objects.create(name="group1")
@@ -313,7 +315,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
THEN:
- Object permissions are set appropriately
"""
user1 = User.objects.create_superuser(username="user1")
user1 = UserFactory(username="user1", superuser=True)
user2 = User.objects.create(username="user2")
group1 = Group.objects.create(name="group1")
@@ -363,7 +365,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
mime_type="application/pdf",
content="this is a document",
)
user1 = User.objects.create_superuser(username="user1")
user1 = UserFactory(username="user1", superuser=True)
user2 = User.objects.create(username="user2")
group1 = Group.objects.create(name="group1")
@@ -413,16 +415,16 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
mime_type="application/pdf",
content="this is a document",
)
user1 = User.objects.create_superuser(username="user1")
user1 = UserFactory(username="user1", superuser=True)
user2 = User.objects.create(username="user2")
group1 = Group.objects.create(name="group1")
doc.owner = user1
doc.save()
assign_perm("view_document", user2, doc)
assign_perm("change_document", user2, doc)
assign_perm("view_document", group1, doc)
assign_perm("change_document", group1, doc)
grant_object(user2, doc, "view_document")
grant_object(user2, doc, "change_document")
grant_object(group1, doc, "view_document")
grant_object(group1, doc, "change_document")
self.client.force_authenticate(user1)
@@ -446,11 +448,9 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
self.assertIn("change_document", get_perms(group1, doc))
def test_document_permissions_change_requires_owner(self) -> None:
owner = User.objects.create_user(username="owner")
editor = User.objects.create_user(username="editor")
editor.user_permissions.add(
*Permission.objects.all(),
)
owner = UserFactory(username="owner")
editor = UserFactory(username="editor")
grant_all_global(editor)
doc = Document.objects.create(
title="Ownered doc",
@@ -460,8 +460,8 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
owner=owner,
)
assign_perm("view_document", editor, doc)
assign_perm("change_document", editor, doc)
grant_object(editor, doc, "view_document")
grant_object(editor, doc, "change_document")
self.client.force_authenticate(editor)
response = self.client.patch(
@@ -499,9 +499,9 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
def test_dynamic_permissions_fields(self) -> None:
user1 = User.objects.create_user(username="user1")
user1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
user2 = User.objects.create_user(username="user2")
user1 = UserFactory(username="user1")
grant_global(user1, "view_document")
user2 = UserFactory(username="user2")
Document.objects.create(title="Test", content="content 1", checksum="1")
doc2 = Document.objects.create(
@@ -523,10 +523,10 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
owner=user1,
)
assign_perm("view_document", user1, doc2)
assign_perm("view_document", user1, doc3)
assign_perm("change_document", user1, doc3)
assign_perm("view_document", user2, doc4)
grant_object(user1, doc2, "view_document")
grant_object(user1, doc3, "view_document")
grant_object(user1, doc3, "change_document")
grant_object(user2, doc4, "view_document")
self.client.force_authenticate(user1)
@@ -574,8 +574,8 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
owned by someone else with no explicit guardian grant -- mirrors
guardian's own ObjectPermissionChecker.has_perm() superuser shortcut.
"""
superuser = User.objects.create_superuser(username="admin")
other_user = User.objects.create_user(username="user2")
superuser = UserFactory(username="admin", superuser=True)
other_user = UserFactory(username="user2")
Document.objects.create(
title="Test",
content="content",
@@ -602,7 +602,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
THEN:
- MFA required error is returned
"""
user1 = User.objects.create_user(username="user1")
user1 = UserFactory(username="user1")
user1.set_password("password")
user1.save()
@@ -626,7 +626,7 @@ class TestApiAuth(DirectoriesMixin, APITestCase):
THEN:
- MFA code is required
"""
user1 = User.objects.create_user(username="user1")
user1 = UserFactory(username="user1")
user1.set_password("password")
user1.save()
@@ -688,7 +688,7 @@ class TestApiUser(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
def test_get_users(self) -> None:
@@ -858,10 +858,8 @@ class TestApiUser(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
regular_user = User.objects.create_user(username="regular_user")
regular_user.user_permissions.add(
*Permission.objects.all(),
)
regular_user = UserFactory(username="regular_user")
grant_all_global(regular_user)
self.client.force_authenticate(regular_user)
Authenticator.objects.create(
user=user1,
@@ -885,9 +883,9 @@ class TestApiUser(DirectoriesMixin, APITestCase):
- Only superusers can change superuser status
"""
user1 = User.objects.create_user(username="user1")
user1.user_permissions.add(*Permission.objects.all())
user2 = User.objects.create_superuser(username="user2")
user1 = UserFactory(username="user1")
grant_all_global(user1)
user2 = UserFactory(username="user2", superuser=True)
self.client.force_authenticate(user1)
@@ -972,9 +970,9 @@ class TestApiUser(DirectoriesMixin, APITestCase):
- Only superusers can change staff status
"""
user1 = User.objects.create_user(username="user1")
user1.user_permissions.add(*Permission.objects.all())
user2 = User.objects.create_superuser(username="user2")
user1 = UserFactory(username="user1")
grant_all_global(user1)
user2 = UserFactory(username="user2", superuser=True)
self.client.force_authenticate(user1)
@@ -1027,7 +1025,7 @@ class TestApiGroup(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
def test_get_groups(self) -> None:
@@ -1128,7 +1126,7 @@ class TestBulkEditObjectPermissions(APITestCase):
def setUp(self) -> None:
super().setUp()
self.temp_admin = User.objects.create_superuser(username="temp_admin")
self.temp_admin = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.temp_admin)
self.t1 = Tag.objects.create(name="t1")
@@ -1276,7 +1274,7 @@ class TestBulkEditObjectPermissions(APITestCase):
},
}
assign_perm("view_tag", self.user3, self.t1)
grant_object(self.user3, self.t1, "view_tag")
self.t1.owner = self.user3
self.t1.save()
@@ -1373,13 +1371,9 @@ class TestBulkEditObjectPermissions(APITestCase):
"""
self.t1.owner = self.user2
self.t1.save()
assign_perm("view_tag", self.user1, self.t1)
assign_perm("change_tag", self.user1, self.t1)
self.user1.user_permissions.add(
*Permission.objects.filter(
codename__in=["view_tag", "change_tag"],
),
)
grant_object(self.user1, self.t1, "view_tag")
grant_object(self.user1, self.t1, "change_tag")
grant_global(self.user1, "view_tag", "change_tag")
user1 = User.objects.get(pk=self.user1.pk)
self.client.force_authenticate(user=user1)
@@ -1426,13 +1420,9 @@ class TestBulkEditObjectPermissions(APITestCase):
"""
owned = Tag.objects.create(name="owned", owner=self.user1)
shared = Tag.objects.create(name="shared", owner=self.user2)
assign_perm("view_tag", self.user1, shared)
assign_perm("change_tag", self.user1, shared)
self.user1.user_permissions.add(
*Permission.objects.filter(
codename__in=["view_tag", "change_tag"],
),
)
grant_object(self.user1, shared, "view_tag")
grant_object(self.user1, shared, "change_tag")
grant_global(self.user1, "view_tag", "change_tag")
user1 = User.objects.get(pk=self.user1.pk)
self.client.force_authenticate(user=user1)
@@ -1473,14 +1463,10 @@ class TestBulkEditObjectPermissions(APITestCase):
"""
self.t1.owner = self.user2
self.t1.save()
assign_perm("view_tag", self.user1, self.t1)
assign_perm("change_tag", self.user1, self.t1)
assign_perm("delete_tag", self.user1, self.t1)
self.user1.user_permissions.add(
*Permission.objects.filter(
codename__in=["view_tag", "change_tag", "delete_tag"],
),
)
grant_object(self.user1, self.t1, "view_tag")
grant_object(self.user1, self.t1, "change_tag")
grant_object(self.user1, self.t1, "delete_tag")
grant_global(self.user1, "view_tag", "change_tag", "delete_tag")
user1 = User.objects.get(pk=self.user1.pk)
self.client.force_authenticate(user=user1)
@@ -1585,7 +1571,7 @@ class TestBulkEditObjectPermissions(APITestCase):
- Request succeeds and null is treated as an empty user list,
so the existing view permission is removed
"""
assign_perm("view_tag", self.user1, self.t1)
grant_object(self.user1, self.t1, "view_tag")
response = self.client.post(
"/api/bulk_edit_objects/",
@@ -1680,7 +1666,7 @@ class TestFullPermissionsFlag(APITestCase):
def setUp(self) -> None:
super().setUp()
self.admin = User.objects.create_superuser(username="admin")
self.admin = UserFactory(username="admin", superuser=True)
def test_full_perms_flag(self) -> None:
"""
@@ -1,3 +1,5 @@
from __future__ import annotations
import unicodedata
from typing import TYPE_CHECKING
from unittest import mock
@@ -7,8 +9,11 @@ import pytest
from django.core.files.uploadedfile import SimpleUploadedFile
if TYPE_CHECKING:
from rest_framework.test import APIClient
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from paperless_testing.dirs import PaperlessDirs
@pytest.fixture()
@@ -18,22 +23,14 @@ def consume_file_mock():
yield m
@pytest.fixture()
def directories(tmp_path, settings, _media_settings):
scratch = tmp_path / "scratch"
scratch.mkdir()
settings.SCRATCH_DIR = scratch
return scratch
@pytest.mark.django_db
class TestPostDocumentNFCNormalization:
def test_nfd_filename_normalized_to_nfc(
self,
admin_client,
admin_client: APIClient,
consume_file_mock: mock.MagicMock,
directories,
):
paperless_dirs: PaperlessDirs,
) -> None:
"""Uploaded file with NFD filename must have its name stored as NFC."""
nfd = unicodedata.normalize("NFD", "Rechnung März.pdf")
nfc = unicodedata.normalize("NFC", "Rechnung März.pdf")
@@ -69,10 +66,10 @@ class TestPostDocumentNFCNormalization:
def test_already_nfc_filename_unchanged(
self,
admin_client,
admin_client: APIClient,
consume_file_mock: mock.MagicMock,
directories,
):
paperless_dirs: PaperlessDirs,
) -> None:
"""Uploaded file with already-NFC filename must pass through unchanged."""
nfc = unicodedata.normalize("NFC", "Invoice_2024.pdf")
+5 -3
View File
@@ -8,7 +8,8 @@ from rest_framework import status
from rest_framework.authtoken.models import Token
from rest_framework.test import APITestCase
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
# see allauth.socialaccount.providers.openid.provider.OpenIDProvider
@@ -55,10 +56,11 @@ class TestApiProfile(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(
self.user = UserFactory(
username="temp_admin",
first_name="firstname",
last_name="surname",
superuser=True,
)
self.client.force_authenticate(user=self.user)
@@ -401,7 +403,7 @@ class TestApiTOTPViews(APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
def test_get_totp(self) -> None:
+40 -46
View File
@@ -7,10 +7,8 @@ import time_machine
from dateutil.relativedelta import relativedelta
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from django.utils import timezone
from guardian.shortcuts import assign_perm
from rest_framework import status
from rest_framework.test import APITestCase
@@ -27,10 +25,13 @@ from documents.models import Tag
from documents.models import Workflow
from documents.search import get_backend
from documents.search import reset_backend
from documents.tests.factories import DocumentFactory
from documents.tests.utils import DirectoriesMixin
from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
pytestmark = pytest.mark.search
@@ -40,7 +41,7 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
super().setUp()
reset_backend()
self.user = User.objects.create_superuser(username="temp_admin")
self.user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=self.user)
def tearDown(self) -> None:
@@ -949,9 +950,9 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
THEN:
- Terms only within docs user has access to are returned
"""
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u1.user_permissions.add(Permission.objects.get(codename="view_document"))
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
grant_global(u1, "view_document")
self.client.force_authenticate(user=u1)
@@ -991,7 +992,7 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data, ["applebaum", "apples"])
assign_perm("view_document", u1, d3)
grant_object(u1, d3, "view_document")
backend.add_or_update(d3)
response = self.client.get("/api/search/autocomplete/?term=app")
@@ -999,10 +1000,10 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertEqual(response.data, ["applebaum", "apples", "appletini"])
def test_search_autocomplete_group_revocation_is_immediate(self) -> None:
user = User.objects.create_user("group-user")
owner = User.objects.create_user("document-owner")
user = UserFactory(username="group-user")
owner = UserFactory(username="document-owner")
group = Group.objects.create(name="temporary-viewers")
user.user_permissions.add(Permission.objects.get(codename="view_document"))
grant_global(user, "view_document")
user.groups.add(group)
document = Document.objects.create(
@@ -1011,7 +1012,7 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
checksum="group-revocation",
owner=owner,
)
assign_perm("view_document", group, document)
grant_object(group, document, "view_document")
get_backend().add_or_update(document)
self.client.force_authenticate(user=user)
@@ -1091,11 +1092,9 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertIsNone(response.data["corrected_query"])
def test_search_spelling_suggestion_suppressed_for_private_terms(self) -> None:
owner = User.objects.create_user("owner")
attacker = User.objects.create_user("attacker")
attacker.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="owner")
attacker = UserFactory(username="attacker")
grant_global(attacker, "view_document")
backend = get_backend()
for i in range(5):
@@ -1222,11 +1221,9 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
THEN:
- The request is rejected
"""
owner = User.objects.create_user("owner")
attacker = User.objects.create_user("attacker")
attacker.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="owner")
attacker = UserFactory(username="attacker")
grant_global(attacker, "view_document")
private_seed = Document.objects.create(
title="private bank statement",
@@ -1534,11 +1531,11 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
- Only owned docs are returned for regular users
- All docs are returned for superuser
"""
superuser = User.objects.create_superuser("superuser")
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
u2.user_permissions.add(*Permission.objects.filter(codename="view_document"))
superuser = UserFactory(username="superuser", superuser=True)
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
grant_global(u1, "view_document")
grant_global(u2, "view_document")
Document.objects.create(checksum="1", content="test 1", owner=u1)
Document.objects.create(checksum="2", content="test 2", owner=u2)
@@ -1588,10 +1585,10 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
THEN:
- Only docs with granted view permissions are returned
"""
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u1.user_permissions.add(*Permission.objects.filter(codename="view_document"))
u2.user_permissions.add(*Permission.objects.filter(codename="view_document"))
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
grant_global(u1, "view_document")
grant_global(u2, "view_document")
d1 = Document.objects.create(checksum="1", content="test 1", owner=u1)
d2 = Document.objects.create(checksum="2", content="test 2", owner=u2)
@@ -1616,9 +1613,9 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
r = self.client.get("/api/documents/?query=test&owner__isnull=true")
self.assertEqual(r.data["count"], 1)
assign_perm("view_document", u1, d2)
assign_perm("view_document", u1, d3)
assign_perm("view_document", u2, d1)
grant_object(u1, d2, "view_document")
grant_object(u1, d3, "view_document")
grant_object(u2, d1, "view_document")
backend.add_or_update(d1)
backend.add_or_update(d2)
@@ -1641,8 +1638,8 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertEqual(r.data["count"], 1)
def test_search_sorting(self) -> None:
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u1 = UserFactory(username="user1")
u2 = UserFactory(username="user2")
c1 = Correspondent.objects.create(name="corres Ax")
c2 = Correspondent.objects.create(name="corres Cx")
c3 = Correspondent.objects.create(name="corres Bx")
@@ -1892,8 +1889,8 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
pk=5,
)
user1 = User.objects.create_user("bank user1")
user2 = User.objects.create_superuser("user2")
user1 = UserFactory(username="bank user1")
user2 = UserFactory(username="user2", superuser=True)
group1 = Group.objects.create(name="bank group1")
Group.objects.create(name="group2")
@@ -1925,7 +1922,7 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
sort_field="",
owner=user2,
)
assign_perm("view_savedview", user1, shared_view)
grant_object(user1, shared_view, "view_savedview")
mail_account1 = MailAccount.objects.create(name="bank mail account 1")
mail_account2 = MailAccount.objects.create(name="mail account 2")
mail_rule1 = MailRule.objects.create(
@@ -2018,12 +2015,9 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertEqual(returned.get(root.id), "latest content")
def test_global_search_filters_owned_mail_objects(self) -> None:
user1 = User.objects.create_user("mail-search-user")
user2 = User.objects.create_user("other-mail-search-user")
user1.user_permissions.add(
Permission.objects.get(codename="view_mailaccount"),
Permission.objects.get(codename="view_mailrule"),
)
user1 = UserFactory(username="mail-search-user")
user2 = UserFactory(username="other-mail-search-user")
grant_global(user1, "view_mailaccount", "view_mailrule")
own_account = MailAccount.objects.create(
name="bank owned account",
+10 -10
View File
@@ -33,7 +33,7 @@ class TestSearchQueryErrorStillBecomesA400:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -68,7 +68,7 @@ class TestLibraryDefectsPropagate:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -98,7 +98,7 @@ class TestLibraryDefectsPropagate:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -141,7 +141,7 @@ class TestSelectionPathsAgreeWithSearch:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -181,7 +181,7 @@ class TestSelectionPathsAgreeWithSearch:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -221,7 +221,7 @@ class TestSelectionPathsAgreeWithSearch:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -259,7 +259,7 @@ class TestSelectionPathsAgreeWithSearch:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -287,7 +287,7 @@ class TestSelectionPathsAgreeWithSearch:
{
"documents": [],
"all": True,
"filters": {"more_like_id": indexed_document.pk},
"filters": {"more_like_id": searchable_document.pk},
},
format="json",
)
@@ -298,7 +298,7 @@ class TestSelectionPathsAgreeWithSearch:
self,
admin_client: APIClient,
monkeypatch: pytest.MonkeyPatch,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -328,7 +328,7 @@ class TestSelectionPathsAgreeWithSearch:
{
"documents": [],
"all": True,
"filters": {"more_like_id": indexed_document.pk},
"filters": {"more_like_id": searchable_document.pk},
},
format="json",
)
@@ -36,7 +36,7 @@ class TestGetSearchEndpointEnforcesTheCap:
def test_query_one_over_the_cap_is_a_400(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -66,7 +66,7 @@ class TestGetSearchEndpointEnforcesTheCap:
def test_query_at_exactly_the_cap_is_accepted(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -86,7 +86,7 @@ class TestGetSearchEndpointEnforcesTheCap:
def test_an_ordinary_query_is_unaffected(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -112,7 +112,7 @@ class TestPostSelectionPathsEnforceTheCap:
def test_bulk_edit_query_one_over_the_cap_is_a_400(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -154,7 +154,7 @@ class TestPostSelectionPathsEnforceTheCap:
self,
bulk_update_task_mock: mock.MagicMock,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -187,7 +187,7 @@ class TestPostSelectionPathsEnforceTheCap:
def test_bulk_download_query_one_over_the_cap_is_a_400(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -236,7 +236,7 @@ class TestGlobalSearchEnforcesTheCapToo:
def test_query_one_over_the_cap_is_a_400(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -260,7 +260,7 @@ class TestGlobalSearchEnforcesTheCapToo:
def test_query_at_exactly_the_cap_is_accepted(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
@@ -38,7 +38,7 @@ class TestUnterminatedBracketReturnsA400:
def test_unterminated_bracket_is_a_400(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
query: str,
) -> None:
"""
@@ -59,7 +59,7 @@ class TestUnterminatedBracketReturnsA400:
def test_properly_closed_bracket_still_searches_cleanly(
self,
admin_client: APIClient,
indexed_document: Document,
searchable_document: Document,
) -> None:
"""
GIVEN:
+7 -11
View File
@@ -5,8 +5,6 @@ from datetime import timedelta
from pathlib import Path
from unittest import mock
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from django.utils import timezone
from rest_framework import status
@@ -14,8 +12,10 @@ from rest_framework.test import APITestCase
from documents.models import PaperlessTask
from documents.permissions import has_system_status_permission
from documents.tests.factories import PaperlessTaskFactory
from paperless import version
from paperless_testing.factories import PaperlessTaskFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
class TestSystemStatus(APITestCase):
@@ -23,9 +23,7 @@ class TestSystemStatus(APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(
username="temp_admin",
)
self.user = UserFactory(username="temp_admin", superuser=True)
self.tmp_dir = Path(tempfile.mkdtemp())
self.override = override_settings(MEDIA_ROOT=self.tmp_dir)
self.override.enable()
@@ -96,7 +94,7 @@ class TestSystemStatus(APITestCase):
response = self.client.get(self.ENDPOINT)
self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)
self.assertEqual(response["WWW-Authenticate"], "Token")
normal_user = User.objects.create_user(username="normal_user")
normal_user = UserFactory(username="normal_user")
self.client.force_login(normal_user)
response = self.client.get(self.ENDPOINT)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
@@ -107,10 +105,8 @@ class TestSystemStatus(APITestCase):
response = self.client.get(self.ENDPOINT)
self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)
user = User.objects.create_user(username="status_user")
user.user_permissions.add(
Permission.objects.get(codename="view_system_monitoring"),
)
user = UserFactory(username="status_user")
grant_global(user, "view_system_monitoring")
self.client.force_login(user)
response = self.client.get(self.ENDPOINT)
+20 -38
View File
@@ -11,22 +11,21 @@ from datetime import timedelta
from unittest import mock
import pytest
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.utils import timezone
from guardian.shortcuts import assign_perm
from rest_framework import status
from rest_framework.test import APIClient
from documents.filters import PaperlessTaskFilterSet
from documents.models import PaperlessTask
from documents.tests.factories import DocumentFactory
from documents.tests.factories import PaperlessTaskFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import PaperlessTaskFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
pytestmark = pytest.mark.api
ENDPOINT = "/api/tasks/"
ACCEPT_V10 = "application/json; version=10"
ACCEPT_V9 = "application/json; version=9"
@@ -346,21 +345,16 @@ class TestGetTasksV10:
self,
admin_user: User,
regular_user: User,
user_client: APIClient,
) -> None:
"""Regular users see their own tasks and unowned (system) tasks; other users' tasks are hidden."""
regular_user.user_permissions.add(
Permission.objects.get(codename="view_paperlesstask"),
)
client = APIClient()
client.force_authenticate(user=regular_user)
client.credentials(HTTP_ACCEPT=ACCEPT_V10)
grant_global(regular_user, "view_paperlesstask")
PaperlessTaskFactory(owner=admin_user) # other user — not visible
unowned_task = PaperlessTaskFactory() # unowned (system task) — visible
own_task = PaperlessTaskFactory(owner=regular_user)
response = client.get(ENDPOINT)
response = user_client.get(ENDPOINT)
assert response.status_code == status.HTTP_200_OK
assert response.data["count"] == 2
@@ -590,9 +584,7 @@ class TestGetTasksV9:
regular_user: User,
) -> None:
"""Non-staff users see their own tasks plus unowned tasks via v9 API."""
regular_user.user_permissions.add(
Permission.objects.get(codename="view_paperlesstask"),
)
grant_global(regular_user, "view_paperlesstask")
client = APIClient()
client.force_authenticate(user=regular_user)
@@ -732,19 +724,17 @@ class TestAcknowledge:
assert response.status_code == status.HTTP_403_FORBIDDEN
def test_succeeds_with_change_permission(self, regular_user: User) -> None:
def test_succeeds_with_change_permission(
self,
regular_user: User,
user_client: APIClient,
) -> None:
"""Users granted change_paperlesstask permission can acknowledge tasks."""
regular_user.user_permissions.add(
Permission.objects.get(codename="change_paperlesstask"),
)
grant_global(regular_user, "change_paperlesstask")
regular_user.save()
client = APIClient()
client.force_authenticate(user=regular_user)
client.credentials(HTTP_ACCEPT=ACCEPT_V10)
task = PaperlessTaskFactory()
response = client.post(
response = user_client.post(
ENDPOINT + "acknowledge/",
{"tasks": [task.id]},
format="json",
@@ -807,9 +797,7 @@ class TestSummaryPermissions:
regular_user,
) -> None:
"""A user with view_system_monitoring but no document permissions can access summary/."""
regular_user.user_permissions.add(
Permission.objects.get(codename="view_system_monitoring"),
)
grant_global(regular_user, "view_system_monitoring")
response = user_client.get(ENDPOINT + "summary/")
@@ -822,9 +810,7 @@ class TestSummaryPermissions:
admin_user,
) -> None:
"""Monitoring user sees aggregate data for all tasks, not just unowned ones."""
regular_user.user_permissions.add(
Permission.objects.get(codename="view_system_monitoring"),
)
grant_global(regular_user, "view_system_monitoring")
PaperlessTaskFactory(
owner=admin_user,
task_type=PaperlessTask.TaskType.CONSUME_FILE,
@@ -845,9 +831,7 @@ class TestSummaryPermissions:
) -> None:
"""A regular user with view_paperlesstask but not view_system_monitoring sees only
their own tasks and unowned tasks in the summary, not other users' tasks."""
regular_user.user_permissions.add(
Permission.objects.get(codename="view_paperlesstask"),
)
grant_global(regular_user, "view_paperlesstask")
PaperlessTaskFactory(
owner=regular_user,
@@ -1012,9 +996,7 @@ class TestDuplicateDocumentsPermissions:
@pytest.fixture()
def user_v9_client(self, regular_user: User) -> APIClient:
regular_user.user_permissions.add(
Permission.objects.get(codename="view_paperlesstask"),
)
grant_global(regular_user, "view_paperlesstask")
client = APIClient()
client.force_authenticate(user=regular_user)
client.credentials(HTTP_ACCEPT=ACCEPT_V9)
@@ -1085,7 +1067,7 @@ class TestDuplicateDocumentsPermissions:
) -> None:
"""A user with explicit guardian view_document permission sees the duplicate_of document."""
doc = DocumentFactory(owner=admin_user, title="Granted Doc")
assign_perm("view_document", regular_user, doc)
grant_object(regular_user, doc, "view_document")
PaperlessTaskFactory(
owner=regular_user,
status=PaperlessTask.Status.SUCCESS,
+9 -9
View File
@@ -1,21 +1,21 @@
from datetime import date
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.core.cache import cache
from rest_framework import status
from rest_framework.test import APITestCase
from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_all_global
class TestTrashAPI(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_user(username="temp_admin")
self.user.user_permissions.add(*Permission.objects.all())
self.user = UserFactory(username="temp_admin")
grant_all_global(self.user)
self.client.force_authenticate(user=self.user)
cache.clear()
@@ -70,7 +70,7 @@ class TestTrashAPI(DirectoriesMixin, APITestCase):
self.assertEqual(Document.global_objects.count(), 0)
def test_trash_list_requires_global_document_view_permission(self) -> None:
user = User.objects.create_user(username="trash_owner")
user = UserFactory(username="trash_owner")
document = Document.objects.create(title="Owned", owner=user)
document.delete()
self.client.force_authenticate(user)
@@ -140,7 +140,7 @@ class TestTrashAPI(DirectoriesMixin, APITestCase):
created=date(2023, 1, 2),
)
document_not_owned.delete()
user2 = User.objects.create_user(username="user2")
user2 = UserFactory(username="user2")
document_u2 = Document.objects.create(
title="Title3",
content="content3",
@@ -158,7 +158,7 @@ class TestTrashAPI(DirectoriesMixin, APITestCase):
self.assertEqual(resp.data["results"][1]["id"], document_u1.pk)
# superuser sees all documents
superuser = User.objects.create_superuser(username="superuser")
superuser = UserFactory(username="superuser", superuser=True)
self.client.force_authenticate(user=superuser)
resp = self.client.get("/api/trash/")
self.assertEqual(resp.status_code, status.HTTP_200_OK)
@@ -174,7 +174,7 @@ class TestTrashAPI(DirectoriesMixin, APITestCase):
- 403 Forbidden
"""
user2 = User.objects.create_user(username="user2")
user2 = UserFactory(username="user2")
document = Document.objects.create(
title="Title",
content="content",
+5 -5
View File
@@ -1,13 +1,13 @@
import json
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from rest_framework import status
from rest_framework.test import APITestCase
from documents.tests.utils import DirectoriesMixin
from paperless.version import __full_version_str__
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
class TestApiUiSettings(DirectoriesMixin, APITestCase):
@@ -15,7 +15,7 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.test_user = User.objects.create_superuser(username="test")
self.test_user = UserFactory(username="test", superuser=True)
self.test_user.first_name = "Test"
self.test_user.last_name = "User"
self.test_user.save()
@@ -91,7 +91,7 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
)
def test_api_set_ui_settings_insufficient_global_permissions(self) -> None:
not_superuser = User.objects.create_user(username="test_not_superuser")
not_superuser = UserFactory(username="test_not_superuser")
self.client.force_authenticate(user=not_superuser)
settings = {
@@ -111,7 +111,7 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
def test_api_set_ui_settings_sufficient_global_permissions(self) -> None:
not_superuser = User.objects.create_user(username="test_not_superuser")
not_superuser = UserFactory(username="test_not_superuser")
not_superuser.user_permissions.add(
*Permission.objects.filter(codename__contains="uisettings"),
)
+3 -2
View File
@@ -14,7 +14,8 @@ from documents.models import Tag
from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
class TestApiWorkflows(DirectoriesMixin, APITestCase):
@@ -25,7 +26,7 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
user = User.objects.create_superuser(username="temp_admin")
user = UserFactory(username="temp_admin", superuser=True)
self.client.force_authenticate(user=user)
self.user2 = User.objects.create(username="user2")
self.user3 = User.objects.create(username="user3")
+75 -75
View File
@@ -2,8 +2,8 @@ import shutil
from collections.abc import Generator
from contextlib import contextmanager
from pathlib import Path
from unittest import mock
import pytest
from django.conf import settings
from django.test import TestCase
from django.test import override_settings
@@ -18,11 +18,11 @@ from documents.models import Document
from documents.models import Tag
from documents.plugins.base import StopConsumeTaskError
from documents.tests.utils import ConsumeTaskMixin
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import DummyProgressManager
from documents.tests.utils import FileSystemAssertsMixin
from documents.tests.utils import SampleDirMixin
from paperless.models import ApplicationConfiguration
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.fakes.progress import FakeProgressManager
class GetReaderPluginMixin:
@@ -31,7 +31,7 @@ class GetReaderPluginMixin:
reader = BarcodePlugin(
ConsumableDocument(DocumentSource.ConsumeFolder, original_file=filepath),
DocumentMetadataOverrides(),
DummyProgressManager(filepath.name, None),
FakeProgressManager(filepath.name, None),
self.dirs.scratch_dir,
"task-id",
)
@@ -86,6 +86,7 @@ class TestBarcode(
self.assertDictEqual(separator_page_numbers, {1: False})
@override_settings(CONSUMER_ENABLE_ASN_BARCODE=True)
@pytest.mark.usefixtures("fake_progress_manager")
def test_asn_barcode_duplicate_in_trash_fails(self) -> None:
"""
GIVEN:
@@ -110,15 +111,14 @@ class TestBarcode(
dupe_asn = settings.SCRATCH_DIR / "barcode-39-asn-123-second.pdf"
shutil.copy(test_file, dupe_asn)
with mock.patch("documents.tasks.ProgressManager", DummyProgressManager):
with self.assertRaisesRegex(ConsumerError, r"ASN 123.*trash"):
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dupe_asn,
),
None,
)
with self.assertRaisesRegex(ConsumerError, r"ASN 123.*trash"):
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dupe_asn,
),
None,
)
@override_settings(
CONSUMER_BARCODE_TIFF_SUPPORT=True,
@@ -606,6 +606,7 @@ class TestBarcodeNewConsume(
TestCase,
):
@override_settings(CONSUMER_ENABLE_BARCODES=True)
@pytest.mark.usefixtures("fake_progress_manager")
def test_consume_barcode_file(self) -> None:
"""
GIVEN:
@@ -624,34 +625,33 @@ class TestBarcodeNewConsume(
overrides = DocumentMetadataOverrides(tag_ids=[1, 2, 9])
with mock.patch("documents.tasks.ProgressManager", DummyProgressManager):
self.assertEqual(
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=temp_copy,
),
overrides,
self.assertEqual(
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=temp_copy,
),
{"reason": "Barcode splitting complete!"},
)
# 2 new document consume tasks created
self.assertEqual(self.consume_file_mock.call_count, 2)
overrides,
),
{"reason": "Barcode splitting complete!"},
)
# 2 new document consume tasks created
self.assertEqual(self.consume_file_mock.call_count, 2)
self.assertIsNotFile(temp_copy)
self.assertIsNotFile(temp_copy)
# Check the split files exist
# Check the original_path is set
# Check the source is unchanged
# Check the overrides are unchanged
for (
new_input_doc,
new_doc_overrides,
) in self.get_all_consume_task_call_args():
self.assertIsFile(new_input_doc.original_file)
self.assertEqual(new_input_doc.original_path, temp_copy)
self.assertEqual(new_input_doc.source, DocumentSource.ConsumeFolder)
self.assertEqual(overrides, new_doc_overrides)
# Check the split files exist
# Check the original_path is set
# Check the source is unchanged
# Check the overrides are unchanged
for (
new_input_doc,
new_doc_overrides,
) in self.get_all_consume_task_call_args():
self.assertIsFile(new_input_doc.original_file)
self.assertEqual(new_input_doc.original_path, temp_copy)
self.assertEqual(new_input_doc.source, DocumentSource.ConsumeFolder)
self.assertEqual(overrides, new_doc_overrides)
class TestAsnBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, TestCase):
@@ -660,7 +660,7 @@ class TestAsnBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
reader = BarcodePlugin(
ConsumableDocument(DocumentSource.ConsumeFolder, original_file=filepath),
DocumentMetadataOverrides(),
DummyProgressManager(filepath.name, None),
FakeProgressManager(filepath.name, None),
self.dirs.scratch_dir,
"task-id",
)
@@ -745,6 +745,7 @@ class TestAsnBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
self.assertEqual(asn, None)
@override_settings(CONSUMER_ENABLE_ASN_BARCODE=True)
@pytest.mark.usefixtures("fake_progress_manager")
def test_consume_barcode_file_asn_assignment(self) -> None:
"""
GIVEN:
@@ -762,19 +763,18 @@ class TestAsnBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
dst = settings.SCRATCH_DIR / "barcode-39-asn-123.pdf"
shutil.copy(test_file, dst)
with mock.patch("documents.tasks.ProgressManager", DummyProgressManager):
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dst,
),
None,
)
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dst,
),
None,
)
document = Document.objects.first()
assert document is not None
document = Document.objects.first()
assert document is not None
self.assertEqual(document.archive_serial_number, 123)
self.assertEqual(document.archive_serial_number, 123)
def test_scan_file_for_qrcode_without_upscale(self) -> None:
"""
@@ -819,7 +819,7 @@ class TestTagBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
reader = BarcodePlugin(
ConsumableDocument(DocumentSource.ConsumeFolder, original_file=filepath),
DocumentMetadataOverrides(),
DummyProgressManager(filepath.name, None),
FakeProgressManager(filepath.name, None),
self.dirs.scratch_dir,
"task-id",
)
@@ -1024,6 +1024,7 @@ class TestTagBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
CELERY_TASK_ALWAYS_EAGER=True,
OCR_MODE="auto",
)
@pytest.mark.usefixtures("fake_progress_manager")
def test_consume_barcode_file_tag_split_and_assignment(self) -> None:
"""
GIVEN:
@@ -1042,34 +1043,33 @@ class TestTagBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
dst = settings.SCRATCH_DIR / "split-by-tag-basic.pdf"
shutil.copy(test_file, dst)
with mock.patch("documents.tasks.ProgressManager", DummyProgressManager):
result = tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dst,
),
None,
)
result = tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dst,
),
None,
)
self.assertEqual(result, {"reason": "Barcode splitting complete!"})
self.assertEqual(result, {"reason": "Barcode splitting complete!"})
documents = Document.objects.all().order_by("id")
self.assertEqual(documents.count(), 3)
documents = Document.objects.all().order_by("id")
self.assertEqual(documents.count(), 3)
doc1 = documents[0]
self.assertEqual(doc1.tags.count(), 0)
doc1 = documents[0]
self.assertEqual(doc1.tags.count(), 0)
doc2 = documents[1]
self.assertEqual(doc2.tags.count(), 1)
_tag_1 = doc2.tags.first()
assert _tag_1 is not None
self.assertEqual(_tag_1.name, "invoice")
doc2 = documents[1]
self.assertEqual(doc2.tags.count(), 1)
_tag_1 = doc2.tags.first()
assert _tag_1 is not None
self.assertEqual(_tag_1.name, "invoice")
doc3 = documents[2]
self.assertEqual(doc3.tags.count(), 1)
_tag_2 = doc3.tags.first()
assert _tag_2 is not None
self.assertEqual(_tag_2.name, "receipt")
doc3 = documents[2]
self.assertEqual(doc3.tags.count(), 1)
_tag_2 = doc3.tags.first()
assert _tag_2 is not None
self.assertEqual(_tag_2.name, "receipt")
@override_settings(
CONSUMER_ENABLE_TAG_BARCODE=True,
+6 -6
View File
@@ -10,7 +10,6 @@ from django.contrib.auth.models import User
from django.db import connection
from django.test import TestCase
from django.test.utils import CaptureQueriesContext
from guardian.shortcuts import assign_perm
from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
@@ -23,7 +22,8 @@ from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.permissions import set_permissions_for_objects
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.permissions import grant_object
class TestBulkEdit(DirectoriesMixin, TestCase):
@@ -440,7 +440,7 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
def test_set_permissions(self, m) -> None:
doc_ids = [self.doc1.id, self.doc2.id, self.doc3.id]
assign_perm("view_document", self.group1, self.doc1)
grant_object(self.group1, self.doc1, "view_document")
permissions = {
"view": {
@@ -482,8 +482,8 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
self.doc1.owner = self.user1
self.doc1.save()
assign_perm("view_document", self.user1, self.doc1)
assign_perm("view_document", self.group1, self.doc1)
grant_object(self.user1, self.doc1, "view_document")
grant_object(self.group1, self.doc1, "view_document")
permissions = {
"view": {
@@ -609,7 +609,7 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
self.doc1.owner = self.user1
self.doc1.save()
self.user1.groups.add(self.group1)
assign_perm("view_document", self.group1, self.doc1)
grant_object(self.group1, self.doc1, "view_document")
bulk_edit.set_permissions(
[self.doc1.id],
+4 -13
View File
@@ -1,5 +1,4 @@
import pickle
import re
import warnings
from datetime import UTC
from datetime import datetime
@@ -28,21 +27,13 @@ from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.factories import DocumentFactory
from documents.tests.factories import TagFactory
from documents.tests.utils import DirectoriesMixin
from documents.tests.helpers import dummy_preprocess
from paperless.settings import CLASSIFIER_LANGUAGES
from paperless.signed_pickle import HMAC_SIZE
from paperless.signed_pickle import signed_pickle_dumps
def dummy_preprocess(content: str) -> str:
"""
Simpler, faster pre-processing for testing purposes
"""
content = content.lower().strip()
content = re.sub(r"\s+", " ", content)
return content
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import TagFactory
class TestClassifier(DirectoriesMixin, TestCase):
+8 -7
View File
@@ -30,11 +30,12 @@ from documents.models import Tag
from documents.parsers import ParseError
from documents.plugins.helpers import ProgressStatusOptions
from documents.tasks import sanity_check
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import DummyProgressManager
from documents.tests.utils import FileSystemAssertsMixin
from documents.tests.utils import GetConsumerMixin
from paperless_mail.models import MailRule
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.fakes.progress import FakeProgressManager
class _BaseNewStyleParser:
@@ -769,14 +770,14 @@ class TestConsumer(
original_modified = timezone.now() - datetime.timedelta(days=1)
Document.objects.filter(pk=root_doc.pk).update(modified=original_modified)
actor = User.objects.create_user(
actor = UserFactory(
username="actor",
email="actor@example.com",
password="password",
)
version_file = self.get_test_file2()
status = DummyProgressManager(version_file.name, None)
status = FakeProgressManager(version_file.name, None)
overrides = DocumentMetadataOverrides(
version_label="v2",
actor_id=actor.pk,
@@ -839,7 +840,7 @@ class TestConsumer(
assert root_doc is not None
version_file = self.get_test_file2()
status = DummyProgressManager(version_file.name, None)
status = FakeProgressManager(version_file.name, None)
overrides = DocumentMetadataOverrides(
filename="valid_pdf_version-upload",
actor_id=999999,
@@ -896,7 +897,7 @@ class TestConsumer(
assert root_doc is not None
def consume_version(version_file: Path) -> Document:
status = DummyProgressManager(version_file.name, None)
status = FakeProgressManager(version_file.name, None)
overrides = DocumentMetadataOverrides()
doc = ConsumableDocument(
DocumentSource.ApiUpload,
@@ -9,11 +9,11 @@ from django.test.utils import CaptureQueriesContext
from rest_framework import status
from documents.models import Document
from documents.tests.factories import DocumentFactory
from documents.versioning import LATEST_VERSION_CONTENT_PREFETCH_ATTR
from documents.versioning import has_prefetched_effective_content
from documents.versioning import latest_version_content_prefetch
from documents.views import DocumentViewSet
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from rest_framework.test import APIClient
+49 -39
View File
@@ -2,8 +2,8 @@ import datetime as dt
import os
import shutil
from pathlib import Path
from unittest import mock
import pytest
from django.test import TestCase
from django.test import override_settings
from pdfminer.high_level import extract_text
@@ -15,22 +15,26 @@ from documents.data_models import ConsumableDocument
from documents.data_models import DocumentSource
from documents.double_sided import STAGING_FILE_NAME
from documents.double_sided import TIMEOUT_MINUTES
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import DummyProgressManager
from documents.tests.utils import FileSystemAssertsMixin
from documents.tests.utils import SampleDirMixin
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
@pytest.mark.usefixtures("fake_progress_manager")
@override_settings(
CONSUMER_RECURSIVE=True,
CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED=True,
)
class TestDoubleSided(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
SAMPLE_DIR = Path(__file__).parent / "samples"
class TestDoubleSided(
DirectoriesMixin,
FileSystemAssertsMixin,
SampleDirMixin,
TestCase,
):
def setUp(self) -> None:
super().setUp()
self.dirs.double_sided_dir = self.dirs.consumption_dir / "double-sided"
self.dirs.double_sided_dir.mkdir()
self.double_sided_dir = self.dirs.consumption_dir / "double-sided"
self.double_sided_dir.mkdir()
self.staging_file = self.dirs.scratch_dir / STAGING_FILE_NAME
def consume_file(self, srcname, dstname: str | Path = "foo.pdf"):
@@ -39,20 +43,16 @@ class TestDoubleSided(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
destination file does not exist afterwards
"""
src = self.SAMPLE_DIR / srcname
dst = self.dirs.double_sided_dir / dstname
dst = self.double_sided_dir / dstname
dst.parent.mkdir(parents=True, exist_ok=True)
shutil.copy(src, dst)
with mock.patch(
"documents.tasks.ProgressManager",
DummyProgressManager,
):
msg = tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dst,
),
None,
)
msg = tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=dst,
),
None,
)
self.assertIsNotFile(dst)
return msg
@@ -214,31 +214,41 @@ class TestDoubleSided(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
self.assertIsNotFile(self.staging_file)
self.assertIsInstance(msg.get("document_id"), int)
def test_subdirectory_upload(self) -> None:
def test_consume_double_sided_in_nested_dir(self) -> None:
"""
GIVEN:
- A staging file exists
WHEN:
- A file gets uploaded into foo/bar/double-sided
or double-sided/foo/bar
- A file is uploaded into foo/bar/double-sided
THEN:
- The collated file gets put into foo/bar
- The collated file is placed into foo/bar
"""
# TODO: parameterize this instead
for path in [
Path("foo") / "bar" / "double-sided",
Path("double-sided") / "foo" / "bar",
]:
with self.subTest(path=str(path)):
# Ensure we get fresh directories for each run
self.tearDown()
self.setUp()
self.create_staging_file()
self.consume_file(
"double-sided-odd.pdf",
Path("foo") / "bar" / "double-sided" / "foo.pdf",
)
self.assertIsFile(
self.dirs.consumption_dir / "foo" / "bar" / "foo-collated.pdf",
)
self.create_staging_file()
self.consume_file("double-sided-odd.pdf", Path(path) / "foo.pdf")
self.assertIsFile(
self.dirs.consumption_dir / "foo" / "bar" / "foo-collated.pdf",
)
def test_consume_double_sided_with_nested_subdir(self) -> None:
"""
GIVEN:
- A staging file exists
WHEN:
- A file is uploaded into double-sided/foo/bar
THEN:
- The collated file is placed into foo/bar
"""
self.create_staging_file()
self.consume_file(
"double-sided-odd.pdf",
Path("double-sided") / "foo" / "bar" / "foo.pdf",
)
self.assertIsFile(
self.dirs.consumption_dir / "foo" / "bar" / "foo-collated.pdf",
)
@override_settings(CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED=False)
def test_disabled_double_sided_dir_upload(self) -> None:
+5 -5
View File
@@ -8,7 +8,6 @@ from unittest import mock
import pytest
from auditlog.context import disable_auditlog
from django.conf import settings
from django.contrib.auth.models import User
from django.db import DatabaseError
from django.db import connection
from django.test import TestCase
@@ -30,9 +29,10 @@ from documents.models import DocumentType
from documents.models import StoragePath
from documents.serialisers import DocumentSerializer
from documents.tasks import empty_trash
from documents.tests.factories import DocumentFactory
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
@@ -1323,7 +1323,7 @@ class TestFilenameGeneration(DirectoriesMixin, TestCase):
- Document without owner returns "none"
"""
u1 = User.objects.create_user("user1")
u1 = UserFactory(username="user1")
owned_doc = Document.objects.create(
title="The Title",
+4 -4
View File
@@ -13,10 +13,10 @@ import pytest
from documents.file_handling import generate_filename
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from documents.tests.factories import StoragePathFactory
from documents.tests.factories import TagFactory
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import StoragePathFactory
from paperless_testing.factories import TagFactory
@pytest.mark.django_db
+2 -2
View File
@@ -20,8 +20,8 @@ if TYPE_CHECKING:
from documents.file_handling import generate_filename
from documents.models import Document
from documents.tasks import update_document_content_maybe_archive_file
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
sample_file: Path = Path(__file__).parent / "samples" / "simple.pdf"
@@ -26,7 +26,6 @@ from django.test import override_settings
from django.utils import timezone
from guardian.models import GroupObjectPermission
from guardian.models import UserObjectPermission
from guardian.shortcuts import assign_perm
from documents.management.commands import document_exporter
from documents.models import Correspondent
@@ -46,11 +45,12 @@ from documents.models import WorkflowTrigger
from documents.sanity_checker import check_sanity
from documents.settings import EXPORTER_FILE_NAME
from documents.settings import EXPORTER_SHARE_LINK_BUNDLE_NAME
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from documents.tests.utils import SampleDirMixin
from documents.tests.utils import paperless_environment
from paperless_mail.models import MailAccount
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.dirs import paperless_environment
from paperless_testing.permissions import grant_object
@pytest.mark.management
@@ -105,8 +105,8 @@ class TestExportImport(
user=self.user,
)
assign_perm("view_document", self.user2, self.d2)
assign_perm("view_document", self.group1, self.d3)
grant_object(self.user2, self.d2, "view_document")
grant_object(self.group1, self.d3, "view_document")
self.t1 = Tag.objects.create(name="t")
self.dt1 = DocumentType.objects.create(name="dt")
+1 -1
View File
@@ -8,7 +8,7 @@ from django.core.management import call_command
from django.test import TestCase
from documents.models import Document
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import DocumentFactory
@pytest.mark.management
@@ -15,9 +15,9 @@ from documents.management.commands.document_importer import _deserialize_record
from documents.models import Document
from documents.settings import EXPORTER_ARCHIVE_NAME
from documents.settings import EXPORTER_FILE_NAME
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from documents.tests.utils import SampleDirMixin
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
@pytest.mark.management
@@ -14,12 +14,12 @@ from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from documents.tests.factories import DocumentTypeFactory
from documents.tests.factories import StoragePathFactory
from documents.tests.factories import TagFactory
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import DocumentTypeFactory
from paperless_testing.factories import StoragePathFactory
from paperless_testing.factories import TagFactory
# ---------------------------------------------------------------------------
# Module-level type aliases
@@ -7,7 +7,7 @@ from django.contrib.auth.models import User
from django.core.management import call_command
from django.test import TestCase
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
@pytest.mark.management
@@ -9,8 +9,8 @@ from django.test import TestCase
from documents.management.commands.document_thumbnails import _process_document
from documents.models import Document
from documents.parsers import get_default_thumbnail
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
@pytest.mark.management
+4 -4
View File
@@ -7,10 +7,10 @@ from documents import matching
from documents.models import Document
from documents.models import MatchingModel
from documents.signals import document_consumption_finished
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from documents.tests.factories import DocumentTypeFactory
from documents.tests.factories import TagFactory
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import DocumentTypeFactory
from paperless_testing.factories import TagFactory
@pytest.fixture(
@@ -2,8 +2,6 @@ import json
from unittest import mock
from auditlog.models import LogEntry
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.test import TestCase
from rest_framework import status
@@ -12,6 +10,8 @@ from rest_framework.test import APITestCase
from documents.bulk_edit import merge_as_versions
from documents.models import Document
from documents.serialisers import MergeDocumentsAsVersionsSerializer
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
class TestMergeDocumentsAsVersionsSerializer(TestCase):
@@ -294,7 +294,7 @@ class TestMergeDocumentsAsVersions(TestCase):
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.search.get_backend")
def test_writes_audit_log_entry(self, *_mocks) -> None:
user = User.objects.create_user(username="merger")
user = UserFactory(username="merger")
root = Document.objects.create(checksum="A", title="Root")
source = Document.objects.create(checksum="B", title="Source")
LogEntry.objects.all().delete()
@@ -335,12 +335,8 @@ class TestMergeDocumentsAsVersions(TestCase):
class TestMergeDocumentsAsVersionsAPI(APITestCase):
def setUp(self) -> None:
self.user = User.objects.create_user(username="user")
self.user.user_permissions.add(
Permission.objects.get(codename="change_document"),
Permission.objects.get(codename="view_document"),
Permission.objects.get(codename="delete_document"),
)
self.user = UserFactory(username="user")
grant_global(self.user, "change_document", "view_document", "delete_document")
self.doc1 = Document.objects.create(
checksum="A",
title="A",
@@ -382,7 +378,7 @@ class TestMergeDocumentsAsVersionsAPI(APITestCase):
@mock.patch("documents.views.bulk_edit.merge_as_versions")
def test_requires_change_permission(self, merge_mock) -> None:
merge_mock.__name__ = "merge_as_versions"
user = User.objects.create_user(username="no-change")
user = UserFactory(username="no-change")
self.doc1.owner = user
self.doc1.save()
self.doc2.owner = user
@@ -405,11 +401,8 @@ class TestMergeDocumentsAsVersionsAPI(APITestCase):
def test_requires_delete_permission(self, merge_mock) -> None:
merge_mock.__name__ = "merge_as_versions"
# Owns them and may change them, but may not make them stop being documents
user = User.objects.create_user(username="no-delete")
user.user_permissions.add(
Permission.objects.get(codename="change_document"),
Permission.objects.get(codename="view_document"),
)
user = UserFactory(username="no-delete")
grant_global(user, "change_document", "view_document")
for doc in (self.doc1, self.doc2):
doc.owner = user
doc.save()
@@ -1,4 +1,4 @@
from documents.tests.utils import TestMigrations
from paperless_testing.migrations import TestMigrations
SAVED_VIEWS_KEY = "saved_views"
DASHBOARD_VIEWS_VISIBLE_IDS_KEY = "dashboard_views_visible_ids"
@@ -7,7 +7,7 @@ from django.conf import settings
from django.db import connection
from django.test import override_settings
from documents.tests.utils import TestMigrations
from paperless_testing.migrations import TestMigrations
def _sha256(data: bytes) -> str:
@@ -1,4 +1,4 @@
from documents.tests.utils import TestMigrations
from paperless_testing.migrations import TestMigrations
class TestMigrateShareLinkBundlePermissions(TestMigrations):
+2 -2
View File
@@ -2,8 +2,8 @@ import pytest
from documents.models import Correspondent
from documents.models import Document
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import DocumentFactory
@pytest.mark.django_db
@@ -1,15 +1,13 @@
from __future__ import annotations
from http import HTTPStatus
from typing import TYPE_CHECKING
from unittest.mock import patch
import pytest
from django.contrib.auth.models import AnonymousUser
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from guardian.shortcuts import assign_perm
from rest_framework.test import APIClient
from documents.matching import match_correspondents
@@ -24,11 +22,17 @@ from documents.permissions import permitted_document_ids
from documents.permissions import permitted_object_ids
from documents.permissions import restrict_queryset_to_visible
from documents.serialisers import _get_viewable_duplicates
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from documents.tests.factories import DocumentTypeFactory
from documents.tests.factories import StoragePathFactory
from documents.tests.factories import TagFactory
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import DocumentTypeFactory
from paperless_testing.factories import StoragePathFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
if TYPE_CHECKING:
from paperless_testing.dirs import PaperlessDirs
def assert_visible_document_ids(actual_ids, *, expected_visible, expected_hidden):
@@ -47,8 +51,8 @@ def assert_visible_document_ids(actual_ids, *, expected_visible, expected_hidden
@pytest.mark.django_db
class TestPermittedDocumentIdsSecurity:
def test_owner_sees_own_document(self):
user = User.objects.create_user(username="alice")
stranger = User.objects.create_user(username="mallory")
user = UserFactory(username="alice")
stranger = UserFactory(username="mallory")
owned = DocumentFactory(owner=user)
strangers_doc = DocumentFactory(owner=stranger)
@@ -61,7 +65,7 @@ class TestPermittedDocumentIdsSecurity:
)
def test_unowned_document_visible_to_everyone(self):
user = User.objects.create_user(username="alice")
user = UserFactory(username="alice")
unowned = DocumentFactory(owner=None)
assert_visible_document_ids(
@@ -71,12 +75,12 @@ class TestPermittedDocumentIdsSecurity:
)
def test_explicit_user_permission_grants_visibility(self):
grantee = User.objects.create_user(username="alice")
stranger = User.objects.create_user(username="mallory")
owner = User.objects.create_user(username="owner")
grantee = UserFactory(username="alice")
stranger = UserFactory(username="mallory")
owner = UserFactory(username="owner")
shared = DocumentFactory(owner=owner)
not_shared = DocumentFactory(owner=owner)
assign_perm("view_document", grantee, shared)
grant_object(grantee, shared, "view_document")
assert_visible_document_ids(
permitted_document_ids(grantee),
@@ -90,13 +94,13 @@ class TestPermittedDocumentIdsSecurity:
)
def test_explicit_group_permission_grants_visibility_to_members_only(self):
owner = User.objects.create_user(username="owner")
member = User.objects.create_user(username="member")
non_member = User.objects.create_user(username="non_member")
owner = UserFactory(username="owner")
member = UserFactory(username="member")
non_member = UserFactory(username="non_member")
group = Group.objects.create(name="finance")
member.groups.add(group)
shared = DocumentFactory(owner=owner)
assign_perm("view_document", group, shared)
grant_object(group, shared, "view_document")
assert_visible_document_ids(
permitted_document_ids(member),
@@ -110,7 +114,7 @@ class TestPermittedDocumentIdsSecurity:
)
def test_soft_deleted_document_excluded_by_default(self):
owner = User.objects.create_user(username="owner")
owner = UserFactory(username="owner")
doc = DocumentFactory(owner=owner)
doc.delete() # soft delete
doc.refresh_from_db()
@@ -126,8 +130,8 @@ class TestPermittedDocumentIdsSecurity:
)
def test_superuser_sees_everything_including_no_perm_documents(self):
superuser = User.objects.create_superuser(username="root")
owner = User.objects.create_user(username="owner")
superuser = UserFactory(username="root", superuser=True)
owner = UserFactory(username="owner")
doc = DocumentFactory(owner=owner)
assert_visible_document_ids(
@@ -137,7 +141,7 @@ class TestPermittedDocumentIdsSecurity:
)
def test_anonymous_user_sees_only_unowned_documents(self):
owner = User.objects.create_user(username="owner")
owner = UserFactory(username="owner")
owned = DocumentFactory(owner=owner)
unowned = DocumentFactory(owner=None)
@@ -151,7 +155,7 @@ class TestPermittedDocumentIdsSecurity:
@pytest.mark.django_db
class TestPermittedDocumentIdsIncludeDeleted:
def test_include_deleted_true_reveals_soft_deleted_owned_document(self):
owner = User.objects.create_user(username="owner")
owner = UserFactory(username="owner")
doc = DocumentFactory(owner=owner)
doc.delete()
@@ -162,8 +166,8 @@ class TestPermittedDocumentIdsIncludeDeleted:
)
def test_include_deleted_true_still_respects_permission_boundary(self):
owner = User.objects.create_user(username="owner")
stranger = User.objects.create_user(username="mallory")
owner = UserFactory(username="owner")
stranger = UserFactory(username="mallory")
doc = DocumentFactory(owner=owner)
doc.delete()
@@ -191,14 +195,12 @@ class TestAiChatAllDocumentsPermissionBoundary:
def test_chat_all_documents_excludes_unshared_document(self, mock_stream_chat):
mock_stream_chat.return_value = iter([b"data"])
owner = User.objects.create_user(username="owner")
asker = User.objects.create_user(username="asker")
asker.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
owner = UserFactory(username="owner")
asker = UserFactory(username="asker")
grant_global(asker, "view_document")
shared = DocumentFactory(owner=owner)
not_shared = DocumentFactory(owner=owner)
assign_perm("view_document", asker, shared)
grant_object(asker, shared, "view_document")
client = APIClient()
client.force_authenticate(user=asker)
@@ -219,13 +221,13 @@ class TestAiChatAllDocumentsPermissionBoundary:
@pytest.mark.django_db
class TestDuplicateDocumentsPermissionBoundary:
def test_get_viewable_duplicates_includes_soft_deleted_but_respects_perms(self):
owner = User.objects.create_user(username="owner")
stranger = User.objects.create_user(username="mallory")
owner = UserFactory(username="owner")
stranger = UserFactory(username="mallory")
original = DocumentFactory(owner=owner, checksum="dupe-checksum")
dup_visible = DocumentFactory(owner=owner, checksum="dupe-checksum")
dup_hidden = DocumentFactory(owner=owner, checksum="dupe-checksum")
dup_hidden.delete() # soft delete, should still be found (include_deleted=True)
assign_perm("view_document", stranger, dup_visible)
grant_object(stranger, dup_visible, "view_document")
result_owner = _get_viewable_duplicates(original, owner)
assert {d.pk for d in result_owner} == {dup_visible.pk, dup_hidden.pk}
@@ -237,13 +239,13 @@ class TestDuplicateDocumentsPermissionBoundary:
@pytest.mark.django_db
class TestPermittedDocumentIdsArbitraryPermission:
def test_change_document_permission_is_distinct_from_view(self):
owner = User.objects.create_user(username="owner")
viewer_only = User.objects.create_user(username="viewer")
editor = User.objects.create_user(username="editor")
owner = UserFactory(username="owner")
viewer_only = UserFactory(username="viewer")
editor = UserFactory(username="editor")
doc = DocumentFactory(owner=owner)
assign_perm("view_document", viewer_only, doc)
assign_perm("change_document", editor, doc)
assign_perm("view_document", editor, doc)
grant_object(viewer_only, doc, "view_document")
grant_object(editor, doc, "change_document")
grant_object(editor, doc, "view_document")
assert_visible_document_ids(
permitted_document_ids(editor, perm="change_document"),
@@ -257,10 +259,10 @@ class TestPermittedDocumentIdsArbitraryPermission:
)
def test_qualified_permission_string_is_normalized_to_codename(self):
owner = User.objects.create_user(username="owner")
editor = User.objects.create_user(username="editor")
owner = UserFactory(username="owner")
editor = UserFactory(username="editor")
doc = DocumentFactory(owner=owner)
assign_perm("change_document", editor, doc)
grant_object(editor, doc, "change_document")
assert_visible_document_ids(
permitted_document_ids(editor, perm="documents.change_document"),
@@ -269,11 +271,11 @@ class TestPermittedDocumentIdsArbitraryPermission:
)
def test_delete_permission_with_include_deleted_for_trash_restore(self):
owner = User.objects.create_user(username="owner")
stranger = User.objects.create_user(username="mallory")
view_only = User.objects.create_user(username="viewer")
owner = UserFactory(username="owner")
stranger = UserFactory(username="mallory")
view_only = UserFactory(username="viewer")
doc = DocumentFactory(owner=owner)
assign_perm("view_document", view_only, doc)
grant_object(view_only, doc, "view_document")
doc.delete()
assert_visible_document_ids(
@@ -307,11 +309,9 @@ class TestEmailDocumentPermissionBoundary:
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
requester.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
grant_global(requester, "view_document")
rest_api_client.force_authenticate(user=requester)
hidden = DocumentFactory(owner=owner)
@@ -339,19 +339,17 @@ class TestBulkEditChangePermissionBoundary:
# permitted document must not be partially applied just because it
# was bundled with a forbidden one, proving the endpoint checks
# every document in the batch rather than only the first/last.
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
# grant the global change_document permission so the object-level
# check (not the global has_perm check) is what's under test
requester.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
grant_global(requester, "change_document")
rest_api_client.force_authenticate(user=requester)
changeable = DocumentFactory(owner=owner)
assign_perm("view_document", requester, changeable)
assign_perm("change_document", requester, changeable) # fully permitted
grant_object(requester, changeable, "view_document")
grant_object(requester, changeable, "change_document") # fully permitted
target = DocumentFactory(owner=owner)
assign_perm("view_document", requester, target) # view only, NOT change
grant_object(requester, target, "view_document") # view only, NOT change
response = rest_api_client.post(
"/api/documents/bulk_edit/",
@@ -369,15 +367,14 @@ class TestBulkEditChangePermissionBoundary:
class TestBulkDownloadPermissionChecksRootDocument:
def test_download_requires_global_view_permission(
self,
rest_api_client,
paperless_dirs,
_media_settings,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
rest_api_client: APIClient,
paperless_dirs: PaperlessDirs,
) -> None:
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
root = DocumentFactory(owner=owner)
root.source_path.write_bytes(b"%PDF-1.4 test")
assign_perm("view_document", requester, root)
grant_object(requester, root, "view_document")
rest_api_client.force_authenticate(user=requester)
response = rest_api_client.post(
@@ -390,21 +387,18 @@ class TestBulkDownloadPermissionChecksRootDocument:
def test_permission_checked_on_root_not_on_version(
self,
rest_api_client,
paperless_dirs,
_media_settings,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
requester.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
rest_api_client: APIClient,
paperless_dirs: PaperlessDirs,
) -> None:
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
grant_global(requester, "view_document")
rest_api_client.force_authenticate(user=requester)
root = DocumentFactory(owner=owner)
# a version of root that the requester has NOT been individually granted
version = DocumentFactory(owner=owner, root_document=root, version_index=1)
version.source_path.write_bytes(b"%PDF-1.4 test")
assign_perm("view_document", requester, root) # granted on ROOT only
grant_object(requester, root, "view_document") # granted on ROOT only
response = rest_api_client.post(
"/api/documents/bulk_download/",
@@ -422,11 +416,9 @@ class TestBulkDownloadPermissionChecksRootDocument:
# root-or-version bug; a user with no grant at all (the old
# `stranger` case) can't tell the two apart, since they're denied
# either way.
version_only_grantee = User.objects.create_user(username="version_only_grantee")
version_only_grantee.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
assign_perm("view_document", version_only_grantee, version)
version_only_grantee = UserFactory(username="version_only_grantee")
grant_global(version_only_grantee, "view_document")
grant_object(version_only_grantee, version, "view_document")
rest_api_client.force_authenticate(user=version_only_grantee)
response = rest_api_client.post(
"/api/documents/bulk_download/",
@@ -445,14 +437,12 @@ class TestTrashRestorePermissionBoundary:
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
requester.user_permissions.add(
Permission.objects.get(codename="delete_document"),
)
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
grant_global(requester, "delete_document")
rest_api_client.force_authenticate(user=requester)
doc = DocumentFactory(owner=owner)
assign_perm("view_document", requester, doc) # view only, NOT delete
grant_object(requester, doc, "view_document") # view only, NOT delete
doc.delete()
response = rest_api_client.post(
@@ -466,14 +456,12 @@ class TestTrashRestorePermissionBoundary:
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
requester.user_permissions.add(
Permission.objects.get(codename="delete_document"),
)
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
grant_global(requester, "delete_document")
rest_api_client.force_authenticate(user=requester)
doc = DocumentFactory(owner=owner)
assign_perm("delete_document", requester, doc)
grant_object(requester, doc, "delete_document")
doc.delete()
response = rest_api_client.post(
@@ -484,11 +472,11 @@ class TestTrashRestorePermissionBoundary:
assert response.status_code == HTTPStatus.OK
def test_restore_requires_global_delete_permission(self, rest_api_client):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
rest_api_client.force_authenticate(user=requester)
doc = DocumentFactory(owner=owner)
assign_perm("delete_document", requester, doc)
grant_object(requester, doc, "delete_document")
doc.delete()
response = rest_api_client.post(
@@ -513,14 +501,12 @@ class TestTrashViewExcludesExplicitlyGrantedDocuments:
"""
def test_explicit_grant_does_not_leak_trashed_document(self, rest_api_client):
owner = User.objects.create_user(username="trash_owner")
grantee = User.objects.create_user(username="trash_grantee")
grantee.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
owner = UserFactory(username="trash_owner")
grantee = UserFactory(username="trash_grantee")
grant_global(grantee, "view_document")
doc = DocumentFactory(owner=owner)
doc.delete() # soft delete
assign_perm("view_document", grantee, doc)
grant_object(grantee, doc, "view_document")
rest_api_client.force_authenticate(user=grantee)
response = rest_api_client.get("/api/trash/")
@@ -542,8 +528,8 @@ class TestTrashViewExcludesExplicitlyGrantedDocuments:
)
class TestPermittedObjectIdsGenericModels:
def test_owner_sees_own_object(self, model, factory, perm):
owner = User.objects.create_user(username=f"owner_{model.__name__}")
stranger = User.objects.create_user(username=f"stranger_{model.__name__}")
owner = UserFactory(username=f"owner_{model.__name__}")
stranger = UserFactory(username=f"stranger_{model.__name__}")
owned = factory(owner=owner)
strangers = factory(owner=stranger)
@@ -556,14 +542,14 @@ class TestPermittedObjectIdsGenericModels:
@pytest.mark.parametrize("is_superuser", [False, True])
def test_inactive_user_sees_nothing(self, model, factory, perm, is_superuser):
suffix = f"{model.__name__}_{is_superuser}"
user = User.objects.create_user(
user = UserFactory(
username=f"inactive_{suffix}",
is_active=False,
is_superuser=is_superuser,
)
other = User.objects.create_user(username=f"other_{suffix}")
other = UserFactory(username=f"other_{suffix}")
granted = factory(owner=other)
assign_perm(perm, user, granted)
grant_object(user, granted, perm)
assert_visible_document_ids(
permitted_object_ids(user, model, perm),
@@ -576,7 +562,7 @@ class TestPermittedObjectIdsGenericModels:
)
def test_unowned_object_visible_to_everyone(self, model, factory, perm):
user = User.objects.create_user(username=f"user_{model.__name__}")
user = UserFactory(username=f"user_{model.__name__}")
unowned = factory(owner=None)
assert_visible_document_ids(
@@ -586,12 +572,12 @@ class TestPermittedObjectIdsGenericModels:
)
def test_explicit_permission_grants_visibility(self, model, factory, perm):
owner = User.objects.create_user(username=f"owner2_{model.__name__}")
grantee = User.objects.create_user(username=f"grantee_{model.__name__}")
stranger = User.objects.create_user(username=f"stranger2_{model.__name__}")
owner = UserFactory(username=f"owner2_{model.__name__}")
grantee = UserFactory(username=f"grantee_{model.__name__}")
stranger = UserFactory(username=f"stranger2_{model.__name__}")
shared = factory(owner=owner)
not_shared = factory(owner=owner)
assign_perm(perm, grantee, shared)
grant_object(grantee, shared, perm)
assert_visible_document_ids(
permitted_object_ids(grantee, model, perm),
@@ -610,13 +596,13 @@ class TestPermittedObjectIdsGenericModels:
factory,
perm,
):
owner = User.objects.create_user(username=f"owner3_{model.__name__}")
member = User.objects.create_user(username=f"member_{model.__name__}")
non_member = User.objects.create_user(username=f"nonmember_{model.__name__}")
owner = UserFactory(username=f"owner3_{model.__name__}")
member = UserFactory(username=f"member_{model.__name__}")
non_member = UserFactory(username=f"nonmember_{model.__name__}")
group = Group.objects.create(name=f"group_{model.__name__}")
member.groups.add(group)
shared = factory(owner=owner)
assign_perm(perm, group, shared)
grant_object(group, shared, perm)
assert_visible_document_ids(
permitted_object_ids(member, model, perm),
@@ -630,8 +616,8 @@ class TestPermittedObjectIdsGenericModels:
)
def test_superuser_sees_everything(self, model, factory, perm):
superuser = User.objects.create_superuser(username=f"root_{model.__name__}")
owner = User.objects.create_user(username=f"owner4_{model.__name__}")
superuser = UserFactory(username=f"root_{model.__name__}", superuser=True)
owner = UserFactory(username=f"owner4_{model.__name__}")
obj = factory(owner=owner)
assert_visible_document_ids(
@@ -644,8 +630,8 @@ class TestPermittedObjectIdsGenericModels:
@pytest.mark.django_db
class TestMatchingRespectsObjectPermissions:
def test_match_tags_only_considers_tags_visible_to_user(self):
owner = User.objects.create_user(username="tag_owner")
classifying_user = User.objects.create_user(username="classifier_user")
owner = UserFactory(username="tag_owner")
classifying_user = UserFactory(username="classifier_user")
visible_tag = TagFactory(
owner=owner,
match="invoice",
@@ -656,7 +642,7 @@ class TestMatchingRespectsObjectPermissions:
match="invoice",
matching_algorithm=Tag.MATCH_LITERAL,
)
assign_perm("view_tag", classifying_user, visible_tag)
grant_object(classifying_user, visible_tag, "view_tag")
doc = DocumentFactory(owner=classifying_user, content="an invoice document")
matched = match_tags(doc, classifier=None, user=classifying_user)
@@ -665,8 +651,8 @@ class TestMatchingRespectsObjectPermissions:
assert hidden_tag.pk not in matched_ids
def test_match_correspondents_only_considers_correspondents_visible_to_user(self):
owner = User.objects.create_user(username="correspondent_owner")
classifying_user = User.objects.create_user(username="classifier_user2")
owner = UserFactory(username="correspondent_owner")
classifying_user = UserFactory(username="classifier_user2")
visible_correspondent = CorrespondentFactory(
owner=owner,
match="invoice",
@@ -677,7 +663,7 @@ class TestMatchingRespectsObjectPermissions:
match="invoice",
matching_algorithm=Correspondent.MATCH_LITERAL,
)
assign_perm("view_correspondent", classifying_user, visible_correspondent)
grant_object(classifying_user, visible_correspondent, "view_correspondent")
doc = DocumentFactory(owner=classifying_user, content="an invoice document")
matched = match_correspondents(doc, classifier=None, user=classifying_user)
@@ -686,8 +672,8 @@ class TestMatchingRespectsObjectPermissions:
assert hidden_correspondent.pk not in matched_ids
def test_match_document_types_only_considers_document_types_visible_to_user(self):
owner = User.objects.create_user(username="document_type_owner")
classifying_user = User.objects.create_user(username="classifier_user3")
owner = UserFactory(username="document_type_owner")
classifying_user = UserFactory(username="classifier_user3")
visible_document_type = DocumentTypeFactory(
owner=owner,
match="invoice",
@@ -698,7 +684,7 @@ class TestMatchingRespectsObjectPermissions:
match="invoice",
matching_algorithm=DocumentType.MATCH_LITERAL,
)
assign_perm("view_documenttype", classifying_user, visible_document_type)
grant_object(classifying_user, visible_document_type, "view_documenttype")
doc = DocumentFactory(owner=classifying_user, content="an invoice document")
matched = match_document_types(doc, classifier=None, user=classifying_user)
@@ -707,8 +693,8 @@ class TestMatchingRespectsObjectPermissions:
assert hidden_document_type.pk not in matched_ids
def test_match_storage_paths_only_considers_storage_paths_visible_to_user(self):
owner = User.objects.create_user(username="storage_path_owner")
classifying_user = User.objects.create_user(username="classifier_user4")
owner = UserFactory(username="storage_path_owner")
classifying_user = UserFactory(username="classifier_user4")
visible_storage_path = StoragePathFactory(
owner=owner,
match="invoice",
@@ -719,7 +705,7 @@ class TestMatchingRespectsObjectPermissions:
match="invoice",
matching_algorithm=StoragePath.MATCH_LITERAL,
)
assign_perm("view_storagepath", classifying_user, visible_storage_path)
grant_object(classifying_user, visible_storage_path, "view_storagepath")
doc = DocumentFactory(owner=classifying_user, content="an invoice document")
matched = match_storage_paths(doc, classifier=None, user=classifying_user)
@@ -731,14 +717,12 @@ class TestMatchingRespectsObjectPermissions:
@pytest.mark.django_db
class TestBulkEditObjectsApplyToAllPermissionBoundary:
def test_apply_to_all_tags_excludes_unpermitted_tag(self, rest_api_client):
owner = User.objects.create_user(username="tags_owner")
requester = User.objects.create_user(username="tags_requester")
new_owner = User.objects.create_user(username="tags_new_owner")
owner = UserFactory(username="tags_owner")
requester = UserFactory(username="tags_requester")
new_owner = UserFactory(username="tags_new_owner")
# grant the global change_tag permission so the object-level
# filtering (not the global has_perm check) is what's under test
requester.user_permissions.add(
Permission.objects.get(codename="change_tag"),
)
grant_global(requester, "change_tag")
rest_api_client.force_authenticate(user=requester)
visible = TagFactory(owner=requester)
hidden = TagFactory(owner=owner)
@@ -771,16 +755,14 @@ class TestBulkEditObjectsApplyToAllPermissionBoundary:
request rather than being silently skipped. Editing permissions is
limited to the owner, same as documents.
"""
owner = User.objects.create_user(username="shared_tags_owner")
requester = User.objects.create_user(username="shared_tags_requester")
requester.user_permissions.add(
Permission.objects.get(codename="change_tag"),
)
owner = UserFactory(username="shared_tags_owner")
requester = UserFactory(username="shared_tags_requester")
grant_global(requester, "change_tag")
rest_api_client.force_authenticate(user=requester)
owned = TagFactory(owner=requester)
shared = TagFactory(owner=owner)
assign_perm("view_tag", requester, shared)
assign_perm("change_tag", requester, shared)
grant_object(requester, shared, "view_tag")
grant_object(requester, shared, "change_tag")
response = rest_api_client.post(
"/api/bulk_edit_objects/",
@@ -831,14 +813,12 @@ class TestBulkEditObjectsTagDescendantPartialPermission:
would pass/fail based on FK cascade behavior, not on whether the
descendant-expansion logic itself respected per-object permissions.
"""
owner = User.objects.create_user(username="tag_hierarchy_owner")
requester = User.objects.create_user(username="tag_hierarchy_requester")
new_owner = User.objects.create_user(username="tag_hierarchy_new_owner")
owner = UserFactory(username="tag_hierarchy_owner")
requester = UserFactory(username="tag_hierarchy_requester")
new_owner = UserFactory(username="tag_hierarchy_new_owner")
# global change_tag permission so the has_perm() gate passes and the
# object-level permitted_object_ids filtering is what's under test
requester.user_permissions.add(
Permission.objects.get(codename="change_tag"),
)
grant_global(requester, "change_tag")
rest_api_client.force_authenticate(user=requester)
parent = TagFactory(owner=requester, name="parent-tag")
@@ -890,7 +870,7 @@ class TestRestrictQuerysetToVisible:
- The queryset is returned unfiltered, rather than
permitted_object_ids(None, ...)'s narrower "unowned rows only"
"""
owner = User.objects.create_user(username="vis_none_owner")
owner = UserFactory(username="vis_none_owner")
tag = TagFactory(owner=owner)
visible = restrict_queryset_to_visible(Tag.objects.all(), None, "view_tag")
@@ -907,8 +887,8 @@ class TestRestrictQuerysetToVisible:
- The queryset is returned unfiltered, skipping the permission
lookup entirely
"""
superuser = User.objects.create_superuser(username="vis_active_super")
owner = User.objects.create_user(username="vis_active_super_owner")
superuser = UserFactory(username="vis_active_super", superuser=True)
owner = UserFactory(username="vis_active_super_owner")
tag = TagFactory(owner=owner)
visible = restrict_queryset_to_visible(
@@ -930,7 +910,7 @@ class TestRestrictQuerysetToVisible:
deactivation has to win over the superuser shortcut, matching
permitted_object_ids's own ordering
"""
user = User.objects.create_user(
user = UserFactory(
username="vis_inactive_super",
is_active=False,
is_superuser=True,
@@ -951,8 +931,8 @@ class TestRestrictQuerysetToVisible:
THEN:
- Only the rows permitted_object_ids() reports are visible
"""
user = User.objects.create_user(username="vis_regular")
other = User.objects.create_user(username="vis_regular_other")
user = UserFactory(username="vis_regular")
other = UserFactory(username="vis_regular_other")
own = TagFactory(owner=user)
hidden = TagFactory(owner=other)
@@ -1,11 +1,11 @@
import pytest
from django.contrib.auth.models import User
from guardian.shortcuts import assign_perm
from rest_framework.test import APIRequestFactory
from documents.filters import PermittedObjectsFilter
from documents.models import Tag
from documents.tests.factories import TagFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_object
class _DummyView:
@@ -15,8 +15,8 @@ class _DummyView:
@pytest.mark.django_db
class TestPermittedObjectsFilter:
def test_superuser_bypasses_filtering_entirely(self):
superuser = User.objects.create_superuser(username="root")
owner = User.objects.create_user(username="owner")
superuser = UserFactory(username="root", superuser=True)
owner = UserFactory(username="owner")
TagFactory(owner=owner)
request = APIRequestFactory().get("/")
request.user = superuser
@@ -29,13 +29,13 @@ class TestPermittedObjectsFilter:
assert result.count() == Tag.objects.count()
def test_non_superuser_sees_only_owned_unowned_and_granted(self):
owner = User.objects.create_user(username="owner")
grantee = User.objects.create_user(username="grantee")
owner = UserFactory(username="owner")
grantee = UserFactory(username="grantee")
owned = TagFactory(owner=grantee)
unowned = TagFactory(owner=None)
granted = TagFactory(owner=owner)
hidden = TagFactory(owner=owner)
assign_perm("view_tag", grantee, granted)
grant_object(grantee, granted, "view_tag")
request = APIRequestFactory().get("/")
request.user = grantee
@@ -49,11 +49,11 @@ class TestPermittedObjectsFilter:
assert hidden.pk not in visible_ids
def test_include_granted_false_excludes_explicitly_shared_objects(self):
owner = User.objects.create_user(username="owner2")
grantee = User.objects.create_user(username="grantee2")
owner = UserFactory(username="owner2")
grantee = UserFactory(username="grantee2")
owned = TagFactory(owner=grantee)
granted = TagFactory(owner=owner)
assign_perm("view_tag", grantee, granted)
grant_object(grantee, granted, "view_tag")
request = APIRequestFactory().get("/")
request.user = grantee
@@ -74,15 +74,15 @@ class TestPermittedObjectsFilter:
[("inactive", False), ("inactive_super", True)],
)
def test_inactive_user_sees_nothing(self, username: str, *, is_superuser: bool):
user = User.objects.create_user(
user = UserFactory(
username=username,
is_active=False,
is_superuser=is_superuser,
)
TagFactory(owner=None)
TagFactory(owner=user)
granted = TagFactory(owner=User.objects.create_user(username=f"o_{username}"))
assign_perm("view_tag", user, granted)
granted = TagFactory(owner=UserFactory(username=f"o_{username}"))
grant_object(user, granted, "view_tag")
request = APIRequestFactory().get("/")
request.user = user
@@ -94,7 +94,7 @@ class TestPermittedObjectsFilter:
assert result.count() == 0
def test_inactive_user_sees_nothing_with_include_granted_false(self):
user = User.objects.create_user(username="inactive_owner", is_active=False)
user = UserFactory(username="inactive_owner", is_active=False)
TagFactory(owner=user)
TagFactory(owner=None)
request = APIRequestFactory().get("/")
+8 -9
View File
@@ -19,7 +19,7 @@ if TYPE_CHECKING:
from collections.abc import Iterable
from documents.models import Document
from documents.tests.conftest import PaperlessDirs
from paperless_testing.dirs import PaperlessDirs
class TestSanityCheckMessages:
@@ -46,14 +46,14 @@ class TestSanityCheckMessages:
class TestCheckSanityNoDocuments:
"""Sanity checks against an empty archive."""
@pytest.mark.usefixtures("_media_settings")
@pytest.mark.usefixtures("paperless_dirs")
def test_no_documents(self) -> None:
messages = check_sanity()
assert not messages.has_error
assert not messages.has_warning
assert messages.total_issue_count == 0
@pytest.mark.usefixtures("_media_settings")
@pytest.mark.usefixtures("paperless_dirs")
def test_no_issues_logs_clean(self, caplog: pytest.LogCaptureFixture) -> None:
messages = check_sanity()
with caplog.at_level(logging.INFO, logger="paperless.sanity_checker"):
@@ -214,18 +214,17 @@ class TestCheckSanityOrphans:
sample_doc: Document,
paperless_dirs: PaperlessDirs,
) -> None:
(paperless_dirs.originals / "orphan.pdf").touch()
(paperless_dirs.originals_dir / "orphan.pdf").touch()
messages = check_sanity()
assert messages.has_warning
assert any("Orphaned file" in m["message"] for m in messages[None])
@pytest.mark.usefixtures("_media_settings")
def test_ignorable_files_not_flagged(
self,
paperless_dirs: PaperlessDirs,
) -> None:
(paperless_dirs.media / ".DS_Store").touch()
(paperless_dirs.media / "desktop.ini").touch()
(paperless_dirs.media_dir / ".DS_Store").touch()
(paperless_dirs.media_dir / "desktop.ini").touch()
messages = check_sanity()
assert not messages.has_warning
@@ -269,13 +268,13 @@ class TestCheckSanityLogMessages:
paperless_dirs: PaperlessDirs,
caplog: pytest.LogCaptureFixture,
) -> None:
(paperless_dirs.originals / "orphan.pdf").touch()
(paperless_dirs.originals_dir / "orphan.pdf").touch()
messages = check_sanity()
with caplog.at_level(logging.WARNING, logger="paperless.sanity_checker"):
messages.log_messages()
assert "Orphaned file" in caplog.text
@pytest.mark.usefixtures("_media_settings")
@pytest.mark.usefixtures("paperless_dirs")
def test_logs_unknown_doc_pk(self, caplog: pytest.LogCaptureFixture) -> None:
"""A doc PK not in the DB logs 'Unknown' as the title."""
messages = check_sanity()
+11 -14
View File
@@ -6,10 +6,8 @@ from pathlib import Path
from unittest import mock
from django.conf import settings
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.utils import timezone
from guardian.shortcuts import assign_perm
from rest_framework import serializers
from rest_framework import status
from rest_framework.test import APITestCase
@@ -20,8 +18,11 @@ from documents.models import ShareLinkBundle
from documents.serialisers import ShareLinkBundleSerializer
from documents.tasks import build_share_link_bundle
from documents.tasks import cleanup_expired_share_link_bundles
from documents.tests.factories import DocumentFactory
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
@@ -29,7 +30,7 @@ class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="bundle_admin")
self.user = UserFactory(username="bundle_admin", superuser=True)
self.client.force_authenticate(self.user)
self.document = DocumentFactory.create()
@@ -55,13 +56,11 @@ class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
self,
delay_mock,
) -> None:
owner = User.objects.create_user(username="document_owner")
requester = User.objects.create_user(username="bundle_creator")
requester.user_permissions.add(
Permission.objects.get(codename="add_sharelinkbundle"),
)
owner = UserFactory(username="document_owner")
requester = UserFactory(username="bundle_creator")
grant_global(requester, "add_sharelinkbundle")
document = DocumentFactory.create(owner=owner)
assign_perm("view_document", requester, document)
grant_object(requester, document, "view_document")
self.client.force_authenticate(requester)
payload = {
"document_ids": [document.pk],
@@ -72,9 +71,7 @@ class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
response = self.client.post(self.ENDPOINT, payload, format="json")
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
requester.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(requester, "view_document")
requester = User.objects.get(pk=requester.pk)
self.client.force_authenticate(requester)
response = self.client.post(self.ENDPOINT, payload, format="json")
+7 -9
View File
@@ -1,7 +1,5 @@
from unittest import mock
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from rest_framework import status
from rest_framework.test import APITestCase
@@ -13,16 +11,16 @@ from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from documents.serialisers import TagSerializer
from documents.signals.handlers import run_workflows
from documents.tests.utils import DirectoriesMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
class TestTagHierarchyPermissions(APITestCase):
def test_children_only_include_visible_tags(self) -> None:
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
requester.user_permissions.add(
Permission.objects.get(codename="view_tag"),
)
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
grant_global(requester, "view_tag")
parent = Tag.objects.create(name="Visible parent", owner=requester)
hidden_child = Tag.objects.create(
name="Hidden child",
@@ -49,7 +47,7 @@ class TestTagHierarchyPermissions(APITestCase):
class TestTagHierarchy(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(username="admin")
self.user = UserFactory(username="admin", superuser=True)
self.client.force_authenticate(user=self.user)
self.parent = Tag.objects.create(name="Parent")
+4 -3
View File
@@ -17,7 +17,8 @@ from documents.signals.handlers import task_failure_handler
from documents.signals.handlers import task_postrun_handler
from documents.signals.handlers import task_prerun_handler
from documents.signals.handlers import task_revoked_handler
from documents.tests.factories import PaperlessTaskFactory
from paperless_testing.factories import PaperlessTaskFactory
from paperless_testing.factories import UserFactory
@pytest.fixture
@@ -34,8 +35,8 @@ def consume_input_doc():
@pytest.fixture
def consume_overrides(django_user_model):
user = django_user_model.objects.create_user(username="testuser")
def consume_overrides():
user = UserFactory(username="testuser")
overrides = mock.MagicMock(spec=DocumentMetadataOverrides)
overrides.owner_id = user.id
return overrides
+38 -3
View File
@@ -17,9 +17,10 @@ from documents.models import Tag
from documents.models import WorkflowAction
from documents.sanity_checker import SanityCheckFailedException
from documents.sanity_checker import SanityCheckMessages
from documents.tests.test_classifier import dummy_preprocess
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from documents.tests.helpers import dummy_preprocess
from paperless_ai.exceptions import LLMBlockedError
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
@pytest.mark.django_db
@@ -555,3 +556,37 @@ class TestApplyAISuggestionsTask(DirectoriesMixin, TestCase):
apply_suggestions.assert_not_called()
self.assertIn("no longer exists", "".join(cm.output))
@override_settings(AI_ENABLED=True)
def test_blocked_request_fails_without_retry(self) -> None:
"""
GIVEN:
- AI enabled and a document with content
- The AI classification call blocked by the outbound request policy
WHEN:
- The task runs through Celery
THEN:
- The workflow code does not swallow the block
- The task fails with LLMBlockedError and is never retried
"""
with (
mock.patch(
"documents.workflows.ai.get_ai_document_classification",
side_effect=LLMBlockedError(
"AI backend request was blocked by the outbound request "
"policy: detail",
),
),
mock.patch.object(
tasks.apply_ai_suggestions,
"retry",
wraps=tasks.apply_ai_suggestions.retry,
) as retry,
):
result = tasks.apply_ai_suggestions.apply(
args=(self.action.pk, self.doc.pk),
)
self.assertTrue(result.failed())
self.assertIsInstance(result.result, LLMBlockedError)
retry.assert_not_called()
@@ -14,8 +14,8 @@ from documents.conditionals import preview_etag
from documents.conditionals import thumbnail_etag
from documents.conditionals import thumbnail_last_modified
from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from documents.versioning import resolve_effective_document_by_pk
from paperless_testing.dirs import DirectoriesMixin
if TYPE_CHECKING:
from rest_framework.request import Request
+71 -40
View File
@@ -15,7 +15,6 @@ from django.test import TestCase
from django.test import override_settings
from django.test.utils import CaptureQueriesContext
from django.utils import timezone
from guardian.shortcuts import assign_perm
from rest_framework import status
from documents.caching import get_llm_suggestion_cache
@@ -29,11 +28,15 @@ from documents.models import StoragePath
from documents.models import Tag
from documents.models import UiSettings
from documents.signals.handlers import update_llm_suggestions_cache
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import read_streaming_response
from paperless.models import ApplicationConfiguration
from paperless_ai.exceptions import LLMBlockedError
from paperless_ai.exceptions import LLMProviderError
from paperless_ai.exceptions import LLMTimeoutError
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
from paperless_testing.http import read_streaming_response
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestViews(DirectoriesMixin, TestCase):
@@ -43,7 +46,7 @@ class TestViews(DirectoriesMixin, TestCase):
ApplicationConfiguration.objects.get_or_create()
def setUp(self) -> None:
self.user = User.objects.create_user("testuser")
self.user = UserFactory(username="testuser")
super().setUp()
def test_login_redirect(self) -> None:
@@ -141,9 +144,7 @@ class TestViews(DirectoriesMixin, TestCase):
codename__contains="sharelink",
)
self.user.user_permissions.add(*sharelink_permissions)
self.user.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(self.user, "view_document")
self.user.save()
self.client.force_login(self.user)
@@ -205,9 +206,7 @@ class TestViews(DirectoriesMixin, TestCase):
codename__contains="sharelink",
)
self.user.user_permissions.add(*sharelink_permissions)
self.user.user_permissions.add(
Permission.objects.get(codename="view_document"),
)
grant_global(self.user, "view_document")
self.client.force_login(self.user)
create_response = self.client.post(
@@ -240,16 +239,16 @@ class TestViews(DirectoriesMixin, TestCase):
group2 = Group.objects.create(name="group2")
group3 = Group.objects.create(name="group3")
t1 = Tag.objects.create(name="invoice", pk=1)
assign_perm("view_tag", self.user, t1)
assign_perm("view_tag", user2, t1)
assign_perm("view_tag", user3, t1)
assign_perm("view_tag", group1, t1)
assign_perm("view_tag", group2, t1)
assign_perm("view_tag", group3, t1)
assign_perm("change_tag", self.user, t1)
assign_perm("change_tag", user2, t1)
assign_perm("change_tag", group1, t1)
assign_perm("change_tag", group2, t1)
grant_object(self.user, t1, "view_tag")
grant_object(user2, t1, "view_tag")
grant_object(user3, t1, "view_tag")
grant_object(group1, t1, "view_tag")
grant_object(group2, t1, "view_tag")
grant_object(group3, t1, "view_tag")
grant_object(self.user, t1, "change_tag")
grant_object(user2, t1, "change_tag")
grant_object(group1, t1, "change_tag")
grant_object(group2, t1, "change_tag")
Tag.objects.create(name="bank statement", pk=2)
d1 = Document.objects.create(
@@ -338,7 +337,7 @@ class TestViews(DirectoriesMixin, TestCase):
class TestAISuggestions(DirectoriesMixin, TestCase):
def setUp(self) -> None:
self.user = User.objects.create_superuser(username="testuser")
self.user = UserFactory(username="testuser", superuser=True)
self.document = Document.objects.create(
title="Test Document",
filename="test.pdf",
@@ -425,14 +424,10 @@ class TestAISuggestions(DirectoriesMixin, TestCase):
requester; the invisible tag id does not leak into either the
matched or suggested tags
"""
tag_owner = User.objects.create_user(username="cache_tag_owner")
tag_owner = UserFactory(username="cache_tag_owner")
invisible_tag = Tag.objects.create(name="cache_restricted", owner=tag_owner)
requester = User.objects.create_user(username="cache_requester")
requester.user_permissions.add(
*Permission.objects.filter(
codename__in=["view_document", "change_document", "view_tag"],
),
)
requester = UserFactory(username="cache_requester")
grant_global(requester, "view_document", "change_document", "view_tag")
mock_get_cache.return_value = MagicMock(
suggestions={
"title": "Untitled",
@@ -637,7 +632,7 @@ class TestAISuggestions(DirectoriesMixin, TestCase):
- The classification runs with the second user's visibility
context without evicting the first user's result
"""
second_user = User.objects.create_superuser(username="second_user")
second_user = UserFactory(username="second_user", superuser=True)
empty_choices = {
"tags": {"existing_ids": [], "new_names": []},
"correspondents": {"existing_ids": [], "new_names": []},
@@ -776,6 +771,48 @@ class TestAISuggestions(DirectoriesMixin, TestCase):
get_llm_suggestion_cache(self.document.pk, backend="openai-like"),
)
@patch("documents.views.get_ai_document_classification")
@override_settings(
AI_ENABLED=True,
LLM_BACKEND="openai-like",
)
def test_ai_suggestions_with_blocked_llm_request(
self,
mock_get_ai_classification,
) -> None:
"""
GIVEN:
- An AI backend request blocked by the outbound request policy
WHEN:
- AI suggestions are requested
THEN:
- 502 is returned with a generic message and nothing is cached
"""
mock_get_ai_classification.side_effect = LLMBlockedError(
"AI backend request was blocked by the outbound request policy: detail",
)
self.client.force_login(user=self.user)
response = self.client.get(
f"/api/documents/{self.document.pk}/ai_suggestions/",
)
self.assertEqual(response.status_code, status.HTTP_502_BAD_GATEWAY)
self.assertEqual(
response.json(),
{
"ai": [
(
"AI backend request was blocked by the outbound request "
"policy. Check logs for details."
),
],
},
)
self.assertIsNone(
get_llm_suggestion_cache(self.document.pk, backend="openai-like"),
)
@patch("documents.views.get_ai_document_classification")
@override_settings(
AI_ENABLED=True,
@@ -875,14 +912,10 @@ class TestAISuggestions(DirectoriesMixin, TestCase):
permission filtering survives the full request path
- it does not appear in either the matched or suggested tags
"""
tag_owner = User.objects.create_user(username="tagowner")
tag_owner = UserFactory(username="tagowner")
invisible_tag = Tag.objects.create(name="restricted", owner=tag_owner)
requester = User.objects.create_user(username="requester")
requester.user_permissions.add(
*Permission.objects.filter(
codename__in=["view_document", "change_document", "view_tag"],
),
)
requester = UserFactory(username="requester")
grant_global(requester, "view_document", "change_document", "view_tag")
mock_get_ai_classification.return_value = {
"title": "Untitled",
@@ -956,7 +989,7 @@ class TestAIChatStreamingView(DirectoriesMixin, TestCase):
ENDPOINT = "/api/documents/chat/"
def setUp(self) -> None:
self.user = User.objects.create_user(username="testuser", password="pass")
self.user = UserFactory(username="testuser", password="pass")
self.client.force_login(user=self.user)
self.document = Document.objects.create(
title="Test Document",
@@ -966,9 +999,7 @@ class TestAIChatStreamingView(DirectoriesMixin, TestCase):
super().setUp()
def grant_view_document_permission(self) -> None:
self.user.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
grant_global(self.user, "view_document")
@override_settings(AI_ENABLED=False)
def test_post_ai_disabled(self) -> None:
File diff suppressed because it is too large Load Diff
+2 -346
View File
@@ -1,242 +1,16 @@
import shutil
import tempfile
import time
import warnings
from collections import namedtuple
from collections.abc import Callable
from collections.abc import Generator
from collections.abc import Iterator
from contextlib import contextmanager
from os import PathLike
from pathlib import Path
from typing import Any
from unittest import mock
import httpx
import pytest
from django.apps import apps
from django.db import connection
from django.db.migrations.executor import MigrationExecutor
from django.http import StreamingHttpResponse
from django.test import TransactionTestCase
from django.test import override_settings
from documents.consumer import AsnCheckPlugin
from documents.consumer import ConsumerPlugin
from documents.consumer import ConsumerPreflightPlugin
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource
from documents.parsers import ParseError
from documents.plugins.helpers import ProgressStatusOptions
def setup_directories():
dirs = namedtuple("Dirs", ())
dirs.data_dir = Path(tempfile.mkdtemp()).resolve()
dirs.scratch_dir = Path(tempfile.mkdtemp()).resolve()
dirs.media_dir = Path(tempfile.mkdtemp()).resolve()
dirs.consumption_dir = Path(tempfile.mkdtemp()).resolve()
dirs.static_dir = Path(tempfile.mkdtemp()).resolve()
dirs.index_dir = dirs.data_dir / "index"
dirs.originals_dir = dirs.media_dir / "documents" / "originals"
dirs.thumbnail_dir = dirs.media_dir / "documents" / "thumbnails"
dirs.archive_dir = dirs.media_dir / "documents" / "archive"
dirs.logging_dir = dirs.data_dir / "log"
dirs.index_dir.mkdir(parents=True, exist_ok=True)
dirs.originals_dir.mkdir(parents=True, exist_ok=True)
dirs.thumbnail_dir.mkdir(parents=True, exist_ok=True)
dirs.archive_dir.mkdir(parents=True, exist_ok=True)
dirs.logging_dir.mkdir(parents=True, exist_ok=True)
dirs.settings_override = override_settings(
DATA_DIR=dirs.data_dir,
SCRATCH_DIR=dirs.scratch_dir,
MEDIA_ROOT=dirs.media_dir,
ORIGINALS_DIR=dirs.originals_dir,
THUMBNAIL_DIR=dirs.thumbnail_dir,
ARCHIVE_DIR=dirs.archive_dir,
CONSUMPTION_DIR=dirs.consumption_dir,
LOGGING_DIR=dirs.logging_dir,
INDEX_DIR=dirs.index_dir,
STATIC_ROOT=dirs.static_dir,
MODEL_FILE=dirs.data_dir / "classification_model.pickle",
MEDIA_LOCK=dirs.media_dir / "media.lock",
)
dirs.settings_override.enable()
return dirs
def remove_dirs(dirs) -> None:
shutil.rmtree(dirs.media_dir, ignore_errors=True)
shutil.rmtree(dirs.data_dir, ignore_errors=True)
shutil.rmtree(dirs.scratch_dir, ignore_errors=True)
shutil.rmtree(dirs.consumption_dir, ignore_errors=True)
shutil.rmtree(dirs.static_dir, ignore_errors=True)
dirs.settings_override.disable()
@contextmanager
def paperless_environment():
dirs = None
try:
dirs = setup_directories()
yield dirs
finally:
if dirs:
remove_dirs(dirs)
def util_call_with_backoff(
method_or_callable: Callable,
args: list | tuple,
*,
skip_on_50x_err=True,
) -> tuple[bool, Any]:
"""
For whatever reason, the images started during the test pipeline like to
segfault sometimes, crash and otherwise fail randomly, when run with the
exact files that usually pass.
So, this function will retry the given method/function up to 3 times, with larger backoff
periods between each attempt, in hopes the issue resolves itself during
one attempt to parse.
This will wait the following:
- Attempt 1 - 20s following failure
- Attempt 2 - 40s following failure
- Attempt 3 - 80s following failure
"""
result = None
succeeded = False
retry_time = 20.0
retry_count = 0
status_codes = []
max_retry_count = 3
while retry_count < max_retry_count and not succeeded:
try:
result = method_or_callable(*args)
succeeded = True
except ParseError as e: # pragma: no cover
cause_exec = e.__cause__
if cause_exec is not None and isinstance(cause_exec, httpx.HTTPStatusError):
status_codes.append(cause_exec.response.status_code)
warnings.warn(
f"HTTP Exception for {cause_exec.request.url} - {cause_exec}",
)
else:
warnings.warn(f"Unexpected error: {e}")
except Exception as e: # pragma: no cover
warnings.warn(f"Unexpected error: {e}")
retry_count = retry_count + 1
time.sleep(retry_time)
retry_time = retry_time * 2.0
if (
not succeeded
and status_codes
and skip_on_50x_err
and all(httpx.codes.is_server_error(code) for code in status_codes)
):
pytest.skip("Repeated HTTP 50x for service") # pragma: no cover
return succeeded, result
def read_streaming_response(response: StreamingHttpResponse) -> bytes:
"""Consume a StreamingHttpResponse/FileResponse and close it."""
content = b"".join(response.streaming_content)
response.close()
return content
class DirectoriesMixin:
"""
Creates and overrides settings for all folders and paths, then ensures
they are cleaned up on exit
"""
def setUp(self) -> None:
from documents.search import reset_backend
reset_backend()
self.dirs = setup_directories()
super().setUp()
def tearDown(self) -> None:
from documents.search import reset_backend
super().tearDown()
reset_backend()
remove_dirs(self.dirs)
class FileSystemAssertsMixin:
"""
Utilities for checks various state information of the file system
"""
def assertIsFile(self, path: PathLike[str] | str) -> None:
self.assertTrue(Path(path).resolve().is_file(), f"File does not exist: {path}")
def assertIsNotFile(self, path: PathLike[str] | str) -> None:
self.assertFalse(Path(path).resolve().is_file(), f"File does exist: {path}")
def assertIsDir(self, path: PathLike[str] | str) -> None:
self.assertTrue(Path(path).resolve().is_dir(), f"Dir does not exist: {path}")
def assertIsNotDir(self, path: PathLike[str] | str) -> None:
self.assertFalse(Path(path).resolve().is_dir(), f"Dir does exist: {path}")
def assertFilesEqual(
self,
path1: PathLike[str] | str,
path2: PathLike[str] | str,
) -> None:
path1 = Path(path1)
path2 = Path(path2)
import hashlib
hash1 = hashlib.sha256(path1.read_bytes()).hexdigest()
hash2 = hashlib.sha256(path2.read_bytes()).hexdigest()
self.assertEqual(hash1, hash2, "File SHA256 mismatch")
def assertFileCountInDir(self, path: PathLike[str] | str, count: int) -> None:
path = Path(path).resolve()
self.assertTrue(path.is_dir(), f"Path {path} is not a directory")
files = [x for x in path.iterdir() if x.is_file()]
self.assertEqual(
len(files),
count,
f"Path {path} contains {len(files)} files instead of {count} files",
)
class ConsumerProgressMixin:
"""
Mocks the Consumer _send_progress, preventing attempts to connect to Redis
and allowing access to its calls for verification
"""
def setUp(self) -> None:
self.send_progress_patcher = mock.patch(
"documents.consumer.Consumer._send_progress",
)
self.send_progress_mock = self.send_progress_patcher.start()
super().setUp()
def tearDown(self) -> None:
super().tearDown()
self.send_progress_patcher.stop()
from paperless_testing.fakes.progress import FakeProgressManager
class ConsumeTaskMixin:
@@ -274,64 +48,6 @@ class ConsumeTaskMixin:
yield (task_kwargs["input_doc"], task_kwargs["overrides"])
class TestMigrations(TransactionTestCase):
@property
def app(self):
return apps.get_containing_app_config(type(self).__module__).name
migrate_from = None
dependencies = None
migrate_to = None
auto_migrate = True
def setUp(self) -> None:
super().setUp()
assert self.migrate_from and self.migrate_to, (
f"TestCase '{type(self).__name__}' must define migrate_from and migrate_to properties"
)
self.migrate_from = [(self.app, self.migrate_from)]
if self.dependencies is not None:
self.migrate_from.extend(self.dependencies)
self.migrate_to = [(self.app, self.migrate_to)]
executor = MigrationExecutor(connection)
old_apps = executor.loader.project_state(self.migrate_from).apps
# Reverse to the original migration
executor.migrate(self.migrate_from)
self.setUpBeforeMigration(old_apps)
self.apps = old_apps
if self.auto_migrate:
self.performMigration()
def performMigration(self) -> None:
# Run the migration to test
executor = MigrationExecutor(connection)
executor.loader.build_graph() # reload.
executor.migrate(self.migrate_to)
self.apps = executor.loader.project_state(self.migrate_to).apps
def setUpBeforeMigration(self, apps) -> None:
pass
def tearDown(self) -> None:
"""
Ensure the database schema is restored to the latest migration after
each migration test, so subsequent tests run against HEAD.
"""
try:
executor = MigrationExecutor(connection)
executor.loader.build_graph()
targets = executor.loader.graph.leaf_nodes()
executor.migrate(targets)
finally:
super().tearDown()
class SampleDirMixin:
SAMPLE_DIR = Path(__file__).parent / "samples"
@@ -348,7 +64,7 @@ class GetConsumerMixin:
mailrule_id: int | None = None,
) -> Generator[ConsumerPlugin, None, None]:
# Store this for verification
self.status = DummyProgressManager(filepath.name, None)
self.status = FakeProgressManager(filepath.name, None)
doc = ConsumableDocument(
source,
original_file=filepath,
@@ -384,63 +100,3 @@ class GetConsumerMixin:
yield reader
finally:
reader.cleanup()
class DummyProgressManager:
"""
A dummy handler for progress management that doesn't actually try to
connect to Redis. Payloads are stored for test assertions if needed.
Use it with
mock.patch("documents.tasks.ProgressManager", DummyProgressManager)
"""
def __init__(self, filename: str, task_id: str | None = None) -> None:
self.filename = filename
self.task_id = task_id
self.payloads = []
def __enter__(self):
self.open()
return self
def __exit__(self, exc_type, exc_val, exc_tb) -> None:
self.close()
def open(self) -> None:
pass
def close(self) -> None:
pass
def send_progress(
self,
status: ProgressStatusOptions,
message: str,
current_progress: int,
max_progress: int,
*,
document_id: int | None = None,
owner_id: int | None = None,
users_can_view: list[int] | None = None,
groups_can_view: list[int] | None = None,
) -> None:
# Ensure the layer is open
self.open()
payload = {
"type": "status_update",
"data": {
"filename": self.filename,
"task_id": self.task_id,
"current_progress": current_progress,
"max_progress": max_progress,
"status": status,
"message": message,
"document_id": document_id,
"owner_id": owner_id,
"users_can_view": users_can_view or [],
"groups_can_view": groups_can_view or [],
},
}
self.payloads.append(payload)
+18
View File
@@ -256,6 +256,7 @@ from paperless.views import StandardPagination
from paperless_ai.ai_classifier import get_ai_document_classification
from paperless_ai.ai_classifier import get_llm_output_language
from paperless_ai.chat import stream_chat_with_documents
from paperless_ai.exceptions import LLMBlockedError
from paperless_ai.exceptions import LLMProviderError
from paperless_ai.exceptions import LLMTimeoutError
from paperless_ai.matching import extract_unmatched_names
@@ -1697,6 +1698,23 @@ class DocumentViewSet(
},
status=status.HTTP_502_BAD_GATEWAY,
)
except LLMBlockedError as exc:
logger.warning(
"AI backend request for document %s was blocked: %s",
doc.pk,
exc,
)
return Response(
{
"ai": [
_(
"AI backend request was blocked by the outbound "
"request policy. Check logs for details.",
),
],
},
status=status.HTTP_502_BAD_GATEWAY,
)
set_llm_suggestions_cache(
doc.pk,
llm_suggestions,
+6 -4
View File
@@ -4,7 +4,8 @@ import httpx
from celery import shared_task
from django.conf import settings
from paperless.network import PinnedHostHTTPTransport
from paperless.network import GuardedHTTPTransport
from paperless.network import OutboundRequestBlockedError
from paperless.network import validate_outbound_http_url
logger = logging.getLogger("paperless.workflows.webhooks")
@@ -14,7 +15,7 @@ logger = logging.getLogger("paperless.workflows.webhooks")
retry_backoff=True,
autoretry_for=(httpx.HTTPStatusError,),
max_retries=3,
throws=(httpx.HTTPError,),
throws=(httpx.HTTPError, OutboundRequestBlockedError),
)
def send_webhook(
url: str,
@@ -29,14 +30,15 @@ def send_webhook(
url,
allowed_schemes=settings.WEBHOOKS_ALLOWED_SCHEMES,
allowed_ports=settings.WEBHOOKS_ALLOWED_PORTS,
# Internal-address checks happen in transport to preserve ConnectError behavior.
# Scheme and port only; the transport enforces the internal-address
# policy at connect time, on the address actually dialled.
allow_internal=True,
)
except ValueError as e:
logger.warning("Webhook blocked: %s", e)
raise
transport = PinnedHostHTTPTransport(
transport = GuardedHTTPTransport(
allow_internal=settings.WEBHOOKS_ALLOW_INTERNAL_REQUESTS,
)
+10 -10
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-09-18 01:29+0000\n"
"POT-Creation-Date: 2026-09-21 19:00+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -1632,7 +1632,7 @@ msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:514 documents/serialisers.py:871
#: documents/serialisers.py:2883 documents/views.py:343 documents/views.py:2726
#: documents/serialisers.py:2885 documents/views.py:343 documents/views.py:2726
#: paperless_mail/serialisers.py:156
msgid "Insufficient permissions."
msgstr ""
@@ -1641,39 +1641,39 @@ msgstr ""
msgid "Invalid color."
msgstr ""
#: documents/serialisers.py:2350
#: documents/serialisers.py:2352
#, python-format
msgid "File type %(type)s not supported"
msgstr ""
#: documents/serialisers.py:2394
#: documents/serialisers.py:2396
#, python-format
msgid "Custom field id must be an integer: %(id)s"
msgstr ""
#: documents/serialisers.py:2401
#: documents/serialisers.py:2403
#, python-format
msgid "Custom field with id %(id)s does not exist"
msgstr ""
#: documents/serialisers.py:2418 documents/serialisers.py:2428
#: documents/serialisers.py:2420 documents/serialisers.py:2430
msgid ""
"Custom fields must be a list of integers or an object mapping ids to values."
msgstr ""
#: documents/serialisers.py:2423
#: documents/serialisers.py:2425
msgid "Some custom fields don't exist or were specified twice."
msgstr ""
#: documents/serialisers.py:2570
#: documents/serialisers.py:2572
msgid "Invalid variable detected."
msgstr ""
#: documents/serialisers.py:2939
#: documents/serialisers.py:2941
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2969 documents/views.py:4763
#: documents/serialisers.py:2971 documents/views.py:4763
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
+519 -158
View File
@@ -1,61 +1,533 @@
import functools
import ipaddress
import logging
import math
import re
import socket
import time
from collections.abc import Callable
from collections.abc import Collection
from collections.abc import Iterable
from enum import StrEnum
from typing import Any
from typing import Final
from typing import Self
from typing import TypeAlias
from urllib.parse import ParseResult
from urllib.parse import urlparse
import anyio
import httpcore
import httpx
# Ranges ipaddress does not report as private, but which routinely front
# internal infrastructure.
# Not exported by httpcore; the guard asserts it is still the async default.
from httpcore._backends.auto import AutoBackend
logger = logging.getLogger("paperless.network")
# requires-python is >=3.11, so no PEP 695 `type` statement.
IPAddress: TypeAlias = ipaddress.IPv4Address | ipaddress.IPv6Address
# Ranges that ipaddress reports as global but which still reach internal hosts.
_NON_PUBLIC_NETWORKS = (
# RFC 6598 shared address space: ISP CGNAT, and the default pod/service
# CIDR on several managed Kubernetes offerings.
ipaddress.ip_network("100.64.0.0/10"),
# RFC 6052 NAT64 well-known prefix: 64:ff9b::7f00:1 is 127.0.0.1 wherever
# a NAT64 gateway exists.
# a NAT64 gateway exists, yet ipaddress classifies the prefix as global.
ipaddress.ip_network("64:ff9b::/96"),
)
def is_public_ip(ip: str | int) -> bool:
try:
obj = ipaddress.ip_address(ip)
return not (
obj.is_private
or obj.is_loopback
or obj.is_link_local
or obj.is_multicast
or obj.is_unspecified
or any(obj in network for network in _NON_PUBLIC_NETWORKS)
class BlockReason(StrEnum):
NON_PUBLIC_ADDRESS = "non_public_address"
UNIX_SOCKET = "unix_socket"
class OutboundRequestBlockedError(Exception):
"""
An outbound connection was refused by policy before any socket was opened.
For NON_PUBLIC_ADDRESS, ``host`` is the name or literal being connected to
and ``address`` the first offending address. For UNIX_SOCKET, ``host`` is
the socket path and ``port`` and ``address`` are None.
``address`` is deliberately left out of the message: the message is logged
and stored on failed tasks, and must not disclose internal addresses.
"""
def __init__(
self,
*,
host: str,
port: int | None,
reason: BlockReason,
address: IPAddress | None = None,
) -> None:
self.host = host
self.port = port
self.reason = reason
self.address = address
target = host if port is None else f"{host}:{port}"
super().__init__(f"Outbound connection to {target} blocked ({reason})")
def __reduce__(self) -> tuple[Callable[..., Self], tuple[object, ...]]:
# Celery rebuilds failed-task exceptions by pickling; keyword-only
# fields cannot be recovered from ``args`` alone.
return (
functools.partial(
type(self),
host=self.host,
port=self.port,
reason=self.reason,
address=self.address,
),
(),
)
except ValueError: # pragma: no cover
return False
def resolve_hostname_ips(hostname: str) -> list[str]:
try:
addr_info = socket.getaddrinfo(hostname, None)
except socket.gaierror as e:
raise ValueError(f"Could not resolve hostname: {hostname}") from e
class HostResolutionError(Exception):
"""The resolver returned no usable addresses for a host."""
ips = [info[4][0] for info in addr_info if info and info[4]]
if not ips:
raise ValueError(f"Could not resolve hostname: {hostname}")
return ips
def __init__(self, *, host: str, detail: str) -> None:
self.host = host
self.detail = detail
super().__init__(f"Could not resolve {host}: {detail}")
def __reduce__(self) -> tuple[Callable[..., Self], tuple[object, ...]]:
return (
functools.partial(type(self), host=self.host, detail=self.detail),
(),
)
def format_host_for_url(host: str) -> str:
def blocked_message(exc: OutboundRequestBlockedError | HostResolutionError) -> str:
"""User-facing text for validation errors, kept stable for existing callers."""
if isinstance(exc, HostResolutionError):
return f"Could not resolve hostname: {exc.host}"
if exc.reason is BlockReason.UNIX_SOCKET:
return "Connection blocked: unix sockets are not permitted"
return f"Connection blocked: {exc.host} resolves to a non-public address"
def is_public_ip(ip: IPAddress) -> bool:
"""
Format IP address for URL use (wrap IPv6 in brackets).
True when ``ip`` is globally routable unicast and not in a range that
ipaddress reports as global but which still reaches internal hosts.
"""
return (
ip.is_global
and not ip.is_multicast
and not any(ip in network for network in _NON_PUBLIC_NETWORKS)
)
# Resolver and clock indirection so tests can fake DNS and time for this module
# without changing how the stock httpcore backends resolve the literals the
# guard dials.
_getaddrinfo = socket.getaddrinfo
_agetaddrinfo = anyio.getaddrinfo
# The clock is a seam because time-machine does not mock monotonic clocks, and
# patching time.monotonic globally would also replace the asyncio event loop's
# own clock, hanging or misfiring its timers for the rest of the test.
_monotonic = time.monotonic
def _collect_addresses(
host: str,
infos: Iterable[tuple[Any, ...]],
) -> tuple[IPAddress, ...]:
# Resolver output is always an address, but a scoped IPv6 answer carries a
# zone id ("fe80::1%1"), which is dropped before classification.
# dict keys keep the first occurrence and resolver order
addresses: dict[IPAddress, None] = {}
for info in infos:
address = ipaddress.ip_address(str(info[4][0]).split("%", 1)[0])
addresses.setdefault(address, None)
if not addresses:
raise HostResolutionError(host=host, detail="no addresses returned")
return tuple(addresses)
def _require_public(
host: str,
port: int | None,
addresses: tuple[IPAddress, ...],
) -> tuple[IPAddress, ...]:
for address in addresses:
if not is_public_ip(address):
raise OutboundRequestBlockedError(
host=host,
port=port,
reason=BlockReason.NON_PUBLIC_ADDRESS,
address=address,
)
return addresses
def resolve_public_addresses(host: str, port: int | None) -> tuple[IPAddress, ...]:
"""
Resolve ``host`` and return its addresses in resolver order, or raise if
any of them is non-public. A name is rejected as a whole; offending
addresses are never filtered out.
IP literals go through the resolver too: getaddrinfo answers them without
a lookup, and validating only its answer means no second parser can read
the host differently from the one that connects.
"""
try:
ip_obj = ipaddress.ip_address(host)
if ip_obj.version == 6:
return f"[{host}]"
return host
except ValueError:
return host
infos = _getaddrinfo(host, port, type=socket.SOCK_STREAM)
except (OSError, UnicodeError) as e:
raise HostResolutionError(host=host, detail=str(e)) from e
return _require_public(host, port, _collect_addresses(host, infos))
async def aresolve_public_addresses(
host: str,
port: int | None,
) -> tuple[IPAddress, ...]:
"""Async variant of resolve_public_addresses."""
try:
infos = await _agetaddrinfo(host, port, type=socket.SOCK_STREAM)
except (OSError, UnicodeError) as e:
raise HostResolutionError(host=host, detail=str(e)) from e
return _require_public(host, port, _collect_addresses(host, infos))
MAX_ADDRESSES_TRIED: Final = 8
MIN_ATTEMPT_TIMEOUT: Final = 2.0
MAX_ATTEMPT_TIMEOUT: Final = 10.0
def _require_positive_timeout(host: str, timeout: float | None) -> None:
# A zero timeout makes the socket non-blocking and a negative one is
# rejected by settimeout; neither can produce a useful connection attempt.
if timeout is not None and timeout <= 0:
raise httpcore.ConnectTimeout(
f"Connect timeout for {host} must be positive, got {timeout}",
)
def _deadline(timeout: float | None) -> float:
return math.inf if timeout is None else _monotonic() + timeout
def _attempt_order(addresses: tuple[IPAddress, ...]) -> list[IPAddress]:
# Alternate address families, starting with the resolver's first family
# (RFC 8305 section 4), so one unreachable family cannot delay the other.
first_version = addresses[0].version
primary = [a for a in addresses if a.version == first_version]
secondary = [a for a in addresses if a.version != first_version]
ordered: list[IPAddress] = []
for index in range(max(len(primary), len(secondary))):
ordered.extend(primary[index : index + 1])
ordered.extend(secondary[index : index + 1])
return ordered[:MAX_ADDRESSES_TRIED]
def _attempt_timeout(remaining: float, attempts_left: int) -> float:
"""
Budget for the next attempt. Once the budget is too small to split, or on
the last address, the attempt gets everything left. Otherwise it gets an
equal share clamped to [MIN, MAX], always leaving MIN for a later attempt.
The floor survives one lost SYN; the ceiling bounds how long a black-holed
address delays the next one.
"""
if attempts_left == 1 or remaining < 2 * MIN_ATTEMPT_TIMEOUT:
return remaining
share = remaining / attempts_left
return min(
MAX_ATTEMPT_TIMEOUT,
max(MIN_ATTEMPT_TIMEOUT, share),
remaining - MIN_ATTEMPT_TIMEOUT,
)
def _as_httpcore_timeout(seconds: float) -> float | None:
return None if math.isinf(seconds) else seconds
def _log_block(error: OutboundRequestBlockedError) -> None:
logger.warning("Blocked outbound connection: %s", error)
def _budget_exhausted(host: str, tried: int, total: int) -> httpcore.ConnectTimeout:
return httpcore.ConnectTimeout(
f"Timed out connecting to {host} after trying {tried} of {total} addresses",
)
def _next_attempt_budget(
host: str,
deadline: float,
candidates: list[IPAddress],
index: int,
) -> float:
"""Budget for the attempt at index, or a timeout if none is left."""
remaining = deadline - _monotonic()
if remaining <= 0:
raise _budget_exhausted(host, index, len(candidates))
return _attempt_timeout(remaining, len(candidates) - index)
def _resolve_for_connect(host: str, port: int) -> tuple[IPAddress, ...]:
try:
return resolve_public_addresses(host, port)
except OutboundRequestBlockedError as e:
_log_block(e)
raise
except HostResolutionError as e:
raise httpcore.ConnectError(str(e)) from e
async def _aresolve_for_connect(
host: str,
port: int,
timeout: float | None,
) -> tuple[IPAddress, ...]:
# The scope closes before dialling; attempts are not nested inside it.
try:
with anyio.fail_after(timeout):
return await aresolve_public_addresses(host, port)
except TimeoutError as e:
raise httpcore.ConnectTimeout(f"Timed out resolving {host}") from e
except OutboundRequestBlockedError as e:
_log_block(e)
raise
except HostResolutionError as e:
raise httpcore.ConnectError(str(e)) from e
class _GuardedSyncBackend(httpcore.NetworkBackend):
"""
Wraps httpcore's sync backend. With internal addresses disallowed, it
resolves the origin host itself, rejects the name if any address is
non-public, and dials the validated literals so the checked address is
the connected one. TLS still verifies against the origin hostname.
"""
def __init__(self, inner: httpcore.NetworkBackend, *, allow_internal: bool) -> None:
self._inner = inner
self._allow_internal = allow_internal
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.NetworkStream:
if self._allow_internal:
return self._inner.connect_tcp(
host,
port,
timeout=timeout,
local_address=local_address,
socket_options=socket_options,
)
_require_positive_timeout(host, timeout)
# Resolution is not charged to the budget, matching the stock backend.
candidates = _attempt_order(_resolve_for_connect(host, port))
deadline = _deadline(timeout)
last_error: httpcore.ConnectError | httpcore.ConnectTimeout | None = None
for index, address in enumerate(candidates):
budget = _next_attempt_budget(host, deadline, candidates, index)
try:
return self._inner.connect_tcp(
str(address),
port,
timeout=_as_httpcore_timeout(budget),
local_address=local_address,
socket_options=socket_options,
)
except (httpcore.ConnectError, httpcore.ConnectTimeout) as e:
logger.debug("Connecting to %s via %s failed: %s", host, address, e)
last_error = e
# candidates is never empty, so every address was tried and failed
raise last_error or _budget_exhausted(host, len(candidates), len(candidates))
def connect_unix_socket(
self,
path: str,
timeout: float | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.NetworkStream:
error = OutboundRequestBlockedError(
host=path,
port=None,
reason=BlockReason.UNIX_SOCKET,
)
_log_block(error)
raise error
def sleep(self, seconds: float) -> None:
self._inner.sleep(seconds)
class _GuardedAsyncBackend(httpcore.AsyncNetworkBackend):
"""Async twin of _GuardedSyncBackend."""
def __init__(
self,
inner: httpcore.AsyncNetworkBackend,
*,
allow_internal: bool,
) -> None:
self._inner = inner
self._allow_internal = allow_internal
async def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.AsyncNetworkStream:
if self._allow_internal:
return await self._inner.connect_tcp(
host,
port,
timeout=timeout,
local_address=local_address,
socket_options=socket_options,
)
_require_positive_timeout(host, timeout)
# Resolution counts against the budget, matching the stock backend.
deadline = _deadline(timeout)
candidates = _attempt_order(await _aresolve_for_connect(host, port, timeout))
last_error: httpcore.ConnectError | httpcore.ConnectTimeout | None = None
for index, address in enumerate(candidates):
budget = _next_attempt_budget(host, deadline, candidates, index)
try:
return await self._inner.connect_tcp(
str(address),
port,
timeout=_as_httpcore_timeout(budget),
local_address=local_address,
socket_options=socket_options,
)
except (httpcore.ConnectError, httpcore.ConnectTimeout) as e:
logger.debug("Connecting to %s via %s failed: %s", host, address, e)
last_error = e
raise last_error or _budget_exhausted(host, len(candidates), len(candidates))
async def connect_unix_socket(
self,
path: str,
timeout: float | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.AsyncNetworkStream:
error = OutboundRequestBlockedError(
host=path,
port=None,
reason=BlockReason.UNIX_SOCKET,
)
_log_block(error)
raise error
async def sleep(self, seconds: float) -> None:
await self._inner.sleep(seconds)
_LAYOUT_ERROR = (
"Unexpected httpx transport layout; refusing to create a transport "
"without the outbound connection guard"
)
class GuardedHTTPTransport(httpx.HTTPTransport):
"""
httpx transport whose connections pass through the outbound guard.
Deliberately accepts no proxy, uds or retries options: a proxy would be
dialled instead of the destination, and a unix socket bypasses TCP
entirely. Adding an option here is a reviewed change, not a pass-through.
"""
def __init__(self, *, allow_internal: bool) -> None:
super().__init__()
# httpx has no public hook for the network backend. Check the exact
# layout before swapping so an httpx or httpcore change fails loudly.
pool = self._pool
if (
type(pool) is not httpcore.ConnectionPool
or type(pool._network_backend) is not httpcore.SyncBackend
):
raise RuntimeError(_LAYOUT_ERROR)
pool._network_backend = _GuardedSyncBackend(
pool._network_backend,
allow_internal=allow_internal,
)
class GuardedAsyncHTTPTransport(httpx.AsyncHTTPTransport):
"""Async twin of GuardedHTTPTransport."""
def __init__(self, *, allow_internal: bool) -> None:
super().__init__()
pool = self._pool
if (
type(pool) is not httpcore.AsyncConnectionPool
or type(pool._network_backend) is not AutoBackend
):
raise RuntimeError(_LAYOUT_ERROR)
pool._network_backend = _GuardedAsyncBackend(
pool._network_backend,
allow_internal=allow_internal,
)
def create_guarded_httpx_client(
url: str,
*,
allow_internal: bool,
timeout: float,
) -> httpx.Client:
"""
Validate ``url`` up front, then build a client that re-checks at connect
time. The up-front check turns static misconfiguration into a ValueError
before any retry layer sees it.
"""
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.Client(
transport=GuardedHTTPTransport(allow_internal=allow_internal),
timeout=timeout,
)
def create_guarded_async_httpx_client(
url: str,
*,
allow_internal: bool,
timeout: float,
) -> httpx.AsyncClient:
"""Async twin of create_guarded_httpx_client."""
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=allow_internal),
timeout=timeout,
)
# urllib3 treats a backslash as ending the authority while urlparse and httpx do
# not, so the host checked here could differ from the one that is dialled.
# Control and whitespace characters are refused for the same reason.
_UNSAFE_URL_CHARS = re.compile(r"[\\\x00-\x1f\x7f\s]")
def _dns_name(url: str) -> str:
"""
The ASCII hostname that httpx and urllib3 look up for ``url``.
urlparse keeps a non-ASCII hostname as typed, and getaddrinfo would then
encode it with the stdlib IDNA 2003 codec. That maps some characters
differently from the IDNA 2008 encoding the HTTP clients use ("faß"
becomes "fass" instead of "xn--fa-hia"), so the check would resolve a
different name from the one that is connected to.
"""
try:
return httpx.URL(url).raw_host.decode("ascii")
except (httpx.InvalidURL, UnicodeError) as e:
raise ValueError("Invalid URL scheme or hostname.") from e
def validate_outbound_http_url(
@@ -81,128 +553,17 @@ def validate_outbound_http_url(
raise ValueError("Destination port not permitted.")
if not allow_internal:
for ip_str in resolve_hostname_ips(parsed.hostname):
if not is_public_ip(ip_str):
raise ValueError(
f"Connection blocked: {parsed.hostname} resolves to a non-public address",
)
if _UNSAFE_URL_CHARS.search(url):
raise ValueError("Invalid URL scheme or hostname.")
host = _dns_name(url)
# HTTP clients may percent-decode the host before resolving it, so the
# checked name could differ from the dialled one. An IPv6 zone id is the
# only legitimate use, and link-local addresses are non-public anyway.
if "%" in host:
raise ValueError("Invalid URL scheme or hostname.")
try:
resolve_public_addresses(host, port)
except (OutboundRequestBlockedError, HostResolutionError) as e:
raise ValueError(blocked_message(e)) from e
return parsed
def _rewrite_request_to_pinned_ip(
request: httpx.Request,
*,
allow_internal: bool,
) -> httpx.Request:
hostname = request.url.host
if not hostname:
raise httpx.ConnectError("No hostname in request URL")
try:
ips = resolve_hostname_ips(hostname)
except ValueError as e:
raise httpx.ConnectError(str(e)) from e
if not allow_internal:
for ip_str in ips:
if not is_public_ip(ip_str):
raise httpx.ConnectError(
f"Connection blocked: {hostname} resolves to a non-public address",
)
ip_str = ips[0]
formatted_ip = format_host_for_url(ip_str)
new_headers = httpx.Headers(request.headers)
if "host" in new_headers:
del new_headers["host"]
host_header = format_host_for_url(hostname)
default_port = 443 if request.url.scheme == "https" else 80
if request.url.port and request.url.port != default_port:
host_header = f"{host_header}:{request.url.port}"
new_headers["Host"] = host_header
new_url = request.url.copy_with(host=formatted_ip)
rewritten_request = httpx.Request(
method=request.method,
url=new_url,
headers=new_headers,
stream=request.stream,
extensions=request.extensions,
)
rewritten_request.extensions["sni_hostname"] = hostname
return rewritten_request
class PinnedHostHTTPTransport(httpx.HTTPTransport):
"""
HTTP transport that resolves/validates hostnames per request and connects to
a vetted IP while preserving the original Host header and TLS SNI hostname.
"""
def __init__(
self,
*args,
allow_internal: bool = False,
**kwargs,
) -> None:
super().__init__(*args, **kwargs)
self.allow_internal = allow_internal
def handle_request(self, request: httpx.Request) -> httpx.Response:
request = _rewrite_request_to_pinned_ip(
request,
allow_internal=self.allow_internal,
)
return super().handle_request(request)
class PinnedHostAsyncHTTPTransport(httpx.AsyncHTTPTransport):
"""
Async variant of PinnedHostHTTPTransport.
"""
def __init__(
self,
*args,
allow_internal: bool = False,
**kwargs,
) -> None:
super().__init__(*args, **kwargs)
self.allow_internal = allow_internal
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
request = _rewrite_request_to_pinned_ip(
request,
allow_internal=self.allow_internal,
)
return await super().handle_async_request(request)
def create_pinned_httpx_client(
url: str,
*,
allow_internal: bool = False,
**kwargs,
) -> httpx.Client:
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.Client(
transport=PinnedHostHTTPTransport(allow_internal=allow_internal),
**kwargs,
)
def create_pinned_async_httpx_client(
url: str,
*,
allow_internal: bool = False,
**kwargs,
) -> httpx.AsyncClient:
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.AsyncClient(
transport=PinnedHostAsyncHTTPTransport(allow_internal=allow_internal),
**kwargs,
)
+5 -1
View File
@@ -394,7 +394,11 @@ class RasterisedDocumentParser:
plain_pdf_path = Path(self.tempdir) / "image_plain.pdf"
try:
convert_kwargs: dict = {}
convert_kwargs: dict = {
# Ignore invalid EXIF orientation values (e.g. 0) instead of
# aborting the conversion; valid values are still applied
"rotation": img2pdf.Rotation.ifvalid,
}
if self.settings.image_dpi is not None:
convert_kwargs["layout_fun"] = img2pdf.get_fixed_dpi_layout_fun(
(self.settings.image_dpi, self.settings.image_dpi),
+4 -4
View File
@@ -22,11 +22,11 @@ if TYPE_CHECKING:
@pytest.fixture(scope="session")
def samples_dir() -> Path:
def parser_samples_dir() -> Path:
"""Absolute path to the shared parser sample files directory.
Sub-package conftest files derive format-specific paths from this root,
e.g. ``samples_dir / "text" / "test.txt"``.
e.g. ``parser_samples_dir / "text" / "test.txt"``.
Returns
-------
@@ -37,7 +37,7 @@ def samples_dir() -> Path:
@pytest.fixture(scope="session")
def tagged_no_text_pdf_file(samples_dir: Path) -> Path:
def tagged_no_text_pdf_file(parser_samples_dir: Path) -> Path:
"""Path to a tagged PDF whose only "text" is pdftotext layout padding.
Reproduces GH #13387: ``/MarkInfo /Marked true`` is set, but the only
@@ -50,7 +50,7 @@ def tagged_no_text_pdf_file(samples_dir: Path) -> Path:
Path
Absolute path to ``tesseract/tagged-but-no-text.pdf``.
"""
return samples_dir / "tesseract" / "tagged-but-no-text.pdf"
return parser_samples_dir / "tesseract" / "tagged-but-no-text.pdf"
@pytest.fixture(autouse=True)
+12 -240
View File
@@ -37,15 +37,15 @@ if TYPE_CHECKING:
@pytest.fixture(scope="session")
def text_samples_dir(samples_dir: Path) -> Path:
def text_samples_dir(parser_samples_dir: Path) -> Path:
"""Absolute path to the text parser sample files directory.
Returns
-------
Path
``<samples_dir>/text/``
``<parser_samples_dir>/text/``
"""
return samples_dir / "text"
return parser_samples_dir / "text"
@pytest.fixture(scope="session")
@@ -175,15 +175,15 @@ def no_engine_settings(
@pytest.fixture(scope="session")
def tika_samples_dir(samples_dir: Path) -> Path:
def tika_samples_dir(parser_samples_dir: Path) -> Path:
"""Absolute path to the Tika parser sample files directory.
Returns
-------
Path
``<samples_dir>/tika/``
``<parser_samples_dir>/tika/``
"""
return samples_dir / "tika"
return parser_samples_dir / "tika"
@pytest.fixture(scope="session")
@@ -258,15 +258,15 @@ def tika_parser() -> Generator[TikaDocumentParser, None, None]:
@pytest.fixture(scope="session")
def mail_samples_dir(samples_dir: Path) -> Path:
def mail_samples_dir(parser_samples_dir: Path) -> Path:
"""Absolute path to the mail parser sample files directory.
Returns
-------
Path
``<samples_dir>/mail/``
``<parser_samples_dir>/mail/``
"""
return samples_dir / "mail"
return parser_samples_dir / "mail"
@pytest.fixture(scope="session")
@@ -421,75 +421,15 @@ def nginx_base_url() -> Generator[str, None, None]:
@pytest.fixture(scope="session")
def tesseract_samples_dir(samples_dir: Path) -> Path:
def tesseract_samples_dir(parser_samples_dir: Path) -> Path:
"""Absolute path to the tesseract parser sample files directory.
Returns
-------
Path
``<samples_dir>/tesseract/``
``<parser_samples_dir>/tesseract/``
"""
return samples_dir / "tesseract"
@pytest.fixture(scope="session")
def document_webp_file(tesseract_samples_dir: Path) -> Path:
"""Path to a WebP document sample file.
Returns
-------
Path
Absolute path to ``tesseract/document.webp``.
"""
return tesseract_samples_dir / "document.webp"
@pytest.fixture(scope="session")
def encrypted_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to an encrypted PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/encrypted.pdf``.
"""
return tesseract_samples_dir / "encrypted.pdf"
@pytest.fixture(scope="session")
def multi_page_digital_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a multi-page digital PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/multi-page-digital.pdf``.
"""
return tesseract_samples_dir / "multi-page-digital.pdf"
@pytest.fixture(scope="session")
def multi_page_images_alpha_rgb_tiff_file(tesseract_samples_dir: Path) -> Path:
"""Path to a multi-page TIFF with alpha channel in RGB.
Returns
-------
Path
Absolute path to ``tesseract/multi-page-images-alpha-rgb.tiff``.
"""
return tesseract_samples_dir / "multi-page-images-alpha-rgb.tiff"
@pytest.fixture(scope="session")
def multi_page_images_alpha_tiff_file(tesseract_samples_dir: Path) -> Path:
"""Path to a multi-page TIFF with alpha channel.
Returns
-------
Path
Absolute path to ``tesseract/multi-page-images-alpha.tiff``.
"""
return tesseract_samples_dir / "multi-page-images-alpha.tiff"
return parser_samples_dir / "tesseract"
@pytest.fixture(scope="session")
@@ -504,90 +444,6 @@ def multi_page_images_pdf_file(tesseract_samples_dir: Path) -> Path:
return tesseract_samples_dir / "multi-page-images.pdf"
@pytest.fixture(scope="session")
def multi_page_images_tiff_file(tesseract_samples_dir: Path) -> Path:
"""Path to a multi-page TIFF sample file.
Returns
-------
Path
Absolute path to ``tesseract/multi-page-images.tiff``.
"""
return tesseract_samples_dir / "multi-page-images.tiff"
@pytest.fixture(scope="session")
def multi_page_mixed_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a multi-page mixed PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/multi-page-mixed.pdf``.
"""
return tesseract_samples_dir / "multi-page-mixed.pdf"
@pytest.fixture(scope="session")
def no_text_alpha_png_file(tesseract_samples_dir: Path) -> Path:
"""Path to a PNG with alpha channel and no text.
Returns
-------
Path
Absolute path to ``tesseract/no-text-alpha.png``.
"""
return tesseract_samples_dir / "no-text-alpha.png"
@pytest.fixture(scope="session")
def rotated_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a rotated PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/rotated.pdf``.
"""
return tesseract_samples_dir / "rotated.pdf"
@pytest.fixture(scope="session")
def rtl_test_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to an RTL test PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/rtl-test.pdf``.
"""
return tesseract_samples_dir / "rtl-test.pdf"
@pytest.fixture(scope="session")
def signed_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a signed PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/signed.pdf``.
"""
return tesseract_samples_dir / "signed.pdf"
@pytest.fixture(scope="session")
def simple_alpha_png_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple PNG with alpha channel.
Returns
-------
Path
Absolute path to ``tesseract/simple-alpha.png``.
"""
return tesseract_samples_dir / "simple-alpha.png"
@pytest.fixture(scope="session")
def simple_digital_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple digital PDF sample file.
@@ -612,54 +468,6 @@ def simple_no_dpi_png_file(tesseract_samples_dir: Path) -> Path:
return tesseract_samples_dir / "simple-no-dpi.png"
@pytest.fixture(scope="session")
def simple_bmp_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple BMP sample file.
Returns
-------
Path
Absolute path to ``tesseract/simple.bmp``.
"""
return tesseract_samples_dir / "simple.bmp"
@pytest.fixture(scope="session")
def simple_gif_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple GIF sample file.
Returns
-------
Path
Absolute path to ``tesseract/simple.gif``.
"""
return tesseract_samples_dir / "simple.gif"
@pytest.fixture(scope="session")
def simple_heic_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple HEIC sample file.
Returns
-------
Path
Absolute path to ``tesseract/simple.heic``.
"""
return tesseract_samples_dir / "simple.heic"
@pytest.fixture(scope="session")
def simple_jpg_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple JPG sample file.
Returns
-------
Path
Absolute path to ``tesseract/simple.jpg``.
"""
return tesseract_samples_dir / "simple.jpg"
@pytest.fixture(scope="session")
def simple_png_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple PNG sample file.
@@ -672,42 +480,6 @@ def simple_png_file(tesseract_samples_dir: Path) -> Path:
return tesseract_samples_dir / "simple.png"
@pytest.fixture(scope="session")
def simple_tif_file(tesseract_samples_dir: Path) -> Path:
"""Path to a simple TIF sample file.
Returns
-------
Path
Absolute path to ``tesseract/simple.tif``.
"""
return tesseract_samples_dir / "simple.tif"
@pytest.fixture(scope="session")
def single_page_mixed_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a single-page mixed PDF sample file.
Returns
-------
Path
Absolute path to ``tesseract/single-page-mixed.pdf``.
"""
return tesseract_samples_dir / "single-page-mixed.pdf"
@pytest.fixture(scope="session")
def with_form_pdf_file(tesseract_samples_dir: Path) -> Path:
"""Path to a PDF with form sample file.
Returns
-------
Path
Absolute path to ``tesseract/with-form.pdf``.
"""
return tesseract_samples_dir / "with-form.pdf"
# ------------------------------------------------------------------
# Tesseract parser instance and settings helpers
# ------------------------------------------------------------------
@@ -18,6 +18,7 @@ import img2pdf
import magic
import pikepdf
import pytest
from PIL import Image
from documents.parsers import ParseError
@@ -139,3 +140,24 @@ class TestConvertImageToPdfa:
tesseract_parser._convert_image_to_pdfa(simple_png_file)
spy.assert_not_called()
def test_invalid_exif_orientation_is_ignored(
self,
tesseract_parser: RasterisedDocumentParser,
tmp_path: Path,
) -> None:
"""
GIVEN: a JPEG with an invalid EXIF orientation value (0)
WHEN: _convert_image_to_pdfa is called
THEN: the invalid value is ignored and a valid PDF is produced
"""
image_path = tmp_path / "invalid_orientation.jpg"
with Image.new("RGB", (120, 80), "white") as image:
exif = image.getexif()
exif[274] = 0 # EXIF tag 274: Orientation
image.save(image_path, exif=exif)
result = tesseract_parser._convert_image_to_pdfa(image_path)
assert result.exists()
assert magic.from_file(str(result), mime=True) == "application/pdf"
@@ -10,8 +10,8 @@ from imagehash import average_hash
from PIL import Image
from pytest_mock import MockerFixture
from documents.tests.utils import util_call_with_backoff
from paperless.parsers.mail import MailDocumentParser
from paperless_testing.retry import util_call_with_backoff
def extract_text(pdf_path: Path) -> str:
@@ -3,14 +3,14 @@ import json
from django.test import TestCase
from django.test import override_settings
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import FileSystemAssertsMixin
from paperless.models import ApplicationConfiguration
from paperless.models import CleanChoices
from paperless.models import ColorConvertChoices
from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices
from paperless.parsers.tesseract import RasterisedDocumentParser
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
class TestParserSettingsFromDb(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
@@ -3,8 +3,8 @@ from pathlib import Path
import pytest
from documents.tests.utils import util_call_with_backoff
from paperless.parsers.tika import TikaDocumentParser
from paperless_testing.retry import util_call_with_backoff
@pytest.mark.skipif(
@@ -2,22 +2,20 @@ import os
from unittest import mock
from django.conf import settings
from django.contrib.auth.models import User
from django.test import override_settings
from rest_framework import status
from rest_framework.test import APITestCase
from documents.tests.utils import DirectoriesMixin
from paperless.settings import _parse_remote_user_settings
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import UserFactory
class TestRemoteUser(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create_superuser(
username="temp_admin",
)
self.user = UserFactory(username="temp_admin", superuser=True)
def test_remote_user(self) -> None:
"""
+6 -5
View File
@@ -15,6 +15,7 @@ from pytest_mock import MockerFixture
from rest_framework.authtoken.models import Token
from paperless.adapter import DrfTokenStrategy
from paperless_testing.factories import UserFactory
@pytest.mark.django_db
@@ -25,7 +26,7 @@ class TestCustomAccountAdapter:
# With no accounts, signups should be allowed
assert adapter.is_open_for_signup(None)
User.objects.create_user("testuser")
UserFactory(username="testuser")
settings.ACCOUNT_ALLOW_SIGNUPS = True
assert adapter.is_open_for_signup(None)
@@ -92,7 +93,7 @@ class TestCustomAccountAdapter:
) -> None:
settings.ACCOUNT_DEFAULT_GROUPS = ["group1", "group2"]
Group.objects.create(name="group1")
user = User.objects.create_user("testuser")
user = UserFactory(username="testuser")
adapter = get_adapter()
form = mocker.MagicMock(
cleaned_data={
@@ -152,7 +153,7 @@ class TestCustomSocialAccountAdapter:
settings.SOCIAL_ACCOUNT_DEFAULT_GROUPS = ["group1", "group2"]
Group.objects.create(name="group1")
adapter = get_social_adapter()
user = User.objects.create_user("testuser")
user = UserFactory(username="testuser")
sociallogin = mocker.MagicMock(user=user)
user = adapter.save_user(HttpRequest(), sociallogin, None)
@@ -187,7 +188,7 @@ class TestDrfTokenStrategy:
THEN:
- A new token is created and its key is returned
"""
user = User.objects.create_user("testuser")
user = UserFactory(username="testuser")
request = HttpRequest()
request.user = user
@@ -207,7 +208,7 @@ class TestDrfTokenStrategy:
THEN:
- The same token key is returned (no new token created)
"""
user = User.objects.create_user("testuser")
user = UserFactory(username="testuser")
existing_token = Token.objects.create(user=user)
request = HttpRequest()
+3 -5
View File
@@ -1,12 +1,13 @@
import uuid
from django.contrib.auth.models import User
from django.test import TestCase
from django.test import override_settings
from django.urls import resolve
from django.urls import reverse
from rest_framework import status
from paperless_testing.factories import UserFactory
class TestApiAuthViews(TestCase):
def test_api_auth_login_uses_allauth_login_view(self):
@@ -24,10 +25,7 @@ class TestApiAuthViews(TestCase):
@override_settings(DISABLE_REGULAR_LOGIN=True)
def test_api_auth_login_respects_disable_regular_login(self):
username = f"testuser-{uuid.uuid4().hex}"
User.objects.create_user(
username=username,
password="testpassword",
)
UserFactory(username=username, password="testpassword")
response = self.client.post(
reverse("rest_framework:login"),
+3 -3
View File
@@ -1,10 +1,10 @@
from django.contrib.auth.models import AnonymousUser
from django.contrib.auth.models import User
from django.test import RequestFactory
from django.test import TestCase
from django.test import override_settings
from paperless.auth import AutoLoginMiddleware
from paperless_testing.factories import UserFactory
@override_settings(AUTO_LOGIN_USERNAME="autologin")
@@ -29,7 +29,7 @@ class TestAutoLoginMiddleware(TestCase):
THEN:
- That user is attached to the request
"""
user = User.objects.create_user(username="autologin")
user = UserFactory(username="autologin")
request = self._process(self.factory.get("/"))
@@ -44,7 +44,7 @@ class TestAutoLoginMiddleware(TestCase):
THEN:
- The request is left anonymous rather than authenticated as them
"""
User.objects.create_user(username="autologin", is_active=False)
UserFactory(username="autologin", is_active=False)
request = self.factory.get("/")
request.user = AnonymousUser()
+9 -38
View File
@@ -1,6 +1,5 @@
import os
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from unittest import mock
@@ -19,35 +18,7 @@ from paperless.checks import check_v3_minimum_upgrade_version
from paperless.checks import debug_mode_check
from paperless.checks import paths_check
from paperless.checks import settings_values_check
@dataclass(frozen=True, slots=True)
class PaperlessTestDirs:
data_dir: Path
media_dir: Path
consumption_dir: Path
# TODO: consolidate with documents/tests/conftest.py PaperlessDirs/paperless_dirs
# once the paperless and documents test suites are ready to share fixtures.
@pytest.fixture()
def directories(tmp_path: Path, settings: Settings) -> PaperlessTestDirs:
data_dir = tmp_path / "data"
media_dir = tmp_path / "media"
consumption_dir = tmp_path / "consumption"
for d in (data_dir, media_dir, consumption_dir):
d.mkdir()
settings.DATA_DIR = data_dir
settings.MEDIA_ROOT = media_dir
settings.CONSUMPTION_DIR = consumption_dir
return PaperlessTestDirs(
data_dir=data_dir,
media_dir=media_dir,
consumption_dir=consumption_dir,
)
from paperless_testing.dirs import PaperlessDirs
class TestChecks:
@@ -58,7 +29,7 @@ class TestChecks:
settings.CONVERT_BINARY = "uuuhh"
assert len(binaries_check(None)) == 1
@pytest.mark.usefixtures("directories")
@pytest.mark.usefixtures("paperless_dirs")
def test_paths_check(self) -> None:
assert paths_check(None) == []
@@ -73,17 +44,17 @@ class TestChecks:
for msg in msgs:
assert msg.msg.endswith("is set but doesn't exist.")
def test_paths_check_no_access(self, directories: PaperlessTestDirs) -> None:
directories.data_dir.chmod(0o000)
directories.media_dir.chmod(0o000)
directories.consumption_dir.chmod(0o000)
def test_paths_check_no_access(self, paperless_dirs: PaperlessDirs) -> None:
paperless_dirs.data_dir.chmod(0o000)
paperless_dirs.media_dir.chmod(0o000)
paperless_dirs.consumption_dir.chmod(0o000)
try:
msgs = paths_check(None)
finally:
directories.data_dir.chmod(0o777)
directories.media_dir.chmod(0o777)
directories.consumption_dir.chmod(0o777)
paperless_dirs.data_dir.chmod(0o777)
paperless_dirs.media_dir.chmod(0o777)
paperless_dirs.consumption_dir.chmod(0o777)
assert len(msgs) == 3
for msg in msgs:
@@ -1,4 +1,4 @@
from documents.tests.utils import TestMigrations
from paperless_testing.migrations import TestMigrations
class TestMigrateSkipArchiveFile(TestMigrations):
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,375 @@
import ipaddress
import os
import httpcore
import httpx
import pytest
from pytest_mock import MockerFixture
from paperless.network import GuardedAsyncHTTPTransport
from paperless.network import GuardedHTTPTransport
from paperless.network import OutboundRequestBlockedError
from paperless.network import create_guarded_httpx_client
from paperless_testing.outbound import DialRecorder
from paperless_testing.outbound import FakeDNS
from paperless_testing.outbound import LocalHTTPServer
from paperless_testing.outbound import running_http_server
class TestGuardedTransportSync:
@pytest.mark.usefixtures("every_address_is_public")
def test_pinned_connection_falls_back_to_next_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A hostname resolving to ::1 then 127.0.0.1
- A server listening on 127.0.0.1 only
- Internal addresses disallowed, with loopback treated as public
WHEN:
- A request is made
THEN:
- ::1 fails, 127.0.0.1 is dialled next and the request succeeds
"""
fake_dns.add("dual-stack.test", "::1", "127.0.0.1")
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
response = client.get(f"http://dual-stack.test:{local_http_server.port}/")
assert response.status_code == 200
assert dial_recorder.hosts() == ["::1", "127.0.0.1"]
def test_allow_internal_uses_stock_resolution(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- Internal addresses allowed
WHEN:
- A request is made to localhost
THEN:
- It succeeds without the guard resolving anything
"""
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=True),
timeout=5.0,
) as client:
response = client.get(f"http://localhost:{local_http_server.port}/")
assert response.status_code == 200
assert fake_dns.lookups == []
@pytest.mark.usefixtures("every_address_is_public")
def test_host_header_is_the_hostname(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- A pinned connection to a named host
WHEN:
- A request is made
THEN:
- The server receives the hostname in Host, not the dialled IP
"""
fake_dns.add("pinned.test", "127.0.0.1")
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
client.get(f"http://pinned.test:{local_http_server.port}/")
assert local_http_server.requests[0].headers["host"] == (
f"pinned.test:{local_http_server.port}"
)
def test_redirect_to_internal_host_is_blocked(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- An allowed origin that redirects to a host resolving to a blocked
address, and a client that follows redirects
WHEN:
- The origin is requested
THEN:
- The redirect hop is blocked without dialling the blocked address
"""
allowed = ipaddress.ip_address("127.0.0.1")
mocker.patch(
"paperless.network.is_public_ip",
side_effect=lambda address: address == allowed,
)
fake_dns.add("origin.test", "127.0.0.1")
fake_dns.add("internal.test", "127.0.0.2")
local_http_server.redirect_to = (
f"http://internal.test:{local_http_server.port}/"
)
with (
httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
follow_redirects=True,
) as client,
pytest.raises(OutboundRequestBlockedError) as exc_info,
):
client.get(f"http://origin.test:{local_http_server.port}/")
assert exc_info.value.address == ipaddress.ip_address("127.0.0.2")
assert dial_recorder.hosts() == ["127.0.0.1"]
assert len(local_http_server.requests) == 1
@pytest.mark.usefixtures("every_address_is_public")
def test_connections_are_not_shared_between_hosts_on_one_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- Two hostnames resolving to the same address
- Internal addresses disallowed, with loopback treated as public
WHEN:
- One client requests the first host twice, then the second host
THEN:
- The first host's connection is reused for its second request
- The second host gets its own connection, so its certificate would
be checked rather than inheriting the first host's session
"""
fake_dns.add("first.test", "127.0.0.1")
fake_dns.add("second.test", "127.0.0.1")
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
client.get(f"http://first.test:{local_http_server.port}/")
client.get(f"http://first.test:{local_http_server.port}/")
client.get(f"http://second.test:{local_http_server.port}/")
assert dial_recorder.hosts() == ["127.0.0.1", "127.0.0.1"]
assert local_http_server.connections == 2
assert [request.headers["host"] for request in local_http_server.requests] == [
f"first.test:{local_http_server.port}",
f"first.test:{local_http_server.port}",
f"second.test:{local_http_server.port}",
]
@pytest.mark.usefixtures("every_address_is_public")
def test_tls_uses_the_hostname_not_the_dialled_address(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A pinned HTTPS connection to a named host
- A plain HTTP server, so the handshake itself fails
WHEN:
- A request is made
THEN:
- The validated address is dialled
- TLS is started with the hostname for SNI and certificate checks
"""
fake_dns.add("pinned.test", "127.0.0.1")
start_tls = mocker.spy(httpcore._backends.sync.SyncStream, "start_tls")
with (
httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client,
pytest.raises(httpx.ConnectError),
):
client.get(f"https://pinned.test:{local_http_server.port}/")
assert dial_recorder.hosts() == ["127.0.0.1"]
start_tls.assert_called_once()
assert start_tls.call_args.kwargs["server_hostname"] == "pinned.test"
@pytest.mark.parametrize(
"host",
[
pytest.param("localhost", id="name"),
pytest.param("2130706433", id="decimal"),
pytest.param("0x7f.1", id="hex-short"),
pytest.param("127.1", id="short-dotted"),
],
)
def test_blocks_internal_host_without_connecting(
self,
local_http_server: LocalHTTPServer,
dial_recorder: DialRecorder,
host: str,
) -> None:
"""
GIVEN:
- Internal addresses disallowed
- A URL whose host reaches loopback, by name or by a
non-canonical spelling of 127.0.0.1
WHEN:
- A request is made through the transport
THEN:
- The resolved address is checked, the request is blocked and the
server never sees a connection
"""
with (
httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client,
pytest.raises(OutboundRequestBlockedError),
):
client.get(f"http://{host}:{local_http_server.port}/")
assert local_http_server.connections == 0
assert dial_recorder.hosts() == []
@pytest.mark.usefixtures("every_address_is_public")
def test_environment_proxy_is_not_used(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- Proxy variables in the environment pointing at a second local server
- Internal addresses disallowed
WHEN:
- A request is made through the production client factory to an
allowed origin
THEN:
- The origin server receives the request directly and the proxy
server never sees a connection
"""
with running_http_server() as proxy_server:
mocker.patch.dict(
os.environ,
{
"HTTP_PROXY": f"http://127.0.0.1:{proxy_server.port}",
"HTTPS_PROXY": f"http://127.0.0.1:{proxy_server.port}",
"ALL_PROXY": f"http://127.0.0.1:{proxy_server.port}",
},
)
fake_dns.add("origin.test", "127.0.0.1")
url = f"http://origin.test:{local_http_server.port}/"
with create_guarded_httpx_client(
url,
allow_internal=False,
timeout=5.0,
) as client:
response = client.get(url)
assert response.status_code == 200
assert len(local_http_server.requests) == 1
assert local_http_server.requests[0].headers["host"] == (
f"origin.test:{local_http_server.port}"
)
assert proxy_server.connections == 0
assert proxy_server.requests == []
assert dial_recorder.hosts() == ["127.0.0.1"]
class TestGuardedTransportAsync:
@pytest.fixture(autouse=True)
def anyio_backend(self) -> str:
return "asyncio"
@pytest.mark.anyio
@pytest.mark.usefixtures("every_address_is_public")
async def test_pinned_connection_falls_back_to_next_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A hostname resolving to ::1 then 127.0.0.1
- A server listening on 127.0.0.1 only
- Internal addresses disallowed, with loopback treated as public
WHEN:
- An async request is made
THEN:
- ::1 fails, 127.0.0.1 is dialled next and the request succeeds
"""
fake_dns.add("dual-stack.test", "::1", "127.0.0.1")
async with httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
response = await client.get(
f"http://dual-stack.test:{local_http_server.port}/",
)
assert response.status_code == 200
assert dial_recorder.hosts() == ["::1", "127.0.0.1"]
@pytest.mark.anyio
async def test_allow_internal_uses_stock_resolution(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- Internal addresses allowed
WHEN:
- An async request is made to localhost
THEN:
- It succeeds without the guard resolving anything
"""
async with httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=True),
timeout=5.0,
) as client:
response = await client.get(f"http://localhost:{local_http_server.port}/")
assert response.status_code == 200
assert fake_dns.lookups == []
@pytest.mark.anyio
async def test_blocks_internal_host_without_connecting(
self,
local_http_server: LocalHTTPServer,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- Internal addresses disallowed
WHEN:
- An async request is made to localhost through the transport
THEN:
- It is blocked and the server never sees a connection
"""
async with httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
with pytest.raises(OutboundRequestBlockedError):
await client.get(f"http://localhost:{local_http_server.port}/")
assert local_http_server.connections == 0
assert dial_recorder.hosts() == []
+23 -63
View File
@@ -1,7 +1,6 @@
from unittest.mock import Mock
from django.contrib.auth.models import Group
from django.contrib.auth.models import User
from django.http import HttpRequest
from django.test import TestCase
from django.test import override_settings
@@ -9,6 +8,7 @@ from django.test import override_settings
from documents.models import UiSettings
from paperless.signals import handle_failed_login
from paperless.signals import handle_social_account_updated
from paperless_testing.factories import UserFactory
class TestFailedLoginLogging(TestCase):
@@ -120,7 +120,7 @@ class TestSyncSocialLoginGroups(TestCase):
- The user's groups are updated to match the social login's groups
"""
group = Group.objects.create(name="group1")
user = User.objects.create_user(username="testuser")
user = UserFactory(username="testuser")
sociallogin = Mock(
user=user,
account=Mock(
@@ -147,7 +147,7 @@ class TestSyncSocialLoginGroups(TestCase):
- The user's groups are not updated
"""
Group.objects.create(name="group1")
user = User.objects.create_user(username="testuser")
user = UserFactory(username="testuser")
sociallogin = Mock(
user=user,
account=Mock(
@@ -180,7 +180,7 @@ class TestSyncSocialLoginGroups(TestCase):
would be rejected for a deactivated user anyway
"""
Group.objects.create(name="admin-group")
user = User.objects.create_user(
user = UserFactory(
username="inactive_user",
is_active=False,
is_superuser=False,
@@ -215,7 +215,7 @@ class TestSyncSocialLoginGroups(TestCase):
- The user's groups are cleared to match the social login's groups
"""
group = Group.objects.create(name="group1")
user = User.objects.create_user(username="testuser")
user = UserFactory(username="testuser")
user.groups.add(group)
user.save()
sociallogin = Mock(
@@ -244,7 +244,7 @@ class TestSyncSocialLoginGroups(TestCase):
- The user's groups are updated using `userinfo.groups`
"""
group = Group.objects.create(name="group1")
user = User.objects.create_user(username="testuser")
user = UserFactory(username="testuser")
sociallogin = Mock(
user=user,
account=Mock(
@@ -275,7 +275,7 @@ class TestSyncSocialLoginGroups(TestCase):
- The user's groups are updated using `id_token.groups`
"""
group = Group.objects.create(name="group1")
user = User.objects.create_user(username="testuser")
user = UserFactory(username="testuser")
sociallogin = Mock(
user=user,
account=Mock(
@@ -310,11 +310,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User is not promoted, since only an exact group match counts
"""
user = User.objects.create_user(
username="testuser",
is_superuser=False,
is_staff=False,
)
user = UserFactory(username="testuser", is_superuser=False, is_staff=False)
sociallogin = Mock(
user=user,
account=Mock(
@@ -345,11 +341,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User becomes superuser and staff
"""
user = User.objects.create_user(
username="testuser_s_e",
is_superuser=False,
is_staff=False,
)
user = UserFactory(username="testuser_s_e", is_superuser=False, is_staff=False)
sociallogin = Mock(
user=user,
account=Mock(
@@ -380,11 +372,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User loses superuser status but preserves staff status if they had it
"""
user = User.objects.create_user(
username="testuser_s_d",
is_superuser=True,
is_staff=True,
)
user = UserFactory(username="testuser_s_d", is_superuser=True, is_staff=True)
sociallogin = Mock(
user=user,
account=Mock(
@@ -415,11 +403,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User becomes staff
"""
user = User.objects.create_user(
username="testuser_st_e",
is_superuser=False,
is_staff=False,
)
user = UserFactory(username="testuser_st_e", is_superuser=False, is_staff=False)
sociallogin = Mock(
user=user,
account=Mock(
@@ -450,11 +434,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User loses staff status
"""
user = User.objects.create_user(
username="testuser_st_d",
is_superuser=False,
is_staff=True,
)
user = UserFactory(username="testuser_st_d", is_superuser=False, is_staff=True)
sociallogin = Mock(
user=user,
account=Mock(
@@ -485,11 +465,7 @@ class TestSyncSocialLoginGroups(TestCase):
- Roles are correctly assigned/revoked according to groups
"""
# Case 1: has both
user = User.objects.create_user(
username="testuser_b_1",
is_superuser=False,
is_staff=False,
)
user = UserFactory(username="testuser_b_1", is_superuser=False, is_staff=False)
sociallogin = Mock(
user=user,
account=Mock(extra_data={"groups": ["admin-group", "staff-group"]}),
@@ -504,11 +480,7 @@ class TestSyncSocialLoginGroups(TestCase):
self.assertTrue(user.is_staff)
# Case 2: has only staff
user2 = User.objects.create_user(
username="testuser_b_2",
is_superuser=True,
is_staff=True,
)
user2 = UserFactory(username="testuser_b_2", is_superuser=True, is_staff=True)
sociallogin2 = Mock(
user=user2,
account=Mock(extra_data={"groups": ["staff-group"]}),
@@ -523,11 +495,7 @@ class TestSyncSocialLoginGroups(TestCase):
self.assertTrue(user2.is_staff)
# Case 3: has neither
user3 = User.objects.create_user(
username="testuser_b_3",
is_superuser=True,
is_staff=True,
)
user3 = UserFactory(username="testuser_b_3", is_superuser=True, is_staff=True)
sociallogin3 = Mock(
user=user3,
account=Mock(extra_data={"groups": ["other-group"]}),
@@ -554,11 +522,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- Existing roles are not modified
"""
user = User.objects.create_user(
username="testuser_n_s",
is_superuser=True,
is_staff=True,
)
user = UserFactory(username="testuser_n_s", is_superuser=True, is_staff=True)
sociallogin = Mock(
user=user,
account=Mock(extra_data={"groups": ["admin-group", "staff-group"]}),
@@ -586,7 +550,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User's superuser status is demoted, matching the group claim exactly
"""
user = User.objects.create_user(
user = UserFactory(
username="local_admin",
password="password123",
is_superuser=True,
@@ -618,11 +582,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User's superuser status is demoted, even though they are the last admin
"""
user = User.objects.create_user(
username="last_admin",
is_superuser=True,
is_staff=True,
)
user = UserFactory(username="last_admin", is_superuser=True, is_staff=True)
user.set_unusable_password()
user.save()
@@ -652,7 +612,7 @@ class TestSyncSocialLoginGroups(TestCase):
THEN:
- User's staff status is demoted, matching the group claim exactly
"""
user = User.objects.create_user(
user = UserFactory(
username="local_staff",
password="password123",
is_superuser=False,
@@ -688,8 +648,8 @@ class TestUserGroupDeletionCleanup(TestCase):
THEN:
- References in ui_settings are cleaned up
"""
user = User.objects.create_user(username="testuser")
user2 = User.objects.create_user(username="testuser2")
user = UserFactory(username="testuser")
user2 = UserFactory(username="testuser2")
group = Group.objects.create(name="testgroup")
ui_settings = UiSettings.objects.create(
@@ -727,8 +687,8 @@ class TestUserGroupDeletionCleanup(TestCase):
THEN:
- Error is logged and the system remains stable
"""
user = User.objects.create_user(username="testuser")
user2 = User.objects.create_user(username="testuser2")
user = UserFactory(username="testuser")
user2 = UserFactory(username="testuser2")
user2_id = user2.id
Group.objects.create(name="testgroup")

Some files were not shown because too many files have changed in this diff Show More