The unittest side of the suite built its temp directory tree with
tempfile.mkdtemp and a manually enabled override_settings, cleaned up only if
tearDown ran. That is now gone. DirectoriesMixin lives alongside the layout it
bridges and does nothing but hand the paperless_dirs fixture to TestCase
subclasses as self.dirs, so both halves of the suite get the same twelve
settings, the same directory shapes and cleanup owned by tmp_path.
The mixin moves to paperless_testing.dirs rather than staying in the documents
test utilities, because modules in paperless and paperless_mail import it
across the app boundary. The thirty-eight consuming modules change only their
import line; self.dirs.scratch_dir and its siblings keep working.
The temp directory layout used by the tests was written out four separate
times: once in the documents conftest, once in the paperless checks tests,
once in a fixture local to the NFC upload tests, and once in the helper
behind the old paperless_environment context manager. Each copy covered a
different subset of the settings, so which directories a test actually got
depended on which copy it happened to reach.
The layout now lives in paperless_testing.dirs. build_paperless_dirs owns
where things go and creates them, dirs_settings maps them onto Django
setting names and is pure, and a paperless_dirs fixture in the root conftest
applies that mapping through pytest-django's settings fixture so every app
can reach it. Tests that need a second environment part way through a test
body use the paperless_environment context manager from the same module,
which expresses the identical layout through override_settings. The three
redundant implementations and the old media settings fixture are gone, and
their consumers now take paperless_dirs.
The model factories lived in the documents test package, but three other
apps needed them. The AI, mail and testing suites all reached across an app
boundary to import from documents.tests.factories, which made a private test
package into a shared dependency.
The factories now live in the shared testing package, where cross-app use is
the intended use.
All four Django apps have test code in common, but the only place to put it
was the documents app's own tests package, so paperless, paperless_ai and
paperless_mail each reached across an app boundary to import helpers and
relied on fixtures that were only defined for the documents test path.
This adds a root src/conftest.py holding the fixtures every app needs and an
new src/paperless_testing package for shared helpers a test names
* Feature: match fuzzy terms in place inside the parsed query
Fuzzy matching was a separate clause OR'd in above the query: a flat bag
of the query's words, re-parsed through tantivy's own parser, blended
beside the exact clause. Nothing around a term reached it, so a fielded
term fuzzed across every default field, a filter did not constrain it, and
an exclusion had to be hoisted back over the whole blend to stop the
clause re-admitting what the query had just excluded.
Widen each leaf where it sits instead, through emit()'s rewrite_leaf hook,
so fielding, negation, AND, REQUIRE and positive filters constrain the
fuzzy match exactly as they constrain the exact one. Each of a leaf's
words becomes a Fuzzy leaf on the leaf's own field, boosted to 0.1, beside
the leaf and any CJK alternative it already had.
* Hello?
QUERY-mode searches blended a separate bigram clause in at the top of the
query, built from the parsed AST's free-text tokens. Because it sat beside
the exact clause rather than inside the query, nothing around a CJK term
constrained its bigram match: an exclusion that was one OR branch's own
condition could never reach it, so "(東京 AND NOT secret) OR bill" still
returned the secret document.
Widen each CJK leaf where it sits instead, through emit()'s rewrite_leaf
hook, so every AND, NOT, REQUIRE, boost and field restriction around the
leaf applies to its bigram match too. Negated leaves are widened on
purpose, so "NOT X" excludes exactly what "X" matches.
* Preprocesses classifier content with Tantivy instead of NLTK
Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.
Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.
* Copies packages instead of hardlinking them in backend CI, some NLTK thing
* Adds a normalization to NFC to better fit what Tantivy expects
BulkEditSerializer's hand-parsed validators only caught the exceptions
their happy paths raised, so wrong-typed input produced a 500 or was
passed through to the task:
- owner: a nonexistent or wrong-typed id raised an uncaught error, and a
boolean was accepted (Django coerces True to pk 1). It is now validated
with PrimaryKeyRelatedField and the validated pk is passed on.
- rotate: null raised TypeError, while true, "90" and 45 were accepted and
failed later in QPDF. Degrees are now an IntegerField plus a
multiple-of-90 check, passing an int on. The dedicated rotate endpoint
and edit_pdf operations share the same multiple-of-90 check.
- split: null raised AttributeError, "0" silently became the last page,
"3-1" gave an empty group that crashed the task, and a range like
"1-5000000" was expanded into a list during the request with no upper
bound. Each range is now checked against 1 <= start <= end <= page_count
before it is built.
SetPermissionsSerializer was a bare DictField, and the legacy bulk_edit
set_permissions parameter bypassed even that by hand-calling
validate_set_permissions() on an unchecked dict. A bool or a non-list in
place of users/groups crashed with a raw TypeError instead of a 400.
It is now a nested serializer (view/change, each with users/groups as
lists of integers), used for owned-object create/update, the legacy
bulk_edit set_permissions parameter, and bulk_edit_objects permissions.
Unknown action keys are rejected: previously a typo like "veiw" was
silently dropped, leaving an empty permission set that could clear
existing grants. An explicit set_permissions null (an owner-only change)
remains a no-op, and an empty bulk_edit_objects permissions dict is still
rejected.
JSONField(binary=True) accepts any JSON value, so a truthy non-dict
(bool/int/list/string) silently passed validation.
Additionally, require string substitutes in barcode_tag_mapping, a non-string
substitute value made the barcode tag substitution fail inside
get_tag_ids, and the tag was silently skipped. barcode_tag_mapping values
must now be strings (empty still means "use the raw barcode").
operations was a ListField with no child, so both EditPdfDocumentsSerializer
and BulkEditSerializer._validate_parameters_edit_pdf hand-checked each entry
with isinstance(). That accepted booleans (isinstance(True, int) is true) and
passed through any extra keys.
Adds PdfEditOperationSerializer (page >= 1, doc >= 0, rotate a multiple of
90) and uses it as the ListField child on both paths, dropping the manual
type checks and the now-redundant negative/page < 1 bounds checks. Rotations
that are not a multiple of 90 previously passed validation and then failed
inside the task when QPDF refused them.
* docs(search): document the query grammar paperless actually supports
* docs(search): cover ?/bracket wildcards and the no-stemmer fallback
* Shorten things up, use tables and less wordy stuff. Fixes some test doc strings that referred to the documentation
* Silly sonarr
* Feature: parse advanced search with whoosh-compat and delete the hand-written translator
* Don't cover these, they exist for defensive, but there's no other current diagnostic
* A mix of more no cover and tests
* Route SCHEMA_FIELD_MISSING into an error, not an HTTP 400
* feat(search): add whoosh-compat, the shared field table and the field registry
* Sonar being useful actually
* Adds the given/when/then commenting
* Trims tests I don't think cover our logic or code or are redundant
* Coverage
* test(search): add pattern normalizer stem-alternates unit tests
* build: bump whoosh-compat to 0.2.0
* Perf: batch guardian permission assignment in bulk-edit
bulk_edit.set_permissions and BulkEditObjectPermissionsView both
looped documents/objects and called set_permissions_for_object per
object, which itself calls guardian's assign_perm/remove_perm once
per (object, user) pair -- ~10-20+ queries per object, scaling with
selection size.
Added set_permissions_for_objects, a bulk equivalent that resolves
existing permission holders once across the whole batch (not once per
object) and applies changes with a small, batch-size-independent
number of queries per action instead of one per (object, user) pair.
* Perf: avoid unnecessary full-row fetches in batch permission assignment
set_permissions_for_objects now takes a model + pks instead of instances,
and identity filtering resolves straight to ids, so bulk-editing
permissions no longer materializes full Document/User/Group rows just to
read their pk/id. Row construction for bulk_create is also chunked to
bound peak memory for very large "apply to all" operations.
* Fix: use .distinct() for existing-grant lookup, drop flaky query-count invariant tests
.distinct() lets the database dedupe identity ids server-side instead of
transferring one row per (object, grantee) match and deduping in Python --
was the dominant cost on a large selection with existing grants.
Also replaced the two query-count-equality tests (bulk_edit and the
bulk_edit_objects API path) with plain functional-correctness checks at
both batch sizes. Hopefully stops that flake.
* Mark empty-pks early-return in set_permissions_for_objects as no-cover
Defensive guard for an edge case (all requested pks already gone/invalid)
rather than a path normal usage exercises; matches the existing
pragma: no cover convention elsewhere in this file.
* Perf: drop speculative row-chunking in bulk permission assignment, keep the query-batching fix
* Resolve every permission action before applying any of them
This fixes the existing issue and resolves the Copilot comment
* Assert permission assignment does not scale with selection size
The two batching tests only checked that permissions came out correct at
5 and 50 objects, so reverting to the old per-object loop would still
have passed. Check sizes as well to prevent that
* perf: skip effective_content annotation on document list unless filtered on
DocumentViewSet.get_queryset() always attached a correlated subquery
resolving each document's latest version content, even though it's only
needed for the deprecated search/title_content/content__* filter params.
Evaluated for every candidate row before pagination's LIMIT, this is
pathological on MariaDB: its default cardinality estimate for the mostly-
NULL root_document_id self-join drives it to a near-full-table scan per
row instead of using the FK index, turning a normal filtered list request
into a multi-second query (root cause of paperless-ngx#13778's report).
Only attach the annotation when a request actually filters on it. The
common case now relies on Document.get_effective_content()'s existing
prefetch-based fallback instead (extended the "versions" prefetch to
include content), which DocumentSerializer.to_representation() now calls
directly instead of checking for the annotation via hasattr().
* fix: address review feedback on effective_content annotation skip
- _needs_effective_content_annotation() now checks for a non-blank,
stripped param value rather than mere key presence, matching how
SearchFilter/TitleContentFilter/EffectiveContentFilter themselves
no-op on a blank value. An empty ?search= or a saved view with a
cleared text filter no longer re-triggers the annotation.
- The "versions" prefetch on DocumentViewSet no longer carries content
for every historical version of every document -- that's unused
bloat for version-heavy documents. Added
latest_version_content_prefetch() (versioning.py), a separate,
windowed prefetch scoped to just the newest version's content per
root, and taught Document.get_effective_content() to check it first.
- DocumentSerializer.to_representation() no longer unconditionally
calls get_effective_content(). Added has_prefetched_effective_content()
(versioning.py) as a cheap upfront check: only resolve version-aware
content when an SQL annotation or a versions prefetch is already on
the instance. TrashView and GlobalSearchView build their own
querysets independently of DocumentViewSet and never display
document content at all (checked both frontend components), so they
now keep showing the document's own, unresolved content with zero
extra queries -- the same behavior as before effective_content
resolution existed, just generalized past the narrow hasattr() check
it replaced.
* Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it
* Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even
* CLean up the new test with the docstrings, handle the fields in one place
* Fun with contenttype and caching. Compare only the
queries spent on the documents themselves or else
A slow writer might create a zero byte file, then take longer than the window to
finish the write. We would then queue a zero byte file for consumption and race the writer
to most likely fail due to still being empty. Instead, drop zero size files at yeild time.
The slow writer may or may not finish, but if it does, there will be a Change.modified
event fired again
* fix(search): resolve index-write permissions and effective content in bulk
Add WriteBatch.add_or_update_ids() and use it in bulk_update_documents
and trash restore, cutting index writes from ~8 queries per document
to a constant handful per batch
* Always these new ones with xdist, try a better condition
* Fix: skip vector store document id filter for unrestricted chat users
ChatStreamingView built an IN filter from every permitted document id
for the "chat over all documents" case, which exceeds the vector
store's SQLite bound-parameter safety limit on installs with more
than ~32700 documents, silently returning no context. For a user who
can see every document (an active superuser), that filter never
narrows anything, so skip it and let the retriever search the whole
index instead.
* Minor improvements from a Claude review
* When a user is unrestricted chatting, still exclude trashed documents using a 'NOT IN' SQL statement. Wire that up where we need it
* Update src/paperless_ai/chat.py
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Trenton H
shamoon
Philipp Defner
dependabot[bot]andTrenton Holmes