* Preprocesses classifier content with Tantivy instead of NLTK
Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.
Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.
* Copies packages instead of hardlinking them in backend CI, some NLTK thing
* Adds a normalization to NFC to better fit what Tantivy expects
BulkEditSerializer's hand-parsed validators only caught the exceptions
their happy paths raised, so wrong-typed input produced a 500 or was
passed through to the task:
- owner: a nonexistent or wrong-typed id raised an uncaught error, and a
boolean was accepted (Django coerces True to pk 1). It is now validated
with PrimaryKeyRelatedField and the validated pk is passed on.
- rotate: null raised TypeError, while true, "90" and 45 were accepted and
failed later in QPDF. Degrees are now an IntegerField plus a
multiple-of-90 check, passing an int on. The dedicated rotate endpoint
and edit_pdf operations share the same multiple-of-90 check.
- split: null raised AttributeError, "0" silently became the last page,
"3-1" gave an empty group that crashed the task, and a range like
"1-5000000" was expanded into a list during the request with no upper
bound. Each range is now checked against 1 <= start <= end <= page_count
before it is built.
SetPermissionsSerializer was a bare DictField, and the legacy bulk_edit
set_permissions parameter bypassed even that by hand-calling
validate_set_permissions() on an unchecked dict. A bool or a non-list in
place of users/groups crashed with a raw TypeError instead of a 400.
It is now a nested serializer (view/change, each with users/groups as
lists of integers), used for owned-object create/update, the legacy
bulk_edit set_permissions parameter, and bulk_edit_objects permissions.
Unknown action keys are rejected: previously a typo like "veiw" was
silently dropped, leaving an empty permission set that could clear
existing grants. An explicit set_permissions null (an owner-only change)
remains a no-op, and an empty bulk_edit_objects permissions dict is still
rejected.
JSONField(binary=True) accepts any JSON value, so a truthy non-dict
(bool/int/list/string) silently passed validation.
Additionally, require string substitutes in barcode_tag_mapping, a non-string
substitute value made the barcode tag substitution fail inside
get_tag_ids, and the tag was silently skipped. barcode_tag_mapping values
must now be strings (empty still means "use the raw barcode").
operations was a ListField with no child, so both EditPdfDocumentsSerializer
and BulkEditSerializer._validate_parameters_edit_pdf hand-checked each entry
with isinstance(). That accepted booleans (isinstance(True, int) is true) and
passed through any extra keys.
Adds PdfEditOperationSerializer (page >= 1, doc >= 0, rotate a multiple of
90) and uses it as the ListField child on both paths, dropping the manual
type checks and the now-redundant negative/page < 1 bounds checks. Rotations
that are not a multiple of 90 previously passed validation and then failed
inside the task when QPDF refused them.
* docs(search): document the query grammar paperless actually supports
* docs(search): cover ?/bracket wildcards and the no-stemmer fallback
* Shorten things up, use tables and less wordy stuff. Fixes some test doc strings that referred to the documentation
* Silly sonarr
* Feature: parse advanced search with whoosh-compat and delete the hand-written translator
* Don't cover these, they exist for defensive, but there's no other current diagnostic
* A mix of more no cover and tests
* Route SCHEMA_FIELD_MISSING into an error, not an HTTP 400
* feat(search): add whoosh-compat, the shared field table and the field registry
* Sonar being useful actually
* Adds the given/when/then commenting
* Trims tests I don't think cover our logic or code or are redundant
* Coverage
* test(search): add pattern normalizer stem-alternates unit tests
* build: bump whoosh-compat to 0.2.0
* Perf: batch guardian permission assignment in bulk-edit
bulk_edit.set_permissions and BulkEditObjectPermissionsView both
looped documents/objects and called set_permissions_for_object per
object, which itself calls guardian's assign_perm/remove_perm once
per (object, user) pair -- ~10-20+ queries per object, scaling with
selection size.
Added set_permissions_for_objects, a bulk equivalent that resolves
existing permission holders once across the whole batch (not once per
object) and applies changes with a small, batch-size-independent
number of queries per action instead of one per (object, user) pair.
* Perf: avoid unnecessary full-row fetches in batch permission assignment
set_permissions_for_objects now takes a model + pks instead of instances,
and identity filtering resolves straight to ids, so bulk-editing
permissions no longer materializes full Document/User/Group rows just to
read their pk/id. Row construction for bulk_create is also chunked to
bound peak memory for very large "apply to all" operations.
* Fix: use .distinct() for existing-grant lookup, drop flaky query-count invariant tests
.distinct() lets the database dedupe identity ids server-side instead of
transferring one row per (object, grantee) match and deduping in Python --
was the dominant cost on a large selection with existing grants.
Also replaced the two query-count-equality tests (bulk_edit and the
bulk_edit_objects API path) with plain functional-correctness checks at
both batch sizes. Hopefully stops that flake.
* Mark empty-pks early-return in set_permissions_for_objects as no-cover
Defensive guard for an edge case (all requested pks already gone/invalid)
rather than a path normal usage exercises; matches the existing
pragma: no cover convention elsewhere in this file.
* Perf: drop speculative row-chunking in bulk permission assignment, keep the query-batching fix
* Resolve every permission action before applying any of them
This fixes the existing issue and resolves the Copilot comment
* Assert permission assignment does not scale with selection size
The two batching tests only checked that permissions came out correct at
5 and 50 objects, so reverting to the old per-object loop would still
have passed. Check sizes as well to prevent that
dependabot[bot]
GitHub Actions
shamoon
Trenton H