* fix(search): resolve index-write permissions and effective content in bulk
Add WriteBatch.add_or_update_ids() and use it in bulk_update_documents
and trash restore, cutting index writes from ~8 queries per document
to a constant handful per batch
* Always these new ones with xdist, try a better condition
* Fix: skip vector store document id filter for unrestricted chat users
ChatStreamingView built an IN filter from every permitted document id
for the "chat over all documents" case, which exceeds the vector
store's SQLite bound-parameter safety limit on installs with more
than ~32700 documents, silently returning no context. For a user who
can see every document (an active superuser), that filter never
narrows anything, so skip it and let the retriever search the whole
index instead.
* Minor improvements from a Claude review
* When a user is unrestricted chatting, still exclude trashed documents using a 'NOT IN' SQL statement. Wire that up where we need it
* Update src/paperless_ai/chat.py
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Security: validate remote OCR endpoint against internal SSRF
Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.
* Validates te outbound url again right before use
* cover empty-value branch of validate_remote_ocr_endpoint because coverage
* re-validate remote OCR endpoint on every outbound request
* Ok! Backend stuff for the remote ocr workflow
* Frotnend workflow stuff
* And docs
* Fix dynamic action fields thing
* Actually, fix the action dropdown thing
* Fix this validation thing, and we have to check existing actions
* Fix migration
* uses_remote_service + allow_remote to allow opt-in / out of remote OCR
* Add to parser dev docs
* remote_ocr_mode config setting
* Checks for remote_ocr_mode and fix import
* Update config.component.spec.ts
* More tests for remote_ocr_mode
* Docs for remote_ocr_mode
* Ok, wire up the remote_ocr_mode with allow_remote for consumer
* Update consumer.py
* Format remote OCR mode check tests
* Use get_choice_from_env
* Backend changes and migration for remote OCR Config
* Backend tests
* Frontend stuff, with sections
* Docs
* Update test_tesseract_parser.py
* Actually we cant use this any more, in case settings are in app config
* Dont mark entire test file for db, use a mock for empty engine settings
AI Suggestions previously invented near-duplicate metadata because the classification
prompt had no knowledge of the installation's own taxonomy. This surfaces
a small, ranked, permission-filtered set of existing tags/document
types/correspondents/storage paths - drawn from the document's RAG
neighbors plus its own already-assigned metadata - so the model prefers
reusing what already exists.
The LLM response schema now returns existing_ids (IDs of reused
candidates) separately from new_names (genuinely new suggestions).
Only new_names goes through localization and fuzzy name-matching;
existing_ids is resolved deterministically and never touched by the
localization pass, so exact matches can no longer be silently
corrupted by translation.
* Feature: Allow configuring the compression type and compression levels during export
Building on the zip export improvements, this now allows users to further configure the
zip to fit their needs. A simple stored zip for speed, or a high compression zstd for
the smallest archive. Full validation of the method and levels at the command line
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Added a new --url argument to specify the base URL of the Paperless instance, allowing matched documents to be displayed as clickable links. Updated the logic to fetch document titles based on the presence of the base URL.
In tracemalloc based profiling, not materializing the whole Document list
reduced memory to approximately 20% of the baseline, with a peak memory
that scaled with the library size. Now, the lazt queryset is used and only
the needed pk value is actually contributing to memory
* feat: add unified PermittedObjectsFilter backed by permitted_object_ids
* refactor: migrate all ViewSets to unified PermittedObjectsFilter
Replace the deprecated ObjectOwnedOrGrantedPermissionsFilter,
DocumentPermissionsFilter, and ObjectOwnedPermissionsFilter aliases
with PermittedObjectsFilter directly across documents/views.py (8
sites, including TrashView's include_granted=False subclass) and
paperless_mail/views.py (3 sites), then delete the now-unreferenced
alias classes from documents/filters.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* docs: document legacy status of get_objects_for_user_owner_aware/has_perms_owner_aware
Stage 4's PermittedObjectsFilter/permitted_object_ids() covers the
queryset-filtering use case, but both functions still have production
callers outside this plan's scope (documents/views.py,
documents/serialisers.py, documents/signals/handlers.py,
paperless_ai/matching.py, paperless_ai/ai_classifier.py). Per Task 20
Step 2, they are kept in place rather than partially deleted, with
docstrings updated to note their legacy status and remaining callers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* Fix: address final review findings for permission-filter unification
- Add a permanent regression test pinning TrashView's include_granted=False
wiring: an explicit view_document grant on a trashed document must not
leak it into /api/trash/ for a non-owner, non-superuser requester.
- Drop the now-dead direct dependency djangorestframework-guardian; the
last rest_framework_guardian import was removed by this branch's
migration onto PermittedObjectsFilter. django-guardian is untouched.
- Replace the hand-maintained, already-stale caller lists in
get_objects_for_user_owner_aware/has_perms_owner_aware docstrings with a
pointer to grep for remaining callers instead.
- In PermittedObjectsFilter.filter_queryset, compute `model` only on the
include_granted=True path that actually uses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* perf: check bulk-edit-objects apply_to_all permissions via DB-side exclude/exists
Materialized the full permitted_object_ids() set into a Python set() just
to check membership for the request's objs queryset -- the same pattern
already fixed at four other sites for Document. This one is used by
apply_to_all, where objs can be an unbounded filtered selection (e.g. all
tags matching a filter) rather than a small request-supplied ID list,
making the wasted materialization worse here than at the sites already
fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Cleans up the comment about why this is still here for now
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: migrate bulk-edit-objects apply_to_all dispatch to permitted_object_ids
Replaces get_objects_for_user_owner_aware/has_perms_owner_aware in the
BulkEditObjectsView apply_to_all dispatch (Tag/Correspondent/DocumentType/
StoragePath) with permitted_object_ids and the resolve-once,
check-membership pattern used elsewhere in this stage. Tag-descendant
expansion logic left untouched. Adds a security test pinning that
apply_to_all excludes objects the requester lacks object-level permission
on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
* test: add tag-descendant partial-permission coverage, verify pre-migration characterization
Adds TestBulkEditObjectsTagDescendantPartialPermission, exercising the
tag-descendant-expansion block in BulkEditObjectsView.post as a
non-superuser with object-level change_tag granted on a parent tag and
one of two children but not the other, confirming the expansion only
pulls in descendants the requester actually has permission on.
Verified both this test and the existing apply_to_all boundary test
pass unchanged against the pre-migration
get_objects_for_user_owner_aware/has_perms_owner_aware code (reverted
via a scratch patch of the prior commit's views.py hunk, then
restored), confirming they characterize genuine pre-existing behavior
rather than something the permitted_object_ids migration made
necessary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: migrate matching.py's 4 permission-filtered lookups to permitted_object_ids
* test: add matching.py permission coverage for correspondents, document types, storage paths
Completes the parametrized coverage started for tags -- proves all 4
matching.py lookups migrated to permitted_object_ids respect
per-object view permissions, not just the tag case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: generalize permitted_document_ids into permitted_object_ids for any model
Implements Task 14 of the permission-filtering consolidation plan:
- Add generic permitted_object_ids(user, model, perm, include_deleted=False)
- Refactor permitted_document_ids to delegate to permitted_object_ids
- Add comprehensive tests for Tag/Correspondent/DocumentType/StoragePath
- Preserve exact public behavior of permitted_document_ids (100% regression-free)
All 38 tests pass (18 existing + 20 new). The include_deleted parameter
correctly handles soft-delete patterns (effective only for Document).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor: add type hints to permitted_object_ids and permitted_document_ids
Add missing type annotations to match the established conventions in this file
(see get_objects_for_user_owner_aware). Also added Model import from django.db.models.
- permitted_object_ids: user: User | None, model: type[Model], return -> QuerySet[int]
- permitted_document_ids: user: User | None, return -> QuerySet[int]
All 38 permission filtering security tests pass; this is a type-annotation-only change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
* refactor: remove redundant deleted_at filter in permitted_object_ids
SoftDeleteManager's own get_queryset() already excludes soft-deleted
rows, so the extra deleted_at__isnull=True filter was dead code.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
shamoon
Trenton H
dependabot[bot]andTrenton Holmes
Max Truxa
Jayden
