AI Suggestions previously invented near-duplicate metadata because the classification
prompt had no knowledge of the installation's own taxonomy. This surfaces
a small, ranked, permission-filtered set of existing tags/document
types/correspondents/storage paths - drawn from the document's RAG
neighbors plus its own already-assigned metadata - so the model prefers
reusing what already exists.
The LLM response schema now returns existing_ids (IDs of reused
candidates) separately from new_names (genuinely new suggestions).
Only new_names goes through localization and fuzzy name-matching;
existing_ids is resolved deterministically and never touched by the
localization pass, so exact matches can no longer be silently
corrupted by translation.
* Feature: Allow configuring the compression type and compression levels during export
Building on the zip export improvements, this now allows users to further configure the
zip to fit their needs. A simple stored zip for speed, or a high compression zstd for
the smallest archive. Full validation of the method and levels at the command line
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Added a new --url argument to specify the base URL of the Paperless instance, allowing matched documents to be displayed as clickable links. Updated the logic to fetch document titles based on the presence of the base URL.
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks
* Updates the documentation to be correct about the respecting now
* merge conflict fixing
In tracemalloc based profiling, not materializing the whole Document list
reduced memory to approximately 20% of the baseline, with a peak memory
that scaled with the library size. Now, the lazt queryset is used and only
the needed pk value is actually contributing to memory
ProcessedMailViewSet.bulk_delete checked permissions inside the delete loop, so an unpermitted id returned 403 only after the mails ahead of it had already been deleted. Resolve the permitted set once via permitted_object_ids and reject before deleting anything, which also drops the per-mail permission queries.
* feat: add unified PermittedObjectsFilter backed by permitted_object_ids
* refactor: migrate all ViewSets to unified PermittedObjectsFilter
Replace the deprecated ObjectOwnedOrGrantedPermissionsFilter,
DocumentPermissionsFilter, and ObjectOwnedPermissionsFilter aliases
with PermittedObjectsFilter directly across documents/views.py (8
sites, including TrashView's include_granted=False subclass) and
paperless_mail/views.py (3 sites), then delete the now-unreferenced
alias classes from documents/filters.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* docs: document legacy status of get_objects_for_user_owner_aware/has_perms_owner_aware
Stage 4's PermittedObjectsFilter/permitted_object_ids() covers the
queryset-filtering use case, but both functions still have production
callers outside this plan's scope (documents/views.py,
documents/serialisers.py, documents/signals/handlers.py,
paperless_ai/matching.py, paperless_ai/ai_classifier.py). Per Task 20
Step 2, they are kept in place rather than partially deleted, with
docstrings updated to note their legacy status and remaining callers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* Fix: address final review findings for permission-filter unification
- Add a permanent regression test pinning TrashView's include_granted=False
wiring: an explicit view_document grant on a trashed document must not
leak it into /api/trash/ for a non-owner, non-superuser requester.
- Drop the now-dead direct dependency djangorestframework-guardian; the
last rest_framework_guardian import was removed by this branch's
migration onto PermittedObjectsFilter. django-guardian is untouched.
- Replace the hand-maintained, already-stale caller lists in
get_objects_for_user_owner_aware/has_perms_owner_aware docstrings with a
pointer to grep for remaining callers instead.
- In PermittedObjectsFilter.filter_queryset, compute `model` only on the
include_granted=True path that actually uses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* perf: check bulk-edit-objects apply_to_all permissions via DB-side exclude/exists
Materialized the full permitted_object_ids() set into a Python set() just
to check membership for the request's objs queryset -- the same pattern
already fixed at four other sites for Document. This one is used by
apply_to_all, where objs can be an unbounded filtered selection (e.g. all
tags matching a filter) rather than a small request-supplied ID list,
making the wasted materialization worse here than at the sites already
fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Cleans up the comment about why this is still here for now
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: migrate bulk-edit-objects apply_to_all dispatch to permitted_object_ids
Replaces get_objects_for_user_owner_aware/has_perms_owner_aware in the
BulkEditObjectsView apply_to_all dispatch (Tag/Correspondent/DocumentType/
StoragePath) with permitted_object_ids and the resolve-once,
check-membership pattern used elsewhere in this stage. Tag-descendant
expansion logic left untouched. Adds a security test pinning that
apply_to_all excludes objects the requester lacks object-level permission
on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
* test: add tag-descendant partial-permission coverage, verify pre-migration characterization
Adds TestBulkEditObjectsTagDescendantPartialPermission, exercising the
tag-descendant-expansion block in BulkEditObjectsView.post as a
non-superuser with object-level change_tag granted on a parent tag and
one of two children but not the other, confirming the expansion only
pulls in descendants the requester actually has permission on.
Verified both this test and the existing apply_to_all boundary test
pass unchanged against the pre-migration
get_objects_for_user_owner_aware/has_perms_owner_aware code (reverted
via a scratch patch of the prior commit's views.py hunk, then
restored), confirming they characterize genuine pre-existing behavior
rather than something the permitted_object_ids migration made
necessary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: migrate matching.py's 4 permission-filtered lookups to permitted_object_ids
* test: add matching.py permission coverage for correspondents, document types, storage paths
Completes the parametrized coverage started for tags -- proves all 4
matching.py lookups migrated to permitted_object_ids respect
per-object view permissions, not just the tag case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: generalize permitted_document_ids into permitted_object_ids for any model
Implements Task 14 of the permission-filtering consolidation plan:
- Add generic permitted_object_ids(user, model, perm, include_deleted=False)
- Refactor permitted_document_ids to delegate to permitted_object_ids
- Add comprehensive tests for Tag/Correspondent/DocumentType/StoragePath
- Preserve exact public behavior of permitted_document_ids (100% regression-free)
All 38 tests pass (18 existing + 20 new). The include_deleted parameter
correctly handles soft-delete patterns (effective only for Document).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor: add type hints to permitted_object_ids and permitted_document_ids
Add missing type annotations to match the established conventions in this file
(see get_objects_for_user_owner_aware). Also added Model import from django.db.models.
- permitted_object_ids: user: User | None, model: type[Model], return -> QuerySet[int]
- permitted_document_ids: user: User | None, return -> QuerySet[int]
All 38 permission filtering security tests pass; this is a type-annotation-only change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
* refactor: remove redundant deleted_at filter in permitted_object_ids
SoftDeleteManager's own get_queryset() already excludes soft-deleted
rows, so the extra deleted_at__isnull=True filter was dead code.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: resolve permitted_document_ids once before email/share loops
Replaces per-document has_perms_owner_aware calls in the email-document
action and bulk share-link-bundle creation with a single
permitted_document_ids(request.user) resolution before the loop,
reducing DB round-trips while preserving identical permission
semantics (including the per-document error message on the bundle
endpoint).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* perf: resolve permitted_document_ids(perm=change_document) once before bulk-edit loops
Migrates the bulk document edit permission check in views.py and the
custom-field DOCUMENTLINK validator in serialisers.py off of
has_perms_owner_aware-per-document loops, resolving
permitted_document_ids(user, perm="change_document") once instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* perf: resolve permitted_document_ids once for root-document version-listing loop
BulkDownloadView.post() previously called has_perms_owner_aware() per row
inside the loop that resolves each document's root and latest version.
Resolve permitted_document_ids(request.user) once before the loop and check
membership by root_doc.pk instead, consistent with the other consolidated
permission-filtering sites.
* test: use HTTPStatus enum instead of bare integers in security test assertions
* perf: resolve permitted_document_ids(perm=delete_document, include_deleted=True) once for trash loop
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* perf: drop unused select_related("owner") from email_documents
The permission check loop that used to call has_perms_owner_aware()
(which read .owner) was already replaced with permitted_document_ids()
resolved once into a set. No other code in email_documents touches
.owner, so the select_related is dead weight.
* test: repurpose inert grant into mixed-batch bulk-edit rejection case
The unrelated view_document grant in
test_bulk_edit_rejects_document_without_change_permission created a
document that was never referenced in the request payload. Turn it
into a genuinely useful case instead: a mixed batch containing one
document the requester is fully permitted to change alongside one
they are not, proving bulk_edit rejects the whole batch when any
document lacks change permission (not just checking the first/last
document in the list).
* test: add version-only-grant case discriminating root-vs-version permission check
The former "stranger" sub-case in
test_permission_checked_on_root_not_on_version had zero grants on either
root or version, so it passed under any implementation, correct or
buggy. Replace it with a user granted view_document on the version
itself (not the root): this only passes if bulk_download truly checks
root-only, catching a regression to "root OR version" that the old
case could never detect.
* perf: check permitted document IDs via DB-side exclude/exists instead of materializing the full set
email_documents, _has_document_permissions, TrashView.post, and
validate_documentlink_targets each resolved permitted_document_ids() into
a full Python set just to check membership for a small, bounded batch of
request document IDs. For a user with broad permitted access that pulls
their entire visible/editable document count into memory and across the
wire regardless of how many documents the request actually touches.
Pushing the membership check into the DB via exclude(...).exists() scales
with the request's batch size instead, without reintroducing the
per-row guardian join pathology from #13276 (confirmed via EXPLAIN
ANALYZE: the permission subplans are hashed once, not re-executed per
outer row).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add perm param to permitted_document_ids for change/delete checks
Widens permitted_document_ids(user, *, include_deleted=False) to
permitted_document_ids(user, *, perm="view_document", include_deleted=False)
so Stage 2 callers can check change_document/delete_document permissions
instead of the hardcoded view_document codename. Default is unchanged for
every existing call site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* test: strengthen delete-permission include_deleted test to discriminate from other grants
test_delete_permission_with_include_deleted_for_trash_restore only checked
an owner and a fully-ungranted stranger, so it never proved perm=
actually discriminates delete_document from other permission grants. Add
a view_only user with view_document (but not delete_document) granted on
the same doc and assert they remain excluded, mirroring the pattern in
test_change_document_permission_is_distinct_from_view.
* fix: normalize qualified permission strings in permitted_document_ids
Guardian's UserObjectPermission/GroupObjectPermission always store a bare
codename, but has_perm()-style callers commonly pass the qualified
"app_label.codename" form. Passing that qualified form here previously
matched zero rows, silently under-permitting. content_type already
disambiguates the codename, so just strip any prefix instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: migrate 5 single-call Document permission sites to permitted_document_ids
Swaps get_objects_for_user_owner_aware(user, "view_document", Document) for
Document.objects.filter(id__in=permitted_document_ids(user)) at 5 read-only,
single-call sites: AI chat "ask all documents", bulk-edit
_resolve_document_ids all:true branch, SelectionDataView permission check,
global search docs bucket, and the statistics endpoint's Document branch.
Confirmed all 3 callers of _resolve_document_ids always use the default
"view_document" codename before swapping. Added a regression test pinning
the AI-chat owner/permission boundary through the real API client, and
updated 2 existing mocked tests in test_views.py that asserted on
get_objects_for_user_owner_aware for the chat endpoint.
* perf: migrate 3 serialisers.py Document permission sites to permitted_document_ids
Migrates _get_viewable_duplicates(), PaperlessTaskSerializer.get_duplicate_documents(),
and the ShareLinkBundle document field queryset to use permitted_document_ids()
instead of get_objects_for_user_owner_aware()/get_objects_for_user(), consolidating
onto the shared permission-filtering helper. The is_staff gate in
get_duplicate_documents() is preserved as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* refactor: remove dead permission_codename param from _resolve_document_ids
The keyword param was unused in the method body since an earlier commit
switched it to call permitted_document_ids(user) internally. None of the
3 call sites passed it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: drop internal task-number reference from AI chat migration test docstring
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
shamoonandGitHub
GitHub Actions
Trenton HandGitHub
Max TruxaandGitHub
JaydenandGitHub
