tags/type/correspondent/storage_path per object
- existing_ids the model returns are now restricted to ids that were
actually offered as candidates in the prompt
- ai_suggestions no longer skips permission filtering on a cache hit.
The cache now stores the raw existing_ids/new_names choices rather than
resolved object ids
- ai_suggestions deduplicates matched objects by id
- Performance updates for keeping things into a queryset instead
AI Suggestions previously invented near-duplicate metadata because the classification
prompt had no knowledge of the installation's own taxonomy. This surfaces
a small, ranked, permission-filtered set of existing tags/document
types/correspondents/storage paths - drawn from the document's RAG
neighbors plus its own already-assigned metadata - so the model prefers
reusing what already exists.
The LLM response schema now returns existing_ids (IDs of reused
candidates) separately from new_names (genuinely new suggestions).
Only new_names goes through localization and fuzzy name-matching;
existing_ids is resolved deterministically and never touched by the
localization pass, so exact matches can no longer be silently
corrupted by translation.
* feat: add unified PermittedObjectsFilter backed by permitted_object_ids
* refactor: migrate all ViewSets to unified PermittedObjectsFilter
Replace the deprecated ObjectOwnedOrGrantedPermissionsFilter,
DocumentPermissionsFilter, and ObjectOwnedPermissionsFilter aliases
with PermittedObjectsFilter directly across documents/views.py (8
sites, including TrashView's include_granted=False subclass) and
paperless_mail/views.py (3 sites), then delete the now-unreferenced
alias classes from documents/filters.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* docs: document legacy status of get_objects_for_user_owner_aware/has_perms_owner_aware
Stage 4's PermittedObjectsFilter/permitted_object_ids() covers the
queryset-filtering use case, but both functions still have production
callers outside this plan's scope (documents/views.py,
documents/serialisers.py, documents/signals/handlers.py,
paperless_ai/matching.py, paperless_ai/ai_classifier.py). Per Task 20
Step 2, they are kept in place rather than partially deleted, with
docstrings updated to note their legacy status and remaining callers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* Fix: address final review findings for permission-filter unification
- Add a permanent regression test pinning TrashView's include_granted=False
wiring: an explicit view_document grant on a trashed document must not
leak it into /api/trash/ for a non-owner, non-superuser requester.
- Drop the now-dead direct dependency djangorestframework-guardian; the
last rest_framework_guardian import was removed by this branch's
migration onto PermittedObjectsFilter. django-guardian is untouched.
- Replace the hand-maintained, already-stale caller lists in
get_objects_for_user_owner_aware/has_perms_owner_aware docstrings with a
pointer to grep for remaining callers instead.
- In PermittedObjectsFilter.filter_queryset, compute `model` only on the
include_granted=True path that actually uses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc
* perf: check bulk-edit-objects apply_to_all permissions via DB-side exclude/exists
Materialized the full permitted_object_ids() set into a Python set() just
to check membership for the request's objs queryset -- the same pattern
already fixed at four other sites for Document. This one is used by
apply_to_all, where objs can be an unbounded filtered selection (e.g. all
tags matching a filter) rather than a small request-supplied ID list,
making the wasted materialization worse here than at the sites already
fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Cleans up the comment about why this is still here for now
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: generalize permitted_document_ids into permitted_object_ids for any model
Implements Task 14 of the permission-filtering consolidation plan:
- Add generic permitted_object_ids(user, model, perm, include_deleted=False)
- Refactor permitted_document_ids to delegate to permitted_object_ids
- Add comprehensive tests for Tag/Correspondent/DocumentType/StoragePath
- Preserve exact public behavior of permitted_document_ids (100% regression-free)
All 38 tests pass (18 existing + 20 new). The include_deleted parameter
correctly handles soft-delete patterns (effective only for Document).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor: add type hints to permitted_object_ids and permitted_document_ids
Add missing type annotations to match the established conventions in this file
(see get_objects_for_user_owner_aware). Also added Model import from django.db.models.
- permitted_object_ids: user: User | None, model: type[Model], return -> QuerySet[int]
- permitted_document_ids: user: User | None, return -> QuerySet[int]
All 38 permission filtering security tests pass; this is a type-annotation-only change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UmMBGW9FKyDgmKRJ5H9rif
* refactor: remove redundant deleted_at filter in permitted_object_ids
SoftDeleteManager's own get_queryset() already excludes soft-deleted
rows, so the extra deleted_at__isnull=True filter was dead code.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add perm param to permitted_document_ids for change/delete checks
Widens permitted_document_ids(user, *, include_deleted=False) to
permitted_document_ids(user, *, perm="view_document", include_deleted=False)
so Stage 2 callers can check change_document/delete_document permissions
instead of the hardcoded view_document codename. Default is unchanged for
every existing call site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* test: strengthen delete-permission include_deleted test to discriminate from other grants
test_delete_permission_with_include_deleted_for_trash_restore only checked
an owner and a fully-ungranted stranger, so it never proved perm=
actually discriminates delete_document from other permission grants. Add
a view_only user with view_document (but not delete_document) granted on
the same doc and assert they remain excluded, mirroring the pattern in
test_change_document_permission_is_distinct_from_view.
* fix: normalize qualified permission strings in permitted_document_ids
Guardian's UserObjectPermission/GroupObjectPermission always store a bare
codename, but has_perm()-style callers commonly pass the qualified
"app_label.codename" form. Passing that qualified form here previously
matched zero rows, silently under-permitting. content_type already
disambiguates the codename, so just strip any prefix instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add include_deleted param to permitted_document_ids
Widens permitted_document_ids to accept an include_deleted keyword-only
flag (default False, preserving current behavior) so later call sites
that need visibility into soft-deleted documents (e.g. trash restore)
can reuse this permission check instead of duplicating it.
* refactor: remove redundant deleted_at filter in permitted_document_ids
Document.objects already applies filter(deleted_at__isnull=True) internally
via SoftDeleteManager.get_queryset(), so the conditional filter was redundant.
Simplify to just use manager.all() in both branches — manager selection alone
ensures correct behavior (Document.objects excludes deleted, Document.global_objects
includes all).
Co-Authored-By: Claude Haiku <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session
---------
Co-authored-by: Claude Haiku <noreply@anthropic.com>
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
* Fix: selection_data re-derives the filtered document set 5 times over
_get_selection_data_for_queryset() (powers ?include_selection_data=true on
the document list and search endpoints) computed document_count for
Correspondent/DocumentType/StoragePath/Tag/CustomField by embedding the
caller's full filtered queryset -- filters plus the permission check -- as
a subquery inside 5 separate Count(filter=Q(documents__in=queryset), ...)
calls. Each one re-evaluates that whole queryset from scratch.
Resolve the document ids once into a concrete list and reuse it across all
five annotations instead. For Tag/CustomField specifically (M2M via a
through-model table), also route through annotate_document_count_by_ids()
-- extracted from the tag/custom-field document_count fix (#13203) -- to
avoid the same Count(filter=Q(id__in=...), distinct=True)-on-an-M2M-relation
anti-pattern diagnosed there.
On a 400k-document/1,000-tag corpus, the correspondents portion alone
previously didn't finish within several minutes (killed twice while
investigating, including one run that left a zombie query still consuming
a CPU 30+ minutes later). All five queries together now complete in
~30-35s. Root-caused from a real report (paperless-ngx#13201) via a
different, already-fixed query (#13205) -- this one hasn't been reported
in the wild yet, found by auditing the same call path.
Depends on #13203 for annotate_document_count_by_ids().
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Address review feedback: drop unnecessary ordering before collecting ids
queryset passed into _get_selection_data_for_queryset() carries the
default/user-specified ordering, which is irrelevant once we're only
collecting a flat id list. Clearing it removes a pointless sort.
No measurable change in benchmarking at 400k documents -- the id-list
materialization/IN-clause cost still dominates -- but it's a free,
strictly-correct cleanup, not just noise-neutral in the other direction.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix (beta): tag/custom-field document_count scales badly with tag count
TagViewSet and CustomFieldViewSet's document_count annotation used a
per-row correlated subquery (annotate_document_count_for_related_queryset),
executed once per tag/custom-field row. Fine at a few dozen rows, but at
~1,000 tags it degrades to a full per-tag GroupAggregate over the tags M2M
table -- 6s+ in production reports, confirmed via EXPLAIN (loops=1000).
Replaced with a single, independent GROUP BY over the through table
(permission filter expressed as a plain WHERE, not an aggregate FILTER),
then injected via Case/When. Also tried a more "obvious" fix -- a plain
Count(filter=Q(id__in=permitted_ids), distinct=True) directly on the M2M
relation -- but that's worse: Postgres fails to plan the id__in check as a
semi-join once a large M2M bridge table is involved, and instead re-checks
subquery membership once per joined row.
Benchmarked against a synthetic corpus (400k documents, 1,000 tags @ ~5/doc,
matching real-world reports in discussion #13161): tag list wall-clock drops
from ~180s to ~8s for a permission-restricted user, ~21s to ~8s for a
superuser. No measurable change at typical home-instance scale (tens of
tags).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Address review feedback: restore validation, scope aggregation to queryset
- Restore the document_count_source_field validation helper dropped during
the refactor -- misconfiguring a viewset (document_count_through set
without document_count_source_field) now fails fast with a clear error
again instead of an obscure runtime failure.
- Restrict annotate_document_count_for_related_queryset()'s through-table
aggregation to rows whose related_object_field is one of the annotated
queryset's pks. No-op for the main tag-list call site (queryset is all
tags), but avoids unnecessary work for narrower callers like the tag
descendants branch in TagViewSet.list().
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
stumpylog
Sebastian SteinbeißerandGitHub