Adds the extensibility design that hoists the bulk-edit operation
definition (today smeared across serialisers.py, views.py, bulk_edit.py
and keyed three different ways) behind a BulkEditOperation registry +
PermissionRequirements value object, with per-operation OpenAPI examples.
Contract-preserving refactor; both docs reviewed across multiple passes
(permission matrix verified, both view call sites accounted for).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the search-error-shapes stub with a full design spec and a TDD
implementation plan for friendlier advanced-search error messages.
Empirically validated against a live Tantivy index: three error families
(UnknownFieldError, InvalidFieldValueError, MalformedQueryError),
proactive numeric validation plus a parse_query backstop, comparison
operators confirmed working, and a parse-based field drift guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DocumentViewSet.get_queryset() carried a blanket .distinct() left over from
an older query shape (predating the version-file feature and the
correlated-subquery rewrite of num_notes/effective_content). Nothing in the
current base queryset can produce duplicate document rows: select_related
is all FK-to-PK, and the permission filter is a boolean id__in predicate,
not a join. id (the PK) is also always selected and inherently unique, so
.distinct() was a structural no-op for correctness.
It was not a no-op for cost. EXPLAIN showed it forcing a full sort-then-
dedupe over the entire permission-visible document set before LIMIT could
apply -- 340k rows for a 25-row page, with the effective_content/num_notes
correlated subqueries re-executed once per row as a result. This was the
dominant cost behind the original report's document overview slowness,
well beyond the get_user_can_change N+1 fixed separately.
Removing it isn't safe on its own, though -- auditing found two existing
filters that rely on an M2M join *without* deduping themselves, previously
papered over by the blanket .distinct():
- InboxFilter: a document with two tags both flagged is_inbox_tag=True
(nothing prevents that) would be returned twice for ?is_in_inbox=true.
- CustomFieldsFilter: a document with multiple custom field instances
matching different OR-ed branches would be returned once per match.
Fixed both locally, matching the pattern ObjectFilter already used for
tags__id__in/custom_fields__id__in. Added regression tests for both --
confirmed they fail without the local .distinct() calls.
Benchmarked against the same 400k-document/1,000-tag corpus: document list
wall-clock for a permission-restricted user drops from ~15.8s to ~2.25s
(~7x), closing most of the previous gap to a superuser's ~1.8s.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix (beta): tag/custom-field document_count scales badly with tag count
TagViewSet and CustomFieldViewSet's document_count annotation used a
per-row correlated subquery (annotate_document_count_for_related_queryset),
executed once per tag/custom-field row. Fine at a few dozen rows, but at
~1,000 tags it degrades to a full per-tag GroupAggregate over the tags M2M
table -- 6s+ in production reports, confirmed via EXPLAIN (loops=1000).
Replaced with a single, independent GROUP BY over the through table
(permission filter expressed as a plain WHERE, not an aggregate FILTER),
then injected via Case/When. Also tried a more "obvious" fix -- a plain
Count(filter=Q(id__in=permitted_ids), distinct=True) directly on the M2M
relation -- but that's worse: Postgres fails to plan the id__in check as a
semi-join once a large M2M bridge table is involved, and instead re-checks
subquery membership once per joined row.
Benchmarked against a synthetic corpus (400k documents, 1,000 tags @ ~5/doc,
matching real-world reports in discussion #13161): tag list wall-clock drops
from ~180s to ~8s for a permission-restricted user, ~21s to ~8s for a
superuser. No measurable change at typical home-instance scale (tens of
tags).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Address review feedback: restore validation, scope aggregation to queryset
- Restore the document_count_source_field validation helper dropped during
the refactor -- misconfiguring a viewset (document_count_through set
without document_count_source_field) now fails fast with a clear error
again instead of an obscure runtime failure.
- Restrict annotate_document_count_for_related_queryset()'s through-table
aggregation to rows whose related_object_field is one of the annotated
queryset's pks. No-op for the main tag-list call site (queryset is all
tags), but avoids unnecessary work for narrower callers like the tag
descendants branch in TagViewSet.list().
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix (beta): batch document user_can_change checks to avoid per-row N+1
DocumentSerializer.get_user_can_change() built a fresh
ObjectPermissionChecker and issued a guardian permission-table query for
every document row not owned by the requesting user -- correct, but O(N)
per page load for any non-superuser viewing documents owned by others.
BulkPermissionMixin already batches this exact lookup (2 queries total,
regardless of page size) for Correspondent/Tag/DocumentType/CustomField,
but was gated behind the rarely-used `full_perms` flag and DocumentViewSet
didn't inherit it at all. Changed the gate to "any list action" (cheap:
just two extra queries per page) and added BulkPermissionMixin to
DocumentViewSet, then updated get_user_can_change to consult that batched
context before falling back to a fresh guardian check.
Preserves guardian's own superuser shortcut explicitly (has_perm() special-
cases is_superuser without a query; the batched-context path doesn't, so it
needed its own check) -- covered by a new regression test, since no
existing test exercised a superuser viewing an other-owned document.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix (beta): don't batch permissions for tantivy search results
UnifiedSearchViewSet.list() returns SearchHit/dict-like objects for
text/title/query/more_like_id search requests, not Document ORM instances.
Adding BulkPermissionMixin to DocumentViewSet (previous commit) meant its
get_serializer_context() ran for search responses too, and its
_get_object_perms() -- which expects real model instances with .pk --
crashed on the dict-like hits with AttributeError, turning every search
request into a 400.
Skip the batching specifically for search requests (existing
_is_search_request() check) by calling past BulkPermissionMixin in the
MRO; non-search list() calls (which return a real Document queryset) are
unaffected and still get the batching.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
stumpylog
shamoonandGitHub
FabianandGitHub
GitHub Actions
