* perf: skip effective_content annotation on document list unless filtered on
DocumentViewSet.get_queryset() always attached a correlated subquery
resolving each document's latest version content, even though it's only
needed for the deprecated search/title_content/content__* filter params.
Evaluated for every candidate row before pagination's LIMIT, this is
pathological on MariaDB: its default cardinality estimate for the mostly-
NULL root_document_id self-join drives it to a near-full-table scan per
row instead of using the FK index, turning a normal filtered list request
into a multi-second query (root cause of paperless-ngx#13778's report).
Only attach the annotation when a request actually filters on it. The
common case now relies on Document.get_effective_content()'s existing
prefetch-based fallback instead (extended the "versions" prefetch to
include content), which DocumentSerializer.to_representation() now calls
directly instead of checking for the annotation via hasattr().
* fix: address review feedback on effective_content annotation skip
- _needs_effective_content_annotation() now checks for a non-blank,
stripped param value rather than mere key presence, matching how
SearchFilter/TitleContentFilter/EffectiveContentFilter themselves
no-op on a blank value. An empty ?search= or a saved view with a
cleared text filter no longer re-triggers the annotation.
- The "versions" prefetch on DocumentViewSet no longer carries content
for every historical version of every document -- that's unused
bloat for version-heavy documents. Added
latest_version_content_prefetch() (versioning.py), a separate,
windowed prefetch scoped to just the newest version's content per
root, and taught Document.get_effective_content() to check it first.
- DocumentSerializer.to_representation() no longer unconditionally
calls get_effective_content(). Added has_prefetched_effective_content()
(versioning.py) as a cheap upfront check: only resolve version-aware
content when an SQL annotation or a versions prefetch is already on
the instance. TrashView and GlobalSearchView build their own
querysets independently of DocumentViewSet and never display
document content at all (checked both frontend components), so they
now keep showing the document's own, unresolved content with zero
extra queries -- the same behavior as before effective_content
resolution existed, just generalized past the narrow hasattr() check
it replaced.
* Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it
* Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even
* CLean up the new test with the docstrings, handle the fields in one place
* Fun with contenttype and caching. Compare only the
queries spent on the documents themselves or else
Django's condition() decorator invokes etag_func and last_modified_func
separately, and the view itself may resolve again -- each call to
resolve_effective_document_by_pk() was redoing the same root/version
DB lookups. Memoize the resolution on the request object so a single
thumb/metadata/preview request resolves the effective document once
instead of up to three times.
Related to paperless-ngx/paperless-ngx#13161.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Trenton H
shamoon
