Performance: skip effective_content annotation on document list unless required (#13789)

* perf: skip effective_content annotation on document list unless filtered on

DocumentViewSet.get_queryset() always attached a correlated subquery
resolving each document's latest version content, even though it's only
needed for the deprecated search/title_content/content__* filter params.
Evaluated for every candidate row before pagination's LIMIT, this is
pathological on MariaDB: its default cardinality estimate for the mostly-
NULL root_document_id self-join drives it to a near-full-table scan per
row instead of using the FK index, turning a normal filtered list request
into a multi-second query (root cause of paperless-ngx#13778's report).

Only attach the annotation when a request actually filters on it. The
common case now relies on Document.get_effective_content()'s existing
prefetch-based fallback instead (extended the "versions" prefetch to
include content), which DocumentSerializer.to_representation() now calls
directly instead of checking for the annotation via hasattr().

* fix: address review feedback on effective_content annotation skip

- _needs_effective_content_annotation() now checks for a non-blank,
  stripped param value rather than mere key presence, matching how
  SearchFilter/TitleContentFilter/EffectiveContentFilter themselves
  no-op on a blank value. An empty ?search= or a saved view with a
  cleared text filter no longer re-triggers the annotation.

- The "versions" prefetch on DocumentViewSet no longer carries content
  for every historical version of every document -- that's unused
  bloat for version-heavy documents. Added
  latest_version_content_prefetch() (versioning.py), a separate,
  windowed prefetch scoped to just the newest version's content per
  root, and taught Document.get_effective_content() to check it first.

- DocumentSerializer.to_representation() no longer unconditionally
  calls get_effective_content(). Added has_prefetched_effective_content()
  (versioning.py) as a cheap upfront check: only resolve version-aware
  content when an SQL annotation or a versions prefetch is already on
  the instance. TrashView and GlobalSearchView build their own
  querysets independently of DocumentViewSet and never display
  document content at all (checked both frontend components), so they
  now keep showing the document's own, unresolved content with zero
  extra queries -- the same behavior as before effective_content
  resolution existed, just generalized past the narrow hasattr() check
  it replaced.

* Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it

* Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even

* CLean up the new test with the docstrings, handle the fields in one place

* Fun with contenttype and caching. Compare only the
queries spent on the documents themselves or else
This commit is contained in:
Trenton H
2026-09-10 12:38:41 -07:00
committed by GitHub
parent 8593f84cae
commit d5f9605daf
5 changed files with 628 additions and 33 deletions
+14
View File
@@ -375,6 +375,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
If the queryset already annotated ``effective_content``, that value is used.
"""
# Here to avoid circular import
from documents.versioning import LATEST_VERSION_CONTENT_PREFETCH_ATTR
from documents.versioning import sort_versions_newest_first
from documents.versioning import versions_newest_first
@@ -384,6 +385,19 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
if self.root_document_id is not None or self.pk is None:
return self.content
latest_version_prefetch = getattr(
self,
LATEST_VERSION_CONTENT_PREFETCH_ATTR,
None,
)
if latest_version_prefetch is not None:
# Empty list means prefetch ran and found no versions — use own content.
return (
latest_version_prefetch[0].content
if latest_version_prefetch
else self.content
)
prefetched_cache = getattr(self, "_prefetched_objects_cache", None)
prefetched_versions = (
prefetched_cache.get("versions")