mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-09-11 04:08:00 +00:00
Performance: skip effective_content annotation on document list unless required (#13789)
* perf: skip effective_content annotation on document list unless filtered on DocumentViewSet.get_queryset() always attached a correlated subquery resolving each document's latest version content, even though it's only needed for the deprecated search/title_content/content__* filter params. Evaluated for every candidate row before pagination's LIMIT, this is pathological on MariaDB: its default cardinality estimate for the mostly- NULL root_document_id self-join drives it to a near-full-table scan per row instead of using the FK index, turning a normal filtered list request into a multi-second query (root cause of paperless-ngx#13778's report). Only attach the annotation when a request actually filters on it. The common case now relies on Document.get_effective_content()'s existing prefetch-based fallback instead (extended the "versions" prefetch to include content), which DocumentSerializer.to_representation() now calls directly instead of checking for the annotation via hasattr(). * fix: address review feedback on effective_content annotation skip - _needs_effective_content_annotation() now checks for a non-blank, stripped param value rather than mere key presence, matching how SearchFilter/TitleContentFilter/EffectiveContentFilter themselves no-op on a blank value. An empty ?search= or a saved view with a cleared text filter no longer re-triggers the annotation. - The "versions" prefetch on DocumentViewSet no longer carries content for every historical version of every document -- that's unused bloat for version-heavy documents. Added latest_version_content_prefetch() (versioning.py), a separate, windowed prefetch scoped to just the newest version's content per root, and taught Document.get_effective_content() to check it first. - DocumentSerializer.to_representation() no longer unconditionally calls get_effective_content(). Added has_prefetched_effective_content() (versioning.py) as a cheap upfront check: only resolve version-aware content when an SQL annotation or a versions prefetch is already on the instance. TrashView and GlobalSearchView build their own querysets independently of DocumentViewSet and never display document content at all (checked both frontend components), so they now keep showing the document's own, unresolved content with zero extra queries -- the same behavior as before effective_content resolution existed, just generalized past the narrow hasattr() check it replaced. * Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it * Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even * CLean up the new test with the docstrings, handle the fields in one place * Fun with contenttype and caching. Compare only the queries spent on the documents themselves or else
This commit is contained in:
@@ -375,6 +375,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
|
||||
If the queryset already annotated ``effective_content``, that value is used.
|
||||
"""
|
||||
# Here to avoid circular import
|
||||
from documents.versioning import LATEST_VERSION_CONTENT_PREFETCH_ATTR
|
||||
from documents.versioning import sort_versions_newest_first
|
||||
from documents.versioning import versions_newest_first
|
||||
|
||||
@@ -384,6 +385,19 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
|
||||
if self.root_document_id is not None or self.pk is None:
|
||||
return self.content
|
||||
|
||||
latest_version_prefetch = getattr(
|
||||
self,
|
||||
LATEST_VERSION_CONTENT_PREFETCH_ATTR,
|
||||
None,
|
||||
)
|
||||
if latest_version_prefetch is not None:
|
||||
# Empty list means prefetch ran and found no versions — use own content.
|
||||
return (
|
||||
latest_version_prefetch[0].content
|
||||
if latest_version_prefetch
|
||||
else self.content
|
||||
)
|
||||
|
||||
prefetched_cache = getattr(self, "_prefetched_objects_cache", None)
|
||||
prefetched_versions = (
|
||||
prefetched_cache.get("versions")
|
||||
|
||||
Reference in New Issue
Block a user