mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-09-11 20:28:01 +00:00
Performance: skip effective_content annotation on document list unless required (#13789)
* perf: skip effective_content annotation on document list unless filtered on DocumentViewSet.get_queryset() always attached a correlated subquery resolving each document's latest version content, even though it's only needed for the deprecated search/title_content/content__* filter params. Evaluated for every candidate row before pagination's LIMIT, this is pathological on MariaDB: its default cardinality estimate for the mostly- NULL root_document_id self-join drives it to a near-full-table scan per row instead of using the FK index, turning a normal filtered list request into a multi-second query (root cause of paperless-ngx#13778's report). Only attach the annotation when a request actually filters on it. The common case now relies on Document.get_effective_content()'s existing prefetch-based fallback instead (extended the "versions" prefetch to include content), which DocumentSerializer.to_representation() now calls directly instead of checking for the annotation via hasattr(). * fix: address review feedback on effective_content annotation skip - _needs_effective_content_annotation() now checks for a non-blank, stripped param value rather than mere key presence, matching how SearchFilter/TitleContentFilter/EffectiveContentFilter themselves no-op on a blank value. An empty ?search= or a saved view with a cleared text filter no longer re-triggers the annotation. - The "versions" prefetch on DocumentViewSet no longer carries content for every historical version of every document -- that's unused bloat for version-heavy documents. Added latest_version_content_prefetch() (versioning.py), a separate, windowed prefetch scoped to just the newest version's content per root, and taught Document.get_effective_content() to check it first. - DocumentSerializer.to_representation() no longer unconditionally calls get_effective_content(). Added has_prefetched_effective_content() (versioning.py) as a cheap upfront check: only resolve version-aware content when an SQL annotation or a versions prefetch is already on the instance. TrashView and GlobalSearchView build their own querysets independently of DocumentViewSet and never display document content at all (checked both frontend components), so they now keep showing the document's own, unresolved content with zero extra queries -- the same behavior as before effective_content resolution existed, just generalized past the narrow hasattr() check it replaced. * Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it * Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even * CLean up the new test with the docstrings, handle the fields in one place * Fun with contenttype and caching. Compare only the queries spent on the documents themselves or else
This commit is contained in:
@@ -89,6 +89,7 @@ from documents.templating.utils import convert_format_str_to_template_format
|
||||
from documents.templating.workflows import validate_workflow_template
|
||||
from documents.validators import uri_validator
|
||||
from documents.validators import url_validator
|
||||
from documents.versioning import has_prefetched_effective_content
|
||||
from documents.versioning import sort_versions_newest_first
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -1152,8 +1153,14 @@ class DocumentSerializer(
|
||||
|
||||
def to_representation(self, instance):
|
||||
doc = super().to_representation(instance)
|
||||
if "content" in self.fields and hasattr(instance, "effective_content"):
|
||||
doc["content"] = getattr(instance, "effective_content") or ""
|
||||
if "content" in self.fields and has_prefetched_effective_content(instance):
|
||||
# Only resolve version-aware content when it's cheap: an SQL
|
||||
# annotation or a versions prefetch is already on the instance.
|
||||
# A caller that set up neither (e.g. TrashView, GlobalSearchView,
|
||||
# which build their own querysets) gets the document's own,
|
||||
# unresolved content instead of paying for an extra per-instance
|
||||
# query -- same as before effective_content resolution existed.
|
||||
doc["content"] = instance.get_effective_content() or ""
|
||||
if self.truncate_content and "content" in self.fields:
|
||||
doc["content"] = doc.get("content")[0:550]
|
||||
return doc
|
||||
|
||||
Reference in New Issue
Block a user