Guardian permission subplans now actually execute (previously
'never executed' due to unowned seed documents short-circuiting the
owner_id IS NULL branch). best_seconds rose from 0.0251s to 31.4648s,
confirming the varchar-cast nested-loop-semi-join pathology is real
and now measured under realistic conditions.
Adds two new functions to harness.py:
- _current_git_ref(): Gets the current git short SHA
- append_profiling_history(): Appends JSON line to profiling/results/history.jsonl
This enables profiling numbers to persist across sessions/dates, with every run
appended to an immutable timeline rather than overwriting per-stage snapshots.
Also adds corresponding test to verify the function writes valid JSON lines.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds standalone profiling tooling (never merged into dev/main): a
persistent, self-healing Postgres 18 container helper
(run_with_postgres.sh/stop_postgres.sh), a scale-profile dataset seeder
(seed.py) whose document counts and guardian permission-row ratios
mirror real bug reports (#13276, #13161), and a typed profiling harness
(harness.py) for timing/query-count comparisons and EXPLAIN ANALYZE
capture, gated by require_postgres() so it never silently runs on
SQLite.
seed.py samples distinct (subject, document) pairs up front rather than
drawing with replacement, since guardian's assign_perm() dedupes on the
(subject, object, permission) triple -- with-replacement sampling
against a small group pool collides heavily (birthday paradox) and
undercounts the target permission-row ratios otherwise.
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library
* Perf: select_related/prefetch_related for the scoped LLM index batch
* Perf: select_related/prefetch_related for the full LLM index rebuild path
* Fix: address Copilot review findings on LLM index scoping
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
* Fix: selection_data re-derives the filtered document set 5 times over
_get_selection_data_for_queryset() (powers ?include_selection_data=true on
the document list and search endpoints) computed document_count for
Correspondent/DocumentType/StoragePath/Tag/CustomField by embedding the
caller's full filtered queryset -- filters plus the permission check -- as
a subquery inside 5 separate Count(filter=Q(documents__in=queryset), ...)
calls. Each one re-evaluates that whole queryset from scratch.
Resolve the document ids once into a concrete list and reuse it across all
five annotations instead. For Tag/CustomField specifically (M2M via a
through-model table), also route through annotate_document_count_by_ids()
-- extracted from the tag/custom-field document_count fix (#13203) -- to
avoid the same Count(filter=Q(id__in=...), distinct=True)-on-an-M2M-relation
anti-pattern diagnosed there.
On a 400k-document/1,000-tag corpus, the correspondents portion alone
previously didn't finish within several minutes (killed twice while
investigating, including one run that left a zombie query still consuming
a CPU 30+ minutes later). All five queries together now complete in
~30-35s. Root-caused from a real report (paperless-ngx#13201) via a
different, already-fixed query (#13205) -- this one hasn't been reported
in the wild yet, found by auditing the same call path.
Depends on #13203 for annotate_document_count_by_ids().
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Address review feedback: drop unnecessary ordering before collecting ids
queryset passed into _get_selection_data_for_queryset() carries the
default/user-specified ordering, which is irrelevant once we're only
collecting a flat id list. Clearing it removes a pointless sort.
No measurable change in benchmarking at 400k documents -- the id-list
materialization/IN-clause cost still dominates -- but it's a free,
strictly-correct cleanup, not just noise-neutral in the other direction.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
query_similar_documents() never excluded the querying document itself, so a document could appear in its own "similar documents" context, duplicating its content into the AI-suggestions prompt and inflating prompt size/tokens unnecessarily. Add NE filter support to the vector store and exclude the source document's id at query time.
Avoids a Postgres error where the clamp UPDATE and the following ALTER TABLE on paperless_mail_mailrule share a transaction, but the table's FKs are deferrable, so pending trigger events block the ALTER.
Also fixes a verbose_name mismatch in migration 0013.
stumpylog
GitHub Actions
shamoonandGitHub
SandroandGitHub
Matthias MastandGitHub
