compact() had no migration guard: on a v1-schema store, document_chunks
reads 0 (freshly created empty) while total_inserts reflects the real
cumulative count, so the bloat check nearly always rebuilt -- silently
losing document_meta (copy_all reads from the empty v1 table) while
schema_version copied across unchanged, leaving the store permanently
unmigratable. compact() now calls has_pending_migration() and no-ops with
a warning instead.
Also folds in five minor final-review findings: drop _rebuild_into's
unused int return, hoist test-local imports to module level in
test_vector_store.py, note in TestMigrations' docstring that its fake
structural migrations only exercise dispatch (not full schema
correctness), restore the comment explaining why _row() requires
document_id, and tighten increment_total_inserts' docstring to not imply
general concurrency safety beyond its single atomic statement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test_migration_never_delegates_to_current_schema_helpers never actually ran
the migration (missing check_and_run_migrations() call) and its source-text
assertion was tautological (the "or DROP TABLE in source" clause was always
true). Now runs the real migration and asserts spy call counts instead:
DocumentChunksTable.create/DocumentMetaTable.create are each called exactly
3 times (construction, rebuild temp file, post-swap reopen -- all via
_open_connection, never from inside apply()), and _create_vec_table is
never called from the migration path.
test_drop_table_clears_modified_times asserted via get_modified_times(),
which short-circuits on table_exists() -- checking only the vec0 table that
drop_table() drops first -- so the assertion held even if
DocumentMetaTable.delete_all() were never called. Now asserts directly
against document_meta and document_chunks row counts.
Rewrites the sqlite-vec vector store's on-disk schema: document_id becomes
an INTEGER vec0 metadata column (was TEXT), modified moves out of vec0 into
a new document_meta side table, and a new document_chunks side table gives
O(1) per-document chunk lookup for delete/upsert instead of a full vec0
scan. compact() now streams document_chunks and document_meta across the
file-swap rebuild too (previously document_meta would have gone silently
empty after the first compaction). drop_table() clears both side tables.
Adds the single frozen m0001_v1_to_v2 migration, converting a real,
historically-shaped v1 store (the shape shipped since v3.0.0) into the v2
shape in one streaming pass, with its own hardcoded DDL rather than
delegating to any "current schema" helper.
SCHEMA_VERSION bumps 1 -> 2.
- Document sqlite3.Row row_factory precondition in module docstring
- Strengthen test_create_is_idempotent to verify populated table survives
- Collapse three roundtrip tests into one @pytest.mark.parametrize
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: unify born-digital PDF detection between archive decision and OCR
should_produce_archive() and RasterisedDocumentParser.parse() each
reimplemented the "does this PDF have real text" check independently,
using different normalization of pdftotext output. Raw pdftotext output
can be non-empty (whitespace/form-feed layout padding) even when there
is no real content, so the two checks could disagree: consumer.py
treated a tagged-but-textless PDF as born-digital and skipped the
archive, while the parser's own (stricter, normalized) check found no
text and ran OCR anyway, leaving the document with no archive despite
real OCR text (GH #13387).
Both call sites now share one predicate, pdf_born_digital_text() in
paperless/parsers/utils.py, so they can no longer drift apart.
* Fix: restore extract_text seam for born-digital detection in parse()
parse() had switched to calling pdf_born_digital_text() directly for
its initial text/born-digital check, bypassing the parser's own
extract_text instance method. That broke test mockability (tests patch
tesseract_parser.extract_text to control the born-digital decision)
and caused CI failures with mismatched OCR call counts and text.
Split pdf_born_digital_text() into is_born_digital_text(text, path,
log) - a pure decision function - and a thin pdf_born_digital_text()
wrapper for callers without text in hand (consumer.should_produce_archive).
parse() now extracts via self.extract_text(None, document_path) and
passes the result to is_born_digital_text(), restoring the seam with
no change to production behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Cleanup: simplify born-digital detection, close#13387 test gap
Simplification pass over the born-digital detection consolidation:
- is_born_digital_text(): drop the has_text temp for an early return.
- consumer.py: standardize the archive-decision log lines on plain
hyphens (was a mix of em-dash and hyphen) and hoist the duplicated
text_length computation.
- Parametrize TestPdfBornDigitalText instead of four near-identical
tests.
Code review follow-up: the existing tests only ever exercised
pdf_born_digital_text() through mocks, so the actual #13387 scenario
(a tagged PDF whose only "text" is layout padding) was never checked
against real pdftotext/pikepdf output - a regression in the
normalize-before-decide logic itself would have gone undetected.
Moved tagged_no_text_pdf_file from parsers/conftest.py up to the
shared paperless/tests/conftest.py (it was previously only visible to
tests under parsers/) and added a non-mocked regression test against
the real sample file.
Also fixed two stale comments in test_consumer.py referencing a
_extract_text_for_archive_check helper that no longer exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: exclude next-period start from relative date-range filters
Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper
bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of
the next period. Use half-open [lo TO hi} for those so e.g. "previous month"
no longer matches the 1st of the current month.
* Adds a regression test down to the second check for the hi range
build_llm_index_text queried Note and CustomFieldInstance (plus its
field FK) per document, uncovered by the earlier correspondent/type/
storage_path/tags prefetch fix. Add notes and custom_fields__field to
the rebuild and scoped document querysets, and read from doc.notes.all()
instead of a fresh Note.objects.filter() so the prefetch is actually used.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library
* Perf: select_related/prefetch_related for the scoped LLM index batch
* Perf: select_related/prefetch_related for the full LLM index rebuild path
* Fix: address Copilot review findings on LLM index scoping
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
* Fix: selection_data re-derives the filtered document set 5 times over
_get_selection_data_for_queryset() (powers ?include_selection_data=true on
the document list and search endpoints) computed document_count for
Correspondent/DocumentType/StoragePath/Tag/CustomField by embedding the
caller's full filtered queryset -- filters plus the permission check -- as
a subquery inside 5 separate Count(filter=Q(documents__in=queryset), ...)
calls. Each one re-evaluates that whole queryset from scratch.
Resolve the document ids once into a concrete list and reuse it across all
five annotations instead. For Tag/CustomField specifically (M2M via a
through-model table), also route through annotate_document_count_by_ids()
-- extracted from the tag/custom-field document_count fix (#13203) -- to
avoid the same Count(filter=Q(id__in=...), distinct=True)-on-an-M2M-relation
anti-pattern diagnosed there.
On a 400k-document/1,000-tag corpus, the correspondents portion alone
previously didn't finish within several minutes (killed twice while
investigating, including one run that left a zombie query still consuming
a CPU 30+ minutes later). All five queries together now complete in
~30-35s. Root-caused from a real report (paperless-ngx#13201) via a
different, already-fixed query (#13205) -- this one hasn't been reported
in the wild yet, found by auditing the same call path.
Depends on #13203 for annotate_document_count_by_ids().
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Address review feedback: drop unnecessary ordering before collecting ids
queryset passed into _get_selection_data_for_queryset() carries the
default/user-specified ordering, which is irrelevant once we're only
collecting a flat id list. Clearing it removes a pointless sort.
No measurable change in benchmarking at 400k documents -- the id-list
materialization/IN-clause cost still dominates -- but it's a free,
strictly-correct cleanup, not just noise-neutral in the other direction.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
query_similar_documents() never excluded the querying document itself, so a document could appear in its own "similar documents" context, duplicating its content into the AI-suggestions prompt and inflating prompt size/tokens unnecessarily. Add NE filter support to the vector store and exclude the source document's id at query time.
Avoids a Postgres error where the clamp UPDATE and the following ALTER TABLE on paperless_mail_mailrule share a transaction, but the table's FKs are deferrable, so pending trigger events block the ALTER.
Also fixes a verbose_name mismatch in migration 0013.
DocumentViewSet.get_queryset() carried a blanket .distinct() left over from
an older query shape (predating the version-file feature and the
correlated-subquery rewrite of num_notes/effective_content). Nothing in the
current base queryset can produce duplicate document rows: select_related
is all FK-to-PK, and the permission filter is a boolean id__in predicate,
not a join. id (the PK) is also always selected and inherently unique, so
.distinct() was a structural no-op for correctness.
It was not a no-op for cost. EXPLAIN showed it forcing a full sort-then-
dedupe over the entire permission-visible document set before LIMIT could
apply -- 340k rows for a 25-row page, with the effective_content/num_notes
correlated subqueries re-executed once per row as a result. This was the
dominant cost behind the original report's document overview slowness,
well beyond the get_user_can_change N+1 fixed separately.
Removing it isn't safe on its own, though -- auditing found two existing
filters that rely on an M2M join *without* deduping themselves, previously
papered over by the blanket .distinct():
- InboxFilter: a document with two tags both flagged is_inbox_tag=True
(nothing prevents that) would be returned twice for ?is_in_inbox=true.
- CustomFieldsFilter: a document with multiple custom field instances
matching different OR-ed branches would be returned once per match.
Fixed both locally, matching the pattern ObjectFilter already used for
tags__id__in/custom_fields__id__in. Added regression tests for both --
confirmed they fail without the local .distinct() calls.
Benchmarked against the same 400k-document/1,000-tag corpus: document list
wall-clock for a permission-restricted user drops from ~15.8s to ~2.25s
(~7x), closing most of the previous gap to a superuser's ~1.8s.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix (beta): tag/custom-field document_count scales badly with tag count
TagViewSet and CustomFieldViewSet's document_count annotation used a
per-row correlated subquery (annotate_document_count_for_related_queryset),
executed once per tag/custom-field row. Fine at a few dozen rows, but at
~1,000 tags it degrades to a full per-tag GroupAggregate over the tags M2M
table -- 6s+ in production reports, confirmed via EXPLAIN (loops=1000).
Replaced with a single, independent GROUP BY over the through table
(permission filter expressed as a plain WHERE, not an aggregate FILTER),
then injected via Case/When. Also tried a more "obvious" fix -- a plain
Count(filter=Q(id__in=permitted_ids), distinct=True) directly on the M2M
relation -- but that's worse: Postgres fails to plan the id__in check as a
semi-join once a large M2M bridge table is involved, and instead re-checks
subquery membership once per joined row.
Benchmarked against a synthetic corpus (400k documents, 1,000 tags @ ~5/doc,
matching real-world reports in discussion #13161): tag list wall-clock drops
from ~180s to ~8s for a permission-restricted user, ~21s to ~8s for a
superuser. No measurable change at typical home-instance scale (tens of
tags).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Address review feedback: restore validation, scope aggregation to queryset
- Restore the document_count_source_field validation helper dropped during
the refactor -- misconfiguring a viewset (document_count_through set
without document_count_source_field) now fails fast with a clear error
again instead of an obscure runtime failure.
- Restrict annotate_document_count_for_related_queryset()'s through-table
aggregation to rows whose related_object_field is one of the annotated
queryset's pks. No-op for the main tag-list call site (queryset is all
tags), but avoids unnecessary work for narrower callers like the tag
descendants branch in TagViewSet.list().
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
stumpylogandClaude Sonnet 5
shamoonandGitHub
Matthias MastandGitHub
