Prompted by an automated simplification review of the branch. Applies the
highest-value findings, all behavior-preserving (full suite still green):
- vector_store.py: extract _rebuild_file() (temp-file lifecycle: open,
populate, swap-in-or-discard-on-failure) and _rebuild_into() (create
table, copy meta, stream rows) out of compact() and the v1->v2
migration, which had duplicated both almost verbatim. The migration
module shrinks to a single _rebuild_into() call instead of reaching
into four PaperlessSqliteVecVectorStore privates.
- vector_store.py: extract _stored_schema_version() out of
has_pending_migration() and check_and_run_migrations(), which
duplicated the same schema_version read and its "missing key means
current" default.
- indexing.py: extract _with_exclusive_access(), collapsing three
identical _exclude_readers()/Timeout/log-and-skip blocks (compaction
in update_llm_index() and llm_index_compact(), the migration check in
_check_and_run_migrations()) to one line each. Also trims
_check_and_run_migrations()'s docstring, which had grown to restate
content already documented on has_pending_migration() and
check_and_run_migrations(), including a claim that was now stale
(llm_index_migrate() is a second consumer of the re-embed signal).
- test_ai_indexing.py: add a mock_store fixture for the
write_store()-yields-a-MagicMock pattern that 8 tests were hand-rolling
identically.
- migrations/__init__.py: consolidate the "how to add a migration"
instructions to one place instead of three.
- docs/administration.md: fix two now-inaccurate claims -- the bare-metal
migrate step doesn't run "automatically" (it's the manual step being
documented), and the self-contradictory "if enabled... no-op if
disabled" phrasing on the same step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Move schema migrations into their own paperless_ai/migrations package,
one module per migration (mNNNN_description.py -- a leading digit isn't
a valid Python identifier, unlike Django's own numbered migrations,
which load via importlib.import_module() rather than a static import
statement), establishing the pattern before more migrations accumulate
in vector_store.py.
- Drop the redundant "if the LLM index is enabled..." clause from the
Docker upgrade note in docs/administration.md -- it's already covered
by the LLM index section below, and calling out just one of several
auto-applied migrations there was incomplete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bulk_edit.py's tag/correspondent/document_type/storage_path/custom_field
helpers write via queryset.update() or direct M2M/through-model bulk
operations, none of which call Document.save() -- so Document.modified's
auto_now never fires. Comparing against get_modified_times() for a
document_ids-scoped update_llm_index() call was silently skipping reindex
of documents whose embedded metadata had just changed via a bulk edit.
The modified-time comparison is now only used for the unscoped,
full-library incremental scan, where it is still needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add document_llmindex migrate, a cheap check-only path (no reindex) safe
to run unconditionally: has_pending_migration() short-circuits to a
metadata-only read once the store is current, so a healthy install pays
almost nothing. If a pending migration would require re-embedding, it
only logs a warning and leaves the index as-is -- re-embedding can be
slow and, for a metered embedding backend, cost money, so it stays a
deliberate manual action (document_llmindex rebuild), never automatic.
Wire it into the Docker image as a new init-llmindex-migrate oneshot
(modeled on init-search-index, gated on init-migrations), and document
the equivalent manual step for bare-metal upgrades.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Only update_llm_index()'s nightly rebuild ran check_and_run_migrations(),
but llm_index_remove_document()/llm_index_add_or_update_document() call
delete()/upsert_document() directly and immediately (on every document
delete/edit). On upgrade, every pre-existing document is "pre-migration"
until the next scheduled rebuild -- up to 24h by default -- so a delete or
edit in that window would find zero document_chunks rows and silently
leave stale/orphaned vec0 rows behind.
Add has_pending_migration(), a cheap metadata-only read that needs no
exclusive access, so normal writes only pay for check_and_run_migrations()'s
exclusive locking when a migration is actually pending -- otherwise they'd
be gated by the same lock compaction uses, which
test_normal_write_is_not_gated_by_the_compaction_lock exists to forbid.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
document_chunks is a plain SQLite table, not a vec0 virtual table, so
(unlike vec0's own TEXT metadata column) it gets normal type-affinity
coercion between the TEXT document ids used elsewhere in this module and
an INTEGER column -- verified empirically. INTEGER keys make the
per-document delete lookup this table exists for cheaper: smaller index
entries and integer rather than text B-tree comparisons.
Cover compact()'s handling of a churned document's final chunk generation,
delete() on a never-indexed document, upsert_document() clearing to an empty
node list, and the v1->v2 migration's embed_model preservation and
idempotency, per repo convention.
vec0 only gets an efficient lookup on a metadata column inside a KNN
query; a plain document_id-filtered DELETE is a full table scan
regardless of index size. Add a document_chunks side table (plain
SQLite, real index) to look up chunk ids for a document, then delete
each by its id primary key instead. Includes a v1 -> v2 migration to
backfill document_chunks for indexes created before this existed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build_llm_index_text queried Note and CustomFieldInstance (plus its
field FK) per document, uncovered by the earlier correspondent/type/
storage_path/tags prefetch fix. Add notes and custom_fields__field to
the rebuild and scoped document querysets, and read from doc.notes.all()
instead of a fresh Note.objects.filter() so the prefetch is actually used.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library
* Perf: select_related/prefetch_related for the scoped LLM index batch
* Perf: select_related/prefetch_related for the full LLM index rebuild path
* Fix: address Copilot review findings on LLM index scoping
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
stumpylogandClaude Sonnet 5
shamoonandGitHub
github-actions[bot]andGitHub
SandroandGitHub
Matthias MastandGitHub