vec0 only inlines TEXT metadata up to 12 bytes (sqlite-vec.c); modified
is an ISO timestamp (e.g. "2026-07-28T12:34:56.789012+00:00", 32 chars),
always over that threshold, so every read of it recompiled and stepped a
fresh SQL statement per row -- uncached, on every full scan
(get_modified_times(), every compact(), every structural migration). It
was also never filtered on inside a KNN query, so it never needed to be
a vec0 column in the first place.
Add document_meta(document_id INTEGER PRIMARY KEY, modified TEXT), one
row per document rather than per chunk (all of a document's chunks share
the same modified value). get_modified_times() drops its per-chunk dedup
loop entirely as a result -- one row read per document, not per chunk.
No separate index needed here (unlike document_chunks): document_id
being the INTEGER PRIMARY KEY makes it the table's own rowid, and every
access pattern is already keyed by it directly.
Schema bump to v3 (m0002), backfilling document_meta from the old vec0
column for existing stores. Along the way:
- _Row/_Vec0Params (NamedTuple) replace bare positional tuples for node
rows and vec0 bind parameters -- still plain tuples as far as
sqlite3.executemany() is concerned, but self-documenting.
- Both new document_meta copy paths (compact()/migration backfill) are
batched via fetchmany(), matching the existing discipline for the
chunk-scoped copy -- small today (one row per document), but the same
bounded-memory principle applies regardless of size.
- Discovered along the way: m0001 delegated to _rebuild_into(), which
always reflects the *current* schema. That was fine while v2 was
current, but silently wrong the moment v3 existed (m0001 would
produce a v3-shaped table instead of its actual v2 shape) -- fixed
upstream on perf/13314-vecstore-point-delete, since it's a defect in
that branch's own code independent of this change.
Second item from VECTOR_STORE_PERF_BACKLOG.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_rebuild_into()/_create_vec_table() always reflect whatever the *current*
schema is -- correct for compact() (which only ever runs on an
already-current-schema store), but wrong for a migration that isn't the
latest one anymore. m0001 delegated to them, which worked fine as long as
v2 was current, but silently breaks the moment a v2 -> v3 migration is
added: m0001 would then produce a v3-shaped table (whatever columns
happen to be current) instead of its actual v2 shape, and the next
migration in the chain would find the columns it expects to migrate from
already gone.
Spell out m0001's own v2-shaped CREATE VIRTUAL TABLE and row copy instead,
so it keeps producing the same output forever, independent of later
schema changes -- the same reasoning already documented on the test
helper _copying_apply(), just not, until now, applied to the real
migration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prompted by an automated simplification review of the branch. Applies the
highest-value findings, all behavior-preserving (full suite still green):
- vector_store.py: extract _rebuild_file() (temp-file lifecycle: open,
populate, swap-in-or-discard-on-failure) and _rebuild_into() (create
table, copy meta, stream rows) out of compact() and the v1->v2
migration, which had duplicated both almost verbatim. The migration
module shrinks to a single _rebuild_into() call instead of reaching
into four PaperlessSqliteVecVectorStore privates.
- vector_store.py: extract _stored_schema_version() out of
has_pending_migration() and check_and_run_migrations(), which
duplicated the same schema_version read and its "missing key means
current" default.
- indexing.py: extract _with_exclusive_access(), collapsing three
identical _exclude_readers()/Timeout/log-and-skip blocks (compaction
in update_llm_index() and llm_index_compact(), the migration check in
_check_and_run_migrations()) to one line each. Also trims
_check_and_run_migrations()'s docstring, which had grown to restate
content already documented on has_pending_migration() and
check_and_run_migrations(), including a claim that was now stale
(llm_index_migrate() is a second consumer of the re-embed signal).
- test_ai_indexing.py: add a mock_store fixture for the
write_store()-yields-a-MagicMock pattern that 8 tests were hand-rolling
identically.
- migrations/__init__.py: consolidate the "how to add a migration"
instructions to one place instead of three.
- docs/administration.md: fix two now-inaccurate claims -- the bare-metal
migrate step doesn't run "automatically" (it's the manual step being
documented), and the self-contradictory "if enabled... no-op if
disabled" phrasing on the same step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Move schema migrations into their own paperless_ai/migrations package,
one module per migration (mNNNN_description.py -- a leading digit isn't
a valid Python identifier, unlike Django's own numbered migrations,
which load via importlib.import_module() rather than a static import
statement), establishing the pattern before more migrations accumulate
in vector_store.py.
- Drop the redundant "if the LLM index is enabled..." clause from the
Docker upgrade note in docs/administration.md -- it's already covered
by the LLM index section below, and calling out just one of several
auto-applied migrations there was incomplete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bulk_edit.py's tag/correspondent/document_type/storage_path/custom_field
helpers write via queryset.update() or direct M2M/through-model bulk
operations, none of which call Document.save() -- so Document.modified's
auto_now never fires. Comparing against get_modified_times() for a
document_ids-scoped update_llm_index() call was silently skipping reindex
of documents whose embedded metadata had just changed via a bulk edit.
The modified-time comparison is now only used for the unscoped,
full-library incremental scan, where it is still needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add document_llmindex migrate, a cheap check-only path (no reindex) safe
to run unconditionally: has_pending_migration() short-circuits to a
metadata-only read once the store is current, so a healthy install pays
almost nothing. If a pending migration would require re-embedding, it
only logs a warning and leaves the index as-is -- re-embedding can be
slow and, for a metered embedding backend, cost money, so it stays a
deliberate manual action (document_llmindex rebuild), never automatic.
Wire it into the Docker image as a new init-llmindex-migrate oneshot
(modeled on init-search-index, gated on init-migrations), and document
the equivalent manual step for bare-metal upgrades.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Only update_llm_index()'s nightly rebuild ran check_and_run_migrations(),
but llm_index_remove_document()/llm_index_add_or_update_document() call
delete()/upsert_document() directly and immediately (on every document
delete/edit). On upgrade, every pre-existing document is "pre-migration"
until the next scheduled rebuild -- up to 24h by default -- so a delete or
edit in that window would find zero document_chunks rows and silently
leave stale/orphaned vec0 rows behind.
Add has_pending_migration(), a cheap metadata-only read that needs no
exclusive access, so normal writes only pay for check_and_run_migrations()'s
exclusive locking when a migration is actually pending -- otherwise they'd
be gated by the same lock compaction uses, which
test_normal_write_is_not_gated_by_the_compaction_lock exists to forbid.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
document_chunks is a plain SQLite table, not a vec0 virtual table, so
(unlike vec0's own TEXT metadata column) it gets normal type-affinity
coercion between the TEXT document ids used elsewhere in this module and
an INTEGER column -- verified empirically. INTEGER keys make the
per-document delete lookup this table exists for cheaper: smaller index
entries and integer rather than text B-tree comparisons.
Cover compact()'s handling of a churned document's final chunk generation,
delete() on a never-indexed document, upsert_document() clearing to an empty
node list, and the v1->v2 migration's embed_model preservation and
idempotency, per repo convention.
vec0 only gets an efficient lookup on a metadata column inside a KNN
query; a plain document_id-filtered DELETE is a full table scan
regardless of index size. Add a document_chunks side table (plain
SQLite, real index) to look up chunk ids for a document, then delete
each by its id primary key instead. Includes a v1 -> v2 migration to
backfill document_chunks for indexes created before this existed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build_llm_index_text queried Note and CustomFieldInstance (plus its
field FK) per document, uncovered by the earlier correspondent/type/
storage_path/tags prefetch fix. Add notes and custom_fields__field to
the rebuild and scoped document querysets, and read from doc.notes.all()
instead of a fresh Note.objects.filter() so the prefetch is actually used.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library
* Perf: select_related/prefetch_related for the scoped LLM index batch
* Perf: select_related/prefetch_related for the full LLM index rebuild path
* Fix: address Copilot review findings on LLM index scoping
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
stumpylogandClaude Sonnet 5
shamoonandGitHub
github-actions[bot]andGitHub
SandroandGitHub
