document_chunks is a plain SQLite table, not a vec0 virtual table, so
(unlike vec0's own TEXT metadata column) it gets normal type-affinity
coercion between the TEXT document ids used elsewhere in this module and
an INTEGER column -- verified empirically. INTEGER keys make the
per-document delete lookup this table exists for cheaper: smaller index
entries and integer rather than text B-tree comparisons.
Cover compact()'s handling of a churned document's final chunk generation,
delete() on a never-indexed document, upsert_document() clearing to an empty
node list, and the v1->v2 migration's embed_model preservation and
idempotency, per repo convention.
vec0 only gets an efficient lookup on a metadata column inside a KNN
query; a plain document_id-filtered DELETE is a full table scan
regardless of index size. Add a document_chunks side table (plain
SQLite, real index) to look up chunk ids for a document, then delete
each by its id primary key instead. Includes a v1 -> v2 migration to
backfill document_chunks for indexes created before this existed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build_llm_index_text queried Note and CustomFieldInstance (plus its
field FK) per document, uncovered by the earlier correspondent/type/
storage_path/tags prefetch fix. Add notes and custom_fields__field to
the rebuild and scoped document querysets, and read from doc.notes.all()
instead of a fresh Note.objects.filter() so the prefetch is actually used.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library
* Perf: select_related/prefetch_related for the scoped LLM index batch
* Perf: select_related/prefetch_related for the full LLM index rebuild path
* Fix: address Copilot review findings on LLM index scoping
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
stumpylog
shamoonandGitHub
github-actions[bot]andGitHub
SandroandGitHub
Matthias MastandGitHub