Delete _get_document_filtered_retriever (74-line custom FAISS retriever
with expanding top_k loop) and rewrite _stream_chat_with_documents to use
a stock VectorIndexRetriever with MetadataFilters(IN). The no-content
pre-check now calls index.vector_store.get_nodes(filters=...) which
returns [] cleanly for un-indexed documents. Move FakeEmbedding and
mock_embed_model fixture to conftest.py so both test_chat.py and
test_ai_indexing.py share them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops migrate_stale_faiss_index (users delete llm_index/ manually on upgrade).
Keeps embedding_dim_mismatch to force a rebuild when the model changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds current_embedding_dim() to embedding.py, migrate_stale_faiss_index()
and embedding_dim_mismatch() to indexing.py, and wires both into
update_llm_index so that stale FAISS directories are wiped on startup and
embedding model changes force a full index rebuild.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace get_or_create_storage_context with get_vector_store() (lazy import
of paperless_ai.vector_store inside the function), rewrite load_or_build_index
to use VectorStoreIndex.from_vector_store, and rewrite vector_store_file_exists
to use store.table_exists(). Add LLM_INDEX_TABLE constant and TYPE_CHECKING-only
import of PaperlessLanceVectorStore. Delete remove_document_docstore_nodes and
rewire llm_index_add_or_update_document, llm_index_remove_document, and
update_llm_index to use upsert_document/delete/drop_table on the LanceDB store.
Serialize tags list as JSON string to satisfy flat_metadata validation. Add
test_get_vector_store_roundtrip, test_add_then_remove_document,
test_update_shrinks_chunks_without_orphans, and the subprocess lazy-import guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Set id_=str(document.id) on the LlamaDocument constructor in
build_document_node so that every chunk node's ref_doc_id equals the
paperless document pk, enabling the LanceDB adapter's delete(str(doc.id))
and doc_id column to work correctly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When upsert_document receives an empty nodes list, delete existing
chunks using the document_id column directly (consistent with the
merge_insert prune predicate) rather than calling delete() which
filters on doc_id. Guard for a missing table to avoid a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fix delete() to use single-quote delimiter consistent with _escape
- Fix _distance comment: L2 not squared-L2
- Fix similarity_top_k zero-guard to use explicit None check
- Replace deprecated table_names() with list_tables().tables (lancedb 0.33)
- Add add() Sequence[BaseNode] signature with collections.abc.Sequence import
- Add test_build_where_or_condition for OR filter branch coverage
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-by-task TDD plan implementing the LanceDB design spec: dependency
swap, the PaperlessLanceVectorStore adapter, atomic merge_insert upsert,
ANN threshold + scalar index + compaction, the indexing/chat/similar
rewires, FAISS migration, and a lazy-import guard test so non-AI paths
(management commands) never drag in llama_index/lancedb/pyarrow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spec for swapping the AI feature's llama-index FAISS StorageContext trio
(FaissVectorStore + SimpleDocumentStore + SimpleIndexStore) for LanceDB via
a custom BasePydanticVectorStore adapter (no llama-index-vector-stores-lancedb,
no pandas).
Covers: disk-resident memory-mapped storage, native merge_insert upsert with
when_not_matched_by_source_delete, MetadataFilters(IN) filtering on a top-level
document_id column, auto IVF ANN threshold (IVF_FLAT fallback), MVCC compaction
via optimize(cleanup_older_than=...), migration, concurrency, and testing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix: Move LLM index lock outside index dir and skip per-doc tasks on bulk update
Two concurrency bugs from #12893:
[P1] Lock file lived inside LLM_INDEX_DIR. A rebuild calls
shutil.rmtree(LLM_INDEX_DIR), deleting the lock while a worker still
held it. A second worker then acquired a fresh lock on the new path and
ran concurrently, defeating serialisation. Move the lock to
DATA_DIR/locks/llm_index.lock (a new settings constant LLM_INDEX_LOCK)
so rmtree cannot touch it. The locks/ dir is created at settings load
time, matching the existing pattern for LOGGING_DIR.
[P2] document_updated was connected to add_or_update_document_in_llm_index
in apps.py. bulk_update_documents() emits document_updated for every
document in the batch, queuing N per-document LLM tasks, and then also
calls update_llm_index(rebuild=False) once at the end. Pass
skip_ai_index=True when sending document_updated from the bulk path so
the handler skips the per-document enqueue; the existing batch call at
the end of bulk_update_documents is the only LLM update for that path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: ghost vectors leave KeyError-prone nodes_dict entries after deletion
docstore.delete_document() removes a node from the docstore but leaves its
entry in index_struct.nodes_dict (the FAISS positional-id to node-UUID map).
A subsequent similarity query resolves the ghost position to the deleted UUID,
finds nothing in fetched_nodes_by_id, and raises KeyError inside
_insert_fetched_nodes_into_query_result.
Purge stale nodes_dict entries after each docstore deletion and re-sync the
mutated index_struct into the kvstore so persist() writes the updated mapping.
Dead FAISS vectors remain in the flat index until the next full rebuild
(IndexFlatL2 is append-only); add a try/except KeyError around
retriever.retrieve() as a defensive fallback for any residual ghost positions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: acquire index lock in query_similar_documents
query_similar_documents() loaded the index and ran the FAISS retriever
without holding the file lock. All write paths (update_llm_index,
llm_index_add_or_update_document, llm_index_remove_document) hold
FileLock(_index_lock_path()), so a concurrent rebuild calling
shutil.rmtree(LLM_INDEX_DIR) while a read is mid-load produces an IOError
or corrupt partial state.
Wrap the load_or_build_index() call and all subsequent retriever work inside
FileLock. The early-return guards (vector_store_file_exists check, empty
allowed_document_ids) remain outside the lock; the DB query for the final
result set also stays outside.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: skip LLM index enqueue on document_updated during version addition
When a document is consumed as a new version of an existing document, the
consumer fires document_consumption_finished (which triggers
add_or_update_document_in_llm_index) and then document_updated for the root
document. Both signals are connected to the same handler, so the root document
was enqueued for LLM indexing twice per version-addition event.
Pass skip_ai_index=True on the consumer's version-addition document_updated
send so the handler's existing guard suppresses the duplicate enqueue.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Test: bulk_update_documents must not enqueue per-doc LLM tasks
With AI enabled, bulk_update_documents() sends document_updated for every
document in the batch. The skip_ai_index=True kwarg (added in the P2 fix)
prevents add_or_update_document_in_llm_index from enqueuing a per-document
task for each one. Only the single update_llm_index call at the end should run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Debug level log sure
* Update src/paperless_ai/indexing.py
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Apply suggestion from @shamoon
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Fix: Remove all nodes for multi-chunk documents in update_llm_index incremental path
The existing_nodes dict comprehension keyed on document_id silently dropped all
but the last node per document, so only that one node was deleted when a
modified document was re-indexed, leaving all other chunks as ghost vectors in
the FAISS index. Switch to a defaultdict(list) that collects every node per
document_id, then iterate and delete all of them before inserting fresh nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Wire document_updated signal to LLM index update handler
Connect document_updated to add_or_update_document_in_llm_index in
DocumentsConfig.ready() so REST API edits (PATCH /api/documents/{id}/)
enqueue an LLM vector store update, matching the existing
document_consumption_finished behavior.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Add file lock around FAISS index mutations to prevent concurrent write corruption
Two concurrent Celery workers calling llm_index_add_or_update_document or
llm_index_remove_document each loaded the same on-disk index independently,
made their own change, and the last writer silently overwrote the first's
update. Wrap both functions and the rebuild/persist body of update_llm_index
in a filelock.FileLock keyed on LLM_INDEX_DIR/index.lock. Add a TOCTOU
comment on queue_llm_index_update_if_needed explaining the residual risk
(duplicate rebuild tasks are wasteful but not corrupting because the lock
serialises the actual write).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Apply _normalize() in extract_unmatched_names to prevent duplicate suggestions
extract_unmatched_names was using .lower() while _match_names_to_queryset
uses _normalize() (which also strips punctuation). A name like "J. Smith"
matched to existing correspondent "J Smith" would still appear in the
unmatched list, causing duplicate object creation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Skip LLM index update gracefully when document has no indexable content
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Persist empty index when all documents are deleted to clear stale FAISS vectors
The early-return guard in update_llm_index fired before persist() when no
documents existed, leaving a stale on-disk FAISS index that returned phantom
hits for deleted document IDs. Now the guard only returns early for the
incremental (rebuild=False) path when no index exists on disk; the rebuild
path always continues through to persist(), producing an empty clean index.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore: Simplify incremental index update — use docs.values() and deduplicate node extend
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Validate and limit chat question input in ChatStreamingView
Add max_length=4000 to ChatStreamingSerializer.q and replace the bare
request.data["q"] read with proper serializer.is_valid(raise_exception=True)
so oversized or missing questions are rejected with HTTP 400 before
reaching the LLM.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Add defensive prompt framing to mark document content as untrusted
* Also adds a system prompt which is treated higher that this is untrusted stuff
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements and tests a retry with backoff + jitter for aquring the index update lock. If we still can't get it, dispatch a celery task to handle it later instead (also with retry)
Signed-off-by: stumpylog <797416+stumpylog@users.noreply.github.com>
stumpylog
shamoon