Task-by-task TDD plan implementing the LanceDB design spec: dependency
swap, the PaperlessLanceVectorStore adapter, atomic merge_insert upsert,
ANN threshold + scalar index + compaction, the indexing/chat/similar
rewires, FAISS migration, and a lazy-import guard test so non-AI paths
(management commands) never drag in llama_index/lancedb/pyarrow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spec for swapping the AI feature's llama-index FAISS StorageContext trio
(FaissVectorStore + SimpleDocumentStore + SimpleIndexStore) for LanceDB via
a custom BasePydanticVectorStore adapter (no llama-index-vector-stores-lancedb,
no pandas).
Covers: disk-resident memory-mapped storage, native merge_insert upsert with
when_not_matched_by_source_delete, MetadataFilters(IN) filtering on a top-level
document_id column, auto IVF ANN threshold (IVF_FLAT fallback), MVCC compaction
via optimize(cleanup_older_than=...), migration, concurrency, and testing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stumpylog