mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-02 00:52:20 +00:00
e5b27df99c
* Feature: schema v2 -- document_chunks/document_meta side tables, document_id INTEGER, point-delete Rewrites the sqlite-vec vector store's on-disk schema: document_id becomes an INTEGER vec0 metadata column (was TEXT), modified moves out of vec0 into a new document_meta side table, and a new document_chunks side table gives O(1) per-document chunk lookup for delete/upsert instead of a full vec0 scan. compact() now streams document_chunks and document_meta across the file-swap rebuild too (previously document_meta would have gone silently empty after the first compaction). drop_table() clears both side tables. Adds the single frozen m0001_v1_to_v2 migration, converting a real, historically-shaped v1 store (the shape shipped since v3.0.0) into the v2 shape in one streaming pass, with its own hardcoded DDL rather than delegating to any "current schema" helper. SCHEMA_VERSION bumps 1 -> 2. * Fix: strengthen two vacuous Task 4 regression tests test_migration_never_delegates_to_current_schema_helpers never actually ran the migration (missing check_and_run_migrations() call) and its source-text assertion was tautological (the "or DROP TABLE in source" clause was always true). Now runs the real migration and asserts spy call counts instead: DocumentChunksTable.create/DocumentMetaTable.create are each called exactly 3 times (construction, rebuild temp file, post-swap reopen -- all via _open_connection, never from inside apply()), and _create_vec_table is never called from the migration path. test_drop_table_clears_modified_times asserted via get_modified_times(), which short-circuits on table_exists() -- checking only the vec0 table that drop_table() drops first -- so the assertion held even if DocumentMetaTable.delete_all() were never called. Now asserts directly against document_meta and document_chunks row counts. * Perf: dedupe table_exists() lookups, atomic insert counter, fewer connections in update_llm_index() * Fix: guard compact() against unmigrated stores, apply final-review cleanups compact() had no migration guard: on a v1-schema store, document_chunks reads 0 (freshly created empty) while total_inserts reflects the real cumulative count, so the bloat check nearly always rebuilt -- silently losing document_meta (copy_all reads from the empty v1 table) while schema_version copied across unchanged, leaving the store permanently unmigratable. compact() now calls has_pending_migration() and no-ops with a warning instead. Also folds in five minor final-review findings: drop _rebuild_into's unused int return, hoist test-local imports to module level in test_vector_store.py, note in TestMigrations' docstring that its fake structural migrations only exercise dispatch (not full schema correctness), restore the comment explaining why _row() requires document_id, and tighten increment_total_inserts' docstring to not imply general concurrency safety beyond its single atomic statement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>