Commit Graph
4 Commits
Author SHA1 Message Date
Trenton H 62d729e301 Fix: Allow AI chat filters over more than 32700 documents (#14754) 2026-11-12 12:08:59 -08:00
Trenton H 68a769b40c Fix: make LLM index migration-check result tri-state, avoid deferred-vs-current ambiguity (#13436)
* Fix: make migration-check result tri-state to avoid deferred-vs-current ambiguity

* Test: cover update_llm_index()'s migration-check-deferred branch

* Refactor: rename COMPACT_BATCH_SIZE to BATCH_SIZE, no longer compact()-specific
2026-08-01 07:20:29 -07:00
Trenton HandClaude Sonnet 5 e5b27df99c Feature: vector store schema v2 document_chunks/document_meta side tables, INTEGER document_id, point-delete (#13435)
* Feature: schema v2 -- document_chunks/document_meta side tables, document_id INTEGER, point-delete

Rewrites the sqlite-vec vector store's on-disk schema: document_id becomes
an INTEGER vec0 metadata column (was TEXT), modified moves out of vec0 into
a new document_meta side table, and a new document_chunks side table gives
O(1) per-document chunk lookup for delete/upsert instead of a full vec0
scan. compact() now streams document_chunks and document_meta across the
file-swap rebuild too (previously document_meta would have gone silently
empty after the first compaction). drop_table() clears both side tables.

Adds the single frozen m0001_v1_to_v2 migration, converting a real,
historically-shaped v1 store (the shape shipped since v3.0.0) into the v2
shape in one streaming pass, with its own hardcoded DDL rather than
delegating to any "current schema" helper.

SCHEMA_VERSION bumps 1 -> 2.

* Fix: strengthen two vacuous Task 4 regression tests

test_migration_never_delegates_to_current_schema_helpers never actually ran
the migration (missing check_and_run_migrations() call) and its source-text
assertion was tautological (the "or DROP TABLE in source" clause was always
true). Now runs the real migration and asserts spy call counts instead:
DocumentChunksTable.create/DocumentMetaTable.create are each called exactly
3 times (construction, rebuild temp file, post-swap reopen -- all via
_open_connection, never from inside apply()), and _create_vec_table is
never called from the migration path.

test_drop_table_clears_modified_times asserted via get_modified_times(),
which short-circuits on table_exists() -- checking only the vec0 table that
drop_table() drops first -- so the assertion held even if
DocumentMetaTable.delete_all() were never called. Now asserts directly
against document_meta and document_chunks row counts.

* Perf: dedupe table_exists() lookups, atomic insert counter, fewer connections in update_llm_index()

* Fix: guard compact() against unmigrated stores, apply final-review cleanups

compact() had no migration guard: on a v1-schema store, document_chunks
reads 0 (freshly created empty) while total_inserts reflects the real
cumulative count, so the bloat check nearly always rebuilt -- silently
losing document_meta (copy_all reads from the empty v1 table) while
schema_version copied across unchanged, leaving the store permanently
unmigratable. compact() now calls has_pending_migration() and no-ops with
a warning instead.

Also folds in five minor final-review findings: drop _rebuild_into's
unused int return, hoist test-local imports to module level in
test_vector_store.py, note in TestMigrations' docstring that its fake
structural migrations only exercise dispatch (not full schema
correctness), restore the comment explaining why _row() requires
document_id, and tighten increment_total_inserts' docstring to not imply
general concurrency safety beyond its single atomic statement.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-01 07:20:29 -07:00
Trenton HandClaude Sonnet 5 95a48dd685 Chore: Vector Store table-gateway module (document_chunks/document_meta/index_meta) (#13428)
* Feature: add table-gateway module for document_chunks/document_meta/index_meta

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Fix: address code review feedback on table gateways

- Document sqlite3.Row row_factory precondition in module docstring
- Strengthen test_create_is_idempotent to verify populated table survives
- Collapse three roundtrip tests into one @pytest.mark.parametrize

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-01 07:20:28 -07:00