Commit Graph
14 Commits
Author SHA1 Message Date
Trenton Holmes 7f5053cbe3 Implements config_mismatch which checks for either a dimension or model name mismatch. Model name is now stored as internal metadata of the vector store 2026-06-06 14:48:50 -07:00
stumpylog 3aa83c9e4c To reduce embedding size, don't store the metadata in the body. Body is content + a few other things, metadata keys hold the metadata 2026-06-05 11:43:42 -07:00
stumpylogandClaude Opus 4.8 17755a2c58 refactor(ai): cleanup pass — naming, batched embedding, remove dead wrappers
- Rename vector_store_file_exists -> llm_index_exists (accurate now)
- Rename _iter_existing_modified -> _stored_modified_times; project away
  vector column (cheap scan) and return dict[doc_id, modified_str] directly
- Drop _index_lock_path() indirection; inline settings.LLM_INDEX_LOCK
- Move LLM_INDEX_LOCK alongside the index dir (drop_table is safe; no rmtree)
- Drop current_embedding_dim() redirect; callers use get_embedding_dim()
- Drop lazy-import explanatory comments (constraint lives in CLAUDE.md)
- Batch embedding calls via get_text_embedding_batch() in all three loops
- get_nodes: raise NotImplementedError for node_ids (was silently ignored)
- has_nodes(): cheap limit(1) existence check; chat.py uses it instead of
  get_nodes() which materialized all matching rows
- conftest: use mocker fixture (pytest-mock) instead of bare patch; add
  LLM_INDEX_LOCK to temp_llm_index_dir override; type-annotate mock_embed_model

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 11:43:42 -07:00
stumpylogandClaude Opus 4.8 b855eba878 feat(ai): dimension guard and FAISS index migration
Adds current_embedding_dim() to embedding.py, migrate_stale_faiss_index()
and embedding_dim_mismatch() to indexing.py, and wires both into
update_llm_index so that stale FAISS directories are wiped on startup and
embedding model changes force a full index rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 11:43:42 -07:00
shamoonandGitHub f6c865bf47 Enhancement: AI LLM chunk size and context window config (#12891) 2026-06-01 17:56:21 +00:00
shamoonandGitHub 01d8fad622 Security: fixes for v3 beta (#12838) 2026-05-26 16:46:23 +00:00
shamoonandGitHub da3e845b8b Fix (beta): normalize long punctuation chunks to improve embedding (#12848) 2026-05-26 09:32:38 -07:00
Trenton HandGitHub da0f25b546 Fix: Use a persistent, writeable location for hugging face models (#12771) 2026-05-09 18:23:11 -07:00
shamoonandGitHub 79d0a04df0 Enhancement: support ollama embeddings (#12753) 2026-05-09 00:06:14 +00:00
shamoonandGitHub 69cb4d06c6 Enhancement (dev): Use OpenAI-like backend (#12668) 2026-04-28 10:06:59 -07:00
shamoon 4d4f30b5f8 Security: validate outbound llm URLs and block internal endpoints 2026-03-16 22:58:16 -07:00
Trenton HandGitHub f5c0c21922 Chore: Lazy imports of the heavy AI modules (#12275) 2026-03-07 12:53:22 -08:00
shamoonandGitHub fa13ca7a42 Fix: pass api_base to OpenAIEmbedding (#12151) 2026-02-23 13:47:32 -08:00
shamoonandGitHub e940764fe0 Feature: Paperless AI (#10319) 2026-01-13 16:24:42 +00:00