imap_tools 1.14.0 adds a uid_list fetch arg that fetches by UID
directly without issuing a SEARCH command, and does its own
bulk-fetch batching. Use it in place of the manual per-batch
AND(uid=...) fetch loop.
DocumentViewSet.get_queryset() annotated num_notes via a LEFT JOIN to
documents_note plus Count(), which requires the database to aggregate
every matching document's note count before it can even sort or paginate
the result -- on every list request, not just filtered ones.
Switched to the same correlated-subquery pattern already used two lines
above for effective_content (Subquery + OuterRef), which Django compiles
to portable SQL across sqlite/postgresql/mariadb rather than a join-based
aggregate.
Benchmarked against a 100k synthetic document corpus (Postgres): the
plain document list request dropped from ~2.2-2.4s to ~0.9-1.0s. Note
that a separate, larger cost remains in the same queryset's plain
.distinct() call, which still forces a full sort over every matching
row regardless of this fix -- tracked separately, not addressed here.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
apply_assignment_to_document() mutated tags directly on the shared document instance via add_nested_tags(), whose m2m_changed signal triggers update_filename_and_move_files() -> instance.refresh_from_db(), discarding any not-yet-saved fields (e.g. storage_path) staged by an earlier-ordered action in the same workflow. Apply tag changes to a freshly-fetched instance instead, matching the pattern already used in apply_removal_to_document().
Adds indexes on Document.checksum, Document.page_count, and a composite
(owner, created) index on Document, plus composite (field, value_*)
indexes on CustomFieldInstance for each typed value column.
Benchmarked against a 100k-document / 200k-custom-field-instance SQLite
dataset: Document.checksum lookups ~69x faster, page_count sort ~40x,
owner+created list queries ~6x. CustomFieldInstance composite indexes
show large gains (10-24x) on realistic narrow-selectivity queries
matching the workflow scheduler and custom-field search code paths.
Django's condition() decorator invokes etag_func and last_modified_func
separately, and the view itself may resolve again -- each call to
resolve_effective_document_by_pk() was redoing the same root/version
DB lookups. Memoize the resolution on the request object so a single
thumb/metadata/preview request resolves the effective document once
instead of up to three times.
Related to paperless-ngx/paperless-ngx#13161.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Tantivy: get permissions by chunks
-40% indexing time compared to previous commit
* Make progress bar process one by one with chunk
-15% indexing time compared to previous commit
* Prefetch FK + iterate over chunk from SQL
Prefetch additional needed data (note user, custom field content)
-20% indexing time compared to previous commit
* Reindex: increase Tantivy heap size from 128 to 512MB
Gains probably vary depending on the machine,
but it seems a sweet spot compatible with low-end hardware.
* Reindex: optimization on permission fetching and autocomplete word set
-10% indexing time compared to previous commit
* Autocomplete analyzer python->rust
Splits words with underscore compared to the python analyzer.
E.g.: "blue_print" -> ["blue", "print"]
It can still be found with the "blue_print" keyword,
as the search string is also split in two words.
-50% indexing time compared to previous commit (indexing is twice faster!)
* Index bigram for CJK content only
Inedxing time slightly longer (~3%),
but since the non-CJK content is not indexed,
bigram searchs will be slightly optimized.
* Fix group-based view_document permissions missing from bulk rebuild
_bulk_get_viewer_ids only queried UserObjectPermission, dropping the
group-permission expansion that get_users_with_perms(with_group_users=True)
performs for the non-batched per-document indexing path. A user who could
only see a document via group membership would lose search access to it
after any full reindex.
Also query GroupObjectPermission and expand group membership to user ids,
matching the existing single-document behavior.
* Yield (document, viewer_ids) pairs from _DocumentViewerStream
Previously _DocumentViewerStream.__iter__ yielded plain Document objects
while the matching viewer ids were exposed through a separate mutable
attribute (viewer_ids_by_pk), overwritten each time the generator crossed
a chunk boundary. rebuild() read that attribute out-of-band per document.
This only worked because the current iter_wrapper (a plain progress-bar
passthrough) happens to consume the stream in strict lock-step with no
lookahead. Any wrapper that buffers, batches, or reorders would silently
pair a document with the wrong chunk's viewer ids. Yield the pair directly
so the association travels with the document regardless of how iter_wrapper
consumes the stream, and drop the now-unneeded viewer_ids_by_pk attribute.
* Add --heap-size-mb CLI arg to document_index reindex
writer_heap_bytes was hardcoded at 512MB with no way to tune it. Expose it
as a manual-rebuild-only CLI arg rather than a settings/env var, per review
feedback, so lower-memory hosts can reduce it without a wider config
surface. Defaults to unset so TantivyBackend.rebuild's own default stays
the single source of truth.
---------
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
* Chore(beta): add sqlite-vec 0.1.9 dependency
Pinned exactly: the 0.1.9 wheels carry no baked SIMD flags (safe on
pre-AVX2 CPUs, the point of this migration); the 0.1.10 alphas bake
-mavx and would reintroduce the #12970 crash class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Test(beta): port vector store tests to sqlite-vec backend
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enhancement(beta): switch AI vector store from LanceDB to sqlite-vec
Fixes the non-AVX2 SIGILL class (#12970) at the root: lancedb is no
longer imported. sqlite-vec 0.1.9 wheels carry no baked SIMD, vec0
metadata columns give parameterized EQ/IN filtering, WAL preserves the
lock-free-reader model, and compact() rebuilds the table because vec0
DELETEs never reclaim space.
Implementation notes vs. the Task 3A draft:
- compact() uses a file-swap approach (new db file + Path.replace) rather
than ALTER TABLE RENAME, which does not cascade to shadow tables in
sqlite-vec 0.1.9 (upstream limitation).
- Bloat is tracked via a cumulative total_inserts counter in index_meta
because the _rowids shadow table does not accumulate deleted rows in
0.1.9 (contrary to the design doc assumption from #54).
- None distances from the zero-vector cosine edge case are mapped to
similarity 0.0 rather than raising TypeError.
- Test suite updated accordingly: _bloat_ratio reads index_meta instead
of _rowids; seed collision in force-compact test fixed (seed=100.0).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enhancement(beta): wire indexing pipeline to the sqlite-vec store
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enhancement(beta): move filename/storage path/ASN to node metadata
Same treatment as title/tags/correspondent in #12944: excluded from
the embedded text, visible to the LLM via metadata prepend. Changes
embedded text for every document, so it ships inside the sqlite-vec
transition, whose forced rebuild re-embeds everything anyway.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Test(beta): cover legacy LanceDB index cleanup and forced rebuild
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): drop lancedb dependency
Fixes#12970: the package whose wheels SIGILL on non-AVX2 CPUs is no
longer installed at all.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): partial pyrefly cleanup on sqlite-vec vector store
- Add MetadataFilter import and isinstance guard in _build_where()
- Add query_embedding None guard in query()
- Fix dict.get() type-checker ambiguity in get_configured_model_name()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): drop automatic LanceDB index cleanup on startup
Leave legacy Lance directory removal to the user rather than deleting it
automatically on first run. Beta policy: user is expected to do a clean
re-embed anyway; no need for the system to silently delete their data.
Remove _cleanup_legacy_lance_index(), the forced-rebuild path that called
it, and the associated tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): ruff format pass on sqlite-vec AI files
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Removes the benchmarking file
* Try to resolve or silence some semgrep. But we're using SQL here, not an ORM and we control the inputs, not users
* Enhancement(beta): add schema migration machinery to sqlite-vec vector store
Adds versioned schema migration support modelled after PR #12968's LanceDB
approach, adapted for sqlite-vec's file-swap compaction pattern.
- SCHEMA_VERSION = 1 written to index_meta at table creation and preserved
through compact()
- Migration dataclass with from_version, to_version, kind ("structural" or
"re-embed"), description, and an optional apply(src, dst, dim) callable
- MIGRATIONS registry (empty at v1 baseline); add entries and bump
SCHEMA_VERSION when the schema changes
- check_and_run_migrations(): structural migrations run via the same
file-swap as compact() (no re-embed); re-embed migrations return True
so the caller forces a full rebuild
- update_llm_index() calls check_and_run_migrations() under the write lock
before any indexing work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): deduplicate vector store internals via helper methods
Extract three helpers to remove copy-paste between compact() and
_run_structural_migration():
- _meta_set_on(conn, key, value): static upsert into any connection's
index_meta; _meta_set() now delegates to it
- _create_vec_table(conn, dim): CREATE VIRTUAL TABLE DDL (carries the
nosemgrep annotation)
- _swap_in_compact(compact_path, db_path): close/replace/reconnect
sequence used by both file-swap callers
Also normalises compact() error-path cleanup to unlink(missing_ok=True).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Adds equality test and no covers some defensive error handling stuff
* Ensures an embed migration stops the migration chain, just in case
* Silence one kind right but not really semgrep
* Trims dead assignment
* Fix(beta): address Copilot review on sqlite-vec vector store
Three findings from the PR review:
- compact() failure cleanup now unlinks the temporary .compact-wal and
.compact-shm files, matching _run_structural_migration(); previously
only the main .compact file was removed.
- _build_where() fails closed (1 = 0) when filters are requested but none
translate, instead of emitting "()" which is invalid SQL; filters scope
document access, so an empty translation must match no rows.
- Drop the unused table_name constructor parameter (all SQL hardcodes
DEFAULT_TABLE_NAME) and its callers in indexing.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Enhancement(beta): guard sqlite-vec compaction swap against concurrent readers
The compaction/migration file swap replaces the database via os.replace,
but the -wal/-shm files are keyed by path, not inode. A reader holding an
open connection across the swap leaves the old WAL aliased onto the new
file; a subsequent write then corrupts the database (reproduced via
PRAGMA integrity_check).
Add a cross-process read/write lock (filelock.ReadWriteLock) over the
index:
- read_store() holds it shared for the whole connection lifetime (and
closes the connection on exit); concurrent readers do not block.
- compaction and the migration check run under an exclusive lock that
drains readers, and skip with an info log on Timeout (maintenance op,
retries next run).
- Normal writes are untouched: WAL gives reader/writer concurrency and
LLM_INDEX_LOCK still serializes writers, so they never block readers.
load_or_build_index() now takes the store from the caller's read_store()
so the lock and connection span the whole retrieval; chat holds it across
the streamed response. Two new settings: LLM_INDEX_RWLOCK and
LLM_INDEX_COMPACTION_LOCK_TIMEOUT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Ensures the store alays cleans up SQLite connections for any operations, even on errors
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
stumpylog
shamoonandGitHub
GitHub Actions
Hugo Beauzée-LuyssenandGitHub
