* Fix: mail rule loops forever when all attachments are duplicates
When every attachment in a mail is rejected as a duplicate, the chord's
header tasks all fail. Celery's default task_allow_error_cb_on_chord_header
skips the error callback in that case, so no ProcessedMail row is ever
created, and the same mail is refetched and reprocessed on every poll for
as long as it stays in the rule's maximum_age window.
* Minor simplifications and cleanup
* Security: validate remote OCR endpoint against internal SSRF
Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.
* Validates te outbound url again right before use
* cover empty-value branch of validate_remote_ocr_endpoint because coverage
* re-validate remote OCR endpoint on every outbound request
* uses_remote_service + allow_remote to allow opt-in / out of remote OCR
* Add to parser dev docs
* remote_ocr_mode config setting
* Checks for remote_ocr_mode and fix import
* Update config.component.spec.ts
* More tests for remote_ocr_mode
* Docs for remote_ocr_mode
* Ok, wire up the remote_ocr_mode with allow_remote for consumer
* Update consumer.py
* Format remote OCR mode check tests
* Use get_choice_from_env
* Backend changes and migration for remote OCR Config
* Backend tests
* Frontend stuff, with sections
* Docs
* Update test_tesseract_parser.py
* Actually we cant use this any more, in case settings are in app config
* Dont mark entire test file for db, use a mock for empty engine settings
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks
* Updates the documentation to be correct about the respecting now
* merge conflict fixing
* Fix: unify born-digital PDF detection between archive decision and OCR
should_produce_archive() and RasterisedDocumentParser.parse() each
reimplemented the "does this PDF have real text" check independently,
using different normalization of pdftotext output. Raw pdftotext output
can be non-empty (whitespace/form-feed layout padding) even when there
is no real content, so the two checks could disagree: consumer.py
treated a tagged-but-textless PDF as born-digital and skipped the
archive, while the parser's own (stricter, normalized) check found no
text and ran OCR anyway, leaving the document with no archive despite
real OCR text (GH #13387).
Both call sites now share one predicate, pdf_born_digital_text() in
paperless/parsers/utils.py, so they can no longer drift apart.
* Fix: restore extract_text seam for born-digital detection in parse()
parse() had switched to calling pdf_born_digital_text() directly for
its initial text/born-digital check, bypassing the parser's own
extract_text instance method. That broke test mockability (tests patch
tesseract_parser.extract_text to control the born-digital decision)
and caused CI failures with mismatched OCR call counts and text.
Split pdf_born_digital_text() into is_born_digital_text(text, path,
log) - a pure decision function - and a thin pdf_born_digital_text()
wrapper for callers without text in hand (consumer.should_produce_archive).
parse() now extracts via self.extract_text(None, document_path) and
passes the result to is_born_digital_text(), restoring the seam with
no change to production behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Cleanup: simplify born-digital detection, close#13387 test gap
Simplification pass over the born-digital detection consolidation:
- is_born_digital_text(): drop the has_text temp for an early return.
- consumer.py: standardize the archive-decision log lines on plain
hyphens (was a mix of em-dash and hyphen) and hoist the duplicated
text_length computation.
- Parametrize TestPdfBornDigitalText instead of four near-identical
tests.
Code review follow-up: the existing tests only ever exercised
pdf_born_digital_text() through mocks, so the actual #13387 scenario
(a tagged PDF whose only "text" is layout padding) was never checked
against real pdftotext/pikepdf output - a regression in the
normalize-before-decide logic itself would have gone undetected.
Moved tagged_no_text_pdf_file from parsers/conftest.py up to the
shared paperless/tests/conftest.py (it was previously only visible to
tests under parsers/) and added a non-mocked regression test against
the real sample file.
Also fixed two stale comments in test_consumer.py referencing a
_extract_text_for_archive_check helper that no longer exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
Avoids a Postgres error where the clamp UPDATE and the following ALTER TABLE on paperless_mail_mailrule share a transaction, but the table's FKs are deferrable, so pending trigger events block the ALTER.
Also fixes a verbose_name mismatch in migration 0013.
* Chore(beta): add sqlite-vec 0.1.9 dependency
Pinned exactly: the 0.1.9 wheels carry no baked SIMD flags (safe on
pre-AVX2 CPUs, the point of this migration); the 0.1.10 alphas bake
-mavx and would reintroduce the #12970 crash class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Test(beta): port vector store tests to sqlite-vec backend
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enhancement(beta): switch AI vector store from LanceDB to sqlite-vec
Fixes the non-AVX2 SIGILL class (#12970) at the root: lancedb is no
longer imported. sqlite-vec 0.1.9 wheels carry no baked SIMD, vec0
metadata columns give parameterized EQ/IN filtering, WAL preserves the
lock-free-reader model, and compact() rebuilds the table because vec0
DELETEs never reclaim space.
Implementation notes vs. the Task 3A draft:
- compact() uses a file-swap approach (new db file + Path.replace) rather
than ALTER TABLE RENAME, which does not cascade to shadow tables in
sqlite-vec 0.1.9 (upstream limitation).
- Bloat is tracked via a cumulative total_inserts counter in index_meta
because the _rowids shadow table does not accumulate deleted rows in
0.1.9 (contrary to the design doc assumption from #54).
- None distances from the zero-vector cosine edge case are mapped to
similarity 0.0 rather than raising TypeError.
- Test suite updated accordingly: _bloat_ratio reads index_meta instead
of _rowids; seed collision in force-compact test fixed (seed=100.0).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enhancement(beta): wire indexing pipeline to the sqlite-vec store
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enhancement(beta): move filename/storage path/ASN to node metadata
Same treatment as title/tags/correspondent in #12944: excluded from
the embedded text, visible to the LLM via metadata prepend. Changes
embedded text for every document, so it ships inside the sqlite-vec
transition, whose forced rebuild re-embeds everything anyway.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Test(beta): cover legacy LanceDB index cleanup and forced rebuild
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): drop lancedb dependency
Fixes#12970: the package whose wheels SIGILL on non-AVX2 CPUs is no
longer installed at all.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): partial pyrefly cleanup on sqlite-vec vector store
- Add MetadataFilter import and isinstance guard in _build_where()
- Add query_embedding None guard in query()
- Fix dict.get() type-checker ambiguity in get_configured_model_name()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): drop automatic LanceDB index cleanup on startup
Leave legacy Lance directory removal to the user rather than deleting it
automatically on first run. Beta policy: user is expected to do a clean
re-embed anyway; no need for the system to silently delete their data.
Remove _cleanup_legacy_lance_index(), the forced-rebuild path that called
it, and the associated tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): ruff format pass on sqlite-vec AI files
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Removes the benchmarking file
* Try to resolve or silence some semgrep. But we're using SQL here, not an ORM and we control the inputs, not users
* Enhancement(beta): add schema migration machinery to sqlite-vec vector store
Adds versioned schema migration support modelled after PR #12968's LanceDB
approach, adapted for sqlite-vec's file-swap compaction pattern.
- SCHEMA_VERSION = 1 written to index_meta at table creation and preserved
through compact()
- Migration dataclass with from_version, to_version, kind ("structural" or
"re-embed"), description, and an optional apply(src, dst, dim) callable
- MIGRATIONS registry (empty at v1 baseline); add entries and bump
SCHEMA_VERSION when the schema changes
- check_and_run_migrations(): structural migrations run via the same
file-swap as compact() (no re-embed); re-embed migrations return True
so the caller forces a full rebuild
- update_llm_index() calls check_and_run_migrations() under the write lock
before any indexing work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Chore(beta): deduplicate vector store internals via helper methods
Extract three helpers to remove copy-paste between compact() and
_run_structural_migration():
- _meta_set_on(conn, key, value): static upsert into any connection's
index_meta; _meta_set() now delegates to it
- _create_vec_table(conn, dim): CREATE VIRTUAL TABLE DDL (carries the
nosemgrep annotation)
- _swap_in_compact(compact_path, db_path): close/replace/reconnect
sequence used by both file-swap callers
Also normalises compact() error-path cleanup to unlink(missing_ok=True).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Adds equality test and no covers some defensive error handling stuff
* Ensures an embed migration stops the migration chain, just in case
* Silence one kind right but not really semgrep
* Trims dead assignment
* Fix(beta): address Copilot review on sqlite-vec vector store
Three findings from the PR review:
- compact() failure cleanup now unlinks the temporary .compact-wal and
.compact-shm files, matching _run_structural_migration(); previously
only the main .compact file was removed.
- _build_where() fails closed (1 = 0) when filters are requested but none
translate, instead of emitting "()" which is invalid SQL; filters scope
document access, so an empty translation must match no rows.
- Drop the unused table_name constructor parameter (all SQL hardcodes
DEFAULT_TABLE_NAME) and its callers in indexing.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Enhancement(beta): guard sqlite-vec compaction swap against concurrent readers
The compaction/migration file swap replaces the database via os.replace,
but the -wal/-shm files are keyed by path, not inode. A reader holding an
open connection across the swap leaves the old WAL aliased onto the new
file; a subsequent write then corrupts the database (reproduced via
PRAGMA integrity_check).
Add a cross-process read/write lock (filelock.ReadWriteLock) over the
index:
- read_store() holds it shared for the whole connection lifetime (and
closes the connection on exit); concurrent readers do not block.
- compaction and the migration check run under an exclusive lock that
drains readers, and skip with an info log on Timeout (maintenance op,
retries next run).
- Normal writes are untouched: WAL gives reader/writer concurrency and
LLM_INDEX_LOCK still serializes writers, so they never block readers.
load_or_build_index() now takes the store from the caller's read_store()
so the lock and connection span the whole retrieval; chat holds it across
the streamed response. Two new settings: LLM_INDEX_RWLOCK and
LLM_INDEX_COMPACTION_LOCK_TIMEOUT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Ensures the store alays cleans up SQLite connections for any operations, even on errors
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Move LLM index lock outside index dir and skip per-doc tasks on bulk update
Two concurrency bugs from #12893:
[P1] Lock file lived inside LLM_INDEX_DIR. A rebuild calls
shutil.rmtree(LLM_INDEX_DIR), deleting the lock while a worker still
held it. A second worker then acquired a fresh lock on the new path and
ran concurrently, defeating serialisation. Move the lock to
DATA_DIR/locks/llm_index.lock (a new settings constant LLM_INDEX_LOCK)
so rmtree cannot touch it. The locks/ dir is created at settings load
time, matching the existing pattern for LOGGING_DIR.
[P2] document_updated was connected to add_or_update_document_in_llm_index
in apps.py. bulk_update_documents() emits document_updated for every
document in the batch, queuing N per-document LLM tasks, and then also
calls update_llm_index(rebuild=False) once at the end. Pass
skip_ai_index=True when sending document_updated from the bulk path so
the handler skips the per-document enqueue; the existing batch call at
the end of bulk_update_documents is the only LLM update for that path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: ghost vectors leave KeyError-prone nodes_dict entries after deletion
docstore.delete_document() removes a node from the docstore but leaves its
entry in index_struct.nodes_dict (the FAISS positional-id to node-UUID map).
A subsequent similarity query resolves the ghost position to the deleted UUID,
finds nothing in fetched_nodes_by_id, and raises KeyError inside
_insert_fetched_nodes_into_query_result.
Purge stale nodes_dict entries after each docstore deletion and re-sync the
mutated index_struct into the kvstore so persist() writes the updated mapping.
Dead FAISS vectors remain in the flat index until the next full rebuild
(IndexFlatL2 is append-only); add a try/except KeyError around
retriever.retrieve() as a defensive fallback for any residual ghost positions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: acquire index lock in query_similar_documents
query_similar_documents() loaded the index and ran the FAISS retriever
without holding the file lock. All write paths (update_llm_index,
llm_index_add_or_update_document, llm_index_remove_document) hold
FileLock(_index_lock_path()), so a concurrent rebuild calling
shutil.rmtree(LLM_INDEX_DIR) while a read is mid-load produces an IOError
or corrupt partial state.
Wrap the load_or_build_index() call and all subsequent retriever work inside
FileLock. The early-return guards (vector_store_file_exists check, empty
allowed_document_ids) remain outside the lock; the DB query for the final
result set also stays outside.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: skip LLM index enqueue on document_updated during version addition
When a document is consumed as a new version of an existing document, the
consumer fires document_consumption_finished (which triggers
add_or_update_document_in_llm_index) and then document_updated for the root
document. Both signals are connected to the same handler, so the root document
was enqueued for LLM indexing twice per version-addition event.
Pass skip_ai_index=True on the consumer's version-addition document_updated
send so the handler's existing guard suppresses the duplicate enqueue.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Test: bulk_update_documents must not enqueue per-doc LLM tasks
With AI enabled, bulk_update_documents() sends document_updated for every
document in the batch. The skip_ai_index=True kwarg (added in the P2 fix)
prevents add_or_update_document_in_llm_index from enqueuing a per-document
task for each one. Only the single update_llm_index call at the end should run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Debug level log sure
* Update src/paperless_ai/indexing.py
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Apply suggestion from @shamoon
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
dependabot[bot]andTrenton Holmes
shamoon
Thomas Steinbachandshamoon
Sebastian Wielandandshamoon
