Prompted by an automated simplification review of the branch. Applies the
highest-value findings, all behavior-preserving (full suite still green):
- vector_store.py: extract _rebuild_file() (temp-file lifecycle: open,
populate, swap-in-or-discard-on-failure) and _rebuild_into() (create
table, copy meta, stream rows) out of compact() and the v1->v2
migration, which had duplicated both almost verbatim. The migration
module shrinks to a single _rebuild_into() call instead of reaching
into four PaperlessSqliteVecVectorStore privates.
- vector_store.py: extract _stored_schema_version() out of
has_pending_migration() and check_and_run_migrations(), which
duplicated the same schema_version read and its "missing key means
current" default.
- indexing.py: extract _with_exclusive_access(), collapsing three
identical _exclude_readers()/Timeout/log-and-skip blocks (compaction
in update_llm_index() and llm_index_compact(), the migration check in
_check_and_run_migrations()) to one line each. Also trims
_check_and_run_migrations()'s docstring, which had grown to restate
content already documented on has_pending_migration() and
check_and_run_migrations(), including a claim that was now stale
(llm_index_migrate() is a second consumer of the re-embed signal).
- test_ai_indexing.py: add a mock_store fixture for the
write_store()-yields-a-MagicMock pattern that 8 tests were hand-rolling
identically.
- migrations/__init__.py: consolidate the "how to add a migration"
instructions to one place instead of three.
- docs/administration.md: fix two now-inaccurate claims -- the bare-metal
migrate step doesn't run "automatically" (it's the manual step being
documented), and the self-contradictory "if enabled... no-op if
disabled" phrasing on the same step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Move schema migrations into their own paperless_ai/migrations package,
one module per migration (mNNNN_description.py -- a leading digit isn't
a valid Python identifier, unlike Django's own numbered migrations,
which load via importlib.import_module() rather than a static import
statement), establishing the pattern before more migrations accumulate
in vector_store.py.
- Drop the redundant "if the LLM index is enabled..." clause from the
Docker upgrade note in docs/administration.md -- it's already covered
by the LLM index section below, and calling out just one of several
auto-applied migrations there was incomplete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add document_llmindex migrate, a cheap check-only path (no reindex) safe
to run unconditionally: has_pending_migration() short-circuits to a
metadata-only read once the store is current, so a healthy install pays
almost nothing. If a pending migration would require re-embedding, it
only logs a warning and leaves the index as-is -- re-embedding can be
slow and, for a metered embedding backend, cost money, so it stays a
deliberate manual action (document_llmindex rebuild), never automatic.
Wire it into the Docker image as a new init-llmindex-migrate oneshot
(modeled on init-search-index, gated on init-migrations), and document
the equivalent manual step for bare-metal upgrades.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Replaces loaddata with streaming bulk_create
Replaces call_command('loaddata') with a streaming implementation that
reads manifest records one at a time via ijson, accumulates per-model
batches up to --batch-size, and flushes via bulk_create. This reduces
peak memory and no longer scales directly with the size of the import.
* fix(importer): avoid guardian lru_cache poisoning; include M2M through tables in check_constraints
clear_cache() inside the import transaction emptied Django's ContentType
manager cache while fixture PKs were live, causing downstream ContentType
lookups to repopulate guardian's separate @lru_cache(None) with
fixture-PK objects. After the TestCase transaction rolled back to
original PKs, guardian's lru_cache held stale fixture ContentType
objects, causing MixedContentTypeError in unrelated subsequent tests.
Remove clear_cache() since it was defending against a theoretical
stale-cache scenario that doesn't occur in a proper same-install restore.
Fix check_constraints() to explicitly include auto-created M2M through
tables (populated by .set() after bulk_create) alongside the model tables,
addressing the gap where join-table FK violations would have gone
undetected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Excludes the consumer and AnonymousUser from any models which might have a FK relation to it. This prevents orphan things like UI setting, which have a relation to no existing user
* Splits into more sub functions for Sonar
* Improvements to the typing of the new functions
* Coverage for some error cases, and removes handling for pk only models. No need to support these
* Final coverage gaps
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Replaces references to docker-compose (the v1 executable) with docker compose (the v2 plugin) as well as fixing up some referenes between the tool vs the command
* Update docs/setup.md
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Replaces references to docker-compose (the v1 executable) with docker compose (the v2 plugin) as well as fixing up some referenes between the tool vs the command
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
stumpylogandClaude Sonnet 5
Katrin LeinweberandGitHub
KilianandGitHub
Malte BreitzmannandGitHub
Kevin DorenandGitHub
Sebastian SteinbeißerandGitHub
omahsandGitHub
RockandGitHub
mechanarchyandshamoon
andstu
Max Truxa