Perf: dedupe table_exists() lookups, atomic insert counter, fewer connections in update_llm_index()

Fix: strengthen two vacuous Task 4 regression tests
test_migration_never_delegates_to_current_schema_helpers never actually ran the migration (missing check_and_run_migrations() call) and its source-text assertion was tautological (the "or DROP TABLE in source" clause was always true). Now runs the real migration and asserts spy call counts instead: DocumentChunksTable.create/DocumentMetaTable.create are each called exactly 3 times (construction, rebuild temp file, post-swap reopen -- all via _open_connection, never from inside apply()), and _create_vec_table is never called from the migration path. test_drop_table_clears_modified_times asserted via get_modified_times(), which short-circuits on table_exists() -- checking only the vec0 table that drop_table() drops first -- so the assertion held even if DocumentMetaTable.delete_all() were never called. Now asserts directly against document_meta and document_chunks row counts.
2026-07-29 23:34:56 +00:00 · 2026-07-29 15:16:08 -07:00 · 2026-07-29 14:55:10 -07:00 · 2026-07-29 14:38:23 -07:00 · 2026-07-29 13:58:57 -07:00 · 2026-07-29 13:51:32 -07:00
28 changed files with 2022 additions and 1672 deletions
@@ -3,7 +3,7 @@

 declare -r log_prefix="[init-llmindex-migrate]"

-echo "${log_prefix} Checking LLM index schema..."
+echo "${log_prefix} Checking for pending LLM index migrations..."
 cd "${PAPERLESS_SRC_DIR}"
 if [[ -n "${USER_IS_NON_ROOT}" ]]; then
 	python3 manage.py document_llmindex migrate
@@ -212,15 +212,15 @@ following:
    This is a no-op if the index is already up to date, so it is safe to
    run on every upgrade.

-5.  Apply any pending LLM index schema migrations.
+5.  Migrate the LLM index if needed.

    ```shell-session
    cd src
    python3 manage.py document_llmindex migrate
    ```

-    This is a no-op if the index is already up to date, or if the LLM index
-    is disabled, so it is safe to run on every upgrade.
+    This is a no-op if the index schema is already current, so it is safe
+    to run on every upgrade.

 ### Database Upgrades

@@ -554,15 +554,6 @@ scheduled task runs.

 Specify `compact` to reclaim space and optimize the on-disk vector store.

-Specify `migrate` to apply any pending index schema migrations without a full reindex.
-This is a no-op if the index is already up to date, so it is safe to run on every
-startup or upgrade; the container's startup sequence runs it automatically, and the
-[bare-metal upgrade steps](#bare-metal-updating) include it as a manual step. If a
-pending migration would require re-embedding every document, `migrate` only logs a
-warning and leaves the index as-is -- re-embedding can be slow and, for a metered
-embedding backend, cost money, so it is never triggered automatically. Run `rebuild`
-yourself when you are ready.
-
 !!! note

    These commands have no effect unless AI is enabled and an embedding backend is
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
 If a "Move to Trash" action is executed in a consume pipeline, the consumption
 will be aborted and the file will be deleted.

+##### Password Removal {#workflow-action-password-removal}
+
+"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
+
+- One or more passwords to try, separated by commas or new lines
+- Each password is tried in order until one successfully unlocks the document
+
+Password removal never modifies a file in place. Instead, once a working password is found, the
+decrypted content is consumed as a new [document version](#document-file-versions), leaving the
+original (still encrypted) version in the document's version history.
+
+**Consumption Started**: because this trigger fires before the document exists yet, the password
+removal itself is deferred until after the initial consumption of the encrypted file has completed.
+OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
+extracted text (unless the file already contained extractable text outside of OCR). Immediately
+afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
+new version of the same document, this time with normal OCR/text extraction applied. In other words,
+a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
+the properly processed version is created.
+
+**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
+already exists, so password removal happens immediately: the decrypted content is queued for
+consumption as a new version right away. Note that if the document's initial consumption also
+happened while it was still encrypted, that original version will likewise be missing OCR text.
+
+**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
+multiple PDF types with different passwords, create separate workflows for each use case.
+
 #### Workflow placeholders

 Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -659,7 +659,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">445</context>
+          <context context-type="linenumber">449</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
@@ -831,7 +831,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">464</context>
+          <context context-type="linenumber">468</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -1355,7 +1355,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">390</context>
+          <context context-type="linenumber">394</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1596,7 +1596,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">165</context>
+          <context context-type="linenumber">169</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2691296884221415710" datatype="html">
@@ -1607,7 +1607,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">170</context>
+          <context context-type="linenumber">174</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1638,7 +1638,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">174</context>
+          <context context-type="linenumber">178</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1669,7 +1669,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">178</context>
+          <context context-type="linenumber">182</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -4914,7 +4914,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">356</context>
+          <context context-type="linenumber">360</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8057014866157903311" datatype="html">
@@ -7760,14 +7760,14 @@
        <source>Details</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">160</context>
+          <context context-type="linenumber">164</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5701618810648052610" datatype="html">
        <source>Title</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">163</context>
+          <context context-type="linenumber">167</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -7790,14 +7790,14 @@
        <source>Date created</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">167</context>
+          <context context-type="linenumber">171</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5607669932062416162" datatype="html">
        <source>Default</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">179</context>
+          <context context-type="linenumber">183</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
@@ -7808,14 +7808,14 @@
        <source>Content</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">286</context>
+          <context context-type="linenumber">290</context>
        </context-group>
      </trans-unit>
      <trans-unit id="218403386307979629" datatype="html">
        <source>Metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">295</context>
+          <context context-type="linenumber">299</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
@@ -7826,147 +7826,147 @@
        <source>Date modified</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">302</context>
+          <context context-type="linenumber">306</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6392918669949841614" datatype="html">
        <source>Date added</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">306</context>
+          <context context-type="linenumber">310</context>
        </context-group>
      </trans-unit>
      <trans-unit id="146828917013192897" datatype="html">
        <source>Media filename</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">310</context>
+          <context context-type="linenumber">314</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4500855521601039868" datatype="html">
        <source>Original filename</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">314</context>
+          <context context-type="linenumber">318</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2659735245739197634" datatype="html">
        <source>Original SHA256 checksum</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">318</context>
+          <context context-type="linenumber">322</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5888243105821763422" datatype="html">
        <source>Original file size</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">322</context>
+          <context context-type="linenumber">326</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2696647325713149563" datatype="html">
        <source>Original mime type</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">326</context>
+          <context context-type="linenumber">330</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6714358112223607756" datatype="html">
        <source>Archive SHA256 checksum</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">331</context>
+          <context context-type="linenumber">335</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6033581412811562084" datatype="html">
        <source>Archive file size</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">337</context>
+          <context context-type="linenumber">341</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6992781481378431874" datatype="html">
        <source>Original document metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">346</context>
+          <context context-type="linenumber">350</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2846565152091361585" datatype="html">
        <source>Archived document metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">349</context>
+          <context context-type="linenumber">353</context>
        </context-group>
      </trans-unit>
      <trans-unit id="7206723502037428235" datatype="html">
        <source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">368,371</context>
+          <context context-type="linenumber">372,375</context>
        </context-group>
      </trans-unit>
      <trans-unit id="186236568870281953" datatype="html">
        <source>History</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">379</context>
+          <context context-type="linenumber">383</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8236092845697214347" datatype="html">
        <source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="cate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/></source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">401,405</context>
+          <context context-type="linenumber">405,409</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6449374629822973702" datatype="html">
        <source>Duplicate documents detected:</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">407</context>
+          <context context-type="linenumber">411</context>
        </context-group>
      </trans-unit>
      <trans-unit id="14058600336670816" datatype="html">
        <source>In trash</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">418</context>
+          <context context-type="linenumber">422</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5129524307369213584" datatype="html">
        <source>Save &amp; next</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">447</context>
+          <context context-type="linenumber">451</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4910102545766233758" datatype="html">
        <source>Save &amp; close</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">449</context>
+          <context context-type="linenumber">453</context>
        </context-group>
      </trans-unit>
      <trans-unit id="3823219296477075982" datatype="html">
        <source>Discard</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">451</context>
+          <context context-type="linenumber">455</context>
        </context-group>
      </trans-unit>
      <trans-unit id="1309556917227148591" datatype="html">
        <source>Document loading...</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">459</context>
+          <context context-type="linenumber">463</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8191371354890763172" datatype="html">
        <source>Enter Password</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">513</context>
+          <context context-type="linenumber">517</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5758784066858623886" datatype="html">
@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
    ;(component as any).applyScale()
    expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
    expect(viewer.currentScale).toBe(2)
+  })

+  it('does not reapply scale for page-only changes', async () => {
+    await initComponent()
+
+    const pdf = (component as any).pdf as { numPages: number }
+    pdf.numPages = 3
+    const viewer = (component as any).pdfViewer as PDFViewer
+    viewer.setDocument(pdf)
    const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
    component.page = 2
-    ;(component as any).lastViewerPage = 2
-    ;(component as any).applyViewerState()
+
+    component.ngOnChanges({
+      page: new SimpleChange(1, 2, false),
+    })
+
+    expect(viewer.currentPageNumber).toBe(2)
    expect((component as any).lastViewerPage).toBeUndefined()
-    expect(applyScaleSpy).toHaveBeenCalled()
+    expect(applyScaleSpy).not.toHaveBeenCalled()
  })

  it('does not reset the viewer when it is already on the requested page', async () => {
@@ -116,7 +116,10 @@ export class PngxPdfViewerComponent
      changes['zoomScale'] ||
      changes['rotation']
    ) {
-      this.applyViewerState()
+      // Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
+      this.applyViewerState(
+        !!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
+      )
    }

    if (changes['searchQuery']) {
@@ -240,7 +243,7 @@ export class PngxPdfViewerComponent
    }
  }

-  private applyViewerState(): void {
+  private applyViewerState(applyScale = true): void {
    if (!this.pdfViewer) {
      return
    }
@@ -264,7 +267,7 @@ export class PngxPdfViewerComponent
    if (this.page === this.lastViewerPage) {
      this.lastViewerPage = undefined
    }
-    if (hasPages) {
+    if (hasPages && applyScale) {
      this.applyScale()
    }
    this.dispatchFindIfReady()
@@ -113,8 +113,8 @@

    <form [formGroup]='documentForm' (ngSubmit)="save()">

-      <div class="btn-toolbar mb-1 border-bottom">
-        <div class="btn-group pb-3">
+      <div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
+        <div class="btn-group me-auto">
          <button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
            <i-bs width="1.2em" height="1.2em" name="x"></i-bs>
          </button>
@@ -127,32 +127,36 @@
        </div>

        <ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
-          <div class="btn-group pb-3 ms-auto">
-            <pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
-              [disabled]="!userCanEdit || suggestionsLoading()"
-              [loading]="suggestionsLoading()"
-              [suggestions]="suggestions()"
-              [aiEnabled]="aiEnabled"
-              (getSuggestions)="getSuggestions()"
-              (addTag)="createTag($event)"
-              (addDocumentType)="createDocumentType($event)"
-              (addCorrespondent)="createCorrespondent($event)">
-            </pngx-suggestions-dropdown>
+          <div class="d-flex gap-2">
+            <div class="btn-group">
+              <pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
+                [disabled]="!userCanEdit || suggestionsLoading()"
+                [loading]="suggestionsLoading()"
+                [suggestions]="suggestions()"
+                [aiEnabled]="aiEnabled"
+                (getSuggestions)="getSuggestions()"
+                (addTag)="createTag($event)"
+                (addDocumentType)="createDocumentType($event)"
+                (addCorrespondent)="createCorrespondent($event)">
+              </pngx-suggestions-dropdown>
+            </div>
+
+            <div class="btn-group">
+              <pngx-custom-fields-dropdown
+                *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
+                [documentId]="documentId()"
+                [disabled]="!userCanEdit"
+                [existingFields]="document()?.custom_fields"
+                (created)="refreshCustomFields()"
+                (added)="addField($event)">
+              </pngx-custom-fields-dropdown>
+            </div>
          </div>

-          <div class="btn-group pb-3 ms-2">
-            <pngx-custom-fields-dropdown
-              *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
-              [documentId]="documentId()"
-              [disabled]="!userCanEdit"
-              [existingFields]="document()?.custom_fields"
-              (created)="refreshCustomFields()"
-              (added)="addField($event)">
-            </pngx-custom-fields-dropdown>
+          <div class="ps-3">
+            <ng-container *ngTemplateOutlet="saveButtons"></ng-container>
          </div>
        </ng-container>
-
-        <ng-container *ngTemplateOutlet="saveButtons"></ng-container>
      </div>

      <ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
@@ -276,7 +280,7 @@
              }
            </div>

-            <div class="d-flex border-top pt-3">
+            <div class="d-flex justify-content-end border-top pt-3">
              <ng-container *ngTemplateOutlet="saveButtons"></ng-container>
            </div>
          </ng-template>
@@ -440,7 +444,7 @@
 </div>

 <ng-template #saveButtons>
-  <div class="btn-group pb-3 ms-4">
+  <div class="btn-group">
    <ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
      <button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
      @if (hasNext()) {
@@ -36,6 +36,9 @@ def send_email(

    TODO: re-evaluate this pending https://code.djangoproject.com/ticket/35581 / https://github.com/django/django/pull/18966
    """
+    if "\r" in subject or "\n" in subject:
+        subject = " ".join(line.strip(" \t") for line in subject.splitlines())
+
    email = EmailMessage(
        subject=subject,
        body=body,
@@ -386,10 +386,19 @@ class Command(CryptMixin, PaperlessCommand):
                raise DeserializationError(
                    f"{model.__name__} has no updatable fields; PK-only models are not supported by the importer",
                )
+            # MySQL/MariaDB support upserts via ON DUPLICATE KEY UPDATE but,
+            # unlike PostgreSQL/SQLite, cannot target a specific unique field
+            # for the conflict -- passing unique_fields there raises
+            # NotSupportedError.
+            unique_fields = (
+                [model._meta.pk.attname]
+                if connection.features.supports_update_conflicts_with_target
+                else None
+            )
            model.objects.bulk_create(  # type: ignore[attr-defined]
                instances,
                update_conflicts=True,
-                unique_fields=[model._meta.pk.attname],
+                unique_fields=unique_fields,
                update_fields=update_fields,
            )
            loaded_models.add(model)
@@ -43,8 +43,16 @@ def _fmt(dt: datetime) -> str:


 def _iso_range(lo: datetime, hi: datetime) -> str:
-    """Format a [lo TO hi] range string in ISO 8601 for Tantivy query syntax."""
-    return f"[{_fmt(lo)} TO {_fmt(hi)}]"
+    """
+    Format a half-open ``[lo TO hi)`` range in ISO 8601 for Tantivy query syntax.
+
+    ``hi`` is always the exclusive ceiling of a computed period (the start of
+    the *next* day/week/month/quarter/year), so the closing bracket must be
+    the Tantivy exclusive-range brace ``}`` rather than ``]`` — otherwise the
+    first instant of the following period (e.g. the 1st of next month) is
+    incorrectly included in the match.
+    """
+    return f"[{_fmt(lo)} TO {_fmt(hi)}}}"


 def _quarter_start(d: date) -> date:
@@ -566,6 +566,17 @@ def translate_range(field: str, lo: str, hi: str, tz: tzinfo) -> str:
        lo_pair, hi_pair = hi_pair, lo_pair

    lo_iso = _fmt(lo_pair[0]) if lo_pair is not None else OPEN_LO
-    hi_iso = _fmt(hi_pair[1]) if hi_pair is not None else OPEN_HI

-    return f"{field}:[{lo_iso} TO {hi_iso}]"
+    # A bound resolves to (floor, ceil) where floor == ceil for an exact instant
+    # (a full ISO datetime, "now", or a "+/-N unit" offset) and floor != ceil for
+    # a coarser period token (year/month/day precision). Only the latter needs a
+    # half-open close: its ceil is the start of the *next* period and must be
+    # excluded, or that instant (e.g. the 1st of next month) wrongly matches.
+    if hi_pair is not None:
+        hi_iso = _fmt(hi_pair[1])
+        hi_close = "]" if hi_pair[0] == hi_pair[1] else "}"
+    else:
+        hi_iso = OPEN_HI
+        hi_close = "]"
+
+    return f"{field}:[{lo_iso} TO {hi_iso}{hi_close}"
@@ -1134,6 +1134,11 @@ def before_task_publish_handler(
        return

    try:
+        # Close stale connections without disrupting a transaction publishing a task
+        for connection in connections.all(initialized_only=True):
+            if not connection.in_atomic_block:
+                connection.close_if_unusable_or_obsolete()
+
        _, task_kwargs, _ = body
        task_id = headers["id"]

@@ -8,8 +8,8 @@ if TYPE_CHECKING:
    from pytest_mock import MockerFixture

 _COMPACT = "documents.management.commands.document_llmindex.llm_index_compact"
-_MIGRATE = "documents.management.commands.document_llmindex.llm_index_migrate"
 _INDEX = "documents.management.commands.document_llmindex.llmindex_index"
+_MIGRATE = "documents.management.commands.document_llmindex.llm_index_migrate"


 class TestDocumentLlmindexCommand:
@@ -32,7 +32,9 @@ AUCKLAND = ZoneInfo("Pacific/Auckland")  # UTC+13 in southern-hemisphere summer


 def _range(result: str, field: str) -> tuple[str, str]:
-    m = re.search(rf"{field}:\[(.+?) TO (.+?)\]", result)
+    # Half-open period ranges close with "}" (exclusive); exact-instant ranges
+    # (full ISO datetimes, "now", relative offsets) close with "]" (inclusive).
+    m = re.search(rf"{field}:\[(.+?) TO (.+?)[\]}}]", result)
    assert m, f"No range for {field!r} in: {result!r}"
    return m.group(1), m.group(2)

@@ -214,27 +214,27 @@ class TestTranslateScalar:
            (
                "created",
                "2020",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
            (
                "created",
                "202003",
-                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
+                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
            ),
            (
                "created",
                "20200115",
-                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
+                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
            ),
            (
                "created",
                "2020-01-15",
-                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
+                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
            ),
            (
                "created",
                "2020-03",
-                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
+                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
            ),
        ],
    )
@@ -248,9 +248,9 @@ class TestTranslateScalar:
        assert exc_info.value.value == "202023"

    def test_keyword_delegates(self) -> None:
-        # keyword path produces a range; just assert it is a created range
+        # keyword path produces a half-open range; just assert it is a created range
        out = translate_scalar("created", "today", UTC)
-        assert out.startswith("created:[") and out.endswith("]")
+        assert out.startswith("created:[") and out.endswith("}")

    def test_14digit_compact_datetime(self) -> None:
        out = translate_scalar("created", "20240115120000", UTC)
@@ -279,21 +279,21 @@ class TestTranslateRange:
    @pytest.mark.parametrize(
        ("lo", "hi", "expected"),
        [
-            ("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"),
+            ("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"),
            (
                "202001",
                "202006",
-                "created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z}",
            ),
            (
                "20200101",
                "20201231",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
            (
                "2020-01-01",
                "2020-12-31",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
        ],
    )
@@ -302,7 +302,7 @@ class TestTranslateRange:

    def test_reversed_swaps(self):
        assert translate_range("created", "2009", "2005", UTC) == (
-            "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"
+            "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"
        )

    def test_open_upper(self):
@@ -311,7 +311,7 @@ class TestTranslateRange:

    def test_open_lower(self):
        out = translate_range("created", "", "2020", UTC)
-        assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z]"
+        assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z}}"

    def test_invalid_bound_raises(self):
        with pytest.raises(InvalidDateQuery) as exc_info:
@@ -334,16 +334,16 @@ class TestTranslateQuery:
        [
            (
                "created:2020",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
            ("tag:foo,bar", "tag:foo AND tag:bar"),
            # 'type' is a user-facing alias rewritten to 'document_type' (the real schema field)
            ("tag:foo,type:bar", "tag:foo AND document_type:bar"),
            (
                "created:[2020 TO 2021],added:[2022 TO 2023]",
-                "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
+                "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
                " AND "
-                "added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z]",
+                "added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z}",
            ),
            # correspondent is not multi-value: comma stays literal inside the value
            ("correspondent:foo,bar", "correspondent:foo,bar"),
@@ -506,7 +506,7 @@ class TestOperatorNormalization:
    def test_date_range_preserved(self) -> None:
        out = translate_query("created:[2020 TO 2021]", UTC)
        # Must not corrupt the ISO range
-        assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
+        assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"

    def test_date_scalar_with_or(self) -> None:
        out = translate_query("created:2020 OR foo", UTC)
@@ -581,42 +581,42 @@ class TestKeywordDateResolution:
        [
            pytest.param(
                "today",
-                "created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z]",
+                "created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z}",
                id="today",
            ),
            pytest.param(
                "yesterday",
-                "created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z]",
+                "created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z}",
                id="yesterday",
            ),
            pytest.param(
                "previous week",
-                "created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z]",
+                "created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z}",
                id="previous-week",
            ),
            pytest.param(
                "this month",
-                "created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z]",
+                "created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z}",
                id="this-month",
            ),
            pytest.param(
                "previous month",
-                "created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z]",
+                "created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z}",
                id="previous-month",
            ),
            pytest.param(
                "this year",
-                "created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z]",
+                "created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z}",
                id="this-year",
            ),
            pytest.param(
                "previous year",
-                "created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
+                "created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
                id="previous-year",
            ),
            pytest.param(
                "previous quarter",
-                "created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
+                "created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
                id="previous-quarter",
            ),
        ],
@@ -637,42 +637,42 @@ class TestKeywordDateResolution:
        [
            pytest.param(
                "today",
-                "added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z]",
+                "added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z}",
                id="today",
            ),
            pytest.param(
                "yesterday",
-                "added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z]",
+                "added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z}",
                id="yesterday",
            ),
            pytest.param(
                "previous week",
-                "added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z]",
+                "added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z}",
                id="previous-week",
            ),
            pytest.param(
                "this month",
-                "added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z]",
+                "added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z}",
                id="this-month",
            ),
            pytest.param(
                "previous month",
-                "added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z]",
+                "added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z}",
                id="previous-month",
            ),
            pytest.param(
                "this year",
-                "added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z]",
+                "added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z}",
                id="this-year",
            ),
            pytest.param(
                "previous year",
-                "added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z]",
+                "added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z}",
                id="previous-year",
            ),
            pytest.param(
                "previous quarter",
-                "added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z]",
+                "added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z}",
                id="previous-quarter",
            ),
        ],
@@ -719,7 +719,7 @@ class TestISODatetimeBounds:
    def test_translate_query_text_before_comma_separated_date_clause(self) -> None:
        result = translate_query("schäfersee,created:previous year", UTC)
        assert result == (
-            "schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]"
+            "schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}"
        )

    def test_invalid_iso_datetime_raises(self) -> None:
@@ -75,7 +75,7 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
                {
                    "documents": [self.doc1.pk, self.doc2.pk],
                    "addresses": "hello@paperless-ngx.com,test@example.com",
-                    "subject": "Bulk email test",
+                    "subject": "Bulk email\n test",
                    "message": "Here are your documents",
                },
            ),
@@ -720,6 +720,48 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
        self.assertEqual(results[0]["id"], 3)
        self.assertEqual(results[0]["title"], "bank statement 3")

+    def test_search_added_previous_month_excludes_next_period_start(self) -> None:
+        """
+        GIVEN:
+            - One document added at the last instant of last month
+            - One document added exactly at the first instant of this month
+        WHEN:
+            - Query for documents added in the previous month
+        THEN:
+            - Only the document from last month is returned; the document dated
+              exactly at the start of this month (the exclusive upper bound of
+              the range) is not
+        """
+        d1 = DocumentFactory.create(
+            title="end of last month",
+            content="last instant of last month",
+            checksum="A",
+            pk=1,
+            added=timezone.make_aware(datetime.datetime(2024, 1, 31, 23, 59, 59)),
+        )
+        d2 = DocumentFactory.create(
+            title="start of this month",
+            content="first instant of this month",
+            checksum="B",
+            pk=2,
+            added=timezone.make_aware(datetime.datetime(2024, 2, 1, 0, 0, 0)),
+        )
+
+        backend = get_backend()
+        backend.add_or_update(d1)
+        backend.add_or_update(d2)
+
+        with time_machine.travel(
+            timezone.make_aware(datetime.datetime(2024, 2, 15, 12, 0, 0)),
+            tick=False,
+        ):
+            response = self.client.get("/api/documents/?query=added:previous month")
+        results = response.data["results"]
+
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0]["id"], 1)
+        self.assertEqual(results[0]["title"], "end of last month")
+
    def test_search_added_invalid_date(self) -> None:
        """
        GIVEN:
@@ -56,6 +56,32 @@ def send_publish(

@pytest.mark.django_db
 class TestBeforeTaskPublishHandler:
+    @mock.patch("documents.signals.handlers.connections.all")
+    def test_closes_old_connections_outside_atomic_blocks(
+        self,
+        connections_all,
+    ) -> None:
+        connection = mock.Mock(in_atomic_block=False)
+        connections_all.return_value = [connection]
+
+        task_id = send_publish("documents.tasks.train_classifier", (), {})
+
+        connection.close_if_unusable_or_obsolete.assert_called_once_with()
+        assert PaperlessTask.objects.filter(task_id=task_id).exists()
+
+    @mock.patch("documents.signals.handlers.connections.all")
+    def test_keeps_connections_open_inside_atomic_blocks(
+        self,
+        connections_all,
+    ) -> None:
+        connection = mock.Mock(in_atomic_block=True)
+        connections_all.return_value = [connection]
+
+        task_id = send_publish("documents.tasks.train_classifier", (), {})
+
+        connection.close_if_unusable_or_obsolete.assert_not_called()
+        assert PaperlessTask.objects.filter(task_id=task_id).exists()
+
    def test_creates_task_for_consume_file(
        self,
        consume_input_doc,
@@ -186,6 +186,21 @@ def write_store(embed_model_name: str | None = None):
        yield store


+def _check_and_run_migrations(store: "PaperlessSqliteVecVectorStore") -> bool:
+    """Run any pending structural migrations, returning True if a pending
+    re-embed migration needs the caller to force a rebuild -- never
+    triggered automatically here. Safe to call before any write, including
+    delete()/upsert_document(): has_pending_migration() (see its docstring)
+    keeps this a no-op, with no exclusive access taken, once the store is
+    current.
+    """
+    if not store.has_pending_migration():
+        return False
+    return bool(
+        _with_exclusive_access("migration check", store.check_and_run_migrations),
+    )
+
+
 def _safe_related_name(document: Document, field: str) -> str | None:
    """
    Returns the ``name`` of a related object (correspondent, document_type,
@@ -342,21 +357,6 @@ def _exclude_document_id_filter(document_id: int | str):
    )


-def _check_and_run_migrations(store: "PaperlessSqliteVecVectorStore") -> bool:
-    """Run any pending structural migrations, returning True if a pending
-    re-embed migration needs the caller to force a rebuild -- never
-    triggered automatically here. Safe to call before any write, including
-    delete()/upsert_document(): has_pending_migration() (see its docstring)
-    keeps this a no-op, with no exclusive access taken, once the store is
-    current.
-    """
-    if not store.has_pending_migration():
-        return False
-    return bool(
-        _with_exclusive_access("migration check", store.check_and_run_migrations),
-    )
-
-
 def update_llm_index(
    *,
    iter_wrapper: IterWrapper[Document] = identity,
@@ -393,12 +393,11 @@ def update_llm_index(
    config = AIConfig()
    model_name = get_configured_model_name(config)

-    if not rebuild and llm_index_exists():
+    if not rebuild:
        with read_store() as store:
-            config_mismatch = store.config_mismatch(model_name)
-        if config_mismatch:
-            logger.warning("Embedding model changed; forcing LLM index rebuild.")
-            rebuild = True
+            if store.table_exists() and store.config_mismatch(model_name):
+                logger.warning("Embedding model changed; forcing LLM index rebuild.")
+                rebuild = True

    if no_documents:
        logger.warning("No documents found to index.")
@@ -421,24 +420,12 @@ def update_llm_index(
                if document_ids is not None
                else documents
            )
-            # When document_ids is given, the caller already knows exactly
-            # which documents to reindex (e.g. a bulk edit) -- trust it and
-            # skip the modified-time comparison entirely. Bulk edits (tags,
-            # correspondent, document type, storage path, custom fields)
-            # write via queryset.update()/M2M bulk operations, which bypass
-            # Document.modified's auto_now, so comparing against
-            # get_modified_times() here would silently skip reindexing
-            # documents whose embedded metadata just changed. The comparison
-            # is only meaningful for the unscoped, full-library scan, where
-            # it avoids re-embedding documents that have not changed.
-            existing = store.get_modified_times() if document_ids is None else None
+            existing = store.get_modified_times()
            changed = 0
            for document in iter_wrapper(scoped_documents):
                doc_id = str(document.id)
-                if existing is not None:
-                    stored_modified = existing.get(doc_id)
-                    if stored_modified == document.modified.isoformat():
-                        continue
+                if existing.get(doc_id) == document.modified.isoformat():
+                    continue
                nodes = build_document_node(document, chunk_size=chunk_size)
                _embed_nodes(nodes, embed_model)
                store.upsert_document(doc_id, nodes)
@@ -464,7 +451,15 @@ def llm_index_add_or_update_document(document: Document):
        _embed_nodes(new_nodes, get_embedding_model(config))

    with write_store(embed_model_name=get_configured_model_name(config)) as store:
-        _check_and_run_migrations(store)
+        needs_reembed = _check_and_run_migrations(store)
+        if needs_reembed:
+            logger.warning(
+                "Skipping incremental LLM index update for document %s: the "
+                "index requires re-embedding first. Run 'document_llmindex "
+                "rebuild' to resolve.",
+                document.id,
+            )
+            return
        store.upsert_document(str(document.id), new_nodes)


@@ -502,7 +497,14 @@ def llm_index_compact() -> None:
 def llm_index_remove_document(document: Document):
    """Remove a document's chunks from the LLM index."""
    with write_store() as store:
-        _check_and_run_migrations(store)
+        if _check_and_run_migrations(store):
+            logger.warning(
+                "Skipping removal of document %s from the LLM index: the "
+                "index requires re-embedding first. Run 'document_llmindex "
+                "rebuild' to resolve.",
+                document.id,
+            )
+            return
        store.delete(str(document.id))


@@ -1,21 +1,24 @@
 """Schema migrations for the sqlite-vec vector store.

 Each migration lives in its own module here, named ``mNNNN_description.py``
-(e.g. ``m0001_add_document_chunks.py`` -- a leading digit isn't a valid
-Python identifier, hence the ``m`` prefix, unlike Django's own numbered
-migrations, which load via a dynamic ``importlib.import_module()`` call
-rather than a static import statement), and registers itself into
-``MIGRATIONS`` at import time. ``vector_store.py`` imports those modules at
-the bottom of the file, purely for that registration side effect, after
-``PaperlessSqliteVecVectorStore`` is fully defined -- migrations need it to
-implement ``apply()`` (see ``Migration`` below).
+(e.g. ``m0001_v1_to_v2.py`` -- a leading digit isn't a valid Python
+identifier, hence the ``m`` prefix, unlike Django's own numbered migrations,
+which load via a dynamic ``importlib.import_module()`` call rather than a
+static import statement), and registers itself into ``MIGRATIONS`` at import
+time. ``vector_store.py`` imports those modules at the bottom of the file,
+purely for that registration side effect, after ``PaperlessSqliteVecVectorStore``
+is fully defined -- migrations need it to implement ``apply()`` (see
+``Migration`` below).

 To add a new migration: add a new ``mNNNN_description.py`` module here that
 imports ``PaperlessSqliteVecVectorStore`` from ``paperless_ai.vector_store``,
-defines its ``apply()`` (most likely just a call to
-``PaperlessSqliteVecVectorStore._rebuild_into()``, see ``Migration`` below),
-and appends a ``Migration`` to ``MIGRATIONS``; then import that module at the
-bottom of ``vector_store.py`` and bump ``SCHEMA_VERSION`` there.
+defines its ``apply()``, and appends a ``Migration`` to ``MIGRATIONS``; then
+import that module at the bottom of ``vector_store.py`` and bump
+``SCHEMA_VERSION`` there. A migration must freeze its own historical DDL for
+any side table its target version depends on (``DROP TABLE IF EXISTS`` +
+its own literal ``CREATE TABLE``/``CREATE INDEX`` statements) rather than
+delegating to any "current schema" helper -- see ``m0001_v1_to_v2.py`` for
+why and the worked example.
 """

 import sqlite3
@@ -31,9 +34,8 @@ class Migration:

    kind="structural": rows are copied into a new-schema file with no
    re-embedding needed.  Supply ``apply(src_conn, dst_conn, dim)``, which
-    must create the vec0 table in ``dst_conn`` and copy ``src_conn``'s rows
-    and index_meta into it -- usually just a call to
-    ``PaperlessSqliteVecVectorStore._rebuild_into(src_conn, dst_conn, dim)``.
+    must create every table its target schema needs in ``dst_conn`` and copy
+    ``src_conn``'s rows and relevant ``index_meta`` keys into it.
    ``schema_version`` is written by the migration runner after ``apply``
    returns, not by ``apply`` itself.

@@ -1,95 +0,0 @@
-import sqlite3
-
-from paperless_ai.migrations import MIGRATIONS
-from paperless_ai.migrations import Migration
-from paperless_ai.vector_store import COMPACT_BATCH_SIZE
-from paperless_ai.vector_store import DEFAULT_TABLE_NAME
-from paperless_ai.vector_store import PaperlessSqliteVecVectorStore
-
-
-def _migrate_v1_to_v2_add_document_chunks(
-    src_conn: sqlite3.Connection,
-    dst_conn: sqlite3.Connection,
-    dim: int,
-) -> None:
-    """v1 -> v2: backfill the document_chunks side table.
-
-    document_chunks (see PaperlessSqliteVecVectorStore._open_connection) lets
-    delete()/upsert_document() find a document's chunk ids without a vec0
-    full table scan on the document_id metadata column. Every row written
-    before this migration predates that table, so without backfilling,
-    deleting a pre-migration document would find zero chunk ids and leave its
-    vec0 rows permanently orphaned.
-
-    Deliberately spells out its own v2-shaped vec0 table and row copy,
-    rather than delegating to PaperlessSqliteVecVectorStore's
-    _create_vec_table()/_rebuild_into()/_copy_rows(): those always reflect
-    whatever the *current* schema is. If a later migration changes that
-    schema (bumping SCHEMA_VERSION again), this migration must keep
-    producing its own historical v2 shape regardless -- otherwise a user
-    upgrading across multiple versions in one go (e.g. v1 straight to v4)
-    would have this migration silently produce a v4-shaped table instead
-    of v2, and the v2 -> v3 migration that runs right after it would find
-    the columns it expects to migrate *from* already gone.
-    """
-    dst_conn.execute(  # nosemgrep: python.sqlalchemy.security.sqlalchemy-execute-raw-query.sqlalchemy-execute-raw-query
-        "CREATE VIRTUAL TABLE "
-        + DEFAULT_TABLE_NAME
-        + " USING vec0("
-        + "id TEXT PRIMARY KEY,"
-        + " document_id TEXT,"
-        + " modified TEXT,"
-        + " +node_content TEXT,"
-        + " embedding float["
-        + str(int(dim))
-        + "] distance_metric=cosine"
-        + ")",
-    )
-    PaperlessSqliteVecVectorStore._meta_set_on(dst_conn, "dim", str(dim))
-    embed_model = PaperlessSqliteVecVectorStore._meta_get_on(src_conn, "embed_model")
-    if embed_model is not None:
-        PaperlessSqliteVecVectorStore._meta_set_on(dst_conn, "embed_model", embed_model)
-
-    dst_conn.execute("BEGIN IMMEDIATE")
-    src_cursor = src_conn.execute(
-        "SELECT id, document_id, modified, node_content, embedding FROM "
-        + DEFAULT_TABLE_NAME,
-    )
-    live = 0
-    while batch := src_cursor.fetchmany(COMPACT_BATCH_SIZE):
-        dst_conn.executemany(
-            "INSERT INTO "
-            + DEFAULT_TABLE_NAME
-            + " (id, document_id, modified, node_content, embedding) "
-            "VALUES (?, ?, ?, ?, ?)",
-            [
-                (
-                    r["id"],
-                    r["document_id"],
-                    r["modified"],
-                    r["node_content"],
-                    bytes(r["embedding"]),
-                )
-                for r in batch
-            ],
-        )
-        dst_conn.executemany(
-            "INSERT INTO document_chunks (chunk_id, document_id) VALUES (?, ?)",
-            [(r["id"], r["document_id"]) for r in batch],
-        )
-        live += len(batch)
-    # This migration only ever copies live rows (like compact()), so the
-    # cumulative counter resets to match -- the new file has no bloat yet.
-    PaperlessSqliteVecVectorStore._meta_set_on(dst_conn, "total_inserts", str(live))
-    dst_conn.execute("COMMIT")
-
-
-MIGRATIONS.append(
-    Migration(
-        from_version=1,
-        to_version=2,
-        kind="structural",
-        description="add document_chunks side table for O(1) per-document deletes",
-        apply=_migrate_v1_to_v2_add_document_chunks,
-    ),
-)
@@ -0,0 +1,118 @@
+import sqlite3
+
+from paperless_ai.migrations import MIGRATIONS
+from paperless_ai.migrations import Migration
+from paperless_ai.tables import ChunkRow
+from paperless_ai.tables import DocumentChunksTable
+from paperless_ai.tables import DocumentMetaRow
+from paperless_ai.tables import DocumentMetaTable
+from paperless_ai.tables import IndexMetaTable
+from paperless_ai.vector_store import COMPACT_BATCH_SIZE
+from paperless_ai.vector_store import DEFAULT_TABLE_NAME
+
+# v1's vec0 shape has never changed since it first shipped and is the ONLY
+# real upgrade path -- no store has ever existed at any intermediate
+# version, so this migration goes straight from that shipped shape to the
+# final v2 target in one pass.
+_V1_SELECT = (
+    "SELECT id, document_id, modified, node_content, embedding FROM "
+    + DEFAULT_TABLE_NAME
+)
+
+
+def _migrate_v1_to_v2(
+    src_conn: sqlite3.Connection,
+    dst_conn: sqlite3.Connection,
+    dim: int,
+) -> None:
+    """v1 -> v2: document_id TEXT -> INTEGER, modified moves out of vec0
+    into document_meta, document_chunks added for O(1) per-document delete.
+
+    Freezes its own v2-shaped vec0/document_chunks/document_meta DDL inline,
+    rather than delegating to the gateway "create table" helpers or the
+    store's own vec0-table builder (all of which always reflect the
+    *current* schema): a later schema version changing any of these tables'
+    shape must not silently change what this migration produces for someone
+    upgrading straight from v1.
+    _open_connection() already created document_chunks/document_meta on
+    dst_conn (reflecting current HEAD) as a side effect of opening it for
+    this migration's rebuild -- DROP them first so this migration's own
+    frozen CREATE TABLE isn't a silent no-op against that. Safe here because
+    dst_conn is a freshly opened, empty rebuild file with nothing written
+    yet.
+    """
+    dst_conn.execute(  # nosemgrep: python.sqlalchemy.security.sqlalchemy-execute-raw-query.sqlalchemy-execute-raw-query
+        "CREATE VIRTUAL TABLE "
+        + DEFAULT_TABLE_NAME
+        + " USING vec0("
+        + "id TEXT PRIMARY KEY,"
+        + " document_id INTEGER,"
+        + " +node_content TEXT,"
+        + " embedding float["
+        + str(int(dim))
+        + "] distance_metric=cosine"
+        + ")",
+    )
+    dst_conn.execute("DROP TABLE IF EXISTS document_chunks")
+    dst_conn.execute(
+        "CREATE TABLE document_chunks "
+        "(chunk_id TEXT PRIMARY KEY, document_id INTEGER NOT NULL)",
+    )
+    dst_conn.execute(
+        "CREATE INDEX idx_document_chunks_document_id ON document_chunks (document_id)",
+    )
+    dst_conn.execute("DROP TABLE IF EXISTS document_meta")
+    dst_conn.execute(
+        "CREATE TABLE document_meta "
+        "(document_id INTEGER PRIMARY KEY, modified TEXT NOT NULL)",
+    )
+
+    IndexMetaTable.set_dim(dst_conn, dim)
+    embed_model = IndexMetaTable.get_embed_model(src_conn)
+    if embed_model is not None:
+        IndexMetaTable.set_embed_model(dst_conn, embed_model)
+
+    dst_conn.execute("BEGIN IMMEDIATE")
+    src_cursor = src_conn.execute(_V1_SELECT)
+    live = 0
+    while batch := src_cursor.fetchmany(COMPACT_BATCH_SIZE):
+        vec0_rows = []
+        chunk_rows = []
+        meta_by_document: dict[int, str] = {}
+        for r in batch:
+            document_id = int(r["document_id"])
+            vec0_rows.append(
+                (r["id"], document_id, r["node_content"], bytes(r["embedding"])),
+            )
+            chunk_rows.append(ChunkRow(r["id"], document_id))
+            meta_by_document[document_id] = str(r["modified"] or "")
+        dst_conn.executemany(
+            "INSERT INTO "
+            + DEFAULT_TABLE_NAME
+            + " (id, document_id, node_content, embedding) VALUES (?, ?, ?, ?)",
+            vec0_rows,
+        )
+        DocumentChunksTable.insert_many(dst_conn, chunk_rows)
+        DocumentMetaTable.upsert_many(
+            dst_conn,
+            (DocumentMetaRow(doc_id, mod) for doc_id, mod in meta_by_document.items()),
+        )
+        live += len(batch)
+    # This migration only ever copies live rows (like compact()), so the
+    # cumulative counter resets to match -- the new file has no bloat yet.
+    IndexMetaTable.reset_total_inserts(dst_conn, live)
+    dst_conn.execute("COMMIT")
+
+
+MIGRATIONS.append(
+    Migration(
+        from_version=1,
+        to_version=2,
+        kind="structural",
+        description=(
+            "document_id TEXT -> INTEGER; move modified into document_meta; "
+            "add document_chunks for O(1) per-document delete"
+        ),
+        apply=_migrate_v1_to_v2,
+    ),
+)
@@ -0,0 +1,242 @@
+"""Thin gateways over the plain relational side tables that sit alongside the
+vec0 table. Each method takes the sqlite3.Connection to operate on
+explicitly, rather than owning one -- the store swaps connections during
+compact()/migration, and migrations always work across two connections
+(src_conn, dst_conn) at once.
+
+PRECONDITION: Callers must set conn.row_factory = sqlite3.Row before passing a
+connection to any of these gateways' read methods. The read methods across all
+three classes (DocumentChunksTable.chunk_ids_for_document, IndexMetaTable._get,
+DocumentMetaTable.all_modified_times, DocumentMetaTable.copy_all) use
+row["column_name"] dictionary-style indexing, which requires sqlite3.Row as the
+row factory -- without it, sqlite3.Row is not set, rows are returned as plain
+tuples, and tuple indices must be integers, raising TypeError.
+"""
+
+import sqlite3
+from collections.abc import Iterable
+from typing import NamedTuple
+
+
+class ChunkRow(NamedTuple):
+    chunk_id: str
+    document_id: int
+
+
+class DocumentMetaRow(NamedTuple):
+    document_id: int
+    modified: str
+
+
+class DocumentChunksTable:
+    """chunk_id -> document_id, indexed by document_id. Gives O(1)
+    per-document chunk lookup that vec0's own document_id metadata column
+    cannot (see PaperlessSqliteVecVectorStore._delete_chunks_by_document_id).
+    """
+
+    @staticmethod
+    def create(conn: sqlite3.Connection) -> None:
+        conn.execute(
+            "CREATE TABLE IF NOT EXISTS document_chunks "
+            "(chunk_id TEXT PRIMARY KEY, document_id INTEGER NOT NULL)",
+        )
+        conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_document_chunks_document_id "
+            "ON document_chunks (document_id)",
+        )
+
+    @staticmethod
+    def insert_many(conn: sqlite3.Connection, rows: Iterable[ChunkRow]) -> None:
+        """rows must already be batch-bounded by the caller (e.g. vec0's own
+        fetchmany() loop) -- this never reads, so it can't itself introduce
+        an unbounded scan, but a whole-table iterable defeats the point."""
+        conn.executemany(
+            "INSERT INTO document_chunks (chunk_id, document_id) VALUES (?, ?)",
+            rows,
+        )
+
+    @staticmethod
+    def chunk_ids_for_document(
+        conn: sqlite3.Connection,
+        document_id: int,
+    ) -> list[str]:
+        return [
+            row["chunk_id"]
+            for row in conn.execute(
+                "SELECT chunk_id FROM document_chunks WHERE document_id = ?",
+                (document_id,),
+            ).fetchall()
+        ]
+
+    @staticmethod
+    def delete_for_document(conn: sqlite3.Connection, document_id: int) -> None:
+        conn.execute(
+            "DELETE FROM document_chunks WHERE document_id = ?",
+            (document_id,),
+        )
+
+    @staticmethod
+    def delete_all(conn: sqlite3.Connection) -> None:
+        conn.execute("DELETE FROM document_chunks")
+
+    @staticmethod
+    def count(conn: sqlite3.Connection) -> int:
+        """Cheap stand-in for vec0's own row count -- see compact()."""
+        return conn.execute("SELECT count(*) FROM document_chunks").fetchone()[0]
+
+
+class DocumentMetaTable:
+    """document_id -> modified, one row per document. Lives outside vec0
+    because vec0 only inlines TEXT metadata up to 12 bytes and `modified`
+    (an ISO timestamp) is always longer.
+    """
+
+    @staticmethod
+    def create(conn: sqlite3.Connection) -> None:
+        conn.execute(
+            "CREATE TABLE IF NOT EXISTS document_meta "
+            "(document_id INTEGER PRIMARY KEY, modified TEXT NOT NULL)",
+        )
+
+    @staticmethod
+    def upsert_many(
+        conn: sqlite3.Connection,
+        rows: Iterable[DocumentMetaRow],
+    ) -> None:
+        conn.executemany(
+            "INSERT INTO document_meta (document_id, modified) VALUES (?, ?) "
+            "ON CONFLICT(document_id) DO UPDATE SET modified = excluded.modified",
+            rows,
+        )
+
+    @staticmethod
+    def delete_for_document(conn: sqlite3.Connection, document_id: int) -> None:
+        conn.execute(
+            "DELETE FROM document_meta WHERE document_id = ?",
+            (document_id,),
+        )
+
+    @staticmethod
+    def delete_all(conn: sqlite3.Connection) -> None:
+        conn.execute("DELETE FROM document_meta")
+
+    @staticmethod
+    def copy_all(
+        src_conn: sqlite3.Connection,
+        dst_conn: sqlite3.Connection,
+        batch_size: int,
+    ) -> None:
+        """Stream document_meta from src_conn into dst_conn in bounded
+        batches. The *only* sanctioned way to move this table across
+        connections (compact()/migrations) -- an unbounded fetchall here
+        would defeat the same OOM-avoidance the vec0 row copy already relies
+        on. batch_size has no default: forces the call site to think about
+        it (pass COMPACT_BATCH_SIZE)."""
+        cursor = src_conn.execute(
+            "SELECT document_id, modified FROM document_meta",
+        )
+        while batch := cursor.fetchmany(batch_size):
+            DocumentMetaTable.upsert_many(
+                dst_conn,
+                (DocumentMetaRow(r["document_id"], r["modified"]) for r in batch),
+            )
+
+    @staticmethod
+    def all_modified_times(conn: sqlite3.Connection) -> dict[str, str]:
+        """Full document_id -> modified map, for get_modified_times()'s
+        public API only. One unbounded read by design (existing behavior).
+        Never use this for cross-connection copying; see copy_all()."""
+        return {
+            str(row["document_id"]): str(row["modified"] or "")
+            for row in conn.execute(
+                "SELECT document_id, modified FROM document_meta",
+            )
+        }
+
+
+class IndexMetaTable:
+    """Typed accessors over index_meta's key/value rows -- replaces
+    PaperlessSqliteVecVectorStore._meta_get_on/_meta_set_on, which returned
+    untyped str | None regardless of whether the key held an int (dim,
+    schema_version, total_inserts) or a string (embed_model).
+    """
+
+    @staticmethod
+    def create(conn: sqlite3.Connection) -> None:
+        conn.execute(
+            "CREATE TABLE IF NOT EXISTS index_meta (key TEXT PRIMARY KEY, value TEXT)",
+        )
+
+    @staticmethod
+    def _get(conn: sqlite3.Connection, key: str) -> str | None:
+        row = conn.execute(
+            "SELECT value FROM index_meta WHERE key = ?",
+            (key,),
+        ).fetchone()
+        return row["value"] if row else None
+
+    @staticmethod
+    def _set(conn: sqlite3.Connection, key: str, value: str) -> None:
+        conn.execute(
+            "INSERT INTO index_meta (key, value) VALUES (?, ?) "
+            "ON CONFLICT(key) DO UPDATE SET value = excluded.value",
+            (key, value),
+        )
+
+    @staticmethod
+    def get_dim(conn: sqlite3.Connection) -> int | None:
+        value = IndexMetaTable._get(conn, "dim")
+        return int(value) if value is not None else None
+
+    @staticmethod
+    def set_dim(conn: sqlite3.Connection, dim: int) -> None:
+        IndexMetaTable._set(conn, "dim", str(dim))
+
+    @staticmethod
+    def get_embed_model(conn: sqlite3.Connection) -> str | None:
+        return IndexMetaTable._get(conn, "embed_model")
+
+    @staticmethod
+    def set_embed_model(conn: sqlite3.Connection, name: str) -> None:
+        IndexMetaTable._set(conn, "embed_model", name)
+
+    @staticmethod
+    def get_schema_version(conn: sqlite3.Connection) -> int | None:
+        value = IndexMetaTable._get(conn, "schema_version")
+        return int(value) if value is not None else None
+
+    @staticmethod
+    def set_schema_version(conn: sqlite3.Connection, version: int) -> None:
+        IndexMetaTable._set(conn, "schema_version", str(version))
+
+    @staticmethod
+    def get_total_inserts(conn: sqlite3.Connection) -> int:
+        value = IndexMetaTable._get(conn, "total_inserts")
+        return int(value) if value is not None else 0
+
+    @staticmethod
+    def increment_total_inserts(conn: sqlite3.Connection, count: int) -> None:
+        """Atomically add ``count`` to the stored counter in one statement
+        (INSERT .. ON CONFLICT DO UPDATE with arithmetic), instead of a
+        separate read-then-write -- called once per add()/upsert_document(),
+        so halving the statement count here is a real, if small, per-call
+        saving. index_meta.value has TEXT affinity, so the incremented
+        result is stored as its text representation -- get_total_inserts()
+        already expects that (int(value)), so this is not a behavior
+        change, only fewer statements.
+        """
+        conn.execute(
+            "INSERT INTO index_meta (key, value) VALUES ('total_inserts', ?) "
+            "ON CONFLICT(key) DO UPDATE SET value = "
+            "CAST(index_meta.value AS INTEGER) + CAST(excluded.value AS INTEGER)",
+            (str(count),),
+        )
+
+    @staticmethod
+    def reset_total_inserts(conn: sqlite3.Connection, count: int) -> None:
+        """Set total_inserts to an absolute value -- distinct from
+        increment_total_inserts(): used by compact()'s rebuild and by
+        m0001_v1_to_v2 after copying live rows into a fresh file, where
+        total_inserts must become exactly the live row count, not add to
+        whatever the source file's counter held."""
+        IndexMetaTable._set(conn, "total_inserts", str(count))
@@ -1,3 +1,4 @@
+import logging
 from pathlib import Path
 from unittest.mock import MagicMock
 from unittest.mock import patch
@@ -21,7 +22,6 @@ from documents.signals import document_consumption_finished
 from documents.signals import document_updated
 from documents.tests.factories import DocumentFactory
 from documents.tests.factories import PaperlessTaskFactory
-from documents.tests.factories import TagFactory
 from paperless.models import ApplicationConfiguration
 from paperless_ai import indexing
 from paperless_ai.tests.conftest import FakeEmbedding
@@ -37,23 +37,6 @@ def real_document(db: None) -> Document:
    )


-@pytest.fixture
-def mock_store(mocker: pytest_mock.MockerFixture) -> MagicMock:
-    """The MagicMock store yielded by every ``with write_store() as store:``
-    block, for tests that only care what indexing.py does with the store,
-    not what the store itself does.
-    """
-    store = mocker.MagicMock()
-    mocker.patch(
-        "paperless_ai.indexing.write_store",
-        return_value=mocker.MagicMock(
-            __enter__=mocker.MagicMock(return_value=store),
-            __exit__=mocker.MagicMock(return_value=False),
-        ),
-    )
-    return store
-
-
@pytest.mark.django_db
 def test_build_document_node(real_document: Document) -> None:
    nodes = indexing.build_document_node(real_document)
@@ -270,6 +253,45 @@ def test_update_llm_index_rebuilds_on_model_name_change(
        assert store.stored_model_name() == "model-b"


+@pytest.mark.django_db
+def test_update_llm_index_merges_exists_and_config_mismatch_reads(
+    temp_llm_index_dir: Path,
+    real_document: Document,
+    mock_embed_model: FakeEmbedding,
+) -> None:
+    # Build an initial index so the second call's table_exists()/
+    # config_mismatch() checks have something real to check against.
+    with patch("documents.models.Document.objects.all") as mock_all:
+        mock_queryset = MagicMock()
+        mock_queryset.exists.return_value = True
+        mock_queryset.__iter__.return_value = iter([real_document])
+        mock_queryset.select_related.return_value = mock_queryset
+        mock_queryset.prefetch_related.return_value = mock_queryset
+        mock_all.return_value = mock_queryset
+        indexing.update_llm_index(rebuild=True)
+
+    with patch("documents.models.Document.objects.all") as mock_all:
+        mock_queryset = MagicMock()
+        mock_queryset.exists.return_value = True
+        mock_queryset.__iter__.return_value = iter([real_document])
+        mock_queryset.select_related.return_value = mock_queryset
+        mock_queryset.prefetch_related.return_value = mock_queryset
+        mock_all.return_value = mock_queryset
+        with patch(
+            "paperless_ai.indexing.read_store",
+            wraps=indexing.read_store,
+        ) as read_store_spy:
+            indexing.update_llm_index(rebuild=False)
+
+    # Documents exist, so the fast-exit check's `no_documents and ...`
+    # short-circuits before ever calling llm_index_exists() -- the only
+    # read_store() call left in this path is the merged table_exists()/
+    # config_mismatch() check. Before this task's fix, that merged check
+    # was two separate read_store() calls (one inside llm_index_exists(),
+    # one for config_mismatch() right after) -- so this asserts 1, not 2.
+    assert read_store_spy.call_count == 1
+
+
@pytest.mark.django_db
 def test_update_llm_index_partial_update(
    temp_llm_index_dir: Path,
@@ -365,62 +387,6 @@ def test_update_llm_index_partial_update(
    assert after[str(doc2.pk)] == before[str(doc2.pk)]


-class TestUpdateLlmIndexScopedDocumentIds:
-    """A document_ids-scoped update must trust the caller and reindex every scoped
-    document, never gating on Document.modified.
-
-    bulk_edit.py's add_tag/remove_tag/modify_tags/set_correspondent/set_document_type/
-    set_storage_path/modify_custom_fields all write via queryset.update() or direct
-    M2M/through-model bulk operations -- none of which call Document.save(), so
-    Document.modified's auto_now never fires. Comparing against
-    get_modified_times() for a document_ids-scoped call would therefore skip
-    reindexing documents whose embedded tags/correspondent/etc. just changed.
-    """
-
-    @pytest.mark.django_db
-    def test_scoped_update_reindexes_despite_unchanged_modified(
-        self,
-        temp_llm_index_dir: Path,
-        mock_embed_model: FakeEmbedding,
-    ) -> None:
-        """A document_ids-scoped update must pick up a tag added via the M2M
-        manager's bulk_create path, even though that leaves modified untouched.
-
-        Steps:
-        1. Build an initial index for a document with no tags.
-        2. Add a tag via a direct through-model bulk_create, mirroring
-           bulk_edit.add_tag -- this does not call Document.save().
-        3. Call update_llm_index(rebuild=False, document_ids=[doc.pk]).
-        4. Assert the stored node metadata now includes the new tag.
-        """
-        # Step 1
-        tag = TagFactory.create(name="Important")
-        doc = DocumentFactory.create(title="Test Document", added=timezone.now())
-        indexing.update_llm_index(rebuild=True)
-        modified_before = doc.modified
-
-        # Step 2: bulk-add the tag the way bulk_edit.add_tag does -- a direct
-        # through-model insert, no Document.save().
-        DocumentTagRelationship = Document.tags.through
-        DocumentTagRelationship.objects.bulk_create(
-            [DocumentTagRelationship(document_id=doc.pk, tag_id=tag.pk)],
-        )
-        doc.refresh_from_db()
-        assert doc.modified == modified_before, (
-            "Precondition failed: expected modified to be unchanged after a "
-            "through-model bulk tag add"
-        )
-
-        # Step 3
-        result = indexing.update_llm_index(rebuild=False, document_ids=[doc.pk])
-        assert result == "LLM index updated successfully."
-
-        # Step 4
-        with indexing.get_vector_store() as store:
-            nodes = store.get_nodes()
-        assert any(tag.name in node.metadata.get("tags", []) for node in nodes)
-
-
@pytest.mark.django_db
 def test_add_or_update_document_updates_existing_entry(
    temp_llm_index_dir: Path,
@@ -779,104 +745,23 @@ class TestLlmIndexAddOrUpdateDocumentEmptyContent:
@pytest.mark.django_db
 def test_llm_index_compact_uses_force(
    temp_llm_index_dir: Path,
-    mock_store: MagicMock,
+    mocker: pytest_mock.MockerFixture,
 ) -> None:
    """compact must use force=True to rebuild the table and reclaim space immediately."""
+    mock_store = mocker.MagicMock()
+    mocker.patch(
+        "paperless_ai.indexing.write_store",
+        return_value=mocker.MagicMock(
+            __enter__=mocker.MagicMock(return_value=mock_store),
+            __exit__=mocker.MagicMock(return_value=False),
+        ),
+    )
+
    indexing.llm_index_compact()

    mock_store.compact.assert_called_once_with(force=True)


-@pytest.mark.django_db
-class TestLlmIndexMigrate:
-    """llm_index_migrate() is the cheap, startup-safe migration check -- see
-    the init-llmindex-migrate container step and the bare-metal upgrade docs.
-    """
-
-    def test_skips_when_llm_index_disabled(
-        self,
-        temp_llm_index_dir: Path,
-        mocker: pytest_mock.MockerFixture,
-    ) -> None:
-        """
-        GIVEN:
-            - The LLM index is disabled
-        WHEN:
-            - llm_index_migrate() is called
-        THEN:
-            - The store is never opened (no stray db file for users who
-              never enabled AI features)
-        """
-        mock_config = mocker.MagicMock()
-        mock_config.llm_index_enabled = False
-        mocker.patch("paperless_ai.indexing.AIConfig", return_value=mock_config)
-        mock_write_store = mocker.patch("paperless_ai.indexing.write_store")
-
-        indexing.llm_index_migrate()
-
-        mock_write_store.assert_not_called()
-
-    def test_runs_pending_structural_migration(
-        self,
-        temp_llm_index_dir: Path,
-        mocker: pytest_mock.MockerFixture,
-        mock_store: MagicMock,
-        caplog: pytest.LogCaptureFixture,
-    ) -> None:
-        """
-        GIVEN:
-            - The LLM index is enabled and a structural migration is pending
-        WHEN:
-            - llm_index_migrate() is called
-        THEN:
-            - check_and_run_migrations() runs, and no re-embed warning is
-              logged (the pending migration was structural, not re-embed)
-        """
-        mock_config = mocker.MagicMock()
-        mock_config.llm_index_enabled = True
-        mocker.patch("paperless_ai.indexing.AIConfig", return_value=mock_config)
-        mock_store.has_pending_migration.return_value = True
-        mock_store.check_and_run_migrations.return_value = False
-
-        with caplog.at_level("WARNING"):
-            indexing.llm_index_migrate()
-
-        mock_store.check_and_run_migrations.assert_called_once()
-        assert "re-embedding" not in caplog.text
-
-    def test_warns_without_rebuilding_when_reembed_pending(
-        self,
-        temp_llm_index_dir: Path,
-        mocker: pytest_mock.MockerFixture,
-        mock_store: MagicMock,
-        caplog: pytest.LogCaptureFixture,
-    ) -> None:
-        """
-        GIVEN:
-            - The LLM index is enabled and a pending migration requires
-              re-embedding
-        WHEN:
-            - llm_index_migrate() is called
-        THEN:
-            - A warning is logged telling the admin to rebuild manually, but
-              no rebuild is triggered automatically -- re-embedding can be
-              slow and, for a metered embedding backend, cost money, so it
-              must be a deliberate user action, never an automatic one
-        """
-        mock_config = mocker.MagicMock()
-        mock_config.llm_index_enabled = True
-        mocker.patch("paperless_ai.indexing.AIConfig", return_value=mock_config)
-        mock_store.has_pending_migration.return_value = True
-        mock_store.check_and_run_migrations.return_value = True
-
-        with caplog.at_level("WARNING"):
-            indexing.llm_index_migrate()
-
-        assert "re-embedding" in caplog.text
-        mock_store.drop_table.assert_not_called()
-        mock_store.add.assert_not_called()
-
-
@pytest.mark.django_db
 class TestLlmIndexLocking:
    """Index mutation functions must go through write_store(), which holds the lock.
@@ -889,9 +774,17 @@ class TestLlmIndexLocking:
        self,
        temp_llm_index_dir: Path,
        mock_embed_model: FakeEmbedding,
-        mock_store: MagicMock,
        mocker: pytest_mock.MockerFixture,
    ) -> None:
+        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = False
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )
        mock_node = MagicMock()
        mock_node.get_content.return_value = "fake node text"
        mocker.patch(
@@ -905,29 +798,25 @@ class TestLlmIndexLocking:

        mock_store.upsert_document.assert_called_once()

-    @pytest.mark.parametrize("has_pending", [True, False])
-    def test_add_or_update_document_runs_migration_check_only_when_pending(
+    def test_add_or_update_document_skips_write_when_reembed_pending(
        self,
        temp_llm_index_dir: Path,
        mock_embed_model: FakeEmbedding,
-        mock_store: MagicMock,
        mocker: pytest_mock.MockerFixture,
-        *,
-        has_pending: bool,
    ) -> None:
+        """A pending re-embed migration must block the incremental write,
+        not let it proceed against a schema that just changed underneath it.
        """
-        GIVEN:
-            - A document to add/update, and a store reporting whether a
-              migration is pending
-        WHEN:
-            - llm_index_add_or_update_document() is called
-        THEN:
-            - check_and_run_migrations() runs only when has_pending_migration()
-              is True, so a normal upsert never pays for the exclusive access
-              that check_and_run_migrations() requires
-            - upsert_document() is called either way
-        """
-        mock_store.has_pending_migration.return_value = has_pending
+        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = True
+        mock_store.check_and_run_migrations.return_value = True
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )
        mock_node = MagicMock()
        mock_node.get_content.return_value = "fake node text"
        mocker.patch(
@@ -935,59 +824,72 @@ class TestLlmIndexLocking:
            return_value=[mock_node],
        )

-        doc = DocumentFactory.build(id=1)
+        doc = MagicMock(spec=Document)
+        doc.id = 1
        indexing.llm_index_add_or_update_document(doc)

-        assert mock_store.check_and_run_migrations.called is has_pending
-        mock_store.upsert_document.assert_called_once()
+        mock_store.upsert_document.assert_not_called()

    def test_remove_document_uses_write_store(
        self,
        temp_llm_index_dir: Path,
-        mock_store: MagicMock,
+        mocker: pytest_mock.MockerFixture,
    ) -> None:
+        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = False
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )
+
        doc = MagicMock(spec=Document)
        doc.id = 1
        indexing.llm_index_remove_document(doc)

        mock_store.delete.assert_called_once_with("1")

-    @pytest.mark.parametrize("has_pending", [True, False])
-    def test_remove_document_runs_migration_check_only_when_pending(
+    def test_remove_document_skips_write_when_reembed_pending(
        self,
        temp_llm_index_dir: Path,
-        mock_store: MagicMock,
-        *,
-        has_pending: bool,
+        mocker: pytest_mock.MockerFixture,
    ) -> None:
+        """A pending re-embed migration must block the delete too, for the
+        same consistency reason as the incremental-update path.
        """
-        GIVEN:
-            - A document to remove, and a store reporting whether a
-              migration is pending
-        WHEN:
-            - llm_index_remove_document() is called
-        THEN:
-            - check_and_run_migrations() runs only when has_pending_migration()
-              is True, so a normal delete never pays for the exclusive access
-              that check_and_run_migrations() requires (see
-              test_normal_write_is_not_gated_by_the_compaction_lock)
-            - delete() is called either way
-        """
-        mock_store.has_pending_migration.return_value = has_pending
+        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = True
+        mock_store.check_and_run_migrations.return_value = True
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )

-        doc = DocumentFactory.build(id=1)
+        doc = MagicMock(spec=Document)
+        doc.id = 1
        indexing.llm_index_remove_document(doc)

-        assert mock_store.check_and_run_migrations.called is has_pending
-        mock_store.delete.assert_called_once_with("1")
+        mock_store.delete.assert_not_called()

    def test_update_llm_index_rebuild_uses_write_store(
        self,
        temp_llm_index_dir: Path,
        mock_embed_model: FakeEmbedding,
-        mock_store: MagicMock,
        mocker: pytest_mock.MockerFixture,
    ) -> None:
+        mock_store = MagicMock()
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )
        mock_qs = MagicMock()
        mock_qs.exists.return_value = True
        mock_qs.__iter__ = MagicMock(return_value=iter([]))
@@ -1046,6 +948,76 @@ class TestVectorStoreIndexing:
            assert rows >= 1


+class TestLlmIndexMigrate:
+    def test_noop_when_ai_disabled(self, mocker: pytest_mock.MockerFixture) -> None:
+        """
+        GIVEN:
+            - AI/LLM index support is disabled in configuration
+        WHEN:
+            - llm_index_migrate() is called
+        THEN:
+            - No store is opened and no migration check runs
+        """
+        mocker.patch(
+            "paperless_ai.indexing.AIConfig",
+            return_value=mocker.Mock(llm_index_enabled=False),
+        )
+        write_store_mock = mocker.patch("paperless_ai.indexing.write_store")
+        indexing.llm_index_migrate()
+        write_store_mock.assert_not_called()
+
+    def test_runs_pending_migration_when_enabled(
+        self,
+        mocker: pytest_mock.MockerFixture,
+    ) -> None:
+        """
+        GIVEN:
+            - AI/LLM index support is enabled
+        WHEN:
+            - llm_index_migrate() is called
+        THEN:
+            - The store is opened for write and a migration check runs
+        """
+        mocker.patch(
+            "paperless_ai.indexing.AIConfig",
+            return_value=mocker.Mock(llm_index_enabled=True),
+        )
+        store_mock = mocker.MagicMock()
+        store_mock.has_pending_migration.return_value = False
+        write_store_cm = mocker.patch("paperless_ai.indexing.write_store")
+        write_store_cm.return_value.__enter__.return_value = store_mock
+        indexing.llm_index_migrate()
+        store_mock.has_pending_migration.assert_called_once()
+
+    def test_logs_warning_when_reembed_needed(
+        self,
+        mocker: pytest_mock.MockerFixture,
+        caplog: pytest.LogCaptureFixture,
+    ) -> None:
+        """
+        GIVEN:
+            - AI/LLM index support is enabled
+            - A pending migration requires re-embedding
+        WHEN:
+            - llm_index_migrate() is called
+        THEN:
+            - A warning directs the operator to run a manual rebuild, since
+              this automatic check must never re-embed on its own
+        """
+        mocker.patch(
+            "paperless_ai.indexing.AIConfig",
+            return_value=mocker.Mock(llm_index_enabled=True),
+        )
+        store_mock = mocker.MagicMock()
+        store_mock.has_pending_migration.return_value = True
+        store_mock.check_and_run_migrations.return_value = True
+        write_store_cm = mocker.patch("paperless_ai.indexing.write_store")
+        write_store_cm.return_value.__enter__.return_value = store_mock
+        with caplog.at_level(logging.WARNING, logger="paperless_ai.indexing"):
+            indexing.llm_index_migrate()
+        assert "requires re-embedding" in caplog.text
+
+
@pytest.mark.django_db
 class TestQuerySimilarDocuments:
    def test_query_similar_documents_respects_allowed_ids(
@@ -0,0 +1,340 @@
+import sqlite3
+from collections.abc import Generator
+
+import pytest
+from pytest_mock import MockerFixture
+
+from paperless_ai.tables import ChunkRow
+from paperless_ai.tables import DocumentChunksTable
+from paperless_ai.tables import DocumentMetaRow
+from paperless_ai.tables import DocumentMetaTable
+from paperless_ai.tables import IndexMetaTable
+
+
+@pytest.fixture
+def conn() -> Generator[sqlite3.Connection, None, None]:
+    connection = sqlite3.connect(":memory:")
+    connection.row_factory = sqlite3.Row
+    try:
+        yield connection
+    finally:
+        connection.close()
+
+
+class TestDocumentChunksTable:
+    def test_create_is_idempotent(self, conn: sqlite3.Connection) -> None:
+        """
+        GIVEN:
+            - A bare sqlite3 connection
+        WHEN:
+            - create() is called, a row is inserted, then create() is called again
+        THEN:
+            - No error is raised and the row survives uncorrupted
+        """
+        DocumentChunksTable.create(conn)
+        DocumentChunksTable.insert_many(conn, [ChunkRow("c1", 1)])
+        DocumentChunksTable.create(conn)
+        assert DocumentChunksTable.chunk_ids_for_document(conn, 1) == ["c1"]
+
+    def test_insert_many_then_lookup_by_document_id(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - An empty document_chunks table
+        WHEN:
+            - Two chunks for document 1 and one for document 2 are inserted
+        THEN:
+            - chunk_ids_for_document returns exactly the matching chunk ids
+        """
+        DocumentChunksTable.create(conn)
+        DocumentChunksTable.insert_many(
+            conn,
+            [ChunkRow("c1", 1), ChunkRow("c2", 1), ChunkRow("c3", 2)],
+        )
+        assert sorted(DocumentChunksTable.chunk_ids_for_document(conn, 1)) == [
+            "c1",
+            "c2",
+        ]
+        assert DocumentChunksTable.chunk_ids_for_document(conn, 2) == ["c3"]
+        assert DocumentChunksTable.chunk_ids_for_document(conn, 999) == []
+
+    def test_delete_for_document_removes_only_that_document(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - Chunks for two different documents
+        WHEN:
+            - delete_for_document() is called for one of them
+        THEN:
+            - Only that document's chunks are removed
+        """
+        DocumentChunksTable.create(conn)
+        DocumentChunksTable.insert_many(
+            conn,
+            [ChunkRow("c1", 1), ChunkRow("c2", 2)],
+        )
+        DocumentChunksTable.delete_for_document(conn, 1)
+        assert DocumentChunksTable.chunk_ids_for_document(conn, 1) == []
+        assert DocumentChunksTable.chunk_ids_for_document(conn, 2) == ["c2"]
+
+    def test_delete_all_clears_every_row(self, conn: sqlite3.Connection) -> None:
+        """
+        GIVEN:
+            - Chunks for multiple documents
+        WHEN:
+            - delete_all() is called
+        THEN:
+            - count() returns 0
+        """
+        DocumentChunksTable.create(conn)
+        DocumentChunksTable.insert_many(
+            conn,
+            [ChunkRow("c1", 1), ChunkRow("c2", 2)],
+        )
+        DocumentChunksTable.delete_all(conn)
+        assert DocumentChunksTable.count(conn) == 0
+
+    def test_count_reflects_live_rows(self, conn: sqlite3.Connection) -> None:
+        """
+        GIVEN:
+            - An empty document_chunks table
+        WHEN:
+            - Rows are inserted then one document's rows are deleted
+        THEN:
+            - count() reflects the remaining row count
+        """
+        DocumentChunksTable.create(conn)
+        DocumentChunksTable.insert_many(
+            conn,
+            [ChunkRow("c1", 1), ChunkRow("c2", 1), ChunkRow("c3", 2)],
+        )
+        assert DocumentChunksTable.count(conn) == 3
+        DocumentChunksTable.delete_for_document(conn, 1)
+        assert DocumentChunksTable.count(conn) == 1
+
+
+class TestDocumentMetaTable:
+    def test_upsert_many_then_all_modified_times(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - An empty document_meta table
+        WHEN:
+            - Two documents' modified timestamps are upserted
+        THEN:
+            - all_modified_times() returns both, keyed by str(document_id)
+        """
+        DocumentMetaTable.create(conn)
+        DocumentMetaTable.upsert_many(
+            conn,
+            [
+                DocumentMetaRow(1, "2026-01-01T00:00:00"),
+                DocumentMetaRow(2, "2026-02-02T00:00:00"),
+            ],
+        )
+        assert DocumentMetaTable.all_modified_times(conn) == {
+            "1": "2026-01-01T00:00:00",
+            "2": "2026-02-02T00:00:00",
+        }
+
+    def test_upsert_many_overwrites_existing_value(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - A document_meta row for document 1
+        WHEN:
+            - upsert_many() is called again with a new modified value for
+              the same document_id
+        THEN:
+            - The stored value is replaced, not duplicated
+        """
+        DocumentMetaTable.create(conn)
+        DocumentMetaTable.upsert_many(conn, [DocumentMetaRow(1, "old")])
+        DocumentMetaTable.upsert_many(conn, [DocumentMetaRow(1, "new")])
+        assert DocumentMetaTable.all_modified_times(conn) == {"1": "new"}
+
+    def test_delete_for_document_removes_only_that_row(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - document_meta rows for two documents
+        WHEN:
+            - delete_for_document() is called for one of them
+        THEN:
+            - Only that document's row is removed
+        """
+        DocumentMetaTable.create(conn)
+        DocumentMetaTable.upsert_many(
+            conn,
+            [DocumentMetaRow(1, "a"), DocumentMetaRow(2, "b")],
+        )
+        DocumentMetaTable.delete_for_document(conn, 1)
+        assert DocumentMetaTable.all_modified_times(conn) == {"2": "b"}
+
+    def test_delete_all_clears_every_row(self, conn: sqlite3.Connection) -> None:
+        """
+        GIVEN:
+            - document_meta rows for multiple documents
+        WHEN:
+            - delete_all() is called
+        THEN:
+            - all_modified_times() returns an empty dict
+        """
+        DocumentMetaTable.create(conn)
+        DocumentMetaTable.upsert_many(
+            conn,
+            [DocumentMetaRow(1, "a"), DocumentMetaRow(2, "b")],
+        )
+        DocumentMetaTable.delete_all(conn)
+        assert DocumentMetaTable.all_modified_times(conn) == {}
+
+    def test_copy_all_streams_every_row_to_destination(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - A source connection with document_meta rows for 5 documents
+            - A separate, empty destination connection
+        WHEN:
+            - copy_all() is called with a batch size smaller than the row
+              count, forcing multiple fetchmany() cycles
+        THEN:
+            - Every row is present on the destination connection
+        """
+        DocumentMetaTable.create(conn)
+        DocumentMetaTable.upsert_many(
+            conn,
+            [DocumentMetaRow(i, f"modified-{i}") for i in range(5)],
+        )
+        dst_conn = sqlite3.connect(":memory:")
+        dst_conn.row_factory = sqlite3.Row
+        try:
+            DocumentMetaTable.create(dst_conn)
+            DocumentMetaTable.copy_all(conn, dst_conn, batch_size=2)
+            assert DocumentMetaTable.all_modified_times(dst_conn) == {
+                str(i): f"modified-{i}" for i in range(5)
+            }
+        finally:
+            dst_conn.close()
+
+
+class TestIndexMetaTable:
+    @pytest.mark.parametrize(
+        ("setter_name", "getter_name", "value"),
+        [
+            ("set_dim", "get_dim", 384),
+            ("set_embed_model", "get_embed_model", "model-a"),
+            ("set_schema_version", "get_schema_version", 2),
+        ],
+    )
+    def test_typed_accessor_roundtrip(
+        self,
+        conn: sqlite3.Connection,
+        setter_name: str,
+        getter_name: str,
+        value: int | str,
+    ) -> None:
+        """
+        GIVEN:
+            - An empty index_meta table
+        WHEN:
+            - A typed accessor's setter is called then the getter is read back
+        THEN:
+            - The same value is returned, correctly typed (int or str)
+        """
+        IndexMetaTable.create(conn)
+        getter = getattr(IndexMetaTable, getter_name)
+        setter = getattr(IndexMetaTable, setter_name)
+        assert getter(conn) is None
+        setter(conn, value)
+        assert getter(conn) == value
+
+    def test_total_inserts_starts_at_zero(self, conn: sqlite3.Connection) -> None:
+        """
+        GIVEN:
+            - An empty index_meta table
+        WHEN:
+            - get_total_inserts() is read before anything is set
+        THEN:
+            - 0 is returned
+        """
+        IndexMetaTable.create(conn)
+        assert IndexMetaTable.get_total_inserts(conn) == 0
+
+    def test_increment_total_inserts_accumulates(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - An empty index_meta table
+        WHEN:
+            - increment_total_inserts() is called twice
+        THEN:
+            - get_total_inserts() returns the running sum
+        """
+        IndexMetaTable.create(conn)
+        IndexMetaTable.increment_total_inserts(conn, 5)
+        IndexMetaTable.increment_total_inserts(conn, 3)
+        assert IndexMetaTable.get_total_inserts(conn) == 8
+
+    def test_increment_total_inserts_is_a_single_statement(
+        self,
+        mocker: MockerFixture,
+    ) -> None:
+        """
+        GIVEN:
+            - An empty index_meta table
+        WHEN:
+            - increment_total_inserts() is called
+        THEN:
+            - Exactly one conn.execute() call is made (a single INSERT ...
+              ON CONFLICT DO UPDATE, not a separate read then write)
+        """
+
+        # sqlite3.Connection is an immutable C extension type with no
+        # instance __dict__, so mocker.spy(conn, "execute") can't shadow
+        # "execute" on a plain connection ("attribute 'execute' is
+        # read-only"). A trivial Python subclass gets a normal instance
+        # __dict__, making the instance spyable while still being a real,
+        # usable sqlite3.Connection.
+        class _SpyableConnection(sqlite3.Connection):
+            pass
+
+        conn = sqlite3.connect(":memory:", factory=_SpyableConnection)
+        try:
+            conn.row_factory = sqlite3.Row
+            IndexMetaTable.create(conn)
+            execute_spy = mocker.spy(conn, "execute")
+            IndexMetaTable.increment_total_inserts(conn, 5)
+            assert execute_spy.call_count == 1
+        finally:
+            conn.close()
+
+    def test_reset_total_inserts_sets_absolute_value(
+        self,
+        conn: sqlite3.Connection,
+    ) -> None:
+        """
+        GIVEN:
+            - A total_inserts counter already at a high value
+        WHEN:
+            - reset_total_inserts() is called with a lower value
+        THEN:
+            - get_total_inserts() returns exactly that value, not a sum
+        """
+        IndexMetaTable.create(conn)
+        IndexMetaTable.increment_total_inserts(conn, 100)
+        IndexMetaTable.reset_total_inserts(conn, 7)
+        assert IndexMetaTable.get_total_inserts(conn) == 7
@@ -8,6 +8,7 @@ from contextlib import contextmanager
 from pathlib import Path
 from types import TracebackType
 from typing import Any
+from typing import NamedTuple

 import sqlite_vec
 from llama_index.core.bridge.pydantic import PrivateAttr
@@ -24,24 +25,20 @@ from llama_index.core.vector_stores.utils import node_to_metadata_dict

 from paperless_ai.migrations import MIGRATIONS
 from paperless_ai.migrations import Migration
+from paperless_ai.tables import ChunkRow
+from paperless_ai.tables import DocumentChunksTable
+from paperless_ai.tables import DocumentMetaRow
+from paperless_ai.tables import DocumentMetaTable
+from paperless_ai.tables import IndexMetaTable

 logger = logging.getLogger("paperless_ai.vector_store")

 DB_FILENAME = "llmindex.db"
 DEFAULT_TABLE_NAME = "documents"

-_INSERT = (
-    "INSERT INTO "
-    + DEFAULT_TABLE_NAME
-    + " (id, document_id, modified, node_content, embedding) VALUES (?, ?, ?, ?, ?)"
-)
-
-_INSERT_CHUNK_INDEX = (
-    "INSERT INTO document_chunks (chunk_id, document_id) VALUES (?, ?)"
-)
-
-# Current schema version. Bump when adding a migration -- see
-# paperless_ai/migrations/__init__.py for the full procedure.
+# Current schema version. Written to index_meta at table creation and bumped
+# whenever a Migration is added to MIGRATIONS. check_and_run_migrations() uses
+# this to decide which migrations to run on an existing store.
 SCHEMA_VERSION = 2

 # compact(): rebuild when the cumulative rowid count exceeds this multiple of
@@ -57,8 +54,23 @@ COMPACT_BATCH_SIZE = 500

 # Filterable vec0 metadata columns. _build_where() only ever receives filter
 # keys we construct ourselves, but allowlisting keeps SQL identifiers safe by
-# construction.
-_FILTER_COLUMNS = frozenset({"document_id", "modified"})
+# construction. "modified" is not here: it is never filtered on, and as of
+# schema v2 it isn't even a vec0 column anymore (see document_meta).
+_FILTER_COLUMNS = frozenset({"document_id"})
+
+
+class _Row(NamedTuple):
+    """One node, ready to write. ``modified`` is not a vec0 column (see
+    document_meta) -- it rides along here because every row-producing call
+    site needs both the vec0 insert values and the document_meta upsert
+    value from the same node.
+    """
+
+    chunk_id: str
+    document_id: int
+    modified: str
+    node_content: str
+    embedding: bytes


 def _pack(embedding: Sequence[float]) -> bytes:
@@ -69,50 +81,30 @@ def _unpack(blob: bytes) -> list[float]:
    return list(struct.unpack(f"{len(blob) // 4}f", blob))


-def _copy_rows(src_conn: sqlite3.Connection, dst_conn: sqlite3.Connection) -> int:
-    """Copy every live vec0 row from ``src_conn`` into ``dst_conn``, recording
-    each one in ``dst_conn``'s document_chunks side table. Returns the number
-    of rows copied. The caller owns ``dst_conn``'s transaction.
-
-    Rows are streamed from the source cursor in batches instead of being
-    materialized all at once, so a large index does not cause an OOM during a
-    routine compaction or migration.
-    """
-    src_cursor = src_conn.execute(
-        "SELECT id, document_id, modified, node_content, embedding FROM "
-        + DEFAULT_TABLE_NAME,
-    )
-    copied = 0
-    while batch := src_cursor.fetchmany(COMPACT_BATCH_SIZE):
-        dst_conn.executemany(
-            _INSERT,
-            [
-                (
-                    r["id"],
-                    r["document_id"],
-                    r["modified"],
-                    r["node_content"],
-                    bytes(r["embedding"]),
-                )
-                for r in batch
-            ],
-        )
-        dst_conn.executemany(
-            _INSERT_CHUNK_INDEX,
-            [(r["id"], r["document_id"]) for r in batch],
-        )
-        copied += len(batch)
-    return copied
+_INSERT = (
+    "INSERT INTO "
+    + DEFAULT_TABLE_NAME
+    + " (id, document_id, node_content, embedding) VALUES (?, ?, ?, ?)"
+)


-def _build_where(filters: MetadataFilters | None) -> tuple[str, list[str]]:
-    """Translate the EQ / IN / NE filters we use into a parameterized SQL clause
-    on vec0 metadata columns. Returns ("", []) when there is nothing to filter.
+def _vec0_params(rows: list[_Row]) -> list[tuple[str, int, str, bytes]]:
+    """``rows``, minus the ``modified`` field vec0 no longer stores."""
+    return [(r.chunk_id, r.document_id, r.node_content, r.embedding) for r in rows]
+
+
+def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
+    """Translate the EQ / IN / NE filters we use into a parameterized SQL
+    clause on vec0 metadata columns. Returns ("", []) when there is nothing
+    to filter. document_id is vec0's only filterable column and is INTEGER;
+    every value is coerced via int() here so callers (which today still pass
+    strings in places, e.g. indexing.py's MetadataFilter construction) don't
+    have to be individually correct -- vec0 doesn't coerce types itself.
    """
    if filters is None or not filters.filters:
        return "", []
    clauses: list[str] = []
-    params: list[str] = []
+    params: list[int] = []
    for f in filters.filters:
        # filters.filters is Union[MetadataFilter, ExactMatchFilter, MetadataFilters];
        # we only build MetadataFilter entries, so skip anything else at runtime.
@@ -121,7 +113,7 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[str]]:
        if f.key not in _FILTER_COLUMNS:  # pragma: no cover - we build the keys
            raise NotImplementedError(f"Unsupported filter column: {f.key}")
        if f.operator == FilterOperator.IN:
-            values = [str(v) for v in f.value]  # type: ignore[union-attr]  # value is list when operator is IN
+            values = [int(v) for v in f.value]  # type: ignore[union-attr]
            if not values:  # pragma: no cover
                clauses.append("1 = 0")
                continue
@@ -130,10 +122,10 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[str]]:
            params.extend(values)
        elif f.operator == FilterOperator.EQ:
            clauses.append(f"{f.key} = ?")
-            params.append(str(f.value))
+            params.append(int(f.value))
        elif f.operator == FilterOperator.NE:
            clauses.append(f"{f.key} != ?")
-            params.append(str(f.value))
+            params.append(int(f.value))
        else:  # pragma: no cover - we only ever build EQ/IN/NE filters
            raise NotImplementedError(f"Unsupported filter operator: {f.operator}")
    if not clauses:
@@ -198,27 +190,21 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        conn.enable_load_extension(False)  # noqa: FBT003
        conn.execute("PRAGMA journal_mode=WAL")
        conn.execute("PRAGMA synchronous=NORMAL")
-        conn.execute(
-            "CREATE TABLE IF NOT EXISTS index_meta (key TEXT PRIMARY KEY, value TEXT)",
-        )
-        # vec0 metadata columns only get an efficient lookup path inside a KNN
-        # (MATCH) query; a plain `WHERE document_id = ?` is a full table scan
-        # regardless of index size. This plain, indexed table is how delete()/
-        # upsert_document() find a document's chunk ids without that scan.
-        # document_id is INTEGER here (unlike vec0's own TEXT metadata
-        # column): this is a normal SQLite table, so standard type affinity
-        # correctly coerces the TEXT document ids written/looked-up
-        # elsewhere in this module -- it does not share vec0's own
-        # metadata-column comparison code, which silently mismatches
-        # non-TEXT bound values instead of coercing them.
-        conn.execute(
-            "CREATE TABLE IF NOT EXISTS document_chunks "
-            "(chunk_id TEXT PRIMARY KEY, document_id INTEGER NOT NULL)",
-        )
-        conn.execute(
-            "CREATE INDEX IF NOT EXISTS idx_document_chunks_document_id "
-            "ON document_chunks (document_id)",
-        )
+        IndexMetaTable.create(conn)
+        # vec0 metadata columns only get an efficient lookup path inside a
+        # KNN (MATCH) query; a plain `WHERE document_id = ?` is a full table
+        # scan regardless of index size. This plain, indexed table is how
+        # delete()/upsert_document() find a document's chunk ids without
+        # that scan.
+        DocumentChunksTable.create(conn)
+        # modified used to be a vec0 metadata column, but vec0 only inlines
+        # TEXT metadata up to 12 bytes -- an ISO timestamp is always longer,
+        # so every read recompiled and stepped a fresh SQL statement per row.
+        # It was never filtered on inside a KNN query either, so it never
+        # needed to be a vec0 column at all. One row per document here (not
+        # per chunk, like document_chunks), since every chunk of a document
+        # shares the same modified value -- see get_modified_times().
+        DocumentMetaTable.create(conn)
        return conn

    @property
@@ -253,28 +239,6 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        else:
            self._conn.execute("COMMIT")

-    @staticmethod
-    def _meta_get_on(conn: sqlite3.Connection, key: str) -> str | None:
-        row = conn.execute(
-            "SELECT value FROM index_meta WHERE key = ?",
-            (key,),
-        ).fetchone()
-        return row["value"] if row else None
-
-    def _meta_get(self, key: str) -> str | None:
-        return self._meta_get_on(self._conn, key)
-
-    @staticmethod
-    def _meta_set_on(conn: sqlite3.Connection, key: str, value: str) -> None:
-        conn.execute(
-            "INSERT INTO index_meta (key, value) VALUES (?, ?) "
-            "ON CONFLICT(key) DO UPDATE SET value = excluded.value",
-            (key, value),
-        )
-
-    def _meta_set(self, key: str, value: str) -> None:
-        self._meta_set_on(self._conn, key, value)
-
    def table_exists(self) -> bool:
        return (
            self._conn.execute(
@@ -287,19 +251,19 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
    def vector_dim(self) -> int | None:
        if not self.table_exists():
            return None
-        value = self._meta_get("dim")
-        return int(value) if value else None
+        return IndexMetaTable.get_dim(self._conn)

    def drop_table(self) -> None:
        self._conn.execute("DROP TABLE IF EXISTS " + DEFAULT_TABLE_NAME)
        self._conn.execute("DELETE FROM index_meta")
-        self._conn.execute("DELETE FROM document_chunks")
+        DocumentChunksTable.delete_all(self._conn)
+        DocumentMetaTable.delete_all(self._conn)

    def stored_model_name(self) -> str | None:
        """Return the embedding model name recorded at table creation, or None."""
        if not self.table_exists():
            return None
-        return self._meta_get("embed_model")
+        return IndexMetaTable.get_embed_model(self._conn)

    def config_mismatch(self, model_name: str) -> bool:
        """True when the stored model name differs from ``model_name``.
@@ -317,14 +281,17 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        # document_id is deliberately a metadata column, NOT a partition key:
        # partition keys change KNN `k` to per-partition semantics under IN
        # filters (asg017/sqlite-vec#142); metadata columns give a correct
-        # global top-k.
+        # global top-k. INTEGER (not TEXT, as in schema v1): EQ/NE/IN
+        # comparisons become a native i64 array compare instead of per-row
+        # strncmp against a 16-byte text view, and this drops the unused
+        # metadatatext shadow table TEXT columns carry. modified is not a
+        # column here at all as of v2 -- see document_meta.
        conn.execute(  # nosemgrep: python.sqlalchemy.security.sqlalchemy-execute-raw-query.sqlalchemy-execute-raw-query
            "CREATE VIRTUAL TABLE "
            + DEFAULT_TABLE_NAME
            + " USING vec0("
            + "id TEXT PRIMARY KEY,"
-            + " document_id TEXT,"
-            + " modified TEXT,"
+            + " document_id INTEGER,"
            + " +node_content TEXT,"
            + " embedding float["
            + str(int(dim))
@@ -334,63 +301,70 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):

    def _create_table(self, dim: int) -> None:
        self._create_vec_table(self._conn, dim)
-        self._meta_set("dim", str(dim))
-        self._meta_set("schema_version", str(SCHEMA_VERSION))
+        IndexMetaTable.set_dim(self._conn, dim)
+        IndexMetaTable.set_schema_version(self._conn, SCHEMA_VERSION)
        if self._embed_model_name:
-            self._meta_set("embed_model", self._embed_model_name)
+            IndexMetaTable.set_embed_model(self._conn, self._embed_model_name)

-    def _ensure_table(self, dim: int) -> None:
-        if not self.table_exists():
+    def _ensure_table(self, dim: int, *, table_exists: bool) -> None:
+        if not table_exists:
            self._create_table(dim)

-    def _row(self, node: BaseNode) -> tuple[str, str, str, str, bytes]:
+    def _row(self, node: BaseNode) -> _Row:
        meta = node_to_metadata_dict(
            node,
            remove_text=False,
            flat_metadata=self.flat_metadata,
        )
-        # vec0 metadata columns reject NULL (asg017/sqlite-vec#141): coerce
-        # every value to a string, with "" as the absent sentinel.
        document_id = node.ref_doc_id or node.metadata.get("document_id")
-        return (
-            node.node_id,
-            str(document_id or ""),
-            str(node.metadata.get("modified") or ""),
-            json.dumps(meta),
-            _pack(node.get_embedding()),
+        return _Row(
+            chunk_id=node.node_id,
+            document_id=int(document_id),
+            modified=str(node.metadata.get("modified") or ""),
+            node_content=json.dumps(meta),
+            embedding=_pack(node.get_embedding()),
        )

-    def _index_chunks(self, rows: list[tuple[str, str, str, str, bytes]]) -> None:
-        """Record each row's (chunk_id, document_id) in the document_chunks
-        side table, kept in lockstep with every insert into the vec0 table."""
-        self._conn.executemany(
-            _INSERT_CHUNK_INDEX,
-            [(chunk_id, document_id) for chunk_id, document_id, *_ in rows],
+    def _index_chunks(self, rows: list[_Row]) -> None:
+        """Record each row's (chunk_id, document_id) in document_chunks, and
+        each row's (document_id, modified) in document_meta -- deduped
+        within the batch, since every chunk of a document shares the same
+        modified value -- kept in lockstep with every insert into the vec0
+        table.
+        """
+        DocumentChunksTable.insert_many(
+            self._conn,
+            (ChunkRow(r.chunk_id, r.document_id) for r in rows),
+        )
+        modified_by_document = {r.document_id: r.modified for r in rows}
+        DocumentMetaTable.upsert_many(
+            self._conn,
+            (
+                DocumentMetaRow(doc_id, mod)
+                for doc_id, mod in modified_by_document.items()
+            ),
        )

-    def _delete_chunks_by_document_id(self, document_id: str) -> None:
+    def _delete_chunks_by_document_id(self, document_id: int) -> None:
        """Delete all of a document's chunks via point-deletes on `id`.

        vec0 has no efficient lookup on the document_id metadata column
-        outside a KNN query (see _open_connection), so a plain
-        `DELETE ... WHERE document_id = ?` is a full table scan regardless of
-        index size. Looking the chunk ids up in document_chunks first (a real
-        indexed lookup) and deleting each by its `id` primary key instead
-        turns that scan into a handful of O(1) point deletes.
+        outside a KNN query, so a plain `DELETE ... WHERE document_id = ?`
+        is a full table scan regardless of index size. Looking the chunk
+        ids up in document_chunks first (a real indexed lookup) and
+        deleting each by its `id` primary key instead turns that scan into
+        a handful of O(1) point deletes.
        """
-        doc_id = str(document_id)
-        chunk_rows = self._conn.execute(
-            "SELECT chunk_id FROM document_chunks WHERE document_id = ?",
-            (doc_id,),
-        ).fetchall()
+        chunk_ids = DocumentChunksTable.chunk_ids_for_document(
+            self._conn,
+            document_id,
+        )
        self._conn.executemany(
            "DELETE FROM " + DEFAULT_TABLE_NAME + " WHERE id = ?",
-            [(row["chunk_id"],) for row in chunk_rows],
-        )
-        self._conn.execute(
-            "DELETE FROM document_chunks WHERE document_id = ?",
-            (doc_id,),
+            [(chunk_id,) for chunk_id in chunk_ids],
        )
+        DocumentChunksTable.delete_for_document(self._conn, document_id)
+        DocumentMetaTable.delete_for_document(self._conn, document_id)

    def _increment_total_inserts(self, count: int) -> None:
        """Increment the cumulative insert counter stored in index_meta.
@@ -400,44 +374,56 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        live_rows exceeds COMPACT_BLOAT_RATIO the table has accumulated
        enough deleted-but-not-freed rows to warrant a rebuild.
        """
-        current = int(self._meta_get("total_inserts") or "0")
-        self._meta_set("total_inserts", str(current + count))
+        IndexMetaTable.increment_total_inserts(self._conn, count)

    def add(self, nodes: Sequence[BaseNode], **add_kwargs: Any) -> list[str]:
        if not nodes:
            return []
        rows = [self._row(node) for node in nodes]
        with self._transaction():
-            self._ensure_table(len(nodes[0].get_embedding()))
-            self._conn.executemany(_INSERT, rows)
+            self._ensure_table(
+                len(nodes[0].get_embedding()),
+                table_exists=self.table_exists(),
+            )
+            self._conn.executemany(_INSERT, _vec0_params(rows))
            self._index_chunks(rows)
            self._increment_total_inserts(len(rows))
        return [node.node_id for node in nodes]

-    def upsert_document(self, document_id: str, nodes: list[BaseNode]) -> list[str]:
+    def upsert_document(
+        self,
+        document_id: int | str,
+        nodes: list[BaseNode],
+    ) -> list[str]:
        """Atomically replace all stored chunks of ``document_id`` with ``nodes``.

        One transaction deletes the document's existing rows and inserts the
-        new set (vec0's INSERT OR REPLACE is broken upstream, #259, so
-        delete+insert it is). WAL readers in other processes see either the
-        old or the new chunk set, never a partial state.
+        new set (vec0's INSERT OR REPLACE is broken upstream, so delete+insert
+        it is). WAL readers in other processes see either the old or the new
+        chunk set, never a partial state.
        """
+        doc_id = int(document_id)
        rows = [self._row(node) for node in nodes]
        with self._transaction():
-            if nodes:
-                self._ensure_table(len(nodes[0].get_embedding()))
-            if self.table_exists():
-                self._delete_chunks_by_document_id(document_id)
+            table_exists = self.table_exists()
+            if nodes and not table_exists:
+                self._ensure_table(
+                    len(nodes[0].get_embedding()),
+                    table_exists=False,
+                )
+                table_exists = True
+            if table_exists:
+                self._delete_chunks_by_document_id(doc_id)
            if rows:
-                self._conn.executemany(_INSERT, rows)
+                self._conn.executemany(_INSERT, _vec0_params(rows))
                self._index_chunks(rows)
                self._increment_total_inserts(len(rows))
        return [node.node_id for node in nodes]

-    def delete(self, ref_doc_id: str, **delete_kwargs: Any) -> None:
+    def delete(self, ref_doc_id: int | str, **delete_kwargs: Any) -> None:
        if self.table_exists():
            with self._transaction():
-                self._delete_chunks_by_document_id(ref_doc_id)
+                self._delete_chunks_by_document_id(int(ref_doc_id))

    def _rows_to_nodes(self, rows: list[sqlite3.Row]) -> list[BaseNode]:
        nodes: list[BaseNode] = []
@@ -507,19 +493,12 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
    def get_modified_times(self) -> dict[str, str]:
        """Return {document_id: stored_modified_isoformat} for all indexed documents.

-        All chunks of a document share the same ``modified`` value, so the
-        first row seen per document is sufficient.
+        document_meta already has exactly one row per document (not per
+        chunk, unlike the vec0 table), so no dedup is needed here.
        """
        if not self.table_exists():
            return {}
-        result: dict[str, str] = {}
-        for row in self._conn.execute(
-            "SELECT document_id, modified FROM " + DEFAULT_TABLE_NAME,
-        ):
-            doc_id = str(row["document_id"])
-            if doc_id not in result:
-                result[doc_id] = str(row["modified"] or "")
-        return result
+        return DocumentMetaTable.all_modified_times(self._conn)

    @property
    def _db_path(self) -> str:
@@ -549,63 +528,25 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
            new_conn.close()
            self._swap_in_compact(compact_path, self._db_path)

-    @staticmethod
-    def _rebuild_into(
-        src_conn: sqlite3.Connection,
-        dst_conn: sqlite3.Connection,
-        dim: int,
-        meta_keys: tuple[str, ...] = ("dim", "embed_model", "schema_version"),
-    ) -> int:
-        """Create the vec0 table in ``dst_conn``, copy ``meta_keys`` from
-        ``src_conn``'s index_meta, and stream every live row across
-        (populating document_chunks as it goes -- see _copy_rows()).
-        Returns the number of rows copied.
-
-        Used by both compact() (default meta_keys: the schema is unchanged)
-        and structural migrations (meta_keys minus "schema_version", which
-        the migration sets to its own target version instead of preserving
-        the source's).
-        """
-        PaperlessSqliteVecVectorStore._create_vec_table(dst_conn, dim)
-        for key in meta_keys:
-            value = PaperlessSqliteVecVectorStore._meta_get_on(src_conn, key)
-            if value is not None:
-                PaperlessSqliteVecVectorStore._meta_set_on(dst_conn, key, value)
-        dst_conn.execute("BEGIN IMMEDIATE")
-        copied = _copy_rows(src_conn, dst_conn)
-        # Reset the cumulative counter: after a rebuild, total_inserts == live.
-        PaperlessSqliteVecVectorStore._meta_set_on(
-            dst_conn,
-            "total_inserts",
-            str(copied),
-        )
-        dst_conn.execute("COMMIT")
-        return copied
-
    def compact(self, *, force: bool = False) -> None:
        """Rebuild the database file to reclaim space left behind by DELETEs.

        vec0 DELETE only invalidates rows; the vector data stays in the file
-        forever (asg017/sqlite-vec#54), and per-document re-indexing is a
-        delete+insert. The cumulative insert counter in ``index_meta`` tracks
-        total rows ever written; when that exceeds ``COMPACT_BLOAT_RATIO`` x
-        the live row count (or when forced), live rows are copied into a fresh
-        database file and swapped in via ``os.replace``.
+        forever, and per-document re-indexing is a delete+insert. The
+        cumulative insert counter in ``index_meta`` tracks total rows ever
+        written; when that exceeds ``COMPACT_BLOAT_RATIO`` x the live row
+        count (or when forced), live rows are copied into a fresh database
+        file and swapped in via ``os.replace``.

        Note: ``ALTER TABLE ... RENAME TO`` on vec0 virtual tables does NOT
-        rename the shadow tables (sqlite-vec upstream limitation), so
-        an in-place rename-based rebuild is not safe.  The file-swap approach
-        is the maintainer-endorsed workaround (asg017/sqlite-vec#205).
+        rename the shadow tables (sqlite-vec upstream limitation), so an
+        in-place rename-based rebuild is not safe. The file-swap approach is
+        the maintainer-endorsed workaround.
        """
        if not self.table_exists():
            return
-        # document_chunks is a plain indexed table kept in lockstep with the
-        # vec0 table (see _index_chunks()/_delete_chunks_by_document_id()),
-        # so it gives an identical live-row count to `SELECT count(*) FROM
-        # documents` without vec0's full table scan -- count(*) has no
-        # equivalent point-lookup fast path, unlike an EQ constraint on `id`.
-        live = self._conn.execute("SELECT count(*) FROM document_chunks").fetchone()[0]
-        total = int(self._meta_get("total_inserts") or str(live))
+        live = DocumentChunksTable.count(self._conn)
+        total = IndexMetaTable.get_total_inserts(self._conn) or live
        if not force and total <= max(live, 1) * COMPACT_BLOAT_RATIO:
            return
        dim = self.vector_dim()
@@ -620,6 +561,60 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        with self._rebuild_file() as new_conn:
            self._rebuild_into(self._conn, new_conn, dim)

+    @staticmethod
+    def _rebuild_into(
+        src_conn: sqlite3.Connection,
+        dst_conn: sqlite3.Connection,
+        dim: int,
+    ) -> int:
+        """Create the vec0 table in ``dst_conn``, copy dim/embed_model from
+        ``src_conn``, and stream every live vec0 row, document_chunks row,
+        and document_meta row across. Returns the number of vec0 rows
+        copied. Used by compact() only -- m0001_v1_to_v2 freezes its own
+        copy loop instead of calling this, since this always reflects the
+        *current* schema (see the migration DDL-freezing rule in the spec).
+        """
+        PaperlessSqliteVecVectorStore._create_vec_table(dst_conn, dim)
+        dim_value = IndexMetaTable.get_dim(src_conn)
+        if dim_value is not None:
+            IndexMetaTable.set_dim(dst_conn, dim_value)
+        embed_model = IndexMetaTable.get_embed_model(src_conn)
+        if embed_model is not None:
+            IndexMetaTable.set_embed_model(dst_conn, embed_model)
+        schema_version = IndexMetaTable.get_schema_version(src_conn)
+        if schema_version is not None:
+            IndexMetaTable.set_schema_version(dst_conn, schema_version)
+
+        dst_conn.execute("BEGIN IMMEDIATE")
+        src_cursor = src_conn.execute(
+            "SELECT id, document_id, node_content, embedding FROM "
+            + DEFAULT_TABLE_NAME,
+        )
+        copied = 0
+        while batch := src_cursor.fetchmany(COMPACT_BATCH_SIZE):
+            dst_conn.executemany(
+                _INSERT,
+                [
+                    (
+                        r["id"],
+                        r["document_id"],
+                        r["node_content"],
+                        bytes(r["embedding"]),
+                    )
+                    for r in batch
+                ],
+            )
+            DocumentChunksTable.insert_many(
+                dst_conn,
+                (ChunkRow(r["id"], r["document_id"]) for r in batch),
+            )
+            copied += len(batch)
+        DocumentMetaTable.copy_all(src_conn, dst_conn, COMPACT_BATCH_SIZE)
+        # Reset the cumulative counter: after a rebuild, total_inserts == live.
+        IndexMetaTable.reset_total_inserts(dst_conn, copied)
+        dst_conn.execute("COMMIT")
+        return copied
+
    def _swap_in_compact(self, compact_path: str, db_path: str) -> None:
        """Atomically replace the live database with the compacted copy."""
        self._conn.close()
@@ -638,8 +633,8 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        """
        if not self.table_exists():
            return None
-        raw = self._meta_get("schema_version")
-        return int(raw) if raw is not None else SCHEMA_VERSION
+        raw_version = IndexMetaTable.get_schema_version(self._conn)
+        return raw_version if raw_version is not None else SCHEMA_VERSION

    def has_pending_migration(self) -> bool:
        """Cheaply check whether a migration is pending, with no exclusive
@@ -705,10 +700,10 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
            raise RuntimeError("Cannot migrate: no stored vector dimension")
        with self._rebuild_file() as new_conn:
            migration.apply(self._conn, new_conn, dim)
-            self._meta_set_on(new_conn, "schema_version", str(migration.to_version))
+            IndexMetaTable.set_schema_version(new_conn, migration.to_version)


-# Registers m0001 into MIGRATIONS; must be at the bottom (needs
+# Registers m0001_v1_to_v2 into MIGRATIONS; must be at the bottom (needs
 # PaperlessSqliteVecVectorStore fully defined) -- see
 # paperless_ai/migrations/__init__.py for the full procedure.
-from paperless_ai.migrations import m0001_add_document_chunks  # noqa: E402, F401
+from paperless_ai.migrations import m0001_v1_to_v2  # noqa: E402, F401
@@ -3735,15 +3735,15 @@ crypto = [

 [[package]]
 name = "pymdown-extensions"
-version = "10.21.3"
+version = "11.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "markdown", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
    { name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/47/67/f1e79672a5f91985577c7984c9709ca110e4fd37fe7fd167b60422e6ccc2/pymdown_extensions-11.0.tar.gz", hash = "sha256:8269cef0247f9e2d0a62fcea10860aba05c1cbab5470fd4b63230b96434dc589", size = 857049, upload-time = "2026-06-23T02:27:45.146Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" },
+    { url = "https://files.pythonhosted.org/packages/af/b6/1ae53367e28b9cffa3be7574e13fbe4589694272fd47710fbdbafd3d63c6/pymdown_extensions-11.0-py3-none-any.whl", hash = "sha256:fbc4acb641814fa9d17521bbd21a5240ef739a662f11c06330c4b78c93e954d6", size = 269415, upload-time = "2026-06-23T02:27:43.826Z" },
 ]

 [[package]]
Author	SHA1	Message	Date
stumpylog	9a4b1402c0	Perf: dedupe table_exists() lookups, atomic insert counter, fewer connections in update_llm_index()	2026-07-29 15:16:08 -07:00
stumpylog	394f81e820	Fix: strengthen two vacuous Task 4 regression tests test_migration_never_delegates_to_current_schema_helpers never actually ran the migration (missing check_and_run_migrations() call) and its source-text assertion was tautological (the "or DROP TABLE in source" clause was always true). Now runs the real migration and asserts spy call counts instead: DocumentChunksTable.create/DocumentMetaTable.create are each called exactly 3 times (construction, rebuild temp file, post-swap reopen -- all via _open_connection, never from inside apply()), and _create_vec_table is never called from the migration path. test_drop_table_clears_modified_times asserted via get_modified_times(), which short-circuits on table_exists() -- checking only the vec0 table that drop_table() drops first -- so the assertion held even if DocumentMetaTable.delete_all() were never called. Now asserts directly against document_meta and document_chunks row counts.	2026-07-29 14:55:10 -07:00
stumpylog	7ac1917f79	Feature: schema v2 -- document_chunks/document_meta side tables, document_id INTEGER, point-delete Rewrites the sqlite-vec vector store's on-disk schema: document_id becomes an INTEGER vec0 metadata column (was TEXT), modified moves out of vec0 into a new document_meta side table, and a new document_chunks side table gives O(1) per-document chunk lookup for delete/upsert instead of a full vec0 scan. compact() now streams document_chunks and document_meta across the file-swap rebuild too (previously document_meta would have gone silently empty after the first compaction). drop_table() clears both side tables. Adds the single frozen m0001_v1_to_v2 migration, converting a real, historically-shaped v1 store (the shape shipped since v3.0.0) into the v2 shape in one streaming pass, with its own hardcoded DDL rather than delegating to any "current schema" helper. SCHEMA_VERSION bumps 1 -> 2.	2026-07-29 14:38:23 -07:00
stumpylog	0464f3c901	Merge branch 'pr2-tables-gateway' into pr3-schema-rewrite	2026-07-29 13:58:57 -07:00
stumpylogandClaude Sonnet 5	7d9ba34582	Fix: address code review feedback on table gateways - Document sqlite3.Row row_factory precondition in module docstring - Strengthen test_create_is_idempotent to verify populated table survives - Collapse three roundtrip tests into one @pytest.mark.parametrize Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:51:32 -07:00
stumpylogandClaude Sonnet 5	91c3d9caab	Feature: add table-gateway module for document_chunks/document_meta/index_meta Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:42:39 -07:00
stumpylog	01f723bae2	Fix: skip incremental index writes when a re-embed migration is pending	2026-07-29 13:23:27 -07:00
stumpylog	49a584607b	Docs: add migrate to document_llmindex choices and bare-metal upgrade steps	2026-07-29 12:33:24 -07:00
stumpylog	ad5af61121	Fix: restore executable bit on init-llmindex-migrate run	2026-07-29 12:20:38 -07:00
stumpylog	5b7341fd3a	Feature: run pending LLM index migrations automatically on startup	2026-07-29 12:19:06 -07:00
stumpylogandClaude Sonnet 5	4d9c0e8bd5	Refactor: extract migration infrastructure, add has_pending_migration() Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 11:58:43 -07:00
shamoonandGitHub	668fa77428	Fix: prevent scale vs page loop in pngx PDF viewer (#13406 )	2026-07-29 09:25:37 -07:00
shamoonandGitHub	5bd72014a6	Fix: handle hidden line breaks in email subjects when sending email (#13402 )	2026-07-29 14:35:30 +00:00
Trenton HandGitHub	bbb9c86ba4	Fix: avoid NotSupportedError from document_importer on MariaDB (#13400 )	2026-07-29 07:14:13 -07:00
GitHub Actions	09c9fa03cf	Auto translate strings	2026-07-28 18:28:12 +00:00
shamoonandGitHub	04779d72bb	Tweak: adjust doc details button toolbar flow (#13382 )	2026-07-28 11:26:37 -07:00
Trenton HandGitHub	1b32b9d678	Fix: exclude next-period start from relative date-range filters (#13381 ) * Fix: exclude next-period start from relative date-range filters Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of the next period. Use half-open [lo TO hi} for those so e.g. "previous month" no longer matches the 1st of the current month. * Adds a regression test down to the second check for the hi range	2026-07-28 16:58:38 +00:00
Trenton HandGitHub	12d318deff	Documentation: Add Password Removal workflow action documentation (#13377 ) Addresses discussion #13373. Documents that password removal creates a new document version via re-consumption of the decrypted file rather than editing in place, and explains why the Consumption Started trigger produces an initial un-OCR'd version followed by a properly processed one.	2026-07-28 09:21:14 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	9c98f30d3f	Chore(deps): Bump pymdown-extensions in the uv group across 1 directory (#13378 ) Bumps the uv group with 1 update in the / directory: [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions). Updates `pymdown-extensions` from 10.21.3 to 11.0 - [Release notes](https://github.com/facelessuser/pymdown-extensions/releases) - [Commits](https://github.com/facelessuser/pymdown-extensions/compare/10.21.3...11.0) --- updated-dependencies: - dependency-name: pymdown-extensions dependency-version: '11.0' dependency-type: indirect dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-07-28 15:29:20 +00:00
shamoonandGitHub	6a1d7b1bca	Fix: close non-atomic db connections in before_task_publish (#13366 )	2026-07-28 07:21:02 -07:00