Fix: skip incremental index writes when a re-embed migration is pending

Docs: add migrate to document_llmindex choices and bare-metal upgrade steps
Fix: restore executable bit on init-llmindex-migrate run
2026-07-30 07:44:54 +00:00 · 2026-07-29 13:23:27 -07:00 · 2026-07-29 12:33:24 -07:00 · 2026-07-29 12:20:38 -07:00 · 2026-07-29 12:19:06 -07:00 · 2026-07-29 11:58:43 -07:00
29 changed files with 655 additions and 185 deletions
@@ -0,0 +1,12 @@
+#!/command/with-contenv /usr/bin/bash
+# shellcheck shell=bash
+
+declare -r log_prefix="[init-llmindex-migrate]"
+
+echo "${log_prefix} Checking for pending LLM index migrations..."
+cd "${PAPERLESS_SRC_DIR}"
+if [[ -n "${USER_IS_NON_ROOT}" ]]; then
+	python3 manage.py document_llmindex migrate
+else
+	s6-setuidgid paperless python3 manage.py document_llmindex migrate
+fi
@@ -0,0 +1 @@
+oneshot
@@ -0,0 +1 @@
+/etc/s6-overlay/s6-rc.d/init-llmindex-migrate/run
@@ -212,6 +212,16 @@ following:
    This is a no-op if the index is already up to date, so it is safe to
    run on every upgrade.

+5.  Migrate the LLM index if needed.
+
+    ```shell-session
+    cd src
+    python3 manage.py document_llmindex migrate
+    ```
+
+    This is a no-op if the index schema is already current, so it is safe
+    to run on every upgrade.
+
 ### Database Upgrades

 Paperless-ngx is compatible with Django-supported versions of PostgreSQL and MariaDB and it is generally
@@ -532,7 +542,7 @@ index is updated automatically on the schedule set by
 can manage it manually:

 ```
-document_llmindex {rebuild,update,compact}
+document_llmindex {rebuild,update,compact,migrate}
 ```

 Specify `rebuild` to build the index from scratch from all documents in the database. Use
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
 If a "Move to Trash" action is executed in a consume pipeline, the consumption
 will be aborted and the file will be deleted.

+##### Password Removal {#workflow-action-password-removal}
+
+"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
+
+- One or more passwords to try, separated by commas or new lines
+- Each password is tried in order until one successfully unlocks the document
+
+Password removal never modifies a file in place. Instead, once a working password is found, the
+decrypted content is consumed as a new [document version](#document-file-versions), leaving the
+original (still encrypted) version in the document's version history.
+
+**Consumption Started**: because this trigger fires before the document exists yet, the password
+removal itself is deferred until after the initial consumption of the encrypted file has completed.
+OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
+extracted text (unless the file already contained extractable text outside of OCR). Immediately
+afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
+new version of the same document, this time with normal OCR/text extraction applied. In other words,
+a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
+the properly processed version is created.
+
+**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
+already exists, so password removal happens immediately: the decrypted content is queued for
+consumption as a new version right away. Note that if the document's initial consumption also
+happened while it was still encrypted, that original version will likewise be missing OCR text.
+
+**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
+multiple PDF types with different passwords, create separate workflows for each use case.
+
 #### Workflow placeholders

 Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -659,7 +659,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">445</context>
+          <context context-type="linenumber">449</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
@@ -831,7 +831,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">464</context>
+          <context context-type="linenumber">468</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -1355,7 +1355,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">390</context>
+          <context context-type="linenumber">394</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1596,7 +1596,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">165</context>
+          <context context-type="linenumber">169</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2691296884221415710" datatype="html">
@@ -1607,7 +1607,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">170</context>
+          <context context-type="linenumber">174</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1638,7 +1638,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">174</context>
+          <context context-type="linenumber">178</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1669,7 +1669,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">178</context>
+          <context context-type="linenumber">182</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -4914,7 +4914,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">356</context>
+          <context context-type="linenumber">360</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8057014866157903311" datatype="html">
@@ -7760,14 +7760,14 @@
        <source>Details</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">160</context>
+          <context context-type="linenumber">164</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5701618810648052610" datatype="html">
        <source>Title</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">163</context>
+          <context context-type="linenumber">167</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -7790,14 +7790,14 @@
        <source>Date created</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">167</context>
+          <context context-type="linenumber">171</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5607669932062416162" datatype="html">
        <source>Default</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">179</context>
+          <context context-type="linenumber">183</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
@@ -7808,14 +7808,14 @@
        <source>Content</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">286</context>
+          <context context-type="linenumber">290</context>
        </context-group>
      </trans-unit>
      <trans-unit id="218403386307979629" datatype="html">
        <source>Metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">295</context>
+          <context context-type="linenumber">299</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
@@ -7826,147 +7826,147 @@
        <source>Date modified</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">302</context>
+          <context context-type="linenumber">306</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6392918669949841614" datatype="html">
        <source>Date added</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">306</context>
+          <context context-type="linenumber">310</context>
        </context-group>
      </trans-unit>
      <trans-unit id="146828917013192897" datatype="html">
        <source>Media filename</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">310</context>
+          <context context-type="linenumber">314</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4500855521601039868" datatype="html">
        <source>Original filename</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">314</context>
+          <context context-type="linenumber">318</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2659735245739197634" datatype="html">
        <source>Original SHA256 checksum</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">318</context>
+          <context context-type="linenumber">322</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5888243105821763422" datatype="html">
        <source>Original file size</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">322</context>
+          <context context-type="linenumber">326</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2696647325713149563" datatype="html">
        <source>Original mime type</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">326</context>
+          <context context-type="linenumber">330</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6714358112223607756" datatype="html">
        <source>Archive SHA256 checksum</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">331</context>
+          <context context-type="linenumber">335</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6033581412811562084" datatype="html">
        <source>Archive file size</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">337</context>
+          <context context-type="linenumber">341</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6992781481378431874" datatype="html">
        <source>Original document metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">346</context>
+          <context context-type="linenumber">350</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2846565152091361585" datatype="html">
        <source>Archived document metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">349</context>
+          <context context-type="linenumber">353</context>
        </context-group>
      </trans-unit>
      <trans-unit id="7206723502037428235" datatype="html">
        <source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">368,371</context>
+          <context context-type="linenumber">372,375</context>
        </context-group>
      </trans-unit>
      <trans-unit id="186236568870281953" datatype="html">
        <source>History</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">379</context>
+          <context context-type="linenumber">383</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8236092845697214347" datatype="html">
        <source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="cate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/></source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">401,405</context>
+          <context context-type="linenumber">405,409</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6449374629822973702" datatype="html">
        <source>Duplicate documents detected:</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">407</context>
+          <context context-type="linenumber">411</context>
        </context-group>
      </trans-unit>
      <trans-unit id="14058600336670816" datatype="html">
        <source>In trash</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">418</context>
+          <context context-type="linenumber">422</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5129524307369213584" datatype="html">
        <source>Save &amp; next</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">447</context>
+          <context context-type="linenumber">451</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4910102545766233758" datatype="html">
        <source>Save &amp; close</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">449</context>
+          <context context-type="linenumber">453</context>
        </context-group>
      </trans-unit>
      <trans-unit id="3823219296477075982" datatype="html">
        <source>Discard</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">451</context>
+          <context context-type="linenumber">455</context>
        </context-group>
      </trans-unit>
      <trans-unit id="1309556917227148591" datatype="html">
        <source>Document loading...</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">459</context>
+          <context context-type="linenumber">463</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8191371354890763172" datatype="html">
        <source>Enter Password</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
-          <context context-type="linenumber">513</context>
+          <context context-type="linenumber">517</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5758784066858623886" datatype="html">
@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
    ;(component as any).applyScale()
    expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
    expect(viewer.currentScale).toBe(2)
+  })

+  it('does not reapply scale for page-only changes', async () => {
+    await initComponent()
+
+    const pdf = (component as any).pdf as { numPages: number }
+    pdf.numPages = 3
+    const viewer = (component as any).pdfViewer as PDFViewer
+    viewer.setDocument(pdf)
    const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
    component.page = 2
-    ;(component as any).lastViewerPage = 2
-    ;(component as any).applyViewerState()
+
+    component.ngOnChanges({
+      page: new SimpleChange(1, 2, false),
+    })
+
+    expect(viewer.currentPageNumber).toBe(2)
    expect((component as any).lastViewerPage).toBeUndefined()
-    expect(applyScaleSpy).toHaveBeenCalled()
+    expect(applyScaleSpy).not.toHaveBeenCalled()
  })

  it('does not reset the viewer when it is already on the requested page', async () => {
@@ -116,7 +116,10 @@ export class PngxPdfViewerComponent
      changes['zoomScale'] ||
      changes['rotation']
    ) {
-      this.applyViewerState()
+      // Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
+      this.applyViewerState(
+        !!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
+      )
    }

    if (changes['searchQuery']) {
@@ -240,7 +243,7 @@ export class PngxPdfViewerComponent
    }
  }

-  private applyViewerState(): void {
+  private applyViewerState(applyScale = true): void {
    if (!this.pdfViewer) {
      return
    }
@@ -264,7 +267,7 @@ export class PngxPdfViewerComponent
    if (this.page === this.lastViewerPage) {
      this.lastViewerPage = undefined
    }
-    if (hasPages) {
+    if (hasPages && applyScale) {
      this.applyScale()
    }
    this.dispatchFindIfReady()
@@ -113,8 +113,8 @@

    <form [formGroup]='documentForm' (ngSubmit)="save()">

-      <div class="btn-toolbar mb-1 border-bottom">
-        <div class="btn-group pb-3">
+      <div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
+        <div class="btn-group me-auto">
          <button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
            <i-bs width="1.2em" height="1.2em" name="x"></i-bs>
          </button>
@@ -127,32 +127,36 @@
        </div>

        <ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
-          <div class="btn-group pb-3 ms-auto">
-            <pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
-              [disabled]="!userCanEdit || suggestionsLoading()"
-              [loading]="suggestionsLoading()"
-              [suggestions]="suggestions()"
-              [aiEnabled]="aiEnabled"
-              (getSuggestions)="getSuggestions()"
-              (addTag)="createTag($event)"
-              (addDocumentType)="createDocumentType($event)"
-              (addCorrespondent)="createCorrespondent($event)">
-            </pngx-suggestions-dropdown>
+          <div class="d-flex gap-2">
+            <div class="btn-group">
+              <pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
+                [disabled]="!userCanEdit || suggestionsLoading()"
+                [loading]="suggestionsLoading()"
+                [suggestions]="suggestions()"
+                [aiEnabled]="aiEnabled"
+                (getSuggestions)="getSuggestions()"
+                (addTag)="createTag($event)"
+                (addDocumentType)="createDocumentType($event)"
+                (addCorrespondent)="createCorrespondent($event)">
+              </pngx-suggestions-dropdown>
+            </div>
+
+            <div class="btn-group">
+              <pngx-custom-fields-dropdown
+                *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
+                [documentId]="documentId()"
+                [disabled]="!userCanEdit"
+                [existingFields]="document()?.custom_fields"
+                (created)="refreshCustomFields()"
+                (added)="addField($event)">
+              </pngx-custom-fields-dropdown>
+            </div>
          </div>

-          <div class="btn-group pb-3 ms-2">
-            <pngx-custom-fields-dropdown
-              *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
-              [documentId]="documentId()"
-              [disabled]="!userCanEdit"
-              [existingFields]="document()?.custom_fields"
-              (created)="refreshCustomFields()"
-              (added)="addField($event)">
-            </pngx-custom-fields-dropdown>
+          <div class="ps-3">
+            <ng-container *ngTemplateOutlet="saveButtons"></ng-container>
          </div>
        </ng-container>
-
-        <ng-container *ngTemplateOutlet="saveButtons"></ng-container>
      </div>

      <ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
@@ -276,7 +280,7 @@
              }
            </div>

-            <div class="d-flex border-top pt-3">
+            <div class="d-flex justify-content-end border-top pt-3">
              <ng-container *ngTemplateOutlet="saveButtons"></ng-container>
            </div>
          </ng-template>
@@ -440,7 +444,7 @@
 </div>

 <ng-template #saveButtons>
-  <div class="btn-group pb-3 ms-4">
+  <div class="btn-group">
    <ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
      <button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
      @if (hasNext()) {
@@ -36,6 +36,9 @@ def send_email(

    TODO: re-evaluate this pending https://code.djangoproject.com/ticket/35581 / https://github.com/django/django/pull/18966
    """
+    if "\r" in subject or "\n" in subject:
+        subject = " ".join(line.strip(" \t") for line in subject.splitlines())
+
    email = EmailMessage(
        subject=subject,
        body=body,
@@ -386,10 +386,19 @@ class Command(CryptMixin, PaperlessCommand):
                raise DeserializationError(
                    f"{model.__name__} has no updatable fields; PK-only models are not supported by the importer",
                )
+            # MySQL/MariaDB support upserts via ON DUPLICATE KEY UPDATE but,
+            # unlike PostgreSQL/SQLite, cannot target a specific unique field
+            # for the conflict -- passing unique_fields there raises
+            # NotSupportedError.
+            unique_fields = (
+                [model._meta.pk.attname]
+                if connection.features.supports_update_conflicts_with_target
+                else None
+            )
            model.objects.bulk_create(  # type: ignore[attr-defined]
                instances,
                update_conflicts=True,
-                unique_fields=[model._meta.pk.attname],
+                unique_fields=unique_fields,
                update_fields=update_fields,
            )
            loaded_models.add(model)
@@ -3,6 +3,7 @@ from typing import Any
 from documents.management.commands.base import PaperlessCommand
 from documents.tasks import llmindex_index
 from paperless_ai.indexing import llm_index_compact
+from paperless_ai.indexing import llm_index_migrate


 class Command(PaperlessCommand):
@@ -13,12 +14,18 @@ class Command(PaperlessCommand):

    def add_arguments(self, parser: Any) -> None:
        super().add_arguments(parser)
-        parser.add_argument("command", choices=["rebuild", "update", "compact"])
+        parser.add_argument(
+            "command",
+            choices=["rebuild", "update", "compact", "migrate"],
+        )

    def handle(self, *args: Any, **options: Any) -> None:
        if options["command"] == "compact":
            llm_index_compact()
            return
+        if options["command"] == "migrate":
+            llm_index_migrate()
+            return
        llmindex_index(
            rebuild=options["command"] == "rebuild",
            iter_wrapper=lambda docs: self.track(
@@ -43,8 +43,16 @@ def _fmt(dt: datetime) -> str:


 def _iso_range(lo: datetime, hi: datetime) -> str:
-    """Format a [lo TO hi] range string in ISO 8601 for Tantivy query syntax."""
-    return f"[{_fmt(lo)} TO {_fmt(hi)}]"
+    """
+    Format a half-open ``[lo TO hi)`` range in ISO 8601 for Tantivy query syntax.
+
+    ``hi`` is always the exclusive ceiling of a computed period (the start of
+    the *next* day/week/month/quarter/year), so the closing bracket must be
+    the Tantivy exclusive-range brace ``}`` rather than ``]`` — otherwise the
+    first instant of the following period (e.g. the 1st of next month) is
+    incorrectly included in the match.
+    """
+    return f"[{_fmt(lo)} TO {_fmt(hi)}}}"


 def _quarter_start(d: date) -> date:
@@ -566,6 +566,17 @@ def translate_range(field: str, lo: str, hi: str, tz: tzinfo) -> str:
        lo_pair, hi_pair = hi_pair, lo_pair

    lo_iso = _fmt(lo_pair[0]) if lo_pair is not None else OPEN_LO
-    hi_iso = _fmt(hi_pair[1]) if hi_pair is not None else OPEN_HI

-    return f"{field}:[{lo_iso} TO {hi_iso}]"
+    # A bound resolves to (floor, ceil) where floor == ceil for an exact instant
+    # (a full ISO datetime, "now", or a "+/-N unit" offset) and floor != ceil for
+    # a coarser period token (year/month/day precision). Only the latter needs a
+    # half-open close: its ceil is the start of the *next* period and must be
+    # excluded, or that instant (e.g. the 1st of next month) wrongly matches.
+    if hi_pair is not None:
+        hi_iso = _fmt(hi_pair[1])
+        hi_close = "]" if hi_pair[0] == hi_pair[1] else "}"
+    else:
+        hi_iso = OPEN_HI
+        hi_close = "]"
+
+    return f"{field}:[{lo_iso} TO {hi_iso}{hi_close}"
@@ -1134,6 +1134,11 @@ def before_task_publish_handler(
        return

    try:
+        # Close stale connections without disrupting a transaction publishing a task
+        for connection in connections.all(initialized_only=True):
+            if not connection.in_atomic_block:
+                connection.close_if_unusable_or_obsolete()
+
        _, task_kwargs, _ = body
        task_id = headers["id"]

@@ -9,6 +9,7 @@ if TYPE_CHECKING:

 _COMPACT = "documents.management.commands.document_llmindex.llm_index_compact"
 _INDEX = "documents.management.commands.document_llmindex.llmindex_index"
+_MIGRATE = "documents.management.commands.document_llmindex.llm_index_migrate"


 class TestDocumentLlmindexCommand:
@@ -17,6 +18,11 @@ class TestDocumentLlmindexCommand:
        call_command("document_llmindex", "compact")
        mock_compact.assert_called_once_with()

+    def test_migrate_calls_llm_index_migrate(self, mocker: MockerFixture) -> None:
+        mock_migrate = mocker.patch(_MIGRATE)
+        call_command("document_llmindex", "migrate")
+        mock_migrate.assert_called_once_with()
+
    def test_rebuild_calls_llmindex_index_with_rebuild_true(
        self,
        mocker: MockerFixture,
@@ -32,7 +32,9 @@ AUCKLAND = ZoneInfo("Pacific/Auckland")  # UTC+13 in southern-hemisphere summer


 def _range(result: str, field: str) -> tuple[str, str]:
-    m = re.search(rf"{field}:\[(.+?) TO (.+?)\]", result)
+    # Half-open period ranges close with "}" (exclusive); exact-instant ranges
+    # (full ISO datetimes, "now", relative offsets) close with "]" (inclusive).
+    m = re.search(rf"{field}:\[(.+?) TO (.+?)[\]}}]", result)
    assert m, f"No range for {field!r} in: {result!r}"
    return m.group(1), m.group(2)

@@ -214,27 +214,27 @@ class TestTranslateScalar:
            (
                "created",
                "2020",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
            (
                "created",
                "202003",
-                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
+                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
            ),
            (
                "created",
                "20200115",
-                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
+                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
            ),
            (
                "created",
                "2020-01-15",
-                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
+                "created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
            ),
            (
                "created",
                "2020-03",
-                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
+                "created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
            ),
        ],
    )
@@ -248,9 +248,9 @@ class TestTranslateScalar:
        assert exc_info.value.value == "202023"

    def test_keyword_delegates(self) -> None:
-        # keyword path produces a range; just assert it is a created range
+        # keyword path produces a half-open range; just assert it is a created range
        out = translate_scalar("created", "today", UTC)
-        assert out.startswith("created:[") and out.endswith("]")
+        assert out.startswith("created:[") and out.endswith("}")

    def test_14digit_compact_datetime(self) -> None:
        out = translate_scalar("created", "20240115120000", UTC)
@@ -279,21 +279,21 @@ class TestTranslateRange:
    @pytest.mark.parametrize(
        ("lo", "hi", "expected"),
        [
-            ("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"),
+            ("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"),
            (
                "202001",
                "202006",
-                "created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z}",
            ),
            (
                "20200101",
                "20201231",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
            (
                "2020-01-01",
                "2020-12-31",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
        ],
    )
@@ -302,7 +302,7 @@ class TestTranslateRange:

    def test_reversed_swaps(self):
        assert translate_range("created", "2009", "2005", UTC) == (
-            "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"
+            "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"
        )

    def test_open_upper(self):
@@ -311,7 +311,7 @@ class TestTranslateRange:

    def test_open_lower(self):
        out = translate_range("created", "", "2020", UTC)
-        assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z]"
+        assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z}}"

    def test_invalid_bound_raises(self):
        with pytest.raises(InvalidDateQuery) as exc_info:
@@ -334,16 +334,16 @@ class TestTranslateQuery:
        [
            (
                "created:2020",
-                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
+                "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
            ),
            ("tag:foo,bar", "tag:foo AND tag:bar"),
            # 'type' is a user-facing alias rewritten to 'document_type' (the real schema field)
            ("tag:foo,type:bar", "tag:foo AND document_type:bar"),
            (
                "created:[2020 TO 2021],added:[2022 TO 2023]",
-                "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
+                "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
                " AND "
-                "added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z]",
+                "added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z}",
            ),
            # correspondent is not multi-value: comma stays literal inside the value
            ("correspondent:foo,bar", "correspondent:foo,bar"),
@@ -506,7 +506,7 @@ class TestOperatorNormalization:
    def test_date_range_preserved(self) -> None:
        out = translate_query("created:[2020 TO 2021]", UTC)
        # Must not corrupt the ISO range
-        assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
+        assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"

    def test_date_scalar_with_or(self) -> None:
        out = translate_query("created:2020 OR foo", UTC)
@@ -581,42 +581,42 @@ class TestKeywordDateResolution:
        [
            pytest.param(
                "today",
-                "created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z]",
+                "created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z}",
                id="today",
            ),
            pytest.param(
                "yesterday",
-                "created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z]",
+                "created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z}",
                id="yesterday",
            ),
            pytest.param(
                "previous week",
-                "created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z]",
+                "created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z}",
                id="previous-week",
            ),
            pytest.param(
                "this month",
-                "created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z]",
+                "created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z}",
                id="this-month",
            ),
            pytest.param(
                "previous month",
-                "created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z]",
+                "created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z}",
                id="previous-month",
            ),
            pytest.param(
                "this year",
-                "created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z]",
+                "created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z}",
                id="this-year",
            ),
            pytest.param(
                "previous year",
-                "created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
+                "created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
                id="previous-year",
            ),
            pytest.param(
                "previous quarter",
-                "created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
+                "created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
                id="previous-quarter",
            ),
        ],
@@ -637,42 +637,42 @@ class TestKeywordDateResolution:
        [
            pytest.param(
                "today",
-                "added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z]",
+                "added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z}",
                id="today",
            ),
            pytest.param(
                "yesterday",
-                "added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z]",
+                "added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z}",
                id="yesterday",
            ),
            pytest.param(
                "previous week",
-                "added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z]",
+                "added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z}",
                id="previous-week",
            ),
            pytest.param(
                "this month",
-                "added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z]",
+                "added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z}",
                id="this-month",
            ),
            pytest.param(
                "previous month",
-                "added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z]",
+                "added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z}",
                id="previous-month",
            ),
            pytest.param(
                "this year",
-                "added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z]",
+                "added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z}",
                id="this-year",
            ),
            pytest.param(
                "previous year",
-                "added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z]",
+                "added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z}",
                id="previous-year",
            ),
            pytest.param(
                "previous quarter",
-                "added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z]",
+                "added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z}",
                id="previous-quarter",
            ),
        ],
@@ -719,7 +719,7 @@ class TestISODatetimeBounds:
    def test_translate_query_text_before_comma_separated_date_clause(self) -> None:
        result = translate_query("schäfersee,created:previous year", UTC)
        assert result == (
-            "schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]"
+            "schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}"
        )

    def test_invalid_iso_datetime_raises(self) -> None:
@@ -75,7 +75,7 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
                {
                    "documents": [self.doc1.pk, self.doc2.pk],
                    "addresses": "hello@paperless-ngx.com,test@example.com",
-                    "subject": "Bulk email test",
+                    "subject": "Bulk email\n test",
                    "message": "Here are your documents",
                },
            ),
@@ -720,6 +720,48 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
        self.assertEqual(results[0]["id"], 3)
        self.assertEqual(results[0]["title"], "bank statement 3")

+    def test_search_added_previous_month_excludes_next_period_start(self) -> None:
+        """
+        GIVEN:
+            - One document added at the last instant of last month
+            - One document added exactly at the first instant of this month
+        WHEN:
+            - Query for documents added in the previous month
+        THEN:
+            - Only the document from last month is returned; the document dated
+              exactly at the start of this month (the exclusive upper bound of
+              the range) is not
+        """
+        d1 = DocumentFactory.create(
+            title="end of last month",
+            content="last instant of last month",
+            checksum="A",
+            pk=1,
+            added=timezone.make_aware(datetime.datetime(2024, 1, 31, 23, 59, 59)),
+        )
+        d2 = DocumentFactory.create(
+            title="start of this month",
+            content="first instant of this month",
+            checksum="B",
+            pk=2,
+            added=timezone.make_aware(datetime.datetime(2024, 2, 1, 0, 0, 0)),
+        )
+
+        backend = get_backend()
+        backend.add_or_update(d1)
+        backend.add_or_update(d2)
+
+        with time_machine.travel(
+            timezone.make_aware(datetime.datetime(2024, 2, 15, 12, 0, 0)),
+            tick=False,
+        ):
+            response = self.client.get("/api/documents/?query=added:previous month")
+        results = response.data["results"]
+
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0]["id"], 1)
+        self.assertEqual(results[0]["title"], "end of last month")
+
    def test_search_added_invalid_date(self) -> None:
        """
        GIVEN:
@@ -56,6 +56,32 @@ def send_publish(

@pytest.mark.django_db
 class TestBeforeTaskPublishHandler:
+    @mock.patch("documents.signals.handlers.connections.all")
+    def test_closes_old_connections_outside_atomic_blocks(
+        self,
+        connections_all,
+    ) -> None:
+        connection = mock.Mock(in_atomic_block=False)
+        connections_all.return_value = [connection]
+
+        task_id = send_publish("documents.tasks.train_classifier", (), {})
+
+        connection.close_if_unusable_or_obsolete.assert_called_once_with()
+        assert PaperlessTask.objects.filter(task_id=task_id).exists()
+
+    @mock.patch("documents.signals.handlers.connections.all")
+    def test_keeps_connections_open_inside_atomic_blocks(
+        self,
+        connections_all,
+    ) -> None:
+        connection = mock.Mock(in_atomic_block=True)
+        connections_all.return_value = [connection]
+
+        task_id = send_publish("documents.tasks.train_classifier", (), {})
+
+        connection.close_if_unusable_or_obsolete.assert_not_called()
+        assert PaperlessTask.objects.filter(task_id=task_id).exists()
+
    def test_creates_task_for_consume_file(
        self,
        consume_input_doc,
@@ -144,6 +144,24 @@ def _exclude_readers():
        lock.close()


+def _with_exclusive_access(operation: str, fn):
+    """Run ``fn()`` with exclusive index access (see ``_exclude_readers()``),
+    for compaction/migration file swaps that must not run while readers are
+    active. Returns ``fn()``'s result, or None (after logging) if active
+    readers do not drain within ``LLM_INDEX_COMPACTION_LOCK_TIMEOUT`` --
+    callers skip the operation this run; it retries next time.
+    """
+    try:
+        with _exclude_readers():
+            return fn()
+    except Timeout:
+        logger.info(
+            "Skipping LLM index %s: index readers are active; will retry next run.",
+            operation,
+        )
+        return None
+
+
@contextmanager
 def write_store(embed_model_name: str | None = None):
    """Acquire the write lock and yield the vector store.
@@ -168,6 +186,21 @@ def write_store(embed_model_name: str | None = None):
        yield store


+def _check_and_run_migrations(store: "PaperlessSqliteVecVectorStore") -> bool:
+    """Run any pending structural migrations, returning True if a pending
+    re-embed migration needs the caller to force a rebuild -- never
+    triggered automatically here. Safe to call before any write, including
+    delete()/upsert_document(): has_pending_migration() (see its docstring)
+    keeps this a no-op, with no exclusive access taken, once the store is
+    current.
+    """
+    if not store.has_pending_migration():
+        return False
+    return bool(
+        _with_exclusive_access("migration check", store.check_and_run_migrations),
+    )
+
+
 def _safe_related_name(document: Document, field: str) -> str | None:
    """
    Returns the ``name`` of a related object (correspondent, document_type,
@@ -339,15 +372,7 @@ def update_llm_index(
    happens, since a rebuild always covers the whole library regardless.
    """
    with write_store() as store:
-        try:
-            with _exclude_readers():
-                needs_reembed = store.check_and_run_migrations()
-        except Timeout:
-            logger.info(
-                "Skipping LLM index migration check: index readers are active; "
-                "will retry next run.",
-            )
-            needs_reembed = False
+        needs_reembed = _check_and_run_migrations(store)
        if needs_reembed:
            logger.warning(
                "LLM index migration requires re-embedding; forcing rebuild.",
@@ -412,14 +437,7 @@ def update_llm_index(
                else "No changes detected in LLM index."
            )

-        try:
-            with _exclude_readers():
-                store.compact()
-        except Timeout:
-            logger.info(
-                "Skipping LLM index compaction: index readers are active; "
-                "will retry next run.",
-            )
+        _with_exclusive_access("compaction", store.compact)
    return msg


@@ -434,25 +452,60 @@ def llm_index_add_or_update_document(document: Document):
        _embed_nodes(new_nodes, get_embedding_model(config))

    with write_store(embed_model_name=get_configured_model_name(config)) as store:
+        needs_reembed = _check_and_run_migrations(store)
+        if needs_reembed:
+            logger.warning(
+                "Skipping incremental LLM index update for document %s: the "
+                "index requires re-embedding first. Run 'document_llmindex "
+                "rebuild' to resolve.",
+                document.id,
+            )
+            return
        store.upsert_document(str(document.id), new_nodes)


+def llm_index_migrate() -> None:
+    """Apply any pending LLM index schema migrations, with no reindex.
+
+    Intended to run unconditionally on every startup (see the
+    init-llmindex-migrate container step and the bare-metal upgrade docs):
+    has_pending_migration() short-circuits to a metadata-only read once the
+    store is current, so a healthy install pays almost nothing here. Only
+    ever applies structural migrations -- a pending re-embed migration is
+    left for the explicit, deliberate rebuild path (``document_llmindex
+    update``/``rebuild``) to resolve, since re-embedding can be slow and,
+    for a metered embedding backend, cost money.
+    """
+    if not AIConfig().llm_index_enabled:
+        return
+    with write_store() as store:
+        needs_reembed = _check_and_run_migrations(store)
+    if needs_reembed:
+        logger.warning(
+            "LLM index requires re-embedding, which this automatic migration "
+            "check will not do on its own -- it can be slow and, for a "
+            "metered embedding backend, cost money. Run "
+            "'document_llmindex rebuild' manually when ready.",
+        )
+
+
 def llm_index_compact() -> None:
    """Compact the index immediately, rebuilding the table to reclaim space."""
    with write_store() as store:
-        try:
-            with _exclude_readers():
-                store.compact(force=True)
-        except Timeout:
-            logger.info(
-                "Skipping LLM index compaction: index readers are active; "
-                "will retry next run.",
-            )
+        _with_exclusive_access("compaction", lambda: store.compact(force=True))


 def llm_index_remove_document(document: Document):
    """Remove a document's chunks from the LLM index."""
    with write_store() as store:
+        if _check_and_run_migrations(store):
+            logger.warning(
+                "Skipping removal of document %s from the LLM index: the "
+                "index requires re-embedding first. Run 'document_llmindex "
+                "rebuild' to resolve.",
+                document.id,
+            )
+            return
        store.delete(str(document.id))


@@ -0,0 +1,60 @@
+"""Schema migrations for the sqlite-vec vector store.
+
+Each migration lives in its own module here, named ``mNNNN_description.py``
+(e.g. ``m0001_v1_to_v2.py`` -- a leading digit isn't a valid Python
+identifier, hence the ``m`` prefix, unlike Django's own numbered migrations,
+which load via a dynamic ``importlib.import_module()`` call rather than a
+static import statement), and registers itself into ``MIGRATIONS`` at import
+time. ``vector_store.py`` imports those modules at the bottom of the file,
+purely for that registration side effect, after ``PaperlessSqliteVecVectorStore``
+is fully defined -- migrations need it to implement ``apply()`` (see
+``Migration`` below).
+
+To add a new migration: add a new ``mNNNN_description.py`` module here that
+imports ``PaperlessSqliteVecVectorStore`` from ``paperless_ai.vector_store``,
+defines its ``apply()``, and appends a ``Migration`` to ``MIGRATIONS``; then
+import that module at the bottom of ``vector_store.py`` and bump
+``SCHEMA_VERSION`` there. A migration must freeze its own historical DDL for
+any side table its target version depends on (``DROP TABLE IF EXISTS`` +
+its own literal ``CREATE TABLE``/``CREATE INDEX`` statements) rather than
+delegating to any "current schema" helper -- see ``m0001_v1_to_v2.py`` for
+why and the worked example.
+"""
+
+import sqlite3
+from collections.abc import Callable
+from dataclasses import dataclass
+from dataclasses import field
+from typing import Literal
+
+
+@dataclass
+class Migration:
+    """A schema migration for the sqlite-vec vector store.
+
+    kind="structural": rows are copied into a new-schema file with no
+    re-embedding needed.  Supply ``apply(src_conn, dst_conn, dim)``, which
+    must create every table its target schema needs in ``dst_conn`` and copy
+    ``src_conn``'s rows and relevant ``index_meta`` keys into it.
+    ``schema_version`` is written by the migration runner after ``apply``
+    returns, not by ``apply`` itself.
+
+    kind="re-embed": the new schema requires fresh embeddings.
+    ``check_and_run_migrations()`` returns True when it encounters one of
+    these so the caller can force a full rebuild (which recreates the table
+    at the current SCHEMA_VERSION).
+    """
+
+    from_version: int
+    to_version: int
+    kind: Literal["structural", "re-embed"]
+    description: str
+    apply: Callable[[sqlite3.Connection, sqlite3.Connection, int], None] | None = field(
+        default=None,
+        repr=False,
+    )
+
+
+# Registry of all schema migrations in order, populated by each migration
+# module's import-time registration (see the module docstring above).
+MIGRATIONS: list[Migration] = []
@@ -1,3 +1,4 @@
+import logging
 from pathlib import Path
 from unittest.mock import MagicMock
 from unittest.mock import patch
@@ -737,6 +738,7 @@ class TestLlmIndexLocking:
        mocker: pytest_mock.MockerFixture,
    ) -> None:
        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = False
        mocker.patch(
            "paperless_ai.indexing.write_store",
            return_value=mocker.MagicMock(
@@ -757,12 +759,45 @@ class TestLlmIndexLocking:

        mock_store.upsert_document.assert_called_once()

+    def test_add_or_update_document_skips_write_when_reembed_pending(
+        self,
+        temp_llm_index_dir: Path,
+        mock_embed_model: FakeEmbedding,
+        mocker: pytest_mock.MockerFixture,
+    ) -> None:
+        """A pending re-embed migration must block the incremental write,
+        not let it proceed against a schema that just changed underneath it.
+        """
+        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = True
+        mock_store.check_and_run_migrations.return_value = True
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )
+        mock_node = MagicMock()
+        mock_node.get_content.return_value = "fake node text"
+        mocker.patch(
+            "paperless_ai.indexing.build_document_node",
+            return_value=[mock_node],
+        )
+
+        doc = MagicMock(spec=Document)
+        doc.id = 1
+        indexing.llm_index_add_or_update_document(doc)
+
+        mock_store.upsert_document.assert_not_called()
+
    def test_remove_document_uses_write_store(
        self,
        temp_llm_index_dir: Path,
        mocker: pytest_mock.MockerFixture,
    ) -> None:
        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = False
        mocker.patch(
            "paperless_ai.indexing.write_store",
            return_value=mocker.MagicMock(
@@ -777,6 +812,31 @@ class TestLlmIndexLocking:

        mock_store.delete.assert_called_once_with("1")

+    def test_remove_document_skips_write_when_reembed_pending(
+        self,
+        temp_llm_index_dir: Path,
+        mocker: pytest_mock.MockerFixture,
+    ) -> None:
+        """A pending re-embed migration must block the delete too, for the
+        same consistency reason as the incremental-update path.
+        """
+        mock_store = MagicMock()
+        mock_store.has_pending_migration.return_value = True
+        mock_store.check_and_run_migrations.return_value = True
+        mocker.patch(
+            "paperless_ai.indexing.write_store",
+            return_value=mocker.MagicMock(
+                __enter__=mocker.MagicMock(return_value=mock_store),
+                __exit__=mocker.MagicMock(return_value=False),
+            ),
+        )
+
+        doc = MagicMock(spec=Document)
+        doc.id = 1
+        indexing.llm_index_remove_document(doc)
+
+        mock_store.delete.assert_not_called()
+
    def test_update_llm_index_rebuild_uses_write_store(
        self,
        temp_llm_index_dir: Path,
@@ -849,6 +909,76 @@ class TestVectorStoreIndexing:
            assert rows >= 1


+class TestLlmIndexMigrate:
+    def test_noop_when_ai_disabled(self, mocker: pytest_mock.MockerFixture) -> None:
+        """
+        GIVEN:
+            - AI/LLM index support is disabled in configuration
+        WHEN:
+            - llm_index_migrate() is called
+        THEN:
+            - No store is opened and no migration check runs
+        """
+        mocker.patch(
+            "paperless_ai.indexing.AIConfig",
+            return_value=mocker.Mock(llm_index_enabled=False),
+        )
+        write_store_mock = mocker.patch("paperless_ai.indexing.write_store")
+        indexing.llm_index_migrate()
+        write_store_mock.assert_not_called()
+
+    def test_runs_pending_migration_when_enabled(
+        self,
+        mocker: pytest_mock.MockerFixture,
+    ) -> None:
+        """
+        GIVEN:
+            - AI/LLM index support is enabled
+        WHEN:
+            - llm_index_migrate() is called
+        THEN:
+            - The store is opened for write and a migration check runs
+        """
+        mocker.patch(
+            "paperless_ai.indexing.AIConfig",
+            return_value=mocker.Mock(llm_index_enabled=True),
+        )
+        store_mock = mocker.MagicMock()
+        store_mock.has_pending_migration.return_value = False
+        write_store_cm = mocker.patch("paperless_ai.indexing.write_store")
+        write_store_cm.return_value.__enter__.return_value = store_mock
+        indexing.llm_index_migrate()
+        store_mock.has_pending_migration.assert_called_once()
+
+    def test_logs_warning_when_reembed_needed(
+        self,
+        mocker: pytest_mock.MockerFixture,
+        caplog: pytest.LogCaptureFixture,
+    ) -> None:
+        """
+        GIVEN:
+            - AI/LLM index support is enabled
+            - A pending migration requires re-embedding
+        WHEN:
+            - llm_index_migrate() is called
+        THEN:
+            - A warning directs the operator to run a manual rebuild, since
+              this automatic check must never re-embed on its own
+        """
+        mocker.patch(
+            "paperless_ai.indexing.AIConfig",
+            return_value=mocker.Mock(llm_index_enabled=True),
+        )
+        store_mock = mocker.MagicMock()
+        store_mock.has_pending_migration.return_value = True
+        store_mock.check_and_run_migrations.return_value = True
+        write_store_cm = mocker.patch("paperless_ai.indexing.write_store")
+        write_store_cm.return_value.__enter__.return_value = store_mock
+        with caplog.at_level(logging.WARNING, logger="paperless_ai.indexing"):
+            indexing.llm_index_migrate()
+        assert "requires re-embedding" in caplog.text
+
+
@pytest.mark.django_db
 class TestQuerySimilarDocuments:
    def test_query_similar_documents_respects_allowed_ids(
@@ -9,11 +9,11 @@ from llama_index.core.vector_stores.types import MetadataFilter
 from llama_index.core.vector_stores.types import MetadataFilters
 from llama_index.core.vector_stores.types import VectorStoreQuery

+from paperless_ai.migrations import MIGRATIONS
+from paperless_ai.migrations import Migration
 from paperless_ai.vector_store import DB_FILENAME
 from paperless_ai.vector_store import DEFAULT_TABLE_NAME
-from paperless_ai.vector_store import MIGRATIONS
 from paperless_ai.vector_store import SCHEMA_VERSION
-from paperless_ai.vector_store import Migration
 from paperless_ai.vector_store import PaperlessSqliteVecVectorStore
 from paperless_ai.vector_store import _build_where

@@ -646,3 +646,50 @@ class TestMigrations:

        assert result is True
        assert self._schema_version(store) == 2
+
+    def test_has_pending_migration_false_when_no_table(
+        self,
+        store: PaperlessSqliteVecVectorStore,
+    ) -> None:
+        """
+        GIVEN:
+            - A vector store with no table created yet
+        WHEN:
+            - has_pending_migration() is checked
+        THEN:
+            - False is returned (nothing to migrate before anything exists)
+        """
+        assert store.has_pending_migration() is False
+
+    def test_has_pending_migration_false_at_current_version(
+        self,
+        store: PaperlessSqliteVecVectorStore,
+    ) -> None:
+        """
+        GIVEN:
+            - A store at the current SCHEMA_VERSION
+        WHEN:
+            - has_pending_migration() is checked
+        THEN:
+            - False is returned
+        """
+        store.add([make_node("a1", "1")])
+        assert store.has_pending_migration() is False
+
+    def test_has_pending_migration_true_when_behind(
+        self,
+        store: PaperlessSqliteVecVectorStore,
+    ) -> None:
+        """
+        GIVEN:
+            - A store whose schema_version has been forced behind SCHEMA_VERSION
+        WHEN:
+            - has_pending_migration() is checked
+        THEN:
+            - True is returned
+        """
+        store.add([make_node("a1", "1")])
+        store.client.execute(
+            "UPDATE index_meta SET value = '0' WHERE key = 'schema_version'",
+        )
+        assert store.has_pending_migration() is True
@@ -2,16 +2,12 @@ import json
 import logging
 import sqlite3
 import struct
-from collections.abc import Callable
 from collections.abc import Iterator
 from collections.abc import Sequence
 from contextlib import contextmanager
-from dataclasses import dataclass
-from dataclasses import field
 from pathlib import Path
 from types import TracebackType
 from typing import Any
-from typing import Literal

 import sqlite_vec
 from llama_index.core.bridge.pydantic import PrivateAttr
@@ -26,6 +22,9 @@ from llama_index.core.vector_stores.types import VectorStoreQueryResult
 from llama_index.core.vector_stores.utils import metadata_dict_to_node
 from llama_index.core.vector_stores.utils import node_to_metadata_dict

+from paperless_ai.migrations import MIGRATIONS
+from paperless_ai.migrations import Migration
+
 logger = logging.getLogger("paperless_ai.vector_store")

 DB_FILENAME = "llmindex.db"
@@ -53,38 +52,6 @@ COMPACT_BATCH_SIZE = 500
 _FILTER_COLUMNS = frozenset({"document_id", "modified"})


-@dataclass
-class Migration:
-    """A schema migration for the sqlite-vec vector store.
-
-    kind="structural": rows are copied into a new-schema file with no
-    re-embedding needed.  Supply ``apply(src_conn, dst_conn, dim)`` which
-    must create the vec0 table in ``dst_conn``, copy all rows from
-    ``src_conn``, and write ``dim`` / ``embed_model`` / ``total_inserts`` to
-    ``dst_conn``'s ``index_meta``.  ``schema_version`` is written by the
-    migration runner after ``apply`` returns.
-
-    kind="re-embed": the new schema requires fresh embeddings.
-    ``check_and_run_migrations()`` returns True when it encounters one of
-    these so the caller can force a full rebuild (which recreates the table
-    at the current SCHEMA_VERSION).
-    """
-
-    from_version: int
-    to_version: int
-    kind: Literal["structural", "re-embed"]
-    description: str
-    apply: Callable[[sqlite3.Connection, sqlite3.Connection, int], None] | None = field(
-        default=None,
-        repr=False,
-    )
-
-
-# Registry of all schema migrations in order. Empty at v1 -- this is the
-# baseline. Add entries here (and bump SCHEMA_VERSION) when the schema changes.
-MIGRATIONS: list[Migration] = []
-
-
 def _pack(embedding: Sequence[float]) -> bytes:
    return struct.pack(f"{len(embedding)}f", *embedding)

@@ -551,6 +518,31 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        Path(compact_path).replace(db_path)
        self._conn = self._open_connection(db_path)

+    def _stored_schema_version(self) -> int | None:
+        """The schema_version recorded in index_meta, or None if no table
+        exists. A missing key (a store predating version tracking) is
+        treated as SCHEMA_VERSION -- i.e. already current -- since no
+        migration in MIGRATIONS targets a version before tracking began.
+        """
+        if not self.table_exists():
+            return None
+        raw = self._meta_get("schema_version")
+        return int(raw) if raw is not None else SCHEMA_VERSION
+
+    def has_pending_migration(self) -> bool:
+        """Cheaply check whether a migration is pending, with no exclusive
+        access needed -- just a metadata read under the connection callers
+        already hold via the write FileLock.
+
+        Callers should only pay for check_and_run_migrations()'s exclusive
+        access (a structural migration's file swap must not run while
+        readers are active) when this returns True, so that the common
+        case -- already at SCHEMA_VERSION -- never contends with readers
+        or a concurrent compaction.
+        """
+        current = self._stored_schema_version()
+        return current is not None and current < SCHEMA_VERSION
+
    def check_and_run_migrations(self) -> bool:
        """Apply any pending schema migrations to the store.

@@ -559,15 +551,13 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
        this method returns True when one is encountered so the caller can
        force a full rebuild (which recreates the table at SCHEMA_VERSION).

-        Must be called under the write FileLock.  No-op when the table does
-        not exist or is already at SCHEMA_VERSION.
+        Must be called under the write FileLock, with readers excluded (see
+        has_pending_migration() for a cheap pre-check that avoids paying for
+        that exclusion in the common case).  No-op when the table does not
+        exist or is already at SCHEMA_VERSION.
        """
-        if not self.table_exists():
-            return False
-
-        raw = self._meta_get("schema_version")
-        current = int(raw) if raw is not None else SCHEMA_VERSION
-        if current >= SCHEMA_VERSION:
+        current = self._stored_schema_version()
+        if current is None or current >= SCHEMA_VERSION:
            return False

        pending = sorted(
@@ -579,7 +569,7 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
            if migration.kind == "re-embed":
                logger.warning(
                    "LLM index schema v%d -> v%d requires re-embedding (%s); "
-                    "forcing full rebuild.",
+                    "the caller must force a rebuild.",
                    migration.from_version,
                    migration.to_version,
                    migration.description,
@@ -3735,15 +3735,15 @@ crypto = [

 [[package]]
 name = "pymdown-extensions"
-version = "10.21.3"
+version = "11.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "markdown", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
    { name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/47/67/f1e79672a5f91985577c7984c9709ca110e4fd37fe7fd167b60422e6ccc2/pymdown_extensions-11.0.tar.gz", hash = "sha256:8269cef0247f9e2d0a62fcea10860aba05c1cbab5470fd4b63230b96434dc589", size = 857049, upload-time = "2026-06-23T02:27:45.146Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" },
+    { url = "https://files.pythonhosted.org/packages/af/b6/1ae53367e28b9cffa3be7574e13fbe4589694272fd47710fbdbafd3d63c6/pymdown_extensions-11.0-py3-none-any.whl", hash = "sha256:fbc4acb641814fa9d17521bbd21a5240ef739a662f11c06330c4b78c93e954d6", size = 269415, upload-time = "2026-06-23T02:27:43.826Z" },
 ]

 [[package]]
Author	SHA1	Message	Date
stumpylog	01f723bae2	Fix: skip incremental index writes when a re-embed migration is pending	2026-07-29 13:23:27 -07:00
stumpylog	49a584607b	Docs: add migrate to document_llmindex choices and bare-metal upgrade steps	2026-07-29 12:33:24 -07:00
stumpylog	ad5af61121	Fix: restore executable bit on init-llmindex-migrate run	2026-07-29 12:20:38 -07:00
stumpylog	5b7341fd3a	Feature: run pending LLM index migrations automatically on startup	2026-07-29 12:19:06 -07:00
stumpylogandClaude Sonnet 5	4d9c0e8bd5	Refactor: extract migration infrastructure, add has_pending_migration() Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 11:58:43 -07:00
shamoonandGitHub	668fa77428	Fix: prevent scale vs page loop in pngx PDF viewer (#13406 )	2026-07-29 09:25:37 -07:00
shamoonandGitHub	5bd72014a6	Fix: handle hidden line breaks in email subjects when sending email (#13402 )	2026-07-29 14:35:30 +00:00
Trenton HandGitHub	bbb9c86ba4	Fix: avoid NotSupportedError from document_importer on MariaDB (#13400 )	2026-07-29 07:14:13 -07:00
GitHub Actions	09c9fa03cf	Auto translate strings	2026-07-28 18:28:12 +00:00
shamoonandGitHub	04779d72bb	Tweak: adjust doc details button toolbar flow (#13382 )	2026-07-28 11:26:37 -07:00
Trenton HandGitHub	1b32b9d678	Fix: exclude next-period start from relative date-range filters (#13381 ) * Fix: exclude next-period start from relative date-range filters Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of the next period. Use half-open [lo TO hi} for those so e.g. "previous month" no longer matches the 1st of the current month. * Adds a regression test down to the second check for the hi range	2026-07-28 16:58:38 +00:00
Trenton HandGitHub	12d318deff	Documentation: Add Password Removal workflow action documentation (#13377 ) Addresses discussion #13373. Documents that password removal creates a new document version via re-consumption of the decrypted file rather than editing in place, and explains why the Consumption Started trigger produces an initial un-OCR'd version followed by a properly processed one.	2026-07-28 09:21:14 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	9c98f30d3f	Chore(deps): Bump pymdown-extensions in the uv group across 1 directory (#13378 ) Bumps the uv group with 1 update in the / directory: [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions). Updates `pymdown-extensions` from 10.21.3 to 11.0 - [Release notes](https://github.com/facelessuser/pymdown-extensions/releases) - [Commits](https://github.com/facelessuser/pymdown-extensions/compare/10.21.3...11.0) --- updated-dependencies: - dependency-name: pymdown-extensions dependency-version: '11.0' dependency-type: indirect dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-07-28 15:29:20 +00:00
shamoonandGitHub	6a1d7b1bca	Fix: close non-atomic db connections in before_task_publish (#13366 )	2026-07-28 07:21:02 -07:00
				`@@ -0,0 +1 @@`
				`/etc/s6-overlay/s6-rc.d/init-llmindex-migrate/run`