Chore: Drops the search shims (#13433)

* Fix: unify born-digital PDF detection between archive decision and OCR

should_produce_archive() and RasterisedDocumentParser.parse() each
reimplemented the "does this PDF have real text" check independently,
using different normalization of pdftotext output. Raw pdftotext output
can be non-empty (whitespace/form-feed layout padding) even when there
is no real content, so the two checks could disagree: consumer.py
treated a tagged-but-textless PDF as born-digital and skipped the
archive, while the parser's own (stricter, normalized) check found no
text and ran OCR anyway, leaving the document with no archive despite
real OCR text (GH #13387).

Both call sites now share one predicate, pdf_born_digital_text() in
paperless/parsers/utils.py, so they can no longer drift apart.

* Fix: restore extract_text seam for born-digital detection in parse()

parse() had switched to calling pdf_born_digital_text() directly for
its initial text/born-digital check, bypassing the parser's own
extract_text instance method. That broke test mockability (tests patch
tesseract_parser.extract_text to control the born-digital decision)
and caused CI failures with mismatched OCR call counts and text.

Split pdf_born_digital_text() into is_born_digital_text(text, path,
log) - a pure decision function - and a thin pdf_born_digital_text()
wrapper for callers without text in hand (consumer.should_produce_archive).
parse() now extracts via self.extract_text(None, document_path) and
passes the result to is_born_digital_text(), restoring the seam with
no change to production behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Cleanup: simplify born-digital detection, close #13387 test gap

Simplification pass over the born-digital detection consolidation:
- is_born_digital_text(): drop the has_text temp for an early return.
- consumer.py: standardize the archive-decision log lines on plain
  hyphens (was a mix of em-dash and hyphen) and hoist the duplicated
  text_length computation.
- Parametrize TestPdfBornDigitalText instead of four near-identical
  tests.

Code review follow-up: the existing tests only ever exercised
pdf_born_digital_text() through mocks, so the actual #13387 scenario
(a tagged PDF whose only "text" is layout padding) was never checked
against real pdftotext/pikepdf output - a regression in the
normalize-before-decide logic itself would have gone undetected.
Moved tagged_no_text_pdf_file from parsers/conftest.py up to the
shared paperless/tests/conftest.py (it was previously only visible to
tests under parsers/) and added a non-mocked regression test against
the real sample file.

Also fixed two stale comments in test_consumer.py referencing a
_extract_text_for_archive_check helper that no longer exists.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

docs/configuration.md

@@ -948,11 +948,10 @@ for display in the web interface.
 
     !!! note
 
-        The **remote OCR parser** (Azure AI) also honors this setting: when
+        The **remote OCR parser** (Azure AI) always produces a searchable
-        no archive is requested (`never`, or `auto` with a born-digital PDF),
+        PDF and stores it as the archive copy, regardless of this setting.
-        the remote engine is skipped entirely and locally-extracted text is
+        `ARCHIVE_FILE_GENERATION=never` has no effect when the remote
-        used instead, avoiding an unnecessary API call and a duplicate text
+        parser handles a document.
-        layer.
 
 #### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
 

docs/migration-v3.md

@@ -187,11 +187,10 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
 
 ### Remote OCR parser
 
-If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
+If you use the **remote OCR parser** (Azure AI), note that it always produces a
-honored the same way as for the local engine: when no archive is requested
+searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
-(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
+has no effect for documents handled by the remote parser - the archive is produced
-entirely and locally-extracted text is used instead, avoiding an unnecessary
+unconditionally by the remote engine.
-API call and a duplicate text layer.
 
 ## Search Index (Whoosh -> Tantivy)
 

src-ui/messages.xlf

@@ -1849,7 +1849,7 @@
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">154</context>
+          <context context-type="linenumber">159</context>
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/filterable-dropdown/filterable-dropdown.component.html</context>
@@ -3972,11 +3972,11 @@
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">96</context>
+          <context context-type="linenumber">101</context>
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">102</context>
+          <context context-type="linenumber">107</context>
         </context-group>
       </trans-unit>
       <trans-unit id="3800326155195149498" datatype="html">
@@ -3987,11 +3987,11 @@
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">97</context>
+          <context context-type="linenumber">102</context>
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">103</context>
+          <context context-type="linenumber">108</context>
         </context-group>
       </trans-unit>
       <trans-unit id="7551700625201096185" datatype="html">
@@ -4002,14 +4002,14 @@
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">119</context>
+          <context context-type="linenumber">124</context>
         </context-group>
       </trans-unit>
       <trans-unit id="3184700926171002527" datatype="html">
         <source>Any</source>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">152</context>
+          <context context-type="linenumber">157</context>
         </context-group>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/filterable-dropdown/filterable-dropdown.component.html</context>
@@ -4020,21 +4020,21 @@
         <source>Not</source>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">157</context>
+          <context context-type="linenumber">162</context>
         </context-group>
       </trans-unit>
       <trans-unit id="6548676277933116532" datatype="html">
         <source>Add query</source>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">176</context>
+          <context context-type="linenumber">181</context>
         </context-group>
       </trans-unit>
       <trans-unit id="5599577087865387184" datatype="html">
         <source>Add expression</source>
         <context-group purpose="location">
           <context context-type="sourcefile">src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html</context>
-          <context context-type="linenumber">179</context>
+          <context context-type="linenumber">184</context>
         </context-group>
       </trans-unit>
       <trans-unit id="6312759212949884929" datatype="html">

src-ui/src/app/components/app-frame/app-frame.component.html

@@ -182,7 +182,7 @@
                     container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
                     <i-bs class="me-2" name="stack"></i-bs><span><ng-container i18n>Attributes</ng-container></span>
                   </a>
-                  @if (!slimSidebarEnabled) {
+                  @if (!slimSidebarEnabled && canSaveSettings) {
                     <button
                       type="button"
                       class="btn btn-link btn-sm text-muted p-0 me-3 attributes-expand-btn"

src-ui/src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.html

@@ -68,6 +68,11 @@
     ></ng-select>
   } @else if (getCustomFieldByID(atom.field)?.data_type === CustomFieldDataType.DocumentLink) {
     <pngx-input-document-link [(ngModel)]="atom.value" class="w-25 form-select doc-link-select p-0" placeholder="Search docs..." i18n-placeholder [minimal]="true"></pngx-input-document-link>
+  } @else if (getCustomFieldByID(atom.field)?.data_type === CustomFieldDataType.Monetary) {
+    <input class="w-25 form-control rounded-end" type="text" inputmode="decimal"
+      [ngModel]="atom.value"
+      (ngModelChange)="setMonetaryValue(atom, $event)"
+      [disabled]="disabled">
   } @else {
     <input class="w-25 form-control rounded-end" type="text" [(ngModel)]="atom.value" [disabled]="disabled">
   }

src-ui/src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.spec.ts

@@ -1,5 +1,6 @@
 import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
 import { provideHttpClientTesting } from '@angular/common/http/testing'
+import { LOCALE_ID } from '@angular/core'
 import { ComponentFixture, TestBed } from '@angular/core/testing'
 import { FormsModule, ReactiveFormsModule } from '@angular/forms'
 import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
@@ -41,6 +42,12 @@ const customFields = [
       ],
     },
   },
+  {
+    id: 3,
+    name: 'Test Monetary Field',
+    data_type: CustomFieldDataType.Monetary,
+    extra_data: { default_currency: 'EUR' },
+  },
 ]
 
 describe('CustomFieldsQueryDropdownComponent', () => {
@@ -61,6 +68,7 @@ describe('CustomFieldsQueryDropdownComponent', () => {
       providers: [
         provideHttpClient(withInterceptorsFromDi()),
         provideHttpClientTesting(),
+        { provide: LOCALE_ID, useValue: 'de' },
       ],
     }).compileComponents()
 
@@ -150,6 +158,22 @@ describe('CustomFieldsQueryDropdownComponent', () => {
     expect(options2).toEqual([])
   })
 
+  it('should normalize localized monetary comparison values', () => {
+    const atom = new CustomFieldQueryAtom([3, 'exact', null])
+
+    component.setMonetaryValue(atom, '1.234,56')
+
+    expect(atom.value).toEqual('1234.56')
+  })
+
+  it('should preserve API-formatted monetary comparison values', () => {
+    const atom = new CustomFieldQueryAtom([3, 'exact', null])
+
+    component.setMonetaryValue(atom, '1234.56')
+
+    expect(atom.value).toEqual('1234.56')
+  })
+
   it('should remove an element from the selection model', () => {
     const expression = new CustomFieldQueryExpression()
     const atom = new CustomFieldQueryAtom()

src-ui/src/app/components/common/custom-fields-query-dropdown/custom-fields-query-dropdown.component.ts

@@ -1,9 +1,14 @@
-import { NgTemplateOutlet } from '@angular/common'
+import {
+  getLocaleNumberSymbol,
+  NgTemplateOutlet,
+  NumberSymbol,
+} from '@angular/common'
 import {
   Component,
   EventEmitter,
   inject,
   Input,
+  LOCALE_ID,
   Output,
   QueryList,
   signal,
@@ -212,6 +217,7 @@ export class CustomFieldQueriesModel {
 })
 export class CustomFieldsQueryDropdownComponent extends LoadingComponentWithPermissions {
   protected customFieldsService = inject(CustomFieldsService)
+  private readonly locale = inject(LOCALE_ID)
 
   public CustomFieldQueryComponentType = CustomFieldQueryElementType
   public CustomFieldQueryOperator = CustomFieldQueryOperator
@@ -376,4 +382,18 @@ export class CustomFieldsQueryDropdownComponent extends LoadingComponentWithPerm
     }
     return []
   }
+
+  setMonetaryValue(atom: CustomFieldQueryAtom, value: string) {
+    // Normalize the decimal symbol e.g. . vs , by locale
+    const decimalSymbol = getLocaleNumberSymbol(
+      this.locale,
+      NumberSymbol.Decimal
+    )
+    if (decimalSymbol !== '.' && value.includes(decimalSymbol)) {
+      const groupSymbol = getLocaleNumberSymbol(this.locale, NumberSymbol.Group)
+      value = value.split(groupSymbol).join('').split(decimalSymbol).join('.')
+    }
+
+    atom.value = value
+  }
 }

src-ui/src/app/components/common/pdf-viewer/pdf-viewer.component.spec.ts

@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
     ;(component as any).applyScale()
     expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
     expect(viewer.currentScale).toBe(2)
+  })
 
+  it('does not reapply scale for page-only changes', async () => {
+    await initComponent()
+
+    const pdf = (component as any).pdf as { numPages: number }
+    pdf.numPages = 3
+    const viewer = (component as any).pdfViewer as PDFViewer
+    viewer.setDocument(pdf)
     const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
     component.page = 2
-    ;(component as any).lastViewerPage = 2
+
-    ;(component as any).applyViewerState()
+    component.ngOnChanges({
+      page: new SimpleChange(1, 2, false),
+    })
+
+    expect(viewer.currentPageNumber).toBe(2)
     expect((component as any).lastViewerPage).toBeUndefined()
-    expect(applyScaleSpy).toHaveBeenCalled()
+    expect(applyScaleSpy).not.toHaveBeenCalled()
   })
 
   it('does not reset the viewer when it is already on the requested page', async () => {

src-ui/src/app/components/common/pdf-viewer/pdf-viewer.component.ts

@@ -116,7 +116,10 @@ export class PngxPdfViewerComponent
       changes['zoomScale'] ||
       changes['rotation']
     ) {
-      this.applyViewerState()
+      // Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
+      this.applyViewerState(
+        !!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
+      )
     }
 
     if (changes['searchQuery']) {
@@ -240,7 +243,7 @@ export class PngxPdfViewerComponent
     }
   }
 
-  private applyViewerState(): void {
+  private applyViewerState(applyScale = true): void {
     if (!this.pdfViewer) {
       return
     }
@@ -264,7 +267,7 @@ export class PngxPdfViewerComponent
     if (this.page === this.lastViewerPage) {
       this.lastViewerPage = undefined
     }
-    if (hasPages) {
+    if (hasPages && applyScale) {
       this.applyScale()
     }
     this.dispatchFindIfReady()

src-ui/src/app/components/manage/document-attributes/document-attributes.component.html

@@ -51,8 +51,8 @@
       *pngxIfPermissions="{ action: PermissionAction.Add, type: activeManagementList.permissionType }">
       <i-bs name="plus-circle" class="me-1"></i-bs><ng-container i18n>Create</ng-container>
     </button>
-  } @else if (activeCustomFields) {
+  } @else if (customFieldsActive) {
-    <button type="button" class="btn btn-sm btn-outline-primary" (click)="activeCustomFields.editField()"
+    <button type="button" class="btn btn-sm btn-outline-primary" (click)="addCustomField()"
       *pngxIfPermissions="{ action: PermissionAction.Add, type: PermissionType.CustomField }">
       <i-bs name="plus-circle" class="me-1"></i-bs><ng-container i18n>Add Field</ng-container>
     </button>

src-ui/src/app/components/manage/document-attributes/document-attributes.component.spec.ts

@@ -18,6 +18,7 @@ import {
   DocumentAttributesComponent,
   DocumentAttributesSectionKind,
 } from './document-attributes.component'
+import { CustomFieldsComponent } from './custom-fields/custom-fields.component'
 import { ManagementListComponent } from './management-list/management-list.component'
 
 @Component({
@@ -207,6 +208,29 @@ describe('DocumentAttributesComponent', () => {
     expect(component.activeSection.kind).toBe(
       DocumentAttributesSectionKind.CustomFields
     )
-    expect(component.activeCustomFields).toBeDefined()
+    const customFields = Object.create(CustomFieldsComponent.prototype)
+    customFields.editField = jest.fn()
+    component.activeOutlet = {
+      componentInstance: customFields,
+    } as any
+    expect(component.activeCustomFields).toBe(customFields)
+
+    component.addCustomField()
+    expect(customFields.editField).toHaveBeenCalled()
+  })
+
+  it('should show the add field button before the custom fields instance is available', async () => {
+    jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
+
+    fixture.detectChanges()
+    component.activeNavID.set(2)
+    await fixture.whenStable()
+
+    expect(component.activeCustomFields).toBeNull()
+    expect(
+      fixture.nativeElement.querySelector(
+        'pngx-page-header .btn-outline-primary'
+      )?.textContent
+    ).toContain('Add Field')
   })
 })

src-ui/src/app/components/manage/document-attributes/document-attributes.component.ts

@@ -163,12 +163,17 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
   }
 
   get activeCustomFields(): CustomFieldsComponent | null {
-    if (this.activeSection?.kind !== DocumentAttributesSectionKind.CustomFields)
+    if (!this.customFieldsActive) return null
-      return null
     const instance = this.activeOutlet?.componentInstance
     return instance instanceof CustomFieldsComponent ? instance : null
   }
 
+  get customFieldsActive(): boolean {
+    return (
+      this.activeSection?.kind === DocumentAttributesSectionKind.CustomFields
+    )
+  }
+
   get activeTabLabel(): string {
     return this.activeSection?.label ?? ''
   }
@@ -224,6 +229,10 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
     this.router.navigate(['attributes', nextSection])
   }
 
+  addCustomField(): void {
+    this.activeCustomFields?.editField(null)
+  }
+
   private getDefaultNavID(): DocumentAttributesNavIDs | null {
     return this.visibleSections[0]?.id ?? null
   }

src/documents/consumer.py

@@ -57,9 +57,7 @@ from paperless.models import ArchiveFileGenerationChoices
 from paperless.parsers import ParserContext
 from paperless.parsers import ParserProtocol
 from paperless.parsers.registry import get_parser_registry
-from paperless.parsers.utils import PDF_TEXT_MIN_LENGTH
+from paperless.parsers.utils import pdf_born_digital_text
-from paperless.parsers.utils import extract_pdf_text
-from paperless.parsers.utils import is_tagged_pdf
 
 LOGGING_NAME: Final[str] = "paperless.consumer"
 
@@ -138,53 +136,45 @@ def should_produce_archive(
 
     # Must produce a PDF so the frontend can display the original format at all.
     if parser.requires_pdf_rendition:
-        _log.debug("Archive: yes — parser requires PDF rendition for frontend display")
+        _log.debug("Archive: yes - parser requires PDF rendition for frontend display")
         return True
 
     # Parser cannot produce an archive (e.g. TextDocumentParser).
     if not parser.can_produce_archive:
-        _log.debug("Archive: no — parser cannot produce archives")
+        _log.debug("Archive: no - parser cannot produce archives")
         return False
 
     generation = OcrConfig().archive_file_generation
 
     if generation == ArchiveFileGenerationChoices.ALWAYS:
-        _log.debug("Archive: yes — ARCHIVE_FILE_GENERATION=always")
+        _log.debug("Archive: yes - ARCHIVE_FILE_GENERATION=always")
         return True
     if generation == ArchiveFileGenerationChoices.NEVER:
-        _log.debug("Archive: no — ARCHIVE_FILE_GENERATION=never")
+        _log.debug("Archive: no - ARCHIVE_FILE_GENERATION=never")
         return False
 
     # auto: produce archives for scanned/image documents; skip for born-digital PDFs.
     if mime_type.startswith("image/"):
-        _log.debug("Archive: yes — image document, ARCHIVE_FILE_GENERATION=auto")
+        _log.debug("Archive: yes - image document, ARCHIVE_FILE_GENERATION=auto")
         return True
     if mime_type == "application/pdf":
-        text = extract_pdf_text(document_path)
+        text, born_digital = pdf_born_digital_text(document_path, log=_log)
-        has_text = text is not None and len(text) > 0
+        text_length = len(text) if text else 0
-        if has_text and is_tagged_pdf(document_path):
+        if born_digital:
             _log.debug(
-                "Archive: no — born-digital PDF (structure tags detected),"
+                "Archive: no - born-digital PDF (text_length=%d),"
                 " ARCHIVE_FILE_GENERATION=auto",
+                text_length,
             )
             return False
-        if text is None or len(text) <= PDF_TEXT_MIN_LENGTH:
-            _log.debug(
-                "Archive: yes — scanned PDF (text_length=%d ≤ %d),"
-                " ARCHIVE_FILE_GENERATION=auto",
-                len(text) if text else 0,
-                PDF_TEXT_MIN_LENGTH,
-            )
-            return True
         _log.debug(
-            "Archive: no — born-digital PDF (text_length=%d > %d),"
+            "Archive: yes - scanned/textless PDF (text_length=%d),"
             " ARCHIVE_FILE_GENERATION=auto",
-            len(text),
+            text_length,
-            PDF_TEXT_MIN_LENGTH,
         )
-        return False
+        return True
     _log.debug(
-        "Archive: no — MIME type %r not eligible for auto archive generation",
+        "Archive: no - MIME type %r not eligible for auto archive generation",
         mime_type,
     )
     return False

src/documents/mail.py

@@ -36,6 +36,9 @@ def send_email(
 
     TODO: re-evaluate this pending https://code.djangoproject.com/ticket/35581 / https://github.com/django/django/pull/18966
     """
+    if "\r" in subject or "\n" in subject:
+        subject = " ".join(line.strip(" \t") for line in subject.splitlines())
+
     email = EmailMessage(
         subject=subject,
         body=body,

src/documents/management/commands/document_importer.py

@@ -386,10 +386,19 @@ class Command(CryptMixin, PaperlessCommand):
                 raise DeserializationError(
                     f"{model.__name__} has no updatable fields; PK-only models are not supported by the importer",
                 )
+            # MySQL/MariaDB support upserts via ON DUPLICATE KEY UPDATE but,
+            # unlike PostgreSQL/SQLite, cannot target a specific unique field
+            # for the conflict -- passing unique_fields there raises
+            # NotSupportedError.
+            unique_fields = (
+                [model._meta.pk.attname]
+                if connection.features.supports_update_conflicts_with_target
+                else None
+            )
             model.objects.bulk_create(  # type: ignore[attr-defined]
                 instances,
                 update_conflicts=True,
-                unique_fields=[model._meta.pk.attname],
+                unique_fields=unique_fields,
                 update_fields=update_fields,
             )
             loaded_models.add(model)

src/documents/management/commands/document_sanity_checker.py

@@ -61,7 +61,7 @@ class Command(PaperlessCommand):
         )
         table.add_column("Level", width=7, no_wrap=True)
         table.add_column("Document", min_width=20)
-        table.add_column("Issue", ratio=1)
+        table.add_column("Issue", ratio=1, overflow="fold")
 
         for doc_pk, doc_messages in messages.iter_messages():
             if doc_pk is not None:

src/documents/search/_query.py

@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import logging
-from datetime import UTC
 from typing import TYPE_CHECKING
 from typing import Final
 
@@ -9,12 +8,6 @@ import regex
 import tantivy
 from django.conf import settings
 
-from documents.search._dates import (
-    _date_only_range,  # noqa: F401 — re-exported for test imports
-)
-from documents.search._dates import (
-    _datetime_range,  # noqa: F401 — re-exported for test imports
-)
 from documents.search._tokenizer import simple_search_tokens
 from documents.search._translate import SearchQueryError
 from documents.search._translate import translate_query
@@ -76,42 +69,6 @@ def _build_cjk_query(
         return None
 
 
-def rewrite_natural_date_keywords(query: str, tz: tzinfo) -> str:
-    """
-    Rewrite natural date syntax to ISO 8601 format for Tantivy compatibility.
-
-    Delegates to ``translate_query`` which handles all date forms, comma
-    expansion, field aliasing, relative ranges, and operator normalization.
-
-    Args:
-        query: Raw user query string
-        tz: Timezone for converting local date boundaries to UTC
-
-    Returns:
-        Query with date syntax rewritten to ISO 8601 ranges
-
-    Note:
-        Bare keywords without field prefixes pass through unchanged.
-    """
-    return translate_query(query, tz)
-
-
-def normalize_query(query: str) -> str:
-    """
-    Normalize query syntax for better search behavior.
-
-    Delegates to ``translate_query`` which handles comma expansion, whitespace
-    collapsing, operator normalization, and field aliasing.
-
-    Args:
-        query: Query string after date rewriting
-
-    Returns:
-        Normalized query string ready for Tantivy parsing
-    """
-    return translate_query(query, UTC)
-
-
 def build_permission_filter(
     schema: tantivy.Schema,
     user: AbstractBaseUser,

src/documents/tests/search/test_query.py

@@ -11,16 +11,15 @@ import pytest
 import tantivy
 import time_machine
 
-from documents.search._query import _date_only_range
+from documents.search._dates import _date_only_range
-from documents.search._query import _datetime_range
+from documents.search._dates import _datetime_range
 from documents.search._query import build_permission_filter
-from documents.search._query import normalize_query
 from documents.search._query import parse_simple_text_highlight_query
 from documents.search._query import parse_user_query
-from documents.search._query import rewrite_natural_date_keywords
 from documents.search._schema import build_schema
 from documents.search._tokenizer import register_tokenizers
 from documents.search._translate import InvalidDateQuery
+from documents.search._translate import translate_query
 
 if TYPE_CHECKING:
     from django.contrib.auth.base_user import AbstractBaseUser
@@ -59,7 +58,7 @@ class TestCreatedDateField:
     )
     @time_machine.travel(datetime(2026, 3, 28, 15, 30, tzinfo=UTC), tick=False)
     def test_today(self, tz: tzinfo, expected_lo: str, expected_hi: str) -> None:
-        lo, hi = _range(rewrite_natural_date_keywords("created:today", tz), "created")
+        lo, hi = _range(translate_query("created:today", tz), "created")
         assert lo == expected_lo
         assert hi == expected_hi
 
@@ -67,7 +66,7 @@ class TestCreatedDateField:
     def test_today_auckland_ahead_of_utc(self) -> None:
         # UTC 03:00 -> Auckland (UTC+13) = 16:00 same date; local date = 2026-03-28
         lo, _ = _range(
-            rewrite_natural_date_keywords("created:today", AUCKLAND),
+            translate_query("created:today", AUCKLAND),
             "created",
         )
         assert lo == "2026-03-28T00:00:00Z"
@@ -129,7 +128,7 @@ class TestCreatedDateField:
     ) -> None:
         # 2026-03-28 is Saturday; Mon-Sun week calculation built into expectations
         query = f"{field}:{keyword}"
-        lo, hi = _range(rewrite_natural_date_keywords(query, UTC), field)
+        lo, hi = _range(translate_query(query, UTC), field)
         assert lo == expected_lo
         assert hi == expected_hi
 
@@ -137,7 +136,7 @@ class TestCreatedDateField:
     def test_this_month_december_wraps_to_next_year(self) -> None:
         # December: next month must roll over to January 1 of next year
         lo, hi = _range(
-            rewrite_natural_date_keywords("created:this month", UTC),
+            translate_query("created:this month", UTC),
             "created",
         )
         assert lo == "2026-12-01T00:00:00Z"
@@ -147,7 +146,7 @@ class TestCreatedDateField:
     def test_last_month_january_wraps_to_previous_year(self) -> None:
         # January: last month must roll back to December 1 of previous year
         lo, hi = _range(
-            rewrite_natural_date_keywords("created:previous month", UTC),
+            translate_query("created:previous month", UTC),
             "created",
         )
         assert lo == "2025-12-01T00:00:00Z"
@@ -156,7 +155,7 @@ class TestCreatedDateField:
     @time_machine.travel(datetime(2026, 7, 15, 12, 0, tzinfo=UTC), tick=False)
     def test_previous_quarter(self) -> None:
         lo, hi = _range(
-            rewrite_natural_date_keywords('created:"previous quarter"', UTC),
+            translate_query('created:"previous quarter"', UTC),
             "created",
         )
         assert lo == "2026-04-01T00:00:00Z"
@@ -176,7 +175,7 @@ class TestDateTimeFields:
     @time_machine.travel(datetime(2026, 3, 28, 15, 30, tzinfo=UTC), tick=False)
     def test_added_today_eastern(self) -> None:
         # EDT = UTC-4; local midnight 2026-03-28 00:00 EDT = 2026-03-28 04:00 UTC
-        lo, hi = _range(rewrite_natural_date_keywords("added:today", EASTERN), "added")
+        lo, hi = _range(translate_query("added:today", EASTERN), "added")
         assert lo == "2026-03-28T04:00:00Z"
         assert hi == "2026-03-29T04:00:00Z"
 
@@ -184,14 +183,14 @@ class TestDateTimeFields:
     def test_added_today_auckland_midnight_crossing(self) -> None:
         # UTC 02:00 on 2026-03-29 -> Auckland (UTC+13) = 2026-03-29 15:00 local
         # Auckland midnight = UTC 2026-03-28 11:00
-        lo, hi = _range(rewrite_natural_date_keywords("added:today", AUCKLAND), "added")
+        lo, hi = _range(translate_query("added:today", AUCKLAND), "added")
         assert lo == "2026-03-28T11:00:00Z"
         assert hi == "2026-03-29T11:00:00Z"
 
     @time_machine.travel(datetime(2026, 3, 28, 15, 0, tzinfo=UTC), tick=False)
     def test_modified_today_utc(self) -> None:
         lo, hi = _range(
-            rewrite_natural_date_keywords("modified:today", UTC),
+            translate_query("modified:today", UTC),
             "modified",
         )
         assert lo == "2026-03-28T00:00:00Z"
@@ -246,14 +245,14 @@ class TestDateTimeFields:
         expected_hi: str,
     ) -> None:
         # 2026-03-28 is Saturday; weekday()==5 so Monday=2026-03-23
-        lo, hi = _range(rewrite_natural_date_keywords(f"added:{keyword}", UTC), "added")
+        lo, hi = _range(translate_query(f"added:{keyword}", UTC), "added")
         assert lo == expected_lo
         assert hi == expected_hi
 
     @time_machine.travel(datetime(2026, 12, 15, 12, 0, tzinfo=UTC), tick=False)
     def test_this_month_december_wraps_to_next_year(self) -> None:
         # December: next month wraps to January of next year
-        lo, hi = _range(rewrite_natural_date_keywords("added:this month", UTC), "added")
+        lo, hi = _range(translate_query("added:this month", UTC), "added")
         assert lo == "2026-12-01T00:00:00Z"
         assert hi == "2027-01-01T00:00:00Z"
 
@@ -261,7 +260,7 @@ class TestDateTimeFields:
     def test_last_month_january_wraps_to_previous_year(self) -> None:
         # January: last month wraps back to December of previous year
         lo, hi = _range(
-            rewrite_natural_date_keywords("added:previous month", UTC),
+            translate_query("added:previous month", UTC),
             "added",
         )
         assert lo == "2025-12-01T00:00:00Z"
@@ -297,7 +296,7 @@ class TestDateTimeFields:
         expected_lo: str,
         expected_hi: str,
     ) -> None:
-        lo, hi = _range(rewrite_natural_date_keywords(query, UTC), "added")
+        lo, hi = _range(translate_query(query, UTC), "added")
         assert lo == expected_lo
         assert hi == expected_hi
 
@@ -311,20 +310,20 @@ class TestWhooshQueryRewriting:
 
     @time_machine.travel(datetime(2026, 3, 28, 15, 0, tzinfo=UTC), tick=False)
     def test_compact_date_shim_rewrites_to_iso(self) -> None:
-        result = rewrite_natural_date_keywords("created:20240115120000", UTC)
+        result = translate_query("created:20240115120000", UTC)
         assert "2024-01-15" in result
         assert "20240115120000" not in result
 
     @time_machine.travel(datetime(2026, 3, 28, 15, 0, tzinfo=UTC), tick=False)
     def test_relative_range_shim_removes_now(self) -> None:
-        result = rewrite_natural_date_keywords("added:[now-7d TO now]", UTC)
+        result = translate_query("added:[now-7d TO now]", UTC)
         assert "now" not in result
         assert "2026-03-" in result
 
     @time_machine.travel(datetime(2026, 3, 28, 12, 0, tzinfo=UTC), tick=False)
     def test_bracket_minus_7_days(self) -> None:
         lo, hi = _range(
-            rewrite_natural_date_keywords("added:[-7 days to now]", UTC),
+            translate_query("added:[-7 days to now]", UTC),
             "added",
         )
         assert lo == "2026-03-21T12:00:00Z"
@@ -333,7 +332,7 @@ class TestWhooshQueryRewriting:
     @time_machine.travel(datetime(2026, 3, 28, 12, 0, tzinfo=UTC), tick=False)
     def test_bracket_minus_1_week(self) -> None:
         lo, hi = _range(
-            rewrite_natural_date_keywords("added:[-1 week to now]", UTC),
+            translate_query("added:[-1 week to now]", UTC),
             "added",
         )
         assert lo == "2026-03-21T12:00:00Z"
@@ -343,7 +342,7 @@ class TestWhooshQueryRewriting:
     def test_bracket_minus_1_month_uses_relativedelta(self) -> None:
         # relativedelta(months=1) from 2026-03-28 = 2026-02-28 (not 29)
         lo, hi = _range(
-            rewrite_natural_date_keywords("created:[-1 month to now]", UTC),
+            translate_query("created:[-1 month to now]", UTC),
             "created",
         )
         assert lo == "2026-02-28T12:00:00Z"
@@ -352,7 +351,7 @@ class TestWhooshQueryRewriting:
     @time_machine.travel(datetime(2026, 3, 28, 12, 0, tzinfo=UTC), tick=False)
     def test_bracket_minus_1_year(self) -> None:
         lo, hi = _range(
-            rewrite_natural_date_keywords("modified:[-1 year to now]", UTC),
+            translate_query("modified:[-1 year to now]", UTC),
             "modified",
         )
         assert lo == "2025-03-28T12:00:00Z"
@@ -361,7 +360,7 @@ class TestWhooshQueryRewriting:
     @time_machine.travel(datetime(2026, 3, 28, 12, 0, tzinfo=UTC), tick=False)
     def test_bracket_plural_unit_hours(self) -> None:
         lo, hi = _range(
-            rewrite_natural_date_keywords("added:[-3 hours to now]", UTC),
+            translate_query("added:[-3 hours to now]", UTC),
             "added",
         )
         assert lo == "2026-03-28T09:00:00Z"
@@ -369,7 +368,7 @@ class TestWhooshQueryRewriting:
 
     @time_machine.travel(datetime(2026, 3, 28, 12, 0, tzinfo=UTC), tick=False)
     def test_bracket_case_insensitive(self) -> None:
-        result = rewrite_natural_date_keywords("added:[-1 WEEK TO NOW]", UTC)
+        result = translate_query("added:[-1 WEEK TO NOW]", UTC)
         assert "now" not in result.lower()
         lo, hi = _range(result, "added")
         assert lo == "2026-03-21T12:00:00Z"
@@ -379,7 +378,7 @@ class TestWhooshQueryRewriting:
     def test_relative_range_swaps_bounds_when_lo_exceeds_hi(self) -> None:
         # [now+1h TO now-1h] has lo > hi before substitution; they must be swapped
         lo, hi = _range(
-            rewrite_natural_date_keywords("added:[now+1h TO now-1h]", UTC),
+            translate_query("added:[now+1h TO now-1h]", UTC),
             "added",
         )
         assert lo == "2026-03-28T11:00:00Z"
@@ -387,14 +386,14 @@ class TestWhooshQueryRewriting:
 
     def test_8digit_created_date_field_always_uses_utc_midnight(self) -> None:
         # created is a DateField: boundaries are always UTC midnight, no TZ offset
-        result = rewrite_natural_date_keywords("created:20231201", EASTERN)
+        result = translate_query("created:20231201", EASTERN)
         lo, hi = _range(result, "created")
         assert lo == "2023-12-01T00:00:00Z"
         assert hi == "2023-12-02T00:00:00Z"
 
     def test_8digit_added_datetime_field_converts_local_midnight_to_utc(self) -> None:
         # added is DateTimeField: midnight Dec 1 Eastern (EST = UTC-5) = 05:00 UTC
-        result = rewrite_natural_date_keywords("added:20231201", EASTERN)
+        result = translate_query("added:20231201", EASTERN)
         lo, hi = _range(result, "added")
         assert lo == "2023-12-01T05:00:00Z"
         assert hi == "2023-12-02T05:00:00Z"
@@ -402,7 +401,7 @@ class TestWhooshQueryRewriting:
     def test_8digit_modified_datetime_field_converts_local_midnight_to_utc(
         self,
     ) -> None:
-        result = rewrite_natural_date_keywords("modified:20231201", EASTERN)
+        result = translate_query("modified:20231201", EASTERN)
         lo, hi = _range(result, "modified")
         assert lo == "2023-12-01T05:00:00Z"
         assert hi == "2023-12-02T05:00:00Z"
@@ -412,7 +411,7 @@ class TestWhooshQueryRewriting:
         # (e.g. month=13) so the API can surface a 400 telling the user the date
         # is malformed instead of silently returning zero results.
         with pytest.raises(InvalidDateQuery) as exc_info:
-            rewrite_natural_date_keywords("added:20231340", UTC)
+            translate_query("added:20231340", UTC)
         assert exc_info.value.field == "added"
         assert exc_info.value.value == "20231340"
 
@@ -579,7 +578,7 @@ class TestYearRangeRewriting:
         expected_lo: str,
         expected_hi: str,
     ) -> None:
-        result = rewrite_natural_date_keywords(query, UTC)
+        result = translate_query(query, UTC)
         lo, hi = _range(result, field)
         assert lo == expected_lo
         assert hi == expected_hi
@@ -587,14 +586,14 @@ class TestYearRangeRewriting:
     def test_reversed_year_range_is_swapped(self) -> None:
         # A reversed range must not yield lo > hi, which Tantivy treats as an
         # empty range (silently zero results). The bounds are swapped instead.
-        result = rewrite_natural_date_keywords("created:[2025 TO 2020]", UTC)
+        result = translate_query("created:[2025 TO 2020]", UTC)
         lo, hi = _range(result, "created")
         assert lo == "2020-01-01T00:00:00Z"
         assert hi == "2026-01-01T00:00:00Z"
 
     def test_year_range_in_complex_boolean_query(self) -> None:
         query = "tag:steuer AND (title:2020 OR (NOT title:2019 AND NOT title:2018 AND created:[2020 TO 2020]))"
-        result = rewrite_natural_date_keywords(query, UTC)
+        result = translate_query(query, UTC)
         lo, hi = _range(result, "created")
         assert lo == "2020-01-01T00:00:00Z"
         assert hi == "2021-01-01T00:00:00Z"
@@ -604,7 +603,7 @@ class TestYearRangeRewriting:
 
     def test_already_iso_date_range_passes_through_unchanged(self) -> None:
         original = "created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]"
-        assert rewrite_natural_date_keywords(original, UTC) == original
+        assert translate_query(original, UTC) == original
 
     def test_8digit_in_brackets_not_matched_as_year_range(self) -> None:
         # [YYYYMMDD TO YYYYMMDD]: the translation layer converts 8-digit bounds to
@@ -613,7 +612,7 @@ class TestYearRangeRewriting:
         # This is the correct and accepted behavior: old compact form becomes a
         # proper Tantivy-parseable ISO range.
         original = "created:[20200101 TO 20201231]"
-        result = rewrite_natural_date_keywords(original, UTC)
+        result = translate_query(original, UTC)
         lo, hi = _range(result, "created")
         assert lo == "2020-01-01T00:00:00Z"
         assert hi == "2021-01-01T00:00:00Z"
@@ -636,7 +635,7 @@ class TestNonDateFieldsNotRewritten:
         ],
     )
     def test_8digit_on_integer_field_passes_through_unchanged(self, query: str) -> None:
-        assert rewrite_natural_date_keywords(query, EASTERN) == query
+        assert translate_query(query, EASTERN) == query
 
     @pytest.mark.parametrize(
         "query",
@@ -650,12 +649,12 @@ class TestNonDateFieldsNotRewritten:
         self,
         query: str,
     ) -> None:
-        assert rewrite_natural_date_keywords(query, UTC) == query
+        assert translate_query(query, UTC) == query
 
     def test_unknown_field_keyword_passes_through_unchanged(self) -> None:
         # foobar is not a date field: 'foobar:today' must not become a date range,
         # which Tantivy would otherwise reject as an unknown/typed field.
-        assert rewrite_natural_date_keywords("foobar:today", UTC) == "foobar:today"
+        assert translate_query("foobar:today", UTC) == "foobar:today"
 
 
 class TestPassthrough:
@@ -663,37 +662,39 @@ class TestPassthrough:
 
     def test_bare_keyword_no_field_prefix_unchanged(self) -> None:
         # Bare 'today' with no field: prefix passes through unchanged
-        result = rewrite_natural_date_keywords("bank statement today", UTC)
+        result = translate_query("bank statement today", UTC)
         assert "today" in result
 
     def test_unrelated_query_unchanged(self) -> None:
-        assert rewrite_natural_date_keywords("title:invoice", UTC) == "title:invoice"
+        assert translate_query("title:invoice", UTC) == "title:invoice"
 
 
 class TestNormalizeQuery:
-    """normalize_query expands comma-separated values and collapses whitespace."""
+    """translate_query expands comma-separated values and collapses whitespace."""
 
     def test_normalize_expands_comma_separated_tags(self) -> None:
-        assert normalize_query("tag:foo,bar") == "tag:foo AND tag:bar"
+        assert translate_query("tag:foo,bar", UTC) == "tag:foo AND tag:bar"
 
     def test_normalize_comma_between_range_expressions(self) -> None:
         # Comma-separated field range expressions (Whoosh v2 syntax) must be
         # converted to AND so Tantivy does not receive an invalid comma.
         q = "created:[2026-01-01T00:00:00Z TO 2026-06-01T00:00:00Z],added:[2026-05-01T00:00:00Z TO 2026-06-01T00:00:00Z]"
-        assert normalize_query(q) == (
+        assert translate_query(q, UTC) == (
             "created:[2026-01-01T00:00:00Z TO 2026-06-01T00:00:00Z]"
             " AND "
             "added:[2026-05-01T00:00:00Z TO 2026-06-01T00:00:00Z]"
         )
 
     def test_normalize_expands_three_values(self) -> None:
-        assert normalize_query("tag:foo,bar,baz") == "tag:foo AND tag:bar AND tag:baz"
+        assert (
+            translate_query("tag:foo,bar,baz", UTC) == "tag:foo AND tag:bar AND tag:baz"
+        )
 
     def test_normalize_collapses_whitespace(self) -> None:
-        assert normalize_query("bank   statement") == "bank statement"
+        assert translate_query("bank   statement", UTC) == "bank statement"
 
     def test_normalize_no_commas_unchanged(self) -> None:
-        assert normalize_query("bank statement") == "bank statement"
+        assert translate_query("bank statement", UTC) == "bank statement"
 
     @pytest.mark.parametrize(
         ("raw", "expected"),
@@ -736,7 +737,7 @@ class TestNormalizeQuery:
         ],
     )
     def test_normalize_strips_dangling_operators(self, raw: str, expected: str) -> None:
-        assert normalize_query(raw) == expected
+        assert translate_query(raw, UTC) == expected
 
     @pytest.mark.parametrize(
         "query",
@@ -748,7 +749,7 @@ class TestNormalizeQuery:
         ],
     )
     def test_normalize_preserves_valid_operators(self, query: str) -> None:
-        assert normalize_query(query) == query
+        assert translate_query(query, UTC) == query
 
 
 class TestParseSimpleTextHighlightQuery:

src/documents/tests/test_api_email.py

@@ -75,7 +75,7 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
                 {
                     "documents": [self.doc1.pk, self.doc2.pk],
                     "addresses": "hello@paperless-ngx.com,test@example.com",
-                    "subject": "Bulk email test",
+                    "subject": "Bulk email\n test",
                     "message": "Here are your documents",
                 },
             ),

src/documents/tests/test_consumer.py

@@ -1329,7 +1329,7 @@ class PreConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
         with self.get_consumer(self.test_file) as c:
             c.run()
             # Verify no pre-consume script subprocess was invoked
-            # (run_subprocess may still be called by _extract_text_for_archive_check)
+            # (run_subprocess may still be called by pdf_born_digital_text via pdftotext)
             script_calls = [
                 call
                 for call in m.call_args_list
@@ -1354,7 +1354,7 @@ class PreConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
                     self.assertTrue(m.called)
 
                     # Find the call that invoked the pre-consume script
-                    # (run_subprocess may also be called by _extract_text_for_archive_check)
+                    # (run_subprocess may also be called by pdf_born_digital_text via pdftotext)
                     script_call = next(
                         call
                         for call in m.call_args_list

src/documents/tests/test_consumer_archive.py

@@ -134,60 +134,32 @@ class TestShouldProduceArchive:
         assert should_produce_archive(parser, mime, Path("/tmp/doc")) is expected
 
     @pytest.mark.parametrize(
-        ("extracted_text", "expected"),
+        ("born_digital", "expected"),
         [
-            pytest.param(
+            pytest.param(True, False, id="born-digital-skips-archive"),
-                "This is a born-digital PDF with lots of text content. " * 10,
+            pytest.param(False, True, id="not-born-digital-produces-archive"),
-                False,
-                id="born-digital-long-text-skips-archive",
-            ),
-            pytest.param(None, True, id="no-text-scanned-produces-archive"),
-            pytest.param("tiny", True, id="short-text-treated-as-scanned"),
         ],
     )
     def test_auto_pdf_archive_decision(
         self,
         mocker: MockerFixture,
         settings,
-        extracted_text: str | None,
+        born_digital: bool,  # noqa: FBT001
         expected: bool,  # noqa: FBT001
     ) -> None:
+        """Archive decision tracks pdf_born_digital_text()'s verdict exactly.
+
+        should_produce_archive() defers entirely to pdf_born_digital_text()
+        for the has-real-text decision, so both callers of that predicate
+        (this function and RasterisedDocumentParser.parse()) always agree.
+        """
         settings.ARCHIVE_FILE_GENERATION = "auto"
-        mocker.patch("documents.consumer.is_tagged_pdf", return_value=False)
+        mocker.patch(
-        mocker.patch("documents.consumer.extract_pdf_text", return_value=extracted_text)
+            "documents.consumer.pdf_born_digital_text",
+            return_value=("some text", born_digital),
+        )
         parser = _parser_instance(can_produce=True, requires_rendition=False)
         assert (
             should_produce_archive(parser, "application/pdf", Path("/tmp/doc.pdf"))
             is expected
         )
-
-    def test_tagged_pdf_skips_archive_in_auto_mode(
-        self,
-        mocker: MockerFixture,
-        settings,
-    ) -> None:
-        """Tagged PDFs (e.g. Word exports) with real text are treated as born-digital, even below PDF_TEXT_MIN_LENGTH."""
-        settings.ARCHIVE_FILE_GENERATION = "auto"
-        mocker.patch("documents.consumer.is_tagged_pdf", return_value=True)
-        mocker.patch("documents.consumer.extract_pdf_text", return_value="tiny")
-        parser = _parser_instance(can_produce=True, requires_rendition=False)
-        assert (
-            should_produce_archive(parser, "application/pdf", Path("/tmp/doc.pdf"))
-            is False
-        )
-
-    def test_tagged_pdf_without_text_produces_archive(
-        self,
-        mocker: MockerFixture,
-        settings,
-    ) -> None:
-        """A tagged PDF with no actual extractable text (e.g. some scanner firmware) is not
-        trusted as born-digital — the tag alone must not bypass OCR."""
-        settings.ARCHIVE_FILE_GENERATION = "auto"
-        mocker.patch("documents.consumer.is_tagged_pdf", return_value=True)
-        mocker.patch("documents.consumer.extract_pdf_text", return_value=None)
-        parser = _parser_instance(can_produce=True, requires_rendition=False)
-        assert (
-            should_produce_archive(parser, "application/pdf", Path("/tmp/doc.pdf"))
-            is True
-        )

src/paperless/parsers/remote.py

@@ -3,9 +3,7 @@ Built-in remote-OCR document parser.
 
 Handles documents by sending them to a configured remote OCR engine
 (currently Azure AI Vision / Document Intelligence) and retrieving both
-the extracted text and a searchable PDF with an embedded text layer. For
+the extracted text and a searchable PDF with an embedded text layer.
-born-digital PDFs that need no archive copy, the remote call is skipped
-entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
 
 When no engine is configured, ``score()`` returns ``None`` so the parser
 is effectively invisible to the registry — the tesseract parser handles
@@ -23,8 +21,6 @@ from typing import Self
 
 from django.conf import settings
 
-from paperless.parsers.utils import extract_pdf_text
-from paperless.parsers.utils import post_process_text
 from paperless.version import __full_version_str__
 
 if TYPE_CHECKING:
@@ -73,11 +69,8 @@ class RemoteDocumentParser:
     """Parse documents via a remote OCR API (currently Azure AI Vision).
 
     This parser sends documents to a remote engine that returns both
-    extracted text and a searchable PDF with an embedded text layer,
+    extracted text and a searchable PDF with an embedded text layer.
-    except when ``parse()`` is called with ``produce_archive=False`` for
+    It does not depend on Tesseract or ocrmypdf.
-    a PDF, in which case the remote call is skipped and only locally
-    extracted text is returned (no archive). It does not depend on
-    Tesseract or ocrmypdf.
 
     Class attributes
     ----------------
@@ -166,11 +159,8 @@ class RemoteDocumentParser:
         Returns
         -------
         bool
-            Always True — the remote engine is capable of returning a PDF
+            Always True — the remote engine always returns a PDF with an
-            with an embedded text layer to serve as the archive copy.
+            embedded text layer that serves as the archive copy.
-            Whether it actually does so for a given document depends on
-            ``produce_archive`` passed to :meth:`parse` (see there for when
-            the remote engine call, and thus archive generation, is skipped).
         """
         return True
 
@@ -227,12 +217,6 @@ class RemoteDocumentParser:
     ) -> None:
         """Send the document to the remote engine and store results.
 
-        When *produce_archive* is False for a PDF, the caller (via
-        ``documents.consumer.should_produce_archive``) has already determined
-        that the document is born-digital and needs no archive — skip the
-        remote engine entirely rather than re-OCRing it and creating a
-        duplicate text layer.
-
         Parameters
         ----------
         document_path:
@@ -240,8 +224,8 @@ class RemoteDocumentParser:
         mime_type:
             Detected MIME type of the document.
         produce_archive:
-            Whether an archive copy is wanted. For PDFs, False skips the
+            Ignored — the remote engine always returns a searchable PDF,
-            remote engine and uses locally-extracted text instead.
+            which is stored as the archive copy regardless of this flag.
         """
         config = RemoteEngineConfig(
             engine=settings.REMOTE_OCR_ENGINE,
@@ -256,16 +240,6 @@ class RemoteDocumentParser:
             self._text = ""
             return
 
-        if not produce_archive and mime_type == "application/pdf":
-            logger.debug(
-                "Remote OCR: skipped — no archive requested, "
-                "using locally-extracted text",
-            )
-            self._text = (
-                post_process_text(extract_pdf_text(document_path, log=logger)) or ""
-            )
-            return
-
         if config.engine == "azureai":
             self._text = self._azure_ai_vision_parse(document_path, config)
 

src/paperless/parsers/tesseract.py

@@ -25,7 +25,7 @@ from paperless.models import CleanChoices
 from paperless.models import ModeChoices
 from paperless.models import OutputTypeChoices
 from paperless.parsers.utils import extract_pdf_text
-from paperless.parsers.utils import pdf_has_digital_text
+from paperless.parsers.utils import is_born_digital_text
 from paperless.parsers.utils import post_process_text
 from paperless.parsers.utils import read_file_handle_unicode_errors
 from paperless.version import __full_version_str__
@@ -509,9 +509,9 @@ class RasterisedDocumentParser:
 
         if mime_type == "application/pdf":
             text_original = self.extract_text(None, document_path)
-            original_has_text = pdf_has_digital_text(
+            original_has_text = is_born_digital_text(
-                document_path,
                 text_original,
+                document_path,
                 log=self.log,
             )
         else:

src/paperless/parsers/utils.py

@@ -65,63 +65,6 @@ def is_tagged_pdf(
         return False
 
 
-def pdf_has_digital_text(
-    path: Path,
-    text: str | None,
-    log: logging.Logger | None = None,
-) -> bool:
-    """Return True if a PDF already has a usable, born-digital text layer.
-
-    Combines the tagged-PDF check with an extracted-text length check.
-    Shared by the tesseract and remote OCR parsers to decide whether
-    OCR_MODE=auto/off should skip (re-)OCRing a document.
-
-    Parameters
-    ----------
-    path:
-        Absolute path to the PDF file.
-    text:
-        Text already extracted from the PDF (e.g. via ``extract_pdf_text``),
-        or ``None``.
-    log:
-        Logger for warnings.  Falls back to the module-level logger when omitted.
-
-    Returns
-    -------
-    bool
-        ``True`` when the document already contains a text layer.
-    """
-    return is_tagged_pdf(path, log=log) or (
-        text is not None and len(text) > PDF_TEXT_MIN_LENGTH
-    )
-
-
-def post_process_text(text: str | None) -> str | None:
-    """Normalise whitespace in extracted OCR/PDF text and strip NUL bytes.
-
-    Parameters
-    ----------
-    text:
-        Raw extracted text, or ``None``.
-
-    Returns
-    -------
-    str | None
-        Cleaned text, or ``None`` when *text* is falsy.
-    """
-    if not text:
-        return None
-
-    collapsed_spaces = re.sub(r"([^\S\r\n]+)", " ", text)
-    no_leading_whitespace = re.sub(r"([\n\r]+)([^\S\n\r]+)", "\\1", collapsed_spaces)
-    no_trailing_whitespace = re.sub(r"([^\S\n\r]+)$", "", no_leading_whitespace)
-
-    # TODO: this needs a rework
-    # replace \0 prevents issues with saving to postgres.
-    # text may contain \0 when this character is present in PDF files.
-    return no_trailing_whitespace.strip().replace("\0", " ")
-
-
 def extract_pdf_text(
     path: Path,
     log: logging.Logger | None = None,
@@ -168,6 +111,88 @@ def extract_pdf_text(
         return None
 
 
+def post_process_text(text: str | None) -> str | None:
+    """Normalize extracted PDF/OCR text: collapse whitespace, strip padding.
+
+    Returns ``None`` for ``None`` or whitespace-only input, so callers can
+    treat "no text" and "only layout padding" the same way.
+    """
+    if not text:
+        return None
+
+    collapsed_spaces = re.sub(r"([^\S\r\n]+)", " ", text)
+    no_leading_whitespace = re.sub(r"([\n\r]+)([^\S\n\r]+)", "\\1", collapsed_spaces)
+    no_trailing_whitespace = re.sub(r"([^\S\n\r]+)$", "", no_leading_whitespace)
+
+    # replace \0 prevents issues with saving to postgres.
+    # text may contain \0 when this character is present in PDF files.
+    result = no_trailing_whitespace.strip().replace("\0", " ")
+    return result or None
+
+
+def is_born_digital_text(
+    text: str | None,
+    path: Path,
+    log: logging.Logger | None = None,
+) -> bool:
+    """Decide whether already-extracted, normalized PDF text counts as born-digital.
+
+    This is the single source of truth for "does this PDF already have real
+    text", used both to decide whether to produce an archive file and to
+    decide whether OCR can be skipped. Both decisions must agree, or a
+    tagged-but-textless PDF can end up with no archive AND a forced OCR pass
+    (see GH #13387): raw ``pdftotext -layout`` output can be non-empty
+    (whitespace/form-feed padding) even when there is no real content, so
+    *text* must already be normalized via :func:`post_process_text`, not the
+    raw extraction.
+
+    Parameters
+    ----------
+    text:
+        The normalized extracted text (or ``None``) to evaluate.
+    path:
+        Absolute path to the PDF file, used for the tagged-PDF check.
+    log:
+        Logger for warnings.  Falls back to the module-level logger when omitted.
+
+    Returns
+    -------
+    bool
+        Whether the PDF counts as born-digital (has real text, and is either
+        tagged or exceeds ``PDF_TEXT_MIN_LENGTH``).
+    """
+    if not text:
+        return False
+    return is_tagged_pdf(path, log=log) or len(text) > PDF_TEXT_MIN_LENGTH
+
+
+def pdf_born_digital_text(
+    path: Path,
+    log: logging.Logger | None = None,
+) -> tuple[str | None, bool]:
+    """Extract a PDF's text and decide whether it should be treated as born-digital.
+
+    Convenience wrapper around :func:`is_born_digital_text` for callers that
+    don't already have the PDF's text extracted (e.g. the archive-generation
+    decision, which runs before any parser has touched the file).
+
+    Parameters
+    ----------
+    path:
+        Absolute path to the PDF file.
+    log:
+        Logger for warnings.  Falls back to the module-level logger when omitted.
+
+    Returns
+    -------
+    tuple[str | None, bool]
+        The normalized extracted text (or ``None``), and whether the PDF
+        counts as born-digital.
+    """
+    text = post_process_text(extract_pdf_text(path, log=log))
+    return text, is_born_digital_text(text, path, log=log)
+
+
 def read_file_handle_unicode_errors(
     filepath: Path,
     log: logging.Logger | None = None,

src/paperless/tests/conftest.py

@@ -36,6 +36,23 @@ def samples_dir() -> Path:
     return (Path(__file__).parent / "samples").resolve()
 
 
+@pytest.fixture(scope="session")
+def tagged_no_text_pdf_file(samples_dir: Path) -> Path:
+    """Path to a tagged PDF whose only "text" is pdftotext layout padding.
+
+    Reproduces GH #13387: ``/MarkInfo /Marked true`` is set, but the only
+    extractable content is a form-feed byte, not real text. Lives here
+    rather than in parsers/conftest.py so both parser tests and
+    paperless/tests/test_parser_utils.py can use it.
+
+    Returns
+    -------
+    Path
+        Absolute path to ``tesseract/tagged-but-no-text.pdf``.
+    """
+    return samples_dir / "tesseract" / "tagged-but-no-text.pdf"
+
+
 @pytest.fixture(autouse=True)
 def clean_registry() -> Generator[None, None, None]:
     """Reset the parser registry before and after every test.

src/paperless/tests/parsers/test_remote_parser.py

@@ -336,117 +336,6 @@ class TestRemoteParserParse:
         assert remote_parser.get_date() is None
 
 
-# ---------------------------------------------------------------------------
-# parse() — produce_archive=False skips the remote engine (PDFs only)
-# ---------------------------------------------------------------------------
-
-
-class TestRemoteParserSkipsWhenNoArchiveWanted:
-    """When the caller has already decided no archive is needed for a PDF
-    (documents.consumer.should_produce_archive), the remote engine call is
-    skipped entirely in favor of locally-extracted text.
-    """
-
-    def test_pdf_skips_azure_when_no_archive_requested(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-    ) -> None:
-        """
-        GIVEN: produce_archive=False for a PDF
-        WHEN:  parse() is called
-        THEN:  Azure is never invoked, no archive is produced, and text
-               comes from local pdftotext extraction
-        """
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        azure_client.begin_analyze_document.assert_not_called()
-        assert remote_parser.get_archive_path() is None
-        assert remote_parser.get_text() != ""
-
-    def test_pdf_no_archive_requested_text_matches_local_extraction(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-        mocker: MockerFixture,
-    ) -> None:
-        """
-        GIVEN: produce_archive=False for a PDF
-        WHEN:  parse() is called
-        THEN:  the returned text is exactly the locally-extracted text,
-               not anything from the (unused) Azure mock
-        """
-        mocker.patch(
-            "paperless.parsers.remote.extract_pdf_text",
-            return_value="Local digital text.",
-        )
-
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        assert remote_parser.get_text() == "Local digital text."
-
-    def test_pdf_no_archive_requested_closes_no_client(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-    ) -> None:
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        azure_client.close.assert_not_called()
-
-    def test_non_pdf_still_calls_azure_when_no_archive_requested(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-    ) -> None:
-        """
-        Images have no local-text fallback, so produce_archive=False does
-        not skip the remote engine for non-PDF MIME types.
-        """
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "image/png",
-            produce_archive=False,
-        )
-
-        azure_client.begin_analyze_document.assert_called_once()
-        assert remote_parser.get_text() == _DEFAULT_TEXT
-
-    @pytest.mark.usefixtures("no_engine_settings")
-    def test_unconfigured_engine_takes_precedence_over_skip(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-    ) -> None:
-        """An unconfigured engine still short-circuits before the
-        produce_archive check, returning empty text as before.
-        """
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        assert remote_parser.get_text() == ""
-        assert remote_parser.get_archive_path() is None
-
-
 # ---------------------------------------------------------------------------
 # parse() — Azure failure path
 # ---------------------------------------------------------------------------

src/paperless/tests/parsers/test_tesseract_parser.py

@@ -21,7 +21,7 @@ from documents.parsers import run_convert
 from paperless.models import ModeChoices
 from paperless.parsers import ParserProtocol
 from paperless.parsers.tesseract import RasterisedDocumentParser
-from paperless.parsers.utils import post_process_text
+from paperless.parsers.utils import is_tagged_pdf
 
 if TYPE_CHECKING:
     from pathlib import Path
@@ -151,36 +151,6 @@ class TestRasterisedDocumentParserLifecycle:
         assert tempdir is not None and not tempdir.exists()
 
 
-# ---------------------------------------------------------------------------
-# post_process_text
-# ---------------------------------------------------------------------------
-
-
-class TestPostProcessText:
-    @pytest.mark.parametrize(
-        ("source", "expected"),
-        [
-            pytest.param(
-                "simple     string",
-                "simple string",
-                id="collapse-spaces",
-            ),
-            pytest.param(
-                "simple    newline\n   testing string",
-                "simple newline\ntesting string",
-                id="preserve-newline",
-            ),
-            pytest.param(
-                "utf-8   строка с пробелами в конце  ",  # noqa: RUF001
-                "utf-8 строка с пробелами в конце",  # noqa: RUF001
-                id="utf8-trailing-spaces",
-            ),
-        ],
-    )
-    def test_post_process_text(self, source: str, expected: str) -> None:
-        assert post_process_text(source) == expected
-
-
 # ---------------------------------------------------------------------------
 # Page count
 # ---------------------------------------------------------------------------
@@ -910,25 +880,25 @@ class TestSkipArchive:
         self,
         mocker: MockerFixture,
         tesseract_parser: RasterisedDocumentParser,
-        tesseract_samples_dir: Path,
+        tagged_no_text_pdf_file: Path,
     ) -> None:
         """
         GIVEN:
-            - A PDF that reports itself as tagged (/MarkInfo /Marked true) but
+            - A real PDF that reports itself as tagged (/MarkInfo /Marked
-              has no actual extractable text (some scanner firmware produces
+              true) but whose only pdftotext output is layout padding (a
-              this — see GitHub issue #13349)
+              lone form-feed byte), not real text (see GitHub issue #13387,
+              originally reported against #13349's tagged-PDF handling)
             - Mode: auto, produce_archive=False
         WHEN:
             - Document is parsed
         THEN:
             - The tag alone is not trusted as "has text"; OCRmyPDF still runs
         """
+        assert is_tagged_pdf(tagged_no_text_pdf_file) is True
         tesseract_parser.settings.mode = ModeChoices.AUTO
-        mocker.patch("paperless.parsers.tesseract.is_tagged_pdf", return_value=True)
-        mocker.patch.object(tesseract_parser, "extract_text", return_value=None)
         mock_ocr = mocker.patch("ocrmypdf.ocr")
         tesseract_parser.parse(
-            tesseract_samples_dir / "multi-page-images.pdf",
+            tagged_no_text_pdf_file,
             "application/pdf",
             produce_archive=False,
         )

src/paperless/tests/test_parser_utils.py

@@ -4,10 +4,18 @@ from __future__ import annotations
 
 import codecs
 from pathlib import Path
+from typing import TYPE_CHECKING
+
+import pytest
 
 from paperless.parsers.utils import is_tagged_pdf
+from paperless.parsers.utils import pdf_born_digital_text
+from paperless.parsers.utils import post_process_text
 from paperless.parsers.utils import read_file_handle_unicode_errors
 
+if TYPE_CHECKING:
+    from pytest_mock import MockerFixture
+
 SAMPLES = Path(__file__).parent / "samples" / "tesseract"
 
 
@@ -60,3 +68,105 @@ class TestIsTaggedPdf:
         bad = tmp_path / "bad.pdf"
         bad.write_bytes(b"not a pdf")
         assert is_tagged_pdf(bad) is False
+
+
+class TestPostProcessText:
+    @pytest.mark.parametrize(
+        ("source", "expected"),
+        [
+            pytest.param(
+                "simple     string",
+                "simple string",
+                id="collapse-spaces",
+            ),
+            pytest.param(
+                "simple    newline\n   testing string",
+                "simple newline\ntesting string",
+                id="preserve-newline",
+            ),
+            pytest.param(
+                "utf-8   строка с пробелами в конце  ",  # noqa: RUF001
+                "utf-8 строка с пробелами в конце",  # noqa: RUF001
+                id="utf8-trailing-spaces",
+            ),
+            pytest.param(None, None, id="none-input"),
+            pytest.param("", None, id="empty-string"),
+            pytest.param("   \n\x0c  \n ", None, id="whitespace-and-formfeed-only"),
+        ],
+    )
+    def test_post_process_text(
+        self,
+        source: str | None,
+        expected: str | None,
+    ) -> None:
+        assert post_process_text(source) == expected
+
+
+class TestPdfBornDigitalText:
+    """Regression coverage for GH #13387.
+
+    should_produce_archive() and RasterisedDocumentParser.parse() must agree
+    on whether a PDF has real text, so both go through this one function.
+    """
+
+    @pytest.mark.parametrize(
+        ("extracted", "tagged", "expected_text", "expected_born_digital"),
+        [
+            pytest.param("tiny", True, "tiny", True, id="tagged-with-real-text"),
+            pytest.param("tiny", False, "tiny", False, id="untagged-below-min-length"),
+            pytest.param(
+                "x" * 51,
+                False,
+                "x" * 51,
+                True,
+                id="untagged-above-min-length",
+            ),
+            pytest.param(None, True, None, False, id="tagged-but-no-text"),
+        ],
+    )
+    def test_born_digital_decision(
+        self,
+        mocker: MockerFixture,
+        tmp_path: Path,
+        extracted: str | None,
+        tagged: bool,  # noqa: FBT001
+        expected_text: str | None,
+        expected_born_digital: bool,  # noqa: FBT001
+    ) -> None:
+        """
+        GIVEN:
+            - A PDF whose pdftotext output and /MarkInfo tag status vary
+        WHEN:
+            - pdf_born_digital_text() is called
+        THEN:
+            - The normalized text and born-digital verdict match; the tag
+              alone never counts as "has text"
+        """
+        mocker.patch(
+            "paperless.parsers.utils.extract_pdf_text",
+            return_value=extracted,
+        )
+        mocker.patch("paperless.parsers.utils.is_tagged_pdf", return_value=tagged)
+        text, born_digital = pdf_born_digital_text(tmp_path / "doc.pdf")
+        assert text == expected_text
+        assert born_digital is expected_born_digital
+
+    def test_tagged_but_textless_pdf_is_not_born_digital(
+        self,
+        tagged_no_text_pdf_file: Path,
+    ) -> None:
+        """
+        GIVEN:
+            - A real PDF that is tagged (/MarkInfo /Marked true) but whose
+              only "text" is layout padding (a stray form-feed byte)
+        WHEN:
+            - pdf_born_digital_text() is called with no mocking
+        THEN:
+            - The normalized text is None and the PDF is not treated as
+              born-digital. The raw, unnormalized pdftotext output is
+              non-empty for this file, which is exactly what caused the
+              archive decision to disagree with the OCR decision in #13387.
+        """
+        text, born_digital = pdf_born_digital_text(tagged_no_text_pdf_file)
+        assert text is None
+        assert born_digital is False

src/paperless_ai/indexing.py

@@ -1,5 +1,6 @@
 import logging
 from collections.abc import Iterable
+from collections.abc import Iterator
 from contextlib import contextmanager
 from datetime import timedelta
 from typing import TYPE_CHECKING
@@ -22,6 +23,7 @@ from paperless_ai.embedding import get_configured_model_name
 from paperless_ai.embedding import get_embedding_model
 
 if TYPE_CHECKING:
+    from django.db.models import QuerySet
     from llama_index.core.schema import BaseNode
 
     from paperless_ai.vector_store import PaperlessSqliteVecVectorStore
@@ -32,6 +34,35 @@ logger = logging.getLogger("paperless_ai.indexing")
 RAG_NUM_OUTPUT = 512
 RAG_CHUNK_OVERLAP = 200
 
+# update_llm_index(): row count per .iterator() batch when streaming
+# documents for a rebuild/update, matching _DocumentViewerStream's chunk
+# size in documents/search/_backend.py.
+_INDEX_STREAM_CHUNK_SIZE = 1000
+
+
+class _StreamedDocuments:
+    """A thin QuerySet wrapper that streams via ``.iterator()`` instead of
+    materializing every row (plus its ``content`` and prefetch caches) into
+    memory at once, while still supporting ``len()`` so ``iter_wrapper``'s
+    progress bar shows a real total instead of falling back to indeterminate.
+    Same shape as ``documents/search/_backend.py``'s ``_DocumentViewerStream``,
+    just without that class's extra per-batch permission lookup -- nothing
+    here needs one.
+    """
+
+    def __init__(self, documents: "QuerySet[Document]") -> None:
+        self._documents = documents
+
+    def __len__(self) -> int:
+        return self._documents.count()
+
+    def __iter__(self) -> Iterator[Document]:
+        # iterator(chunk_size=...) streams from a server-side cursor instead
+        # of materializing the whole queryset in memory; since Django 4.1 it
+        # still honours prefetch_related, running the prefetches one batch
+        # at a time.
+        return iter(self._documents.iterator(chunk_size=_INDEX_STREAM_CHUNK_SIZE))
+
 
 def queue_llm_index_update_if_needed(*, rebuild: bool, reason: str) -> bool:
     # NOTE: The check-then-enqueue sequence below is non-atomic (TOCTOU): two
@@ -385,7 +416,7 @@ def update_llm_index(
         if rebuild or not store.table_exists():
             logger.info("Rebuilding LLM index.")
             store.drop_table()
-            for document in iter_wrapper(documents):
+            for document in iter_wrapper(_StreamedDocuments(documents)):
                 nodes = build_document_node(document, chunk_size=chunk_size)
                 _embed_nodes(nodes, embed_model)
                 store.add(nodes)
@@ -398,7 +429,7 @@ def update_llm_index(
             )
             existing = store.get_modified_times()
             changed = 0
-            for document in iter_wrapper(scoped_documents):
+            for document in iter_wrapper(_StreamedDocuments(scoped_documents)):
                 doc_id = str(document.id)
                 if existing.get(doc_id) == document.modified.isoformat():
                     continue

src/paperless_ai/tests/test_ai_indexing.py

@@ -186,6 +186,45 @@ def test_truncate_embedding_query_returns_single_chunk() -> None:
     assert "word199" not in result
 
 
+class TestStreamedDocuments:
+    """_StreamedDocuments streams via .iterator() instead of materializing
+    the whole queryset (plus its content and prefetch caches) in memory at
+    once, while still supporting len() so a progress bar wrapped around it
+    shows a real total.
+    """
+
+    def test_len_and_iter_delegate_to_streaming_queryset_methods(
+        self,
+        mocker: pytest_mock.MockerFixture,
+    ) -> None:
+        """
+        GIVEN:
+            - A mock queryset
+        WHEN:
+            - A _StreamedDocuments wrapping it is measured and iterated
+        THEN:
+            - len() uses count() (not a materializing len()), and iteration
+              uses .iterator(chunk_size=...) (not plain iteration, which
+              would materialize prefetches for the whole queryset at once)
+        """
+        mock_queryset = mocker.MagicMock()
+        mock_queryset.count.return_value = 42
+        mock_queryset.iterator.return_value = iter(["doc-1", "doc-2"])
+        streamed = indexing._StreamedDocuments(mock_queryset)
+
+        assert len(streamed) == 42
+        assert list(streamed) == ["doc-1", "doc-2"]
+        # count.call_count isn't asserted exactly: list()'s own size-hint
+        # optimization calls len(streamed) again internally, on top of the
+        # explicit len() call above -- both legitimately delegate to
+        # count(), so only the delegation itself (not the call count) is
+        # the thing being verified here.
+        mock_queryset.count.assert_called_with()
+        mock_queryset.iterator.assert_called_once_with(
+            chunk_size=indexing._INDEX_STREAM_CHUNK_SIZE,
+        )
+
+
 @pytest.mark.django_db
 def test_update_llm_index(
     temp_llm_index_dir: Path,