Compare commits

..
Author SHA1 Message Date
Trenton HandGitHub 6a02b87dde Feature: Updates remote OCR parser to respect the OCR mode setting (#13408)
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks

* Updates the documentation to be correct about the respecting now

* merge conflict fixing
2026-08-11 19:23:46 +00:00
Trenton HandGitHub 59a2651804 Fix: pass document chat queries as a QuerySet instead of a materialized list (#13638)
In tracemalloc based profiling, not materializing the whole Document list
reduced memory to approximately 20% of the baseline, with a peak memory
that scaled with the library size.  Now, the lazt queryset is used and only
the needed pk value is actually contributing to memory
2026-08-11 15:25:08 +00:00
GitHub Actions a99f63e059 Auto translate strings 2026-08-11 14:11:20 +00:00
shamoonandGitHub 939cb52f6e Fix: add pagination to saved views management page (#13646) 2026-08-11 07:08:30 -07:00
shamoonandGitHub 855669ddf9 Fix: fixes for workflow assign custom field values (#13630) 2026-08-10 07:38:07 -07:00
66 changed files with 507 additions and 1734 deletions
+1 -2
View File
@@ -301,8 +301,7 @@ The following methods are supported:
- `delete`
- No `parameters` required
- `reprocess`
- Optional `parameters`: `{ "remote_ocr": true }` to send the documents to the
remote OCR engine, see [Remote OCR](usage.md#remote-ocr). Defaults to false.
- No `parameters` required
- `set_permissions`
- Requires `parameters`:
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
+5 -16
View File
@@ -948,10 +948,11 @@ for display in the web interface.
!!! note
The **remote OCR parser** (Azure AI) always produces a searchable
PDF and stores it as the archive copy, regardless of this setting.
`ARCHIVE_FILE_GENERATION=never` has no effect when the remote
parser handles a document.
The **remote OCR parser** (Azure AI) also honors this setting: when
no archive is requested (`never`, or `auto` with a born-digital PDF),
the remote engine is skipped entirely and locally-extracted text is
used instead, avoiding an unnecessary API call and a duplicate text
layer.
#### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
@@ -2047,18 +2048,6 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to None.
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
: Which documents are sent to the remote OCR engine.
- `always`: every document of a supported file type is sent to the remote
engine, bypassing the local OCR engine.
- `workflow_only`: documents are processed locally unless a workflow
explicitly enables remote OCR for them, letting you use the remote engine
selectively.
Defaults to "always".
## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
-14
View File
@@ -456,20 +456,6 @@ def score(
return 10
```
**Remote services**
If your parser sends document content to a remote service, declare it:
```python
class MyCustomParser:
uses_remote_service = True
```
Paperless-ngx excludes such parsers when the document being consumed has not
been marked for remote processing, so users can keep remote OCR off by default
and enable it selectively with a workflow. Parsers that do not declare the
attribute are treated as fully local and are always considered.
**Archive and rendition flags**
```python
+5 -4
View File
@@ -187,10 +187,11 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
### Remote OCR parser
If you use the **remote OCR parser** (Azure AI), note that it always produces a
searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
has no effect for documents handled by the remote parser - the archive is produced
unconditionally by the remote engine.
If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
honored the same way as for the local engine: when no archive is requested
(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
entirely and locally-extracted text is used instead, avoiding an unnecessary
API call and a duplicate text layer.
## Search Index (Whoosh -> Tantivy)
+4 -23
View File
@@ -576,7 +576,9 @@ The following workflow action types are available:
- Tags, correspondent, document type and storage path
- Document owner
- View and / or edit permissions to users or groups
- Custom fields. Note that no value for the field will be set
- Custom fields, optionally with a value. If no value is set, the field is only added to the
document and any value it may already have is left untouched. If a value is set, it will
overwrite an existing value of that field on the document.
##### Removal {#workflow-action-removal}
@@ -648,19 +650,6 @@ happened while it was still encrypted, that original version will likewise be mi
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
multiple PDF types with different passwords, create separate workflows for each use case.
##### Remote OCR {#workflow-action-remote-ocr}
"Remote OCR" actions send the document to the configured remote OCR engine instead of processing it
locally. To use remote OCR selectively, set the [remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE)
to `workflow_only` then add this action to a workflow that matches only the documents you
want sent to the remote engine. See [Remote OCR](#remote-ocr) for the engine setup. The action only works with
a **Consumption Started** trigger.
The action takes no options, its presence is what enables remote OCR for a matching document.
If the remote engine is not configured, or does not support the document's file type, the document is
processed locally instead and a warning is written to the log.
#### Workflow placeholders
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -1097,19 +1086,11 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
settings can be supplied as environment variables or via **Application Configuration**.
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
or page limitations (e.g. with a free tier).
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
unless a [remote OCR workflow action](#workflow-action-remote-ocr) enables it for them, so you can limit the remote
engine to particular documents.
Setting the mode to `workflow_only` also allows the **Reprocess** actions to selectively use remote OCR for individual documents.
## Architecture
Paperless-ngx consists of the following components:
+8 -8
View File
@@ -599,7 +599,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
<context context-type="linenumber">84,85</context>
<context context-type="linenumber">85,86</context>
</context-group>
</trans-unit>
<trans-unit id="3768927257183755959" datatype="html">
@@ -670,7 +670,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
<context context-type="linenumber">85,86</context>
<context context-type="linenumber">86,87</context>
</context-group>
</trans-unit>
<trans-unit id="5079885666748292382" datatype="html">
@@ -9907,7 +9907,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.ts</context>
<context context-type="linenumber">296</context>
<context context-type="linenumber">314</context>
</context-group>
</trans-unit>
<trans-unit id="2620006875434695386" datatype="html">
@@ -10234,7 +10234,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.ts</context>
<context context-type="linenumber">290</context>
<context context-type="linenumber">308</context>
</context-group>
</trans-unit>
<trans-unit id="3501895737484542570" datatype="html">
@@ -10346,28 +10346,28 @@
<source>Saved view &quot;<x id="PH" equiv-text="savedView.name"/>&quot; deleted.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.ts</context>
<context context-type="linenumber">160</context>
<context context-type="linenumber">178</context>
</context-group>
</trans-unit>
<trans-unit id="1660419335376265526" datatype="html">
<source>Views saved successfully.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.ts</context>
<context context-type="linenumber">237</context>
<context context-type="linenumber">255</context>
</context-group>
</trans-unit>
<trans-unit id="1699877326523238632" datatype="html">
<source>Error while saving views.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.ts</context>
<context context-type="linenumber">242</context>
<context context-type="linenumber">260</context>
</context-group>
</trans-unit>
<trans-unit id="4919025779187821586" datatype="html">
<source>Note: Sharing saved views does not share the underlying documents.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.ts</context>
<context context-type="linenumber">278</context>
<context context-type="linenumber">296</context>
</context-group>
</trans-unit>
<trans-unit id="1229748338333965418" datatype="html">
@@ -14,12 +14,8 @@
<a ngbNavLink>{{category}}</a>
<ng-template ngbNavContent>
<div class="p-3">
@for (section of getCategorySections(category); track section) {
@if (section) {
<h5 class="mt-4 mb-3">{{section}}</h5>
}
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category, section); track option.key) {
@for (option of getCategoryOptions(category); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
@@ -55,7 +51,6 @@
</div>
}
</div>
}
</div>
</ng-template>
</li>
@@ -8,11 +8,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgSelectModule } from '@ng-select/ng-select'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of, throwError } from 'rxjs'
import {
ConfigCategory,
ConfigSection,
OutputTypeConfig,
} from 'src/app/data/paperless-config'
import { OutputTypeConfig } from 'src/app/data/paperless-config'
import { ConfigService } from 'src/app/services/config.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -162,24 +158,4 @@ describe('ConfigComponent', () => {
component.resetOption('barcodes_enabled')
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
})
it('should group options into sections within a category, or not', () => {
const sections = component.getCategorySections(ConfigCategory.OCR)
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
expect(
component
.getCategoryOptions(ConfigCategory.OCR)
.map((option) => option.key)
).toContain('output_type')
expect(
component
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
.map((option) => option.key)
).toEqual([
'remote_ocr_engine',
'remote_ocr_api_key',
'remote_ocr_endpoint',
'remote_ocr_mode',
])
})
})
@@ -74,20 +74,8 @@ export class ConfigComponent
return Object.values(ConfigCategory)
}
getCategorySections(category: string): string[] {
return [
...new Set(
PaperlessConfigOptions.filter((o) => o.category === category).map(
(o) => o.section ?? null // null means no section
)
),
]
}
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
return PaperlessConfigOptions.filter(
(o) => o.category === category && (o.section ?? null) === section
)
getCategoryOptions(category: string): ConfigOption[] {
return PaperlessConfigOptions.filter((o) => o.category === category)
}
initialConfig: PaperlessConfig
@@ -1,28 +0,0 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (messageBold) {
<p class="text-break"><b>{{messageBold}}</b></p>
}
@if (message) {
<p class="mb-0 text-break" [innerHTML]="message"></p>
}
@if (showRemoteOcr) {
<div class="form-check mt-3">
<input class="form-check-input" type="checkbox" id="reprocessRemoteOcr" [(ngModel)]="remoteOcr" />
<label class="form-check-label" for="reprocessRemoteOcr" i18n>Use remote OCR</label>
<div class="form-text" i18n>Sends the document to the configured remote OCR service, which may incur costs.</div>
</div>
}
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
{{btnCaption}}
</button>
</div>
@@ -1,72 +0,0 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { RemoteOCRModeConfig } from 'src/app/data/paperless-config'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { SettingsService } from 'src/app/services/settings.service'
import { ReprocessConfirmDialogComponent } from './reprocess-confirm-dialog.component'
describe('ReprocessConfirmDialogComponent', () => {
let component: ReprocessConfirmDialogComponent
let fixture: ComponentFixture<ReprocessConfirmDialogComponent>
let settingsService: SettingsService
const createComponent = (configured: boolean, mode: string) => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, configured)
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_MODE, mode)
fixture = TestBed.createComponent(ReprocessConfirmDialogComponent)
component = fixture.componentInstance
fixture.detectChanges()
}
beforeEach(async () => {
TestBed.configureTestingModule({
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
imports: [ReprocessConfirmDialogComponent],
}).compileComponents()
settingsService = TestBed.inject(SettingsService)
})
it('should not request remote OCR by default', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.remoteOcr).toBeFalsy()
})
it('should not offer remote OCR when no engine is configured', () => {
createComponent(false, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should not offer remote OCR when it already handles every document', () => {
createComponent(true, RemoteOCRModeConfig.ALWAYS)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should offer remote OCR when configured and selective', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeTruthy()
const checkbox = fixture.nativeElement.querySelector('#reprocessRemoteOcr')
expect(checkbox).not.toBeNull()
checkbox.click()
fixture.detectChanges()
expect(component.remoteOcr).toBeTruthy()
})
})
@@ -1,20 +0,0 @@
import { Component, inject } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { SettingsService } from 'src/app/services/settings.service'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-reprocess-confirm-dialog',
templateUrl: './reprocess-confirm-dialog.component.html',
imports: [FormsModule],
})
export class ReprocessConfirmDialogComponent extends ConfirmDialogComponent {
private settings = inject(SettingsService)
remoteOcr: boolean = false
public get showRemoteOcr(): boolean {
// Hidden when it is not configured, or when it already handles every document anyway.
return this.settings.remoteOCRIsSelectable
}
}
@@ -455,13 +455,6 @@
</div>
</div>
}
@case (WorkflowActionType.RemoteOcr) {
<div class="row">
<div class="col">
<p class="text-muted small" i18n>The document will be sent to the configured remote OCR service. May incur costs.</p>
</div>
</div>
}
}
</div>
</ng-template>
@@ -29,7 +29,6 @@ import {
DocumentSource,
WorkflowTriggerType,
} from 'src/app/data/workflow-trigger'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
@@ -225,12 +224,7 @@ describe('WorkflowEditDialogComponent', () => {
).toEqual('Document Added')
expect(component.getTriggerTypeOptionName(null)).toEqual('')
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
// Remote OCR is absent until the workflow has a consumption trigger
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.RemoteOcr
)
)
expect(component.actionTypeOptions).toEqual(WORKFLOW_ACTION_OPTIONS)
expect(
component.getActionTypeOptionName(WorkflowActionType.Assignment)
).toEqual('Assignment')
@@ -243,104 +237,7 @@ describe('WorkflowEditDialogComponent', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(false)
component.ngOnInit()
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) =>
a.id !== WorkflowActionType.Email &&
a.id !== WorkflowActionType.RemoteOcr
)
)
})
it('should offer remote OCR only for consumption workflows', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// A consumption trigger makes the action reachable
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Any other trigger type runs after the document has been parsed
component.object = {
name: 'Workflow 2',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should offer remote OCR on a trigger added to a new workflow', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
component.ngOnInit()
// Nothing for the action to apply to yet
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
// addTrigger creates the form field with emitEvent false, so the options
// have to be computed on read rather than cached from valueChanges
component.addTrigger()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Switching that trigger to a type that runs after parsing removes it
component.triggerFields
.at(0)
.get('type')
.setValue(WorkflowTriggerType.DocumentAdded)
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should keep remote OCR listed when an action already uses it', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [{ type: WorkflowActionType.RemoteOcr }],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
})
it('should not offer remote OCR when no engine is configured', () => {
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
WORKFLOW_ACTION_OPTIONS.filter((a) => a.id !== WorkflowActionType.Email)
)
})
@@ -148,10 +148,6 @@ export const WORKFLOW_ACTION_OPTIONS = [
id: WorkflowActionType.MoveToTrash,
name: $localize`Move to trash`,
},
{
id: WorkflowActionType.RemoteOcr,
name: $localize`Remote OCR`,
},
]
export enum TriggerFilterType {
@@ -508,6 +504,8 @@ export class WorkflowEditDialogComponent
expandedItem: number = null
readonly allowedActionTypes = signal([])
private readonly triggerFilterOptionsMap = new WeakMap<
FormArray,
TriggerFilterOption[]
@@ -550,40 +548,13 @@ export class WorkflowEditDialogComponent
this.checkRemovalActionFields.bind(this)
)
this.checkRemovalActionFields(this.objectForm.value)
}
private allowedActionTypes: typeof WORKFLOW_ACTION_OPTIONS = null
private getAllowedActionTypes() {
let allowed = WORKFLOW_ACTION_OPTIONS
if (!this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.Email)
}
// Remote OCR is decided before the document is parsed, so it is only
// offered for workflows that run at consumption.
const formWorkflow: Workflow = this.objectForm?.value
const remoteOcrUsable =
this.settingsService.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
(formWorkflow?.triggers?.some(
(trigger) => trigger.type === WorkflowTriggerType.Consumption
) ||
formWorkflow?.actions?.some(
(action) => action.type === WorkflowActionType.RemoteOcr
))
if (!remoteOcrUsable) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.RemoteOcr)
}
if (
this.allowedActionTypes?.length === allowed.length &&
this.allowedActionTypes.every((a, i) => a.id === allowed[i].id)
) {
return this.allowedActionTypes
}
this.allowedActionTypes = allowed
return allowed
this.allowedActionTypes.set(
this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)
? WORKFLOW_ACTION_OPTIONS
: WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.Email
)
)
}
private checkRemovalActionFields(formWorkflow: Workflow) {
@@ -1308,8 +1279,7 @@ export class WorkflowEditDialogComponent
get actionTypeOptions() {
this.settingsService.trackChanges()
// Computed on read rather than cached
return this.getAllowedActionTypes()
return this.allowedActionTypes()
}
getActionTypeOptionName(type: WorkflowActionType): string {
@@ -52,10 +52,10 @@ describe('CustomFieldsValuesComponent', () => {
})
it('should set selectedFields and map values correctly', () => {
component.value = { 1: 'value1' }
component.selectedFields = [1, 2]
expect(component.selectedFields).toEqual([1, 2])
expect(component.value).toEqual({ 1: 'value1', 2: null })
component.value = { 1: 'value1', 3: 0, 4: false }
component.selectedFields = [1, 2, 3, 4]
expect(component.selectedFields).toEqual([1, 2, 3, 4])
expect(component.value).toEqual({ 1: 'value1', 2: null, 3: 0, 4: false })
})
it('should return the correct custom field by id', () => {
@@ -77,7 +77,7 @@ export class CustomFieldsValuesComponent extends AbstractInputComponent<Object>
this._selectedFields = newFields
// map the selected fields to an object with field_id as key and value as value
this.value = newFields.reduce((acc, fieldId) => {
acc[fieldId] = this.value?.[fieldId] || null
acc[fieldId] = this.value?.[fieldId] ?? null
return acc
}, {})
this.onChange(this.value)
@@ -963,24 +963,12 @@ describe('DocumentDetailComponent', () => {
component.reprocess()
const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, false)
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] })
expect(modalSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should pass remote OCR choice when reprocessing', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
reprocessSpy.mockReturnValue(of(true))
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
component.reprocess()
openModal.componentInstance.remoteOcr = true
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, true)
})
it('should show error if redo ocr call fails', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
@@ -97,7 +97,6 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
import * as UTIF from 'utif'
import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
@@ -1403,7 +1402,7 @@ export class DocumentDetailComponent
}
reprocess() {
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Reprocess confirm`
@@ -1414,10 +1413,7 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
this.documentsService
.reprocessDocuments(
{ documents: [this.document().id] },
modal.componentInstance.remoteOcr
)
.reprocessDocuments({ documents: [this.document().id] })
.subscribe({
next: () => {
this.toastService.showInfo(
@@ -1122,7 +1122,6 @@ describe('BulkEditorComponent', () => {
req.flush(true)
expect(req.request.body).toEqual({
documents: [3, 4],
remote_ocr: false,
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
@@ -51,7 +51,6 @@ import { ToastService } from 'src/app/services/toast.service'
import { flattenTags } from 'src/app/utils/flatten-tags'
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
@@ -910,7 +909,7 @@ export class BulkEditorComponent
}
reprocessSelected() {
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Reprocess confirm`
@@ -924,10 +923,7 @@ export class BulkEditorComponent
modal.componentInstance.buttonsEnabled = false
this.executeDocumentAction(
modal,
this.documentService.reprocessDocuments(
this.getSelectionQuery(),
modal.componentInstance.remoteOcr
)
this.documentService.reprocessDocuments(this.getSelectionQuery())
)
})
}
@@ -7,7 +7,7 @@
</pngx-page-header>
<form [formGroup]="savedViewsForm" (ngSubmit)="save()">
<ul class="list-group mb-3" formGroupName="savedViews">
@for (view of savedViews(); track view) {
@for (view of pagedSavedViews(); track view) {
<li class="list-group-item py-3">
<div [formGroupName]="view.id">
<div class="row">
@@ -81,6 +81,11 @@
}
</ul>
<div class="d-flex align-items-center mb-3">
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
@if (savedViews()?.length > pageSize) {
<ngb-pagination class="ms-auto" [pageSize]="pageSize" [collectionSize]="savedViews().length" [page]="page()" [maxSize]="5" (pageChange)="page.set($event)" size="sm" aria-label="Pagination"></ngb-pagination>
}
</div>
</form>
@@ -4,6 +4,7 @@ import { provideHttpClientTesting } from '@angular/common/http/testing'
import { signal } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { NgbModal, NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { Subject, of, throwError } from 'rxjs'
@@ -222,6 +223,44 @@ describe('SavedViewsComponent', () => {
).toEqual(view.show_on_dashboard)
})
it('should page saved views, clamp the page if views are removed', () => {
const manyViews = Array.from({ length: 30 }, (_, i) => ({
id: i + 1,
name: `view${i + 1}`,
})) as SavedView[]
const listSpy = jest.spyOn(savedViewService, 'list').mockReturnValue(
of({
all: manyViews.map((v) => v.id),
count: manyViews.length,
results: manyViews.concat([]),
})
)
component.ngOnInit()
fixture.detectChanges()
expect(listSpy).toHaveBeenCalledWith(1, 100000, null, false, {
full_perms: true,
})
expect(component.pagedSavedViews()).toHaveLength(25)
expect(fixture.debugElement.query(By.css('ngb-pagination'))).not.toBeNull()
// all views have controls, not just the current page
expect(
Object.keys(component.savedViewsForm.get('savedViews').value)
).toHaveLength(30)
component.page.set(2)
expect(component.pagedSavedViews()).toHaveLength(5)
listSpy.mockReturnValue(
of({
all: manyViews.slice(0, 25).map((v) => v.id),
count: 25,
results: manyViews.slice(0, 25),
})
)
component.ngOnInit()
expect(component.page()).toEqual(1)
})
it('should support editing permissions', () => {
const confirmClicked = new Subject<any>()
const modalRef = {
@@ -1,12 +1,19 @@
import { AsyncPipe } from '@angular/common'
import { Component, OnDestroy, OnInit, inject, signal } from '@angular/core'
import {
Component,
OnDestroy,
OnInit,
computed,
inject,
signal,
} from '@angular/core'
import {
FormControl,
FormGroup,
FormsModule,
ReactiveFormsModule,
} from '@angular/forms'
import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { NgbModal, NgbPaginationModule } from '@ng-bootstrap/ng-bootstrap'
import { dirtyCheck } from '@ngneat/dirty-check-forms'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { BehaviorSubject, Observable, of, switchMap, takeUntil } from 'rxjs'
@@ -42,6 +49,7 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
FormsModule,
ReactiveFormsModule,
AsyncPipe,
NgbPaginationModule,
NgxBootstrapIconsModule,
],
})
@@ -58,6 +66,14 @@ export class SavedViewsComponent
DisplayMode = DisplayMode
readonly savedViews = signal<SavedView[]>(undefined)
readonly page = signal(1)
public readonly pageSize = 25
// All views are loaded at init, so paging is only for display
readonly pagedSavedViews = computed(() => {
const start = (this.page() - 1) * this.pageSize
return this.savedViews()?.slice(start, start + this.pageSize)
})
private savedViewsGroup = new FormGroup({})
public savedViewsForm: FormGroup = new FormGroup({
savedViews: this.savedViewsGroup,
@@ -84,9 +100,11 @@ export class SavedViewsComponent
private reloadViews(): void {
this.loading.set(true)
this.savedViewService
.list(null, null, null, false, { full_perms: true })
.list(1, 100000, null, false, { full_perms: true })
.subscribe((r) => {
this.savedViews.set(r.results)
const pageCount = Math.ceil(r.results.length / this.pageSize)
this.page.update((page) => Math.min(page, Math.max(1, pageCount)))
this.initialize()
})
}
-55
View File
@@ -54,10 +54,6 @@ export const ConfigCategory = {
AI: $localize`AI Settings`,
}
export const ConfigSection = {
RemoteOCR: $localize`Remote OCR`,
}
export const LLMEmbeddingBackendConfig = {
OPENAI_LIKE: 'openai-like',
HUGGINGFACE: 'huggingface',
@@ -69,15 +65,6 @@ export const LLMBackendConfig = {
OLLAMA: 'ollama',
}
export const RemoteOCREngineConfig = {
AZURE_AI: 'azureai',
}
export const RemoteOCRModeConfig = {
ALWAYS: 'always',
WORKFLOW_ONLY: 'workflow_only',
}
export interface ConfigOption {
key: string
title: string
@@ -85,7 +72,6 @@ export interface ConfigOption {
choices?: Array<{ id: string; name: string }>
config_key?: string
category: string
section?: string
note?: string
}
@@ -195,43 +181,6 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_OCR_USER_ARGS',
category: ConfigCategory.OCR,
},
{
key: 'remote_ocr_engine',
title: $localize`Remote OCR Engine`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCREngineConfig),
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
},
{
key: 'remote_ocr_api_key',
title: $localize`Remote OCR API Key`,
type: ConfigOptionType.Password,
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
},
{
key: 'remote_ocr_endpoint',
title: $localize`Remote OCR Endpoint`,
type: ConfigOptionType.String,
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Required when using the Azure AI engine.`,
},
{
key: 'remote_ocr_mode',
title: $localize`Remote OCR Mode`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCRModeConfig),
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
},
{
key: 'app_logo',
title: $localize`Application Logo`,
@@ -449,10 +398,6 @@ export interface PaperlessConfig extends ObjectWithId {
barcode_enable_tag: boolean
barcode_tag_mapping: object
barcode_tag_split: boolean
remote_ocr_engine: string
remote_ocr_api_key: string
remote_ocr_endpoint: string
remote_ocr_mode: string
ai_enabled: boolean
llm_embedding_backend: string
llm_embedding_model: string
-13
View File
@@ -1,6 +1,5 @@
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
import { RemoteOCRModeConfig } from './paperless-config'
import { User } from './user'
export interface UiSettings {
@@ -95,8 +94,6 @@ export const SETTINGS_KEYS = {
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
EMAIL_ENABLED: 'email_enabled',
AI_ENABLED: 'ai_enabled',
REMOTE_OCR_CONFIGURED: 'remote_ocr:configured',
REMOTE_OCR_MODE: 'remote_ocr:mode',
}
export const SETTINGS: UiSetting[] = [
@@ -350,14 +347,4 @@ export const SETTINGS: UiSetting[] = [
type: 'string',
default: PdfEditorEditMode.Create,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_CONFIGURED,
type: 'boolean',
default: false,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_MODE,
type: 'string',
default: RemoteOCRModeConfig.ALWAYS,
},
]
-1
View File
@@ -7,7 +7,6 @@ export enum WorkflowActionType {
Webhook = 4,
PasswordRemoval = 5,
MoveToTrash = 6,
RemoteOcr = 7,
}
export interface WorkflowActionEmail extends ObjectWithId {
@@ -284,21 +284,6 @@ describe(`DocumentService`, () => {
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: false,
})
})
it('should request remote OCR when reprocessing with it enabled', () => {
const ids = [1, 2, 3]
subscription = service
.reprocessDocuments({ documents: ids }, true)
.subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/reprocess/`
)
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: true,
})
})
@@ -349,13 +349,9 @@ export class DocumentService extends AbstractPaperlessService<Document> {
})
}
reprocessDocuments(
selection: DocumentSelectionQuery,
remoteOcr: boolean = false
) {
reprocessDocuments(selection: DocumentSelectionQuery) {
return this.http.post(this.getResourceUrl(null, 'reprocess'), {
...selection,
remote_ocr: remoteOcr,
})
}
@@ -13,7 +13,6 @@ import { environment } from 'src/environments/environment'
import { CustomFieldDataType } from '../data/custom-field'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { SavedView } from '../data/saved-view'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
import { PermissionsService } from './permissions.service'
import { CustomFieldsService } from './rest/custom-fields.service'
@@ -435,26 +434,4 @@ describe('SettingsService', () => {
).name
).toEqual(customFields[0].name)
})
it('should offer remote OCR only when configured and selective', () => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, false)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
// configured, but already handling every document
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, true)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.ALWAYS
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeTruthy()
})
})
@@ -19,7 +19,6 @@ import {
} from 'src/app/utils/color'
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SavedView } from '../data/saved-view'
import {
PAPERLESS_GREEN_HEX,
@@ -688,17 +687,6 @@ export class SettingsService {
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
}
/**
* Offering remote OCR as a choice only makes sense when an engine
* is configured but is not already handling every document.
*/
get remoteOCRIsSelectable(): boolean {
return (
this.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
this.get(SETTINGS_KEYS.REMOTE_OCR_MODE) !== RemoteOCRModeConfig.ALWAYS
)
}
offerTour(): boolean {
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
}
+2 -8
View File
@@ -394,16 +394,10 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
return "OK"
def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
"""
Re-run parsing for the given documents.
Consumption workflows do not run here, so ``remote_ocr`` is how the user
asks for the remote engine when it is not configured to handle everything.
"""
def reprocess(doc_ids: list[int]) -> Literal["OK"]:
for document_id in doc_ids:
update_document_content_maybe_archive_file.apply_async(
kwargs={"document_id": document_id, "remote_ocr": remote_ocr},
kwargs={"document_id": document_id},
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
)
-18
View File
@@ -53,7 +53,6 @@ from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess
from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ArchiveFileGenerationChoices
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
@@ -452,19 +451,12 @@ class ConsumerPlugin(
except Exception as e:
self.log.error(f"Error attempting to clean PDF: {e}")
# Workflows have already run at this point, so the metadata knows
# whether this document was singled out for remote OCR
allow_remote = (
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
)
# Based on the mime type, get the parser for that type
parser_class: type[ParserProtocol] | None = (
get_parser_registry().get_parser_for_file(
mime_type,
self.filename,
self.working_copy,
allow_remote=allow_remote,
)
)
if not parser_class:
@@ -473,16 +465,6 @@ class ConsumerPlugin(
f"Unsupported mime type {mime_type}",
)
if self.metadata.remote_ocr and not getattr(
parser_class,
"uses_remote_service",
False,
):
self.log.warning(
"Remote OCR was requested for this document but no remote "
"parser is available for it, processing locally instead.",
)
# Notify all listeners that we're going to do some work.
document_consumption_started.send(
-3
View File
@@ -34,7 +34,6 @@ class DocumentMetadataOverrides:
skip_asn_if_exists: bool = False
version_label: str | None = None
actor_id: int | None = None
remote_ocr: bool = False
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
"""
@@ -58,8 +57,6 @@ class DocumentMetadataOverrides:
self.actor_id = other.actor_id
if other.skip_asn_if_exists:
self.skip_asn_if_exists = True
if other.remote_ocr:
self.remote_ocr = True
if other.version_label is not None:
self.version_label = other.version_label
@@ -1,30 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 17:27
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0022_add_perf_indexes"),
]
operations = [
migrations.AlterField(
model_name="workflowaction",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Assignment"),
(2, "Removal"),
(3, "Email"),
(4, "Webhook"),
(5, "Password removal"),
(6, "Move to trash"),
(7, "Remote OCR"),
],
default=1,
verbose_name="Workflow Action Type",
),
),
]
-4
View File
@@ -1599,10 +1599,6 @@ class WorkflowAction(models.Model):
6,
_("Move to trash"),
)
REMOTE_OCR = (
7,
_("Remote OCR"),
)
type = models.PositiveSmallIntegerField(
_("Workflow Action Type"),
+8 -30
View File
@@ -1744,7 +1744,7 @@ class DeleteDocumentsSerializer(DocumentSelectionSerializer):
class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
remote_ocr = serializers.BooleanField(required=False, default=False)
pass
class BulkEditSerializer(
@@ -2086,13 +2086,6 @@ class BulkEditSerializer(
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
)
def _validate_parameters_reprocess(self, parameters) -> None:
if "remote_ocr" in parameters:
if not isinstance(parameters["remote_ocr"], bool):
raise serializers.ValidationError("remote_ocr must be a boolean")
else:
parameters["remote_ocr"] = False
def validate_parameters_remove_password(self, parameters):
if "password" not in parameters:
raise serializers.ValidationError("password not specified")
@@ -2157,8 +2150,6 @@ class BulkEditSerializer(
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
elif method == bulk_edit.remove_password:
self.validate_parameters_remove_password(parameters)
elif method == bulk_edit.reprocess:
self._validate_parameters_reprocess(parameters)
return attrs
@@ -3222,6 +3213,13 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
{"assign_title": f'Invalid f-string detected: "{e.args[0]}"'},
)
if attrs.get("assign_custom_fields_values"):
# Empty strings treated as None to avoid unexpected behavior
attrs["assign_custom_fields_values"] = {
field_id: (None if value == "" else value)
for field_id, value in attrs["assign_custom_fields_values"].items()
}
if (
"type" in attrs
and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL
@@ -3277,26 +3275,6 @@ class WorkflowSerializer(serializers.ModelSerializer[Workflow]):
"actions",
]
def validate(self, attrs):
attrs = super().validate(attrs)
triggers = attrs.get("triggers") or []
actions = attrs.get("actions") or []
# Remote OCR can only work with consumption triggers
if any(
action.get("type") == WorkflowAction.WorkflowActionType.REMOTE_OCR
for action in actions
) and not any(
trigger.get("type") == WorkflowTrigger.WorkflowTriggerType.CONSUMPTION
for trigger in triggers
):
raise serializers.ValidationError(
"Remote OCR actions require a consumption started trigger",
)
return attrs
def update_triggers_and_actions(
self,
instance: Workflow,
-11
View File
@@ -971,17 +971,6 @@ def run_workflows(
)
elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH:
has_move_to_trash_action = True
elif action.type == WorkflowAction.WorkflowActionType.REMOTE_OCR:
if use_overrides and overrides:
overrides.remote_ocr = True
else:
# If a workflow has a consumption trigger *and* another type,
# the document has already been parsed by the time the other one fires
logger.debug(
"Remote OCR action only applies to consumption "
"triggers, ignoring",
extra={"group": logging_group},
)
if not use_overrides:
# limit title to 128 characters
+1 -10
View File
@@ -66,7 +66,6 @@ from documents.utils import compute_checksum
from documents.utils import identity
from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig
from paperless.config import RemoteOCRConfig
from paperless.logging import consume_task_id
from paperless.parsers import ParserContext
from paperless.parsers.registry import get_parser_registry
@@ -338,17 +337,10 @@ def bulk_update_documents(document_ids) -> None:
@shared_task
def update_document_content_maybe_archive_file(
document_id,
*,
remote_ocr: bool = False,
) -> None:
def update_document_content_maybe_archive_file(document_id) -> None:
"""
Re-creates OCR content and thumbnail for a document, and archive file if
it exists.
Remote OCR is used only when the engine is configured to handle everything
or if explicitly asked for via ``remote_ocr``.
"""
document = Document.objects.get(id=document_id)
@@ -358,7 +350,6 @@ def update_document_content_maybe_archive_file(
mime_type,
document.original_filename or "",
document.source_path,
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
)
if not parser_class:
@@ -72,10 +72,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"barcode_enable_tag": None,
"barcode_tag_mapping": None,
"barcode_tag_split": None,
"remote_ocr_engine": None,
"remote_ocr_api_key": None,
"remote_ocr_endpoint": None,
"remote_ocr_mode": None,
"ai_enabled": False,
"llm_embedding_backend": None,
"llm_embedding_model": None,
@@ -874,49 +870,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
config.refresh_from_db()
self.assertEqual(config.llm_api_key, None)
def test_update_remote_ocr_api_key(self) -> None:
"""
GIVEN:
- Existing config with remote_ocr_api_key specified
WHEN:
- API to update remote_ocr_api_key is called with all *s
- API to update remote_ocr_api_key is called with empty string
THEN:
- remote_ocr_api_key is unchanged
- remote_ocr_api_key is set to None
"""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_api_key = "1234567890"
config.save()
# Test with all *
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "*" * 32,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, "1234567890")
# Test with empty string
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, None)
def test_enable_ai_index_triggers_update(self) -> None:
"""
GIVEN:
+1 -46
View File
@@ -532,29 +532,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": False})
@mock.patch("documents.views.bulk_edit.reprocess")
def test_reprocess_documents_endpoint_remote_ocr(self, m) -> None:
"""
GIVEN:
- API data to reprocess a document with remote OCR requested
WHEN:
- API is called
THEN:
- reprocess is called with remote_ocr=True
"""
self.setup_mock(m, "reprocess")
response = self.client.post(
"/api/documents/reprocess/",
json.dumps({"documents": [self.doc1.id], "remote_ocr": True}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
self.assertEqual(len(kwargs), 0)
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None:
@@ -1575,29 +1553,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
),
)
def test_legacy_bulk_edit_reprocess_invalid_remote_ocr(self) -> None:
"""
GIVEN:
- The deprecated bulk_edit endpoint with a non-boolean remote_ocr
WHEN:
- API is called
THEN:
- The request is rejected rather than passed through to the task
"""
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc1.id],
"method": "reprocess",
"parameters": {"remote_ocr": "yes please"},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
@mock.patch("documents.views.bulk_edit.edit_pdf")
def test_edit_pdf(self, m) -> None:
self.setup_mock(m, "edit_pdf")
@@ -60,10 +60,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
},
"email_enabled": False,
"ai_enabled": False,
"remote_ocr": {
"configured": False,
"mode": "always",
},
},
)
@@ -158,50 +154,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
str(response.data["settings"]),
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="somekey",
REMOTE_OCR_ENDPOINT="https://example.cognitiveservices.azure.com",
REMOTE_OCR_MODE="workflow_only",
)
def test_settings_reports_remote_ocr_when_configured(self) -> None:
"""
GIVEN:
- A fully configured remote OCR engine in workflow_only mode
WHEN:
- The ui_settings endpoint is called
THEN:
- The UI is told remote OCR is available and selective, so it can
offer it where it would actually change something
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
response.data["settings"]["remote_ocr"],
{"configured": True, "mode": "workflow_only"},
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
def test_settings_reports_remote_ocr_incompletely_configured(self) -> None:
"""
GIVEN:
- An engine named but missing its endpoint and API key
WHEN:
- The ui_settings endpoint is called
THEN:
- It is reported as not configured, matching what the parser
registry will actually do
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertFalse(response.data["settings"]["remote_ocr"]["configured"])
@override_settings(
OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
GMAIL_OAUTH_CLIENT_ID="abc123",
+9 -76
View File
@@ -390,82 +390,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(Workflow.objects.count(), 1)
def test_api_create_remote_ocr_action_requires_consumption_trigger(
self,
) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- No consumption started trigger, so the action could never run
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
existing_count = Workflow.objects.count()
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR too late",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Workflow.objects.count(), existing_count)
def test_api_create_remote_ocr_action_with_consumption_trigger(self) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- A consumption started trigger alongside another trigger type
WHEN:
- API is called
THEN:
- The workflow is created, the action applies to consumption only
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR on consume",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
"filter_filename": "*.pdf",
},
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
def test_api_create_workflow_trigger_action_empty_fields(self) -> None:
"""
GIVEN:
@@ -498,6 +422,11 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
json.dumps(
{
"assign_title": "",
"assign_custom_fields": [self.cf1.id, self.cf2.id],
"assign_custom_fields_values": {
str(self.cf1.id): "",
str(self.cf2.id): 0,
},
},
),
content_type="application/json",
@@ -505,6 +434,10 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
action = WorkflowAction.objects.get(id=response.data["id"])
self.assertIsNone(action.assign_title)
self.assertEqual(
action.assign_custom_fields_values,
{str(self.cf1.id): None, str(self.cf2.id): 0},
)
response = self.client.post(
self.ENDPOINT_TRIGGERS,
-53
View File
@@ -1782,56 +1782,3 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIn("wrong password", str(exc.exception))
self.assertIn("Error removing password from document", cm.output[0])
class TestBulkEditReprocess(DirectoriesMixin, TestCase):
def setUp(self) -> None:
super().setUp()
self.doc = Document.objects.create(
title="test",
checksum="A",
mime_type="application/pdf",
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_defaults_to_local(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that says nothing about remote OCR
WHEN:
- reprocess is called
THEN:
- The task is queued without asking for the remote engine
"""
result = bulk_edit.reprocess([self.doc.id])
self.assertEqual(result, "OK")
mock_task.apply_async.assert_called_once()
_, kwargs = mock_task.apply_async.call_args
self.assertEqual(
kwargs["kwargs"],
{"document_id": self.doc.id, "remote_ocr": False},
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_passes_remote_ocr(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that explicitly asks for remote OCR
WHEN:
- reprocess is called
THEN:
- The request is forwarded to the task for every document
"""
other = Document.objects.create(
title="test2",
checksum="B",
mime_type="application/pdf",
)
bulk_edit.reprocess([self.doc.id, other.id], remote_ocr=True)
self.assertEqual(mock_task.apply_async.call_count, 2)
for call in mock_task.apply_async.call_args_list:
self.assertTrue(call.kwargs["kwargs"]["remote_ocr"])
-80
View File
@@ -1559,72 +1559,6 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
consumer.run_post_consume_script(doc)
class TestConsumerRemoteOCR(
DirectoriesMixin,
FileSystemAssertsMixin,
GetConsumerMixin,
TestCase,
):
"""
The consumer resolves the remote OCR mode and the per-document request from
workflows into the allow_remote flag it hands to the parser registry.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.consumer.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
self.addCleanup(patcher.stop)
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
src = (
Path(__file__).parent
/ "samples"
/ "documents"
/ "originals"
/ "0000001.pdf"
)
dst = self.dirs.scratch_dir / "sample.pdf"
shutil.copy(src, dst)
with self.get_consumer(dst, overrides=overrides) as consumer:
consumer.run()
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
"""
GIVEN: Remote OCR mode is 'always'.
WHEN: A document is consumed without any workflow asking for it.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A document is consumed and nothing asked for remote OCR.
THEN: The remote parser is excluded.
"""
self.assertFalse(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A workflow set remote_ocr on the metadata overrides.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
)
class TestMetadataOverrides(TestCase):
def test_update_skip_asn_if_exists(self) -> None:
base = DocumentMetadataOverrides()
@@ -1632,20 +1566,6 @@ class TestMetadataOverrides(TestCase):
base.update(incoming)
self.assertTrue(base.skip_asn_if_exists)
def test_update_remote_ocr(self) -> None:
base = DocumentMetadataOverrides()
base.update(DocumentMetadataOverrides(remote_ocr=True))
self.assertTrue(base.remote_ocr)
def test_update_remote_ocr_is_not_unset(self) -> None:
"""
A later workflow that says nothing must not undo an earlier one that
asked for remote OCR.
"""
base = DocumentMetadataOverrides(remote_ocr=True)
base.update(DocumentMetadataOverrides())
self.assertTrue(base.remote_ocr)
def test_update_actor_and_version_label(self) -> None:
base = DocumentMetadataOverrides(
actor_id=1,
-39
View File
@@ -287,45 +287,6 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
"""
Consumption workflows do not run on reprocess, so the remote parser is
used only in 'always' mode or when the caller explicitly asks for it.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.tasks.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = None
self.addCleanup(patcher.stop)
self.doc = Document.objects.create(
title="test",
content="my document",
checksum="wow",
mime_type="application/pdf",
)
def _allow_remote(self, **kwargs) -> bool:
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return call_kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
self.assertTrue(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
self.assertFalse(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
self.assertTrue(self._allow_remote(remote_ocr=True))
class TestAIIndex(DirectoriesMixin, TestCase):
@override_settings(
AI_ENABLED=True,
+49 -79
View File
@@ -2000,6 +2000,55 @@ class TestWorkflows(
r"Doc added in \w{3,}",
) # Match any 3-letter month name
def test_document_updated_workflow_existing_custom_field_empty_value(self) -> None:
"""
GIVEN:
- Existing workflow with UPDATED trigger and action that assigns a custom field
with an empty value
WHEN:
- Document is updated that already contains the field with a value
THEN:
- The existing value is left untouched, see GH #13627
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
filter_has_document_type=self.dt,
)
action = WorkflowAction.objects.create()
action.assign_custom_fields.add(self.cf1)
action.assign_custom_fields_values = {self.cf1.pk: ""}
action.save()
w = Workflow.objects.create(
name="Workflow 1",
order=0,
)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
correspondent=self.c,
original_filename="sample.pdf",
)
CustomFieldInstance.objects.create(
document=doc,
field=self.cf1,
value_text="existing value",
)
superuser = User.objects.create_superuser("superuser")
self.client.force_authenticate(user=superuser)
self.client.patch(
f"/api/documents/{doc.id}/",
{"document_type": self.dt.id},
format="json",
)
doc.refresh_from_db()
self.assertEqual(doc.custom_fields.get(field=self.cf1).value, "existing value")
def test_document_updated_workflow_existing_custom_field(self) -> None:
"""
GIVEN:
@@ -5360,82 +5409,3 @@ class TestDateWorkflowLocalization(
document = Document.objects.first()
assert document is not None
assert document.title == expected_title
class TestRemoteOCRWorkflowAction(DirectoriesMixin, SampleDirMixin, APITestCase):
def _make_workflow(self, trigger_type) -> None:
trigger = WorkflowTrigger.objects.create(type=trigger_type)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
def test_consumption_trigger_requests_remote_ocr(self) -> None:
"""
GIVEN:
- A consumption workflow with a remote OCR action
WHEN:
- A matching document is consumed
THEN:
- The overrides ask for remote OCR, which is what the consumer
reads when choosing a parser
"""
self._make_workflow(WorkflowTrigger.WorkflowTriggerType.CONSUMPTION)
test_file = shutil.copy(
self.SAMPLE_DIR / "simple.pdf",
self.dirs.scratch_dir / "simple.pdf",
)
overrides = DocumentMetadataOverrides()
run_workflows(
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=test_file,
),
overrides=overrides,
)
self.assertTrue(overrides.remote_ocr)
def test_other_trigger_types_are_ignored(self) -> None:
"""
GIVEN:
- A workflow with a remote OCR action that also has a
non-consumption trigger, which is a valid combination
WHEN:
- The non-consumption trigger fires
THEN:
- The action is skipped, since the document has already been
parsed by this point
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
)
updated_trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger, updated_trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
original_filename="sample.pdf",
)
with self.assertLogs("paperless.handlers", level="DEBUG") as cm:
run_workflows(
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
doc,
)
self.assertIn("only applies to consumption triggers", "".join(cm.output))
+1 -8
View File
@@ -236,10 +236,8 @@ from paperless import version
from paperless.celery import app as celery_app
from paperless.config import AIConfig
from paperless.config import GeneralConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ApplicationConfiguration
from paperless.parsers.registry import get_parser_registry
from paperless.parsers.remote import RemoteEngineConfig
from paperless.serialisers import GroupSerializer
from paperless.serialisers import UserSerializer
from paperless.views import StandardPagination
@@ -2269,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions")
documents = [document]
documents = Document.objects.filter(pk=document.pk)
else:
documents = Document.objects.filter(
id__in=permitted_document_ids(request.user),
@@ -4012,11 +4010,6 @@ class UiSettingsView(GenericAPIView[Any]):
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
ui_settings["remote_ocr"] = {
"configured": RemoteEngineConfig.from_app_config().engine_is_valid(),
"mode": RemoteOCRConfig().remote_ocr_mode,
}
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
manager = PaperlessMailOAuth2Manager()
if settings.GMAIL_OAUTH_ENABLED:
+2 -1
View File
@@ -105,7 +105,8 @@ def apply_assignment_to_document(
field=field,
document=document,
).first()
if instance and args[value_field_name] is not None:
# empty string is indistinguishable from no value in the UI
if instance and args[value_field_name] not in (None, ""):
setattr(instance, value_field_name, args[value_field_name])
instance.save()
elif not instance:
+3 -17
View File
@@ -339,30 +339,16 @@ def check_deprecated_v2_ocr_env_vars(
@register()
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
# Import here because checks.py runs before the app registry is ready
from paperless.models import RemoteOCRMode
errors = []
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
):
errors.append(
return [
Error(
"Azure AI remote parser requires endpoint and API key to be configured.",
),
)
]
valid_modes = {mode.value for mode in RemoteOCRMode}
if settings.REMOTE_OCR_MODE not in valid_modes:
errors.append(
Error(
f"PAPERLESS_REMOTE_OCR_MODE is set to {settings.REMOTE_OCR_MODE!r}, "
f"expected one of {sorted(valid_modes)}.",
),
)
return errors
return []
def get_tesseract_langs():
-40
View File
@@ -9,7 +9,6 @@ from paperless.models import CleanChoices
from paperless.models import ColorConvertChoices
from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices
from paperless.models import RemoteOCRMode
@dataclasses.dataclass
@@ -186,45 +185,6 @@ class GeneralConfig(BaseConfig):
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
@dataclasses.dataclass
class RemoteOCRConfig(BaseConfig):
"""
Settings for the remote (cloud) OCR parser
"""
remote_ocr_engine: str | None = dataclasses.field(init=False)
remote_ocr_api_key: str | None = dataclasses.field(init=False)
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
def __post_init__(self) -> None:
app_config = self._get_config_instance()
self.remote_ocr_engine = (
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
)
self.remote_ocr_api_key = (
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
)
self.remote_ocr_endpoint = (
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
)
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
settings.REMOTE_OCR_MODE,
)
@property
def remote_ocr_by_default(self) -> bool:
"""
Whether every supported document goes to the remote engine.
When False the remote engine is used only for documents that
explicitly asked for it, i.e. a workflow matched during consumption or
the user ticked the box when reprocessing.
"""
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
@dataclasses.dataclass
class AIConfig(BaseConfig):
"""
@@ -1,44 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 14:37
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_api_key",
field=models.CharField(
blank=True,
max_length=1024,
null=True,
verbose_name="Sets the remote OCR API key",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_endpoint",
field=models.CharField(
blank=True,
max_length=256,
null=True,
verbose_name="Sets the remote OCR endpoint",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_engine",
field=models.CharField(
blank=True,
choices=[("azureai", "Azure AI Document Intelligence")],
max_length=32,
null=True,
verbose_name="Sets the remote OCR engine",
),
),
]
@@ -1,27 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 15:43
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_mode",
field=models.CharField(
blank=True,
choices=[
("always", "All supported documents"),
("workflow_only", "Only when a workflow enables it"),
],
max_length=32,
null=True,
verbose_name="Sets which documents are sent to the remote OCR engine",
),
),
]
-55
View File
@@ -74,23 +74,6 @@ class ColorConvertChoices(models.TextChoices):
CMYK = ("CMYK", _("CMYK"))
class RemoteOCREngine(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_ENGINE
"""
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
class RemoteOCRMode(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_MODE
"""
ALWAYS = ("always", _("All supported documents"))
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
class LLMEmbeddingBackend(models.TextChoices):
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
HUGGINGFACE = ("huggingface", _("Huggingface"))
@@ -303,44 +286,6 @@ class ApplicationConfiguration(AbstractSingletonModel):
null=True,
)
"""
Settings for the remote OCR parser
"""
# PAPERLESS_REMOTE_OCR_ENGINE
remote_ocr_engine = models.CharField(
verbose_name=_("Sets the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCREngine.choices,
)
# PAPERLESS_REMOTE_OCR_API_KEY
remote_ocr_api_key = models.CharField(
verbose_name=_("Sets the remote OCR API key"),
blank=True,
null=True,
max_length=1024,
)
# PAPERLESS_REMOTE_OCR_ENDPOINT
remote_ocr_endpoint = models.CharField(
verbose_name=_("Sets the remote OCR endpoint"),
blank=True,
null=True,
max_length=256,
)
# PAPERLESS_REMOTE_OCR_MODE
remote_ocr_mode = models.CharField(
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCRMode.choices,
)
"""
AI related settings
"""
-9
View File
@@ -134,11 +134,6 @@ class ParserProtocol(Protocol):
Author or organisation name.
url : str
URL for documentation, source code, or issue tracker.
Parsers that send document content to a remote service should additionally
set ``uses_remote_service = True`` so the registry can exclude them when
remote processing has not been requested for a document. The attribute is
optional so a parser that omits it is treated as fully local.
"""
# ------------------------------------------------------------------
@@ -150,10 +145,6 @@ class ParserProtocol(Protocol):
author: str
url: str
# NOTE: uses_remote_service is not declared here, the registry reads it
# with getattr(cls, ..., False) for backwards-compatibility with existing
# parsers
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
-14
View File
@@ -334,8 +334,6 @@ class ParserRegistry:
mime_type: str,
filename: str,
path: Path | None = None,
*,
allow_remote: bool = True,
) -> type[ParserProtocol] | None:
"""Return the best parser class for the given file, or None.
@@ -361,11 +359,6 @@ class ParserRegistry:
path:
Optional filesystem path to the file. Forwarded to each
parser's score method.
allow_remote:
When False, parsers that declare ``uses_remote_service = True``
are excluded from consideration, so a document is never sent to
a remote service. Parsers that do not declare the attribute
are treated as local and are always considered.
Returns
-------
@@ -381,13 +374,6 @@ class ParserRegistry:
if mime_type not in parser_class.supported_mime_types():
continue
if not allow_remote and getattr(
parser_class,
"uses_remote_service",
False,
):
continue
score = parser_class.score(mime_type, filename, path)
if score is None:
continue
+43 -26
View File
@@ -3,7 +3,9 @@ Built-in remote-OCR document parser.
Handles documents by sending them to a configured remote OCR engine
(currently Azure AI Vision / Document Intelligence) and retrieving both
the extracted text and a searchable PDF with an embedded text layer.
the extracted text and a searchable PDF with an embedded text layer. For
born-digital PDFs that need no archive copy, the remote call is skipped
entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
When no engine is configured, ``score()`` returns ``None`` so the parser
is effectively invisible to the registry the tesseract parser handles
@@ -22,6 +24,8 @@ from typing import Self
from django.conf import settings
from documents.parsers import ParseError
from paperless.parsers.utils import extract_pdf_text
from paperless.parsers.utils import post_process_text
from paperless.version import __full_version_str__
if TYPE_CHECKING:
@@ -57,18 +61,6 @@ class RemoteEngineConfig:
self.api_key = api_key
self.endpoint = endpoint
@classmethod
def from_app_config(cls) -> Self:
"""Build the config from the app config, falling back to the env."""
from paperless.config import RemoteOCRConfig
app_config = RemoteOCRConfig()
return cls(
engine=app_config.remote_ocr_engine,
api_key=app_config.remote_ocr_api_key,
endpoint=app_config.remote_ocr_endpoint,
)
def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured."""
return (
@@ -82,8 +74,11 @@ class RemoteDocumentParser:
"""Parse documents via a remote OCR API (currently Azure AI Vision).
This parser sends documents to a remote engine that returns both
extracted text and a searchable PDF with an embedded text layer.
It does not depend on Tesseract or ocrmypdf.
extracted text and a searchable PDF with an embedded text layer,
except when ``parse()`` is called with ``produce_archive=False`` for
a PDF, in which case the remote call is skipped and only locally
extracted text is returned (no archive). It does not depend on
Tesseract or ocrmypdf.
Class attributes
----------------
@@ -95,9 +90,6 @@ class RemoteDocumentParser:
Maintainer name.
url : str
Issue tracker / source URL.
uses_remote_service : bool
Content is sent to a remote service, True so that the registry
can skip this parser if remote processing was not requested.
"""
name: str = "Paperless-ngx Remote OCR Parser"
@@ -105,8 +97,6 @@ class RemoteDocumentParser:
author: str = "Paperless-ngx Contributors"
url: str = "https://github.com/paperless-ngx/paperless-ngx"
uses_remote_service: bool = True
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
@@ -155,7 +145,11 @@ class RemoteDocumentParser:
20 when the remote engine is configured and the MIME type is
supported, otherwise None.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
return None
if mime_type not in _SUPPORTED_MIME_TYPES:
@@ -173,8 +167,11 @@ class RemoteDocumentParser:
Returns
-------
bool
Always True the remote engine always returns a PDF with an
embedded text layer that serves as the archive copy.
Always True the remote engine is capable of returning a PDF
with an embedded text layer to serve as the archive copy.
Whether it actually does so for a given document depends on
``produce_archive`` passed to :meth:`parse` (see there for when
the remote engine call, and thus archive generation, is skipped).
"""
return True
@@ -231,6 +228,12 @@ class RemoteDocumentParser:
) -> None:
"""Send the document to the remote engine and store results.
When *produce_archive* is False for a PDF, the caller (via
``documents.consumer.should_produce_archive``) has already determined
that the document is born-digital and needs no archive skip the
remote engine entirely rather than re-OCRing it and creating a
duplicate text layer.
Parameters
----------
document_path:
@@ -238,10 +241,14 @@ class RemoteDocumentParser:
mime_type:
Detected MIME type of the document.
produce_archive:
Ignored the remote engine always returns a searchable PDF,
which is stored as the archive copy regardless of this flag.
Whether an archive copy is wanted. For PDFs, False skips the
remote engine and uses locally-extracted text instead.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
logger.warning(
@@ -250,6 +257,16 @@ class RemoteDocumentParser:
self._text = ""
return
if not produce_archive and mime_type == "application/pdf":
logger.debug(
"Remote OCR: skipped — no archive requested, "
"using locally-extracted text",
)
self._text = (
post_process_text(extract_pdf_text(document_path, log=logger)) or ""
)
return
if config.engine == "azureai":
self._text = self._azure_ai_vision_parse(document_path, config)
+5 -14
View File
@@ -219,13 +219,6 @@ class ApplicationConfigurationSerializer(
allow_null=True,
max_length=1024,
)
remote_ocr_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
max_length=1024,
)
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
def run_validation(self, data):
# Empty strings treated as None to avoid unexpected behavior
@@ -237,13 +230,11 @@ class ApplicationConfigurationSerializer(
data["language"] = None
if "llm_output_language" in data and data["llm_output_language"] == "":
data["llm_output_language"] = None
for field in self.OBFUSCATED_FIELDS:
if field in data and data[field] is not None:
if data[field] == "":
data[field] = None
# Not a real value, don't overwrite the stored one
elif len(data[field].replace("*", "")) == 0:
del data[field]
if "llm_api_key" in data and data["llm_api_key"] is not None:
if data["llm_api_key"] == "":
data["llm_api_key"] = None
elif len(data["llm_api_key"].replace("*", "")) == 0:
del data["llm_api_key"]
return super().run_validation(data)
def update(self, instance, validated_data):
-1
View File
@@ -1197,7 +1197,6 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
REMOTE_OCR_MODE = os.getenv("PAPERLESS_REMOTE_OCR_MODE", "always")
################################################################################
# AI Settings #
+111 -17
View File
@@ -21,7 +21,6 @@ from unittest.mock import Mock
import pytest
from documents.parsers import ParseError
from paperless.models import ApplicationConfiguration
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
from paperless.parsers.remote import RemoteDocumentParser
@@ -34,10 +33,6 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture
# Remote ocr config from ApplicationConfiguration needs DB access
pytestmark = pytest.mark.django_db
# ---------------------------------------------------------------------------
# Module-local fixtures
# ---------------------------------------------------------------------------
@@ -232,18 +227,6 @@ class TestRemoteParserScore:
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert score is not None and score > 10
@pytest.mark.usefixtures("no_engine_settings")
def test_score_uses_app_config_when_env_unset(self) -> None:
"""The app config alone is enough to activate the parser."""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_engine = "azureai"
config.remote_ocr_api_key = "app-config-key"
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
config.save()
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
# ---------------------------------------------------------------------------
# Properties
@@ -354,6 +337,117 @@ class TestRemoteParserParse:
assert remote_parser.get_date() is None
# ---------------------------------------------------------------------------
# parse() — produce_archive=False skips the remote engine (PDFs only)
# ---------------------------------------------------------------------------
class TestRemoteParserSkipsWhenNoArchiveWanted:
"""When the caller has already decided no archive is needed for a PDF
(documents.consumer.should_produce_archive), the remote engine call is
skipped entirely in favor of locally-extracted text.
"""
def test_pdf_skips_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: Azure is never invoked, no archive is produced, and text
comes from local pdftotext extraction
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_not_called()
assert remote_parser.get_archive_path() is None
assert remote_parser.get_text() != ""
def test_pdf_no_archive_requested_text_matches_local_extraction(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
mocker: MockerFixture,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: the returned text is exactly the locally-extracted text,
not anything from the (unused) Azure mock
"""
mocker.patch(
"paperless.parsers.remote.extract_pdf_text",
return_value="Local digital text.",
)
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == "Local digital text."
def test_pdf_no_archive_requested_closes_no_client(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.close.assert_not_called()
def test_non_pdf_still_calls_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
Images have no local-text fallback, so produce_archive=False does
not skip the remote engine for non-PDF MIME types.
"""
remote_parser.parse(
simple_digital_pdf_file,
"image/png",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_called_once()
assert remote_parser.get_text() == _DEFAULT_TEXT
@pytest.mark.usefixtures("no_engine_settings")
def test_unconfigured_engine_takes_precedence_over_skip(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
) -> None:
"""An unconfigured engine still short-circuits before the
produce_archive check, returning empty text as before.
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == ""
assert remote_parser.get_archive_path() is None
# ---------------------------------------------------------------------------
# parse() — Azure failure path
# ---------------------------------------------------------------------------
@@ -1277,8 +1277,6 @@ class TestParserFileTypes:
# ---------------------------------------------------------------------------
# Remote ocr config from ApplicationConfiguration needs DB access
@pytest.mark.django_db
class TestRasterisedDocumentParserRegistry:
def test_registered_in_defaults(self) -> None:
from paperless.parsers.registry import ParserRegistry
-17
View File
@@ -655,23 +655,6 @@ class TestRemoteParserChecks:
in msg.msg
)
def test_valid_mode(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
settings.REMOTE_OCR_MODE = "workflow_only"
msgs = check_remote_parser_configured(None)
assert len(msgs) == 0
def test_invalid_mode(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
settings.REMOTE_OCR_MODE = "sometimes"
msgs = check_remote_parser_configured(None)
assert len(msgs) == 1
assert "PAPERLESS_REMOTE_OCR_MODE is set to 'sometimes'" in msgs[0].msg
class TestTesseractChecks:
def test_default_language(self) -> None:
-118
View File
@@ -468,124 +468,6 @@ class TestParserRegistryGetParserForFile:
assert result is AcceptingBuiltin
class TestParserRegistryRemoteParsers:
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
@staticmethod
def _remote_parser_cls() -> type:
class RemoteParser:
name = "remote"
version = "1.0"
author = "A"
url = "https://example.com/remote"
uses_remote_service = True
@classmethod
def supported_mime_types(cls):
return {"text/plain": ".txt"}
@classmethod
def score(cls, mime_type, filename, path=None):
return 20
return RemoteParser
def test_remote_parser_wins_when_remote_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=True.
THEN: The remote parser is returned.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=True,
)
assert result is remote_parser_cls
def test_remote_parser_skipped_when_remote_not_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: The local parser is returned despite its lower score.
"""
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
"""
GIVEN: A registry whose only candidate declares uses_remote_service.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: None is returned the remote parser is never used as a
fallback when remote processing was not requested.
"""
registry = ParserRegistry()
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is None
def test_parser_without_attribute_treated_as_local(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A third-party parser predating uses_remote_service, so it does
not declare the attribute at all.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: It is still considered, i.e. treated as fully local, rather
than raising AttributeError.
"""
assert not hasattr(dummy_parser_cls, "uses_remote_service")
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_remote_allowed_by_default(self) -> None:
"""
GIVEN: A registry containing only a remote parser.
WHEN: get_parser_for_file() is called without allow_remote.
THEN: The remote parser is returned callers that do not opt in to
the filter keep the previous behaviour.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file("text/plain", "readme.txt")
assert result is remote_parser_cls
class TestDiscover:
"""Verify entrypoint discovery in ParserRegistry.discover()."""
@@ -1,113 +0,0 @@
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from django.test import override_settings
from paperless.config import RemoteOCRConfig
from paperless.models import RemoteOCRMode
if TYPE_CHECKING:
from unittest.mock import MagicMock
@pytest.fixture()
def null_app_config(mocker) -> MagicMock:
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
return mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
@pytest.fixture()
def make_remote_ocr_config(mocker):
def _make(app_config, **django_settings_overrides):
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
with override_settings(**django_settings_overrides):
return RemoteOCRConfig()
return _make
class TestRemoteOCRConfig:
def test_falls_back_to_settings(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "env-key"
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_app_config_takes_precedence(
self,
make_remote_ocr_config,
mocker,
) -> None:
app_config = mocker.MagicMock(
remote_ocr_engine="azureai",
remote_ocr_api_key="db-key",
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
)
cfg = make_remote_ocr_config(
app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "db-key"
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_unset_everywhere(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
assert cfg.remote_ocr_engine is None
assert cfg.remote_ocr_api_key is None
assert cfg.remote_ocr_endpoint is None
class TestRemoteOCRByDefault:
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_by_default is True
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_by_default is False
+21 -6
View File
@@ -2,6 +2,8 @@ import json
import logging
import sys
from django.db.models import QuerySet
from documents.models import Document
from paperless.config import AIConfig
from paperless_ai.client import AIClient
@@ -82,10 +84,21 @@ def _build_document_reference(
def _get_document_references(
documents: list[Document],
documents: QuerySet[Document],
top_nodes: list,
) -> list[dict[str, int | str]]:
allowed_documents = {doc.pk: doc for doc in documents}
candidate_ids: set[int] = set()
for node in top_nodes:
try:
candidate_ids.add(int(node.metadata["document_id"]))
except (KeyError, TypeError, ValueError): # pragma: no cover
continue
if not candidate_ids:
return []
allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
references: list[dict[str, int | str]] = []
seen_document_ids: set[int] = set()
@@ -119,7 +132,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
def stream_chat_with_documents(
query_str: str,
documents: list[Document],
documents: QuerySet[Document],
output_language: str | None = None,
):
try:
@@ -135,10 +148,10 @@ def stream_chat_with_documents(
def _stream_chat_with_documents(
query_str: str,
documents: list[Document],
documents: QuerySet[Document],
output_language: str | None = None,
):
if not documents:
if not documents.exists():
yield CHAT_NO_CONTENT_MESSAGE
return
@@ -148,7 +161,9 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig()
filters = _document_id_filters(str(doc.pk) for doc in documents)
filters = _document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
# Hold the shared read lock for the whole operation: the query engine
# retrieves from the vector store again during synthesis, so the connection
+101 -46
View File
@@ -3,10 +3,12 @@ from unittest.mock import MagicMock
from unittest.mock import patch
import pytest
from django.db.models.signals import post_init
from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode
from documents.models import Document
from documents.tests.factories import DocumentFactory
from paperless_ai import chat
from paperless_ai import indexing
@@ -36,16 +38,6 @@ def patch_embed_nodes():
yield mock_embed_nodes
@pytest.fixture
def mock_document():
doc = MagicMock()
doc.pk = 1
doc.title = "Test Document"
doc.filename = "test_file.pdf"
doc.content = "This is the document content."
return doc
def assert_chat_output(
output: list[str],
*,
@@ -61,6 +53,13 @@ def assert_chat_output(
}
def _fake_documents_queryset(pks: list[int]) -> MagicMock:
qs = MagicMock()
qs.exists.return_value = bool(pks)
qs.values_list.return_value = pks
return qs
@pytest.mark.parametrize(
("output_language", "expected_language_line"),
[
@@ -107,9 +106,10 @@ def test_build_refine_prompt(
@pytest.mark.django_db
def test_stream_chat_with_one_document_retrieval(
mock_document,
patch_embed_nodes,
) -> None:
document = DocumentFactory.create(title="Test Document", content="ignored")
documents = Document.objects.filter(pk=document.pk)
with (
patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -124,22 +124,19 @@ def test_stream_chat_with_one_document_retrieval(
mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock()
mock_node = TextNode(
text="This is node content.",
metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
)
mock_index = MagicMock()
# Simulate get_nodes returning nodes (content exists)
mock_index.vector_store.get_nodes.return_value = [mock_node]
mock_index.vector_store.get_nodes.return_value = [
TextNode(
text="This is node content.",
metadata={"document_id": str(document.pk), "title": "Test Document"},
),
]
mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [
MagicMock(
metadata={
"document_id": str(mock_document.pk),
"title": "Test Document",
},
metadata={"document_id": str(document.pk), "title": "Test Document"},
),
]
@@ -153,7 +150,7 @@ def test_stream_chat_with_one_document_retrieval(
"llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance,
):
output = list(stream_chat_with_documents("What is this?", [mock_document]))
output = list(stream_chat_with_documents("What is this?", documents))
mock_query_engine.query.assert_called_once_with("What is this?")
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
@@ -166,13 +163,16 @@ def test_stream_chat_with_one_document_retrieval(
output,
expected_chunks=["chunk1", "chunk2"],
expected_references=[
{"id": mock_document.pk, "title": "Test Document"},
{"id": document.pk, "title": "Test Document"},
],
)
@pytest.mark.django_db
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
doc1 = DocumentFactory.create(title="Document 1", content="ignored")
doc2 = DocumentFactory.create(title="Document 2", content="ignored")
documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
with (
patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock()
mock_node1 = TextNode(
text="Content for doc 1.",
metadata={"document_id": "1", "title": "Document 1"},
)
mock_node2 = TextNode(
text="Content for doc 2.",
metadata={"document_id": "2", "title": "Document 2"},
)
mock_index = MagicMock()
# Simulate get_nodes returning nodes (content exists)
mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2]
mock_index.vector_store.get_nodes.return_value = [
TextNode(
text="Content for doc 1.",
metadata={"document_id": str(doc1.pk), "title": "Document 1"},
),
TextNode(
text="Content for doc 2.",
metadata={"document_id": str(doc2.pk), "title": "Document 2"},
),
]
mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [
MagicMock(metadata={"document_id": "1", "title": "Document 1"}),
MagicMock(metadata={"document_id": "2", "title": "Document 2"}),
MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
]
mock_response_stream = MagicMock()
@@ -210,14 +210,11 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_query_engine_cls.return_value = mock_query_engine
mock_query_engine.query.return_value = mock_response_stream
doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
with patch(
"llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance,
):
output = list(stream_chat_with_documents("What's up?", [doc1, doc2]))
output = list(stream_chat_with_documents("What's up?", documents))
mock_query_engine.query.assert_called_once_with("What's up?")
patch_embed_nodes.assert_not_called()
@@ -225,15 +222,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
output,
expected_chunks=["chunk1", "chunk2"],
expected_references=[
{"id": 1, "title": "Document 1"},
{"id": 2, "title": "Document 2"},
{"id": doc1.pk, "title": "Document 1"},
{"id": doc2.pk, "title": "Document 2"},
],
)
def test_stream_chat_empty_document_list() -> None:
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
output = list(stream_chat_with_documents("Any info?", []))
output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
mock_load_index.assert_not_called()
assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -253,7 +250,9 @@ def test_stream_chat_no_matching_nodes() -> None:
mock_index.vector_store.get_nodes.return_value = []
mock_load_index.return_value = mock_index
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -282,7 +281,9 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
)
mock_retriever_cls.return_value = mock_retriever
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
assert output == [CHAT_ERROR_MESSAGE]
assert "Failed to stream document chat response" in caplog.text
@@ -298,7 +299,12 @@ class TestStreamChatRetrieval:
) -> None:
doc = DocumentFactory.create(content="hello world")
# Nothing indexed for this document yet.
out = list(chat.stream_chat_with_documents("question?", [doc]))
out = list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=doc.pk),
),
)
assert chat.CHAT_NO_CONTENT_MESSAGE in out
def test_chat_filter_contains_only_requested_document_ids(
@@ -332,7 +338,12 @@ class TestStreamChatRetrieval:
side_effect=capture_retriever,
)
list(chat.stream_chat_with_documents("question?", [included]))
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=included.pk),
),
)
assert captured_filters, "VectorIndexRetriever was never constructed"
filt = captured_filters[0]
@@ -340,3 +351,47 @@ class TestStreamChatRetrieval:
filter_values = filt.filters[0].value
assert str(included.pk) in filter_values
assert str(excluded.pk) not in filter_values
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
django_assert_num_queries,
) -> None:
"""Building references must not hydrate every document the caller is
permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
the retriever actually returned nodes for.
"""
referenced = DocumentFactory.create(title="Referenced Document")
# Many more documents are "accessible" but never referenced by a node.
DocumentFactory.create_batch(200)
documents = Document.objects.all()
top_nodes = [
MagicMock(
metadata={
"document_id": str(referenced.pk),
"title": "Referenced Document",
},
),
]
hydrated_count = 0
def _count_hydration(sender, instance, **kwargs):
nonlocal hydrated_count
hydrated_count += 1
post_init.connect(_count_hydration, sender=Document)
try:
# One query: `documents.filter(pk__in=candidate_ids)` for the single
# referenced id. No query should scale with the 200 unreferenced documents.
with django_assert_num_queries(1):
references = chat._get_document_references(documents, top_nodes)
finally:
post_init.disconnect(_count_hydration, sender=Document)
# The bug this guards against: the old code hydrated all 201 accessible
# documents via `{doc.pk: doc for doc in documents}` before filtering by
# top_nodes. Only the referenced document should ever be constructed.
assert hydrated_count == 1
assert references == [{"id": referenced.pk, "title": "Referenced Document"}]