Compare commits

..
Author SHA1 Message Date
shamoon 44a822cb10 Actually, fix the action dropdown thing 2026-08-10 20:29:33 -07:00
shamoon 6df70e62f7 Fix dynamic action fields thing 2026-08-10 11:11:32 -07:00
shamoon c478d7bb5d And docs 2026-08-10 10:57:39 -07:00
shamoon ef54e0564a Frotnend workflow stuff 2026-08-10 10:57:35 -07:00
shamoon 67628401c8 Ok! Backend stuff for the remote ocr workflow 2026-08-10 10:43:26 -07:00
shamoon 5c96b38f4e Docs 2026-08-10 10:13:24 -07:00
shamoon 21ac856e3f Ok, frontend reprocess remote option 2026-08-10 10:13:23 -07:00
shamoon 37d6b02ebc Backend stuff for remote ocr reprocess, add to bulk edit pass in from ui settings 2026-08-10 10:13:23 -07:00
shamoon 7456b52e84 Update consumer.py 2026-08-10 10:13:18 -07:00
shamoon 6a392ea099 Ok, wire up the remote_ocr_mode with allow_remote for consumer 2026-08-10 10:13:18 -07:00
shamoon 2032ad1341 Docs for remote_ocr_mode 2026-08-10 10:13:18 -07:00
shamoon cc8fee91c4 More tests for remote_ocr_mode 2026-08-10 10:13:18 -07:00
shamoon a5d46a883e Update config.component.spec.ts 2026-08-10 10:13:18 -07:00
shamoon b0e1793093 Checks for remote_ocr_mode and fix import 2026-08-10 10:13:17 -07:00
shamoon 7e466d1f71 remote_ocr_mode config setting 2026-08-10 10:13:17 -07:00
shamoon 7b69a178c0 Add to parser dev docs 2026-08-10 10:13:17 -07:00
shamoon 22cd13a8a9 uses_remote_service + allow_remote to allow opt-in / out of remote OCR 2026-08-10 10:13:17 -07:00
shamoon 72a4676be0 Update test_tesseract_parser.py 2026-08-10 10:11:14 -07:00
shamoon 6673144d23 Docs 2026-08-10 10:11:13 -07:00
shamoon 994a84cf92 Frontend stuff, with sections 2026-08-10 10:11:13 -07:00
shamoon 654ce5d8f3 Backend tests 2026-08-10 10:11:13 -07:00
shamoon 5d5e9b6db4 Backend changes and migration for remote OCR Config 2026-08-10 10:11:13 -07:00
GitHub Actions 62089df2d8 Auto translate strings 2026-08-10 02:26:58 +00:00
Trenton HandGitHub 5e5f6a88a3 Fix: deny deactivated users in permission filtering and auto-login (#13623)
* Fix: Hardening sweep, ensure a user is active, not just authenticated

* Missed this test
2026-08-10 02:25:06 +00:00
shamoonandGitHub 02e6c49c62 Chore: update fpdf2 to 2.8.8 (#13629) 2026-08-10 02:07:13 +00:00
shamoonandGitHub 3be64da4cb Chore: udpate pnpm, add blockExoticSubdeps (#13628) 2026-08-09 18:59:26 -07:00
75 changed files with 1817 additions and 758 deletions
-15
View File
@@ -299,8 +299,6 @@ optional arguments:
-sm, --split-manifest -sm, --split-manifest
-z, --zip -z, --zip
-zn, --zip-name -zn, --zip-name
--zip-compression
--zip-compression-level
--data-only --data-only
--no-progress-bar --no-progress-bar
--passphrase --passphrase
@@ -363,19 +361,6 @@ If `-z` or `--zip` is provided, the export will be a zip file
in the target directory, named according to the current local date or the in the target directory, named according to the current local date or the
value set in `-zn` or `--zip-name`. value set in `-zn` or `--zip-name`.
The compression method for the zip can be set with `--zip-compression`
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
`--zip-compression-level` (deflated: 09, bzip2: 19, zstd: -2222; ignored
for `stored` and `lzma`). Both options require `--zip`.
!!! warning
`zstd` compression requires Python 3.14 or newer on **both** the machine
creating the export and any machine importing it. An archive compressed with
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
imported on a runtime that lacks the codec; the importer will refuse it with
a clear error. The default `deflated` is universally readable.
If `--data-only` is provided, only the database will be exported. This option is intended If `--data-only` is provided, only the database will be exported. This option is intended
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory. to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
+2 -1
View File
@@ -301,7 +301,8 @@ The following methods are supported:
- `delete` - `delete`
- No `parameters` required - No `parameters` required
- `reprocess` - `reprocess`
- No `parameters` required - Optional `parameters`: `{ "remote_ocr": true }` to send the documents to the
remote OCR engine, see [Remote OCR](usage.md#remote-ocr). Defaults to false.
- `set_permissions` - `set_permissions`
- Requires `parameters`: - Requires `parameters`:
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or - `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
+12
View File
@@ -2047,6 +2047,18 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to None. Defaults to None.
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
: Which documents are sent to the remote OCR engine.
- `always`: every document of a supported file type is sent to the remote
engine, bypassing the local OCR engine.
- `workflow_only`: documents are processed locally unless a workflow
explicitly enables remote OCR for them, letting you use the remote engine
selectively.
Defaults to "always".
## AI {#ai} ## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED} #### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
+14
View File
@@ -456,6 +456,20 @@ def score(
return 10 return 10
``` ```
**Remote services**
If your parser sends document content to a remote service, declare it:
```python
class MyCustomParser:
uses_remote_service = True
```
Paperless-ngx excludes such parsers when the document being consumed has not
been marked for remote processing, so users can keep remote OCR off by default
and enable it selectively with a workflow. Parsers that do not declare the
attribute are treated as fully local and are always considered.
**Archive and rendition flags** **Archive and rendition flags**
```python ```python
+22 -1
View File
@@ -648,6 +648,19 @@ happened while it was still encrypted, that original version will likewise be mi
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle **Current limitation**: Passwords are stored as a simple list without descriptions. To handle
multiple PDF types with different passwords, create separate workflows for each use case. multiple PDF types with different passwords, create separate workflows for each use case.
##### Remote OCR {#workflow-action-remote-ocr}
"Remote OCR" actions send the document to the configured remote OCR engine instead of processing it
locally. To use remote OCR selectively, set the [remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE)
to `workflow_only` then add this action to a workflow that matches only the documents you
want sent to the remote engine. See [Remote OCR](#remote-ocr) for the engine setup. The action only works with
a **Consumption Started** trigger.
The action takes no options, its presence is what enables remote OCR for a matching document.
If the remote engine is not configured, or does not support the document's file type, the document is
processed locally instead and a warning is written to the log.
#### Workflow placeholders #### Workflow placeholders
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/). Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -1084,11 +1097,19 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence). [Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
settings can be supplied as environment variables or via **Application Configuration**.
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
or page limitations (e.g. with a free tier). or page limitations (e.g. with a free tier).
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
unless a [remote OCR workflow action](#workflow-action-remote-ocr) enables it for them, so you can limit the remote
engine to particular documents.
Setting the mode to `workflow_only` also allows the **Reprocess** actions to selectively use remote OCR for individual documents.
## Architecture ## Architecture
Paperless-ngx consists of the following components: Paperless-ngx consists of the following components:
+1 -1
View File
@@ -66,5 +66,5 @@
"ts-node": "~10.9.1", "ts-node": "~10.9.1",
"typescript": "^6.0.3" "typescript": "^6.0.3"
}, },
"packageManager": "pnpm@10.26.0" "packageManager": "pnpm@11.15.1"
} }
+1
View File
@@ -5,6 +5,7 @@ trustPolicy: no-downgrade
trustPolicyExclude: trustPolicyExclude:
- "chokidar@4.0.3" - "chokidar@4.0.3"
- "semver@6.3.1 || 5.7.2" - "semver@6.3.1 || 5.7.2"
blockExoticSubdeps: true
allowBuilds: allowBuilds:
"@parcel/watcher": true "@parcel/watcher": true
canvas: true canvas: true
@@ -14,43 +14,48 @@
<a ngbNavLink>{{category}}</a> <a ngbNavLink>{{category}}</a>
<ng-template ngbNavContent> <ng-template ngbNavContent>
<div class="p-3"> <div class="p-3">
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2"> @for (section of getCategorySections(category); track section) {
@for (option of getCategoryOptions(category); track option.key) { @if (section) {
<div class="col"> <h5 class="mt-4 mb-3">{{section}}</h5>
<div class="card bg-light"> }
<div class="card-body"> <div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
<div class="card-title d-flex align-items-center"> @for (option of getCategoryOptions(category, section); track option.key) {
<h6 class="mb-0"> <div class="col">
{{option.title}} <div class="card bg-light">
</h6> <div class="card-body">
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer"> <div class="card-title d-flex align-items-center">
<i-bs name="info-circle"></i-bs> <h6 class="mb-0">
</a> {{option.title}}
@if (isSet(option.key)) { </h6>
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)"> <a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container> <i-bs name="info-circle"></i-bs>
</button> </a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
} }
</div> </div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
}
</div> </div>
</div> </div>
</div> }
} </div>
</div> }
</div> </div>
</ng-template> </ng-template>
</li> </li>
@@ -8,7 +8,11 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgSelectModule } from '@ng-select/ng-select' import { NgSelectModule } from '@ng-select/ng-select'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons' import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of, throwError } from 'rxjs' import { of, throwError } from 'rxjs'
import { OutputTypeConfig } from 'src/app/data/paperless-config' import {
ConfigCategory,
ConfigSection,
OutputTypeConfig,
} from 'src/app/data/paperless-config'
import { ConfigService } from 'src/app/services/config.service' import { ConfigService } from 'src/app/services/config.service'
import { SettingsService } from 'src/app/services/settings.service' import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service' import { ToastService } from 'src/app/services/toast.service'
@@ -158,4 +162,24 @@ describe('ConfigComponent', () => {
component.resetOption('barcodes_enabled') component.resetOption('barcodes_enabled')
expect(component.configForm.get('barcodes_enabled').value).toBeNull() expect(component.configForm.get('barcodes_enabled').value).toBeNull()
}) })
it('should group options into sections within a category, or not', () => {
const sections = component.getCategorySections(ConfigCategory.OCR)
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
expect(
component
.getCategoryOptions(ConfigCategory.OCR)
.map((option) => option.key)
).toContain('output_type')
expect(
component
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
.map((option) => option.key)
).toEqual([
'remote_ocr_engine',
'remote_ocr_api_key',
'remote_ocr_endpoint',
'remote_ocr_mode',
])
})
}) })
@@ -74,8 +74,20 @@ export class ConfigComponent
return Object.values(ConfigCategory) return Object.values(ConfigCategory)
} }
getCategoryOptions(category: string): ConfigOption[] { getCategorySections(category: string): string[] {
return PaperlessConfigOptions.filter((o) => o.category === category) return [
...new Set(
PaperlessConfigOptions.filter((o) => o.category === category).map(
(o) => o.section ?? null // null means no section
)
),
]
}
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
return PaperlessConfigOptions.filter(
(o) => o.category === category && (o.section ?? null) === section
)
} }
initialConfig: PaperlessConfig initialConfig: PaperlessConfig
@@ -0,0 +1,28 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (messageBold) {
<p class="text-break"><b>{{messageBold}}</b></p>
}
@if (message) {
<p class="mb-0 text-break" [innerHTML]="message"></p>
}
@if (showRemoteOcr) {
<div class="form-check mt-3">
<input class="form-check-input" type="checkbox" id="reprocessRemoteOcr" [(ngModel)]="remoteOcr" />
<label class="form-check-label" for="reprocessRemoteOcr" i18n>Use remote OCR</label>
<div class="form-text" i18n>Sends the document to the configured remote OCR service, which may incur costs.</div>
</div>
}
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
{{btnCaption}}
</button>
</div>
@@ -0,0 +1,72 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { RemoteOCRModeConfig } from 'src/app/data/paperless-config'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { SettingsService } from 'src/app/services/settings.service'
import { ReprocessConfirmDialogComponent } from './reprocess-confirm-dialog.component'
describe('ReprocessConfirmDialogComponent', () => {
let component: ReprocessConfirmDialogComponent
let fixture: ComponentFixture<ReprocessConfirmDialogComponent>
let settingsService: SettingsService
const createComponent = (configured: boolean, mode: string) => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, configured)
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_MODE, mode)
fixture = TestBed.createComponent(ReprocessConfirmDialogComponent)
component = fixture.componentInstance
fixture.detectChanges()
}
beforeEach(async () => {
TestBed.configureTestingModule({
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
imports: [ReprocessConfirmDialogComponent],
}).compileComponents()
settingsService = TestBed.inject(SettingsService)
})
it('should not request remote OCR by default', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.remoteOcr).toBeFalsy()
})
it('should not offer remote OCR when no engine is configured', () => {
createComponent(false, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should not offer remote OCR when it already handles every document', () => {
createComponent(true, RemoteOCRModeConfig.ALWAYS)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should offer remote OCR when configured and selective', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeTruthy()
const checkbox = fixture.nativeElement.querySelector('#reprocessRemoteOcr')
expect(checkbox).not.toBeNull()
checkbox.click()
fixture.detectChanges()
expect(component.remoteOcr).toBeTruthy()
})
})
@@ -0,0 +1,20 @@
import { Component, inject } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { SettingsService } from 'src/app/services/settings.service'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-reprocess-confirm-dialog',
templateUrl: './reprocess-confirm-dialog.component.html',
imports: [FormsModule],
})
export class ReprocessConfirmDialogComponent extends ConfirmDialogComponent {
private settings = inject(SettingsService)
remoteOcr: boolean = false
public get showRemoteOcr(): boolean {
// Hidden when it is not configured, or when it already handles every document anyway.
return this.settings.remoteOCRIsSelectable
}
}
@@ -455,6 +455,13 @@
</div> </div>
</div> </div>
} }
@case (WorkflowActionType.RemoteOcr) {
<div class="row">
<div class="col">
<p class="text-muted small" i18n>The document will be sent to the configured remote OCR service. May incur costs.</p>
</div>
</div>
}
} }
</div> </div>
</ng-template> </ng-template>
@@ -29,6 +29,7 @@ import {
DocumentSource, DocumentSource,
WorkflowTriggerType, WorkflowTriggerType,
} from 'src/app/data/workflow-trigger' } from 'src/app/data/workflow-trigger'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive' import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive' import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CorrespondentService } from 'src/app/services/rest/correspondent.service' import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
@@ -224,7 +225,12 @@ describe('WorkflowEditDialogComponent', () => {
).toEqual('Document Added') ).toEqual('Document Added')
expect(component.getTriggerTypeOptionName(null)).toEqual('') expect(component.getTriggerTypeOptionName(null)).toEqual('')
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS) expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
expect(component.actionTypeOptions).toEqual(WORKFLOW_ACTION_OPTIONS) // Remote OCR is absent until the workflow has a consumption trigger
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.RemoteOcr
)
)
expect( expect(
component.getActionTypeOptionName(WorkflowActionType.Assignment) component.getActionTypeOptionName(WorkflowActionType.Assignment)
).toEqual('Assignment') ).toEqual('Assignment')
@@ -237,7 +243,104 @@ describe('WorkflowEditDialogComponent', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(false) jest.spyOn(settingsService, 'get').mockReturnValue(false)
component.ngOnInit() component.ngOnInit()
expect(component.actionTypeOptions).toEqual( expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter((a) => a.id !== WorkflowActionType.Email) WORKFLOW_ACTION_OPTIONS.filter(
(a) =>
a.id !== WorkflowActionType.Email &&
a.id !== WorkflowActionType.RemoteOcr
)
)
})
it('should offer remote OCR only for consumption workflows', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// A consumption trigger makes the action reachable
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Any other trigger type runs after the document has been parsed
component.object = {
name: 'Workflow 2',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should offer remote OCR on a trigger added to a new workflow', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
component.ngOnInit()
// Nothing for the action to apply to yet
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
// addTrigger creates the form field with emitEvent false, so the options
// have to be computed on read rather than cached from valueChanges
component.addTrigger()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Switching that trigger to a type that runs after parsing removes it
component.triggerFields
.at(0)
.get('type')
.setValue(WorkflowTriggerType.DocumentAdded)
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should keep remote OCR listed when an action already uses it', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [{ type: WorkflowActionType.RemoteOcr }],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
})
it('should not offer remote OCR when no engine is configured', () => {
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
) )
}) })
@@ -148,6 +148,10 @@ export const WORKFLOW_ACTION_OPTIONS = [
id: WorkflowActionType.MoveToTrash, id: WorkflowActionType.MoveToTrash,
name: $localize`Move to trash`, name: $localize`Move to trash`,
}, },
{
id: WorkflowActionType.RemoteOcr,
name: $localize`Remote OCR`,
},
] ]
export enum TriggerFilterType { export enum TriggerFilterType {
@@ -504,8 +508,6 @@ export class WorkflowEditDialogComponent
expandedItem: number = null expandedItem: number = null
readonly allowedActionTypes = signal([])
private readonly triggerFilterOptionsMap = new WeakMap< private readonly triggerFilterOptionsMap = new WeakMap<
FormArray, FormArray,
TriggerFilterOption[] TriggerFilterOption[]
@@ -548,13 +550,40 @@ export class WorkflowEditDialogComponent
this.checkRemovalActionFields.bind(this) this.checkRemovalActionFields.bind(this)
) )
this.checkRemovalActionFields(this.objectForm.value) this.checkRemovalActionFields(this.objectForm.value)
this.allowedActionTypes.set( }
this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)
? WORKFLOW_ACTION_OPTIONS private allowedActionTypes: typeof WORKFLOW_ACTION_OPTIONS = null
: WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.Email private getAllowedActionTypes() {
) let allowed = WORKFLOW_ACTION_OPTIONS
)
if (!this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.Email)
}
// Remote OCR is decided before the document is parsed, so it is only
// offered for workflows that run at consumption.
const formWorkflow: Workflow = this.objectForm?.value
const remoteOcrUsable =
this.settingsService.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
(formWorkflow?.triggers?.some(
(trigger) => trigger.type === WorkflowTriggerType.Consumption
) ||
formWorkflow?.actions?.some(
(action) => action.type === WorkflowActionType.RemoteOcr
))
if (!remoteOcrUsable) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.RemoteOcr)
}
if (
this.allowedActionTypes?.length === allowed.length &&
this.allowedActionTypes.every((a, i) => a.id === allowed[i].id)
) {
return this.allowedActionTypes
}
this.allowedActionTypes = allowed
return allowed
} }
private checkRemovalActionFields(formWorkflow: Workflow) { private checkRemovalActionFields(formWorkflow: Workflow) {
@@ -1279,7 +1308,8 @@ export class WorkflowEditDialogComponent
get actionTypeOptions() { get actionTypeOptions() {
this.settingsService.trackChanges() this.settingsService.trackChanges()
return this.allowedActionTypes() // Computed on read rather than cached
return this.getAllowedActionTypes()
} }
getActionTypeOptionName(type: WorkflowActionType): string { getActionTypeOptionName(type: WorkflowActionType): string {
@@ -963,12 +963,24 @@ describe('DocumentDetailComponent', () => {
component.reprocess() component.reprocess()
const modalCloseSpy = jest.spyOn(openModal, 'close') const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.confirmClicked.next() openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }) expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, false)
expect(modalSpy).toHaveBeenCalled() expect(modalSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalled() expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled() expect(modalCloseSpy).toHaveBeenCalled()
}) })
it('should pass remote OCR choice when reprocessing', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
reprocessSpy.mockReturnValue(of(true))
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
component.reprocess()
openModal.componentInstance.remoteOcr = true
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, true)
})
it('should show error if redo ocr call fails', () => { it('should show error if redo ocr call fails', () => {
initNormally() initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments') const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
@@ -97,6 +97,7 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
import * as UTIF from 'utif' import * as UTIF from 'utif'
import { DocumentDetailFieldID } from '../admin/settings/settings.component' import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component' import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component' import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component' import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component' import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
@@ -1402,7 +1403,7 @@ export class DocumentDetailComponent
} }
reprocess() { reprocess() {
let modal = this.modalService.open(ConfirmDialogComponent, { let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
backdrop: 'static', backdrop: 'static',
}) })
modal.componentInstance.title = $localize`Reprocess confirm` modal.componentInstance.title = $localize`Reprocess confirm`
@@ -1413,7 +1414,10 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked.subscribe(() => { modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false modal.componentInstance.buttonsEnabled = false
this.documentsService this.documentsService
.reprocessDocuments({ documents: [this.document().id] }) .reprocessDocuments(
{ documents: [this.document().id] },
modal.componentInstance.remoteOcr
)
.subscribe({ .subscribe({
next: () => { next: () => {
this.toastService.showInfo( this.toastService.showInfo(
@@ -1122,6 +1122,7 @@ describe('BulkEditorComponent', () => {
req.flush(true) req.flush(true)
expect(req.request.body).toEqual({ expect(req.request.body).toEqual({
documents: [3, 4], documents: [3, 4],
remote_ocr: false,
}) })
httpTestingController.match( httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true` `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
@@ -51,6 +51,7 @@ import { ToastService } from 'src/app/services/toast.service'
import { flattenTags } from 'src/app/utils/flatten-tags' import { flattenTags } from 'src/app/utils/flatten-tags'
import { queryParamsFromFilterRules } from 'src/app/utils/query-params' import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component' import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component' import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component' import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component' import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
@@ -909,7 +910,7 @@ export class BulkEditorComponent
} }
reprocessSelected() { reprocessSelected() {
let modal = this.modalService.open(ConfirmDialogComponent, { let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
backdrop: 'static', backdrop: 'static',
}) })
modal.componentInstance.title = $localize`Reprocess confirm` modal.componentInstance.title = $localize`Reprocess confirm`
@@ -923,7 +924,10 @@ export class BulkEditorComponent
modal.componentInstance.buttonsEnabled = false modal.componentInstance.buttonsEnabled = false
this.executeDocumentAction( this.executeDocumentAction(
modal, modal,
this.documentService.reprocessDocuments(this.getSelectionQuery()) this.documentService.reprocessDocuments(
this.getSelectionQuery(),
modal.componentInstance.remoteOcr
)
) )
}) })
} }
+55
View File
@@ -54,6 +54,10 @@ export const ConfigCategory = {
AI: $localize`AI Settings`, AI: $localize`AI Settings`,
} }
export const ConfigSection = {
RemoteOCR: $localize`Remote OCR`,
}
export const LLMEmbeddingBackendConfig = { export const LLMEmbeddingBackendConfig = {
OPENAI_LIKE: 'openai-like', OPENAI_LIKE: 'openai-like',
HUGGINGFACE: 'huggingface', HUGGINGFACE: 'huggingface',
@@ -65,6 +69,15 @@ export const LLMBackendConfig = {
OLLAMA: 'ollama', OLLAMA: 'ollama',
} }
export const RemoteOCREngineConfig = {
AZURE_AI: 'azureai',
}
export const RemoteOCRModeConfig = {
ALWAYS: 'always',
WORKFLOW_ONLY: 'workflow_only',
}
export interface ConfigOption { export interface ConfigOption {
key: string key: string
title: string title: string
@@ -72,6 +85,7 @@ export interface ConfigOption {
choices?: Array<{ id: string; name: string }> choices?: Array<{ id: string; name: string }>
config_key?: string config_key?: string
category: string category: string
section?: string
note?: string note?: string
} }
@@ -181,6 +195,43 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_OCR_USER_ARGS', config_key: 'PAPERLESS_OCR_USER_ARGS',
category: ConfigCategory.OCR, category: ConfigCategory.OCR,
}, },
{
key: 'remote_ocr_engine',
title: $localize`Remote OCR Engine`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCREngineConfig),
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
},
{
key: 'remote_ocr_api_key',
title: $localize`Remote OCR API Key`,
type: ConfigOptionType.Password,
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
},
{
key: 'remote_ocr_endpoint',
title: $localize`Remote OCR Endpoint`,
type: ConfigOptionType.String,
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Required when using the Azure AI engine.`,
},
{
key: 'remote_ocr_mode',
title: $localize`Remote OCR Mode`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCRModeConfig),
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
},
{ {
key: 'app_logo', key: 'app_logo',
title: $localize`Application Logo`, title: $localize`Application Logo`,
@@ -398,6 +449,10 @@ export interface PaperlessConfig extends ObjectWithId {
barcode_enable_tag: boolean barcode_enable_tag: boolean
barcode_tag_mapping: object barcode_tag_mapping: object
barcode_tag_split: boolean barcode_tag_split: boolean
remote_ocr_engine: string
remote_ocr_api_key: string
remote_ocr_endpoint: string
remote_ocr_mode: string
ai_enabled: boolean ai_enabled: boolean
llm_embedding_backend: string llm_embedding_backend: string
llm_embedding_model: string llm_embedding_model: string
+13
View File
@@ -1,5 +1,6 @@
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode' import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types' import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
import { RemoteOCRModeConfig } from './paperless-config'
import { User } from './user' import { User } from './user'
export interface UiSettings { export interface UiSettings {
@@ -94,6 +95,8 @@ export const SETTINGS_KEYS = {
OUTLOOK_OAUTH_URL: 'outlook_oauth_url', OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
EMAIL_ENABLED: 'email_enabled', EMAIL_ENABLED: 'email_enabled',
AI_ENABLED: 'ai_enabled', AI_ENABLED: 'ai_enabled',
REMOTE_OCR_CONFIGURED: 'remote_ocr:configured',
REMOTE_OCR_MODE: 'remote_ocr:mode',
} }
export const SETTINGS: UiSetting[] = [ export const SETTINGS: UiSetting[] = [
@@ -347,4 +350,14 @@ export const SETTINGS: UiSetting[] = [
type: 'string', type: 'string',
default: PdfEditorEditMode.Create, default: PdfEditorEditMode.Create,
}, },
{
key: SETTINGS_KEYS.REMOTE_OCR_CONFIGURED,
type: 'boolean',
default: false,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_MODE,
type: 'string',
default: RemoteOCRModeConfig.ALWAYS,
},
] ]
+1
View File
@@ -7,6 +7,7 @@ export enum WorkflowActionType {
Webhook = 4, Webhook = 4,
PasswordRemoval = 5, PasswordRemoval = 5,
MoveToTrash = 6, MoveToTrash = 6,
RemoteOcr = 7,
} }
export interface WorkflowActionEmail extends ObjectWithId { export interface WorkflowActionEmail extends ObjectWithId {
@@ -284,6 +284,21 @@ describe(`DocumentService`, () => {
expect(req.request.method).toEqual('POST') expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({ expect(req.request.body).toEqual({
documents: ids, documents: ids,
remote_ocr: false,
})
})
it('should request remote OCR when reprocessing with it enabled', () => {
const ids = [1, 2, 3]
subscription = service
.reprocessDocuments({ documents: ids }, true)
.subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/reprocess/`
)
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: true,
}) })
}) })
@@ -349,9 +349,13 @@ export class DocumentService extends AbstractPaperlessService<Document> {
}) })
} }
reprocessDocuments(selection: DocumentSelectionQuery) { reprocessDocuments(
selection: DocumentSelectionQuery,
remoteOcr: boolean = false
) {
return this.http.post(this.getResourceUrl(null, 'reprocess'), { return this.http.post(this.getResourceUrl(null, 'reprocess'), {
...selection, ...selection,
remote_ocr: remoteOcr,
}) })
} }
@@ -13,6 +13,7 @@ import { environment } from 'src/environments/environment'
import { CustomFieldDataType } from '../data/custom-field' import { CustomFieldDataType } from '../data/custom-field'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document' import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { SavedView } from '../data/saved-view' import { SavedView } from '../data/saved-view'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings' import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
import { PermissionsService } from './permissions.service' import { PermissionsService } from './permissions.service'
import { CustomFieldsService } from './rest/custom-fields.service' import { CustomFieldsService } from './rest/custom-fields.service'
@@ -434,4 +435,26 @@ describe('SettingsService', () => {
).name ).name
).toEqual(customFields[0].name) ).toEqual(customFields[0].name)
}) })
it('should offer remote OCR only when configured and selective', () => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, false)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
// configured, but already handling every document
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, true)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.ALWAYS
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeTruthy()
})
}) })
@@ -19,6 +19,7 @@ import {
} from 'src/app/utils/color' } from 'src/app/utils/color'
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment' import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document' import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SavedView } from '../data/saved-view' import { SavedView } from '../data/saved-view'
import { import {
PAPERLESS_GREEN_HEX, PAPERLESS_GREEN_HEX,
@@ -687,6 +688,17 @@ export class SettingsService {
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED) return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
} }
/**
* Offering remote OCR as a choice only makes sense when an engine
* is configured but is not already handling every document.
*/
get remoteOCRIsSelectable(): boolean {
return (
this.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
this.get(SETTINGS_KEYS.REMOTE_OCR_MODE) !== RemoteOCRModeConfig.ALWAYS
)
}
offerTour(): boolean { offerTour(): boolean {
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE) return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
} }
+8 -2
View File
@@ -394,10 +394,16 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
return "OK" return "OK"
def reprocess(doc_ids: list[int]) -> Literal["OK"]: def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
"""
Re-run parsing for the given documents.
Consumption workflows do not run here, so ``remote_ocr`` is how the user
asks for the remote engine when it is not configured to handle everything.
"""
for document_id in doc_ids: for document_id in doc_ids:
update_document_content_maybe_archive_file.apply_async( update_document_content_maybe_archive_file.apply_async(
kwargs={"document_id": document_id}, kwargs={"document_id": document_id, "remote_ocr": remote_ocr},
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL}, headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
) )
+18
View File
@@ -53,6 +53,7 @@ from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess from documents.utils import run_subprocess
from paperless.config import OcrConfig from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ArchiveFileGenerationChoices from paperless.models import ArchiveFileGenerationChoices
from paperless.parsers import ParserContext from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol from paperless.parsers import ParserProtocol
@@ -451,12 +452,19 @@ class ConsumerPlugin(
except Exception as e: except Exception as e:
self.log.error(f"Error attempting to clean PDF: {e}") self.log.error(f"Error attempting to clean PDF: {e}")
# Workflows have already run at this point, so the metadata knows
# whether this document was singled out for remote OCR
allow_remote = (
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
)
# Based on the mime type, get the parser for that type # Based on the mime type, get the parser for that type
parser_class: type[ParserProtocol] | None = ( parser_class: type[ParserProtocol] | None = (
get_parser_registry().get_parser_for_file( get_parser_registry().get_parser_for_file(
mime_type, mime_type,
self.filename, self.filename,
self.working_copy, self.working_copy,
allow_remote=allow_remote,
) )
) )
if not parser_class: if not parser_class:
@@ -465,6 +473,16 @@ class ConsumerPlugin(
f"Unsupported mime type {mime_type}", f"Unsupported mime type {mime_type}",
) )
if self.metadata.remote_ocr and not getattr(
parser_class,
"uses_remote_service",
False,
):
self.log.warning(
"Remote OCR was requested for this document but no remote "
"parser is available for it, processing locally instead.",
)
# Notify all listeners that we're going to do some work. # Notify all listeners that we're going to do some work.
document_consumption_started.send( document_consumption_started.send(
+3
View File
@@ -34,6 +34,7 @@ class DocumentMetadataOverrides:
skip_asn_if_exists: bool = False skip_asn_if_exists: bool = False
version_label: str | None = None version_label: str | None = None
actor_id: int | None = None actor_id: int | None = None
remote_ocr: bool = False
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides": def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
""" """
@@ -57,6 +58,8 @@ class DocumentMetadataOverrides:
self.actor_id = other.actor_id self.actor_id = other.actor_id
if other.skip_asn_if_exists: if other.skip_asn_if_exists:
self.skip_asn_if_exists = True self.skip_asn_if_exists = True
if other.remote_ocr:
self.remote_ocr = True
if other.version_label is not None: if other.version_label is not None:
self.version_label = other.version_label self.version_label = other.version_label
-108
View File
@@ -1,108 +0,0 @@
from __future__ import annotations
import importlib
import zipfile
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
# availability is enforced separately in compression_available().
COMPRESSION_CHOICES: tuple[str, ...] = (
"stored",
"deflated",
"bzip2",
"lzma",
"zstd",
)
# Method name -> zipfile compression constant (zstd only when supported).
COMPRESSION_METHODS: dict[str, int] = {
"stored": zipfile.ZIP_STORED,
"deflated": zipfile.ZIP_DEFLATED,
"bzip2": zipfile.ZIP_BZIP2,
"lzma": zipfile.ZIP_LZMA,
}
if ZSTD is not None:
COMPRESSION_METHODS["zstd"] = ZSTD
# Inclusive (min, max) level bounds per method; None => level not applicable.
# Verified on CPython 3.14.3.
#
# zstd's raw library bounds are (-131072, 22)
# (compression.zstd.CompressionParameter.compression_level.bounds()) — the
# minimum is an internal implementation constant (-ZSTD_TARGETLENGTH_MAX),
# not a meaningful distinct "level"; deeper negative values than -22 buy
# nothing over -22 in practice. We expose the conventional zstd CLI range
# instead of the raw library bounds.
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
"stored": None,
"deflated": (0, 9),
"bzip2": (1, 9),
"lzma": None,
"zstd": (-22, 22),
}
# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
# zstd id that zipfile can still read.
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
zipfile.ZIP_STORED: "stored",
zipfile.ZIP_DEFLATED: "deflated",
zipfile.ZIP_BZIP2: "bzip2",
zipfile.ZIP_LZMA: "lzma",
93: "zstd",
20: "zstd",
}
def compression_available(method: str) -> bool:
"""Whether the running interpreter can actually use the given method."""
if method in ("stored", "deflated"):
# zlib is a hard CPython dependency; stored needs nothing.
return True
if method == "bzip2":
return _module_importable("bz2")
if method == "lzma":
return _module_importable("lzma")
if method == "zstd":
return ZSTD is not None and _module_importable("compression.zstd")
return False
def _module_importable(name: str) -> bool:
try:
importlib.import_module(name)
except ImportError:
return False
return True
def level_error(method: str, level: int | None) -> str | None:
"""Return a human message if (method, level) is invalid, else None."""
if level is None:
return None
bounds = LEVEL_BOUNDS[method]
if bounds is None:
return f"--zip-compression-level has no effect for '{method}'"
low, high = bounds
if not (low <= level <= high):
return (
f"--zip-compression-level for '{method}' must be between {low} and {high}"
)
return None
def compress_type_readable(compress_type: int) -> bool:
"""Whether this interpreter can decompress an entry of the given type."""
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
if method is None:
return False
return compression_available(method)
def unreadable_method_names(compress_types: set[int]) -> set[str]:
"""Map a set of compress_type ids to human method names for error messages."""
names: set[str] = set()
for ct in compress_types:
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
return names
+2 -13
View File
@@ -243,21 +243,11 @@ class ZipExportSink(ExportSink):
added as an entry at finalize (a zip entry cannot be interleaved with others). added as an entry at finalize (a zip entry cannot be interleaved with others).
""" """
def __init__( def __init__(self, target: Path, zip_name: str, *, delete: bool = False) -> None:
self,
target: Path,
zip_name: str,
*,
delete: bool = False,
compression: int = zipfile.ZIP_DEFLATED,
compresslevel: int | None = None,
) -> None:
self._target = target.resolve() self._target = target.resolve()
self._zip_path = (self._target / zip_name).with_suffix(".zip") self._zip_path = (self._target / zip_name).with_suffix(".zip")
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp") self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
self._delete = delete self._delete = delete
self._compression = compression
self._compresslevel = compresslevel
self._zip: zipfile.ZipFile | None = None self._zip: zipfile.ZipFile | None = None
self._dirs: set[str] = set() self._dirs: set[str] = set()
self._pending_manifest: tuple[Path, str] | None = None self._pending_manifest: tuple[Path, str] | None = None
@@ -268,8 +258,7 @@ class ZipExportSink(ExportSink):
self._zip = zipfile.ZipFile( self._zip = zipfile.ZipFile(
self._tmp_path, self._tmp_path,
"w", "w",
compression=self._compression, compression=zipfile.ZIP_DEFLATED,
compresslevel=self._compresslevel,
allowZip64=True, allowZip64=True,
) )
+6
View File
@@ -1047,6 +1047,12 @@ class PermittedObjectsFilter(BaseFilterBackend):
perm_codename: str | None = None perm_codename: str | None = None
def filter_queryset(self, request, queryset, view): def filter_queryset(self, request, queryset, view):
# Before the superuser and owner-only paths, neither of which consults
# permitted_object_ids. Scoped to authenticated users so anonymous
# access (AnonymousUser.is_active is False) keeps its existing
# unowned-only behaviour.
if request.user.is_authenticated and not request.user.is_active:
return queryset.none()
if request.user.is_superuser: if request.user.is_superuser:
return queryset return queryset
if not self.include_granted: if not self.include_granted:
@@ -29,11 +29,6 @@ if TYPE_CHECKING:
if settings.AUDIT_LOG_ENABLED: if settings.AUDIT_LOG_ENABLED:
from auditlog.models import LogEntry from auditlog.models import LogEntry
from documents.export.compression import COMPRESSION_CHOICES
from documents.export.compression import COMPRESSION_METHODS
from documents.export.compression import ZSTD
from documents.export.compression import compression_available
from documents.export.compression import level_error
from documents.export.sinks import DirectoryExportSink from documents.export.sinks import DirectoryExportSink
from documents.export.sinks import ExportSink from documents.export.sinks import ExportSink
from documents.export.sinks import StreamingManifestWriter from documents.export.sinks import StreamingManifestWriter
@@ -197,28 +192,6 @@ class Command(CryptMixin, PaperlessCommand):
help="Sets the export zip file name", help="Sets the export zip file name",
) )
parser.add_argument(
"--zip-compression",
choices=COMPRESSION_CHOICES,
default=None,
help=(
"Compression method for the export zip (requires --zip). "
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
"exporting and importing machine."
),
)
parser.add_argument(
"--zip-compression-level",
type=int,
default=None,
help=(
"Compression level for the export zip (requires --zip). "
"deflated: 0-9, bzip2: 1-9, zstd: -22..22; ignored for "
"stored/lzma."
),
)
parser.add_argument( parser.add_argument(
"--data-only", "--data-only",
default=False, default=False,
@@ -274,39 +247,12 @@ class Command(CryptMixin, PaperlessCommand):
if not os.access(self.target, os.W_OK): if not os.access(self.target, os.W_OK):
raise CommandError("That path doesn't appear to be writable") raise CommandError("That path doesn't appear to be writable")
zip_compression: str | None = options["zip_compression"]
zip_compression_level: int | None = options["zip_compression_level"]
if not self.zip_export and (
zip_compression is not None or zip_compression_level is not None
):
raise CommandError(
"--zip-compression and --zip-compression-level require --zip",
)
compression_method = zip_compression or "deflated"
if self.zip_export:
if not compression_available(compression_method):
if compression_method == "zstd" and ZSTD is None:
raise CommandError(
"zstd compression requires Python 3.14 or newer",
)
raise CommandError(
f"Compression method '{compression_method}' is not "
f"available on this Python runtime",
)
level_msg = level_error(compression_method, zip_compression_level)
if level_msg is not None:
raise CommandError(level_msg)
sink: ExportSink sink: ExportSink
if self.zip_export: if self.zip_export:
sink = ZipExportSink( sink = ZipExportSink(
self.target, self.target,
options["zip_name"], options["zip_name"],
delete=self.delete, delete=self.delete,
compression=COMPRESSION_METHODS[compression_method],
compresslevel=zip_compression_level,
) )
else: else:
sink = DirectoryExportSink( sink = DirectoryExportSink(
@@ -32,8 +32,6 @@ from django.db.models.signals import post_save
from filelock import FileLock from filelock import FileLock
from guardian.shortcuts import clear_ct_cache from guardian.shortcuts import clear_ct_cache
from documents.export.compression import compress_type_readable
from documents.export.compression import unreadable_method_names
from documents.file_handling import create_source_path_directory from documents.file_handling import create_source_path_directory
from documents.management.commands.base import PaperlessCommand from documents.management.commands.base import PaperlessCommand
from documents.management.commands.mixins import CryptMixin from documents.management.commands.mixins import CryptMixin
@@ -462,20 +460,6 @@ class Command(CryptMixin, PaperlessCommand):
with tempfile.TemporaryDirectory() as tmp_dir: with tempfile.TemporaryDirectory() as tmp_dir:
if is_zipfile(self.source): if is_zipfile(self.source):
with ZipFile(self.source) as zf: with ZipFile(self.source) as zf:
unsupported = {
info.compress_type
for info in zf.infolist()
if not compress_type_readable(info.compress_type)
}
if unsupported:
names = sorted(unreadable_method_names(unsupported))
message = (
f"This archive uses compression this Python cannot "
f"read ({', '.join(names)})."
)
if "zstd" in names:
message += " zstd archives require Python 3.14+."
raise CommandError(message)
zf.extractall(tmp_dir) zf.extractall(tmp_dir)
self.source = Path(tmp_dir) self.source = Path(tmp_dir)
self._run_import() self._run_import()
@@ -0,0 +1,30 @@
# Generated by Django 5.2.16 on 2026-08-10 17:27
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0022_add_perf_indexes"),
]
operations = [
migrations.AlterField(
model_name="workflowaction",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Assignment"),
(2, "Removal"),
(3, "Email"),
(4, "Webhook"),
(5, "Password removal"),
(6, "Move to trash"),
(7, "Remote OCR"),
],
default=1,
verbose_name="Workflow Action Type",
),
),
]
+4
View File
@@ -1599,6 +1599,10 @@ class WorkflowAction(models.Model):
6, 6,
_("Move to trash"), _("Move to trash"),
) )
REMOTE_OCR = (
7,
_("Remote OCR"),
)
type = models.PositiveSmallIntegerField( type = models.PositiveSmallIntegerField(
_("Workflow Action Type"), _("Workflow Action Type"),
+18 -3
View File
@@ -54,11 +54,15 @@ class PaperlessObjectPermissions(DjangoObjectPermissions):
class PaperlessAdminPermissions(BasePermission): class PaperlessAdminPermissions(BasePermission):
def has_permission(self, request, view): def has_permission(self, request, view):
return request.user.is_staff return request.user.is_active and request.user.is_staff
def has_global_statistics_permission(user: User | None) -> bool: def has_global_statistics_permission(user: User | None) -> bool:
if user is None or not getattr(user, "is_authenticated", False): if (
user is None
or not getattr(user, "is_active", False)
or not getattr(user, "is_authenticated", False)
):
return False return False
return getattr(user, "is_superuser", False) or user.has_perm( return getattr(user, "is_superuser", False) or user.has_perm(
@@ -67,7 +71,11 @@ def has_global_statistics_permission(user: User | None) -> bool:
def has_system_status_permission(user: User | None) -> bool: def has_system_status_permission(user: User | None) -> bool:
if user is None or not getattr(user, "is_authenticated", False): if (
user is None
or not getattr(user, "is_active", False)
or not getattr(user, "is_authenticated", False)
):
return False return False
return ( return (
@@ -188,6 +196,13 @@ def permitted_object_ids(
if user is None or not getattr(user, "is_authenticated", False): if user is None or not getattr(user, "is_authenticated", False):
return base_qs.filter(owner__isnull=True).values_list("id", flat=True) return base_qs.filter(owner__isnull=True).values_list("id", flat=True)
# Deactivated users get nothing, deactivated superusers included, so this
# has to come before the superuser shortcut. guardian's
# ObjectPermissionChecker denies inactive users, but get_objects_for_user
# (the pattern this replaces) does not, so it would not be inherited.
if not getattr(user, "is_active", False):
return base_qs.none().values_list("id", flat=True)
if getattr(user, "is_superuser", False): if getattr(user, "is_superuser", False):
return base_qs.values_list("id", flat=True) return base_qs.values_list("id", flat=True)
+30 -1
View File
@@ -1744,7 +1744,7 @@ class DeleteDocumentsSerializer(DocumentSelectionSerializer):
class ReprocessDocumentsSerializer(DocumentSelectionSerializer): class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
pass remote_ocr = serializers.BooleanField(required=False, default=False)
class BulkEditSerializer( class BulkEditSerializer(
@@ -2086,6 +2086,13 @@ class BulkEditSerializer(
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.", f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
) )
def _validate_parameters_reprocess(self, parameters) -> None:
if "remote_ocr" in parameters:
if not isinstance(parameters["remote_ocr"], bool):
raise serializers.ValidationError("remote_ocr must be a boolean")
else:
parameters["remote_ocr"] = False
def validate_parameters_remove_password(self, parameters): def validate_parameters_remove_password(self, parameters):
if "password" not in parameters: if "password" not in parameters:
raise serializers.ValidationError("password not specified") raise serializers.ValidationError("password not specified")
@@ -2150,6 +2157,8 @@ class BulkEditSerializer(
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0]) self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
elif method == bulk_edit.remove_password: elif method == bulk_edit.remove_password:
self.validate_parameters_remove_password(parameters) self.validate_parameters_remove_password(parameters)
elif method == bulk_edit.reprocess:
self._validate_parameters_reprocess(parameters)
return attrs return attrs
@@ -3268,6 +3277,26 @@ class WorkflowSerializer(serializers.ModelSerializer[Workflow]):
"actions", "actions",
] ]
def validate(self, attrs):
attrs = super().validate(attrs)
triggers = attrs.get("triggers") or []
actions = attrs.get("actions") or []
# Remote OCR can only work with consumption triggers
if any(
action.get("type") == WorkflowAction.WorkflowActionType.REMOTE_OCR
for action in actions
) and not any(
trigger.get("type") == WorkflowTrigger.WorkflowTriggerType.CONSUMPTION
for trigger in triggers
):
raise serializers.ValidationError(
"Remote OCR actions require a consumption started trigger",
)
return attrs
def update_triggers_and_actions( def update_triggers_and_actions(
self, self,
instance: Workflow, instance: Workflow,
+11
View File
@@ -971,6 +971,17 @@ def run_workflows(
) )
elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH: elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH:
has_move_to_trash_action = True has_move_to_trash_action = True
elif action.type == WorkflowAction.WorkflowActionType.REMOTE_OCR:
if use_overrides and overrides:
overrides.remote_ocr = True
else:
# If a workflow has a consumption trigger *and* another type,
# the document has already been parsed by the time the other one fires
logger.debug(
"Remote OCR action only applies to consumption "
"triggers, ignoring",
extra={"group": logging_group},
)
if not use_overrides: if not use_overrides:
# limit title to 128 characters # limit title to 128 characters
+10 -1
View File
@@ -66,6 +66,7 @@ from documents.utils import compute_checksum
from documents.utils import identity from documents.utils import identity
from documents.workflows.utils import get_workflows_for_trigger from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig from paperless.config import AIConfig
from paperless.config import RemoteOCRConfig
from paperless.logging import consume_task_id from paperless.logging import consume_task_id
from paperless.parsers import ParserContext from paperless.parsers import ParserContext
from paperless.parsers.registry import get_parser_registry from paperless.parsers.registry import get_parser_registry
@@ -337,10 +338,17 @@ def bulk_update_documents(document_ids) -> None:
@shared_task @shared_task
def update_document_content_maybe_archive_file(document_id) -> None: def update_document_content_maybe_archive_file(
document_id,
*,
remote_ocr: bool = False,
) -> None:
""" """
Re-creates OCR content and thumbnail for a document, and archive file if Re-creates OCR content and thumbnail for a document, and archive file if
it exists. it exists.
Remote OCR is used only when the engine is configured to handle everything
or if explicitly asked for via ``remote_ocr``.
""" """
document = Document.objects.get(id=document_id) document = Document.objects.get(id=document_id)
@@ -350,6 +358,7 @@ def update_document_content_maybe_archive_file(document_id) -> None:
mime_type, mime_type,
document.original_filename or "", document.original_filename or "",
document.source_path, document.source_path,
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
) )
if not parser_class: if not parser_class:
@@ -1,190 +0,0 @@
import sys
import zipfile
import pytest
from documents.export import compression
class TestCompressionMethods:
def test_choices_always_include_zstd(self) -> None:
"""
GIVEN:
- The compression policy module's CLI choices list
WHEN:
- Read on any runtime
THEN:
- zstd is always present; availability is checked separately so
argparse never hides it based on the current Python version
"""
assert compression.COMPRESSION_CHOICES == (
"stored",
"deflated",
"bzip2",
"lzma",
"zstd",
)
@pytest.mark.parametrize(
("name", "constant"),
[
("stored", zipfile.ZIP_STORED),
("deflated", zipfile.ZIP_DEFLATED),
("bzip2", zipfile.ZIP_BZIP2),
("lzma", zipfile.ZIP_LZMA),
],
)
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
"""
GIVEN:
- A compression method name
WHEN:
- Looked up in COMPRESSION_METHODS
THEN:
- It maps to the matching zipfile compression constant
"""
assert compression.COMPRESSION_METHODS[name] == constant
def test_stored_and_deflated_always_available(self) -> None:
"""
GIVEN:
- The stored and deflated compression methods
WHEN:
- Checked with compression_available()
THEN:
- Both are always available (zlib is a hard CPython dependency)
"""
assert compression.compression_available("stored")
assert compression.compression_available("deflated")
def test_zstd_availability_tracks_runtime(self) -> None:
"""
GIVEN:
- The zstd compression method
WHEN:
- Checked with compression_available() on this runtime
THEN:
- Availability matches whether Python is 3.14+
"""
expected: bool = sys.version_info >= (3, 14)
assert compression.compression_available("zstd") == expected
class TestLevelError:
@pytest.mark.parametrize(
("method", "level"),
[
("deflated", 0),
("deflated", 9),
("bzip2", 1),
("bzip2", 9),
("zstd", -22),
("zstd", 22),
("deflated", None),
("stored", None),
],
)
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
"""
GIVEN:
- A method and a level within its valid bounds (or no level)
WHEN:
- Checked with level_error()
THEN:
- No error message is returned
"""
assert compression.level_error(method, level) is None
@pytest.mark.parametrize(
("method", "level"),
[
("deflated", 10),
("deflated", -1),
("bzip2", 0),
("bzip2", 10),
("zstd", -23),
("zstd", 23),
],
)
def test_out_of_range_levels_return_message(
self,
method: str,
level: int,
) -> None:
"""
GIVEN:
- A method and a level outside its valid bounds
WHEN:
- Checked with level_error()
THEN:
- An error message naming the valid range is returned
"""
msg: str | None = compression.level_error(method, level)
assert msg is not None
assert "between" in msg
@pytest.mark.parametrize("method", ["stored", "lzma"])
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
"""
GIVEN:
- A method that ignores compression level (stored, lzma)
WHEN:
- A level is passed to level_error() anyway
THEN:
- An error message noting the level has no effect is returned
"""
msg: str | None = compression.level_error(method, 5)
assert msg is not None
assert "no effect" in msg
class TestCompressTypeReadable:
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
"""
GIVEN:
- A stored or deflated compress_type id
WHEN:
- Checked with compress_type_readable()
THEN:
- It is always readable
"""
assert compression.compress_type_readable(ct)
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
"""
GIVEN:
- The current (93) and legacy (20) zstd compress_type ids
WHEN:
- Checked with compress_type_readable() on this runtime
THEN:
- Readability matches whether Python is 3.14+
"""
# 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
expected: bool = sys.version_info >= (3, 14)
assert compression.compress_type_readable(93) == expected
assert compression.compress_type_readable(20) == expected
def test_unknown_compress_type_is_unreadable(self) -> None:
"""
GIVEN:
- An unrecognized compress_type id
WHEN:
- Checked with compress_type_readable()
THEN:
- It is reported as unreadable
"""
assert not compression.compress_type_readable(9999)
def test_unreadable_method_names_lists_methods(self) -> None:
"""
GIVEN:
- A set containing an unknown compress_type id
WHEN:
- Passed to unreadable_method_names()
THEN:
- It is reported generically as "method <id>"
"""
# An unknown method id maps to no name and is reported generically.
names: set[str] = compression.unreadable_method_names({9999})
assert names == {"method 9999"}
-43
View File
@@ -5,7 +5,6 @@ import zipfile
from pathlib import Path from pathlib import Path
import pytest import pytest
import pytest_mock
from pytest_django.fixtures import SettingsWrapper from pytest_django.fixtures import SettingsWrapper
from documents.export.sinks import DirectoryExportSink from documents.export.sinks import DirectoryExportSink
@@ -306,48 +305,6 @@ class TestZipExportSink:
assert not (target / "export.zip").exists() assert not (target / "export.zip").exists()
class TestZipExportSinkCompression:
@pytest.mark.parametrize(
("method", "constant"),
[
("stored", zipfile.ZIP_STORED),
("deflated", zipfile.ZIP_DEFLATED),
("bzip2", zipfile.ZIP_BZIP2),
("lzma", zipfile.ZIP_LZMA),
],
)
def test_compression_and_level_forwarded_to_zipfile(
self,
mocker: pytest_mock.MockerFixture,
tmp_path: Path,
method: str,
constant: int,
) -> None:
"""
GIVEN:
- A ZipExportSink constructed with a compression method and level
WHEN:
- The sink is opened
THEN:
- zipfile.ZipFile is constructed with those values forwarded
unchanged (whether ZipFile actually compresses is Python's own
contract, not ours, so this checks the call args, not a real
archive)
"""
target: Path = tmp_path / "out"
target.mkdir()
zip_cls = mocker.patch("documents.export.sinks.zipfile.ZipFile")
sink = ZipExportSink(target, "export", compression=constant, compresslevel=5)
sink._open()
zip_cls.assert_called_once_with(
mocker.ANY,
"w",
compression=constant,
compresslevel=5,
allowZip64=True,
)
class TestStreamContract: class TestStreamContract:
@pytest.fixture(params=["dir", "zip"]) @pytest.fixture(params=["dir", "zip"])
def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink: def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink:
@@ -72,6 +72,10 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"barcode_enable_tag": None, "barcode_enable_tag": None,
"barcode_tag_mapping": None, "barcode_tag_mapping": None,
"barcode_tag_split": None, "barcode_tag_split": None,
"remote_ocr_engine": None,
"remote_ocr_api_key": None,
"remote_ocr_endpoint": None,
"remote_ocr_mode": None,
"ai_enabled": False, "ai_enabled": False,
"llm_embedding_backend": None, "llm_embedding_backend": None,
"llm_embedding_model": None, "llm_embedding_model": None,
@@ -870,6 +874,49 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
config.refresh_from_db() config.refresh_from_db()
self.assertEqual(config.llm_api_key, None) self.assertEqual(config.llm_api_key, None)
def test_update_remote_ocr_api_key(self) -> None:
"""
GIVEN:
- Existing config with remote_ocr_api_key specified
WHEN:
- API to update remote_ocr_api_key is called with all *s
- API to update remote_ocr_api_key is called with empty string
THEN:
- remote_ocr_api_key is unchanged
- remote_ocr_api_key is set to None
"""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_api_key = "1234567890"
config.save()
# Test with all *
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "*" * 32,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, "1234567890")
# Test with empty string
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, None)
def test_enable_ai_index_triggers_update(self) -> None: def test_enable_ai_index_triggers_update(self) -> None:
""" """
GIVEN: GIVEN:
+46 -1
View File
@@ -532,7 +532,29 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m.assert_called_once() m.assert_called_once()
args, kwargs = m.call_args args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id]) self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(len(kwargs), 0) self.assertEqual(kwargs, {"remote_ocr": False})
@mock.patch("documents.views.bulk_edit.reprocess")
def test_reprocess_documents_endpoint_remote_ocr(self, m) -> None:
"""
GIVEN:
- API data to reprocess a document with remote OCR requested
WHEN:
- API is called
THEN:
- reprocess is called with remote_ocr=True
"""
self.setup_mock(m, "reprocess")
response = self.client.post(
"/api/documents/reprocess/",
json.dumps({"documents": [self.doc1.id], "remote_ocr": True}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
@mock.patch("documents.serialisers.bulk_edit.set_storage_path") @mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None: def test_api_set_storage_path(self, m) -> None:
@@ -1553,6 +1575,29 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
), ),
) )
def test_legacy_bulk_edit_reprocess_invalid_remote_ocr(self) -> None:
"""
GIVEN:
- The deprecated bulk_edit endpoint with a non-boolean remote_ocr
WHEN:
- API is called
THEN:
- The request is rejected rather than passed through to the task
"""
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc1.id],
"method": "reprocess",
"parameters": {"remote_ocr": "yes please"},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
@mock.patch("documents.views.bulk_edit.edit_pdf") @mock.patch("documents.views.bulk_edit.edit_pdf")
def test_edit_pdf(self, m) -> None: def test_edit_pdf(self, m) -> None:
self.setup_mock(m, "edit_pdf") self.setup_mock(m, "edit_pdf")
@@ -60,6 +60,10 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
}, },
"email_enabled": False, "email_enabled": False,
"ai_enabled": False, "ai_enabled": False,
"remote_ocr": {
"configured": False,
"mode": "always",
},
}, },
) )
@@ -154,6 +158,50 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
str(response.data["settings"]), str(response.data["settings"]),
) )
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="somekey",
REMOTE_OCR_ENDPOINT="https://example.cognitiveservices.azure.com",
REMOTE_OCR_MODE="workflow_only",
)
def test_settings_reports_remote_ocr_when_configured(self) -> None:
"""
GIVEN:
- A fully configured remote OCR engine in workflow_only mode
WHEN:
- The ui_settings endpoint is called
THEN:
- The UI is told remote OCR is available and selective, so it can
offer it where it would actually change something
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
response.data["settings"]["remote_ocr"],
{"configured": True, "mode": "workflow_only"},
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
def test_settings_reports_remote_ocr_incompletely_configured(self) -> None:
"""
GIVEN:
- An engine named but missing its endpoint and API key
WHEN:
- The ui_settings endpoint is called
THEN:
- It is reported as not configured, matching what the parser
registry will actually do
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertFalse(response.data["settings"]["remote_ocr"]["configured"])
@override_settings( @override_settings(
OAUTH_CALLBACK_BASE_URL="http://localhost:8000", OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
GMAIL_OAUTH_CLIENT_ID="abc123", GMAIL_OAUTH_CLIENT_ID="abc123",
+76
View File
@@ -390,6 +390,82 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(Workflow.objects.count(), 1) self.assertEqual(Workflow.objects.count(), 1)
def test_api_create_remote_ocr_action_requires_consumption_trigger(
self,
) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- No consumption started trigger, so the action could never run
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
existing_count = Workflow.objects.count()
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR too late",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Workflow.objects.count(), existing_count)
def test_api_create_remote_ocr_action_with_consumption_trigger(self) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- A consumption started trigger alongside another trigger type
WHEN:
- API is called
THEN:
- The workflow is created, the action applies to consumption only
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR on consume",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
"filter_filename": "*.pdf",
},
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
def test_api_create_workflow_trigger_action_empty_fields(self) -> None: def test_api_create_workflow_trigger_action_empty_fields(self) -> None:
""" """
GIVEN: GIVEN:
+53
View File
@@ -1782,3 +1782,56 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIn("wrong password", str(exc.exception)) self.assertIn("wrong password", str(exc.exception))
self.assertIn("Error removing password from document", cm.output[0]) self.assertIn("Error removing password from document", cm.output[0])
class TestBulkEditReprocess(DirectoriesMixin, TestCase):
def setUp(self) -> None:
super().setUp()
self.doc = Document.objects.create(
title="test",
checksum="A",
mime_type="application/pdf",
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_defaults_to_local(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that says nothing about remote OCR
WHEN:
- reprocess is called
THEN:
- The task is queued without asking for the remote engine
"""
result = bulk_edit.reprocess([self.doc.id])
self.assertEqual(result, "OK")
mock_task.apply_async.assert_called_once()
_, kwargs = mock_task.apply_async.call_args
self.assertEqual(
kwargs["kwargs"],
{"document_id": self.doc.id, "remote_ocr": False},
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_passes_remote_ocr(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that explicitly asks for remote OCR
WHEN:
- reprocess is called
THEN:
- The request is forwarded to the task for every document
"""
other = Document.objects.create(
title="test2",
checksum="B",
mime_type="application/pdf",
)
bulk_edit.reprocess([self.doc.id, other.id], remote_ocr=True)
self.assertEqual(mock_task.apply_async.call_count, 2)
for call in mock_task.apply_async.call_args_list:
self.assertTrue(call.kwargs["kwargs"]["remote_ocr"])
+80
View File
@@ -1559,6 +1559,72 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
consumer.run_post_consume_script(doc) consumer.run_post_consume_script(doc)
class TestConsumerRemoteOCR(
DirectoriesMixin,
FileSystemAssertsMixin,
GetConsumerMixin,
TestCase,
):
"""
The consumer resolves the remote OCR mode and the per-document request from
workflows into the allow_remote flag it hands to the parser registry.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.consumer.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
self.addCleanup(patcher.stop)
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
src = (
Path(__file__).parent
/ "samples"
/ "documents"
/ "originals"
/ "0000001.pdf"
)
dst = self.dirs.scratch_dir / "sample.pdf"
shutil.copy(src, dst)
with self.get_consumer(dst, overrides=overrides) as consumer:
consumer.run()
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
"""
GIVEN: Remote OCR mode is 'always'.
WHEN: A document is consumed without any workflow asking for it.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A document is consumed and nothing asked for remote OCR.
THEN: The remote parser is excluded.
"""
self.assertFalse(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A workflow set remote_ocr on the metadata overrides.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
)
class TestMetadataOverrides(TestCase): class TestMetadataOverrides(TestCase):
def test_update_skip_asn_if_exists(self) -> None: def test_update_skip_asn_if_exists(self) -> None:
base = DocumentMetadataOverrides() base = DocumentMetadataOverrides()
@@ -1566,6 +1632,20 @@ class TestMetadataOverrides(TestCase):
base.update(incoming) base.update(incoming)
self.assertTrue(base.skip_asn_if_exists) self.assertTrue(base.skip_asn_if_exists)
def test_update_remote_ocr(self) -> None:
base = DocumentMetadataOverrides()
base.update(DocumentMetadataOverrides(remote_ocr=True))
self.assertTrue(base.remote_ocr)
def test_update_remote_ocr_is_not_unset(self) -> None:
"""
A later workflow that says nothing must not undo an earlier one that
asked for remote OCR.
"""
base = DocumentMetadataOverrides(remote_ocr=True)
base.update(DocumentMetadataOverrides())
self.assertTrue(base.remote_ocr)
def test_update_actor_and_version_label(self) -> None: def test_update_actor_and_version_label(self) -> None:
base = DocumentMetadataOverrides( base = DocumentMetadataOverrides(
actor_id=1, actor_id=1,
@@ -6,8 +6,6 @@ from datetime import timedelta
from io import StringIO from io import StringIO
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
from zipfile import ZIP_DEFLATED
from zipfile import ZIP_LZMA
from zipfile import ZipFile from zipfile import ZipFile
import pytest import pytest
@@ -1080,186 +1078,6 @@ class TestExportImport(
skip_checks=True, skip_checks=True,
) )
def test_compression_flags_require_zip(self) -> None:
"""
GIVEN:
- A request to export without --zip
WHEN:
- --zip-compression or --zip-compression-level is passed anyway
THEN:
- A CommandError is raised (the flags are meaningless without --zip)
"""
for args in (
["--zip-compression", "lzma"],
["--zip-compression-level", "5"],
):
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
*args,
skip_checks=True,
)
def test_zip_compression_level_out_of_range_raises(self) -> None:
"""
GIVEN:
- A request to export to a zip file
WHEN:
- --zip-compression-level is outside the chosen method's valid range
THEN:
- A CommandError is raised
"""
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"deflated",
"--zip-compression-level",
"99",
skip_checks=True,
)
def test_zip_compression_level_rejected_for_stored(self) -> None:
"""
GIVEN:
- A request to export to a zip file with --zip-compression stored
WHEN:
- --zip-compression-level is also passed
THEN:
- A CommandError is raised (stored ignores level entirely)
"""
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"stored",
"--zip-compression-level",
"5",
skip_checks=True,
)
def test_zip_compression_level_rejected_for_lzma(self) -> None:
"""
GIVEN:
- A request to export to a zip file with --zip-compression lzma
WHEN:
- --zip-compression-level is also passed
THEN:
- A CommandError is raised (lzma ignores level entirely)
"""
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"lzma",
"--zip-compression-level",
"5",
skip_checks=True,
)
def test_zstd_unavailable_raises_friendly_error(self) -> None:
"""
GIVEN:
- A Python runtime without zstd support (< 3.14)
WHEN:
- --zip-compression zstd is requested
THEN:
- A CommandError naming the Python version requirement is raised
zstd availability is mocked rather than relying on the actual
runtime: on a Python 3.14+ CI leg, ZSTD is not None, so without the
mock this check is skipped and the command falls through into the
real export, which fails on missing document files instead of
raising the expected CommandError.
"""
with (
mock.patch(
"documents.management.commands.document_exporter.ZSTD",
None,
),
mock.patch(
"documents.management.commands.document_exporter.compression_available",
return_value=False,
),
self.assertRaises(CommandError) as e,
):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"zstd",
skip_checks=True,
)
self.assertIn("3.14", str(e.exception))
def test_zip_compression_flag_resolves_to_sink_constant(self) -> None:
"""
GIVEN:
- A request to export to a zip file with --zip-compression lzma
WHEN:
- The export runs
THEN:
- ZipExportSink is constructed with the resolved ZIP_LZMA constant
(whether zipfile actually compresses with the chosen method is
Python's own contract, and ZipExportSink's own tests already
cover the forwarding; what this command owns is resolving the
CLI string to the right constant, so assert that resolution
directly)
"""
with mock.patch(
"documents.management.commands.document_exporter.ZipExportSink",
) as sink_cls:
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"lzma",
skip_checks=True,
)
sink_cls.assert_called_once_with(
mock.ANY,
mock.ANY,
delete=False,
compression=ZIP_LZMA,
compresslevel=None,
)
def test_default_zip_compression_resolves_to_deflate(self) -> None:
"""
GIVEN:
- A request to export to a zip file with no --zip-compression flag
WHEN:
- The export runs
THEN:
- ZipExportSink is constructed with the default ZIP_DEFLATED
constant and compresslevel=None, matching pre-existing behavior
"""
with mock.patch(
"documents.management.commands.document_exporter.ZipExportSink",
) as sink_cls:
call_command(
"document_exporter",
self.target,
"--zip",
skip_checks=True,
)
sink_cls.assert_called_once_with(
mock.ANY,
mock.ANY,
delete=False,
compression=ZIP_DEFLATED,
compresslevel=None,
)
@pytest.mark.management @pytest.mark.management
class TestCryptExportImport( class TestCryptExportImport(
@@ -525,35 +525,6 @@ class TestCommandImport(
self.assertEqual(doc.tags.count(), 1) self.assertEqual(doc.tags.count(), 1)
self.assertEqual(doc.tags.first().name, "batch-flush-tag") self.assertEqual(doc.tags.first().name, "batch-flush-tag")
def test_import_rejects_unreadable_compression(self) -> None:
"""
GIVEN:
- A zip archive with an entry whose compression this Python can't read
WHEN:
- Import is attempted
THEN:
- A CommandError naming the issue is raised, before extraction
"""
import zipfile
from unittest import mock
archive = Path(self.dirs.scratch_dir) / "export.zip"
with zipfile.ZipFile(archive, "w") as zf:
zf.writestr("manifest.json", "[]")
with mock.patch(
"documents.management.commands.document_importer.compress_type_readable",
return_value=False,
):
with self.assertRaises(CommandError) as e:
call_command(
"document_importer",
str(archive),
"--no-progress-bar",
skip_checks=True,
)
self.assertIn("compression", str(e.exception))
@pytest.mark.management @pytest.mark.management
@pytest.mark.django_db @pytest.mark.django_db
@@ -496,6 +496,28 @@ class TestPermittedObjectIdsGenericModels:
expected_hidden=[strangers.pk], expected_hidden=[strangers.pk],
) )
@pytest.mark.parametrize("is_superuser", [False, True])
def test_inactive_user_sees_nothing(self, model, factory, perm, is_superuser):
suffix = f"{model.__name__}_{is_superuser}"
user = User.objects.create_user(
username=f"inactive_{suffix}",
is_active=False,
is_superuser=is_superuser,
)
other = User.objects.create_user(username=f"other_{suffix}")
granted = factory(owner=other)
assign_perm(perm, user, granted)
assert_visible_document_ids(
permitted_object_ids(user, model, perm),
expected_visible=[],
expected_hidden=[
factory(owner=None).pk,
factory(owner=user).pk,
granted.pk,
],
)
def test_unowned_object_visible_to_everyone(self, model, factory, perm): def test_unowned_object_visible_to_everyone(self, model, factory, perm):
user = User.objects.create_user(username=f"user_{model.__name__}") user = User.objects.create_user(username=f"user_{model.__name__}")
unowned = factory(owner=None) unowned = factory(owner=None)
@@ -68,3 +68,44 @@ class TestPermittedObjectsFilter:
visible_ids = set(result.values_list("id", flat=True)) visible_ids = set(result.values_list("id", flat=True))
assert visible_ids == {owned.pk} assert visible_ids == {owned.pk}
assert granted.pk not in visible_ids assert granted.pk not in visible_ids
@pytest.mark.parametrize(
("username", "is_superuser"),
[("inactive", False), ("inactive_super", True)],
)
def test_inactive_user_sees_nothing(self, username: str, *, is_superuser: bool):
user = User.objects.create_user(
username=username,
is_active=False,
is_superuser=is_superuser,
)
TagFactory(owner=None)
TagFactory(owner=user)
granted = TagFactory(owner=User.objects.create_user(username=f"o_{username}"))
assign_perm("view_tag", user, granted)
request = APIRequestFactory().get("/")
request.user = user
result = PermittedObjectsFilter().filter_queryset(
request,
Tag.objects.all(),
_DummyView(),
)
assert result.count() == 0
def test_inactive_user_sees_nothing_with_include_granted_false(self):
user = User.objects.create_user(username="inactive_owner", is_active=False)
TagFactory(owner=user)
TagFactory(owner=None)
request = APIRequestFactory().get("/")
request.user = user
class _OwnerOnlyFilter(PermittedObjectsFilter):
include_granted = False
result = _OwnerOnlyFilter().filter_queryset(
request,
Tag.objects.all(),
_DummyView(),
)
assert result.count() == 0
+39
View File
@@ -287,6 +287,45 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test") self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
"""
Consumption workflows do not run on reprocess, so the remote parser is
used only in 'always' mode or when the caller explicitly asks for it.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.tasks.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = None
self.addCleanup(patcher.stop)
self.doc = Document.objects.create(
title="test",
content="my document",
checksum="wow",
mime_type="application/pdf",
)
def _allow_remote(self, **kwargs) -> bool:
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return call_kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
self.assertTrue(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
self.assertFalse(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
self.assertTrue(self._allow_remote(remote_ocr=True))
class TestAIIndex(DirectoriesMixin, TestCase): class TestAIIndex(DirectoriesMixin, TestCase):
@override_settings( @override_settings(
AI_ENABLED=True, AI_ENABLED=True,
+79
View File
@@ -5360,3 +5360,82 @@ class TestDateWorkflowLocalization(
document = Document.objects.first() document = Document.objects.first()
assert document is not None assert document is not None
assert document.title == expected_title assert document.title == expected_title
class TestRemoteOCRWorkflowAction(DirectoriesMixin, SampleDirMixin, APITestCase):
def _make_workflow(self, trigger_type) -> None:
trigger = WorkflowTrigger.objects.create(type=trigger_type)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
def test_consumption_trigger_requests_remote_ocr(self) -> None:
"""
GIVEN:
- A consumption workflow with a remote OCR action
WHEN:
- A matching document is consumed
THEN:
- The overrides ask for remote OCR, which is what the consumer
reads when choosing a parser
"""
self._make_workflow(WorkflowTrigger.WorkflowTriggerType.CONSUMPTION)
test_file = shutil.copy(
self.SAMPLE_DIR / "simple.pdf",
self.dirs.scratch_dir / "simple.pdf",
)
overrides = DocumentMetadataOverrides()
run_workflows(
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=test_file,
),
overrides=overrides,
)
self.assertTrue(overrides.remote_ocr)
def test_other_trigger_types_are_ignored(self) -> None:
"""
GIVEN:
- A workflow with a remote OCR action that also has a
non-consumption trigger, which is a valid combination
WHEN:
- The non-consumption trigger fires
THEN:
- The action is skipped, since the document has already been
parsed by this point
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
)
updated_trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger, updated_trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
original_filename="sample.pdf",
)
with self.assertLogs("paperless.handlers", level="DEBUG") as cm:
run_workflows(
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
doc,
)
self.assertIn("only applies to consumption triggers", "".join(cm.output))
+7
View File
@@ -236,8 +236,10 @@ from paperless import version
from paperless.celery import app as celery_app from paperless.celery import app as celery_app
from paperless.config import AIConfig from paperless.config import AIConfig
from paperless.config import GeneralConfig from paperless.config import GeneralConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ApplicationConfiguration from paperless.models import ApplicationConfiguration
from paperless.parsers.registry import get_parser_registry from paperless.parsers.registry import get_parser_registry
from paperless.parsers.remote import RemoteEngineConfig
from paperless.serialisers import GroupSerializer from paperless.serialisers import GroupSerializer
from paperless.serialisers import UserSerializer from paperless.serialisers import UserSerializer
from paperless.views import StandardPagination from paperless.views import StandardPagination
@@ -4010,6 +4012,11 @@ class UiSettingsView(GenericAPIView[Any]):
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
ui_settings["remote_ocr"] = {
"configured": RemoteEngineConfig.from_app_config().engine_is_valid(),
"mode": RemoteOCRConfig().remote_ocr_mode,
}
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED: if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
manager = PaperlessMailOAuth2Manager() manager = PaperlessMailOAuth2Manager()
if settings.GMAIL_OAUTH_ENABLED: if settings.GMAIL_OAUTH_ENABLED:
+2 -2
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr "" msgstr ""
"Project-Id-Version: paperless-ngx\n" "Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n" "Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-08-08 14:28+0000\n" "POT-Creation-Date: 2026-08-10 02:25+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n" "PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n" "Last-Translator: \n"
"Language-Team: English\n" "Language-Team: English\n"
@@ -53,7 +53,7 @@ msgstr ""
msgid "Maximum nesting depth exceeded." msgid "Maximum nesting depth exceeded."
msgstr "" msgstr ""
#: documents/filters.py:1073 #: documents/filters.py:1079
msgid "Custom field not found" msgid "Custom field not found"
msgstr "" msgstr ""
+4 -1
View File
@@ -19,7 +19,10 @@ class AutoLoginMiddleware(MiddlewareMixin):
if request.path.startswith("/api/token/") and request.method == "POST": if request.path.startswith("/api/token/") and request.method == "POST":
return None return None
try: try:
request.user = User.objects.get(username=settings.AUTO_LOGIN_USERNAME) request.user = User.objects.get(
username=settings.AUTO_LOGIN_USERNAME,
is_active=True,
)
auth.login( auth.login(
request=request, request=request,
user=request.user, user=request.user,
+17 -3
View File
@@ -339,16 +339,30 @@ def check_deprecated_v2_ocr_env_vars(
@register() @register()
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]: def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
# Import here because checks.py runs before the app registry is ready
from paperless.models import RemoteOCRMode
errors = []
if settings.REMOTE_OCR_ENGINE == "azureai" and not ( if settings.REMOTE_OCR_ENGINE == "azureai" and not (
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
): ):
return [ errors.append(
Error( Error(
"Azure AI remote parser requires endpoint and API key to be configured.", "Azure AI remote parser requires endpoint and API key to be configured.",
), ),
] )
return [] valid_modes = {mode.value for mode in RemoteOCRMode}
if settings.REMOTE_OCR_MODE not in valid_modes:
errors.append(
Error(
f"PAPERLESS_REMOTE_OCR_MODE is set to {settings.REMOTE_OCR_MODE!r}, "
f"expected one of {sorted(valid_modes)}.",
),
)
return errors
def get_tesseract_langs(): def get_tesseract_langs():
+40
View File
@@ -9,6 +9,7 @@ from paperless.models import CleanChoices
from paperless.models import ColorConvertChoices from paperless.models import ColorConvertChoices
from paperless.models import ModeChoices from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices from paperless.models import OutputTypeChoices
from paperless.models import RemoteOCRMode
@dataclasses.dataclass @dataclasses.dataclass
@@ -185,6 +186,45 @@ class GeneralConfig(BaseConfig):
self.app_logo = app_config.app_logo.url if app_config.app_logo else None self.app_logo = app_config.app_logo.url if app_config.app_logo else None
@dataclasses.dataclass
class RemoteOCRConfig(BaseConfig):
"""
Settings for the remote (cloud) OCR parser
"""
remote_ocr_engine: str | None = dataclasses.field(init=False)
remote_ocr_api_key: str | None = dataclasses.field(init=False)
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
def __post_init__(self) -> None:
app_config = self._get_config_instance()
self.remote_ocr_engine = (
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
)
self.remote_ocr_api_key = (
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
)
self.remote_ocr_endpoint = (
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
)
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
settings.REMOTE_OCR_MODE,
)
@property
def remote_ocr_by_default(self) -> bool:
"""
Whether every supported document goes to the remote engine.
When False the remote engine is used only for documents that
explicitly asked for it, i.e. a workflow matched during consumption or
the user ticked the box when reprocessing.
"""
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
@dataclasses.dataclass @dataclasses.dataclass
class AIConfig(BaseConfig): class AIConfig(BaseConfig):
""" """
@@ -0,0 +1,44 @@
# Generated by Django 5.2.16 on 2026-08-10 14:37
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_api_key",
field=models.CharField(
blank=True,
max_length=1024,
null=True,
verbose_name="Sets the remote OCR API key",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_endpoint",
field=models.CharField(
blank=True,
max_length=256,
null=True,
verbose_name="Sets the remote OCR endpoint",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_engine",
field=models.CharField(
blank=True,
choices=[("azureai", "Azure AI Document Intelligence")],
max_length=32,
null=True,
verbose_name="Sets the remote OCR engine",
),
),
]
@@ -0,0 +1,27 @@
# Generated by Django 5.2.16 on 2026-08-10 15:43
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_mode",
field=models.CharField(
blank=True,
choices=[
("always", "All supported documents"),
("workflow_only", "Only when a workflow enables it"),
],
max_length=32,
null=True,
verbose_name="Sets which documents are sent to the remote OCR engine",
),
),
]
+55
View File
@@ -74,6 +74,23 @@ class ColorConvertChoices(models.TextChoices):
CMYK = ("CMYK", _("CMYK")) CMYK = ("CMYK", _("CMYK"))
class RemoteOCREngine(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_ENGINE
"""
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
class RemoteOCRMode(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_MODE
"""
ALWAYS = ("always", _("All supported documents"))
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
class LLMEmbeddingBackend(models.TextChoices): class LLMEmbeddingBackend(models.TextChoices):
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible")) OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
HUGGINGFACE = ("huggingface", _("Huggingface")) HUGGINGFACE = ("huggingface", _("Huggingface"))
@@ -286,6 +303,44 @@ class ApplicationConfiguration(AbstractSingletonModel):
null=True, null=True,
) )
"""
Settings for the remote OCR parser
"""
# PAPERLESS_REMOTE_OCR_ENGINE
remote_ocr_engine = models.CharField(
verbose_name=_("Sets the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCREngine.choices,
)
# PAPERLESS_REMOTE_OCR_API_KEY
remote_ocr_api_key = models.CharField(
verbose_name=_("Sets the remote OCR API key"),
blank=True,
null=True,
max_length=1024,
)
# PAPERLESS_REMOTE_OCR_ENDPOINT
remote_ocr_endpoint = models.CharField(
verbose_name=_("Sets the remote OCR endpoint"),
blank=True,
null=True,
max_length=256,
)
# PAPERLESS_REMOTE_OCR_MODE
remote_ocr_mode = models.CharField(
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCRMode.choices,
)
""" """
AI related settings AI related settings
""" """
+9
View File
@@ -134,6 +134,11 @@ class ParserProtocol(Protocol):
Author or organisation name. Author or organisation name.
url : str url : str
URL for documentation, source code, or issue tracker. URL for documentation, source code, or issue tracker.
Parsers that send document content to a remote service should additionally
set ``uses_remote_service = True`` so the registry can exclude them when
remote processing has not been requested for a document. The attribute is
optional so a parser that omits it is treated as fully local.
""" """
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@@ -145,6 +150,10 @@ class ParserProtocol(Protocol):
author: str author: str
url: str url: str
# NOTE: uses_remote_service is not declared here, the registry reads it
# with getattr(cls, ..., False) for backwards-compatibility with existing
# parsers
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Class methods # Class methods
# ------------------------------------------------------------------ # ------------------------------------------------------------------
+14
View File
@@ -334,6 +334,8 @@ class ParserRegistry:
mime_type: str, mime_type: str,
filename: str, filename: str,
path: Path | None = None, path: Path | None = None,
*,
allow_remote: bool = True,
) -> type[ParserProtocol] | None: ) -> type[ParserProtocol] | None:
"""Return the best parser class for the given file, or None. """Return the best parser class for the given file, or None.
@@ -359,6 +361,11 @@ class ParserRegistry:
path: path:
Optional filesystem path to the file. Forwarded to each Optional filesystem path to the file. Forwarded to each
parser's score method. parser's score method.
allow_remote:
When False, parsers that declare ``uses_remote_service = True``
are excluded from consideration, so a document is never sent to
a remote service. Parsers that do not declare the attribute
are treated as local and are always considered.
Returns Returns
------- -------
@@ -374,6 +381,13 @@ class ParserRegistry:
if mime_type not in parser_class.supported_mime_types(): if mime_type not in parser_class.supported_mime_types():
continue continue
if not allow_remote and getattr(
parser_class,
"uses_remote_service",
False,
):
continue
score = parser_class.score(mime_type, filename, path) score = parser_class.score(mime_type, filename, path)
if score is None: if score is None:
continue continue
+19 -10
View File
@@ -57,6 +57,18 @@ class RemoteEngineConfig:
self.api_key = api_key self.api_key = api_key
self.endpoint = endpoint self.endpoint = endpoint
@classmethod
def from_app_config(cls) -> Self:
"""Build the config from the app config, falling back to the env."""
from paperless.config import RemoteOCRConfig
app_config = RemoteOCRConfig()
return cls(
engine=app_config.remote_ocr_engine,
api_key=app_config.remote_ocr_api_key,
endpoint=app_config.remote_ocr_endpoint,
)
def engine_is_valid(self) -> bool: def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured.""" """Return True when the engine is known and fully configured."""
return ( return (
@@ -83,6 +95,9 @@ class RemoteDocumentParser:
Maintainer name. Maintainer name.
url : str url : str
Issue tracker / source URL. Issue tracker / source URL.
uses_remote_service : bool
Content is sent to a remote service, True so that the registry
can skip this parser if remote processing was not requested.
""" """
name: str = "Paperless-ngx Remote OCR Parser" name: str = "Paperless-ngx Remote OCR Parser"
@@ -90,6 +105,8 @@ class RemoteDocumentParser:
author: str = "Paperless-ngx Contributors" author: str = "Paperless-ngx Contributors"
url: str = "https://github.com/paperless-ngx/paperless-ngx" url: str = "https://github.com/paperless-ngx/paperless-ngx"
uses_remote_service: bool = True
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Class methods # Class methods
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@@ -138,11 +155,7 @@ class RemoteDocumentParser:
20 when the remote engine is configured and the MIME type is 20 when the remote engine is configured and the MIME type is
supported, otherwise None. supported, otherwise None.
""" """
config = RemoteEngineConfig( config = RemoteEngineConfig.from_app_config()
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid(): if not config.engine_is_valid():
return None return None
if mime_type not in _SUPPORTED_MIME_TYPES: if mime_type not in _SUPPORTED_MIME_TYPES:
@@ -228,11 +241,7 @@ class RemoteDocumentParser:
Ignored the remote engine always returns a searchable PDF, Ignored the remote engine always returns a searchable PDF,
which is stored as the archive copy regardless of this flag. which is stored as the archive copy regardless of this flag.
""" """
config = RemoteEngineConfig( config = RemoteEngineConfig.from_app_config()
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid(): if not config.engine_is_valid():
logger.warning( logger.warning(
+14 -5
View File
@@ -219,6 +219,13 @@ class ApplicationConfigurationSerializer(
allow_null=True, allow_null=True,
max_length=1024, max_length=1024,
) )
remote_ocr_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
max_length=1024,
)
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
def run_validation(self, data): def run_validation(self, data):
# Empty strings treated as None to avoid unexpected behavior # Empty strings treated as None to avoid unexpected behavior
@@ -230,11 +237,13 @@ class ApplicationConfigurationSerializer(
data["language"] = None data["language"] = None
if "llm_output_language" in data and data["llm_output_language"] == "": if "llm_output_language" in data and data["llm_output_language"] == "":
data["llm_output_language"] = None data["llm_output_language"] = None
if "llm_api_key" in data and data["llm_api_key"] is not None: for field in self.OBFUSCATED_FIELDS:
if data["llm_api_key"] == "": if field in data and data[field] is not None:
data["llm_api_key"] = None if data[field] == "":
elif len(data["llm_api_key"].replace("*", "")) == 0: data[field] = None
del data["llm_api_key"] # Not a real value, don't overwrite the stored one
elif len(data[field].replace("*", "")) == 0:
del data[field]
return super().run_validation(data) return super().run_validation(data)
def update(self, instance, validated_data): def update(self, instance, validated_data):
+1
View File
@@ -1197,6 +1197,7 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE") REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY") REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT") REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
REMOTE_OCR_MODE = os.getenv("PAPERLESS_REMOTE_OCR_MODE", "always")
################################################################################ ################################################################################
# AI Settings # # AI Settings #
@@ -21,6 +21,7 @@ from unittest.mock import Mock
import pytest import pytest
from documents.parsers import ParseError from documents.parsers import ParseError
from paperless.models import ApplicationConfiguration
from paperless.parsers import ParserContext from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol from paperless.parsers import ParserProtocol
from paperless.parsers.remote import RemoteDocumentParser from paperless.parsers.remote import RemoteDocumentParser
@@ -33,6 +34,10 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture from pytest_mock import MockerFixture
# Remote ocr config from ApplicationConfiguration needs DB access
pytestmark = pytest.mark.django_db
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Module-local fixtures # Module-local fixtures
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -227,6 +232,18 @@ class TestRemoteParserScore:
score = RemoteDocumentParser.score("application/pdf", "doc.pdf") score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert score is not None and score > 10 assert score is not None and score > 10
@pytest.mark.usefixtures("no_engine_settings")
def test_score_uses_app_config_when_env_unset(self) -> None:
"""The app config alone is enough to activate the parser."""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_engine = "azureai"
config.remote_ocr_api_key = "app-config-key"
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
config.save()
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Properties # Properties
@@ -1277,6 +1277,8 @@ class TestParserFileTypes:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Remote ocr config from ApplicationConfiguration needs DB access
@pytest.mark.django_db
class TestRasterisedDocumentParserRegistry: class TestRasterisedDocumentParserRegistry:
def test_registered_in_defaults(self) -> None: def test_registered_in_defaults(self) -> None:
from paperless.parsers.registry import ParserRegistry from paperless.parsers.registry import ParserRegistry
@@ -0,0 +1,53 @@
from django.contrib.auth.models import AnonymousUser
from django.contrib.auth.models import User
from django.test import RequestFactory
from django.test import TestCase
from django.test import override_settings
from paperless.auth import AutoLoginMiddleware
@override_settings(AUTO_LOGIN_USERNAME="autologin")
class TestAutoLoginMiddleware(TestCase):
def setUp(self) -> None:
super().setUp()
self.factory = RequestFactory()
self.middleware = AutoLoginMiddleware(lambda request: None)
def _process(self, request):
# login() needs a session to write to
request.session = self.client.session
self.middleware.process_request(request)
return request
def test_active_user_is_logged_in(self) -> None:
"""
GIVEN:
- AUTO_LOGIN_USERNAME names an active user
WHEN:
- A request is processed by the middleware
THEN:
- That user is attached to the request
"""
user = User.objects.create_user(username="autologin")
request = self._process(self.factory.get("/"))
self.assertEqual(request.user, user)
def test_deactivated_user_is_not_logged_in(self) -> None:
"""
GIVEN:
- AUTO_LOGIN_USERNAME names a user who has been deactivated
WHEN:
- A request is processed by the middleware
THEN:
- The request is left anonymous rather than authenticated as them
"""
User.objects.create_user(username="autologin", is_active=False)
request = self.factory.get("/")
request.user = AnonymousUser()
self._process(request)
self.assertFalse(request.user.is_authenticated)
+17
View File
@@ -655,6 +655,23 @@ class TestRemoteParserChecks:
in msg.msg in msg.msg
) )
def test_valid_mode(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
settings.REMOTE_OCR_MODE = "workflow_only"
msgs = check_remote_parser_configured(None)
assert len(msgs) == 0
def test_invalid_mode(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
settings.REMOTE_OCR_MODE = "sometimes"
msgs = check_remote_parser_configured(None)
assert len(msgs) == 1
assert "PAPERLESS_REMOTE_OCR_MODE is set to 'sometimes'" in msgs[0].msg
class TestTesseractChecks: class TestTesseractChecks:
def test_default_language(self) -> None: def test_default_language(self) -> None:
+118
View File
@@ -468,6 +468,124 @@ class TestParserRegistryGetParserForFile:
assert result is AcceptingBuiltin assert result is AcceptingBuiltin
class TestParserRegistryRemoteParsers:
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
@staticmethod
def _remote_parser_cls() -> type:
class RemoteParser:
name = "remote"
version = "1.0"
author = "A"
url = "https://example.com/remote"
uses_remote_service = True
@classmethod
def supported_mime_types(cls):
return {"text/plain": ".txt"}
@classmethod
def score(cls, mime_type, filename, path=None):
return 20
return RemoteParser
def test_remote_parser_wins_when_remote_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=True.
THEN: The remote parser is returned.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=True,
)
assert result is remote_parser_cls
def test_remote_parser_skipped_when_remote_not_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: The local parser is returned despite its lower score.
"""
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
"""
GIVEN: A registry whose only candidate declares uses_remote_service.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: None is returned the remote parser is never used as a
fallback when remote processing was not requested.
"""
registry = ParserRegistry()
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is None
def test_parser_without_attribute_treated_as_local(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A third-party parser predating uses_remote_service, so it does
not declare the attribute at all.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: It is still considered, i.e. treated as fully local, rather
than raising AttributeError.
"""
assert not hasattr(dummy_parser_cls, "uses_remote_service")
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_remote_allowed_by_default(self) -> None:
"""
GIVEN: A registry containing only a remote parser.
WHEN: get_parser_for_file() is called without allow_remote.
THEN: The remote parser is returned callers that do not opt in to
the filter keep the previous behaviour.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file("text/plain", "readme.txt")
assert result is remote_parser_cls
class TestDiscover: class TestDiscover:
"""Verify entrypoint discovery in ParserRegistry.discover().""" """Verify entrypoint discovery in ParserRegistry.discover()."""
@@ -0,0 +1,113 @@
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from django.test import override_settings
from paperless.config import RemoteOCRConfig
from paperless.models import RemoteOCRMode
if TYPE_CHECKING:
from unittest.mock import MagicMock
@pytest.fixture()
def null_app_config(mocker) -> MagicMock:
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
return mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
@pytest.fixture()
def make_remote_ocr_config(mocker):
def _make(app_config, **django_settings_overrides):
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
with override_settings(**django_settings_overrides):
return RemoteOCRConfig()
return _make
class TestRemoteOCRConfig:
def test_falls_back_to_settings(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "env-key"
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_app_config_takes_precedence(
self,
make_remote_ocr_config,
mocker,
) -> None:
app_config = mocker.MagicMock(
remote_ocr_engine="azureai",
remote_ocr_api_key="db-key",
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
)
cfg = make_remote_ocr_config(
app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "db-key"
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_unset_everywhere(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
assert cfg.remote_ocr_engine is None
assert cfg.remote_ocr_api_key is None
assert cfg.remote_ocr_endpoint is None
class TestRemoteOCRByDefault:
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_by_default is True
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_by_default is False
Generated
+22 -22
View File
@@ -1298,16 +1298,16 @@ wheels = [
[[package]] [[package]]
name = "fpdf2" name = "fpdf2"
version = "2.8.7" version = "2.8.8"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/27/f2/72feae0b2827ed38013e4307b14f95bf0b3d124adfef4d38a7d57533f7be/fpdf2-2.8.7.tar.gz", hash = "sha256:7060ccee5a9c7ab0a271fb765a36a23639f83ef8996c34e3d46af0a17ede57f9", size = 362351, upload-time = "2026-02-28T05:39:16.456Z" } sdist = { url = "https://files.pythonhosted.org/packages/1e/bc/8fd4321aed40cadadddc8f311c65b6082346b252bca048f7b476d8f35d72/fpdf2-2.8.8.tar.gz", hash = "sha256:9e94e155e85e8053329a9a1fce8b566fd7a7c5bb79e98a1a3952d379b947c5b9", size = 374689, upload-time = "2026-08-09T23:32:45.334Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/66/0a/cf50ecffa1e3747ed9380a3adfc829259f1f86b3fdbd9e505af789003141/fpdf2-2.8.7-py3-none-any.whl", hash = "sha256:d391fc508a3ce02fc43a577c830cda4fe6f37646f2d143d489839940932fbc19", size = 327056, upload-time = "2026-02-28T05:39:14.619Z" }, { url = "https://files.pythonhosted.org/packages/f5/be/af012eda9507494f28b99b077423806c43a11573eb6225dd46f19ae2d263/fpdf2-2.8.8-py3-none-any.whl", hash = "sha256:3557a478fc577a929c94aace9666aed4dcc432b5ab6764232e6a59f1ccd75f17", size = 337000, upload-time = "2026-08-09T23:32:43.728Z" },
] ]
[[package]] [[package]]
@@ -2927,8 +2927,8 @@ dependencies = [
{ name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" }, { name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" }, { name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -4511,8 +4511,8 @@ dependencies = [
{ name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" }, { name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" }, { name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -4957,18 +4957,17 @@ name = "torch"
version = "2.13.0" version = "2.13.0"
source = { registry = "https://download.pytorch.org/whl/cpu" } source = { registry = "https://download.pytorch.org/whl/cpu" }
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and sys_platform == 'darwin'",
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'", "python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
"python_full_version < '3.12' and sys_platform == 'darwin'", "python_full_version < '3.12' and sys_platform == 'darwin'",
] ]
dependencies = [ dependencies = [
{ name = "filelock", marker = "sys_platform == 'darwin'" }, { name = "filelock", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "fsspec", marker = "sys_platform == 'darwin'" }, { name = "fsspec", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "jinja2", marker = "sys_platform == 'darwin'" }, { name = "jinja2", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "networkx", marker = "sys_platform == 'darwin'" }, { name = "networkx", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "setuptools", marker = "sys_platform == 'darwin'" }, { name = "setuptools", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "sympy", marker = "sys_platform == 'darwin'" }, { name = "sympy", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
{ name = "typing-extensions", marker = "sys_platform == 'darwin'" }, { name = "typing-extensions", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
] ]
wheels = [ wheels = [
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" }, { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" },
@@ -4983,6 +4982,7 @@ name = "torch"
version = "2.13.0+cpu" version = "2.13.0+cpu"
source = { registry = "https://download.pytorch.org/whl/cpu" } source = { registry = "https://download.pytorch.org/whl/cpu" }
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and sys_platform == 'darwin'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'", "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'", "python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"python_full_version >= '3.15' and sys_platform == 'linux'", "python_full_version >= '3.15' and sys_platform == 'linux'",
@@ -4990,13 +4990,13 @@ resolution-markers = [
"python_full_version < '3.12' and sys_platform == 'linux'", "python_full_version < '3.12' and sys_platform == 'linux'",
] ]
dependencies = [ dependencies = [
{ name = "filelock", marker = "sys_platform == 'linux'" }, { name = "filelock", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "fsspec", marker = "sys_platform == 'linux'" }, { name = "fsspec", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "jinja2", marker = "sys_platform == 'linux'" }, { name = "jinja2", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "networkx", marker = "sys_platform == 'linux'" }, { name = "networkx", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "setuptools", marker = "sys_platform == 'linux'" }, { name = "setuptools", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "sympy", marker = "sys_platform == 'linux'" }, { name = "sympy", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
{ name = "typing-extensions", marker = "sys_platform == 'linux'" }, { name = "typing-extensions", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
] ]
wheels = [ wheels = [
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" }, { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" },