mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-11 13:23:21 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cdda191f91 |
@@ -299,6 +299,8 @@ optional arguments:
|
|||||||
-sm, --split-manifest
|
-sm, --split-manifest
|
||||||
-z, --zip
|
-z, --zip
|
||||||
-zn, --zip-name
|
-zn, --zip-name
|
||||||
|
--zip-compression
|
||||||
|
--zip-compression-level
|
||||||
--data-only
|
--data-only
|
||||||
--no-progress-bar
|
--no-progress-bar
|
||||||
--passphrase
|
--passphrase
|
||||||
@@ -361,6 +363,19 @@ If `-z` or `--zip` is provided, the export will be a zip file
|
|||||||
in the target directory, named according to the current local date or the
|
in the target directory, named according to the current local date or the
|
||||||
value set in `-zn` or `--zip-name`.
|
value set in `-zn` or `--zip-name`.
|
||||||
|
|
||||||
|
The compression method for the zip can be set with `--zip-compression`
|
||||||
|
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
|
||||||
|
`--zip-compression-level` (deflated: 0–9, bzip2: 1–9, zstd: -22–22; ignored
|
||||||
|
for `stored` and `lzma`). Both options require `--zip`.
|
||||||
|
|
||||||
|
!!! warning
|
||||||
|
|
||||||
|
`zstd` compression requires Python 3.14 or newer on **both** the machine
|
||||||
|
creating the export and any machine importing it. An archive compressed with
|
||||||
|
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
|
||||||
|
imported on a runtime that lacks the codec; the importer will refuse it with
|
||||||
|
a clear error. The default `deflated` is universally readable.
|
||||||
|
|
||||||
If `--data-only` is provided, only the database will be exported. This option is intended
|
If `--data-only` is provided, only the database will be exported. This option is intended
|
||||||
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
|
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
|
||||||
|
|
||||||
|
|||||||
+1
-2
@@ -301,8 +301,7 @@ The following methods are supported:
|
|||||||
- `delete`
|
- `delete`
|
||||||
- No `parameters` required
|
- No `parameters` required
|
||||||
- `reprocess`
|
- `reprocess`
|
||||||
- Optional `parameters`: `{ "remote_ocr": true }` to send the documents to the
|
- No `parameters` required
|
||||||
remote OCR engine, see [Remote OCR](usage.md#remote-ocr). Defaults to false.
|
|
||||||
- `set_permissions`
|
- `set_permissions`
|
||||||
- Requires `parameters`:
|
- Requires `parameters`:
|
||||||
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
|
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
|
||||||
|
|||||||
@@ -2047,18 +2047,6 @@ password. All of these options come from their similarly-named [Django settings]
|
|||||||
|
|
||||||
Defaults to None.
|
Defaults to None.
|
||||||
|
|
||||||
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
|
|
||||||
|
|
||||||
: Which documents are sent to the remote OCR engine.
|
|
||||||
|
|
||||||
- `always`: every document of a supported file type is sent to the remote
|
|
||||||
engine, bypassing the local OCR engine.
|
|
||||||
- `workflow_only`: documents are processed locally unless a workflow
|
|
||||||
explicitly enables remote OCR for them, letting you use the remote engine
|
|
||||||
selectively.
|
|
||||||
|
|
||||||
Defaults to "always".
|
|
||||||
|
|
||||||
## AI {#ai}
|
## AI {#ai}
|
||||||
|
|
||||||
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
|
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
|
||||||
|
|||||||
@@ -456,20 +456,6 @@ def score(
|
|||||||
return 10
|
return 10
|
||||||
```
|
```
|
||||||
|
|
||||||
**Remote services**
|
|
||||||
|
|
||||||
If your parser sends document content to a remote service, declare it:
|
|
||||||
|
|
||||||
```python
|
|
||||||
class MyCustomParser:
|
|
||||||
uses_remote_service = True
|
|
||||||
```
|
|
||||||
|
|
||||||
Paperless-ngx excludes such parsers when the document being consumed has not
|
|
||||||
been marked for remote processing, so users can keep remote OCR off by default
|
|
||||||
and enable it selectively with a workflow. Parsers that do not declare the
|
|
||||||
attribute are treated as fully local and are always considered.
|
|
||||||
|
|
||||||
**Archive and rendition flags**
|
**Archive and rendition flags**
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|||||||
+1
-22
@@ -648,19 +648,6 @@ happened while it was still encrypted, that original version will likewise be mi
|
|||||||
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
|
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
|
||||||
multiple PDF types with different passwords, create separate workflows for each use case.
|
multiple PDF types with different passwords, create separate workflows for each use case.
|
||||||
|
|
||||||
##### Remote OCR {#workflow-action-remote-ocr}
|
|
||||||
|
|
||||||
"Remote OCR" actions send the document to the configured remote OCR engine instead of processing it
|
|
||||||
locally. To use remote OCR selectively, set the [remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE)
|
|
||||||
to `workflow_only` then add this action to a workflow that matches only the documents you
|
|
||||||
want sent to the remote engine. See [Remote OCR](#remote-ocr) for the engine setup. The action only works with
|
|
||||||
a **Consumption Started** trigger.
|
|
||||||
|
|
||||||
The action takes no options, its presence is what enables remote OCR for a matching document.
|
|
||||||
|
|
||||||
If the remote engine is not configured, or does not support the document's file type, the document is
|
|
||||||
processed locally instead and a warning is written to the log.
|
|
||||||
|
|
||||||
#### Workflow placeholders
|
#### Workflow placeholders
|
||||||
|
|
||||||
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
|
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
|
||||||
@@ -1097,19 +1084,11 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
|
|||||||
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
|
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
|
||||||
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
|
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
|
||||||
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
|
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
|
||||||
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
|
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
|
||||||
settings can be supplied as environment variables or via **Application Configuration**.
|
|
||||||
|
|
||||||
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
|
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
|
||||||
or page limitations (e.g. with a free tier).
|
or page limitations (e.g. with a free tier).
|
||||||
|
|
||||||
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
|
|
||||||
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
|
|
||||||
unless a [remote OCR workflow action](#workflow-action-remote-ocr) enables it for them, so you can limit the remote
|
|
||||||
engine to particular documents.
|
|
||||||
|
|
||||||
Setting the mode to `workflow_only` also allows the **Reprocess** actions to selectively use remote OCR for individual documents.
|
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
Paperless-ngx consists of the following components:
|
Paperless-ngx consists of the following components:
|
||||||
|
|||||||
+1
-1
@@ -66,5 +66,5 @@
|
|||||||
"ts-node": "~10.9.1",
|
"ts-node": "~10.9.1",
|
||||||
"typescript": "^6.0.3"
|
"typescript": "^6.0.3"
|
||||||
},
|
},
|
||||||
"packageManager": "pnpm@11.15.1"
|
"packageManager": "pnpm@10.26.0"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ trustPolicy: no-downgrade
|
|||||||
trustPolicyExclude:
|
trustPolicyExclude:
|
||||||
- "chokidar@4.0.3"
|
- "chokidar@4.0.3"
|
||||||
- "semver@6.3.1 || 5.7.2"
|
- "semver@6.3.1 || 5.7.2"
|
||||||
blockExoticSubdeps: true
|
|
||||||
allowBuilds:
|
allowBuilds:
|
||||||
"@parcel/watcher": true
|
"@parcel/watcher": true
|
||||||
canvas: true
|
canvas: true
|
||||||
|
|||||||
@@ -14,12 +14,8 @@
|
|||||||
<a ngbNavLink>{{category}}</a>
|
<a ngbNavLink>{{category}}</a>
|
||||||
<ng-template ngbNavContent>
|
<ng-template ngbNavContent>
|
||||||
<div class="p-3">
|
<div class="p-3">
|
||||||
@for (section of getCategorySections(category); track section) {
|
|
||||||
@if (section) {
|
|
||||||
<h5 class="mt-4 mb-3">{{section}}</h5>
|
|
||||||
}
|
|
||||||
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
|
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
|
||||||
@for (option of getCategoryOptions(category, section); track option.key) {
|
@for (option of getCategoryOptions(category); track option.key) {
|
||||||
<div class="col">
|
<div class="col">
|
||||||
<div class="card bg-light">
|
<div class="card bg-light">
|
||||||
<div class="card-body">
|
<div class="card-body">
|
||||||
@@ -55,7 +51,6 @@
|
|||||||
</div>
|
</div>
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
}
|
|
||||||
</div>
|
</div>
|
||||||
</ng-template>
|
</ng-template>
|
||||||
</li>
|
</li>
|
||||||
|
|||||||
@@ -8,11 +8,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
|
|||||||
import { NgSelectModule } from '@ng-select/ng-select'
|
import { NgSelectModule } from '@ng-select/ng-select'
|
||||||
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
||||||
import { of, throwError } from 'rxjs'
|
import { of, throwError } from 'rxjs'
|
||||||
import {
|
import { OutputTypeConfig } from 'src/app/data/paperless-config'
|
||||||
ConfigCategory,
|
|
||||||
ConfigSection,
|
|
||||||
OutputTypeConfig,
|
|
||||||
} from 'src/app/data/paperless-config'
|
|
||||||
import { ConfigService } from 'src/app/services/config.service'
|
import { ConfigService } from 'src/app/services/config.service'
|
||||||
import { SettingsService } from 'src/app/services/settings.service'
|
import { SettingsService } from 'src/app/services/settings.service'
|
||||||
import { ToastService } from 'src/app/services/toast.service'
|
import { ToastService } from 'src/app/services/toast.service'
|
||||||
@@ -162,24 +158,4 @@ describe('ConfigComponent', () => {
|
|||||||
component.resetOption('barcodes_enabled')
|
component.resetOption('barcodes_enabled')
|
||||||
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
|
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should group options into sections within a category, or not', () => {
|
|
||||||
const sections = component.getCategorySections(ConfigCategory.OCR)
|
|
||||||
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
|
|
||||||
expect(
|
|
||||||
component
|
|
||||||
.getCategoryOptions(ConfigCategory.OCR)
|
|
||||||
.map((option) => option.key)
|
|
||||||
).toContain('output_type')
|
|
||||||
expect(
|
|
||||||
component
|
|
||||||
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
|
|
||||||
.map((option) => option.key)
|
|
||||||
).toEqual([
|
|
||||||
'remote_ocr_engine',
|
|
||||||
'remote_ocr_api_key',
|
|
||||||
'remote_ocr_endpoint',
|
|
||||||
'remote_ocr_mode',
|
|
||||||
])
|
|
||||||
})
|
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -74,20 +74,8 @@ export class ConfigComponent
|
|||||||
return Object.values(ConfigCategory)
|
return Object.values(ConfigCategory)
|
||||||
}
|
}
|
||||||
|
|
||||||
getCategorySections(category: string): string[] {
|
getCategoryOptions(category: string): ConfigOption[] {
|
||||||
return [
|
return PaperlessConfigOptions.filter((o) => o.category === category)
|
||||||
...new Set(
|
|
||||||
PaperlessConfigOptions.filter((o) => o.category === category).map(
|
|
||||||
(o) => o.section ?? null // null means no section
|
|
||||||
)
|
|
||||||
),
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
|
|
||||||
return PaperlessConfigOptions.filter(
|
|
||||||
(o) => o.category === category && (o.section ?? null) === section
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
initialConfig: PaperlessConfig
|
initialConfig: PaperlessConfig
|
||||||
|
|||||||
-28
@@ -1,28 +0,0 @@
|
|||||||
<div class="modal-header">
|
|
||||||
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
|
|
||||||
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
<div class="modal-body">
|
|
||||||
@if (messageBold) {
|
|
||||||
<p class="text-break"><b>{{messageBold}}</b></p>
|
|
||||||
}
|
|
||||||
@if (message) {
|
|
||||||
<p class="mb-0 text-break" [innerHTML]="message"></p>
|
|
||||||
}
|
|
||||||
@if (showRemoteOcr) {
|
|
||||||
<div class="form-check mt-3">
|
|
||||||
<input class="form-check-input" type="checkbox" id="reprocessRemoteOcr" [(ngModel)]="remoteOcr" />
|
|
||||||
<label class="form-check-label" for="reprocessRemoteOcr" i18n>Use remote OCR</label>
|
|
||||||
<div class="form-text" i18n>Sends the document to the configured remote OCR service, which may incur costs.</div>
|
|
||||||
</div>
|
|
||||||
}
|
|
||||||
</div>
|
|
||||||
<div class="modal-footer">
|
|
||||||
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
|
|
||||||
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
|
|
||||||
</button>
|
|
||||||
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
|
|
||||||
{{btnCaption}}
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
-72
@@ -1,72 +0,0 @@
|
|||||||
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
|
|
||||||
import { provideHttpClientTesting } from '@angular/common/http/testing'
|
|
||||||
import { ComponentFixture, TestBed } from '@angular/core/testing'
|
|
||||||
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
|
|
||||||
import { RemoteOCRModeConfig } from 'src/app/data/paperless-config'
|
|
||||||
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
|
|
||||||
import { SettingsService } from 'src/app/services/settings.service'
|
|
||||||
import { ReprocessConfirmDialogComponent } from './reprocess-confirm-dialog.component'
|
|
||||||
|
|
||||||
describe('ReprocessConfirmDialogComponent', () => {
|
|
||||||
let component: ReprocessConfirmDialogComponent
|
|
||||||
let fixture: ComponentFixture<ReprocessConfirmDialogComponent>
|
|
||||||
let settingsService: SettingsService
|
|
||||||
|
|
||||||
const createComponent = (configured: boolean, mode: string) => {
|
|
||||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, configured)
|
|
||||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_MODE, mode)
|
|
||||||
|
|
||||||
fixture = TestBed.createComponent(ReprocessConfirmDialogComponent)
|
|
||||||
component = fixture.componentInstance
|
|
||||||
fixture.detectChanges()
|
|
||||||
}
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
TestBed.configureTestingModule({
|
|
||||||
providers: [
|
|
||||||
NgbActiveModal,
|
|
||||||
provideHttpClient(withInterceptorsFromDi()),
|
|
||||||
provideHttpClientTesting(),
|
|
||||||
],
|
|
||||||
imports: [ReprocessConfirmDialogComponent],
|
|
||||||
}).compileComponents()
|
|
||||||
|
|
||||||
settingsService = TestBed.inject(SettingsService)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not request remote OCR by default', () => {
|
|
||||||
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
|
|
||||||
|
|
||||||
expect(component.remoteOcr).toBeFalsy()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not offer remote OCR when no engine is configured', () => {
|
|
||||||
createComponent(false, RemoteOCRModeConfig.WORKFLOW_ONLY)
|
|
||||||
|
|
||||||
expect(component.showRemoteOcr).toBeFalsy()
|
|
||||||
expect(
|
|
||||||
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
|
|
||||||
).toBeNull()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not offer remote OCR when it already handles every document', () => {
|
|
||||||
createComponent(true, RemoteOCRModeConfig.ALWAYS)
|
|
||||||
|
|
||||||
expect(component.showRemoteOcr).toBeFalsy()
|
|
||||||
expect(
|
|
||||||
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
|
|
||||||
).toBeNull()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should offer remote OCR when configured and selective', () => {
|
|
||||||
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
|
|
||||||
|
|
||||||
expect(component.showRemoteOcr).toBeTruthy()
|
|
||||||
const checkbox = fixture.nativeElement.querySelector('#reprocessRemoteOcr')
|
|
||||||
expect(checkbox).not.toBeNull()
|
|
||||||
|
|
||||||
checkbox.click()
|
|
||||||
fixture.detectChanges()
|
|
||||||
expect(component.remoteOcr).toBeTruthy()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
-20
@@ -1,20 +0,0 @@
|
|||||||
import { Component, inject } from '@angular/core'
|
|
||||||
import { FormsModule } from '@angular/forms'
|
|
||||||
import { SettingsService } from 'src/app/services/settings.service'
|
|
||||||
import { ConfirmDialogComponent } from '../confirm-dialog.component'
|
|
||||||
|
|
||||||
@Component({
|
|
||||||
selector: 'pngx-reprocess-confirm-dialog',
|
|
||||||
templateUrl: './reprocess-confirm-dialog.component.html',
|
|
||||||
imports: [FormsModule],
|
|
||||||
})
|
|
||||||
export class ReprocessConfirmDialogComponent extends ConfirmDialogComponent {
|
|
||||||
private settings = inject(SettingsService)
|
|
||||||
|
|
||||||
remoteOcr: boolean = false
|
|
||||||
|
|
||||||
public get showRemoteOcr(): boolean {
|
|
||||||
// Hidden when it is not configured, or when it already handles every document anyway.
|
|
||||||
return this.settings.remoteOCRIsSelectable
|
|
||||||
}
|
|
||||||
}
|
|
||||||
-7
@@ -455,13 +455,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
}
|
}
|
||||||
@case (WorkflowActionType.RemoteOcr) {
|
|
||||||
<div class="row">
|
|
||||||
<div class="col">
|
|
||||||
<p class="text-muted small" i18n>The document will be sent to the configured remote OCR service. May incur costs.</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
</ng-template>
|
</ng-template>
|
||||||
|
|||||||
+2
-105
@@ -29,7 +29,6 @@ import {
|
|||||||
DocumentSource,
|
DocumentSource,
|
||||||
WorkflowTriggerType,
|
WorkflowTriggerType,
|
||||||
} from 'src/app/data/workflow-trigger'
|
} from 'src/app/data/workflow-trigger'
|
||||||
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
|
|
||||||
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
|
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
|
||||||
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
|
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
|
||||||
import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
|
import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
|
||||||
@@ -225,12 +224,7 @@ describe('WorkflowEditDialogComponent', () => {
|
|||||||
).toEqual('Document Added')
|
).toEqual('Document Added')
|
||||||
expect(component.getTriggerTypeOptionName(null)).toEqual('')
|
expect(component.getTriggerTypeOptionName(null)).toEqual('')
|
||||||
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
|
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
|
||||||
// Remote OCR is absent until the workflow has a consumption trigger
|
expect(component.actionTypeOptions).toEqual(WORKFLOW_ACTION_OPTIONS)
|
||||||
expect(component.actionTypeOptions).toEqual(
|
|
||||||
WORKFLOW_ACTION_OPTIONS.filter(
|
|
||||||
(a) => a.id !== WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
)
|
|
||||||
expect(
|
expect(
|
||||||
component.getActionTypeOptionName(WorkflowActionType.Assignment)
|
component.getActionTypeOptionName(WorkflowActionType.Assignment)
|
||||||
).toEqual('Assignment')
|
).toEqual('Assignment')
|
||||||
@@ -243,104 +237,7 @@ describe('WorkflowEditDialogComponent', () => {
|
|||||||
jest.spyOn(settingsService, 'get').mockReturnValue(false)
|
jest.spyOn(settingsService, 'get').mockReturnValue(false)
|
||||||
component.ngOnInit()
|
component.ngOnInit()
|
||||||
expect(component.actionTypeOptions).toEqual(
|
expect(component.actionTypeOptions).toEqual(
|
||||||
WORKFLOW_ACTION_OPTIONS.filter(
|
WORKFLOW_ACTION_OPTIONS.filter((a) => a.id !== WorkflowActionType.Email)
|
||||||
(a) =>
|
|
||||||
a.id !== WorkflowActionType.Email &&
|
|
||||||
a.id !== WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should offer remote OCR only for consumption workflows', () => {
|
|
||||||
jest.spyOn(settingsService, 'get').mockReturnValue(true)
|
|
||||||
|
|
||||||
// A consumption trigger makes the action reachable
|
|
||||||
component.object = {
|
|
||||||
name: 'Workflow 1',
|
|
||||||
order: 0,
|
|
||||||
enabled: true,
|
|
||||||
triggers: [{ type: WorkflowTriggerType.Consumption }],
|
|
||||||
actions: [],
|
|
||||||
} as Workflow
|
|
||||||
component.ngOnInit()
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
|
|
||||||
// Any other trigger type runs after the document has been parsed
|
|
||||||
component.object = {
|
|
||||||
name: 'Workflow 2',
|
|
||||||
order: 0,
|
|
||||||
enabled: true,
|
|
||||||
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
|
|
||||||
actions: [],
|
|
||||||
} as Workflow
|
|
||||||
component.ngOnInit()
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should offer remote OCR on a trigger added to a new workflow', () => {
|
|
||||||
jest.spyOn(settingsService, 'get').mockReturnValue(true)
|
|
||||||
component.ngOnInit()
|
|
||||||
|
|
||||||
// Nothing for the action to apply to yet
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
|
|
||||||
// addTrigger creates the form field with emitEvent false, so the options
|
|
||||||
// have to be computed on read rather than cached from valueChanges
|
|
||||||
component.addTrigger()
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
|
|
||||||
// Switching that trigger to a type that runs after parsing removes it
|
|
||||||
component.triggerFields
|
|
||||||
.at(0)
|
|
||||||
.get('type')
|
|
||||||
.setValue(WorkflowTriggerType.DocumentAdded)
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should keep remote OCR listed when an action already uses it', () => {
|
|
||||||
jest.spyOn(settingsService, 'get').mockReturnValue(true)
|
|
||||||
|
|
||||||
// Otherwise changing the trigger would silently blank the selection
|
|
||||||
component.object = {
|
|
||||||
name: 'Workflow 1',
|
|
||||||
order: 0,
|
|
||||||
enabled: true,
|
|
||||||
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
|
|
||||||
actions: [{ type: WorkflowActionType.RemoteOcr }],
|
|
||||||
} as Workflow
|
|
||||||
component.ngOnInit()
|
|
||||||
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should not offer remote OCR when no engine is configured', () => {
|
|
||||||
jest
|
|
||||||
.spyOn(settingsService, 'get')
|
|
||||||
.mockImplementation((key) => key !== SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
|
|
||||||
|
|
||||||
component.object = {
|
|
||||||
name: 'Workflow 1',
|
|
||||||
order: 0,
|
|
||||||
enabled: true,
|
|
||||||
triggers: [{ type: WorkflowTriggerType.Consumption }],
|
|
||||||
actions: [],
|
|
||||||
} as Workflow
|
|
||||||
component.ngOnInit()
|
|
||||||
|
|
||||||
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
|
|
||||||
WorkflowActionType.RemoteOcr
|
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
+10
-40
@@ -148,10 +148,6 @@ export const WORKFLOW_ACTION_OPTIONS = [
|
|||||||
id: WorkflowActionType.MoveToTrash,
|
id: WorkflowActionType.MoveToTrash,
|
||||||
name: $localize`Move to trash`,
|
name: $localize`Move to trash`,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
id: WorkflowActionType.RemoteOcr,
|
|
||||||
name: $localize`Remote OCR`,
|
|
||||||
},
|
|
||||||
]
|
]
|
||||||
|
|
||||||
export enum TriggerFilterType {
|
export enum TriggerFilterType {
|
||||||
@@ -508,6 +504,8 @@ export class WorkflowEditDialogComponent
|
|||||||
|
|
||||||
expandedItem: number = null
|
expandedItem: number = null
|
||||||
|
|
||||||
|
readonly allowedActionTypes = signal([])
|
||||||
|
|
||||||
private readonly triggerFilterOptionsMap = new WeakMap<
|
private readonly triggerFilterOptionsMap = new WeakMap<
|
||||||
FormArray,
|
FormArray,
|
||||||
TriggerFilterOption[]
|
TriggerFilterOption[]
|
||||||
@@ -550,40 +548,13 @@ export class WorkflowEditDialogComponent
|
|||||||
this.checkRemovalActionFields.bind(this)
|
this.checkRemovalActionFields.bind(this)
|
||||||
)
|
)
|
||||||
this.checkRemovalActionFields(this.objectForm.value)
|
this.checkRemovalActionFields(this.objectForm.value)
|
||||||
}
|
this.allowedActionTypes.set(
|
||||||
|
this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)
|
||||||
private allowedActionTypes: typeof WORKFLOW_ACTION_OPTIONS = null
|
? WORKFLOW_ACTION_OPTIONS
|
||||||
|
: WORKFLOW_ACTION_OPTIONS.filter(
|
||||||
private getAllowedActionTypes() {
|
(a) => a.id !== WorkflowActionType.Email
|
||||||
let allowed = WORKFLOW_ACTION_OPTIONS
|
)
|
||||||
|
)
|
||||||
if (!this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)) {
|
|
||||||
allowed = allowed.filter((a) => a.id !== WorkflowActionType.Email)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Remote OCR is decided before the document is parsed, so it is only
|
|
||||||
// offered for workflows that run at consumption.
|
|
||||||
const formWorkflow: Workflow = this.objectForm?.value
|
|
||||||
const remoteOcrUsable =
|
|
||||||
this.settingsService.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
|
|
||||||
(formWorkflow?.triggers?.some(
|
|
||||||
(trigger) => trigger.type === WorkflowTriggerType.Consumption
|
|
||||||
) ||
|
|
||||||
formWorkflow?.actions?.some(
|
|
||||||
(action) => action.type === WorkflowActionType.RemoteOcr
|
|
||||||
))
|
|
||||||
if (!remoteOcrUsable) {
|
|
||||||
allowed = allowed.filter((a) => a.id !== WorkflowActionType.RemoteOcr)
|
|
||||||
}
|
|
||||||
|
|
||||||
if (
|
|
||||||
this.allowedActionTypes?.length === allowed.length &&
|
|
||||||
this.allowedActionTypes.every((a, i) => a.id === allowed[i].id)
|
|
||||||
) {
|
|
||||||
return this.allowedActionTypes
|
|
||||||
}
|
|
||||||
this.allowedActionTypes = allowed
|
|
||||||
return allowed
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private checkRemovalActionFields(formWorkflow: Workflow) {
|
private checkRemovalActionFields(formWorkflow: Workflow) {
|
||||||
@@ -1308,8 +1279,7 @@ export class WorkflowEditDialogComponent
|
|||||||
|
|
||||||
get actionTypeOptions() {
|
get actionTypeOptions() {
|
||||||
this.settingsService.trackChanges()
|
this.settingsService.trackChanges()
|
||||||
// Computed on read rather than cached
|
return this.allowedActionTypes()
|
||||||
return this.getAllowedActionTypes()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
getActionTypeOptionName(type: WorkflowActionType): string {
|
getActionTypeOptionName(type: WorkflowActionType): string {
|
||||||
|
|||||||
@@ -963,24 +963,12 @@ describe('DocumentDetailComponent', () => {
|
|||||||
component.reprocess()
|
component.reprocess()
|
||||||
const modalCloseSpy = jest.spyOn(openModal, 'close')
|
const modalCloseSpy = jest.spyOn(openModal, 'close')
|
||||||
openModal.componentInstance.confirmClicked.next()
|
openModal.componentInstance.confirmClicked.next()
|
||||||
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, false)
|
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] })
|
||||||
expect(modalSpy).toHaveBeenCalled()
|
expect(modalSpy).toHaveBeenCalled()
|
||||||
expect(toastSpy).toHaveBeenCalled()
|
expect(toastSpy).toHaveBeenCalled()
|
||||||
expect(modalCloseSpy).toHaveBeenCalled()
|
expect(modalCloseSpy).toHaveBeenCalled()
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should pass remote OCR choice when reprocessing', () => {
|
|
||||||
initNormally()
|
|
||||||
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
|
|
||||||
reprocessSpy.mockReturnValue(of(true))
|
|
||||||
let openModal: NgbModalRef
|
|
||||||
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
|
||||||
component.reprocess()
|
|
||||||
openModal.componentInstance.remoteOcr = true
|
|
||||||
openModal.componentInstance.confirmClicked.next()
|
|
||||||
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should show error if redo ocr call fails', () => {
|
it('should show error if redo ocr call fails', () => {
|
||||||
initNormally()
|
initNormally()
|
||||||
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
|
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
|
||||||
|
|||||||
@@ -97,7 +97,6 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
|
|||||||
import * as UTIF from 'utif'
|
import * as UTIF from 'utif'
|
||||||
import { DocumentDetailFieldID } from '../admin/settings/settings.component'
|
import { DocumentDetailFieldID } from '../admin/settings/settings.component'
|
||||||
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
|
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
|
||||||
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
|
|
||||||
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
|
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
|
||||||
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
|
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
|
||||||
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
|
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
|
||||||
@@ -1403,7 +1402,7 @@ export class DocumentDetailComponent
|
|||||||
}
|
}
|
||||||
|
|
||||||
reprocess() {
|
reprocess() {
|
||||||
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
|
let modal = this.modalService.open(ConfirmDialogComponent, {
|
||||||
backdrop: 'static',
|
backdrop: 'static',
|
||||||
})
|
})
|
||||||
modal.componentInstance.title = $localize`Reprocess confirm`
|
modal.componentInstance.title = $localize`Reprocess confirm`
|
||||||
@@ -1414,10 +1413,7 @@ export class DocumentDetailComponent
|
|||||||
modal.componentInstance.confirmClicked.subscribe(() => {
|
modal.componentInstance.confirmClicked.subscribe(() => {
|
||||||
modal.componentInstance.buttonsEnabled = false
|
modal.componentInstance.buttonsEnabled = false
|
||||||
this.documentsService
|
this.documentsService
|
||||||
.reprocessDocuments(
|
.reprocessDocuments({ documents: [this.document().id] })
|
||||||
{ documents: [this.document().id] },
|
|
||||||
modal.componentInstance.remoteOcr
|
|
||||||
)
|
|
||||||
.subscribe({
|
.subscribe({
|
||||||
next: () => {
|
next: () => {
|
||||||
this.toastService.showInfo(
|
this.toastService.showInfo(
|
||||||
|
|||||||
@@ -1122,7 +1122,6 @@ describe('BulkEditorComponent', () => {
|
|||||||
req.flush(true)
|
req.flush(true)
|
||||||
expect(req.request.body).toEqual({
|
expect(req.request.body).toEqual({
|
||||||
documents: [3, 4],
|
documents: [3, 4],
|
||||||
remote_ocr: false,
|
|
||||||
})
|
})
|
||||||
httpTestingController.match(
|
httpTestingController.match(
|
||||||
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
|
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
|
||||||
|
|||||||
@@ -51,7 +51,6 @@ import { ToastService } from 'src/app/services/toast.service'
|
|||||||
import { flattenTags } from 'src/app/utils/flatten-tags'
|
import { flattenTags } from 'src/app/utils/flatten-tags'
|
||||||
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
|
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
|
||||||
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
|
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
|
||||||
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
|
|
||||||
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
|
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
|
||||||
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
|
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
|
||||||
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
|
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
|
||||||
@@ -910,7 +909,7 @@ export class BulkEditorComponent
|
|||||||
}
|
}
|
||||||
|
|
||||||
reprocessSelected() {
|
reprocessSelected() {
|
||||||
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
|
let modal = this.modalService.open(ConfirmDialogComponent, {
|
||||||
backdrop: 'static',
|
backdrop: 'static',
|
||||||
})
|
})
|
||||||
modal.componentInstance.title = $localize`Reprocess confirm`
|
modal.componentInstance.title = $localize`Reprocess confirm`
|
||||||
@@ -924,10 +923,7 @@ export class BulkEditorComponent
|
|||||||
modal.componentInstance.buttonsEnabled = false
|
modal.componentInstance.buttonsEnabled = false
|
||||||
this.executeDocumentAction(
|
this.executeDocumentAction(
|
||||||
modal,
|
modal,
|
||||||
this.documentService.reprocessDocuments(
|
this.documentService.reprocessDocuments(this.getSelectionQuery())
|
||||||
this.getSelectionQuery(),
|
|
||||||
modal.componentInstance.remoteOcr
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -54,10 +54,6 @@ export const ConfigCategory = {
|
|||||||
AI: $localize`AI Settings`,
|
AI: $localize`AI Settings`,
|
||||||
}
|
}
|
||||||
|
|
||||||
export const ConfigSection = {
|
|
||||||
RemoteOCR: $localize`Remote OCR`,
|
|
||||||
}
|
|
||||||
|
|
||||||
export const LLMEmbeddingBackendConfig = {
|
export const LLMEmbeddingBackendConfig = {
|
||||||
OPENAI_LIKE: 'openai-like',
|
OPENAI_LIKE: 'openai-like',
|
||||||
HUGGINGFACE: 'huggingface',
|
HUGGINGFACE: 'huggingface',
|
||||||
@@ -69,15 +65,6 @@ export const LLMBackendConfig = {
|
|||||||
OLLAMA: 'ollama',
|
OLLAMA: 'ollama',
|
||||||
}
|
}
|
||||||
|
|
||||||
export const RemoteOCREngineConfig = {
|
|
||||||
AZURE_AI: 'azureai',
|
|
||||||
}
|
|
||||||
|
|
||||||
export const RemoteOCRModeConfig = {
|
|
||||||
ALWAYS: 'always',
|
|
||||||
WORKFLOW_ONLY: 'workflow_only',
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface ConfigOption {
|
export interface ConfigOption {
|
||||||
key: string
|
key: string
|
||||||
title: string
|
title: string
|
||||||
@@ -85,7 +72,6 @@ export interface ConfigOption {
|
|||||||
choices?: Array<{ id: string; name: string }>
|
choices?: Array<{ id: string; name: string }>
|
||||||
config_key?: string
|
config_key?: string
|
||||||
category: string
|
category: string
|
||||||
section?: string
|
|
||||||
note?: string
|
note?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -195,43 +181,6 @@ export const PaperlessConfigOptions: ConfigOption[] = [
|
|||||||
config_key: 'PAPERLESS_OCR_USER_ARGS',
|
config_key: 'PAPERLESS_OCR_USER_ARGS',
|
||||||
category: ConfigCategory.OCR,
|
category: ConfigCategory.OCR,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
key: 'remote_ocr_engine',
|
|
||||||
title: $localize`Remote OCR Engine`,
|
|
||||||
type: ConfigOptionType.Select,
|
|
||||||
choices: mapToItems(RemoteOCREngineConfig),
|
|
||||||
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
|
|
||||||
category: ConfigCategory.OCR,
|
|
||||||
section: ConfigSection.RemoteOCR,
|
|
||||||
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
key: 'remote_ocr_api_key',
|
|
||||||
title: $localize`Remote OCR API Key`,
|
|
||||||
type: ConfigOptionType.Password,
|
|
||||||
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
|
|
||||||
category: ConfigCategory.OCR,
|
|
||||||
section: ConfigSection.RemoteOCR,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
key: 'remote_ocr_endpoint',
|
|
||||||
title: $localize`Remote OCR Endpoint`,
|
|
||||||
type: ConfigOptionType.String,
|
|
||||||
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
|
|
||||||
category: ConfigCategory.OCR,
|
|
||||||
section: ConfigSection.RemoteOCR,
|
|
||||||
note: $localize`Required when using the Azure AI engine.`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
key: 'remote_ocr_mode',
|
|
||||||
title: $localize`Remote OCR Mode`,
|
|
||||||
type: ConfigOptionType.Select,
|
|
||||||
choices: mapToItems(RemoteOCRModeConfig),
|
|
||||||
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
|
|
||||||
category: ConfigCategory.OCR,
|
|
||||||
section: ConfigSection.RemoteOCR,
|
|
||||||
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
key: 'app_logo',
|
key: 'app_logo',
|
||||||
title: $localize`Application Logo`,
|
title: $localize`Application Logo`,
|
||||||
@@ -449,10 +398,6 @@ export interface PaperlessConfig extends ObjectWithId {
|
|||||||
barcode_enable_tag: boolean
|
barcode_enable_tag: boolean
|
||||||
barcode_tag_mapping: object
|
barcode_tag_mapping: object
|
||||||
barcode_tag_split: boolean
|
barcode_tag_split: boolean
|
||||||
remote_ocr_engine: string
|
|
||||||
remote_ocr_api_key: string
|
|
||||||
remote_ocr_endpoint: string
|
|
||||||
remote_ocr_mode: string
|
|
||||||
ai_enabled: boolean
|
ai_enabled: boolean
|
||||||
llm_embedding_backend: string
|
llm_embedding_backend: string
|
||||||
llm_embedding_model: string
|
llm_embedding_model: string
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
|
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
|
||||||
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
|
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
|
||||||
import { RemoteOCRModeConfig } from './paperless-config'
|
|
||||||
import { User } from './user'
|
import { User } from './user'
|
||||||
|
|
||||||
export interface UiSettings {
|
export interface UiSettings {
|
||||||
@@ -95,8 +94,6 @@ export const SETTINGS_KEYS = {
|
|||||||
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
|
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
|
||||||
EMAIL_ENABLED: 'email_enabled',
|
EMAIL_ENABLED: 'email_enabled',
|
||||||
AI_ENABLED: 'ai_enabled',
|
AI_ENABLED: 'ai_enabled',
|
||||||
REMOTE_OCR_CONFIGURED: 'remote_ocr:configured',
|
|
||||||
REMOTE_OCR_MODE: 'remote_ocr:mode',
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export const SETTINGS: UiSetting[] = [
|
export const SETTINGS: UiSetting[] = [
|
||||||
@@ -350,14 +347,4 @@ export const SETTINGS: UiSetting[] = [
|
|||||||
type: 'string',
|
type: 'string',
|
||||||
default: PdfEditorEditMode.Create,
|
default: PdfEditorEditMode.Create,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
key: SETTINGS_KEYS.REMOTE_OCR_CONFIGURED,
|
|
||||||
type: 'boolean',
|
|
||||||
default: false,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
key: SETTINGS_KEYS.REMOTE_OCR_MODE,
|
|
||||||
type: 'string',
|
|
||||||
default: RemoteOCRModeConfig.ALWAYS,
|
|
||||||
},
|
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ export enum WorkflowActionType {
|
|||||||
Webhook = 4,
|
Webhook = 4,
|
||||||
PasswordRemoval = 5,
|
PasswordRemoval = 5,
|
||||||
MoveToTrash = 6,
|
MoveToTrash = 6,
|
||||||
RemoteOcr = 7,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface WorkflowActionEmail extends ObjectWithId {
|
export interface WorkflowActionEmail extends ObjectWithId {
|
||||||
|
|||||||
@@ -284,21 +284,6 @@ describe(`DocumentService`, () => {
|
|||||||
expect(req.request.method).toEqual('POST')
|
expect(req.request.method).toEqual('POST')
|
||||||
expect(req.request.body).toEqual({
|
expect(req.request.body).toEqual({
|
||||||
documents: ids,
|
documents: ids,
|
||||||
remote_ocr: false,
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('should request remote OCR when reprocessing with it enabled', () => {
|
|
||||||
const ids = [1, 2, 3]
|
|
||||||
subscription = service
|
|
||||||
.reprocessDocuments({ documents: ids }, true)
|
|
||||||
.subscribe()
|
|
||||||
const req = httpTestingController.expectOne(
|
|
||||||
`${environment.apiBaseUrl}${endpoint}/reprocess/`
|
|
||||||
)
|
|
||||||
expect(req.request.body).toEqual({
|
|
||||||
documents: ids,
|
|
||||||
remote_ocr: true,
|
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
@@ -349,13 +349,9 @@ export class DocumentService extends AbstractPaperlessService<Document> {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
reprocessDocuments(
|
reprocessDocuments(selection: DocumentSelectionQuery) {
|
||||||
selection: DocumentSelectionQuery,
|
|
||||||
remoteOcr: boolean = false
|
|
||||||
) {
|
|
||||||
return this.http.post(this.getResourceUrl(null, 'reprocess'), {
|
return this.http.post(this.getResourceUrl(null, 'reprocess'), {
|
||||||
...selection,
|
...selection,
|
||||||
remote_ocr: remoteOcr,
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ import { environment } from 'src/environments/environment'
|
|||||||
import { CustomFieldDataType } from '../data/custom-field'
|
import { CustomFieldDataType } from '../data/custom-field'
|
||||||
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
||||||
import { SavedView } from '../data/saved-view'
|
import { SavedView } from '../data/saved-view'
|
||||||
import { RemoteOCRModeConfig } from '../data/paperless-config'
|
|
||||||
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
|
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
|
||||||
import { PermissionsService } from './permissions.service'
|
import { PermissionsService } from './permissions.service'
|
||||||
import { CustomFieldsService } from './rest/custom-fields.service'
|
import { CustomFieldsService } from './rest/custom-fields.service'
|
||||||
@@ -435,26 +434,4 @@ describe('SettingsService', () => {
|
|||||||
).name
|
).name
|
||||||
).toEqual(customFields[0].name)
|
).toEqual(customFields[0].name)
|
||||||
})
|
})
|
||||||
it('should offer remote OCR only when configured and selective', () => {
|
|
||||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, false)
|
|
||||||
settingsService.set(
|
|
||||||
SETTINGS_KEYS.REMOTE_OCR_MODE,
|
|
||||||
RemoteOCRModeConfig.WORKFLOW_ONLY
|
|
||||||
)
|
|
||||||
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
|
|
||||||
|
|
||||||
// configured, but already handling every document
|
|
||||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, true)
|
|
||||||
settingsService.set(
|
|
||||||
SETTINGS_KEYS.REMOTE_OCR_MODE,
|
|
||||||
RemoteOCRModeConfig.ALWAYS
|
|
||||||
)
|
|
||||||
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
|
|
||||||
|
|
||||||
settingsService.set(
|
|
||||||
SETTINGS_KEYS.REMOTE_OCR_MODE,
|
|
||||||
RemoteOCRModeConfig.WORKFLOW_ONLY
|
|
||||||
)
|
|
||||||
expect(settingsService.remoteOCRIsSelectable).toBeTruthy()
|
|
||||||
})
|
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -19,7 +19,6 @@ import {
|
|||||||
} from 'src/app/utils/color'
|
} from 'src/app/utils/color'
|
||||||
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
|
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
|
||||||
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
||||||
import { RemoteOCRModeConfig } from '../data/paperless-config'
|
|
||||||
import { SavedView } from '../data/saved-view'
|
import { SavedView } from '../data/saved-view'
|
||||||
import {
|
import {
|
||||||
PAPERLESS_GREEN_HEX,
|
PAPERLESS_GREEN_HEX,
|
||||||
@@ -688,17 +687,6 @@ export class SettingsService {
|
|||||||
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
|
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Offering remote OCR as a choice only makes sense when an engine
|
|
||||||
* is configured but is not already handling every document.
|
|
||||||
*/
|
|
||||||
get remoteOCRIsSelectable(): boolean {
|
|
||||||
return (
|
|
||||||
this.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
|
|
||||||
this.get(SETTINGS_KEYS.REMOTE_OCR_MODE) !== RemoteOCRModeConfig.ALWAYS
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
offerTour(): boolean {
|
offerTour(): boolean {
|
||||||
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
|
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -394,16 +394,10 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
|
|||||||
return "OK"
|
return "OK"
|
||||||
|
|
||||||
|
|
||||||
def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
|
def reprocess(doc_ids: list[int]) -> Literal["OK"]:
|
||||||
"""
|
|
||||||
Re-run parsing for the given documents.
|
|
||||||
|
|
||||||
Consumption workflows do not run here, so ``remote_ocr`` is how the user
|
|
||||||
asks for the remote engine when it is not configured to handle everything.
|
|
||||||
"""
|
|
||||||
for document_id in doc_ids:
|
for document_id in doc_ids:
|
||||||
update_document_content_maybe_archive_file.apply_async(
|
update_document_content_maybe_archive_file.apply_async(
|
||||||
kwargs={"document_id": document_id, "remote_ocr": remote_ocr},
|
kwargs={"document_id": document_id},
|
||||||
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
|
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -53,7 +53,6 @@ from documents.utils import copy_basic_file_stats
|
|||||||
from documents.utils import copy_file_with_basic_stats
|
from documents.utils import copy_file_with_basic_stats
|
||||||
from documents.utils import run_subprocess
|
from documents.utils import run_subprocess
|
||||||
from paperless.config import OcrConfig
|
from paperless.config import OcrConfig
|
||||||
from paperless.config import RemoteOCRConfig
|
|
||||||
from paperless.models import ArchiveFileGenerationChoices
|
from paperless.models import ArchiveFileGenerationChoices
|
||||||
from paperless.parsers import ParserContext
|
from paperless.parsers import ParserContext
|
||||||
from paperless.parsers import ParserProtocol
|
from paperless.parsers import ParserProtocol
|
||||||
@@ -452,19 +451,12 @@ class ConsumerPlugin(
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.log.error(f"Error attempting to clean PDF: {e}")
|
self.log.error(f"Error attempting to clean PDF: {e}")
|
||||||
|
|
||||||
# Workflows have already run at this point, so the metadata knows
|
|
||||||
# whether this document was singled out for remote OCR
|
|
||||||
allow_remote = (
|
|
||||||
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
|
|
||||||
)
|
|
||||||
|
|
||||||
# Based on the mime type, get the parser for that type
|
# Based on the mime type, get the parser for that type
|
||||||
parser_class: type[ParserProtocol] | None = (
|
parser_class: type[ParserProtocol] | None = (
|
||||||
get_parser_registry().get_parser_for_file(
|
get_parser_registry().get_parser_for_file(
|
||||||
mime_type,
|
mime_type,
|
||||||
self.filename,
|
self.filename,
|
||||||
self.working_copy,
|
self.working_copy,
|
||||||
allow_remote=allow_remote,
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
if not parser_class:
|
if not parser_class:
|
||||||
@@ -473,16 +465,6 @@ class ConsumerPlugin(
|
|||||||
f"Unsupported mime type {mime_type}",
|
f"Unsupported mime type {mime_type}",
|
||||||
)
|
)
|
||||||
|
|
||||||
if self.metadata.remote_ocr and not getattr(
|
|
||||||
parser_class,
|
|
||||||
"uses_remote_service",
|
|
||||||
False,
|
|
||||||
):
|
|
||||||
self.log.warning(
|
|
||||||
"Remote OCR was requested for this document but no remote "
|
|
||||||
"parser is available for it, processing locally instead.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Notify all listeners that we're going to do some work.
|
# Notify all listeners that we're going to do some work.
|
||||||
|
|
||||||
document_consumption_started.send(
|
document_consumption_started.send(
|
||||||
|
|||||||
@@ -34,7 +34,6 @@ class DocumentMetadataOverrides:
|
|||||||
skip_asn_if_exists: bool = False
|
skip_asn_if_exists: bool = False
|
||||||
version_label: str | None = None
|
version_label: str | None = None
|
||||||
actor_id: int | None = None
|
actor_id: int | None = None
|
||||||
remote_ocr: bool = False
|
|
||||||
|
|
||||||
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
|
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
|
||||||
"""
|
"""
|
||||||
@@ -58,8 +57,6 @@ class DocumentMetadataOverrides:
|
|||||||
self.actor_id = other.actor_id
|
self.actor_id = other.actor_id
|
||||||
if other.skip_asn_if_exists:
|
if other.skip_asn_if_exists:
|
||||||
self.skip_asn_if_exists = True
|
self.skip_asn_if_exists = True
|
||||||
if other.remote_ocr:
|
|
||||||
self.remote_ocr = True
|
|
||||||
if other.version_label is not None:
|
if other.version_label is not None:
|
||||||
self.version_label = other.version_label
|
self.version_label = other.version_label
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,108 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
|
||||||
|
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
|
||||||
|
|
||||||
|
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
|
||||||
|
# availability is enforced separately in compression_available().
|
||||||
|
COMPRESSION_CHOICES: tuple[str, ...] = (
|
||||||
|
"stored",
|
||||||
|
"deflated",
|
||||||
|
"bzip2",
|
||||||
|
"lzma",
|
||||||
|
"zstd",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Method name -> zipfile compression constant (zstd only when supported).
|
||||||
|
COMPRESSION_METHODS: dict[str, int] = {
|
||||||
|
"stored": zipfile.ZIP_STORED,
|
||||||
|
"deflated": zipfile.ZIP_DEFLATED,
|
||||||
|
"bzip2": zipfile.ZIP_BZIP2,
|
||||||
|
"lzma": zipfile.ZIP_LZMA,
|
||||||
|
}
|
||||||
|
if ZSTD is not None:
|
||||||
|
COMPRESSION_METHODS["zstd"] = ZSTD
|
||||||
|
|
||||||
|
# Inclusive (min, max) level bounds per method; None => level not applicable.
|
||||||
|
# Verified on CPython 3.14.3.
|
||||||
|
#
|
||||||
|
# zstd's raw library bounds are (-131072, 22)
|
||||||
|
# (compression.zstd.CompressionParameter.compression_level.bounds()) — the
|
||||||
|
# minimum is an internal implementation constant (-ZSTD_TARGETLENGTH_MAX),
|
||||||
|
# not a meaningful distinct "level"; deeper negative values than -22 buy
|
||||||
|
# nothing over -22 in practice. We expose the conventional zstd CLI range
|
||||||
|
# instead of the raw library bounds.
|
||||||
|
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
|
||||||
|
"stored": None,
|
||||||
|
"deflated": (0, 9),
|
||||||
|
"bzip2": (1, 9),
|
||||||
|
"lzma": None,
|
||||||
|
"zstd": (-22, 22),
|
||||||
|
}
|
||||||
|
|
||||||
|
# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
|
||||||
|
# zstd id that zipfile can still read.
|
||||||
|
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
|
||||||
|
zipfile.ZIP_STORED: "stored",
|
||||||
|
zipfile.ZIP_DEFLATED: "deflated",
|
||||||
|
zipfile.ZIP_BZIP2: "bzip2",
|
||||||
|
zipfile.ZIP_LZMA: "lzma",
|
||||||
|
93: "zstd",
|
||||||
|
20: "zstd",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def compression_available(method: str) -> bool:
|
||||||
|
"""Whether the running interpreter can actually use the given method."""
|
||||||
|
if method in ("stored", "deflated"):
|
||||||
|
# zlib is a hard CPython dependency; stored needs nothing.
|
||||||
|
return True
|
||||||
|
if method == "bzip2":
|
||||||
|
return _module_importable("bz2")
|
||||||
|
if method == "lzma":
|
||||||
|
return _module_importable("lzma")
|
||||||
|
if method == "zstd":
|
||||||
|
return ZSTD is not None and _module_importable("compression.zstd")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _module_importable(name: str) -> bool:
|
||||||
|
try:
|
||||||
|
importlib.import_module(name)
|
||||||
|
except ImportError:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def level_error(method: str, level: int | None) -> str | None:
|
||||||
|
"""Return a human message if (method, level) is invalid, else None."""
|
||||||
|
if level is None:
|
||||||
|
return None
|
||||||
|
bounds = LEVEL_BOUNDS[method]
|
||||||
|
if bounds is None:
|
||||||
|
return f"--zip-compression-level has no effect for '{method}'"
|
||||||
|
low, high = bounds
|
||||||
|
if not (low <= level <= high):
|
||||||
|
return (
|
||||||
|
f"--zip-compression-level for '{method}' must be between {low} and {high}"
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def compress_type_readable(compress_type: int) -> bool:
|
||||||
|
"""Whether this interpreter can decompress an entry of the given type."""
|
||||||
|
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
|
||||||
|
if method is None:
|
||||||
|
return False
|
||||||
|
return compression_available(method)
|
||||||
|
|
||||||
|
|
||||||
|
def unreadable_method_names(compress_types: set[int]) -> set[str]:
|
||||||
|
"""Map a set of compress_type ids to human method names for error messages."""
|
||||||
|
names: set[str] = set()
|
||||||
|
for ct in compress_types:
|
||||||
|
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
|
||||||
|
return names
|
||||||
@@ -243,11 +243,21 @@ class ZipExportSink(ExportSink):
|
|||||||
added as an entry at finalize (a zip entry cannot be interleaved with others).
|
added as an entry at finalize (a zip entry cannot be interleaved with others).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, target: Path, zip_name: str, *, delete: bool = False) -> None:
|
def __init__(
|
||||||
|
self,
|
||||||
|
target: Path,
|
||||||
|
zip_name: str,
|
||||||
|
*,
|
||||||
|
delete: bool = False,
|
||||||
|
compression: int = zipfile.ZIP_DEFLATED,
|
||||||
|
compresslevel: int | None = None,
|
||||||
|
) -> None:
|
||||||
self._target = target.resolve()
|
self._target = target.resolve()
|
||||||
self._zip_path = (self._target / zip_name).with_suffix(".zip")
|
self._zip_path = (self._target / zip_name).with_suffix(".zip")
|
||||||
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
|
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
|
||||||
self._delete = delete
|
self._delete = delete
|
||||||
|
self._compression = compression
|
||||||
|
self._compresslevel = compresslevel
|
||||||
self._zip: zipfile.ZipFile | None = None
|
self._zip: zipfile.ZipFile | None = None
|
||||||
self._dirs: set[str] = set()
|
self._dirs: set[str] = set()
|
||||||
self._pending_manifest: tuple[Path, str] | None = None
|
self._pending_manifest: tuple[Path, str] | None = None
|
||||||
@@ -258,7 +268,8 @@ class ZipExportSink(ExportSink):
|
|||||||
self._zip = zipfile.ZipFile(
|
self._zip = zipfile.ZipFile(
|
||||||
self._tmp_path,
|
self._tmp_path,
|
||||||
"w",
|
"w",
|
||||||
compression=zipfile.ZIP_DEFLATED,
|
compression=self._compression,
|
||||||
|
compresslevel=self._compresslevel,
|
||||||
allowZip64=True,
|
allowZip64=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -1047,12 +1047,6 @@ class PermittedObjectsFilter(BaseFilterBackend):
|
|||||||
perm_codename: str | None = None
|
perm_codename: str | None = None
|
||||||
|
|
||||||
def filter_queryset(self, request, queryset, view):
|
def filter_queryset(self, request, queryset, view):
|
||||||
# Before the superuser and owner-only paths, neither of which consults
|
|
||||||
# permitted_object_ids. Scoped to authenticated users so anonymous
|
|
||||||
# access (AnonymousUser.is_active is False) keeps its existing
|
|
||||||
# unowned-only behaviour.
|
|
||||||
if request.user.is_authenticated and not request.user.is_active:
|
|
||||||
return queryset.none()
|
|
||||||
if request.user.is_superuser:
|
if request.user.is_superuser:
|
||||||
return queryset
|
return queryset
|
||||||
if not self.include_granted:
|
if not self.include_granted:
|
||||||
|
|||||||
@@ -29,6 +29,11 @@ if TYPE_CHECKING:
|
|||||||
if settings.AUDIT_LOG_ENABLED:
|
if settings.AUDIT_LOG_ENABLED:
|
||||||
from auditlog.models import LogEntry
|
from auditlog.models import LogEntry
|
||||||
|
|
||||||
|
from documents.export.compression import COMPRESSION_CHOICES
|
||||||
|
from documents.export.compression import COMPRESSION_METHODS
|
||||||
|
from documents.export.compression import ZSTD
|
||||||
|
from documents.export.compression import compression_available
|
||||||
|
from documents.export.compression import level_error
|
||||||
from documents.export.sinks import DirectoryExportSink
|
from documents.export.sinks import DirectoryExportSink
|
||||||
from documents.export.sinks import ExportSink
|
from documents.export.sinks import ExportSink
|
||||||
from documents.export.sinks import StreamingManifestWriter
|
from documents.export.sinks import StreamingManifestWriter
|
||||||
@@ -192,6 +197,28 @@ class Command(CryptMixin, PaperlessCommand):
|
|||||||
help="Sets the export zip file name",
|
help="Sets the export zip file name",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--zip-compression",
|
||||||
|
choices=COMPRESSION_CHOICES,
|
||||||
|
default=None,
|
||||||
|
help=(
|
||||||
|
"Compression method for the export zip (requires --zip). "
|
||||||
|
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
|
||||||
|
"exporting and importing machine."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--zip-compression-level",
|
||||||
|
type=int,
|
||||||
|
default=None,
|
||||||
|
help=(
|
||||||
|
"Compression level for the export zip (requires --zip). "
|
||||||
|
"deflated: 0-9, bzip2: 1-9, zstd: -22..22; ignored for "
|
||||||
|
"stored/lzma."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--data-only",
|
"--data-only",
|
||||||
default=False,
|
default=False,
|
||||||
@@ -247,12 +274,39 @@ class Command(CryptMixin, PaperlessCommand):
|
|||||||
if not os.access(self.target, os.W_OK):
|
if not os.access(self.target, os.W_OK):
|
||||||
raise CommandError("That path doesn't appear to be writable")
|
raise CommandError("That path doesn't appear to be writable")
|
||||||
|
|
||||||
|
zip_compression: str | None = options["zip_compression"]
|
||||||
|
zip_compression_level: int | None = options["zip_compression_level"]
|
||||||
|
|
||||||
|
if not self.zip_export and (
|
||||||
|
zip_compression is not None or zip_compression_level is not None
|
||||||
|
):
|
||||||
|
raise CommandError(
|
||||||
|
"--zip-compression and --zip-compression-level require --zip",
|
||||||
|
)
|
||||||
|
|
||||||
|
compression_method = zip_compression or "deflated"
|
||||||
|
if self.zip_export:
|
||||||
|
if not compression_available(compression_method):
|
||||||
|
if compression_method == "zstd" and ZSTD is None:
|
||||||
|
raise CommandError(
|
||||||
|
"zstd compression requires Python 3.14 or newer",
|
||||||
|
)
|
||||||
|
raise CommandError(
|
||||||
|
f"Compression method '{compression_method}' is not "
|
||||||
|
f"available on this Python runtime",
|
||||||
|
)
|
||||||
|
level_msg = level_error(compression_method, zip_compression_level)
|
||||||
|
if level_msg is not None:
|
||||||
|
raise CommandError(level_msg)
|
||||||
|
|
||||||
sink: ExportSink
|
sink: ExportSink
|
||||||
if self.zip_export:
|
if self.zip_export:
|
||||||
sink = ZipExportSink(
|
sink = ZipExportSink(
|
||||||
self.target,
|
self.target,
|
||||||
options["zip_name"],
|
options["zip_name"],
|
||||||
delete=self.delete,
|
delete=self.delete,
|
||||||
|
compression=COMPRESSION_METHODS[compression_method],
|
||||||
|
compresslevel=zip_compression_level,
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
sink = DirectoryExportSink(
|
sink = DirectoryExportSink(
|
||||||
|
|||||||
@@ -32,6 +32,8 @@ from django.db.models.signals import post_save
|
|||||||
from filelock import FileLock
|
from filelock import FileLock
|
||||||
from guardian.shortcuts import clear_ct_cache
|
from guardian.shortcuts import clear_ct_cache
|
||||||
|
|
||||||
|
from documents.export.compression import compress_type_readable
|
||||||
|
from documents.export.compression import unreadable_method_names
|
||||||
from documents.file_handling import create_source_path_directory
|
from documents.file_handling import create_source_path_directory
|
||||||
from documents.management.commands.base import PaperlessCommand
|
from documents.management.commands.base import PaperlessCommand
|
||||||
from documents.management.commands.mixins import CryptMixin
|
from documents.management.commands.mixins import CryptMixin
|
||||||
@@ -460,6 +462,20 @@ class Command(CryptMixin, PaperlessCommand):
|
|||||||
with tempfile.TemporaryDirectory() as tmp_dir:
|
with tempfile.TemporaryDirectory() as tmp_dir:
|
||||||
if is_zipfile(self.source):
|
if is_zipfile(self.source):
|
||||||
with ZipFile(self.source) as zf:
|
with ZipFile(self.source) as zf:
|
||||||
|
unsupported = {
|
||||||
|
info.compress_type
|
||||||
|
for info in zf.infolist()
|
||||||
|
if not compress_type_readable(info.compress_type)
|
||||||
|
}
|
||||||
|
if unsupported:
|
||||||
|
names = sorted(unreadable_method_names(unsupported))
|
||||||
|
message = (
|
||||||
|
f"This archive uses compression this Python cannot "
|
||||||
|
f"read ({', '.join(names)})."
|
||||||
|
)
|
||||||
|
if "zstd" in names:
|
||||||
|
message += " zstd archives require Python 3.14+."
|
||||||
|
raise CommandError(message)
|
||||||
zf.extractall(tmp_dir)
|
zf.extractall(tmp_dir)
|
||||||
self.source = Path(tmp_dir)
|
self.source = Path(tmp_dir)
|
||||||
self._run_import()
|
self._run_import()
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
# Generated by Django 5.2.16 on 2026-08-10 17:27
|
|
||||||
|
|
||||||
from django.db import migrations
|
|
||||||
from django.db import models
|
|
||||||
|
|
||||||
|
|
||||||
class Migration(migrations.Migration):
|
|
||||||
dependencies = [
|
|
||||||
("documents", "0022_add_perf_indexes"),
|
|
||||||
]
|
|
||||||
|
|
||||||
operations = [
|
|
||||||
migrations.AlterField(
|
|
||||||
model_name="workflowaction",
|
|
||||||
name="type",
|
|
||||||
field=models.PositiveSmallIntegerField(
|
|
||||||
choices=[
|
|
||||||
(1, "Assignment"),
|
|
||||||
(2, "Removal"),
|
|
||||||
(3, "Email"),
|
|
||||||
(4, "Webhook"),
|
|
||||||
(5, "Password removal"),
|
|
||||||
(6, "Move to trash"),
|
|
||||||
(7, "Remote OCR"),
|
|
||||||
],
|
|
||||||
default=1,
|
|
||||||
verbose_name="Workflow Action Type",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
]
|
|
||||||
@@ -1599,10 +1599,6 @@ class WorkflowAction(models.Model):
|
|||||||
6,
|
6,
|
||||||
_("Move to trash"),
|
_("Move to trash"),
|
||||||
)
|
)
|
||||||
REMOTE_OCR = (
|
|
||||||
7,
|
|
||||||
_("Remote OCR"),
|
|
||||||
)
|
|
||||||
|
|
||||||
type = models.PositiveSmallIntegerField(
|
type = models.PositiveSmallIntegerField(
|
||||||
_("Workflow Action Type"),
|
_("Workflow Action Type"),
|
||||||
|
|||||||
@@ -54,15 +54,11 @@ class PaperlessObjectPermissions(DjangoObjectPermissions):
|
|||||||
|
|
||||||
class PaperlessAdminPermissions(BasePermission):
|
class PaperlessAdminPermissions(BasePermission):
|
||||||
def has_permission(self, request, view):
|
def has_permission(self, request, view):
|
||||||
return request.user.is_active and request.user.is_staff
|
return request.user.is_staff
|
||||||
|
|
||||||
|
|
||||||
def has_global_statistics_permission(user: User | None) -> bool:
|
def has_global_statistics_permission(user: User | None) -> bool:
|
||||||
if (
|
if user is None or not getattr(user, "is_authenticated", False):
|
||||||
user is None
|
|
||||||
or not getattr(user, "is_active", False)
|
|
||||||
or not getattr(user, "is_authenticated", False)
|
|
||||||
):
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
return getattr(user, "is_superuser", False) or user.has_perm(
|
return getattr(user, "is_superuser", False) or user.has_perm(
|
||||||
@@ -71,11 +67,7 @@ def has_global_statistics_permission(user: User | None) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
def has_system_status_permission(user: User | None) -> bool:
|
def has_system_status_permission(user: User | None) -> bool:
|
||||||
if (
|
if user is None or not getattr(user, "is_authenticated", False):
|
||||||
user is None
|
|
||||||
or not getattr(user, "is_active", False)
|
|
||||||
or not getattr(user, "is_authenticated", False)
|
|
||||||
):
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
return (
|
return (
|
||||||
@@ -196,13 +188,6 @@ def permitted_object_ids(
|
|||||||
if user is None or not getattr(user, "is_authenticated", False):
|
if user is None or not getattr(user, "is_authenticated", False):
|
||||||
return base_qs.filter(owner__isnull=True).values_list("id", flat=True)
|
return base_qs.filter(owner__isnull=True).values_list("id", flat=True)
|
||||||
|
|
||||||
# Deactivated users get nothing, deactivated superusers included, so this
|
|
||||||
# has to come before the superuser shortcut. guardian's
|
|
||||||
# ObjectPermissionChecker denies inactive users, but get_objects_for_user
|
|
||||||
# (the pattern this replaces) does not, so it would not be inherited.
|
|
||||||
if not getattr(user, "is_active", False):
|
|
||||||
return base_qs.none().values_list("id", flat=True)
|
|
||||||
|
|
||||||
if getattr(user, "is_superuser", False):
|
if getattr(user, "is_superuser", False):
|
||||||
return base_qs.values_list("id", flat=True)
|
return base_qs.values_list("id", flat=True)
|
||||||
|
|
||||||
|
|||||||
@@ -1744,7 +1744,7 @@ class DeleteDocumentsSerializer(DocumentSelectionSerializer):
|
|||||||
|
|
||||||
|
|
||||||
class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
|
class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
|
||||||
remote_ocr = serializers.BooleanField(required=False, default=False)
|
pass
|
||||||
|
|
||||||
|
|
||||||
class BulkEditSerializer(
|
class BulkEditSerializer(
|
||||||
@@ -2086,13 +2086,6 @@ class BulkEditSerializer(
|
|||||||
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
|
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
|
||||||
)
|
)
|
||||||
|
|
||||||
def _validate_parameters_reprocess(self, parameters) -> None:
|
|
||||||
if "remote_ocr" in parameters:
|
|
||||||
if not isinstance(parameters["remote_ocr"], bool):
|
|
||||||
raise serializers.ValidationError("remote_ocr must be a boolean")
|
|
||||||
else:
|
|
||||||
parameters["remote_ocr"] = False
|
|
||||||
|
|
||||||
def validate_parameters_remove_password(self, parameters):
|
def validate_parameters_remove_password(self, parameters):
|
||||||
if "password" not in parameters:
|
if "password" not in parameters:
|
||||||
raise serializers.ValidationError("password not specified")
|
raise serializers.ValidationError("password not specified")
|
||||||
@@ -2157,8 +2150,6 @@ class BulkEditSerializer(
|
|||||||
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
|
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
|
||||||
elif method == bulk_edit.remove_password:
|
elif method == bulk_edit.remove_password:
|
||||||
self.validate_parameters_remove_password(parameters)
|
self.validate_parameters_remove_password(parameters)
|
||||||
elif method == bulk_edit.reprocess:
|
|
||||||
self._validate_parameters_reprocess(parameters)
|
|
||||||
|
|
||||||
return attrs
|
return attrs
|
||||||
|
|
||||||
@@ -3277,26 +3268,6 @@ class WorkflowSerializer(serializers.ModelSerializer[Workflow]):
|
|||||||
"actions",
|
"actions",
|
||||||
]
|
]
|
||||||
|
|
||||||
def validate(self, attrs):
|
|
||||||
attrs = super().validate(attrs)
|
|
||||||
|
|
||||||
triggers = attrs.get("triggers") or []
|
|
||||||
actions = attrs.get("actions") or []
|
|
||||||
|
|
||||||
# Remote OCR can only work with consumption triggers
|
|
||||||
if any(
|
|
||||||
action.get("type") == WorkflowAction.WorkflowActionType.REMOTE_OCR
|
|
||||||
for action in actions
|
|
||||||
) and not any(
|
|
||||||
trigger.get("type") == WorkflowTrigger.WorkflowTriggerType.CONSUMPTION
|
|
||||||
for trigger in triggers
|
|
||||||
):
|
|
||||||
raise serializers.ValidationError(
|
|
||||||
"Remote OCR actions require a consumption started trigger",
|
|
||||||
)
|
|
||||||
|
|
||||||
return attrs
|
|
||||||
|
|
||||||
def update_triggers_and_actions(
|
def update_triggers_and_actions(
|
||||||
self,
|
self,
|
||||||
instance: Workflow,
|
instance: Workflow,
|
||||||
|
|||||||
@@ -971,17 +971,6 @@ def run_workflows(
|
|||||||
)
|
)
|
||||||
elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH:
|
elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH:
|
||||||
has_move_to_trash_action = True
|
has_move_to_trash_action = True
|
||||||
elif action.type == WorkflowAction.WorkflowActionType.REMOTE_OCR:
|
|
||||||
if use_overrides and overrides:
|
|
||||||
overrides.remote_ocr = True
|
|
||||||
else:
|
|
||||||
# If a workflow has a consumption trigger *and* another type,
|
|
||||||
# the document has already been parsed by the time the other one fires
|
|
||||||
logger.debug(
|
|
||||||
"Remote OCR action only applies to consumption "
|
|
||||||
"triggers, ignoring",
|
|
||||||
extra={"group": logging_group},
|
|
||||||
)
|
|
||||||
|
|
||||||
if not use_overrides:
|
if not use_overrides:
|
||||||
# limit title to 128 characters
|
# limit title to 128 characters
|
||||||
|
|||||||
+1
-10
@@ -66,7 +66,6 @@ from documents.utils import compute_checksum
|
|||||||
from documents.utils import identity
|
from documents.utils import identity
|
||||||
from documents.workflows.utils import get_workflows_for_trigger
|
from documents.workflows.utils import get_workflows_for_trigger
|
||||||
from paperless.config import AIConfig
|
from paperless.config import AIConfig
|
||||||
from paperless.config import RemoteOCRConfig
|
|
||||||
from paperless.logging import consume_task_id
|
from paperless.logging import consume_task_id
|
||||||
from paperless.parsers import ParserContext
|
from paperless.parsers import ParserContext
|
||||||
from paperless.parsers.registry import get_parser_registry
|
from paperless.parsers.registry import get_parser_registry
|
||||||
@@ -338,17 +337,10 @@ def bulk_update_documents(document_ids) -> None:
|
|||||||
|
|
||||||
|
|
||||||
@shared_task
|
@shared_task
|
||||||
def update_document_content_maybe_archive_file(
|
def update_document_content_maybe_archive_file(document_id) -> None:
|
||||||
document_id,
|
|
||||||
*,
|
|
||||||
remote_ocr: bool = False,
|
|
||||||
) -> None:
|
|
||||||
"""
|
"""
|
||||||
Re-creates OCR content and thumbnail for a document, and archive file if
|
Re-creates OCR content and thumbnail for a document, and archive file if
|
||||||
it exists.
|
it exists.
|
||||||
|
|
||||||
Remote OCR is used only when the engine is configured to handle everything
|
|
||||||
or if explicitly asked for via ``remote_ocr``.
|
|
||||||
"""
|
"""
|
||||||
document = Document.objects.get(id=document_id)
|
document = Document.objects.get(id=document_id)
|
||||||
|
|
||||||
@@ -358,7 +350,6 @@ def update_document_content_maybe_archive_file(
|
|||||||
mime_type,
|
mime_type,
|
||||||
document.original_filename or "",
|
document.original_filename or "",
|
||||||
document.source_path,
|
document.source_path,
|
||||||
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if not parser_class:
|
if not parser_class:
|
||||||
|
|||||||
@@ -0,0 +1,190 @@
|
|||||||
|
import sys
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from documents.export import compression
|
||||||
|
|
||||||
|
|
||||||
|
class TestCompressionMethods:
|
||||||
|
def test_choices_always_include_zstd(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- The compression policy module's CLI choices list
|
||||||
|
WHEN:
|
||||||
|
- Read on any runtime
|
||||||
|
THEN:
|
||||||
|
- zstd is always present; availability is checked separately so
|
||||||
|
argparse never hides it based on the current Python version
|
||||||
|
"""
|
||||||
|
assert compression.COMPRESSION_CHOICES == (
|
||||||
|
"stored",
|
||||||
|
"deflated",
|
||||||
|
"bzip2",
|
||||||
|
"lzma",
|
||||||
|
"zstd",
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("name", "constant"),
|
||||||
|
[
|
||||||
|
("stored", zipfile.ZIP_STORED),
|
||||||
|
("deflated", zipfile.ZIP_DEFLATED),
|
||||||
|
("bzip2", zipfile.ZIP_BZIP2),
|
||||||
|
("lzma", zipfile.ZIP_LZMA),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A compression method name
|
||||||
|
WHEN:
|
||||||
|
- Looked up in COMPRESSION_METHODS
|
||||||
|
THEN:
|
||||||
|
- It maps to the matching zipfile compression constant
|
||||||
|
"""
|
||||||
|
assert compression.COMPRESSION_METHODS[name] == constant
|
||||||
|
|
||||||
|
def test_stored_and_deflated_always_available(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- The stored and deflated compression methods
|
||||||
|
WHEN:
|
||||||
|
- Checked with compression_available()
|
||||||
|
THEN:
|
||||||
|
- Both are always available (zlib is a hard CPython dependency)
|
||||||
|
"""
|
||||||
|
assert compression.compression_available("stored")
|
||||||
|
assert compression.compression_available("deflated")
|
||||||
|
|
||||||
|
def test_zstd_availability_tracks_runtime(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- The zstd compression method
|
||||||
|
WHEN:
|
||||||
|
- Checked with compression_available() on this runtime
|
||||||
|
THEN:
|
||||||
|
- Availability matches whether Python is 3.14+
|
||||||
|
"""
|
||||||
|
expected: bool = sys.version_info >= (3, 14)
|
||||||
|
assert compression.compression_available("zstd") == expected
|
||||||
|
|
||||||
|
|
||||||
|
class TestLevelError:
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("method", "level"),
|
||||||
|
[
|
||||||
|
("deflated", 0),
|
||||||
|
("deflated", 9),
|
||||||
|
("bzip2", 1),
|
||||||
|
("bzip2", 9),
|
||||||
|
("zstd", -22),
|
||||||
|
("zstd", 22),
|
||||||
|
("deflated", None),
|
||||||
|
("stored", None),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A method and a level within its valid bounds (or no level)
|
||||||
|
WHEN:
|
||||||
|
- Checked with level_error()
|
||||||
|
THEN:
|
||||||
|
- No error message is returned
|
||||||
|
"""
|
||||||
|
assert compression.level_error(method, level) is None
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("method", "level"),
|
||||||
|
[
|
||||||
|
("deflated", 10),
|
||||||
|
("deflated", -1),
|
||||||
|
("bzip2", 0),
|
||||||
|
("bzip2", 10),
|
||||||
|
("zstd", -23),
|
||||||
|
("zstd", 23),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_out_of_range_levels_return_message(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
level: int,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A method and a level outside its valid bounds
|
||||||
|
WHEN:
|
||||||
|
- Checked with level_error()
|
||||||
|
THEN:
|
||||||
|
- An error message naming the valid range is returned
|
||||||
|
"""
|
||||||
|
msg: str | None = compression.level_error(method, level)
|
||||||
|
assert msg is not None
|
||||||
|
assert "between" in msg
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("method", ["stored", "lzma"])
|
||||||
|
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A method that ignores compression level (stored, lzma)
|
||||||
|
WHEN:
|
||||||
|
- A level is passed to level_error() anyway
|
||||||
|
THEN:
|
||||||
|
- An error message noting the level has no effect is returned
|
||||||
|
"""
|
||||||
|
msg: str | None = compression.level_error(method, 5)
|
||||||
|
assert msg is not None
|
||||||
|
assert "no effect" in msg
|
||||||
|
|
||||||
|
|
||||||
|
class TestCompressTypeReadable:
|
||||||
|
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
|
||||||
|
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A stored or deflated compress_type id
|
||||||
|
WHEN:
|
||||||
|
- Checked with compress_type_readable()
|
||||||
|
THEN:
|
||||||
|
- It is always readable
|
||||||
|
"""
|
||||||
|
assert compression.compress_type_readable(ct)
|
||||||
|
|
||||||
|
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- The current (93) and legacy (20) zstd compress_type ids
|
||||||
|
WHEN:
|
||||||
|
- Checked with compress_type_readable() on this runtime
|
||||||
|
THEN:
|
||||||
|
- Readability matches whether Python is 3.14+
|
||||||
|
"""
|
||||||
|
# 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
|
||||||
|
expected: bool = sys.version_info >= (3, 14)
|
||||||
|
assert compression.compress_type_readable(93) == expected
|
||||||
|
assert compression.compress_type_readable(20) == expected
|
||||||
|
|
||||||
|
def test_unknown_compress_type_is_unreadable(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- An unrecognized compress_type id
|
||||||
|
WHEN:
|
||||||
|
- Checked with compress_type_readable()
|
||||||
|
THEN:
|
||||||
|
- It is reported as unreadable
|
||||||
|
"""
|
||||||
|
assert not compression.compress_type_readable(9999)
|
||||||
|
|
||||||
|
def test_unreadable_method_names_lists_methods(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A set containing an unknown compress_type id
|
||||||
|
WHEN:
|
||||||
|
- Passed to unreadable_method_names()
|
||||||
|
THEN:
|
||||||
|
- It is reported generically as "method <id>"
|
||||||
|
"""
|
||||||
|
# An unknown method id maps to no name and is reported generically.
|
||||||
|
names: set[str] = compression.unreadable_method_names({9999})
|
||||||
|
assert names == {"method 9999"}
|
||||||
@@ -5,6 +5,7 @@ import zipfile
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
import pytest_mock
|
||||||
from pytest_django.fixtures import SettingsWrapper
|
from pytest_django.fixtures import SettingsWrapper
|
||||||
|
|
||||||
from documents.export.sinks import DirectoryExportSink
|
from documents.export.sinks import DirectoryExportSink
|
||||||
@@ -305,6 +306,48 @@ class TestZipExportSink:
|
|||||||
assert not (target / "export.zip").exists()
|
assert not (target / "export.zip").exists()
|
||||||
|
|
||||||
|
|
||||||
|
class TestZipExportSinkCompression:
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("method", "constant"),
|
||||||
|
[
|
||||||
|
("stored", zipfile.ZIP_STORED),
|
||||||
|
("deflated", zipfile.ZIP_DEFLATED),
|
||||||
|
("bzip2", zipfile.ZIP_BZIP2),
|
||||||
|
("lzma", zipfile.ZIP_LZMA),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_compression_and_level_forwarded_to_zipfile(
|
||||||
|
self,
|
||||||
|
mocker: pytest_mock.MockerFixture,
|
||||||
|
tmp_path: Path,
|
||||||
|
method: str,
|
||||||
|
constant: int,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A ZipExportSink constructed with a compression method and level
|
||||||
|
WHEN:
|
||||||
|
- The sink is opened
|
||||||
|
THEN:
|
||||||
|
- zipfile.ZipFile is constructed with those values forwarded
|
||||||
|
unchanged (whether ZipFile actually compresses is Python's own
|
||||||
|
contract, not ours, so this checks the call args, not a real
|
||||||
|
archive)
|
||||||
|
"""
|
||||||
|
target: Path = tmp_path / "out"
|
||||||
|
target.mkdir()
|
||||||
|
zip_cls = mocker.patch("documents.export.sinks.zipfile.ZipFile")
|
||||||
|
sink = ZipExportSink(target, "export", compression=constant, compresslevel=5)
|
||||||
|
sink._open()
|
||||||
|
zip_cls.assert_called_once_with(
|
||||||
|
mocker.ANY,
|
||||||
|
"w",
|
||||||
|
compression=constant,
|
||||||
|
compresslevel=5,
|
||||||
|
allowZip64=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TestStreamContract:
|
class TestStreamContract:
|
||||||
@pytest.fixture(params=["dir", "zip"])
|
@pytest.fixture(params=["dir", "zip"])
|
||||||
def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink:
|
def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink:
|
||||||
|
|||||||
@@ -72,10 +72,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
|||||||
"barcode_enable_tag": None,
|
"barcode_enable_tag": None,
|
||||||
"barcode_tag_mapping": None,
|
"barcode_tag_mapping": None,
|
||||||
"barcode_tag_split": None,
|
"barcode_tag_split": None,
|
||||||
"remote_ocr_engine": None,
|
|
||||||
"remote_ocr_api_key": None,
|
|
||||||
"remote_ocr_endpoint": None,
|
|
||||||
"remote_ocr_mode": None,
|
|
||||||
"ai_enabled": False,
|
"ai_enabled": False,
|
||||||
"llm_embedding_backend": None,
|
"llm_embedding_backend": None,
|
||||||
"llm_embedding_model": None,
|
"llm_embedding_model": None,
|
||||||
@@ -874,49 +870,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
|||||||
config.refresh_from_db()
|
config.refresh_from_db()
|
||||||
self.assertEqual(config.llm_api_key, None)
|
self.assertEqual(config.llm_api_key, None)
|
||||||
|
|
||||||
def test_update_remote_ocr_api_key(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- Existing config with remote_ocr_api_key specified
|
|
||||||
WHEN:
|
|
||||||
- API to update remote_ocr_api_key is called with all *s
|
|
||||||
- API to update remote_ocr_api_key is called with empty string
|
|
||||||
THEN:
|
|
||||||
- remote_ocr_api_key is unchanged
|
|
||||||
- remote_ocr_api_key is set to None
|
|
||||||
"""
|
|
||||||
config = ApplicationConfiguration.objects.first()
|
|
||||||
assert config is not None
|
|
||||||
config.remote_ocr_api_key = "1234567890"
|
|
||||||
config.save()
|
|
||||||
|
|
||||||
# Test with all *
|
|
||||||
response = self.client.patch(
|
|
||||||
f"{self.ENDPOINT}1/",
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"remote_ocr_api_key": "*" * 32,
|
|
||||||
},
|
|
||||||
),
|
|
||||||
content_type="application/json",
|
|
||||||
)
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
|
||||||
config.refresh_from_db()
|
|
||||||
self.assertEqual(config.remote_ocr_api_key, "1234567890")
|
|
||||||
# Test with empty string
|
|
||||||
response = self.client.patch(
|
|
||||||
f"{self.ENDPOINT}1/",
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"remote_ocr_api_key": "",
|
|
||||||
},
|
|
||||||
),
|
|
||||||
content_type="application/json",
|
|
||||||
)
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
|
||||||
config.refresh_from_db()
|
|
||||||
self.assertEqual(config.remote_ocr_api_key, None)
|
|
||||||
|
|
||||||
def test_enable_ai_index_triggers_update(self) -> None:
|
def test_enable_ai_index_triggers_update(self) -> None:
|
||||||
"""
|
"""
|
||||||
GIVEN:
|
GIVEN:
|
||||||
|
|||||||
@@ -532,29 +532,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
|
|||||||
m.assert_called_once()
|
m.assert_called_once()
|
||||||
args, kwargs = m.call_args
|
args, kwargs = m.call_args
|
||||||
self.assertEqual(args[0], [self.doc1.id])
|
self.assertEqual(args[0], [self.doc1.id])
|
||||||
self.assertEqual(kwargs, {"remote_ocr": False})
|
self.assertEqual(len(kwargs), 0)
|
||||||
|
|
||||||
@mock.patch("documents.views.bulk_edit.reprocess")
|
|
||||||
def test_reprocess_documents_endpoint_remote_ocr(self, m) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- API data to reprocess a document with remote OCR requested
|
|
||||||
WHEN:
|
|
||||||
- API is called
|
|
||||||
THEN:
|
|
||||||
- reprocess is called with remote_ocr=True
|
|
||||||
"""
|
|
||||||
self.setup_mock(m, "reprocess")
|
|
||||||
response = self.client.post(
|
|
||||||
"/api/documents/reprocess/",
|
|
||||||
json.dumps({"documents": [self.doc1.id], "remote_ocr": True}),
|
|
||||||
content_type="application/json",
|
|
||||||
)
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
|
||||||
m.assert_called_once()
|
|
||||||
args, kwargs = m.call_args
|
|
||||||
self.assertEqual(args[0], [self.doc1.id])
|
|
||||||
self.assertEqual(kwargs, {"remote_ocr": True})
|
|
||||||
|
|
||||||
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
|
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
|
||||||
def test_api_set_storage_path(self, m) -> None:
|
def test_api_set_storage_path(self, m) -> None:
|
||||||
@@ -1575,29 +1553,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_legacy_bulk_edit_reprocess_invalid_remote_ocr(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- The deprecated bulk_edit endpoint with a non-boolean remote_ocr
|
|
||||||
WHEN:
|
|
||||||
- API is called
|
|
||||||
THEN:
|
|
||||||
- The request is rejected rather than passed through to the task
|
|
||||||
"""
|
|
||||||
response = self.client.post(
|
|
||||||
"/api/documents/bulk_edit/",
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"documents": [self.doc1.id],
|
|
||||||
"method": "reprocess",
|
|
||||||
"parameters": {"remote_ocr": "yes please"},
|
|
||||||
},
|
|
||||||
),
|
|
||||||
content_type="application/json",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
|
||||||
|
|
||||||
@mock.patch("documents.views.bulk_edit.edit_pdf")
|
@mock.patch("documents.views.bulk_edit.edit_pdf")
|
||||||
def test_edit_pdf(self, m) -> None:
|
def test_edit_pdf(self, m) -> None:
|
||||||
self.setup_mock(m, "edit_pdf")
|
self.setup_mock(m, "edit_pdf")
|
||||||
|
|||||||
@@ -60,10 +60,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
|
|||||||
},
|
},
|
||||||
"email_enabled": False,
|
"email_enabled": False,
|
||||||
"ai_enabled": False,
|
"ai_enabled": False,
|
||||||
"remote_ocr": {
|
|
||||||
"configured": False,
|
|
||||||
"mode": "always",
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -158,50 +154,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
|
|||||||
str(response.data["settings"]),
|
str(response.data["settings"]),
|
||||||
)
|
)
|
||||||
|
|
||||||
@override_settings(
|
|
||||||
REMOTE_OCR_ENGINE="azureai",
|
|
||||||
REMOTE_OCR_API_KEY="somekey",
|
|
||||||
REMOTE_OCR_ENDPOINT="https://example.cognitiveservices.azure.com",
|
|
||||||
REMOTE_OCR_MODE="workflow_only",
|
|
||||||
)
|
|
||||||
def test_settings_reports_remote_ocr_when_configured(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A fully configured remote OCR engine in workflow_only mode
|
|
||||||
WHEN:
|
|
||||||
- The ui_settings endpoint is called
|
|
||||||
THEN:
|
|
||||||
- The UI is told remote OCR is available and selective, so it can
|
|
||||||
offer it where it would actually change something
|
|
||||||
"""
|
|
||||||
response = self.client.get(self.ENDPOINT, format="json")
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
|
||||||
self.assertEqual(
|
|
||||||
response.data["settings"]["remote_ocr"],
|
|
||||||
{"configured": True, "mode": "workflow_only"},
|
|
||||||
)
|
|
||||||
|
|
||||||
@override_settings(
|
|
||||||
REMOTE_OCR_ENGINE="azureai",
|
|
||||||
REMOTE_OCR_API_KEY=None,
|
|
||||||
REMOTE_OCR_ENDPOINT=None,
|
|
||||||
)
|
|
||||||
def test_settings_reports_remote_ocr_incompletely_configured(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- An engine named but missing its endpoint and API key
|
|
||||||
WHEN:
|
|
||||||
- The ui_settings endpoint is called
|
|
||||||
THEN:
|
|
||||||
- It is reported as not configured, matching what the parser
|
|
||||||
registry will actually do
|
|
||||||
"""
|
|
||||||
response = self.client.get(self.ENDPOINT, format="json")
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
|
||||||
self.assertFalse(response.data["settings"]["remote_ocr"]["configured"])
|
|
||||||
|
|
||||||
@override_settings(
|
@override_settings(
|
||||||
OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
|
OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
|
||||||
GMAIL_OAUTH_CLIENT_ID="abc123",
|
GMAIL_OAUTH_CLIENT_ID="abc123",
|
||||||
|
|||||||
@@ -390,82 +390,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
|
|||||||
|
|
||||||
self.assertEqual(Workflow.objects.count(), 1)
|
self.assertEqual(Workflow.objects.count(), 1)
|
||||||
|
|
||||||
def test_api_create_remote_ocr_action_requires_consumption_trigger(
|
|
||||||
self,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- API request to create a workflow with a remote OCR action
|
|
||||||
- No consumption started trigger, so the action could never run
|
|
||||||
WHEN:
|
|
||||||
- API is called
|
|
||||||
THEN:
|
|
||||||
- Correct HTTP 400 response
|
|
||||||
- No objects are created
|
|
||||||
"""
|
|
||||||
existing_count = Workflow.objects.count()
|
|
||||||
|
|
||||||
response = self.client.post(
|
|
||||||
self.ENDPOINT,
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"name": "Remote OCR too late",
|
|
||||||
"order": 1,
|
|
||||||
"triggers": [
|
|
||||||
{
|
|
||||||
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
|
|
||||||
},
|
|
||||||
],
|
|
||||||
"actions": [
|
|
||||||
{
|
|
||||||
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
|
|
||||||
},
|
|
||||||
],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
content_type="application/json",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
|
||||||
self.assertEqual(Workflow.objects.count(), existing_count)
|
|
||||||
|
|
||||||
def test_api_create_remote_ocr_action_with_consumption_trigger(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- API request to create a workflow with a remote OCR action
|
|
||||||
- A consumption started trigger alongside another trigger type
|
|
||||||
WHEN:
|
|
||||||
- API is called
|
|
||||||
THEN:
|
|
||||||
- The workflow is created, the action applies to consumption only
|
|
||||||
"""
|
|
||||||
response = self.client.post(
|
|
||||||
self.ENDPOINT,
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"name": "Remote OCR on consume",
|
|
||||||
"order": 1,
|
|
||||||
"triggers": [
|
|
||||||
{
|
|
||||||
"type": WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
|
|
||||||
"filter_filename": "*.pdf",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
|
|
||||||
},
|
|
||||||
],
|
|
||||||
"actions": [
|
|
||||||
{
|
|
||||||
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
|
|
||||||
},
|
|
||||||
],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
content_type="application/json",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
|
|
||||||
|
|
||||||
def test_api_create_workflow_trigger_action_empty_fields(self) -> None:
|
def test_api_create_workflow_trigger_action_empty_fields(self) -> None:
|
||||||
"""
|
"""
|
||||||
GIVEN:
|
GIVEN:
|
||||||
|
|||||||
@@ -1782,56 +1782,3 @@ class TestPDFActions(DirectoriesMixin, TestCase):
|
|||||||
|
|
||||||
self.assertIn("wrong password", str(exc.exception))
|
self.assertIn("wrong password", str(exc.exception))
|
||||||
self.assertIn("Error removing password from document", cm.output[0])
|
self.assertIn("Error removing password from document", cm.output[0])
|
||||||
|
|
||||||
|
|
||||||
class TestBulkEditReprocess(DirectoriesMixin, TestCase):
|
|
||||||
def setUp(self) -> None:
|
|
||||||
super().setUp()
|
|
||||||
|
|
||||||
self.doc = Document.objects.create(
|
|
||||||
title="test",
|
|
||||||
checksum="A",
|
|
||||||
mime_type="application/pdf",
|
|
||||||
)
|
|
||||||
|
|
||||||
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
|
|
||||||
def test_reprocess_defaults_to_local(self, mock_task: mock.Mock) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A reprocess request that says nothing about remote OCR
|
|
||||||
WHEN:
|
|
||||||
- reprocess is called
|
|
||||||
THEN:
|
|
||||||
- The task is queued without asking for the remote engine
|
|
||||||
"""
|
|
||||||
result = bulk_edit.reprocess([self.doc.id])
|
|
||||||
|
|
||||||
self.assertEqual(result, "OK")
|
|
||||||
mock_task.apply_async.assert_called_once()
|
|
||||||
_, kwargs = mock_task.apply_async.call_args
|
|
||||||
self.assertEqual(
|
|
||||||
kwargs["kwargs"],
|
|
||||||
{"document_id": self.doc.id, "remote_ocr": False},
|
|
||||||
)
|
|
||||||
|
|
||||||
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
|
|
||||||
def test_reprocess_passes_remote_ocr(self, mock_task: mock.Mock) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A reprocess request that explicitly asks for remote OCR
|
|
||||||
WHEN:
|
|
||||||
- reprocess is called
|
|
||||||
THEN:
|
|
||||||
- The request is forwarded to the task for every document
|
|
||||||
"""
|
|
||||||
other = Document.objects.create(
|
|
||||||
title="test2",
|
|
||||||
checksum="B",
|
|
||||||
mime_type="application/pdf",
|
|
||||||
)
|
|
||||||
|
|
||||||
bulk_edit.reprocess([self.doc.id, other.id], remote_ocr=True)
|
|
||||||
|
|
||||||
self.assertEqual(mock_task.apply_async.call_count, 2)
|
|
||||||
for call in mock_task.apply_async.call_args_list:
|
|
||||||
self.assertTrue(call.kwargs["kwargs"]["remote_ocr"])
|
|
||||||
|
|||||||
@@ -1559,72 +1559,6 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
|
|||||||
consumer.run_post_consume_script(doc)
|
consumer.run_post_consume_script(doc)
|
||||||
|
|
||||||
|
|
||||||
class TestConsumerRemoteOCR(
|
|
||||||
DirectoriesMixin,
|
|
||||||
FileSystemAssertsMixin,
|
|
||||||
GetConsumerMixin,
|
|
||||||
TestCase,
|
|
||||||
):
|
|
||||||
"""
|
|
||||||
The consumer resolves the remote OCR mode and the per-document request from
|
|
||||||
workflows into the allow_remote flag it hands to the parser registry.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def setUp(self) -> None:
|
|
||||||
super().setUp()
|
|
||||||
|
|
||||||
patcher = mock.patch("documents.consumer.get_parser_registry")
|
|
||||||
self.mock_registry = patcher.start()
|
|
||||||
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
|
|
||||||
self.addCleanup(patcher.stop)
|
|
||||||
|
|
||||||
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
|
|
||||||
src = (
|
|
||||||
Path(__file__).parent
|
|
||||||
/ "samples"
|
|
||||||
/ "documents"
|
|
||||||
/ "originals"
|
|
||||||
/ "0000001.pdf"
|
|
||||||
)
|
|
||||||
dst = self.dirs.scratch_dir / "sample.pdf"
|
|
||||||
shutil.copy(src, dst)
|
|
||||||
|
|
||||||
with self.get_consumer(dst, overrides=overrides) as consumer:
|
|
||||||
consumer.run()
|
|
||||||
|
|
||||||
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
|
|
||||||
return kwargs["allow_remote"]
|
|
||||||
|
|
||||||
@override_settings(REMOTE_OCR_MODE="always")
|
|
||||||
def test_always_mode_allows_remote(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: Remote OCR mode is 'always'.
|
|
||||||
WHEN: A document is consumed without any workflow asking for it.
|
|
||||||
THEN: The registry is allowed to pick the remote parser.
|
|
||||||
"""
|
|
||||||
self.assertTrue(self._consume())
|
|
||||||
|
|
||||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
|
||||||
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: Remote OCR mode is 'workflow_only'.
|
|
||||||
WHEN: A document is consumed and nothing asked for remote OCR.
|
|
||||||
THEN: The remote parser is excluded.
|
|
||||||
"""
|
|
||||||
self.assertFalse(self._consume())
|
|
||||||
|
|
||||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
|
||||||
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: Remote OCR mode is 'workflow_only'.
|
|
||||||
WHEN: A workflow set remote_ocr on the metadata overrides.
|
|
||||||
THEN: The registry is allowed to pick the remote parser.
|
|
||||||
"""
|
|
||||||
self.assertTrue(
|
|
||||||
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TestMetadataOverrides(TestCase):
|
class TestMetadataOverrides(TestCase):
|
||||||
def test_update_skip_asn_if_exists(self) -> None:
|
def test_update_skip_asn_if_exists(self) -> None:
|
||||||
base = DocumentMetadataOverrides()
|
base = DocumentMetadataOverrides()
|
||||||
@@ -1632,20 +1566,6 @@ class TestMetadataOverrides(TestCase):
|
|||||||
base.update(incoming)
|
base.update(incoming)
|
||||||
self.assertTrue(base.skip_asn_if_exists)
|
self.assertTrue(base.skip_asn_if_exists)
|
||||||
|
|
||||||
def test_update_remote_ocr(self) -> None:
|
|
||||||
base = DocumentMetadataOverrides()
|
|
||||||
base.update(DocumentMetadataOverrides(remote_ocr=True))
|
|
||||||
self.assertTrue(base.remote_ocr)
|
|
||||||
|
|
||||||
def test_update_remote_ocr_is_not_unset(self) -> None:
|
|
||||||
"""
|
|
||||||
A later workflow that says nothing must not undo an earlier one that
|
|
||||||
asked for remote OCR.
|
|
||||||
"""
|
|
||||||
base = DocumentMetadataOverrides(remote_ocr=True)
|
|
||||||
base.update(DocumentMetadataOverrides())
|
|
||||||
self.assertTrue(base.remote_ocr)
|
|
||||||
|
|
||||||
def test_update_actor_and_version_label(self) -> None:
|
def test_update_actor_and_version_label(self) -> None:
|
||||||
base = DocumentMetadataOverrides(
|
base = DocumentMetadataOverrides(
|
||||||
actor_id=1,
|
actor_id=1,
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ from datetime import timedelta
|
|||||||
from io import StringIO
|
from io import StringIO
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from unittest import mock
|
from unittest import mock
|
||||||
|
from zipfile import ZIP_DEFLATED
|
||||||
|
from zipfile import ZIP_LZMA
|
||||||
from zipfile import ZipFile
|
from zipfile import ZipFile
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -1078,6 +1080,186 @@ class TestExportImport(
|
|||||||
skip_checks=True,
|
skip_checks=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_compression_flags_require_zip(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A request to export without --zip
|
||||||
|
WHEN:
|
||||||
|
- --zip-compression or --zip-compression-level is passed anyway
|
||||||
|
THEN:
|
||||||
|
- A CommandError is raised (the flags are meaningless without --zip)
|
||||||
|
"""
|
||||||
|
for args in (
|
||||||
|
["--zip-compression", "lzma"],
|
||||||
|
["--zip-compression-level", "5"],
|
||||||
|
):
|
||||||
|
with self.assertRaises(CommandError):
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
*args,
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_zip_compression_level_out_of_range_raises(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A request to export to a zip file
|
||||||
|
WHEN:
|
||||||
|
- --zip-compression-level is outside the chosen method's valid range
|
||||||
|
THEN:
|
||||||
|
- A CommandError is raised
|
||||||
|
"""
|
||||||
|
with self.assertRaises(CommandError):
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
"--zip",
|
||||||
|
"--zip-compression",
|
||||||
|
"deflated",
|
||||||
|
"--zip-compression-level",
|
||||||
|
"99",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_zip_compression_level_rejected_for_stored(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A request to export to a zip file with --zip-compression stored
|
||||||
|
WHEN:
|
||||||
|
- --zip-compression-level is also passed
|
||||||
|
THEN:
|
||||||
|
- A CommandError is raised (stored ignores level entirely)
|
||||||
|
"""
|
||||||
|
with self.assertRaises(CommandError):
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
"--zip",
|
||||||
|
"--zip-compression",
|
||||||
|
"stored",
|
||||||
|
"--zip-compression-level",
|
||||||
|
"5",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_zip_compression_level_rejected_for_lzma(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A request to export to a zip file with --zip-compression lzma
|
||||||
|
WHEN:
|
||||||
|
- --zip-compression-level is also passed
|
||||||
|
THEN:
|
||||||
|
- A CommandError is raised (lzma ignores level entirely)
|
||||||
|
"""
|
||||||
|
with self.assertRaises(CommandError):
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
"--zip",
|
||||||
|
"--zip-compression",
|
||||||
|
"lzma",
|
||||||
|
"--zip-compression-level",
|
||||||
|
"5",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_zstd_unavailable_raises_friendly_error(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A Python runtime without zstd support (< 3.14)
|
||||||
|
WHEN:
|
||||||
|
- --zip-compression zstd is requested
|
||||||
|
THEN:
|
||||||
|
- A CommandError naming the Python version requirement is raised
|
||||||
|
|
||||||
|
zstd availability is mocked rather than relying on the actual
|
||||||
|
runtime: on a Python 3.14+ CI leg, ZSTD is not None, so without the
|
||||||
|
mock this check is skipped and the command falls through into the
|
||||||
|
real export, which fails on missing document files instead of
|
||||||
|
raising the expected CommandError.
|
||||||
|
"""
|
||||||
|
with (
|
||||||
|
mock.patch(
|
||||||
|
"documents.management.commands.document_exporter.ZSTD",
|
||||||
|
None,
|
||||||
|
),
|
||||||
|
mock.patch(
|
||||||
|
"documents.management.commands.document_exporter.compression_available",
|
||||||
|
return_value=False,
|
||||||
|
),
|
||||||
|
self.assertRaises(CommandError) as e,
|
||||||
|
):
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
"--zip",
|
||||||
|
"--zip-compression",
|
||||||
|
"zstd",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
self.assertIn("3.14", str(e.exception))
|
||||||
|
|
||||||
|
def test_zip_compression_flag_resolves_to_sink_constant(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A request to export to a zip file with --zip-compression lzma
|
||||||
|
WHEN:
|
||||||
|
- The export runs
|
||||||
|
THEN:
|
||||||
|
- ZipExportSink is constructed with the resolved ZIP_LZMA constant
|
||||||
|
(whether zipfile actually compresses with the chosen method is
|
||||||
|
Python's own contract, and ZipExportSink's own tests already
|
||||||
|
cover the forwarding; what this command owns is resolving the
|
||||||
|
CLI string to the right constant, so assert that resolution
|
||||||
|
directly)
|
||||||
|
"""
|
||||||
|
with mock.patch(
|
||||||
|
"documents.management.commands.document_exporter.ZipExportSink",
|
||||||
|
) as sink_cls:
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
"--zip",
|
||||||
|
"--zip-compression",
|
||||||
|
"lzma",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
sink_cls.assert_called_once_with(
|
||||||
|
mock.ANY,
|
||||||
|
mock.ANY,
|
||||||
|
delete=False,
|
||||||
|
compression=ZIP_LZMA,
|
||||||
|
compresslevel=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_default_zip_compression_resolves_to_deflate(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A request to export to a zip file with no --zip-compression flag
|
||||||
|
WHEN:
|
||||||
|
- The export runs
|
||||||
|
THEN:
|
||||||
|
- ZipExportSink is constructed with the default ZIP_DEFLATED
|
||||||
|
constant and compresslevel=None, matching pre-existing behavior
|
||||||
|
"""
|
||||||
|
with mock.patch(
|
||||||
|
"documents.management.commands.document_exporter.ZipExportSink",
|
||||||
|
) as sink_cls:
|
||||||
|
call_command(
|
||||||
|
"document_exporter",
|
||||||
|
self.target,
|
||||||
|
"--zip",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
sink_cls.assert_called_once_with(
|
||||||
|
mock.ANY,
|
||||||
|
mock.ANY,
|
||||||
|
delete=False,
|
||||||
|
compression=ZIP_DEFLATED,
|
||||||
|
compresslevel=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.management
|
@pytest.mark.management
|
||||||
class TestCryptExportImport(
|
class TestCryptExportImport(
|
||||||
|
|||||||
@@ -525,6 +525,35 @@ class TestCommandImport(
|
|||||||
self.assertEqual(doc.tags.count(), 1)
|
self.assertEqual(doc.tags.count(), 1)
|
||||||
self.assertEqual(doc.tags.first().name, "batch-flush-tag")
|
self.assertEqual(doc.tags.first().name, "batch-flush-tag")
|
||||||
|
|
||||||
|
def test_import_rejects_unreadable_compression(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A zip archive with an entry whose compression this Python can't read
|
||||||
|
WHEN:
|
||||||
|
- Import is attempted
|
||||||
|
THEN:
|
||||||
|
- A CommandError naming the issue is raised, before extraction
|
||||||
|
"""
|
||||||
|
import zipfile
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
archive = Path(self.dirs.scratch_dir) / "export.zip"
|
||||||
|
with zipfile.ZipFile(archive, "w") as zf:
|
||||||
|
zf.writestr("manifest.json", "[]")
|
||||||
|
|
||||||
|
with mock.patch(
|
||||||
|
"documents.management.commands.document_importer.compress_type_readable",
|
||||||
|
return_value=False,
|
||||||
|
):
|
||||||
|
with self.assertRaises(CommandError) as e:
|
||||||
|
call_command(
|
||||||
|
"document_importer",
|
||||||
|
str(archive),
|
||||||
|
"--no-progress-bar",
|
||||||
|
skip_checks=True,
|
||||||
|
)
|
||||||
|
self.assertIn("compression", str(e.exception))
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.management
|
@pytest.mark.management
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
|
|||||||
@@ -496,28 +496,6 @@ class TestPermittedObjectIdsGenericModels:
|
|||||||
expected_hidden=[strangers.pk],
|
expected_hidden=[strangers.pk],
|
||||||
)
|
)
|
||||||
|
|
||||||
@pytest.mark.parametrize("is_superuser", [False, True])
|
|
||||||
def test_inactive_user_sees_nothing(self, model, factory, perm, is_superuser):
|
|
||||||
suffix = f"{model.__name__}_{is_superuser}"
|
|
||||||
user = User.objects.create_user(
|
|
||||||
username=f"inactive_{suffix}",
|
|
||||||
is_active=False,
|
|
||||||
is_superuser=is_superuser,
|
|
||||||
)
|
|
||||||
other = User.objects.create_user(username=f"other_{suffix}")
|
|
||||||
granted = factory(owner=other)
|
|
||||||
assign_perm(perm, user, granted)
|
|
||||||
|
|
||||||
assert_visible_document_ids(
|
|
||||||
permitted_object_ids(user, model, perm),
|
|
||||||
expected_visible=[],
|
|
||||||
expected_hidden=[
|
|
||||||
factory(owner=None).pk,
|
|
||||||
factory(owner=user).pk,
|
|
||||||
granted.pk,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_unowned_object_visible_to_everyone(self, model, factory, perm):
|
def test_unowned_object_visible_to_everyone(self, model, factory, perm):
|
||||||
user = User.objects.create_user(username=f"user_{model.__name__}")
|
user = User.objects.create_user(username=f"user_{model.__name__}")
|
||||||
unowned = factory(owner=None)
|
unowned = factory(owner=None)
|
||||||
|
|||||||
@@ -68,44 +68,3 @@ class TestPermittedObjectsFilter:
|
|||||||
visible_ids = set(result.values_list("id", flat=True))
|
visible_ids = set(result.values_list("id", flat=True))
|
||||||
assert visible_ids == {owned.pk}
|
assert visible_ids == {owned.pk}
|
||||||
assert granted.pk not in visible_ids
|
assert granted.pk not in visible_ids
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("username", "is_superuser"),
|
|
||||||
[("inactive", False), ("inactive_super", True)],
|
|
||||||
)
|
|
||||||
def test_inactive_user_sees_nothing(self, username: str, *, is_superuser: bool):
|
|
||||||
user = User.objects.create_user(
|
|
||||||
username=username,
|
|
||||||
is_active=False,
|
|
||||||
is_superuser=is_superuser,
|
|
||||||
)
|
|
||||||
TagFactory(owner=None)
|
|
||||||
TagFactory(owner=user)
|
|
||||||
granted = TagFactory(owner=User.objects.create_user(username=f"o_{username}"))
|
|
||||||
assign_perm("view_tag", user, granted)
|
|
||||||
request = APIRequestFactory().get("/")
|
|
||||||
request.user = user
|
|
||||||
|
|
||||||
result = PermittedObjectsFilter().filter_queryset(
|
|
||||||
request,
|
|
||||||
Tag.objects.all(),
|
|
||||||
_DummyView(),
|
|
||||||
)
|
|
||||||
assert result.count() == 0
|
|
||||||
|
|
||||||
def test_inactive_user_sees_nothing_with_include_granted_false(self):
|
|
||||||
user = User.objects.create_user(username="inactive_owner", is_active=False)
|
|
||||||
TagFactory(owner=user)
|
|
||||||
TagFactory(owner=None)
|
|
||||||
request = APIRequestFactory().get("/")
|
|
||||||
request.user = user
|
|
||||||
|
|
||||||
class _OwnerOnlyFilter(PermittedObjectsFilter):
|
|
||||||
include_granted = False
|
|
||||||
|
|
||||||
result = _OwnerOnlyFilter().filter_queryset(
|
|
||||||
request,
|
|
||||||
Tag.objects.all(),
|
|
||||||
_DummyView(),
|
|
||||||
)
|
|
||||||
assert result.count() == 0
|
|
||||||
|
|||||||
@@ -287,45 +287,6 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
|
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
|
||||||
|
|
||||||
|
|
||||||
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
|
|
||||||
"""
|
|
||||||
Consumption workflows do not run on reprocess, so the remote parser is
|
|
||||||
used only in 'always' mode or when the caller explicitly asks for it.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def setUp(self) -> None:
|
|
||||||
super().setUp()
|
|
||||||
|
|
||||||
patcher = mock.patch("documents.tasks.get_parser_registry")
|
|
||||||
self.mock_registry = patcher.start()
|
|
||||||
self.mock_registry.return_value.get_parser_for_file.return_value = None
|
|
||||||
self.addCleanup(patcher.stop)
|
|
||||||
|
|
||||||
self.doc = Document.objects.create(
|
|
||||||
title="test",
|
|
||||||
content="my document",
|
|
||||||
checksum="wow",
|
|
||||||
mime_type="application/pdf",
|
|
||||||
)
|
|
||||||
|
|
||||||
def _allow_remote(self, **kwargs) -> bool:
|
|
||||||
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
|
|
||||||
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
|
|
||||||
return call_kwargs["allow_remote"]
|
|
||||||
|
|
||||||
@override_settings(REMOTE_OCR_MODE="always")
|
|
||||||
def test_always_mode_allows_remote(self) -> None:
|
|
||||||
self.assertTrue(self._allow_remote())
|
|
||||||
|
|
||||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
|
||||||
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
|
|
||||||
self.assertFalse(self._allow_remote())
|
|
||||||
|
|
||||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
|
||||||
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
|
|
||||||
self.assertTrue(self._allow_remote(remote_ocr=True))
|
|
||||||
|
|
||||||
|
|
||||||
class TestAIIndex(DirectoriesMixin, TestCase):
|
class TestAIIndex(DirectoriesMixin, TestCase):
|
||||||
@override_settings(
|
@override_settings(
|
||||||
AI_ENABLED=True,
|
AI_ENABLED=True,
|
||||||
|
|||||||
@@ -5360,82 +5360,3 @@ class TestDateWorkflowLocalization(
|
|||||||
document = Document.objects.first()
|
document = Document.objects.first()
|
||||||
assert document is not None
|
assert document is not None
|
||||||
assert document.title == expected_title
|
assert document.title == expected_title
|
||||||
|
|
||||||
|
|
||||||
class TestRemoteOCRWorkflowAction(DirectoriesMixin, SampleDirMixin, APITestCase):
|
|
||||||
def _make_workflow(self, trigger_type) -> None:
|
|
||||||
trigger = WorkflowTrigger.objects.create(type=trigger_type)
|
|
||||||
action = WorkflowAction.objects.create(
|
|
||||||
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
|
|
||||||
)
|
|
||||||
w = Workflow.objects.create(name="Remote OCR", order=0)
|
|
||||||
w.triggers.add(trigger)
|
|
||||||
w.actions.add(action)
|
|
||||||
w.save()
|
|
||||||
|
|
||||||
def test_consumption_trigger_requests_remote_ocr(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A consumption workflow with a remote OCR action
|
|
||||||
WHEN:
|
|
||||||
- A matching document is consumed
|
|
||||||
THEN:
|
|
||||||
- The overrides ask for remote OCR, which is what the consumer
|
|
||||||
reads when choosing a parser
|
|
||||||
"""
|
|
||||||
self._make_workflow(WorkflowTrigger.WorkflowTriggerType.CONSUMPTION)
|
|
||||||
|
|
||||||
test_file = shutil.copy(
|
|
||||||
self.SAMPLE_DIR / "simple.pdf",
|
|
||||||
self.dirs.scratch_dir / "simple.pdf",
|
|
||||||
)
|
|
||||||
overrides = DocumentMetadataOverrides()
|
|
||||||
|
|
||||||
run_workflows(
|
|
||||||
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
|
|
||||||
ConsumableDocument(
|
|
||||||
source=DocumentSource.ConsumeFolder,
|
|
||||||
original_file=test_file,
|
|
||||||
),
|
|
||||||
overrides=overrides,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertTrue(overrides.remote_ocr)
|
|
||||||
|
|
||||||
def test_other_trigger_types_are_ignored(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A workflow with a remote OCR action that also has a
|
|
||||||
non-consumption trigger, which is a valid combination
|
|
||||||
WHEN:
|
|
||||||
- The non-consumption trigger fires
|
|
||||||
THEN:
|
|
||||||
- The action is skipped, since the document has already been
|
|
||||||
parsed by this point
|
|
||||||
"""
|
|
||||||
trigger = WorkflowTrigger.objects.create(
|
|
||||||
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
|
|
||||||
)
|
|
||||||
updated_trigger = WorkflowTrigger.objects.create(
|
|
||||||
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
|
|
||||||
)
|
|
||||||
action = WorkflowAction.objects.create(
|
|
||||||
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
|
|
||||||
)
|
|
||||||
w = Workflow.objects.create(name="Remote OCR", order=0)
|
|
||||||
w.triggers.add(trigger, updated_trigger)
|
|
||||||
w.actions.add(action)
|
|
||||||
w.save()
|
|
||||||
|
|
||||||
doc = Document.objects.create(
|
|
||||||
title="sample test",
|
|
||||||
original_filename="sample.pdf",
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertLogs("paperless.handlers", level="DEBUG") as cm:
|
|
||||||
run_workflows(
|
|
||||||
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
|
|
||||||
doc,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertIn("only applies to consumption triggers", "".join(cm.output))
|
|
||||||
|
|||||||
@@ -236,10 +236,8 @@ from paperless import version
|
|||||||
from paperless.celery import app as celery_app
|
from paperless.celery import app as celery_app
|
||||||
from paperless.config import AIConfig
|
from paperless.config import AIConfig
|
||||||
from paperless.config import GeneralConfig
|
from paperless.config import GeneralConfig
|
||||||
from paperless.config import RemoteOCRConfig
|
|
||||||
from paperless.models import ApplicationConfiguration
|
from paperless.models import ApplicationConfiguration
|
||||||
from paperless.parsers.registry import get_parser_registry
|
from paperless.parsers.registry import get_parser_registry
|
||||||
from paperless.parsers.remote import RemoteEngineConfig
|
|
||||||
from paperless.serialisers import GroupSerializer
|
from paperless.serialisers import GroupSerializer
|
||||||
from paperless.serialisers import UserSerializer
|
from paperless.serialisers import UserSerializer
|
||||||
from paperless.views import StandardPagination
|
from paperless.views import StandardPagination
|
||||||
@@ -4012,11 +4010,6 @@ class UiSettingsView(GenericAPIView[Any]):
|
|||||||
|
|
||||||
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
|
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
|
||||||
|
|
||||||
ui_settings["remote_ocr"] = {
|
|
||||||
"configured": RemoteEngineConfig.from_app_config().engine_is_valid(),
|
|
||||||
"mode": RemoteOCRConfig().remote_ocr_mode,
|
|
||||||
}
|
|
||||||
|
|
||||||
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
|
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
|
||||||
manager = PaperlessMailOAuth2Manager()
|
manager = PaperlessMailOAuth2Manager()
|
||||||
if settings.GMAIL_OAUTH_ENABLED:
|
if settings.GMAIL_OAUTH_ENABLED:
|
||||||
|
|||||||
@@ -2,7 +2,7 @@ msgid ""
|
|||||||
msgstr ""
|
msgstr ""
|
||||||
"Project-Id-Version: paperless-ngx\n"
|
"Project-Id-Version: paperless-ngx\n"
|
||||||
"Report-Msgid-Bugs-To: \n"
|
"Report-Msgid-Bugs-To: \n"
|
||||||
"POT-Creation-Date: 2026-08-10 02:25+0000\n"
|
"POT-Creation-Date: 2026-08-08 14:28+0000\n"
|
||||||
"PO-Revision-Date: 2022-02-17 04:17\n"
|
"PO-Revision-Date: 2022-02-17 04:17\n"
|
||||||
"Last-Translator: \n"
|
"Last-Translator: \n"
|
||||||
"Language-Team: English\n"
|
"Language-Team: English\n"
|
||||||
@@ -53,7 +53,7 @@ msgstr ""
|
|||||||
msgid "Maximum nesting depth exceeded."
|
msgid "Maximum nesting depth exceeded."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:1079
|
#: documents/filters.py:1073
|
||||||
msgid "Custom field not found"
|
msgid "Custom field not found"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
|
|||||||
@@ -19,10 +19,7 @@ class AutoLoginMiddleware(MiddlewareMixin):
|
|||||||
if request.path.startswith("/api/token/") and request.method == "POST":
|
if request.path.startswith("/api/token/") and request.method == "POST":
|
||||||
return None
|
return None
|
||||||
try:
|
try:
|
||||||
request.user = User.objects.get(
|
request.user = User.objects.get(username=settings.AUTO_LOGIN_USERNAME)
|
||||||
username=settings.AUTO_LOGIN_USERNAME,
|
|
||||||
is_active=True,
|
|
||||||
)
|
|
||||||
auth.login(
|
auth.login(
|
||||||
request=request,
|
request=request,
|
||||||
user=request.user,
|
user=request.user,
|
||||||
|
|||||||
+3
-17
@@ -339,30 +339,16 @@ def check_deprecated_v2_ocr_env_vars(
|
|||||||
|
|
||||||
@register()
|
@register()
|
||||||
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
|
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
|
||||||
# Import here because checks.py runs before the app registry is ready
|
|
||||||
from paperless.models import RemoteOCRMode
|
|
||||||
|
|
||||||
errors = []
|
|
||||||
|
|
||||||
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
|
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
|
||||||
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
|
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
|
||||||
):
|
):
|
||||||
errors.append(
|
return [
|
||||||
Error(
|
Error(
|
||||||
"Azure AI remote parser requires endpoint and API key to be configured.",
|
"Azure AI remote parser requires endpoint and API key to be configured.",
|
||||||
),
|
),
|
||||||
)
|
]
|
||||||
|
|
||||||
valid_modes = {mode.value for mode in RemoteOCRMode}
|
return []
|
||||||
if settings.REMOTE_OCR_MODE not in valid_modes:
|
|
||||||
errors.append(
|
|
||||||
Error(
|
|
||||||
f"PAPERLESS_REMOTE_OCR_MODE is set to {settings.REMOTE_OCR_MODE!r}, "
|
|
||||||
f"expected one of {sorted(valid_modes)}.",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
return errors
|
|
||||||
|
|
||||||
|
|
||||||
def get_tesseract_langs():
|
def get_tesseract_langs():
|
||||||
|
|||||||
@@ -9,7 +9,6 @@ from paperless.models import CleanChoices
|
|||||||
from paperless.models import ColorConvertChoices
|
from paperless.models import ColorConvertChoices
|
||||||
from paperless.models import ModeChoices
|
from paperless.models import ModeChoices
|
||||||
from paperless.models import OutputTypeChoices
|
from paperless.models import OutputTypeChoices
|
||||||
from paperless.models import RemoteOCRMode
|
|
||||||
|
|
||||||
|
|
||||||
@dataclasses.dataclass
|
@dataclasses.dataclass
|
||||||
@@ -186,45 +185,6 @@ class GeneralConfig(BaseConfig):
|
|||||||
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
||||||
|
|
||||||
|
|
||||||
@dataclasses.dataclass
|
|
||||||
class RemoteOCRConfig(BaseConfig):
|
|
||||||
"""
|
|
||||||
Settings for the remote (cloud) OCR parser
|
|
||||||
"""
|
|
||||||
|
|
||||||
remote_ocr_engine: str | None = dataclasses.field(init=False)
|
|
||||||
remote_ocr_api_key: str | None = dataclasses.field(init=False)
|
|
||||||
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
|
|
||||||
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
app_config = self._get_config_instance()
|
|
||||||
|
|
||||||
self.remote_ocr_engine = (
|
|
||||||
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
|
|
||||||
)
|
|
||||||
self.remote_ocr_api_key = (
|
|
||||||
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
|
|
||||||
)
|
|
||||||
self.remote_ocr_endpoint = (
|
|
||||||
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
|
|
||||||
)
|
|
||||||
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
|
|
||||||
settings.REMOTE_OCR_MODE,
|
|
||||||
)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def remote_ocr_by_default(self) -> bool:
|
|
||||||
"""
|
|
||||||
Whether every supported document goes to the remote engine.
|
|
||||||
|
|
||||||
When False the remote engine is used only for documents that
|
|
||||||
explicitly asked for it, i.e. a workflow matched during consumption or
|
|
||||||
the user ticked the box when reprocessing.
|
|
||||||
"""
|
|
||||||
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
|
|
||||||
|
|
||||||
|
|
||||||
@dataclasses.dataclass
|
@dataclasses.dataclass
|
||||||
class AIConfig(BaseConfig):
|
class AIConfig(BaseConfig):
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -1,44 +0,0 @@
|
|||||||
# Generated by Django 5.2.16 on 2026-08-10 14:37
|
|
||||||
|
|
||||||
from django.db import migrations
|
|
||||||
from django.db import models
|
|
||||||
|
|
||||||
|
|
||||||
class Migration(migrations.Migration):
|
|
||||||
dependencies = [
|
|
||||||
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
|
|
||||||
]
|
|
||||||
|
|
||||||
operations = [
|
|
||||||
migrations.AddField(
|
|
||||||
model_name="applicationconfiguration",
|
|
||||||
name="remote_ocr_api_key",
|
|
||||||
field=models.CharField(
|
|
||||||
blank=True,
|
|
||||||
max_length=1024,
|
|
||||||
null=True,
|
|
||||||
verbose_name="Sets the remote OCR API key",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
migrations.AddField(
|
|
||||||
model_name="applicationconfiguration",
|
|
||||||
name="remote_ocr_endpoint",
|
|
||||||
field=models.CharField(
|
|
||||||
blank=True,
|
|
||||||
max_length=256,
|
|
||||||
null=True,
|
|
||||||
verbose_name="Sets the remote OCR endpoint",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
migrations.AddField(
|
|
||||||
model_name="applicationconfiguration",
|
|
||||||
name="remote_ocr_engine",
|
|
||||||
field=models.CharField(
|
|
||||||
blank=True,
|
|
||||||
choices=[("azureai", "Azure AI Document Intelligence")],
|
|
||||||
max_length=32,
|
|
||||||
null=True,
|
|
||||||
verbose_name="Sets the remote OCR engine",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
]
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Generated by Django 5.2.16 on 2026-08-10 15:43
|
|
||||||
|
|
||||||
from django.db import migrations
|
|
||||||
from django.db import models
|
|
||||||
|
|
||||||
|
|
||||||
class Migration(migrations.Migration):
|
|
||||||
dependencies = [
|
|
||||||
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
|
|
||||||
]
|
|
||||||
|
|
||||||
operations = [
|
|
||||||
migrations.AddField(
|
|
||||||
model_name="applicationconfiguration",
|
|
||||||
name="remote_ocr_mode",
|
|
||||||
field=models.CharField(
|
|
||||||
blank=True,
|
|
||||||
choices=[
|
|
||||||
("always", "All supported documents"),
|
|
||||||
("workflow_only", "Only when a workflow enables it"),
|
|
||||||
],
|
|
||||||
max_length=32,
|
|
||||||
null=True,
|
|
||||||
verbose_name="Sets which documents are sent to the remote OCR engine",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
]
|
|
||||||
@@ -74,23 +74,6 @@ class ColorConvertChoices(models.TextChoices):
|
|||||||
CMYK = ("CMYK", _("CMYK"))
|
CMYK = ("CMYK", _("CMYK"))
|
||||||
|
|
||||||
|
|
||||||
class RemoteOCREngine(models.TextChoices):
|
|
||||||
"""
|
|
||||||
Matches to PAPERLESS_REMOTE_OCR_ENGINE
|
|
||||||
"""
|
|
||||||
|
|
||||||
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
|
|
||||||
|
|
||||||
|
|
||||||
class RemoteOCRMode(models.TextChoices):
|
|
||||||
"""
|
|
||||||
Matches to PAPERLESS_REMOTE_OCR_MODE
|
|
||||||
"""
|
|
||||||
|
|
||||||
ALWAYS = ("always", _("All supported documents"))
|
|
||||||
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
|
|
||||||
|
|
||||||
|
|
||||||
class LLMEmbeddingBackend(models.TextChoices):
|
class LLMEmbeddingBackend(models.TextChoices):
|
||||||
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
|
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
|
||||||
HUGGINGFACE = ("huggingface", _("Huggingface"))
|
HUGGINGFACE = ("huggingface", _("Huggingface"))
|
||||||
@@ -303,44 +286,6 @@ class ApplicationConfiguration(AbstractSingletonModel):
|
|||||||
null=True,
|
null=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
"""
|
|
||||||
Settings for the remote OCR parser
|
|
||||||
"""
|
|
||||||
|
|
||||||
# PAPERLESS_REMOTE_OCR_ENGINE
|
|
||||||
remote_ocr_engine = models.CharField(
|
|
||||||
verbose_name=_("Sets the remote OCR engine"),
|
|
||||||
blank=True,
|
|
||||||
null=True,
|
|
||||||
max_length=32,
|
|
||||||
choices=RemoteOCREngine.choices,
|
|
||||||
)
|
|
||||||
|
|
||||||
# PAPERLESS_REMOTE_OCR_API_KEY
|
|
||||||
remote_ocr_api_key = models.CharField(
|
|
||||||
verbose_name=_("Sets the remote OCR API key"),
|
|
||||||
blank=True,
|
|
||||||
null=True,
|
|
||||||
max_length=1024,
|
|
||||||
)
|
|
||||||
|
|
||||||
# PAPERLESS_REMOTE_OCR_ENDPOINT
|
|
||||||
remote_ocr_endpoint = models.CharField(
|
|
||||||
verbose_name=_("Sets the remote OCR endpoint"),
|
|
||||||
blank=True,
|
|
||||||
null=True,
|
|
||||||
max_length=256,
|
|
||||||
)
|
|
||||||
|
|
||||||
# PAPERLESS_REMOTE_OCR_MODE
|
|
||||||
remote_ocr_mode = models.CharField(
|
|
||||||
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
|
|
||||||
blank=True,
|
|
||||||
null=True,
|
|
||||||
max_length=32,
|
|
||||||
choices=RemoteOCRMode.choices,
|
|
||||||
)
|
|
||||||
|
|
||||||
"""
|
"""
|
||||||
AI related settings
|
AI related settings
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -134,11 +134,6 @@ class ParserProtocol(Protocol):
|
|||||||
Author or organisation name.
|
Author or organisation name.
|
||||||
url : str
|
url : str
|
||||||
URL for documentation, source code, or issue tracker.
|
URL for documentation, source code, or issue tracker.
|
||||||
|
|
||||||
Parsers that send document content to a remote service should additionally
|
|
||||||
set ``uses_remote_service = True`` so the registry can exclude them when
|
|
||||||
remote processing has not been requested for a document. The attribute is
|
|
||||||
optional so a parser that omits it is treated as fully local.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -150,10 +145,6 @@ class ParserProtocol(Protocol):
|
|||||||
author: str
|
author: str
|
||||||
url: str
|
url: str
|
||||||
|
|
||||||
# NOTE: uses_remote_service is not declared here, the registry reads it
|
|
||||||
# with getattr(cls, ..., False) for backwards-compatibility with existing
|
|
||||||
# parsers
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Class methods
|
# Class methods
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|||||||
@@ -334,8 +334,6 @@ class ParserRegistry:
|
|||||||
mime_type: str,
|
mime_type: str,
|
||||||
filename: str,
|
filename: str,
|
||||||
path: Path | None = None,
|
path: Path | None = None,
|
||||||
*,
|
|
||||||
allow_remote: bool = True,
|
|
||||||
) -> type[ParserProtocol] | None:
|
) -> type[ParserProtocol] | None:
|
||||||
"""Return the best parser class for the given file, or None.
|
"""Return the best parser class for the given file, or None.
|
||||||
|
|
||||||
@@ -361,11 +359,6 @@ class ParserRegistry:
|
|||||||
path:
|
path:
|
||||||
Optional filesystem path to the file. Forwarded to each
|
Optional filesystem path to the file. Forwarded to each
|
||||||
parser's score method.
|
parser's score method.
|
||||||
allow_remote:
|
|
||||||
When False, parsers that declare ``uses_remote_service = True``
|
|
||||||
are excluded from consideration, so a document is never sent to
|
|
||||||
a remote service. Parsers that do not declare the attribute
|
|
||||||
are treated as local and are always considered.
|
|
||||||
|
|
||||||
Returns
|
Returns
|
||||||
-------
|
-------
|
||||||
@@ -381,13 +374,6 @@ class ParserRegistry:
|
|||||||
if mime_type not in parser_class.supported_mime_types():
|
if mime_type not in parser_class.supported_mime_types():
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if not allow_remote and getattr(
|
|
||||||
parser_class,
|
|
||||||
"uses_remote_service",
|
|
||||||
False,
|
|
||||||
):
|
|
||||||
continue
|
|
||||||
|
|
||||||
score = parser_class.score(mime_type, filename, path)
|
score = parser_class.score(mime_type, filename, path)
|
||||||
if score is None:
|
if score is None:
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -57,18 +57,6 @@ class RemoteEngineConfig:
|
|||||||
self.api_key = api_key
|
self.api_key = api_key
|
||||||
self.endpoint = endpoint
|
self.endpoint = endpoint
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_app_config(cls) -> Self:
|
|
||||||
"""Build the config from the app config, falling back to the env."""
|
|
||||||
from paperless.config import RemoteOCRConfig
|
|
||||||
|
|
||||||
app_config = RemoteOCRConfig()
|
|
||||||
return cls(
|
|
||||||
engine=app_config.remote_ocr_engine,
|
|
||||||
api_key=app_config.remote_ocr_api_key,
|
|
||||||
endpoint=app_config.remote_ocr_endpoint,
|
|
||||||
)
|
|
||||||
|
|
||||||
def engine_is_valid(self) -> bool:
|
def engine_is_valid(self) -> bool:
|
||||||
"""Return True when the engine is known and fully configured."""
|
"""Return True when the engine is known and fully configured."""
|
||||||
return (
|
return (
|
||||||
@@ -95,9 +83,6 @@ class RemoteDocumentParser:
|
|||||||
Maintainer name.
|
Maintainer name.
|
||||||
url : str
|
url : str
|
||||||
Issue tracker / source URL.
|
Issue tracker / source URL.
|
||||||
uses_remote_service : bool
|
|
||||||
Content is sent to a remote service, True so that the registry
|
|
||||||
can skip this parser if remote processing was not requested.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
name: str = "Paperless-ngx Remote OCR Parser"
|
name: str = "Paperless-ngx Remote OCR Parser"
|
||||||
@@ -105,8 +90,6 @@ class RemoteDocumentParser:
|
|||||||
author: str = "Paperless-ngx Contributors"
|
author: str = "Paperless-ngx Contributors"
|
||||||
url: str = "https://github.com/paperless-ngx/paperless-ngx"
|
url: str = "https://github.com/paperless-ngx/paperless-ngx"
|
||||||
|
|
||||||
uses_remote_service: bool = True
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Class methods
|
# Class methods
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -155,7 +138,11 @@ class RemoteDocumentParser:
|
|||||||
20 when the remote engine is configured and the MIME type is
|
20 when the remote engine is configured and the MIME type is
|
||||||
supported, otherwise None.
|
supported, otherwise None.
|
||||||
"""
|
"""
|
||||||
config = RemoteEngineConfig.from_app_config()
|
config = RemoteEngineConfig(
|
||||||
|
engine=settings.REMOTE_OCR_ENGINE,
|
||||||
|
api_key=settings.REMOTE_OCR_API_KEY,
|
||||||
|
endpoint=settings.REMOTE_OCR_ENDPOINT,
|
||||||
|
)
|
||||||
if not config.engine_is_valid():
|
if not config.engine_is_valid():
|
||||||
return None
|
return None
|
||||||
if mime_type not in _SUPPORTED_MIME_TYPES:
|
if mime_type not in _SUPPORTED_MIME_TYPES:
|
||||||
@@ -241,7 +228,11 @@ class RemoteDocumentParser:
|
|||||||
Ignored — the remote engine always returns a searchable PDF,
|
Ignored — the remote engine always returns a searchable PDF,
|
||||||
which is stored as the archive copy regardless of this flag.
|
which is stored as the archive copy regardless of this flag.
|
||||||
"""
|
"""
|
||||||
config = RemoteEngineConfig.from_app_config()
|
config = RemoteEngineConfig(
|
||||||
|
engine=settings.REMOTE_OCR_ENGINE,
|
||||||
|
api_key=settings.REMOTE_OCR_API_KEY,
|
||||||
|
endpoint=settings.REMOTE_OCR_ENDPOINT,
|
||||||
|
)
|
||||||
|
|
||||||
if not config.engine_is_valid():
|
if not config.engine_is_valid():
|
||||||
logger.warning(
|
logger.warning(
|
||||||
|
|||||||
@@ -219,13 +219,6 @@ class ApplicationConfigurationSerializer(
|
|||||||
allow_null=True,
|
allow_null=True,
|
||||||
max_length=1024,
|
max_length=1024,
|
||||||
)
|
)
|
||||||
remote_ocr_api_key = ObfuscatedPasswordField(
|
|
||||||
required=False,
|
|
||||||
allow_null=True,
|
|
||||||
max_length=1024,
|
|
||||||
)
|
|
||||||
|
|
||||||
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
|
|
||||||
|
|
||||||
def run_validation(self, data):
|
def run_validation(self, data):
|
||||||
# Empty strings treated as None to avoid unexpected behavior
|
# Empty strings treated as None to avoid unexpected behavior
|
||||||
@@ -237,13 +230,11 @@ class ApplicationConfigurationSerializer(
|
|||||||
data["language"] = None
|
data["language"] = None
|
||||||
if "llm_output_language" in data and data["llm_output_language"] == "":
|
if "llm_output_language" in data and data["llm_output_language"] == "":
|
||||||
data["llm_output_language"] = None
|
data["llm_output_language"] = None
|
||||||
for field in self.OBFUSCATED_FIELDS:
|
if "llm_api_key" in data and data["llm_api_key"] is not None:
|
||||||
if field in data and data[field] is not None:
|
if data["llm_api_key"] == "":
|
||||||
if data[field] == "":
|
data["llm_api_key"] = None
|
||||||
data[field] = None
|
elif len(data["llm_api_key"].replace("*", "")) == 0:
|
||||||
# Not a real value, don't overwrite the stored one
|
del data["llm_api_key"]
|
||||||
elif len(data[field].replace("*", "")) == 0:
|
|
||||||
del data[field]
|
|
||||||
return super().run_validation(data)
|
return super().run_validation(data)
|
||||||
|
|
||||||
def update(self, instance, validated_data):
|
def update(self, instance, validated_data):
|
||||||
|
|||||||
@@ -1197,7 +1197,6 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
|
|||||||
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
|
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
|
||||||
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
|
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
|
||||||
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
|
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
|
||||||
REMOTE_OCR_MODE = os.getenv("PAPERLESS_REMOTE_OCR_MODE", "always")
|
|
||||||
|
|
||||||
################################################################################
|
################################################################################
|
||||||
# AI Settings #
|
# AI Settings #
|
||||||
|
|||||||
@@ -21,7 +21,6 @@ from unittest.mock import Mock
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from documents.parsers import ParseError
|
from documents.parsers import ParseError
|
||||||
from paperless.models import ApplicationConfiguration
|
|
||||||
from paperless.parsers import ParserContext
|
from paperless.parsers import ParserContext
|
||||||
from paperless.parsers import ParserProtocol
|
from paperless.parsers import ParserProtocol
|
||||||
from paperless.parsers.remote import RemoteDocumentParser
|
from paperless.parsers.remote import RemoteDocumentParser
|
||||||
@@ -34,10 +33,6 @@ if TYPE_CHECKING:
|
|||||||
from pytest_mock import MockerFixture
|
from pytest_mock import MockerFixture
|
||||||
|
|
||||||
|
|
||||||
# Remote ocr config from ApplicationConfiguration needs DB access
|
|
||||||
pytestmark = pytest.mark.django_db
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Module-local fixtures
|
# Module-local fixtures
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -232,18 +227,6 @@ class TestRemoteParserScore:
|
|||||||
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
|
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
|
||||||
assert score is not None and score > 10
|
assert score is not None and score > 10
|
||||||
|
|
||||||
@pytest.mark.usefixtures("no_engine_settings")
|
|
||||||
def test_score_uses_app_config_when_env_unset(self) -> None:
|
|
||||||
"""The app config alone is enough to activate the parser."""
|
|
||||||
config = ApplicationConfiguration.objects.first()
|
|
||||||
assert config is not None
|
|
||||||
config.remote_ocr_engine = "azureai"
|
|
||||||
config.remote_ocr_api_key = "app-config-key"
|
|
||||||
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
|
|
||||||
config.save()
|
|
||||||
|
|
||||||
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Properties
|
# Properties
|
||||||
|
|||||||
@@ -1277,8 +1277,6 @@ class TestParserFileTypes:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
# Remote ocr config from ApplicationConfiguration needs DB access
|
|
||||||
@pytest.mark.django_db
|
|
||||||
class TestRasterisedDocumentParserRegistry:
|
class TestRasterisedDocumentParserRegistry:
|
||||||
def test_registered_in_defaults(self) -> None:
|
def test_registered_in_defaults(self) -> None:
|
||||||
from paperless.parsers.registry import ParserRegistry
|
from paperless.parsers.registry import ParserRegistry
|
||||||
|
|||||||
@@ -1,53 +0,0 @@
|
|||||||
from django.contrib.auth.models import AnonymousUser
|
|
||||||
from django.contrib.auth.models import User
|
|
||||||
from django.test import RequestFactory
|
|
||||||
from django.test import TestCase
|
|
||||||
from django.test import override_settings
|
|
||||||
|
|
||||||
from paperless.auth import AutoLoginMiddleware
|
|
||||||
|
|
||||||
|
|
||||||
@override_settings(AUTO_LOGIN_USERNAME="autologin")
|
|
||||||
class TestAutoLoginMiddleware(TestCase):
|
|
||||||
def setUp(self) -> None:
|
|
||||||
super().setUp()
|
|
||||||
self.factory = RequestFactory()
|
|
||||||
self.middleware = AutoLoginMiddleware(lambda request: None)
|
|
||||||
|
|
||||||
def _process(self, request):
|
|
||||||
# login() needs a session to write to
|
|
||||||
request.session = self.client.session
|
|
||||||
self.middleware.process_request(request)
|
|
||||||
return request
|
|
||||||
|
|
||||||
def test_active_user_is_logged_in(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- AUTO_LOGIN_USERNAME names an active user
|
|
||||||
WHEN:
|
|
||||||
- A request is processed by the middleware
|
|
||||||
THEN:
|
|
||||||
- That user is attached to the request
|
|
||||||
"""
|
|
||||||
user = User.objects.create_user(username="autologin")
|
|
||||||
|
|
||||||
request = self._process(self.factory.get("/"))
|
|
||||||
|
|
||||||
self.assertEqual(request.user, user)
|
|
||||||
|
|
||||||
def test_deactivated_user_is_not_logged_in(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- AUTO_LOGIN_USERNAME names a user who has been deactivated
|
|
||||||
WHEN:
|
|
||||||
- A request is processed by the middleware
|
|
||||||
THEN:
|
|
||||||
- The request is left anonymous rather than authenticated as them
|
|
||||||
"""
|
|
||||||
User.objects.create_user(username="autologin", is_active=False)
|
|
||||||
|
|
||||||
request = self.factory.get("/")
|
|
||||||
request.user = AnonymousUser()
|
|
||||||
self._process(request)
|
|
||||||
|
|
||||||
self.assertFalse(request.user.is_authenticated)
|
|
||||||
@@ -655,23 +655,6 @@ class TestRemoteParserChecks:
|
|||||||
in msg.msg
|
in msg.msg
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_valid_mode(self, settings: SettingsWrapper) -> None:
|
|
||||||
settings.REMOTE_OCR_ENGINE = None
|
|
||||||
settings.REMOTE_OCR_MODE = "workflow_only"
|
|
||||||
|
|
||||||
msgs = check_remote_parser_configured(None)
|
|
||||||
|
|
||||||
assert len(msgs) == 0
|
|
||||||
|
|
||||||
def test_invalid_mode(self, settings: SettingsWrapper) -> None:
|
|
||||||
settings.REMOTE_OCR_ENGINE = None
|
|
||||||
settings.REMOTE_OCR_MODE = "sometimes"
|
|
||||||
|
|
||||||
msgs = check_remote_parser_configured(None)
|
|
||||||
|
|
||||||
assert len(msgs) == 1
|
|
||||||
assert "PAPERLESS_REMOTE_OCR_MODE is set to 'sometimes'" in msgs[0].msg
|
|
||||||
|
|
||||||
|
|
||||||
class TestTesseractChecks:
|
class TestTesseractChecks:
|
||||||
def test_default_language(self) -> None:
|
def test_default_language(self) -> None:
|
||||||
|
|||||||
@@ -468,124 +468,6 @@ class TestParserRegistryGetParserForFile:
|
|||||||
assert result is AcceptingBuiltin
|
assert result is AcceptingBuiltin
|
||||||
|
|
||||||
|
|
||||||
class TestParserRegistryRemoteParsers:
|
|
||||||
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _remote_parser_cls() -> type:
|
|
||||||
class RemoteParser:
|
|
||||||
name = "remote"
|
|
||||||
version = "1.0"
|
|
||||||
author = "A"
|
|
||||||
url = "https://example.com/remote"
|
|
||||||
uses_remote_service = True
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def supported_mime_types(cls):
|
|
||||||
return {"text/plain": ".txt"}
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def score(cls, mime_type, filename, path=None):
|
|
||||||
return 20
|
|
||||||
|
|
||||||
return RemoteParser
|
|
||||||
|
|
||||||
def test_remote_parser_wins_when_remote_allowed(
|
|
||||||
self,
|
|
||||||
dummy_parser_cls: type,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
|
|
||||||
WHEN: get_parser_for_file() is called with allow_remote=True.
|
|
||||||
THEN: The remote parser is returned.
|
|
||||||
"""
|
|
||||||
remote_parser_cls = self._remote_parser_cls()
|
|
||||||
registry = ParserRegistry()
|
|
||||||
registry.register_builtin(dummy_parser_cls)
|
|
||||||
registry.register_builtin(remote_parser_cls)
|
|
||||||
|
|
||||||
result = registry.get_parser_for_file(
|
|
||||||
"text/plain",
|
|
||||||
"readme.txt",
|
|
||||||
allow_remote=True,
|
|
||||||
)
|
|
||||||
assert result is remote_parser_cls
|
|
||||||
|
|
||||||
def test_remote_parser_skipped_when_remote_not_allowed(
|
|
||||||
self,
|
|
||||||
dummy_parser_cls: type,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
|
|
||||||
WHEN: get_parser_for_file() is called with allow_remote=False.
|
|
||||||
THEN: The local parser is returned despite its lower score.
|
|
||||||
"""
|
|
||||||
registry = ParserRegistry()
|
|
||||||
registry.register_builtin(dummy_parser_cls)
|
|
||||||
registry.register_builtin(self._remote_parser_cls())
|
|
||||||
|
|
||||||
result = registry.get_parser_for_file(
|
|
||||||
"text/plain",
|
|
||||||
"readme.txt",
|
|
||||||
allow_remote=False,
|
|
||||||
)
|
|
||||||
assert result is dummy_parser_cls
|
|
||||||
|
|
||||||
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: A registry whose only candidate declares uses_remote_service.
|
|
||||||
WHEN: get_parser_for_file() is called with allow_remote=False.
|
|
||||||
THEN: None is returned — the remote parser is never used as a
|
|
||||||
fallback when remote processing was not requested.
|
|
||||||
"""
|
|
||||||
registry = ParserRegistry()
|
|
||||||
registry.register_builtin(self._remote_parser_cls())
|
|
||||||
|
|
||||||
result = registry.get_parser_for_file(
|
|
||||||
"text/plain",
|
|
||||||
"readme.txt",
|
|
||||||
allow_remote=False,
|
|
||||||
)
|
|
||||||
assert result is None
|
|
||||||
|
|
||||||
def test_parser_without_attribute_treated_as_local(
|
|
||||||
self,
|
|
||||||
dummy_parser_cls: type,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: A third-party parser predating uses_remote_service, so it does
|
|
||||||
not declare the attribute at all.
|
|
||||||
WHEN: get_parser_for_file() is called with allow_remote=False.
|
|
||||||
THEN: It is still considered, i.e. treated as fully local, rather
|
|
||||||
than raising AttributeError.
|
|
||||||
"""
|
|
||||||
assert not hasattr(dummy_parser_cls, "uses_remote_service")
|
|
||||||
|
|
||||||
registry = ParserRegistry()
|
|
||||||
registry.register_builtin(dummy_parser_cls)
|
|
||||||
|
|
||||||
result = registry.get_parser_for_file(
|
|
||||||
"text/plain",
|
|
||||||
"readme.txt",
|
|
||||||
allow_remote=False,
|
|
||||||
)
|
|
||||||
assert result is dummy_parser_cls
|
|
||||||
|
|
||||||
def test_remote_allowed_by_default(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN: A registry containing only a remote parser.
|
|
||||||
WHEN: get_parser_for_file() is called without allow_remote.
|
|
||||||
THEN: The remote parser is returned — callers that do not opt in to
|
|
||||||
the filter keep the previous behaviour.
|
|
||||||
"""
|
|
||||||
remote_parser_cls = self._remote_parser_cls()
|
|
||||||
registry = ParserRegistry()
|
|
||||||
registry.register_builtin(remote_parser_cls)
|
|
||||||
|
|
||||||
result = registry.get_parser_for_file("text/plain", "readme.txt")
|
|
||||||
assert result is remote_parser_cls
|
|
||||||
|
|
||||||
|
|
||||||
class TestDiscover:
|
class TestDiscover:
|
||||||
"""Verify entrypoint discovery in ParserRegistry.discover()."""
|
"""Verify entrypoint discovery in ParserRegistry.discover()."""
|
||||||
|
|
||||||
|
|||||||
@@ -1,113 +0,0 @@
|
|||||||
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import TYPE_CHECKING
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from django.test import override_settings
|
|
||||||
|
|
||||||
from paperless.config import RemoteOCRConfig
|
|
||||||
from paperless.models import RemoteOCRMode
|
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
|
||||||
from unittest.mock import MagicMock
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture()
|
|
||||||
def null_app_config(mocker) -> MagicMock:
|
|
||||||
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
|
|
||||||
return mocker.MagicMock(
|
|
||||||
remote_ocr_engine=None,
|
|
||||||
remote_ocr_api_key=None,
|
|
||||||
remote_ocr_endpoint=None,
|
|
||||||
remote_ocr_mode=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture()
|
|
||||||
def make_remote_ocr_config(mocker):
|
|
||||||
def _make(app_config, **django_settings_overrides):
|
|
||||||
mocker.patch(
|
|
||||||
"paperless.config.BaseConfig._get_config_instance",
|
|
||||||
return_value=app_config,
|
|
||||||
)
|
|
||||||
with override_settings(**django_settings_overrides):
|
|
||||||
return RemoteOCRConfig()
|
|
||||||
|
|
||||||
return _make
|
|
||||||
|
|
||||||
|
|
||||||
class TestRemoteOCRConfig:
|
|
||||||
def test_falls_back_to_settings(
|
|
||||||
self,
|
|
||||||
make_remote_ocr_config,
|
|
||||||
null_app_config,
|
|
||||||
) -> None:
|
|
||||||
cfg = make_remote_ocr_config(
|
|
||||||
null_app_config,
|
|
||||||
REMOTE_OCR_ENGINE="azureai",
|
|
||||||
REMOTE_OCR_API_KEY="env-key",
|
|
||||||
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
|
|
||||||
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
|
|
||||||
)
|
|
||||||
assert cfg.remote_ocr_engine == "azureai"
|
|
||||||
assert cfg.remote_ocr_api_key == "env-key"
|
|
||||||
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
|
|
||||||
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
|
|
||||||
|
|
||||||
def test_app_config_takes_precedence(
|
|
||||||
self,
|
|
||||||
make_remote_ocr_config,
|
|
||||||
mocker,
|
|
||||||
) -> None:
|
|
||||||
app_config = mocker.MagicMock(
|
|
||||||
remote_ocr_engine="azureai",
|
|
||||||
remote_ocr_api_key="db-key",
|
|
||||||
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
|
|
||||||
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
|
|
||||||
)
|
|
||||||
cfg = make_remote_ocr_config(
|
|
||||||
app_config,
|
|
||||||
REMOTE_OCR_ENGINE=None,
|
|
||||||
REMOTE_OCR_API_KEY="env-key",
|
|
||||||
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
|
|
||||||
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
|
|
||||||
)
|
|
||||||
assert cfg.remote_ocr_engine == "azureai"
|
|
||||||
assert cfg.remote_ocr_api_key == "db-key"
|
|
||||||
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
|
|
||||||
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
|
|
||||||
|
|
||||||
def test_unset_everywhere(
|
|
||||||
self,
|
|
||||||
make_remote_ocr_config,
|
|
||||||
null_app_config,
|
|
||||||
) -> None:
|
|
||||||
cfg = make_remote_ocr_config(
|
|
||||||
null_app_config,
|
|
||||||
REMOTE_OCR_ENGINE=None,
|
|
||||||
REMOTE_OCR_API_KEY=None,
|
|
||||||
REMOTE_OCR_ENDPOINT=None,
|
|
||||||
)
|
|
||||||
assert cfg.remote_ocr_engine is None
|
|
||||||
assert cfg.remote_ocr_api_key is None
|
|
||||||
assert cfg.remote_ocr_endpoint is None
|
|
||||||
|
|
||||||
|
|
||||||
class TestRemoteOCRByDefault:
|
|
||||||
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
|
|
||||||
cfg = make_remote_ocr_config(
|
|
||||||
null_app_config,
|
|
||||||
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert cfg.remote_ocr_by_default is True
|
|
||||||
|
|
||||||
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
|
|
||||||
cfg = make_remote_ocr_config(
|
|
||||||
null_app_config,
|
|
||||||
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert cfg.remote_ocr_by_default is False
|
|
||||||
@@ -1298,16 +1298,16 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "fpdf2"
|
name = "fpdf2"
|
||||||
version = "2.8.8"
|
version = "2.8.7"
|
||||||
source = { registry = "https://pypi.org/simple" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
]
|
]
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/1e/bc/8fd4321aed40cadadddc8f311c65b6082346b252bca048f7b476d8f35d72/fpdf2-2.8.8.tar.gz", hash = "sha256:9e94e155e85e8053329a9a1fce8b566fd7a7c5bb79e98a1a3952d379b947c5b9", size = 374689, upload-time = "2026-08-09T23:32:45.334Z" }
|
sdist = { url = "https://files.pythonhosted.org/packages/27/f2/72feae0b2827ed38013e4307b14f95bf0b3d124adfef4d38a7d57533f7be/fpdf2-2.8.7.tar.gz", hash = "sha256:7060ccee5a9c7ab0a271fb765a36a23639f83ef8996c34e3d46af0a17ede57f9", size = 362351, upload-time = "2026-02-28T05:39:16.456Z" }
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/f5/be/af012eda9507494f28b99b077423806c43a11573eb6225dd46f19ae2d263/fpdf2-2.8.8-py3-none-any.whl", hash = "sha256:3557a478fc577a929c94aace9666aed4dcc432b5ab6764232e6a59f1ccd75f17", size = 337000, upload-time = "2026-08-09T23:32:43.728Z" },
|
{ url = "https://files.pythonhosted.org/packages/66/0a/cf50ecffa1e3747ed9380a3adfc829259f1f86b3fdbd9e505af789003141/fpdf2-2.8.7-py3-none-any.whl", hash = "sha256:d391fc508a3ce02fc43a577c830cda4fe6f37646f2d143d489839940932fbc19", size = 327056, upload-time = "2026-02-28T05:39:14.619Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -2927,8 +2927,8 @@ dependencies = [
|
|||||||
{ name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
|
||||||
{ name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
@@ -4511,8 +4511,8 @@ dependencies = [
|
|||||||
{ name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
|
||||||
{ name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
@@ -4957,17 +4957,18 @@ name = "torch"
|
|||||||
version = "2.13.0"
|
version = "2.13.0"
|
||||||
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
||||||
resolution-markers = [
|
resolution-markers = [
|
||||||
|
"python_full_version >= '3.15' and sys_platform == 'darwin'",
|
||||||
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
|
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
|
||||||
"python_full_version < '3.12' and sys_platform == 'darwin'",
|
"python_full_version < '3.12' and sys_platform == 'darwin'",
|
||||||
]
|
]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "filelock", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "filelock", marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "fsspec", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "fsspec", marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "jinja2", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "jinja2", marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "networkx", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "networkx", marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "setuptools", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "setuptools", marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "sympy", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "sympy", marker = "sys_platform == 'darwin'" },
|
||||||
{ name = "typing-extensions", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
{ name = "typing-extensions", marker = "sys_platform == 'darwin'" },
|
||||||
]
|
]
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" },
|
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" },
|
||||||
@@ -4982,7 +4983,6 @@ name = "torch"
|
|||||||
version = "2.13.0+cpu"
|
version = "2.13.0+cpu"
|
||||||
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
||||||
resolution-markers = [
|
resolution-markers = [
|
||||||
"python_full_version >= '3.15' and sys_platform == 'darwin'",
|
|
||||||
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
|
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
|
||||||
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
|
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
|
||||||
"python_full_version >= '3.15' and sys_platform == 'linux'",
|
"python_full_version >= '3.15' and sys_platform == 'linux'",
|
||||||
@@ -4990,13 +4990,13 @@ resolution-markers = [
|
|||||||
"python_full_version < '3.12' and sys_platform == 'linux'",
|
"python_full_version < '3.12' and sys_platform == 'linux'",
|
||||||
]
|
]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "filelock", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "filelock", marker = "sys_platform == 'linux'" },
|
||||||
{ name = "fsspec", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "fsspec", marker = "sys_platform == 'linux'" },
|
||||||
{ name = "jinja2", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "jinja2", marker = "sys_platform == 'linux'" },
|
||||||
{ name = "networkx", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "networkx", marker = "sys_platform == 'linux'" },
|
||||||
{ name = "setuptools", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "setuptools", marker = "sys_platform == 'linux'" },
|
||||||
{ name = "sympy", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "sympy", marker = "sys_platform == 'linux'" },
|
||||||
{ name = "typing-extensions", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
{ name = "typing-extensions", marker = "sys_platform == 'linux'" },
|
||||||
]
|
]
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" },
|
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" },
|
||||||
|
|||||||
Reference in New Issue
Block a user