mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-12 22:03:19 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f839db6ecf | ||
|
|
4789fe9a52 | ||
|
|
e150c8c7c0 | ||
|
|
879cd4a30a | ||
|
|
6a02b87dde | ||
|
|
59a2651804 | ||
|
|
a99f63e059 | ||
|
|
939cb52f6e | ||
|
|
855669ddf9 |
@@ -699,6 +699,7 @@ document_fuzzy_match [--ratio] [--processes N]
|
||||
| --ratio | No | 85.0 | a number between 0 and 100, setting how similar a document must be for it to be reported. Higher numbers mean more similarity. |
|
||||
| --processes | No | 1/4 of system cores | Number of processes to use for matching. Setting 1 disables multiple processes |
|
||||
| --delete | No | False | If provided, one document of a matched pair above the ratio will be deleted. |
|
||||
| --url | No | blank | If an instance URL is provided, the output table will show URLs to each documents instead of the document ID and name. |
|
||||
|
||||
!!! warning
|
||||
|
||||
|
||||
+1
-2
@@ -301,8 +301,7 @@ The following methods are supported:
|
||||
- `delete`
|
||||
- No `parameters` required
|
||||
- `reprocess`
|
||||
- Optional `parameters`: `{ "remote_ocr": true }` to send the documents to the
|
||||
remote OCR engine, see [Remote OCR](usage.md#remote-ocr). Defaults to false.
|
||||
- No `parameters` required
|
||||
- `set_permissions`
|
||||
- Requires `parameters`:
|
||||
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
|
||||
|
||||
+5
-16
@@ -948,10 +948,11 @@ for display in the web interface.
|
||||
|
||||
!!! note
|
||||
|
||||
The **remote OCR parser** (Azure AI) always produces a searchable
|
||||
PDF and stores it as the archive copy, regardless of this setting.
|
||||
`ARCHIVE_FILE_GENERATION=never` has no effect when the remote
|
||||
parser handles a document.
|
||||
The **remote OCR parser** (Azure AI) also honors this setting: when
|
||||
no archive is requested (`never`, or `auto` with a born-digital PDF),
|
||||
the remote engine is skipped entirely and locally-extracted text is
|
||||
used instead, avoiding an unnecessary API call and a duplicate text
|
||||
layer.
|
||||
|
||||
#### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
|
||||
|
||||
@@ -2047,18 +2048,6 @@ password. All of these options come from their similarly-named [Django settings]
|
||||
|
||||
Defaults to None.
|
||||
|
||||
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
|
||||
|
||||
: Which documents are sent to the remote OCR engine.
|
||||
|
||||
- `always`: every document of a supported file type is sent to the remote
|
||||
engine, bypassing the local OCR engine.
|
||||
- `workflow_only`: documents are processed locally unless a workflow
|
||||
explicitly enables remote OCR for them, letting you use the remote engine
|
||||
selectively.
|
||||
|
||||
Defaults to "always".
|
||||
|
||||
## AI {#ai}
|
||||
|
||||
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
|
||||
|
||||
@@ -456,20 +456,6 @@ def score(
|
||||
return 10
|
||||
```
|
||||
|
||||
**Remote services**
|
||||
|
||||
If your parser sends document content to a remote service, declare it:
|
||||
|
||||
```python
|
||||
class MyCustomParser:
|
||||
uses_remote_service = True
|
||||
```
|
||||
|
||||
Paperless-ngx excludes such parsers when the document being consumed has not
|
||||
been marked for remote processing, so users can keep remote OCR off by default
|
||||
and enable it selectively with a workflow. Parsers that do not declare the
|
||||
attribute are treated as fully local and are always considered.
|
||||
|
||||
**Archive and rendition flags**
|
||||
|
||||
```python
|
||||
|
||||
@@ -187,10 +187,11 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
|
||||
|
||||
### Remote OCR parser
|
||||
|
||||
If you use the **remote OCR parser** (Azure AI), note that it always produces a
|
||||
searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
|
||||
has no effect for documents handled by the remote parser - the archive is produced
|
||||
unconditionally by the remote engine.
|
||||
If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
|
||||
honored the same way as for the local engine: when no archive is requested
|
||||
(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
|
||||
entirely and locally-extracted text is used instead, avoiding an unnecessary
|
||||
API call and a duplicate text layer.
|
||||
|
||||
## Search Index (Whoosh -> Tantivy)
|
||||
|
||||
|
||||
+4
-9
@@ -576,7 +576,9 @@ The following workflow action types are available:
|
||||
- Tags, correspondent, document type and storage path
|
||||
- Document owner
|
||||
- View and / or edit permissions to users or groups
|
||||
- Custom fields. Note that no value for the field will be set
|
||||
- Custom fields, optionally with a value. If no value is set, the field is only added to the
|
||||
document and any value it may already have is left untouched. If a value is set, it will
|
||||
overwrite an existing value of that field on the document.
|
||||
|
||||
##### Removal {#workflow-action-removal}
|
||||
|
||||
@@ -1084,18 +1086,11 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
|
||||
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
|
||||
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
|
||||
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
|
||||
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
|
||||
settings can be supplied as environment variables or via **Application Configuration**.
|
||||
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
|
||||
|
||||
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
|
||||
or page limitations (e.g. with a free tier).
|
||||
|
||||
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
|
||||
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
|
||||
unless a workflow explicitly enables remote OCR for them, so you can limit the remote engine to particular documents.
|
||||
|
||||
Setting the mode to `workflow_only` also allows the **Reprocess** actions to selectively use remote OCR for individual documents.
|
||||
|
||||
## Architecture
|
||||
|
||||
Paperless-ngx consists of the following components:
|
||||
|
||||
+491
-99
File diff suppressed because it is too large
Load Diff
@@ -14,48 +14,43 @@
|
||||
<a ngbNavLink>{{category}}</a>
|
||||
<ng-template ngbNavContent>
|
||||
<div class="p-3">
|
||||
@for (section of getCategorySections(category); track section) {
|
||||
@if (section) {
|
||||
<h5 class="mt-4 mb-3">{{section}}</h5>
|
||||
}
|
||||
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
|
||||
@for (option of getCategoryOptions(category, section); track option.key) {
|
||||
<div class="col">
|
||||
<div class="card bg-light">
|
||||
<div class="card-body">
|
||||
<div class="card-title d-flex align-items-center">
|
||||
<h6 class="mb-0">
|
||||
{{option.title}}
|
||||
</h6>
|
||||
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
|
||||
<i-bs name="info-circle"></i-bs>
|
||||
</a>
|
||||
@if (isSet(option.key)) {
|
||||
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
|
||||
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
|
||||
</button>
|
||||
}
|
||||
</div>
|
||||
<div class="mb-n3">
|
||||
@switch (option.type) {
|
||||
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
|
||||
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
|
||||
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
|
||||
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
||||
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
||||
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
|
||||
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
|
||||
}
|
||||
</div>
|
||||
@if (option.note) {
|
||||
<div class="form-text fst-italic">{{option.note}}</div>
|
||||
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
|
||||
@for (option of getCategoryOptions(category); track option.key) {
|
||||
<div class="col">
|
||||
<div class="card bg-light">
|
||||
<div class="card-body">
|
||||
<div class="card-title d-flex align-items-center">
|
||||
<h6 class="mb-0">
|
||||
{{option.title}}
|
||||
</h6>
|
||||
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
|
||||
<i-bs name="info-circle"></i-bs>
|
||||
</a>
|
||||
@if (isSet(option.key)) {
|
||||
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
|
||||
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
|
||||
</button>
|
||||
}
|
||||
</div>
|
||||
<div class="mb-n3">
|
||||
@switch (option.type) {
|
||||
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
|
||||
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
|
||||
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
|
||||
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
||||
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
||||
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
|
||||
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
|
||||
}
|
||||
</div>
|
||||
@if (option.note) {
|
||||
<div class="form-text fst-italic">{{option.note}}</div>
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
</ng-template>
|
||||
</li>
|
||||
|
||||
@@ -8,11 +8,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
|
||||
import { NgSelectModule } from '@ng-select/ng-select'
|
||||
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
||||
import { of, throwError } from 'rxjs'
|
||||
import {
|
||||
ConfigCategory,
|
||||
ConfigSection,
|
||||
OutputTypeConfig,
|
||||
} from 'src/app/data/paperless-config'
|
||||
import { OutputTypeConfig } from 'src/app/data/paperless-config'
|
||||
import { ConfigService } from 'src/app/services/config.service'
|
||||
import { SettingsService } from 'src/app/services/settings.service'
|
||||
import { ToastService } from 'src/app/services/toast.service'
|
||||
@@ -162,24 +158,4 @@ describe('ConfigComponent', () => {
|
||||
component.resetOption('barcodes_enabled')
|
||||
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
|
||||
})
|
||||
|
||||
it('should group options into sections within a category, or not', () => {
|
||||
const sections = component.getCategorySections(ConfigCategory.OCR)
|
||||
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
|
||||
expect(
|
||||
component
|
||||
.getCategoryOptions(ConfigCategory.OCR)
|
||||
.map((option) => option.key)
|
||||
).toContain('output_type')
|
||||
expect(
|
||||
component
|
||||
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
|
||||
.map((option) => option.key)
|
||||
).toEqual([
|
||||
'remote_ocr_engine',
|
||||
'remote_ocr_api_key',
|
||||
'remote_ocr_endpoint',
|
||||
'remote_ocr_mode',
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -74,20 +74,8 @@ export class ConfigComponent
|
||||
return Object.values(ConfigCategory)
|
||||
}
|
||||
|
||||
getCategorySections(category: string): string[] {
|
||||
return [
|
||||
...new Set(
|
||||
PaperlessConfigOptions.filter((o) => o.category === category).map(
|
||||
(o) => o.section ?? null // null means no section
|
||||
)
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
|
||||
return PaperlessConfigOptions.filter(
|
||||
(o) => o.category === category && (o.section ?? null) === section
|
||||
)
|
||||
getCategoryOptions(category: string): ConfigOption[] {
|
||||
return PaperlessConfigOptions.filter((o) => o.category === category)
|
||||
}
|
||||
|
||||
initialConfig: PaperlessConfig
|
||||
|
||||
@@ -111,7 +111,7 @@
|
||||
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="view.name"
|
||||
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
|
||||
popoverClass="popover-slim">
|
||||
<i-bs class="me-2" name="funnel"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
|
||||
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
|
||||
@if (showSidebarCounts && !slimSidebarEnabled) {
|
||||
<span class="badge bg-info text-dark ms-2 d-inline">{{ savedViewService.getDocumentCount(view) }}</span>
|
||||
}
|
||||
|
||||
-28
@@ -1,28 +0,0 @@
|
||||
<div class="modal-header">
|
||||
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
|
||||
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
|
||||
</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
@if (messageBold) {
|
||||
<p class="text-break"><b>{{messageBold}}</b></p>
|
||||
}
|
||||
@if (message) {
|
||||
<p class="mb-0 text-break" [innerHTML]="message"></p>
|
||||
}
|
||||
@if (showRemoteOcr) {
|
||||
<div class="form-check mt-3">
|
||||
<input class="form-check-input" type="checkbox" id="reprocessRemoteOcr" [(ngModel)]="remoteOcr" />
|
||||
<label class="form-check-label" for="reprocessRemoteOcr" i18n>Use remote OCR</label>
|
||||
<div class="form-text" i18n>Sends the document to the configured remote OCR service, which may incur costs.</div>
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
|
||||
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
|
||||
</button>
|
||||
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
|
||||
{{btnCaption}}
|
||||
</button>
|
||||
</div>
|
||||
-72
@@ -1,72 +0,0 @@
|
||||
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
|
||||
import { provideHttpClientTesting } from '@angular/common/http/testing'
|
||||
import { ComponentFixture, TestBed } from '@angular/core/testing'
|
||||
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
|
||||
import { RemoteOCRModeConfig } from 'src/app/data/paperless-config'
|
||||
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
|
||||
import { SettingsService } from 'src/app/services/settings.service'
|
||||
import { ReprocessConfirmDialogComponent } from './reprocess-confirm-dialog.component'
|
||||
|
||||
describe('ReprocessConfirmDialogComponent', () => {
|
||||
let component: ReprocessConfirmDialogComponent
|
||||
let fixture: ComponentFixture<ReprocessConfirmDialogComponent>
|
||||
let settingsService: SettingsService
|
||||
|
||||
const createComponent = (configured: boolean, mode: string) => {
|
||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, configured)
|
||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_MODE, mode)
|
||||
|
||||
fixture = TestBed.createComponent(ReprocessConfirmDialogComponent)
|
||||
component = fixture.componentInstance
|
||||
fixture.detectChanges()
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
TestBed.configureTestingModule({
|
||||
providers: [
|
||||
NgbActiveModal,
|
||||
provideHttpClient(withInterceptorsFromDi()),
|
||||
provideHttpClientTesting(),
|
||||
],
|
||||
imports: [ReprocessConfirmDialogComponent],
|
||||
}).compileComponents()
|
||||
|
||||
settingsService = TestBed.inject(SettingsService)
|
||||
})
|
||||
|
||||
it('should not request remote OCR by default', () => {
|
||||
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
|
||||
|
||||
expect(component.remoteOcr).toBeFalsy()
|
||||
})
|
||||
|
||||
it('should not offer remote OCR when no engine is configured', () => {
|
||||
createComponent(false, RemoteOCRModeConfig.WORKFLOW_ONLY)
|
||||
|
||||
expect(component.showRemoteOcr).toBeFalsy()
|
||||
expect(
|
||||
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
|
||||
).toBeNull()
|
||||
})
|
||||
|
||||
it('should not offer remote OCR when it already handles every document', () => {
|
||||
createComponent(true, RemoteOCRModeConfig.ALWAYS)
|
||||
|
||||
expect(component.showRemoteOcr).toBeFalsy()
|
||||
expect(
|
||||
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
|
||||
).toBeNull()
|
||||
})
|
||||
|
||||
it('should offer remote OCR when configured and selective', () => {
|
||||
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
|
||||
|
||||
expect(component.showRemoteOcr).toBeTruthy()
|
||||
const checkbox = fixture.nativeElement.querySelector('#reprocessRemoteOcr')
|
||||
expect(checkbox).not.toBeNull()
|
||||
|
||||
checkbox.click()
|
||||
fixture.detectChanges()
|
||||
expect(component.remoteOcr).toBeTruthy()
|
||||
})
|
||||
})
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
import { Component, inject } from '@angular/core'
|
||||
import { FormsModule } from '@angular/forms'
|
||||
import { SettingsService } from 'src/app/services/settings.service'
|
||||
import { ConfirmDialogComponent } from '../confirm-dialog.component'
|
||||
|
||||
@Component({
|
||||
selector: 'pngx-reprocess-confirm-dialog',
|
||||
templateUrl: './reprocess-confirm-dialog.component.html',
|
||||
imports: [FormsModule],
|
||||
})
|
||||
export class ReprocessConfirmDialogComponent extends ConfirmDialogComponent {
|
||||
private settings = inject(SettingsService)
|
||||
|
||||
remoteOcr: boolean = false
|
||||
|
||||
public get showRemoteOcr(): boolean {
|
||||
// Hidden when it is not configured, or when it already handles every document anyway.
|
||||
return this.settings.remoteOCRIsSelectable
|
||||
}
|
||||
}
|
||||
+4
-4
@@ -52,10 +52,10 @@ describe('CustomFieldsValuesComponent', () => {
|
||||
})
|
||||
|
||||
it('should set selectedFields and map values correctly', () => {
|
||||
component.value = { 1: 'value1' }
|
||||
component.selectedFields = [1, 2]
|
||||
expect(component.selectedFields).toEqual([1, 2])
|
||||
expect(component.value).toEqual({ 1: 'value1', 2: null })
|
||||
component.value = { 1: 'value1', 3: 0, 4: false }
|
||||
component.selectedFields = [1, 2, 3, 4]
|
||||
expect(component.selectedFields).toEqual([1, 2, 3, 4])
|
||||
expect(component.value).toEqual({ 1: 'value1', 2: null, 3: 0, 4: false })
|
||||
})
|
||||
|
||||
it('should return the correct custom field by id', () => {
|
||||
|
||||
+1
-1
@@ -77,7 +77,7 @@ export class CustomFieldsValuesComponent extends AbstractInputComponent<Object>
|
||||
this._selectedFields = newFields
|
||||
// map the selected fields to an object with field_id as key and value as value
|
||||
this.value = newFields.reduce((acc, fieldId) => {
|
||||
acc[fieldId] = this.value?.[fieldId] || null
|
||||
acc[fieldId] = this.value?.[fieldId] ?? null
|
||||
return acc
|
||||
}, {})
|
||||
this.onChange(this.value)
|
||||
|
||||
@@ -36,7 +36,16 @@
|
||||
(focus)="clearLastSearchTerm()"
|
||||
(clear)="clearLastSearchTerm()"
|
||||
(blur)="onBlur()">
|
||||
<ng-template ng-label-tmp let-item="item">
|
||||
@if (iconField && item[iconField]) {
|
||||
<i-bs class="me-2" [name]="item[iconField]"></i-bs>
|
||||
}
|
||||
<span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
|
||||
</ng-template>
|
||||
<ng-template ng-option-tmp let-item="item">
|
||||
@if (iconField && item[iconField]) {
|
||||
<i-bs class="me-2" [name]="item[iconField]"></i-bs>
|
||||
}
|
||||
<span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
|
||||
</ng-template>
|
||||
</ng-select>
|
||||
|
||||
@@ -35,7 +35,7 @@ import { AbstractInputComponent } from '../abstract-input'
|
||||
NgxBootstrapIconsModule,
|
||||
],
|
||||
})
|
||||
export class SelectComponent extends AbstractInputComponent<number> {
|
||||
export class SelectComponent extends AbstractInputComponent<number | string> {
|
||||
constructor() {
|
||||
super()
|
||||
this.addItemRef = this.addItem.bind(this)
|
||||
@@ -100,6 +100,9 @@ export class SelectComponent extends AbstractInputComponent<number> {
|
||||
@Input()
|
||||
bindLabel: string = 'name'
|
||||
|
||||
@Input()
|
||||
iconField: string
|
||||
|
||||
public searchFn = (term: string, item: any): boolean =>
|
||||
matchesSearchText(item?.[this.bindLabel], term)
|
||||
|
||||
|
||||
+1
@@ -1,6 +1,7 @@
|
||||
<pngx-widget-frame
|
||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Document }"
|
||||
[title]="savedView.name"
|
||||
[titleIcon]="savedView.icon || 'funnel'"
|
||||
[loading]="false"
|
||||
[draggable]="savedView"
|
||||
>
|
||||
|
||||
+6
-1
@@ -8,7 +8,12 @@
|
||||
<i-bs name="grip-vertical"></i-bs>
|
||||
</div>
|
||||
}
|
||||
<h6 class="card-title mb-0">{{title()}}</h6>
|
||||
<h6 class="card-title mb-0">
|
||||
@if (titleIcon()) {
|
||||
<i-bs class="me-2" [name]="titleIcon()"></i-bs>
|
||||
}
|
||||
{{title()}}
|
||||
</h6>
|
||||
<ng-content select="[title-badge]"></ng-content>
|
||||
@if (badge() !== null && badge() !== undefined) {
|
||||
<span class="badge bg-info text-dark ms-2">{{badge()}}</span>
|
||||
|
||||
@@ -16,6 +16,8 @@ export class WidgetFrameComponent implements AfterViewInit {
|
||||
|
||||
title = input<string>()
|
||||
|
||||
titleIcon = input<string>()
|
||||
|
||||
draggable = input<any>()
|
||||
|
||||
cardless = input(false)
|
||||
|
||||
@@ -963,24 +963,12 @@ describe('DocumentDetailComponent', () => {
|
||||
component.reprocess()
|
||||
const modalCloseSpy = jest.spyOn(openModal, 'close')
|
||||
openModal.componentInstance.confirmClicked.next()
|
||||
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, false)
|
||||
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] })
|
||||
expect(modalSpy).toHaveBeenCalled()
|
||||
expect(toastSpy).toHaveBeenCalled()
|
||||
expect(modalCloseSpy).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('should pass remote OCR choice when reprocessing', () => {
|
||||
initNormally()
|
||||
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
|
||||
reprocessSpy.mockReturnValue(of(true))
|
||||
let openModal: NgbModalRef
|
||||
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
||||
component.reprocess()
|
||||
openModal.componentInstance.remoteOcr = true
|
||||
openModal.componentInstance.confirmClicked.next()
|
||||
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, true)
|
||||
})
|
||||
|
||||
it('should show error if redo ocr call fails', () => {
|
||||
initNormally()
|
||||
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
|
||||
|
||||
@@ -97,7 +97,6 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
|
||||
import * as UTIF from 'utif'
|
||||
import { DocumentDetailFieldID } from '../admin/settings/settings.component'
|
||||
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
|
||||
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
|
||||
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
|
||||
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
|
||||
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
|
||||
@@ -1403,7 +1402,7 @@ export class DocumentDetailComponent
|
||||
}
|
||||
|
||||
reprocess() {
|
||||
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
|
||||
let modal = this.modalService.open(ConfirmDialogComponent, {
|
||||
backdrop: 'static',
|
||||
})
|
||||
modal.componentInstance.title = $localize`Reprocess confirm`
|
||||
@@ -1414,10 +1413,7 @@ export class DocumentDetailComponent
|
||||
modal.componentInstance.confirmClicked.subscribe(() => {
|
||||
modal.componentInstance.buttonsEnabled = false
|
||||
this.documentsService
|
||||
.reprocessDocuments(
|
||||
{ documents: [this.document().id] },
|
||||
modal.componentInstance.remoteOcr
|
||||
)
|
||||
.reprocessDocuments({ documents: [this.document().id] })
|
||||
.subscribe({
|
||||
next: () => {
|
||||
this.toastService.showInfo(
|
||||
|
||||
@@ -1122,7 +1122,6 @@ describe('BulkEditorComponent', () => {
|
||||
req.flush(true)
|
||||
expect(req.request.body).toEqual({
|
||||
documents: [3, 4],
|
||||
remote_ocr: false,
|
||||
})
|
||||
httpTestingController.match(
|
||||
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
|
||||
|
||||
@@ -51,7 +51,6 @@ import { ToastService } from 'src/app/services/toast.service'
|
||||
import { flattenTags } from 'src/app/utils/flatten-tags'
|
||||
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
|
||||
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
|
||||
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
|
||||
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
|
||||
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
|
||||
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
|
||||
@@ -910,7 +909,7 @@ export class BulkEditorComponent
|
||||
}
|
||||
|
||||
reprocessSelected() {
|
||||
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
|
||||
let modal = this.modalService.open(ConfirmDialogComponent, {
|
||||
backdrop: 'static',
|
||||
})
|
||||
modal.componentInstance.title = $localize`Reprocess confirm`
|
||||
@@ -924,10 +923,7 @@ export class BulkEditorComponent
|
||||
modal.componentInstance.buttonsEnabled = false
|
||||
this.executeDocumentAction(
|
||||
modal,
|
||||
this.documentService.reprocessDocuments(
|
||||
this.getSelectionQuery(),
|
||||
modal.componentInstance.remoteOcr
|
||||
)
|
||||
this.documentService.reprocessDocuments(this.getSelectionQuery())
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -97,7 +97,9 @@
|
||||
<div class="dropdown-menu shadow dropdown-menu-right" ngbDropdownMenu>
|
||||
@if (!list.activeSavedViewId) {
|
||||
@for (view of savedViewService.allViews; track view) {
|
||||
<button ngbDropdownItem (click)="loadViewConfig(view.id)">{{view.name}}</button>
|
||||
<button ngbDropdownItem (click)="loadViewConfig(view.id)">
|
||||
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs>{{view.name}}
|
||||
</button>
|
||||
}
|
||||
@if (savedViewService.allViews.length > 0) {
|
||||
<div class="dropdown-divider"></div>
|
||||
|
||||
@@ -457,6 +457,7 @@ export class DocumentListComponent
|
||||
modal.componentInstance.buttonsEnabled.set(false)
|
||||
let savedView: SavedView = {
|
||||
name: formValue.name,
|
||||
icon: formValue.icon,
|
||||
filter_rules: this.list.filterRules,
|
||||
sort_reverse: this.list.sortReverse,
|
||||
sort_field: this.list.sortField,
|
||||
|
||||
+8
@@ -6,6 +6,14 @@
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<pngx-input-text i18n-title title="Name" formControlName="name" [error]="error()?.name" autocomplete="off"></pngx-input-text>
|
||||
<pngx-input-select
|
||||
i18n-title
|
||||
title="Icon"
|
||||
formControlName="icon"
|
||||
[items]="savedViewIcons"
|
||||
iconField="icon"
|
||||
[error]="error()?.icon">
|
||||
</pngx-input-select>
|
||||
<pngx-input-check i18n-title title="Show in sidebar" formControlName="showInSideBar"></pngx-input-check>
|
||||
<pngx-input-check i18n-title title="Show on dashboard" formControlName="showOnDashboard"></pngx-input-check>
|
||||
<pngx-permissions-form accordion="true" formControlName="permissions_form"></pngx-permissions-form>
|
||||
|
||||
+5
@@ -9,6 +9,7 @@ import { CheckComponent } from '../../common/input/check/check.component'
|
||||
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
|
||||
import { PermissionsGroupComponent } from '../../common/input/permissions/permissions-group/permissions-group.component'
|
||||
import { PermissionsUserComponent } from '../../common/input/permissions/permissions-user/permissions-user.component'
|
||||
import { SelectComponent } from '../../common/input/select/select.component'
|
||||
import { TextComponent } from '../../common/input/text/text.component'
|
||||
import { SaveViewConfigDialogComponent } from './save-view-config-dialog.component'
|
||||
|
||||
@@ -40,6 +41,7 @@ describe('SaveViewConfigDialogComponent', () => {
|
||||
ReactiveFormsModule,
|
||||
SaveViewConfigDialogComponent,
|
||||
TextComponent,
|
||||
SelectComponent,
|
||||
CheckComponent,
|
||||
PermissionsFormComponent,
|
||||
PermissionsUserComponent,
|
||||
@@ -63,6 +65,7 @@ describe('SaveViewConfigDialogComponent', () => {
|
||||
expect(component.defaultName()).toEqual(name)
|
||||
expect(result).toEqual({
|
||||
name,
|
||||
icon: 'funnel',
|
||||
showInSideBar: false,
|
||||
showOnDashboard: false,
|
||||
})
|
||||
@@ -94,6 +97,7 @@ describe('SaveViewConfigDialogComponent', () => {
|
||||
component.save()
|
||||
expect(result).toEqual({
|
||||
name,
|
||||
icon: 'funnel',
|
||||
showInSideBar: true,
|
||||
showOnDashboard: true,
|
||||
})
|
||||
@@ -113,6 +117,7 @@ describe('SaveViewConfigDialogComponent', () => {
|
||||
component.save()
|
||||
expect(result).toEqual({
|
||||
name: '',
|
||||
icon: 'funnel',
|
||||
showInSideBar: false,
|
||||
showOnDashboard: false,
|
||||
permissions_form: permissions,
|
||||
|
||||
+9
@@ -13,9 +13,14 @@ import {
|
||||
ReactiveFormsModule,
|
||||
} from '@angular/forms'
|
||||
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
|
||||
import {
|
||||
DEFAULT_SAVED_VIEW_ICON,
|
||||
SAVED_VIEW_ICONS,
|
||||
} from 'src/app/data/saved-view-icons'
|
||||
import { User } from 'src/app/data/user'
|
||||
import { CheckComponent } from '../../common/input/check/check.component'
|
||||
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
|
||||
import { SelectComponent } from '../../common/input/select/select.component'
|
||||
import { TextComponent } from '../../common/input/text/text.component'
|
||||
|
||||
@Component({
|
||||
@@ -24,6 +29,7 @@ import { TextComponent } from '../../common/input/text/text.component'
|
||||
styleUrls: ['./save-view-config-dialog.component.scss'],
|
||||
imports: [
|
||||
CheckComponent,
|
||||
SelectComponent,
|
||||
TextComponent,
|
||||
PermissionsFormComponent,
|
||||
FormsModule,
|
||||
@@ -41,6 +47,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
|
||||
public saveClicked = new EventEmitter()
|
||||
|
||||
users: User[]
|
||||
readonly savedViewIcons = SAVED_VIEW_ICONS
|
||||
|
||||
setDefaultName(value: string) {
|
||||
this.defaultName.set(value)
|
||||
@@ -49,6 +56,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
|
||||
|
||||
saveViewConfigForm = new FormGroup({
|
||||
name: new FormControl(''),
|
||||
icon: new FormControl(DEFAULT_SAVED_VIEW_ICON),
|
||||
showInSideBar: new FormControl(false),
|
||||
showOnDashboard: new FormControl(false),
|
||||
permissions_form: new FormControl(null),
|
||||
@@ -65,6 +73,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
|
||||
const formValue = this.saveViewConfigForm.value
|
||||
const saveViewConfig = {
|
||||
name: formValue.name,
|
||||
icon: formValue.icon,
|
||||
showInSideBar: formValue.showInSideBar,
|
||||
showOnDashboard: formValue.showOnDashboard,
|
||||
}
|
||||
|
||||
@@ -7,15 +7,24 @@
|
||||
</pngx-page-header>
|
||||
<form [formGroup]="savedViewsForm" (ngSubmit)="save()">
|
||||
<ul class="list-group mb-3" formGroupName="savedViews">
|
||||
@for (view of savedViews(); track view) {
|
||||
@for (view of pagedSavedViews(); track view) {
|
||||
<li class="list-group-item py-3">
|
||||
<div [formGroupName]="view.id">
|
||||
<div class="row">
|
||||
<div class="col">
|
||||
<div class="col-md">
|
||||
<pngx-input-text title="Name" formControlName="name"></pngx-input-text>
|
||||
</div>
|
||||
<div class="col-md">
|
||||
<pngx-input-select
|
||||
i18n-title
|
||||
title="Icon"
|
||||
formControlName="icon"
|
||||
[items]="savedViewIcons"
|
||||
iconField="icon">
|
||||
</pngx-input-select>
|
||||
</div>
|
||||
@if (canSaveSettings) {
|
||||
<div class="col">
|
||||
<div class="col-md">
|
||||
<div class="form-check form-switch mt-3">
|
||||
<input type="checkbox" class="form-check-input" id="show_on_dashboard_{{view.id}}" formControlName="show_on_dashboard">
|
||||
<label class="form-check-label" for="show_on_dashboard_{{view.id}}" i18n>Show on dashboard</label>
|
||||
@@ -81,6 +90,11 @@
|
||||
}
|
||||
</ul>
|
||||
|
||||
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
|
||||
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
|
||||
<div class="d-flex align-items-center mb-3">
|
||||
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
|
||||
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
|
||||
@if (savedViews()?.length > pageSize) {
|
||||
<ngb-pagination class="ms-auto" [pageSize]="pageSize" [collectionSize]="savedViews().length" [page]="page()" [maxSize]="5" (pageChange)="page.set($event)" size="sm" aria-label="Pagination"></ngb-pagination>
|
||||
}
|
||||
</div>
|
||||
</form>
|
||||
|
||||
@@ -4,6 +4,7 @@ import { provideHttpClientTesting } from '@angular/common/http/testing'
|
||||
import { signal } from '@angular/core'
|
||||
import { ComponentFixture, TestBed } from '@angular/core/testing'
|
||||
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
|
||||
import { By } from '@angular/platform-browser'
|
||||
import { NgbModal, NgbModule } from '@ng-bootstrap/ng-bootstrap'
|
||||
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
||||
import { Subject, of, throwError } from 'rxjs'
|
||||
@@ -25,8 +26,20 @@ import { PageHeaderComponent } from '../../common/page-header/page-header.compon
|
||||
import { SavedViewsComponent } from './saved-views.component'
|
||||
|
||||
const savedViews = [
|
||||
{ id: 1, name: 'view1', show_in_sidebar: true, show_on_dashboard: true },
|
||||
{ id: 2, name: 'view2', show_in_sidebar: false, show_on_dashboard: false },
|
||||
{
|
||||
id: 1,
|
||||
name: 'view1',
|
||||
icon: 'archive',
|
||||
show_in_sidebar: true,
|
||||
show_on_dashboard: true,
|
||||
},
|
||||
{
|
||||
id: 2,
|
||||
name: 'view2',
|
||||
icon: 'funnel',
|
||||
show_in_sidebar: false,
|
||||
show_on_dashboard: false,
|
||||
},
|
||||
]
|
||||
|
||||
describe('SavedViewsComponent', () => {
|
||||
@@ -157,6 +170,24 @@ describe('SavedViewsComponent', () => {
|
||||
expect(patchBody.show_in_sidebar).toBeUndefined()
|
||||
})
|
||||
|
||||
it('should persist a changed icon', () => {
|
||||
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
|
||||
const view = savedViews[0]
|
||||
const iconControl = component.savedViewsForm
|
||||
.get('savedViews')
|
||||
.get(view.id.toString())
|
||||
.get('icon')
|
||||
|
||||
iconControl.setValue('bell')
|
||||
iconControl.markAsDirty()
|
||||
component.save()
|
||||
|
||||
expect(patchSpy.mock.calls[0][0][0]).toMatchObject({
|
||||
id: view.id,
|
||||
icon: 'bell',
|
||||
})
|
||||
})
|
||||
|
||||
it('should persist visibility changes to user settings', () => {
|
||||
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
|
||||
const updateVisibilitySpy = jest
|
||||
@@ -222,6 +253,44 @@ describe('SavedViewsComponent', () => {
|
||||
).toEqual(view.show_on_dashboard)
|
||||
})
|
||||
|
||||
it('should page saved views, clamp the page if views are removed', () => {
|
||||
const manyViews = Array.from({ length: 30 }, (_, i) => ({
|
||||
id: i + 1,
|
||||
name: `view${i + 1}`,
|
||||
})) as SavedView[]
|
||||
const listSpy = jest.spyOn(savedViewService, 'list').mockReturnValue(
|
||||
of({
|
||||
all: manyViews.map((v) => v.id),
|
||||
count: manyViews.length,
|
||||
results: manyViews.concat([]),
|
||||
})
|
||||
)
|
||||
component.ngOnInit()
|
||||
fixture.detectChanges()
|
||||
expect(listSpy).toHaveBeenCalledWith(1, 100000, null, false, {
|
||||
full_perms: true,
|
||||
})
|
||||
expect(component.pagedSavedViews()).toHaveLength(25)
|
||||
expect(fixture.debugElement.query(By.css('ngb-pagination'))).not.toBeNull()
|
||||
// all views have controls, not just the current page
|
||||
expect(
|
||||
Object.keys(component.savedViewsForm.get('savedViews').value)
|
||||
).toHaveLength(30)
|
||||
|
||||
component.page.set(2)
|
||||
expect(component.pagedSavedViews()).toHaveLength(5)
|
||||
|
||||
listSpy.mockReturnValue(
|
||||
of({
|
||||
all: manyViews.slice(0, 25).map((v) => v.id),
|
||||
count: 25,
|
||||
results: manyViews.slice(0, 25),
|
||||
})
|
||||
)
|
||||
component.ngOnInit()
|
||||
expect(component.page()).toEqual(1)
|
||||
})
|
||||
|
||||
it('should support editing permissions', () => {
|
||||
const confirmClicked = new Subject<any>()
|
||||
const modalRef = {
|
||||
|
||||
@@ -1,18 +1,29 @@
|
||||
import { AsyncPipe } from '@angular/common'
|
||||
import { Component, OnDestroy, OnInit, inject, signal } from '@angular/core'
|
||||
import {
|
||||
Component,
|
||||
OnDestroy,
|
||||
OnInit,
|
||||
computed,
|
||||
inject,
|
||||
signal,
|
||||
} from '@angular/core'
|
||||
import {
|
||||
FormControl,
|
||||
FormGroup,
|
||||
FormsModule,
|
||||
ReactiveFormsModule,
|
||||
} from '@angular/forms'
|
||||
import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
|
||||
import { NgbModal, NgbPaginationModule } from '@ng-bootstrap/ng-bootstrap'
|
||||
import { dirtyCheck } from '@ngneat/dirty-check-forms'
|
||||
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
||||
import { BehaviorSubject, Observable, of, switchMap, takeUntil } from 'rxjs'
|
||||
import { PermissionsDialogComponent } from 'src/app/components/common/permissions-dialog/permissions-dialog.component'
|
||||
import { DisplayMode } from 'src/app/data/document'
|
||||
import { SavedView } from 'src/app/data/saved-view'
|
||||
import {
|
||||
DEFAULT_SAVED_VIEW_ICON,
|
||||
SAVED_VIEW_ICONS,
|
||||
} from 'src/app/data/saved-view-icons'
|
||||
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
|
||||
import {
|
||||
PermissionAction,
|
||||
@@ -25,6 +36,7 @@ import { ToastService } from 'src/app/services/toast.service'
|
||||
import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-button.component'
|
||||
import { DragDropSelectComponent } from '../../common/input/drag-drop-select/drag-drop-select.component'
|
||||
import { NumberComponent } from '../../common/input/number/number.component'
|
||||
import { SelectComponent } from '../../common/input/select/select.component'
|
||||
import { TextComponent } from '../../common/input/text/text.component'
|
||||
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
|
||||
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
|
||||
@@ -36,12 +48,14 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
|
||||
PageHeaderComponent,
|
||||
ConfirmButtonComponent,
|
||||
NumberComponent,
|
||||
SelectComponent,
|
||||
TextComponent,
|
||||
IfPermissionsDirective,
|
||||
DragDropSelectComponent,
|
||||
FormsModule,
|
||||
ReactiveFormsModule,
|
||||
AsyncPipe,
|
||||
NgbPaginationModule,
|
||||
NgxBootstrapIconsModule,
|
||||
],
|
||||
})
|
||||
@@ -56,8 +70,17 @@ export class SavedViewsComponent
|
||||
private readonly modalService = inject(NgbModal)
|
||||
|
||||
DisplayMode = DisplayMode
|
||||
readonly savedViewIcons = SAVED_VIEW_ICONS
|
||||
|
||||
readonly savedViews = signal<SavedView[]>(undefined)
|
||||
readonly page = signal(1)
|
||||
public readonly pageSize = 25
|
||||
// All views are loaded at init, so paging is only for display
|
||||
readonly pagedSavedViews = computed(() => {
|
||||
const start = (this.page() - 1) * this.pageSize
|
||||
return this.savedViews()?.slice(start, start + this.pageSize)
|
||||
})
|
||||
|
||||
private savedViewsGroup = new FormGroup({})
|
||||
public savedViewsForm: FormGroup = new FormGroup({
|
||||
savedViews: this.savedViewsGroup,
|
||||
@@ -84,9 +107,11 @@ export class SavedViewsComponent
|
||||
private reloadViews(): void {
|
||||
this.loading.set(true)
|
||||
this.savedViewService
|
||||
.list(null, null, null, false, { full_perms: true })
|
||||
.list(1, 100000, null, false, { full_perms: true })
|
||||
.subscribe((r) => {
|
||||
this.savedViews.set(r.results)
|
||||
const pageCount = Math.ceil(r.results.length / this.pageSize)
|
||||
this.page.update((page) => Math.min(page, Math.max(1, pageCount)))
|
||||
this.initialize()
|
||||
})
|
||||
}
|
||||
@@ -110,6 +135,7 @@ export class SavedViewsComponent
|
||||
storeData.savedViews[view.id.toString()] = {
|
||||
id: view.id,
|
||||
name: view.name,
|
||||
icon: view.icon ?? DEFAULT_SAVED_VIEW_ICON,
|
||||
show_on_dashboard: view.show_on_dashboard,
|
||||
show_in_sidebar: view.show_in_sidebar,
|
||||
page_size: view.page_size,
|
||||
@@ -122,6 +148,7 @@ export class SavedViewsComponent
|
||||
new FormGroup({
|
||||
id: new FormControl({ value: null, disabled: !canEdit }),
|
||||
name: new FormControl({ value: null, disabled: !canEdit }),
|
||||
icon: new FormControl({ value: null, disabled: !canEdit }),
|
||||
show_on_dashboard: new FormControl({
|
||||
value: null,
|
||||
disabled: false,
|
||||
@@ -200,6 +227,7 @@ export class SavedViewsComponent
|
||||
|
||||
const modelFieldsChanged =
|
||||
group.get('name')?.dirty ||
|
||||
group.get('icon')?.dirty ||
|
||||
group.get('page_size')?.dirty ||
|
||||
group.get('display_mode')?.dirty ||
|
||||
group.get('display_fields')?.dirty
|
||||
|
||||
@@ -54,10 +54,6 @@ export const ConfigCategory = {
|
||||
AI: $localize`AI Settings`,
|
||||
}
|
||||
|
||||
export const ConfigSection = {
|
||||
RemoteOCR: $localize`Remote OCR`,
|
||||
}
|
||||
|
||||
export const LLMEmbeddingBackendConfig = {
|
||||
OPENAI_LIKE: 'openai-like',
|
||||
HUGGINGFACE: 'huggingface',
|
||||
@@ -69,15 +65,6 @@ export const LLMBackendConfig = {
|
||||
OLLAMA: 'ollama',
|
||||
}
|
||||
|
||||
export const RemoteOCREngineConfig = {
|
||||
AZURE_AI: 'azureai',
|
||||
}
|
||||
|
||||
export const RemoteOCRModeConfig = {
|
||||
ALWAYS: 'always',
|
||||
WORKFLOW_ONLY: 'workflow_only',
|
||||
}
|
||||
|
||||
export interface ConfigOption {
|
||||
key: string
|
||||
title: string
|
||||
@@ -85,7 +72,6 @@ export interface ConfigOption {
|
||||
choices?: Array<{ id: string; name: string }>
|
||||
config_key?: string
|
||||
category: string
|
||||
section?: string
|
||||
note?: string
|
||||
}
|
||||
|
||||
@@ -195,43 +181,6 @@ export const PaperlessConfigOptions: ConfigOption[] = [
|
||||
config_key: 'PAPERLESS_OCR_USER_ARGS',
|
||||
category: ConfigCategory.OCR,
|
||||
},
|
||||
{
|
||||
key: 'remote_ocr_engine',
|
||||
title: $localize`Remote OCR Engine`,
|
||||
type: ConfigOptionType.Select,
|
||||
choices: mapToItems(RemoteOCREngineConfig),
|
||||
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
|
||||
category: ConfigCategory.OCR,
|
||||
section: ConfigSection.RemoteOCR,
|
||||
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
|
||||
},
|
||||
{
|
||||
key: 'remote_ocr_api_key',
|
||||
title: $localize`Remote OCR API Key`,
|
||||
type: ConfigOptionType.Password,
|
||||
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
|
||||
category: ConfigCategory.OCR,
|
||||
section: ConfigSection.RemoteOCR,
|
||||
},
|
||||
{
|
||||
key: 'remote_ocr_endpoint',
|
||||
title: $localize`Remote OCR Endpoint`,
|
||||
type: ConfigOptionType.String,
|
||||
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
|
||||
category: ConfigCategory.OCR,
|
||||
section: ConfigSection.RemoteOCR,
|
||||
note: $localize`Required when using the Azure AI engine.`,
|
||||
},
|
||||
{
|
||||
key: 'remote_ocr_mode',
|
||||
title: $localize`Remote OCR Mode`,
|
||||
type: ConfigOptionType.Select,
|
||||
choices: mapToItems(RemoteOCRModeConfig),
|
||||
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
|
||||
category: ConfigCategory.OCR,
|
||||
section: ConfigSection.RemoteOCR,
|
||||
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
|
||||
},
|
||||
{
|
||||
key: 'app_logo',
|
||||
title: $localize`Application Logo`,
|
||||
@@ -449,10 +398,6 @@ export interface PaperlessConfig extends ObjectWithId {
|
||||
barcode_enable_tag: boolean
|
||||
barcode_tag_mapping: object
|
||||
barcode_tag_split: boolean
|
||||
remote_ocr_engine: string
|
||||
remote_ocr_api_key: string
|
||||
remote_ocr_endpoint: string
|
||||
remote_ocr_mode: string
|
||||
ai_enabled: boolean
|
||||
llm_embedding_backend: string
|
||||
llm_embedding_model: string
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
export const DEFAULT_SAVED_VIEW_ICON = 'funnel'
|
||||
|
||||
export const SAVED_VIEW_ICONS = [
|
||||
{ id: 'archive', name: $localize`Archive`, icon: 'archive' },
|
||||
{ id: 'bank', name: $localize`Bank`, icon: 'bank' },
|
||||
{ id: 'basket', name: $localize`Basket`, icon: 'basket' },
|
||||
{ id: 'bell', name: $localize`Bell`, icon: 'bell' },
|
||||
{ id: 'bookmark', name: $localize`Bookmark`, icon: 'bookmark' },
|
||||
{ id: 'boxes', name: $localize`Boxes`, icon: 'boxes' },
|
||||
{ id: 'briefcase', name: $localize`Briefcase`, icon: 'briefcase' },
|
||||
{ id: 'building', name: $localize`Building`, icon: 'building' },
|
||||
{ id: 'calculator', name: $localize`Calculator`, icon: 'calculator' },
|
||||
{ id: 'calendar', name: $localize`Calendar`, icon: 'calendar' },
|
||||
{ id: 'camera', name: $localize`Camera`, icon: 'camera' },
|
||||
{
|
||||
id: 'card-checklist',
|
||||
name: $localize`Checklist`,
|
||||
icon: 'card-checklist',
|
||||
},
|
||||
{ id: 'cash', name: $localize`Cash`, icon: 'cash' },
|
||||
{ id: 'chat-left-text', name: $localize`Chat`, icon: 'chat-left-text' },
|
||||
{ id: 'check-circle', name: $localize`Check`, icon: 'check-circle' },
|
||||
{ id: 'clipboard', name: $localize`Clipboard`, icon: 'clipboard' },
|
||||
{ id: 'clock-history', name: $localize`Clock`, icon: 'clock-history' },
|
||||
{ id: 'credit-card', name: $localize`Credit card`, icon: 'credit-card' },
|
||||
{ id: 'download', name: $localize`Download`, icon: 'download' },
|
||||
{ id: 'envelope', name: $localize`Envelope`, icon: 'envelope' },
|
||||
{
|
||||
id: 'exclamation-triangle',
|
||||
name: $localize`Warning`,
|
||||
icon: 'exclamation-triangle',
|
||||
},
|
||||
{ id: 'file-earmark', name: $localize`File`, icon: 'file-earmark' },
|
||||
{
|
||||
id: 'file-earmark-check',
|
||||
name: $localize`Checked file`,
|
||||
icon: 'file-earmark-check',
|
||||
},
|
||||
{
|
||||
id: 'file-earmark-lock',
|
||||
name: $localize`Locked file`,
|
||||
icon: 'file-earmark-lock',
|
||||
},
|
||||
{
|
||||
id: 'file-earmark-medical',
|
||||
name: $localize`Medical file`,
|
||||
icon: 'file-earmark-medical',
|
||||
},
|
||||
{
|
||||
id: 'file-earmark-person',
|
||||
name: $localize`Person file`,
|
||||
icon: 'file-earmark-person',
|
||||
},
|
||||
{
|
||||
id: 'file-earmark-spreadsheet',
|
||||
name: $localize`Spreadsheet`,
|
||||
icon: 'file-earmark-spreadsheet',
|
||||
},
|
||||
{ id: 'file-text', name: $localize`Text file`, icon: 'file-text' },
|
||||
{ id: 'files', name: $localize`Files`, icon: 'files' },
|
||||
{ id: 'folder', name: $localize`Folder`, icon: 'folder' },
|
||||
{ id: 'funnel', name: $localize`Filter`, icon: 'funnel' },
|
||||
{ id: 'gear', name: $localize`Gear`, icon: 'gear' },
|
||||
{ id: 'globe2', name: $localize`Globe`, icon: 'globe2' },
|
||||
{ id: 'hash', name: $localize`Hash`, icon: 'hash' },
|
||||
{ id: 'heart', name: $localize`Heart`, icon: 'heart' },
|
||||
{ id: 'house', name: $localize`House`, icon: 'house' },
|
||||
{ id: 'inbox', name: $localize`Inbox`, icon: 'inbox' },
|
||||
{ id: 'journals', name: $localize`Journals`, icon: 'journals' },
|
||||
{ id: 'list-task', name: $localize`Task list`, icon: 'list-task' },
|
||||
{ id: 'newspaper', name: $localize`Newspaper`, icon: 'newspaper' },
|
||||
{ id: 'paperclip', name: $localize`Attachment`, icon: 'paperclip' },
|
||||
{ id: 'people', name: $localize`People`, icon: 'people' },
|
||||
{ id: 'person', name: $localize`Person`, icon: 'person' },
|
||||
{ id: 'printer', name: $localize`Printer`, icon: 'printer' },
|
||||
{ id: 'receipt', name: $localize`Receipt`, icon: 'receipt' },
|
||||
{ id: 'safe', name: $localize`Safe`, icon: 'safe' },
|
||||
{ id: 'search', name: $localize`Search`, icon: 'search' },
|
||||
{ id: 'send', name: $localize`Send`, icon: 'send' },
|
||||
{ id: 'shop', name: $localize`Shop`, icon: 'shop' },
|
||||
{ id: 'stack', name: $localize`Stack`, icon: 'stack' },
|
||||
{ id: 'stars', name: $localize`Stars`, icon: 'stars' },
|
||||
{ id: 'tag', name: $localize`Tag`, icon: 'tag' },
|
||||
{ id: 'tags', name: $localize`Tags`, icon: 'tags' },
|
||||
{ id: 'telephone', name: $localize`Telephone`, icon: 'telephone' },
|
||||
{ id: 'truck', name: $localize`Truck`, icon: 'truck' },
|
||||
{ id: 'upc-scan', name: $localize`Barcode`, icon: 'upc-scan' },
|
||||
{ id: 'wallet2', name: $localize`Wallet`, icon: 'wallet2' },
|
||||
]
|
||||
@@ -5,6 +5,8 @@ import { ObjectWithPermissions } from './object-with-permissions'
|
||||
export interface SavedView extends ObjectWithPermissions {
|
||||
name?: string
|
||||
|
||||
icon?: string
|
||||
|
||||
show_on_dashboard?: boolean
|
||||
|
||||
show_in_sidebar?: boolean
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
|
||||
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
|
||||
import { RemoteOCRModeConfig } from './paperless-config'
|
||||
import { User } from './user'
|
||||
|
||||
export interface UiSettings {
|
||||
@@ -95,8 +94,6 @@ export const SETTINGS_KEYS = {
|
||||
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
|
||||
EMAIL_ENABLED: 'email_enabled',
|
||||
AI_ENABLED: 'ai_enabled',
|
||||
REMOTE_OCR_CONFIGURED: 'remote_ocr:configured',
|
||||
REMOTE_OCR_MODE: 'remote_ocr:mode',
|
||||
}
|
||||
|
||||
export const SETTINGS: UiSetting[] = [
|
||||
@@ -350,14 +347,4 @@ export const SETTINGS: UiSetting[] = [
|
||||
type: 'string',
|
||||
default: PdfEditorEditMode.Create,
|
||||
},
|
||||
{
|
||||
key: SETTINGS_KEYS.REMOTE_OCR_CONFIGURED,
|
||||
type: 'boolean',
|
||||
default: false,
|
||||
},
|
||||
{
|
||||
key: SETTINGS_KEYS.REMOTE_OCR_MODE,
|
||||
type: 'string',
|
||||
default: RemoteOCRModeConfig.ALWAYS,
|
||||
},
|
||||
]
|
||||
|
||||
@@ -284,21 +284,6 @@ describe(`DocumentService`, () => {
|
||||
expect(req.request.method).toEqual('POST')
|
||||
expect(req.request.body).toEqual({
|
||||
documents: ids,
|
||||
remote_ocr: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('should request remote OCR when reprocessing with it enabled', () => {
|
||||
const ids = [1, 2, 3]
|
||||
subscription = service
|
||||
.reprocessDocuments({ documents: ids }, true)
|
||||
.subscribe()
|
||||
const req = httpTestingController.expectOne(
|
||||
`${environment.apiBaseUrl}${endpoint}/reprocess/`
|
||||
)
|
||||
expect(req.request.body).toEqual({
|
||||
documents: ids,
|
||||
remote_ocr: true,
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -349,13 +349,9 @@ export class DocumentService extends AbstractPaperlessService<Document> {
|
||||
})
|
||||
}
|
||||
|
||||
reprocessDocuments(
|
||||
selection: DocumentSelectionQuery,
|
||||
remoteOcr: boolean = false
|
||||
) {
|
||||
reprocessDocuments(selection: DocumentSelectionQuery) {
|
||||
return this.http.post(this.getResourceUrl(null, 'reprocess'), {
|
||||
...selection,
|
||||
remote_ocr: remoteOcr,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -13,7 +13,6 @@ import { environment } from 'src/environments/environment'
|
||||
import { CustomFieldDataType } from '../data/custom-field'
|
||||
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
||||
import { SavedView } from '../data/saved-view'
|
||||
import { RemoteOCRModeConfig } from '../data/paperless-config'
|
||||
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
|
||||
import { PermissionsService } from './permissions.service'
|
||||
import { CustomFieldsService } from './rest/custom-fields.service'
|
||||
@@ -435,26 +434,4 @@ describe('SettingsService', () => {
|
||||
).name
|
||||
).toEqual(customFields[0].name)
|
||||
})
|
||||
it('should offer remote OCR only when configured and selective', () => {
|
||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, false)
|
||||
settingsService.set(
|
||||
SETTINGS_KEYS.REMOTE_OCR_MODE,
|
||||
RemoteOCRModeConfig.WORKFLOW_ONLY
|
||||
)
|
||||
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
|
||||
|
||||
// configured, but already handling every document
|
||||
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, true)
|
||||
settingsService.set(
|
||||
SETTINGS_KEYS.REMOTE_OCR_MODE,
|
||||
RemoteOCRModeConfig.ALWAYS
|
||||
)
|
||||
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
|
||||
|
||||
settingsService.set(
|
||||
SETTINGS_KEYS.REMOTE_OCR_MODE,
|
||||
RemoteOCRModeConfig.WORKFLOW_ONLY
|
||||
)
|
||||
expect(settingsService.remoteOCRIsSelectable).toBeTruthy()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -19,7 +19,6 @@ import {
|
||||
} from 'src/app/utils/color'
|
||||
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
|
||||
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
||||
import { RemoteOCRModeConfig } from '../data/paperless-config'
|
||||
import { SavedView } from '../data/saved-view'
|
||||
import {
|
||||
PAPERLESS_GREEN_HEX,
|
||||
@@ -688,17 +687,6 @@ export class SettingsService {
|
||||
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
|
||||
}
|
||||
|
||||
/**
|
||||
* Offering remote OCR as a choice only makes sense when an engine
|
||||
* is configured but is not already handling every document.
|
||||
*/
|
||||
get remoteOCRIsSelectable(): boolean {
|
||||
return (
|
||||
this.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
|
||||
this.get(SETTINGS_KEYS.REMOTE_OCR_MODE) !== RemoteOCRModeConfig.ALWAYS
|
||||
)
|
||||
}
|
||||
|
||||
offerTour(): boolean {
|
||||
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
|
||||
}
|
||||
|
||||
@@ -35,19 +35,27 @@ import {
|
||||
arrowRightShort,
|
||||
arrowUpRight,
|
||||
asterisk,
|
||||
bank,
|
||||
basket,
|
||||
bell,
|
||||
bodyText,
|
||||
bookmark,
|
||||
boxArrowUp,
|
||||
boxArrowUpRight,
|
||||
boxes,
|
||||
braces,
|
||||
briefcase,
|
||||
building,
|
||||
calculator,
|
||||
calendar,
|
||||
calendarEvent,
|
||||
calendarEventFill,
|
||||
camera,
|
||||
cardChecklist,
|
||||
cardHeading,
|
||||
caretDown,
|
||||
caretUp,
|
||||
cash,
|
||||
chatLeftText,
|
||||
chatSquareDots,
|
||||
check,
|
||||
@@ -65,6 +73,7 @@ import {
|
||||
clipboardCheckFill,
|
||||
clipboardFill,
|
||||
clockHistory,
|
||||
creditCard,
|
||||
dash,
|
||||
dashCircle,
|
||||
diagram3,
|
||||
@@ -83,9 +92,12 @@ import {
|
||||
fileEarmarkDiff,
|
||||
fileEarmarkFill,
|
||||
fileEarmarkLock,
|
||||
fileEarmarkMedical,
|
||||
fileEarmarkMinus,
|
||||
fileEarmarkPerson,
|
||||
fileEarmarkPlus,
|
||||
fileEarmarkRichtext,
|
||||
fileEarmarkSpreadsheet,
|
||||
fileText,
|
||||
files,
|
||||
filter,
|
||||
@@ -93,12 +105,15 @@ import {
|
||||
folderFill,
|
||||
funnel,
|
||||
gear,
|
||||
globe2,
|
||||
google,
|
||||
grid,
|
||||
gripVertical,
|
||||
hash,
|
||||
hddStack,
|
||||
heart,
|
||||
house,
|
||||
inbox,
|
||||
infoCircle,
|
||||
journals,
|
||||
link,
|
||||
@@ -106,7 +121,9 @@ import {
|
||||
listTask,
|
||||
listUl,
|
||||
microsoft,
|
||||
newspaper,
|
||||
nodePlus,
|
||||
paperclip,
|
||||
pencil,
|
||||
people,
|
||||
peopleFill,
|
||||
@@ -121,9 +138,12 @@ import {
|
||||
plusCircle,
|
||||
printer,
|
||||
questionCircle,
|
||||
receipt,
|
||||
safe,
|
||||
scissors,
|
||||
search,
|
||||
send,
|
||||
shop,
|
||||
slashCircle,
|
||||
sliders2Vertical,
|
||||
sortAlphaDown,
|
||||
@@ -133,14 +153,17 @@ import {
|
||||
tag,
|
||||
tagFill,
|
||||
tags,
|
||||
telephone,
|
||||
textIndentLeft,
|
||||
textLeft,
|
||||
threeDots,
|
||||
threeDotsVertical,
|
||||
trash,
|
||||
truck,
|
||||
uiRadios,
|
||||
unlock,
|
||||
upcScan,
|
||||
wallet2,
|
||||
windowStack,
|
||||
x,
|
||||
xCircle,
|
||||
@@ -258,15 +281,22 @@ const icons = {
|
||||
arrowRightShort,
|
||||
arrowUpRight,
|
||||
asterisk,
|
||||
bank,
|
||||
basket,
|
||||
bell,
|
||||
braces,
|
||||
bodyText,
|
||||
bookmark,
|
||||
boxArrowUp,
|
||||
boxArrowUpRight,
|
||||
boxes,
|
||||
briefcase,
|
||||
building,
|
||||
calculator,
|
||||
calendar,
|
||||
calendarEvent,
|
||||
calendarEventFill,
|
||||
camera,
|
||||
cardChecklist,
|
||||
cardHeading,
|
||||
caretDown,
|
||||
@@ -288,6 +318,8 @@ const icons = {
|
||||
clipboardCheckFill,
|
||||
clipboardFill,
|
||||
clockHistory,
|
||||
cash,
|
||||
creditCard,
|
||||
dash,
|
||||
dashCircle,
|
||||
diagram3,
|
||||
@@ -306,9 +338,12 @@ const icons = {
|
||||
fileEarmarkDiff,
|
||||
fileEarmarkFill,
|
||||
fileEarmarkLock,
|
||||
fileEarmarkMedical,
|
||||
fileEarmarkMinus,
|
||||
fileEarmarkPerson,
|
||||
fileEarmarkPlus,
|
||||
fileEarmarkRichtext,
|
||||
fileEarmarkSpreadsheet,
|
||||
files,
|
||||
fileText,
|
||||
filter,
|
||||
@@ -316,12 +351,15 @@ const icons = {
|
||||
folderFill,
|
||||
funnel,
|
||||
gear,
|
||||
globe2,
|
||||
google,
|
||||
grid,
|
||||
gripVertical,
|
||||
hash,
|
||||
hddStack,
|
||||
heart,
|
||||
house,
|
||||
inbox,
|
||||
infoCircle,
|
||||
journals,
|
||||
link,
|
||||
@@ -329,8 +367,10 @@ const icons = {
|
||||
listTask,
|
||||
listUl,
|
||||
microsoft,
|
||||
newspaper,
|
||||
nodePlus,
|
||||
pencil,
|
||||
paperclip,
|
||||
people,
|
||||
peopleFill,
|
||||
person,
|
||||
@@ -344,10 +384,13 @@ const icons = {
|
||||
plusCircle,
|
||||
printer,
|
||||
questionCircle,
|
||||
receipt,
|
||||
safe,
|
||||
scissors,
|
||||
search,
|
||||
send,
|
||||
slashCircle,
|
||||
shop,
|
||||
sliders2Vertical,
|
||||
sortAlphaDown,
|
||||
sortAlphaUpAlt,
|
||||
@@ -358,12 +401,15 @@ const icons = {
|
||||
tags,
|
||||
textIndentLeft,
|
||||
textLeft,
|
||||
telephone,
|
||||
threeDots,
|
||||
threeDotsVertical,
|
||||
trash,
|
||||
truck,
|
||||
uiRadios,
|
||||
unlock,
|
||||
upcScan,
|
||||
wallet2,
|
||||
windowStack,
|
||||
x,
|
||||
xCircle,
|
||||
|
||||
@@ -394,16 +394,10 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
|
||||
return "OK"
|
||||
|
||||
|
||||
def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
|
||||
"""
|
||||
Re-run parsing for the given documents.
|
||||
|
||||
Consumption workflows do not run here, so ``remote_ocr`` is how the user
|
||||
asks for the remote engine when it is not configured to handle everything.
|
||||
"""
|
||||
def reprocess(doc_ids: list[int]) -> Literal["OK"]:
|
||||
for document_id in doc_ids:
|
||||
update_document_content_maybe_archive_file.apply_async(
|
||||
kwargs={"document_id": document_id, "remote_ocr": remote_ocr},
|
||||
kwargs={"document_id": document_id},
|
||||
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
|
||||
)
|
||||
|
||||
|
||||
@@ -53,7 +53,6 @@ from documents.utils import copy_basic_file_stats
|
||||
from documents.utils import copy_file_with_basic_stats
|
||||
from documents.utils import run_subprocess
|
||||
from paperless.config import OcrConfig
|
||||
from paperless.config import RemoteOCRConfig
|
||||
from paperless.models import ArchiveFileGenerationChoices
|
||||
from paperless.parsers import ParserContext
|
||||
from paperless.parsers import ParserProtocol
|
||||
@@ -452,19 +451,12 @@ class ConsumerPlugin(
|
||||
except Exception as e:
|
||||
self.log.error(f"Error attempting to clean PDF: {e}")
|
||||
|
||||
# Workflows have already run at this point, so the metadata knows
|
||||
# whether this document was singled out for remote OCR
|
||||
allow_remote = (
|
||||
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
|
||||
)
|
||||
|
||||
# Based on the mime type, get the parser for that type
|
||||
parser_class: type[ParserProtocol] | None = (
|
||||
get_parser_registry().get_parser_for_file(
|
||||
mime_type,
|
||||
self.filename,
|
||||
self.working_copy,
|
||||
allow_remote=allow_remote,
|
||||
)
|
||||
)
|
||||
if not parser_class:
|
||||
|
||||
@@ -34,7 +34,6 @@ class DocumentMetadataOverrides:
|
||||
skip_asn_if_exists: bool = False
|
||||
version_label: str | None = None
|
||||
actor_id: int | None = None
|
||||
remote_ocr: bool = False
|
||||
|
||||
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
|
||||
"""
|
||||
@@ -58,8 +57,6 @@ class DocumentMetadataOverrides:
|
||||
self.actor_id = other.actor_id
|
||||
if other.skip_asn_if_exists:
|
||||
self.skip_asn_if_exists = True
|
||||
if other.remote_ocr:
|
||||
self.remote_ocr = True
|
||||
if other.version_label is not None:
|
||||
self.version_label = other.version_label
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ class Command(PaperlessCommand):
|
||||
"--ratio",
|
||||
default=85.0,
|
||||
type=float,
|
||||
help="Ratio to consider documents a match",
|
||||
help="Ratio to consider documents a match (0.0 - 100.0)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--delete",
|
||||
@@ -69,6 +69,17 @@ class Command(PaperlessCommand):
|
||||
action="store_true",
|
||||
help="Skip the confirmation prompt when used with --delete",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--url",
|
||||
default=None,
|
||||
type=str,
|
||||
help=(
|
||||
"Base URL of the Paperless instance (e.g. "
|
||||
"http://localhost:8000 or https://paperless.local). If set, matched "
|
||||
"documents are shown as clickable (usually ctrl+click) links to "
|
||||
"<url>/documents/<id>/details instead of by title."
|
||||
),
|
||||
)
|
||||
|
||||
def _render_results(
|
||||
self,
|
||||
@@ -76,6 +87,7 @@ class Command(PaperlessCommand):
|
||||
*,
|
||||
opt_ratio: float,
|
||||
do_delete: bool,
|
||||
base_url: str | None = None,
|
||||
) -> list[int]:
|
||||
"""Render match results as a Rich table. Returns list of PKs to delete."""
|
||||
if not matches:
|
||||
@@ -88,13 +100,22 @@ class Command(PaperlessCommand):
|
||||
)
|
||||
return []
|
||||
|
||||
# Fetch titles for matched documents in a single query.
|
||||
all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
|
||||
titles: dict[int, str] = dict(
|
||||
Document.objects.filter(pk__in=all_pks)
|
||||
.only("pk", "title")
|
||||
.values_list("pk", "title"),
|
||||
)
|
||||
# Fetch titles for matched documents in a single query, unless we're
|
||||
# going to show URLs instead.
|
||||
titles: dict[int, str] = {}
|
||||
if not base_url:
|
||||
all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
|
||||
titles = dict(
|
||||
Document.objects.filter(pk__in=all_pks)
|
||||
.only("pk", "title")
|
||||
.values_list("pk", "title"),
|
||||
)
|
||||
|
||||
def _cell(pk: int) -> str:
|
||||
if base_url:
|
||||
doc_url = f"{base_url.rstrip('/')}/documents/{pk}/details"
|
||||
return f"[link={doc_url}]{doc_url}[/link]"
|
||||
return f"[dim]#{pk}[/dim] {titles.get(pk, 'Unknown')}"
|
||||
|
||||
table = Table(
|
||||
title=f"Fuzzy Matches (threshold: {opt_ratio:.1f}%)",
|
||||
@@ -124,8 +145,8 @@ class Command(PaperlessCommand):
|
||||
|
||||
table.add_row(
|
||||
str(i),
|
||||
f"[dim]#{pk_a}[/dim] {titles.get(pk_a, 'Unknown')}",
|
||||
f"[dim]#{pk_b}[/dim] {titles.get(pk_b, 'Unknown')}",
|
||||
_cell(pk_a),
|
||||
_cell(pk_b),
|
||||
Text(f"{ratio:.1f}%", style=ratio_style),
|
||||
)
|
||||
maybe_delete_ids.append(pk_b)
|
||||
@@ -208,6 +229,7 @@ class Command(PaperlessCommand):
|
||||
matches,
|
||||
opt_ratio=opt_ratio,
|
||||
do_delete=options["delete"],
|
||||
base_url=options["url"],
|
||||
)
|
||||
|
||||
if options["delete"] and maybe_delete_ids:
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
from django.db import migrations
|
||||
from django.db import models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("documents", "0022_add_perf_indexes"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="savedview",
|
||||
name="icon",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("archive", "Archive"),
|
||||
("bank", "Bank"),
|
||||
("basket", "Basket"),
|
||||
("bell", "Bell"),
|
||||
("bookmark", "Bookmark"),
|
||||
("boxes", "Boxes"),
|
||||
("briefcase", "Briefcase"),
|
||||
("building", "Building"),
|
||||
("calculator", "Calculator"),
|
||||
("calendar", "Calendar"),
|
||||
("camera", "Camera"),
|
||||
("card-checklist", "Checklist"),
|
||||
("cash", "Cash"),
|
||||
("chat-left-text", "Chat"),
|
||||
("check-circle", "Check"),
|
||||
("clipboard", "Clipboard"),
|
||||
("clock-history", "Clock"),
|
||||
("credit-card", "Credit card"),
|
||||
("download", "Download"),
|
||||
("envelope", "Envelope"),
|
||||
("exclamation-triangle", "Warning"),
|
||||
("file-earmark", "File"),
|
||||
("file-earmark-check", "Checked file"),
|
||||
("file-earmark-lock", "Locked file"),
|
||||
("file-earmark-medical", "Medical file"),
|
||||
("file-earmark-person", "Person file"),
|
||||
("file-earmark-spreadsheet", "Spreadsheet"),
|
||||
("file-text", "Text file"),
|
||||
("files", "Files"),
|
||||
("folder", "Folder"),
|
||||
("funnel", "Filter"),
|
||||
("gear", "Gear"),
|
||||
("globe2", "Globe"),
|
||||
("hash", "Hash"),
|
||||
("heart", "Heart"),
|
||||
("house", "House"),
|
||||
("inbox", "Inbox"),
|
||||
("journals", "Journals"),
|
||||
("list-task", "Task list"),
|
||||
("newspaper", "Newspaper"),
|
||||
("paperclip", "Attachment"),
|
||||
("people", "People"),
|
||||
("person", "Person"),
|
||||
("printer", "Printer"),
|
||||
("receipt", "Receipt"),
|
||||
("safe", "Safe"),
|
||||
("search", "Search"),
|
||||
("send", "Send"),
|
||||
("shop", "Shop"),
|
||||
("stack", "Stack"),
|
||||
("stars", "Stars"),
|
||||
("tag", "Tag"),
|
||||
("tags", "Tags"),
|
||||
("telephone", "Telephone"),
|
||||
("truck", "Truck"),
|
||||
("upc-scan", "Barcode"),
|
||||
("wallet2", "Wallet"),
|
||||
],
|
||||
default="funnel",
|
||||
max_length=64,
|
||||
verbose_name="icon",
|
||||
),
|
||||
),
|
||||
]
|
||||
@@ -519,6 +519,68 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
|
||||
|
||||
|
||||
class SavedView(ModelWithOwner):
|
||||
class Icon(models.TextChoices):
|
||||
ARCHIVE = ("archive", _("Archive"))
|
||||
BANK = ("bank", _("Bank"))
|
||||
BASKET = ("basket", _("Basket"))
|
||||
BELL = ("bell", _("Bell"))
|
||||
BOOKMARK = ("bookmark", _("Bookmark"))
|
||||
BOXES = ("boxes", _("Boxes"))
|
||||
BRIEFCASE = ("briefcase", _("Briefcase"))
|
||||
BUILDING = ("building", _("Building"))
|
||||
CALCULATOR = ("calculator", _("Calculator"))
|
||||
CALENDAR = ("calendar", _("Calendar"))
|
||||
CAMERA = ("camera", _("Camera"))
|
||||
CARD_CHECKLIST = ("card-checklist", _("Checklist"))
|
||||
CASH = ("cash", _("Cash"))
|
||||
CHAT_LEFT_TEXT = ("chat-left-text", _("Chat"))
|
||||
CHECK_CIRCLE = ("check-circle", _("Check"))
|
||||
CLIPBOARD = ("clipboard", _("Clipboard"))
|
||||
CLOCK_HISTORY = ("clock-history", _("Clock"))
|
||||
CREDIT_CARD = ("credit-card", _("Credit card"))
|
||||
DOWNLOAD = ("download", _("Download"))
|
||||
ENVELOPE = ("envelope", _("Envelope"))
|
||||
EXCLAMATION_TRIANGLE = ("exclamation-triangle", _("Warning"))
|
||||
FILE_EARMARK = ("file-earmark", _("File"))
|
||||
FILE_EARMARK_CHECK = ("file-earmark-check", _("Checked file"))
|
||||
FILE_EARMARK_LOCK = ("file-earmark-lock", _("Locked file"))
|
||||
FILE_EARMARK_MEDICAL = ("file-earmark-medical", _("Medical file"))
|
||||
FILE_EARMARK_PERSON = ("file-earmark-person", _("Person file"))
|
||||
FILE_EARMARK_SPREADSHEET = (
|
||||
"file-earmark-spreadsheet",
|
||||
_("Spreadsheet"),
|
||||
)
|
||||
FILE_TEXT = ("file-text", _("Text file"))
|
||||
FILES = ("files", _("Files"))
|
||||
FOLDER = ("folder", _("Folder"))
|
||||
FUNNEL = ("funnel", _("Filter"))
|
||||
GEAR = ("gear", _("Gear"))
|
||||
GLOBE = ("globe2", _("Globe"))
|
||||
HASH = ("hash", _("Hash"))
|
||||
HEART = ("heart", _("Heart"))
|
||||
HOUSE = ("house", _("House"))
|
||||
INBOX = ("inbox", _("Inbox"))
|
||||
JOURNALS = ("journals", _("Journals"))
|
||||
LIST_TASK = ("list-task", _("Task list"))
|
||||
NEWSPAPER = ("newspaper", _("Newspaper"))
|
||||
PAPERCLIP = ("paperclip", _("Attachment"))
|
||||
PEOPLE = ("people", _("People"))
|
||||
PERSON = ("person", _("Person"))
|
||||
PRINTER = ("printer", _("Printer"))
|
||||
RECEIPT = ("receipt", _("Receipt"))
|
||||
SAFE = ("safe", _("Safe"))
|
||||
SEARCH = ("search", _("Search"))
|
||||
SEND = ("send", _("Send"))
|
||||
SHOP = ("shop", _("Shop"))
|
||||
STACK = ("stack", _("Stack"))
|
||||
STARS = ("stars", _("Stars"))
|
||||
TAG = ("tag", _("Tag"))
|
||||
TAGS = ("tags", _("Tags"))
|
||||
TELEPHONE = ("telephone", _("Telephone"))
|
||||
TRUCK = ("truck", _("Truck"))
|
||||
UPC_SCAN = ("upc-scan", _("Barcode"))
|
||||
WALLET = ("wallet2", _("Wallet"))
|
||||
|
||||
class DisplayMode(models.TextChoices):
|
||||
TABLE = ("table", _("Table"))
|
||||
SMALL_CARDS = ("smallCards", _("Small Cards"))
|
||||
@@ -541,6 +603,13 @@ class SavedView(ModelWithOwner):
|
||||
|
||||
name = models.CharField(_("name"), max_length=128)
|
||||
|
||||
icon = models.CharField(
|
||||
_("icon"),
|
||||
max_length=64,
|
||||
choices=Icon.choices,
|
||||
default=Icon.FUNNEL,
|
||||
)
|
||||
|
||||
sort_field = models.CharField(
|
||||
_("sort field"),
|
||||
max_length=128,
|
||||
|
||||
@@ -1383,6 +1383,7 @@ class SavedViewSerializer(OwnedObjectSerializer):
|
||||
fields = [
|
||||
"id",
|
||||
"name",
|
||||
"icon",
|
||||
"sort_field",
|
||||
"sort_reverse",
|
||||
"filter_rules",
|
||||
@@ -1744,7 +1745,7 @@ class DeleteDocumentsSerializer(DocumentSelectionSerializer):
|
||||
|
||||
|
||||
class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
|
||||
remote_ocr = serializers.BooleanField(required=False, default=False)
|
||||
pass
|
||||
|
||||
|
||||
class BulkEditSerializer(
|
||||
@@ -2086,13 +2087,6 @@ class BulkEditSerializer(
|
||||
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
|
||||
)
|
||||
|
||||
def _validate_parameters_reprocess(self, parameters) -> None:
|
||||
if "remote_ocr" in parameters:
|
||||
if not isinstance(parameters["remote_ocr"], bool):
|
||||
raise serializers.ValidationError("remote_ocr must be a boolean")
|
||||
else:
|
||||
parameters["remote_ocr"] = False
|
||||
|
||||
def validate_parameters_remove_password(self, parameters):
|
||||
if "password" not in parameters:
|
||||
raise serializers.ValidationError("password not specified")
|
||||
@@ -2157,8 +2151,6 @@ class BulkEditSerializer(
|
||||
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
|
||||
elif method == bulk_edit.remove_password:
|
||||
self.validate_parameters_remove_password(parameters)
|
||||
elif method == bulk_edit.reprocess:
|
||||
self._validate_parameters_reprocess(parameters)
|
||||
|
||||
return attrs
|
||||
|
||||
@@ -3222,6 +3214,13 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
|
||||
{"assign_title": f'Invalid f-string detected: "{e.args[0]}"'},
|
||||
)
|
||||
|
||||
if attrs.get("assign_custom_fields_values"):
|
||||
# Empty strings treated as None to avoid unexpected behavior
|
||||
attrs["assign_custom_fields_values"] = {
|
||||
field_id: (None if value == "" else value)
|
||||
for field_id, value in attrs["assign_custom_fields_values"].items()
|
||||
}
|
||||
|
||||
if (
|
||||
"type" in attrs
|
||||
and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL
|
||||
|
||||
+1
-10
@@ -66,7 +66,6 @@ from documents.utils import compute_checksum
|
||||
from documents.utils import identity
|
||||
from documents.workflows.utils import get_workflows_for_trigger
|
||||
from paperless.config import AIConfig
|
||||
from paperless.config import RemoteOCRConfig
|
||||
from paperless.logging import consume_task_id
|
||||
from paperless.parsers import ParserContext
|
||||
from paperless.parsers.registry import get_parser_registry
|
||||
@@ -338,17 +337,10 @@ def bulk_update_documents(document_ids) -> None:
|
||||
|
||||
|
||||
@shared_task
|
||||
def update_document_content_maybe_archive_file(
|
||||
document_id,
|
||||
*,
|
||||
remote_ocr: bool = False,
|
||||
) -> None:
|
||||
def update_document_content_maybe_archive_file(document_id) -> None:
|
||||
"""
|
||||
Re-creates OCR content and thumbnail for a document, and archive file if
|
||||
it exists.
|
||||
|
||||
Remote OCR is used only when the engine is configured to handle everything
|
||||
or if explicitly asked for via ``remote_ocr``.
|
||||
"""
|
||||
document = Document.objects.get(id=document_id)
|
||||
|
||||
@@ -358,7 +350,6 @@ def update_document_content_maybe_archive_file(
|
||||
mime_type,
|
||||
document.original_filename or "",
|
||||
document.source_path,
|
||||
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
|
||||
)
|
||||
|
||||
if not parser_class:
|
||||
|
||||
@@ -72,10 +72,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
||||
"barcode_enable_tag": None,
|
||||
"barcode_tag_mapping": None,
|
||||
"barcode_tag_split": None,
|
||||
"remote_ocr_engine": None,
|
||||
"remote_ocr_api_key": None,
|
||||
"remote_ocr_endpoint": None,
|
||||
"remote_ocr_mode": None,
|
||||
"ai_enabled": False,
|
||||
"llm_embedding_backend": None,
|
||||
"llm_embedding_model": None,
|
||||
@@ -874,49 +870,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
||||
config.refresh_from_db()
|
||||
self.assertEqual(config.llm_api_key, None)
|
||||
|
||||
def test_update_remote_ocr_api_key(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- Existing config with remote_ocr_api_key specified
|
||||
WHEN:
|
||||
- API to update remote_ocr_api_key is called with all *s
|
||||
- API to update remote_ocr_api_key is called with empty string
|
||||
THEN:
|
||||
- remote_ocr_api_key is unchanged
|
||||
- remote_ocr_api_key is set to None
|
||||
"""
|
||||
config = ApplicationConfiguration.objects.first()
|
||||
assert config is not None
|
||||
config.remote_ocr_api_key = "1234567890"
|
||||
config.save()
|
||||
|
||||
# Test with all *
|
||||
response = self.client.patch(
|
||||
f"{self.ENDPOINT}1/",
|
||||
json.dumps(
|
||||
{
|
||||
"remote_ocr_api_key": "*" * 32,
|
||||
},
|
||||
),
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
config.refresh_from_db()
|
||||
self.assertEqual(config.remote_ocr_api_key, "1234567890")
|
||||
# Test with empty string
|
||||
response = self.client.patch(
|
||||
f"{self.ENDPOINT}1/",
|
||||
json.dumps(
|
||||
{
|
||||
"remote_ocr_api_key": "",
|
||||
},
|
||||
),
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
config.refresh_from_db()
|
||||
self.assertEqual(config.remote_ocr_api_key, None)
|
||||
|
||||
def test_enable_ai_index_triggers_update(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -532,29 +532,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
|
||||
m.assert_called_once()
|
||||
args, kwargs = m.call_args
|
||||
self.assertEqual(args[0], [self.doc1.id])
|
||||
self.assertEqual(kwargs, {"remote_ocr": False})
|
||||
|
||||
@mock.patch("documents.views.bulk_edit.reprocess")
|
||||
def test_reprocess_documents_endpoint_remote_ocr(self, m) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- API data to reprocess a document with remote OCR requested
|
||||
WHEN:
|
||||
- API is called
|
||||
THEN:
|
||||
- reprocess is called with remote_ocr=True
|
||||
"""
|
||||
self.setup_mock(m, "reprocess")
|
||||
response = self.client.post(
|
||||
"/api/documents/reprocess/",
|
||||
json.dumps({"documents": [self.doc1.id], "remote_ocr": True}),
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
m.assert_called_once()
|
||||
args, kwargs = m.call_args
|
||||
self.assertEqual(args[0], [self.doc1.id])
|
||||
self.assertEqual(kwargs, {"remote_ocr": True})
|
||||
self.assertEqual(len(kwargs), 0)
|
||||
|
||||
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
|
||||
def test_api_set_storage_path(self, m) -> None:
|
||||
@@ -1575,29 +1553,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
|
||||
),
|
||||
)
|
||||
|
||||
def test_legacy_bulk_edit_reprocess_invalid_remote_ocr(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- The deprecated bulk_edit endpoint with a non-boolean remote_ocr
|
||||
WHEN:
|
||||
- API is called
|
||||
THEN:
|
||||
- The request is rejected rather than passed through to the task
|
||||
"""
|
||||
response = self.client.post(
|
||||
"/api/documents/bulk_edit/",
|
||||
json.dumps(
|
||||
{
|
||||
"documents": [self.doc1.id],
|
||||
"method": "reprocess",
|
||||
"parameters": {"remote_ocr": "yes please"},
|
||||
},
|
||||
),
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||
|
||||
@mock.patch("documents.views.bulk_edit.edit_pdf")
|
||||
def test_edit_pdf(self, m) -> None:
|
||||
self.setup_mock(m, "edit_pdf")
|
||||
|
||||
@@ -2905,18 +2905,20 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
|
||||
|
||||
v1 = SavedView.objects.get(name="test")
|
||||
self.assertEqual(v1.sort_field, "created2")
|
||||
self.assertEqual(v1.icon, SavedView.Icon.FUNNEL)
|
||||
self.assertEqual(v1.filter_rules.count(), 1)
|
||||
self.assertEqual(v1.owner, self.user)
|
||||
|
||||
response = self.client.patch(
|
||||
f"/api/saved_views/{v1.id}/",
|
||||
{"sort_reverse": True},
|
||||
{"sort_reverse": True, "icon": SavedView.Icon.RECEIPT},
|
||||
format="json",
|
||||
)
|
||||
|
||||
v1 = SavedView.objects.get(id=v1.id)
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
self.assertTrue(v1.sort_reverse)
|
||||
self.assertEqual(v1.icon, SavedView.Icon.RECEIPT)
|
||||
self.assertEqual(v1.filter_rules.count(), 1)
|
||||
|
||||
view["filter_rules"] = [{"rule_type": 12, "value": "secret"}]
|
||||
@@ -2936,6 +2938,13 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
|
||||
v1 = SavedView.objects.get(id=v1.id)
|
||||
self.assertEqual(v1.filter_rules.count(), 0)
|
||||
|
||||
response = self.client.patch(
|
||||
f"/api/saved_views/{v1.id}/",
|
||||
{"icon": "not-an-icon"},
|
||||
format="json",
|
||||
)
|
||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||
|
||||
def test_saved_view_display_options(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -60,10 +60,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
|
||||
},
|
||||
"email_enabled": False,
|
||||
"ai_enabled": False,
|
||||
"remote_ocr": {
|
||||
"configured": False,
|
||||
"mode": "always",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
@@ -158,50 +154,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
|
||||
str(response.data["settings"]),
|
||||
)
|
||||
|
||||
@override_settings(
|
||||
REMOTE_OCR_ENGINE="azureai",
|
||||
REMOTE_OCR_API_KEY="somekey",
|
||||
REMOTE_OCR_ENDPOINT="https://example.cognitiveservices.azure.com",
|
||||
REMOTE_OCR_MODE="workflow_only",
|
||||
)
|
||||
def test_settings_reports_remote_ocr_when_configured(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A fully configured remote OCR engine in workflow_only mode
|
||||
WHEN:
|
||||
- The ui_settings endpoint is called
|
||||
THEN:
|
||||
- The UI is told remote OCR is available and selective, so it can
|
||||
offer it where it would actually change something
|
||||
"""
|
||||
response = self.client.get(self.ENDPOINT, format="json")
|
||||
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
self.assertEqual(
|
||||
response.data["settings"]["remote_ocr"],
|
||||
{"configured": True, "mode": "workflow_only"},
|
||||
)
|
||||
|
||||
@override_settings(
|
||||
REMOTE_OCR_ENGINE="azureai",
|
||||
REMOTE_OCR_API_KEY=None,
|
||||
REMOTE_OCR_ENDPOINT=None,
|
||||
)
|
||||
def test_settings_reports_remote_ocr_incompletely_configured(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- An engine named but missing its endpoint and API key
|
||||
WHEN:
|
||||
- The ui_settings endpoint is called
|
||||
THEN:
|
||||
- It is reported as not configured, matching what the parser
|
||||
registry will actually do
|
||||
"""
|
||||
response = self.client.get(self.ENDPOINT, format="json")
|
||||
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
self.assertFalse(response.data["settings"]["remote_ocr"]["configured"])
|
||||
|
||||
@override_settings(
|
||||
OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
|
||||
GMAIL_OAUTH_CLIENT_ID="abc123",
|
||||
|
||||
@@ -422,6 +422,11 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
|
||||
json.dumps(
|
||||
{
|
||||
"assign_title": "",
|
||||
"assign_custom_fields": [self.cf1.id, self.cf2.id],
|
||||
"assign_custom_fields_values": {
|
||||
str(self.cf1.id): "",
|
||||
str(self.cf2.id): 0,
|
||||
},
|
||||
},
|
||||
),
|
||||
content_type="application/json",
|
||||
@@ -429,6 +434,10 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
|
||||
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
|
||||
action = WorkflowAction.objects.get(id=response.data["id"])
|
||||
self.assertIsNone(action.assign_title)
|
||||
self.assertEqual(
|
||||
action.assign_custom_fields_values,
|
||||
{str(self.cf1.id): None, str(self.cf2.id): 0},
|
||||
)
|
||||
|
||||
response = self.client.post(
|
||||
self.ENDPOINT_TRIGGERS,
|
||||
|
||||
@@ -1782,56 +1782,3 @@ class TestPDFActions(DirectoriesMixin, TestCase):
|
||||
|
||||
self.assertIn("wrong password", str(exc.exception))
|
||||
self.assertIn("Error removing password from document", cm.output[0])
|
||||
|
||||
|
||||
class TestBulkEditReprocess(DirectoriesMixin, TestCase):
|
||||
def setUp(self) -> None:
|
||||
super().setUp()
|
||||
|
||||
self.doc = Document.objects.create(
|
||||
title="test",
|
||||
checksum="A",
|
||||
mime_type="application/pdf",
|
||||
)
|
||||
|
||||
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
|
||||
def test_reprocess_defaults_to_local(self, mock_task: mock.Mock) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A reprocess request that says nothing about remote OCR
|
||||
WHEN:
|
||||
- reprocess is called
|
||||
THEN:
|
||||
- The task is queued without asking for the remote engine
|
||||
"""
|
||||
result = bulk_edit.reprocess([self.doc.id])
|
||||
|
||||
self.assertEqual(result, "OK")
|
||||
mock_task.apply_async.assert_called_once()
|
||||
_, kwargs = mock_task.apply_async.call_args
|
||||
self.assertEqual(
|
||||
kwargs["kwargs"],
|
||||
{"document_id": self.doc.id, "remote_ocr": False},
|
||||
)
|
||||
|
||||
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
|
||||
def test_reprocess_passes_remote_ocr(self, mock_task: mock.Mock) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A reprocess request that explicitly asks for remote OCR
|
||||
WHEN:
|
||||
- reprocess is called
|
||||
THEN:
|
||||
- The request is forwarded to the task for every document
|
||||
"""
|
||||
other = Document.objects.create(
|
||||
title="test2",
|
||||
checksum="B",
|
||||
mime_type="application/pdf",
|
||||
)
|
||||
|
||||
bulk_edit.reprocess([self.doc.id, other.id], remote_ocr=True)
|
||||
|
||||
self.assertEqual(mock_task.apply_async.call_count, 2)
|
||||
for call in mock_task.apply_async.call_args_list:
|
||||
self.assertTrue(call.kwargs["kwargs"]["remote_ocr"])
|
||||
|
||||
@@ -1559,72 +1559,6 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
|
||||
consumer.run_post_consume_script(doc)
|
||||
|
||||
|
||||
class TestConsumerRemoteOCR(
|
||||
DirectoriesMixin,
|
||||
FileSystemAssertsMixin,
|
||||
GetConsumerMixin,
|
||||
TestCase,
|
||||
):
|
||||
"""
|
||||
The consumer resolves the remote OCR mode and the per-document request from
|
||||
workflows into the allow_remote flag it hands to the parser registry.
|
||||
"""
|
||||
|
||||
def setUp(self) -> None:
|
||||
super().setUp()
|
||||
|
||||
patcher = mock.patch("documents.consumer.get_parser_registry")
|
||||
self.mock_registry = patcher.start()
|
||||
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
|
||||
self.addCleanup(patcher.stop)
|
||||
|
||||
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
|
||||
src = (
|
||||
Path(__file__).parent
|
||||
/ "samples"
|
||||
/ "documents"
|
||||
/ "originals"
|
||||
/ "0000001.pdf"
|
||||
)
|
||||
dst = self.dirs.scratch_dir / "sample.pdf"
|
||||
shutil.copy(src, dst)
|
||||
|
||||
with self.get_consumer(dst, overrides=overrides) as consumer:
|
||||
consumer.run()
|
||||
|
||||
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
|
||||
return kwargs["allow_remote"]
|
||||
|
||||
@override_settings(REMOTE_OCR_MODE="always")
|
||||
def test_always_mode_allows_remote(self) -> None:
|
||||
"""
|
||||
GIVEN: Remote OCR mode is 'always'.
|
||||
WHEN: A document is consumed without any workflow asking for it.
|
||||
THEN: The registry is allowed to pick the remote parser.
|
||||
"""
|
||||
self.assertTrue(self._consume())
|
||||
|
||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
|
||||
"""
|
||||
GIVEN: Remote OCR mode is 'workflow_only'.
|
||||
WHEN: A document is consumed and nothing asked for remote OCR.
|
||||
THEN: The remote parser is excluded.
|
||||
"""
|
||||
self.assertFalse(self._consume())
|
||||
|
||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
|
||||
"""
|
||||
GIVEN: Remote OCR mode is 'workflow_only'.
|
||||
WHEN: A workflow set remote_ocr on the metadata overrides.
|
||||
THEN: The registry is allowed to pick the remote parser.
|
||||
"""
|
||||
self.assertTrue(
|
||||
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
|
||||
)
|
||||
|
||||
|
||||
class TestMetadataOverrides(TestCase):
|
||||
def test_update_skip_asn_if_exists(self) -> None:
|
||||
base = DocumentMetadataOverrides()
|
||||
@@ -1632,20 +1566,6 @@ class TestMetadataOverrides(TestCase):
|
||||
base.update(incoming)
|
||||
self.assertTrue(base.skip_asn_if_exists)
|
||||
|
||||
def test_update_remote_ocr(self) -> None:
|
||||
base = DocumentMetadataOverrides()
|
||||
base.update(DocumentMetadataOverrides(remote_ocr=True))
|
||||
self.assertTrue(base.remote_ocr)
|
||||
|
||||
def test_update_remote_ocr_is_not_unset(self) -> None:
|
||||
"""
|
||||
A later workflow that says nothing must not undo an earlier one that
|
||||
asked for remote OCR.
|
||||
"""
|
||||
base = DocumentMetadataOverrides(remote_ocr=True)
|
||||
base.update(DocumentMetadataOverrides())
|
||||
self.assertTrue(base.remote_ocr)
|
||||
|
||||
def test_update_actor_and_version_label(self) -> None:
|
||||
base = DocumentMetadataOverrides(
|
||||
actor_id=1,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import os
|
||||
from io import StringIO
|
||||
from unittest.mock import patch
|
||||
|
||||
@@ -41,7 +42,7 @@ class TestFuzzyMatchCommand(TestCase):
|
||||
|
||||
def test_invalid_ratio_upper_limit(self) -> None:
|
||||
"""
|
||||
GIVEN:s
|
||||
GIVEN:
|
||||
- Invalid ratio above upper
|
||||
WHEN:
|
||||
- Command is called
|
||||
@@ -108,6 +109,45 @@ class TestFuzzyMatchCommand(TestCase):
|
||||
stdout, _ = self.call_command("--processes", "1")
|
||||
self.assertIn("Found 1 matching pair(s)", stdout)
|
||||
|
||||
def test_with_matches_and_url(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- 2 documents exist
|
||||
- Similarity between content is 86.667
|
||||
- --url is provided
|
||||
WHEN:
|
||||
- Command is called with --url
|
||||
THEN:
|
||||
- 1 match is returned from doc 1 to doc 2
|
||||
- No match from doc 2 to doc 1 reported
|
||||
- Output contains clickable links to the documents instead of titles
|
||||
"""
|
||||
# Content similarity is 86.667
|
||||
Document.objects.create(
|
||||
checksum="BEEFCAFE",
|
||||
title="A",
|
||||
content="first document scanned by bob",
|
||||
mime_type="application/pdf",
|
||||
filename="test.pdf",
|
||||
)
|
||||
Document.objects.create(
|
||||
checksum="DEADBEAF",
|
||||
title="A",
|
||||
content="first document scanned by alice",
|
||||
mime_type="application/pdf",
|
||||
filename="other_test.pdf",
|
||||
)
|
||||
with patch.dict(os.environ, {"COLUMNS": "200"}):
|
||||
stdout, _ = self.call_command(
|
||||
"--processes",
|
||||
"1",
|
||||
"--url",
|
||||
"http://localhost:8000",
|
||||
)
|
||||
self.assertIn("Found 1 matching pair(s)", stdout)
|
||||
self.assertIn("http://localhost:8000/documents/1/details", stdout)
|
||||
self.assertIn("http://localhost:8000/documents/2/details", stdout)
|
||||
|
||||
def test_with_3_matches(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -287,45 +287,6 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
||||
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
|
||||
|
||||
|
||||
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
|
||||
"""
|
||||
Consumption workflows do not run on reprocess, so the remote parser is
|
||||
used only in 'always' mode or when the caller explicitly asks for it.
|
||||
"""
|
||||
|
||||
def setUp(self) -> None:
|
||||
super().setUp()
|
||||
|
||||
patcher = mock.patch("documents.tasks.get_parser_registry")
|
||||
self.mock_registry = patcher.start()
|
||||
self.mock_registry.return_value.get_parser_for_file.return_value = None
|
||||
self.addCleanup(patcher.stop)
|
||||
|
||||
self.doc = Document.objects.create(
|
||||
title="test",
|
||||
content="my document",
|
||||
checksum="wow",
|
||||
mime_type="application/pdf",
|
||||
)
|
||||
|
||||
def _allow_remote(self, **kwargs) -> bool:
|
||||
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
|
||||
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
|
||||
return call_kwargs["allow_remote"]
|
||||
|
||||
@override_settings(REMOTE_OCR_MODE="always")
|
||||
def test_always_mode_allows_remote(self) -> None:
|
||||
self.assertTrue(self._allow_remote())
|
||||
|
||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
|
||||
self.assertFalse(self._allow_remote())
|
||||
|
||||
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
|
||||
self.assertTrue(self._allow_remote(remote_ocr=True))
|
||||
|
||||
|
||||
class TestAIIndex(DirectoriesMixin, TestCase):
|
||||
@override_settings(
|
||||
AI_ENABLED=True,
|
||||
|
||||
@@ -2000,6 +2000,55 @@ class TestWorkflows(
|
||||
r"Doc added in \w{3,}",
|
||||
) # Match any 3-letter month name
|
||||
|
||||
def test_document_updated_workflow_existing_custom_field_empty_value(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- Existing workflow with UPDATED trigger and action that assigns a custom field
|
||||
with an empty value
|
||||
WHEN:
|
||||
- Document is updated that already contains the field with a value
|
||||
THEN:
|
||||
- The existing value is left untouched, see GH #13627
|
||||
"""
|
||||
trigger = WorkflowTrigger.objects.create(
|
||||
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
|
||||
filter_has_document_type=self.dt,
|
||||
)
|
||||
action = WorkflowAction.objects.create()
|
||||
action.assign_custom_fields.add(self.cf1)
|
||||
action.assign_custom_fields_values = {self.cf1.pk: ""}
|
||||
action.save()
|
||||
w = Workflow.objects.create(
|
||||
name="Workflow 1",
|
||||
order=0,
|
||||
)
|
||||
w.triggers.add(trigger)
|
||||
w.actions.add(action)
|
||||
w.save()
|
||||
|
||||
doc = Document.objects.create(
|
||||
title="sample test",
|
||||
correspondent=self.c,
|
||||
original_filename="sample.pdf",
|
||||
)
|
||||
CustomFieldInstance.objects.create(
|
||||
document=doc,
|
||||
field=self.cf1,
|
||||
value_text="existing value",
|
||||
)
|
||||
|
||||
superuser = User.objects.create_superuser("superuser")
|
||||
self.client.force_authenticate(user=superuser)
|
||||
|
||||
self.client.patch(
|
||||
f"/api/documents/{doc.id}/",
|
||||
{"document_type": self.dt.id},
|
||||
format="json",
|
||||
)
|
||||
|
||||
doc.refresh_from_db()
|
||||
self.assertEqual(doc.custom_fields.get(field=self.cf1).value, "existing value")
|
||||
|
||||
def test_document_updated_workflow_existing_custom_field(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -236,10 +236,8 @@ from paperless import version
|
||||
from paperless.celery import app as celery_app
|
||||
from paperless.config import AIConfig
|
||||
from paperless.config import GeneralConfig
|
||||
from paperless.config import RemoteOCRConfig
|
||||
from paperless.models import ApplicationConfiguration
|
||||
from paperless.parsers.registry import get_parser_registry
|
||||
from paperless.parsers.remote import RemoteEngineConfig
|
||||
from paperless.serialisers import GroupSerializer
|
||||
from paperless.serialisers import UserSerializer
|
||||
from paperless.views import StandardPagination
|
||||
@@ -2269,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
|
||||
if not has_perms_owner_aware(request.user, "view_document", document):
|
||||
return HttpResponseForbidden("Insufficient permissions")
|
||||
|
||||
documents = [document]
|
||||
documents = Document.objects.filter(pk=document.pk)
|
||||
else:
|
||||
documents = Document.objects.filter(
|
||||
id__in=permitted_document_ids(request.user),
|
||||
@@ -4012,11 +4010,6 @@ class UiSettingsView(GenericAPIView[Any]):
|
||||
|
||||
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
|
||||
|
||||
ui_settings["remote_ocr"] = {
|
||||
"configured": RemoteEngineConfig.from_app_config().engine_is_valid(),
|
||||
"mode": RemoteOCRConfig().remote_ocr_mode,
|
||||
}
|
||||
|
||||
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
|
||||
manager = PaperlessMailOAuth2Manager()
|
||||
if settings.GMAIL_OAUTH_ENABLED:
|
||||
|
||||
@@ -105,7 +105,8 @@ def apply_assignment_to_document(
|
||||
field=field,
|
||||
document=document,
|
||||
).first()
|
||||
if instance and args[value_field_name] is not None:
|
||||
# empty string is indistinguishable from no value in the UI
|
||||
if instance and args[value_field_name] not in (None, ""):
|
||||
setattr(instance, value_field_name, args[value_field_name])
|
||||
instance.save()
|
||||
elif not instance:
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+3
-17
@@ -339,30 +339,16 @@ def check_deprecated_v2_ocr_env_vars(
|
||||
|
||||
@register()
|
||||
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
|
||||
# Import here because checks.py runs before the app registry is ready
|
||||
from paperless.models import RemoteOCRMode
|
||||
|
||||
errors = []
|
||||
|
||||
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
|
||||
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
|
||||
):
|
||||
errors.append(
|
||||
return [
|
||||
Error(
|
||||
"Azure AI remote parser requires endpoint and API key to be configured.",
|
||||
),
|
||||
)
|
||||
]
|
||||
|
||||
valid_modes = {mode.value for mode in RemoteOCRMode}
|
||||
if settings.REMOTE_OCR_MODE not in valid_modes:
|
||||
errors.append(
|
||||
Error(
|
||||
f"PAPERLESS_REMOTE_OCR_MODE is set to {settings.REMOTE_OCR_MODE!r}, "
|
||||
f"expected one of {sorted(valid_modes)}.",
|
||||
),
|
||||
)
|
||||
|
||||
return errors
|
||||
return []
|
||||
|
||||
|
||||
def get_tesseract_langs():
|
||||
|
||||
@@ -9,7 +9,6 @@ from paperless.models import CleanChoices
|
||||
from paperless.models import ColorConvertChoices
|
||||
from paperless.models import ModeChoices
|
||||
from paperless.models import OutputTypeChoices
|
||||
from paperless.models import RemoteOCRMode
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
@@ -186,45 +185,6 @@ class GeneralConfig(BaseConfig):
|
||||
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class RemoteOCRConfig(BaseConfig):
|
||||
"""
|
||||
Settings for the remote (cloud) OCR parser
|
||||
"""
|
||||
|
||||
remote_ocr_engine: str | None = dataclasses.field(init=False)
|
||||
remote_ocr_api_key: str | None = dataclasses.field(init=False)
|
||||
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
|
||||
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
app_config = self._get_config_instance()
|
||||
|
||||
self.remote_ocr_engine = (
|
||||
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
|
||||
)
|
||||
self.remote_ocr_api_key = (
|
||||
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
|
||||
)
|
||||
self.remote_ocr_endpoint = (
|
||||
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
|
||||
)
|
||||
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
|
||||
settings.REMOTE_OCR_MODE,
|
||||
)
|
||||
|
||||
@property
|
||||
def remote_ocr_by_default(self) -> bool:
|
||||
"""
|
||||
Whether every supported document goes to the remote engine.
|
||||
|
||||
When False the remote engine is used only for documents that
|
||||
explicitly asked for it, i.e. a workflow matched during consumption or
|
||||
the user ticked the box when reprocessing.
|
||||
"""
|
||||
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class AIConfig(BaseConfig):
|
||||
"""
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
# Generated by Django 5.2.16 on 2026-08-10 14:37
|
||||
|
||||
from django.db import migrations
|
||||
from django.db import models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="applicationconfiguration",
|
||||
name="remote_ocr_api_key",
|
||||
field=models.CharField(
|
||||
blank=True,
|
||||
max_length=1024,
|
||||
null=True,
|
||||
verbose_name="Sets the remote OCR API key",
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="applicationconfiguration",
|
||||
name="remote_ocr_endpoint",
|
||||
field=models.CharField(
|
||||
blank=True,
|
||||
max_length=256,
|
||||
null=True,
|
||||
verbose_name="Sets the remote OCR endpoint",
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="applicationconfiguration",
|
||||
name="remote_ocr_engine",
|
||||
field=models.CharField(
|
||||
blank=True,
|
||||
choices=[("azureai", "Azure AI Document Intelligence")],
|
||||
max_length=32,
|
||||
null=True,
|
||||
verbose_name="Sets the remote OCR engine",
|
||||
),
|
||||
),
|
||||
]
|
||||
@@ -1,27 +0,0 @@
|
||||
# Generated by Django 5.2.16 on 2026-08-10 15:43
|
||||
|
||||
from django.db import migrations
|
||||
from django.db import models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="applicationconfiguration",
|
||||
name="remote_ocr_mode",
|
||||
field=models.CharField(
|
||||
blank=True,
|
||||
choices=[
|
||||
("always", "All supported documents"),
|
||||
("workflow_only", "Only when a workflow enables it"),
|
||||
],
|
||||
max_length=32,
|
||||
null=True,
|
||||
verbose_name="Sets which documents are sent to the remote OCR engine",
|
||||
),
|
||||
),
|
||||
]
|
||||
@@ -74,23 +74,6 @@ class ColorConvertChoices(models.TextChoices):
|
||||
CMYK = ("CMYK", _("CMYK"))
|
||||
|
||||
|
||||
class RemoteOCREngine(models.TextChoices):
|
||||
"""
|
||||
Matches to PAPERLESS_REMOTE_OCR_ENGINE
|
||||
"""
|
||||
|
||||
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
|
||||
|
||||
|
||||
class RemoteOCRMode(models.TextChoices):
|
||||
"""
|
||||
Matches to PAPERLESS_REMOTE_OCR_MODE
|
||||
"""
|
||||
|
||||
ALWAYS = ("always", _("All supported documents"))
|
||||
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
|
||||
|
||||
|
||||
class LLMEmbeddingBackend(models.TextChoices):
|
||||
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
|
||||
HUGGINGFACE = ("huggingface", _("Huggingface"))
|
||||
@@ -303,44 +286,6 @@ class ApplicationConfiguration(AbstractSingletonModel):
|
||||
null=True,
|
||||
)
|
||||
|
||||
"""
|
||||
Settings for the remote OCR parser
|
||||
"""
|
||||
|
||||
# PAPERLESS_REMOTE_OCR_ENGINE
|
||||
remote_ocr_engine = models.CharField(
|
||||
verbose_name=_("Sets the remote OCR engine"),
|
||||
blank=True,
|
||||
null=True,
|
||||
max_length=32,
|
||||
choices=RemoteOCREngine.choices,
|
||||
)
|
||||
|
||||
# PAPERLESS_REMOTE_OCR_API_KEY
|
||||
remote_ocr_api_key = models.CharField(
|
||||
verbose_name=_("Sets the remote OCR API key"),
|
||||
blank=True,
|
||||
null=True,
|
||||
max_length=1024,
|
||||
)
|
||||
|
||||
# PAPERLESS_REMOTE_OCR_ENDPOINT
|
||||
remote_ocr_endpoint = models.CharField(
|
||||
verbose_name=_("Sets the remote OCR endpoint"),
|
||||
blank=True,
|
||||
null=True,
|
||||
max_length=256,
|
||||
)
|
||||
|
||||
# PAPERLESS_REMOTE_OCR_MODE
|
||||
remote_ocr_mode = models.CharField(
|
||||
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
|
||||
blank=True,
|
||||
null=True,
|
||||
max_length=32,
|
||||
choices=RemoteOCRMode.choices,
|
||||
)
|
||||
|
||||
"""
|
||||
AI related settings
|
||||
"""
|
||||
|
||||
@@ -134,11 +134,6 @@ class ParserProtocol(Protocol):
|
||||
Author or organisation name.
|
||||
url : str
|
||||
URL for documentation, source code, or issue tracker.
|
||||
|
||||
Parsers that send document content to a remote service should additionally
|
||||
set ``uses_remote_service = True`` so the registry can exclude them when
|
||||
remote processing has not been requested for a document. The attribute is
|
||||
optional so a parser that omits it is treated as fully local.
|
||||
"""
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -150,10 +145,6 @@ class ParserProtocol(Protocol):
|
||||
author: str
|
||||
url: str
|
||||
|
||||
# NOTE: uses_remote_service is not declared here, the registry reads it
|
||||
# with getattr(cls, ..., False) for backwards-compatibility with existing
|
||||
# parsers
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Class methods
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@@ -334,8 +334,6 @@ class ParserRegistry:
|
||||
mime_type: str,
|
||||
filename: str,
|
||||
path: Path | None = None,
|
||||
*,
|
||||
allow_remote: bool = True,
|
||||
) -> type[ParserProtocol] | None:
|
||||
"""Return the best parser class for the given file, or None.
|
||||
|
||||
@@ -361,11 +359,6 @@ class ParserRegistry:
|
||||
path:
|
||||
Optional filesystem path to the file. Forwarded to each
|
||||
parser's score method.
|
||||
allow_remote:
|
||||
When False, parsers that declare ``uses_remote_service = True``
|
||||
are excluded from consideration, so a document is never sent to
|
||||
a remote service. Parsers that do not declare the attribute
|
||||
are treated as local and are always considered.
|
||||
|
||||
Returns
|
||||
-------
|
||||
@@ -381,13 +374,6 @@ class ParserRegistry:
|
||||
if mime_type not in parser_class.supported_mime_types():
|
||||
continue
|
||||
|
||||
if not allow_remote and getattr(
|
||||
parser_class,
|
||||
"uses_remote_service",
|
||||
False,
|
||||
):
|
||||
continue
|
||||
|
||||
score = parser_class.score(mime_type, filename, path)
|
||||
if score is None:
|
||||
continue
|
||||
|
||||
@@ -3,7 +3,9 @@ Built-in remote-OCR document parser.
|
||||
|
||||
Handles documents by sending them to a configured remote OCR engine
|
||||
(currently Azure AI Vision / Document Intelligence) and retrieving both
|
||||
the extracted text and a searchable PDF with an embedded text layer.
|
||||
the extracted text and a searchable PDF with an embedded text layer. For
|
||||
born-digital PDFs that need no archive copy, the remote call is skipped
|
||||
entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
|
||||
|
||||
When no engine is configured, ``score()`` returns ``None`` so the parser
|
||||
is effectively invisible to the registry — the tesseract parser handles
|
||||
@@ -22,6 +24,8 @@ from typing import Self
|
||||
from django.conf import settings
|
||||
|
||||
from documents.parsers import ParseError
|
||||
from paperless.parsers.utils import extract_pdf_text
|
||||
from paperless.parsers.utils import post_process_text
|
||||
from paperless.version import __full_version_str__
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -57,18 +61,6 @@ class RemoteEngineConfig:
|
||||
self.api_key = api_key
|
||||
self.endpoint = endpoint
|
||||
|
||||
@classmethod
|
||||
def from_app_config(cls) -> Self:
|
||||
"""Build the config from the app config, falling back to the env."""
|
||||
from paperless.config import RemoteOCRConfig
|
||||
|
||||
app_config = RemoteOCRConfig()
|
||||
return cls(
|
||||
engine=app_config.remote_ocr_engine,
|
||||
api_key=app_config.remote_ocr_api_key,
|
||||
endpoint=app_config.remote_ocr_endpoint,
|
||||
)
|
||||
|
||||
def engine_is_valid(self) -> bool:
|
||||
"""Return True when the engine is known and fully configured."""
|
||||
return (
|
||||
@@ -82,8 +74,11 @@ class RemoteDocumentParser:
|
||||
"""Parse documents via a remote OCR API (currently Azure AI Vision).
|
||||
|
||||
This parser sends documents to a remote engine that returns both
|
||||
extracted text and a searchable PDF with an embedded text layer.
|
||||
It does not depend on Tesseract or ocrmypdf.
|
||||
extracted text and a searchable PDF with an embedded text layer,
|
||||
except when ``parse()`` is called with ``produce_archive=False`` for
|
||||
a PDF, in which case the remote call is skipped and only locally
|
||||
extracted text is returned (no archive). It does not depend on
|
||||
Tesseract or ocrmypdf.
|
||||
|
||||
Class attributes
|
||||
----------------
|
||||
@@ -95,9 +90,6 @@ class RemoteDocumentParser:
|
||||
Maintainer name.
|
||||
url : str
|
||||
Issue tracker / source URL.
|
||||
uses_remote_service : bool
|
||||
Content is sent to a remote service, True so that the registry
|
||||
can skip this parser if remote processing was not requested.
|
||||
"""
|
||||
|
||||
name: str = "Paperless-ngx Remote OCR Parser"
|
||||
@@ -105,8 +97,6 @@ class RemoteDocumentParser:
|
||||
author: str = "Paperless-ngx Contributors"
|
||||
url: str = "https://github.com/paperless-ngx/paperless-ngx"
|
||||
|
||||
uses_remote_service: bool = True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Class methods
|
||||
# ------------------------------------------------------------------
|
||||
@@ -155,7 +145,11 @@ class RemoteDocumentParser:
|
||||
20 when the remote engine is configured and the MIME type is
|
||||
supported, otherwise None.
|
||||
"""
|
||||
config = RemoteEngineConfig.from_app_config()
|
||||
config = RemoteEngineConfig(
|
||||
engine=settings.REMOTE_OCR_ENGINE,
|
||||
api_key=settings.REMOTE_OCR_API_KEY,
|
||||
endpoint=settings.REMOTE_OCR_ENDPOINT,
|
||||
)
|
||||
if not config.engine_is_valid():
|
||||
return None
|
||||
if mime_type not in _SUPPORTED_MIME_TYPES:
|
||||
@@ -173,8 +167,11 @@ class RemoteDocumentParser:
|
||||
Returns
|
||||
-------
|
||||
bool
|
||||
Always True — the remote engine always returns a PDF with an
|
||||
embedded text layer that serves as the archive copy.
|
||||
Always True — the remote engine is capable of returning a PDF
|
||||
with an embedded text layer to serve as the archive copy.
|
||||
Whether it actually does so for a given document depends on
|
||||
``produce_archive`` passed to :meth:`parse` (see there for when
|
||||
the remote engine call, and thus archive generation, is skipped).
|
||||
"""
|
||||
return True
|
||||
|
||||
@@ -231,6 +228,12 @@ class RemoteDocumentParser:
|
||||
) -> None:
|
||||
"""Send the document to the remote engine and store results.
|
||||
|
||||
When *produce_archive* is False for a PDF, the caller (via
|
||||
``documents.consumer.should_produce_archive``) has already determined
|
||||
that the document is born-digital and needs no archive — skip the
|
||||
remote engine entirely rather than re-OCRing it and creating a
|
||||
duplicate text layer.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
document_path:
|
||||
@@ -238,10 +241,14 @@ class RemoteDocumentParser:
|
||||
mime_type:
|
||||
Detected MIME type of the document.
|
||||
produce_archive:
|
||||
Ignored — the remote engine always returns a searchable PDF,
|
||||
which is stored as the archive copy regardless of this flag.
|
||||
Whether an archive copy is wanted. For PDFs, False skips the
|
||||
remote engine and uses locally-extracted text instead.
|
||||
"""
|
||||
config = RemoteEngineConfig.from_app_config()
|
||||
config = RemoteEngineConfig(
|
||||
engine=settings.REMOTE_OCR_ENGINE,
|
||||
api_key=settings.REMOTE_OCR_API_KEY,
|
||||
endpoint=settings.REMOTE_OCR_ENDPOINT,
|
||||
)
|
||||
|
||||
if not config.engine_is_valid():
|
||||
logger.warning(
|
||||
@@ -250,6 +257,16 @@ class RemoteDocumentParser:
|
||||
self._text = ""
|
||||
return
|
||||
|
||||
if not produce_archive and mime_type == "application/pdf":
|
||||
logger.debug(
|
||||
"Remote OCR: skipped — no archive requested, "
|
||||
"using locally-extracted text",
|
||||
)
|
||||
self._text = (
|
||||
post_process_text(extract_pdf_text(document_path, log=logger)) or ""
|
||||
)
|
||||
return
|
||||
|
||||
if config.engine == "azureai":
|
||||
self._text = self._azure_ai_vision_parse(document_path, config)
|
||||
|
||||
|
||||
@@ -219,13 +219,6 @@ class ApplicationConfigurationSerializer(
|
||||
allow_null=True,
|
||||
max_length=1024,
|
||||
)
|
||||
remote_ocr_api_key = ObfuscatedPasswordField(
|
||||
required=False,
|
||||
allow_null=True,
|
||||
max_length=1024,
|
||||
)
|
||||
|
||||
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
|
||||
|
||||
def run_validation(self, data):
|
||||
# Empty strings treated as None to avoid unexpected behavior
|
||||
@@ -237,13 +230,11 @@ class ApplicationConfigurationSerializer(
|
||||
data["language"] = None
|
||||
if "llm_output_language" in data and data["llm_output_language"] == "":
|
||||
data["llm_output_language"] = None
|
||||
for field in self.OBFUSCATED_FIELDS:
|
||||
if field in data and data[field] is not None:
|
||||
if data[field] == "":
|
||||
data[field] = None
|
||||
# Not a real value, don't overwrite the stored one
|
||||
elif len(data[field].replace("*", "")) == 0:
|
||||
del data[field]
|
||||
if "llm_api_key" in data and data["llm_api_key"] is not None:
|
||||
if data["llm_api_key"] == "":
|
||||
data["llm_api_key"] = None
|
||||
elif len(data["llm_api_key"].replace("*", "")) == 0:
|
||||
del data["llm_api_key"]
|
||||
return super().run_validation(data)
|
||||
|
||||
def update(self, instance, validated_data):
|
||||
|
||||
@@ -1197,7 +1197,6 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
|
||||
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
|
||||
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
|
||||
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
|
||||
REMOTE_OCR_MODE = os.getenv("PAPERLESS_REMOTE_OCR_MODE", "always")
|
||||
|
||||
################################################################################
|
||||
# AI Settings #
|
||||
|
||||
@@ -21,7 +21,6 @@ from unittest.mock import Mock
|
||||
import pytest
|
||||
|
||||
from documents.parsers import ParseError
|
||||
from paperless.models import ApplicationConfiguration
|
||||
from paperless.parsers import ParserContext
|
||||
from paperless.parsers import ParserProtocol
|
||||
from paperless.parsers.remote import RemoteDocumentParser
|
||||
@@ -34,10 +33,6 @@ if TYPE_CHECKING:
|
||||
from pytest_mock import MockerFixture
|
||||
|
||||
|
||||
# Remote ocr config from ApplicationConfiguration needs DB access
|
||||
pytestmark = pytest.mark.django_db
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-local fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -232,18 +227,6 @@ class TestRemoteParserScore:
|
||||
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
|
||||
assert score is not None and score > 10
|
||||
|
||||
@pytest.mark.usefixtures("no_engine_settings")
|
||||
def test_score_uses_app_config_when_env_unset(self) -> None:
|
||||
"""The app config alone is enough to activate the parser."""
|
||||
config = ApplicationConfiguration.objects.first()
|
||||
assert config is not None
|
||||
config.remote_ocr_engine = "azureai"
|
||||
config.remote_ocr_api_key = "app-config-key"
|
||||
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
|
||||
config.save()
|
||||
|
||||
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Properties
|
||||
@@ -354,6 +337,117 @@ class TestRemoteParserParse:
|
||||
assert remote_parser.get_date() is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse() — produce_archive=False skips the remote engine (PDFs only)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRemoteParserSkipsWhenNoArchiveWanted:
|
||||
"""When the caller has already decided no archive is needed for a PDF
|
||||
(documents.consumer.should_produce_archive), the remote engine call is
|
||||
skipped entirely in favor of locally-extracted text.
|
||||
"""
|
||||
|
||||
def test_pdf_skips_azure_when_no_archive_requested(
|
||||
self,
|
||||
remote_parser: RemoteDocumentParser,
|
||||
simple_digital_pdf_file: Path,
|
||||
azure_client: Mock,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN: produce_archive=False for a PDF
|
||||
WHEN: parse() is called
|
||||
THEN: Azure is never invoked, no archive is produced, and text
|
||||
comes from local pdftotext extraction
|
||||
"""
|
||||
remote_parser.parse(
|
||||
simple_digital_pdf_file,
|
||||
"application/pdf",
|
||||
produce_archive=False,
|
||||
)
|
||||
|
||||
azure_client.begin_analyze_document.assert_not_called()
|
||||
assert remote_parser.get_archive_path() is None
|
||||
assert remote_parser.get_text() != ""
|
||||
|
||||
def test_pdf_no_archive_requested_text_matches_local_extraction(
|
||||
self,
|
||||
remote_parser: RemoteDocumentParser,
|
||||
simple_digital_pdf_file: Path,
|
||||
azure_client: Mock,
|
||||
mocker: MockerFixture,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN: produce_archive=False for a PDF
|
||||
WHEN: parse() is called
|
||||
THEN: the returned text is exactly the locally-extracted text,
|
||||
not anything from the (unused) Azure mock
|
||||
"""
|
||||
mocker.patch(
|
||||
"paperless.parsers.remote.extract_pdf_text",
|
||||
return_value="Local digital text.",
|
||||
)
|
||||
|
||||
remote_parser.parse(
|
||||
simple_digital_pdf_file,
|
||||
"application/pdf",
|
||||
produce_archive=False,
|
||||
)
|
||||
|
||||
assert remote_parser.get_text() == "Local digital text."
|
||||
|
||||
def test_pdf_no_archive_requested_closes_no_client(
|
||||
self,
|
||||
remote_parser: RemoteDocumentParser,
|
||||
simple_digital_pdf_file: Path,
|
||||
azure_client: Mock,
|
||||
) -> None:
|
||||
remote_parser.parse(
|
||||
simple_digital_pdf_file,
|
||||
"application/pdf",
|
||||
produce_archive=False,
|
||||
)
|
||||
|
||||
azure_client.close.assert_not_called()
|
||||
|
||||
def test_non_pdf_still_calls_azure_when_no_archive_requested(
|
||||
self,
|
||||
remote_parser: RemoteDocumentParser,
|
||||
simple_digital_pdf_file: Path,
|
||||
azure_client: Mock,
|
||||
) -> None:
|
||||
"""
|
||||
Images have no local-text fallback, so produce_archive=False does
|
||||
not skip the remote engine for non-PDF MIME types.
|
||||
"""
|
||||
remote_parser.parse(
|
||||
simple_digital_pdf_file,
|
||||
"image/png",
|
||||
produce_archive=False,
|
||||
)
|
||||
|
||||
azure_client.begin_analyze_document.assert_called_once()
|
||||
assert remote_parser.get_text() == _DEFAULT_TEXT
|
||||
|
||||
@pytest.mark.usefixtures("no_engine_settings")
|
||||
def test_unconfigured_engine_takes_precedence_over_skip(
|
||||
self,
|
||||
remote_parser: RemoteDocumentParser,
|
||||
simple_digital_pdf_file: Path,
|
||||
) -> None:
|
||||
"""An unconfigured engine still short-circuits before the
|
||||
produce_archive check, returning empty text as before.
|
||||
"""
|
||||
remote_parser.parse(
|
||||
simple_digital_pdf_file,
|
||||
"application/pdf",
|
||||
produce_archive=False,
|
||||
)
|
||||
|
||||
assert remote_parser.get_text() == ""
|
||||
assert remote_parser.get_archive_path() is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse() — Azure failure path
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1277,8 +1277,6 @@ class TestParserFileTypes:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Remote ocr config from ApplicationConfiguration needs DB access
|
||||
@pytest.mark.django_db
|
||||
class TestRasterisedDocumentParserRegistry:
|
||||
def test_registered_in_defaults(self) -> None:
|
||||
from paperless.parsers.registry import ParserRegistry
|
||||
|
||||
@@ -655,23 +655,6 @@ class TestRemoteParserChecks:
|
||||
in msg.msg
|
||||
)
|
||||
|
||||
def test_valid_mode(self, settings: SettingsWrapper) -> None:
|
||||
settings.REMOTE_OCR_ENGINE = None
|
||||
settings.REMOTE_OCR_MODE = "workflow_only"
|
||||
|
||||
msgs = check_remote_parser_configured(None)
|
||||
|
||||
assert len(msgs) == 0
|
||||
|
||||
def test_invalid_mode(self, settings: SettingsWrapper) -> None:
|
||||
settings.REMOTE_OCR_ENGINE = None
|
||||
settings.REMOTE_OCR_MODE = "sometimes"
|
||||
|
||||
msgs = check_remote_parser_configured(None)
|
||||
|
||||
assert len(msgs) == 1
|
||||
assert "PAPERLESS_REMOTE_OCR_MODE is set to 'sometimes'" in msgs[0].msg
|
||||
|
||||
|
||||
class TestTesseractChecks:
|
||||
def test_default_language(self) -> None:
|
||||
|
||||
@@ -468,124 +468,6 @@ class TestParserRegistryGetParserForFile:
|
||||
assert result is AcceptingBuiltin
|
||||
|
||||
|
||||
class TestParserRegistryRemoteParsers:
|
||||
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
|
||||
|
||||
@staticmethod
|
||||
def _remote_parser_cls() -> type:
|
||||
class RemoteParser:
|
||||
name = "remote"
|
||||
version = "1.0"
|
||||
author = "A"
|
||||
url = "https://example.com/remote"
|
||||
uses_remote_service = True
|
||||
|
||||
@classmethod
|
||||
def supported_mime_types(cls):
|
||||
return {"text/plain": ".txt"}
|
||||
|
||||
@classmethod
|
||||
def score(cls, mime_type, filename, path=None):
|
||||
return 20
|
||||
|
||||
return RemoteParser
|
||||
|
||||
def test_remote_parser_wins_when_remote_allowed(
|
||||
self,
|
||||
dummy_parser_cls: type,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
|
||||
WHEN: get_parser_for_file() is called with allow_remote=True.
|
||||
THEN: The remote parser is returned.
|
||||
"""
|
||||
remote_parser_cls = self._remote_parser_cls()
|
||||
registry = ParserRegistry()
|
||||
registry.register_builtin(dummy_parser_cls)
|
||||
registry.register_builtin(remote_parser_cls)
|
||||
|
||||
result = registry.get_parser_for_file(
|
||||
"text/plain",
|
||||
"readme.txt",
|
||||
allow_remote=True,
|
||||
)
|
||||
assert result is remote_parser_cls
|
||||
|
||||
def test_remote_parser_skipped_when_remote_not_allowed(
|
||||
self,
|
||||
dummy_parser_cls: type,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
|
||||
WHEN: get_parser_for_file() is called with allow_remote=False.
|
||||
THEN: The local parser is returned despite its lower score.
|
||||
"""
|
||||
registry = ParserRegistry()
|
||||
registry.register_builtin(dummy_parser_cls)
|
||||
registry.register_builtin(self._remote_parser_cls())
|
||||
|
||||
result = registry.get_parser_for_file(
|
||||
"text/plain",
|
||||
"readme.txt",
|
||||
allow_remote=False,
|
||||
)
|
||||
assert result is dummy_parser_cls
|
||||
|
||||
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
|
||||
"""
|
||||
GIVEN: A registry whose only candidate declares uses_remote_service.
|
||||
WHEN: get_parser_for_file() is called with allow_remote=False.
|
||||
THEN: None is returned — the remote parser is never used as a
|
||||
fallback when remote processing was not requested.
|
||||
"""
|
||||
registry = ParserRegistry()
|
||||
registry.register_builtin(self._remote_parser_cls())
|
||||
|
||||
result = registry.get_parser_for_file(
|
||||
"text/plain",
|
||||
"readme.txt",
|
||||
allow_remote=False,
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_parser_without_attribute_treated_as_local(
|
||||
self,
|
||||
dummy_parser_cls: type,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN: A third-party parser predating uses_remote_service, so it does
|
||||
not declare the attribute at all.
|
||||
WHEN: get_parser_for_file() is called with allow_remote=False.
|
||||
THEN: It is still considered, i.e. treated as fully local, rather
|
||||
than raising AttributeError.
|
||||
"""
|
||||
assert not hasattr(dummy_parser_cls, "uses_remote_service")
|
||||
|
||||
registry = ParserRegistry()
|
||||
registry.register_builtin(dummy_parser_cls)
|
||||
|
||||
result = registry.get_parser_for_file(
|
||||
"text/plain",
|
||||
"readme.txt",
|
||||
allow_remote=False,
|
||||
)
|
||||
assert result is dummy_parser_cls
|
||||
|
||||
def test_remote_allowed_by_default(self) -> None:
|
||||
"""
|
||||
GIVEN: A registry containing only a remote parser.
|
||||
WHEN: get_parser_for_file() is called without allow_remote.
|
||||
THEN: The remote parser is returned — callers that do not opt in to
|
||||
the filter keep the previous behaviour.
|
||||
"""
|
||||
remote_parser_cls = self._remote_parser_cls()
|
||||
registry = ParserRegistry()
|
||||
registry.register_builtin(remote_parser_cls)
|
||||
|
||||
result = registry.get_parser_for_file("text/plain", "readme.txt")
|
||||
assert result is remote_parser_cls
|
||||
|
||||
|
||||
class TestDiscover:
|
||||
"""Verify entrypoint discovery in ParserRegistry.discover()."""
|
||||
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
from django.test import override_settings
|
||||
|
||||
from paperless.config import RemoteOCRConfig
|
||||
from paperless.models import RemoteOCRMode
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def null_app_config(mocker) -> MagicMock:
|
||||
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
|
||||
return mocker.MagicMock(
|
||||
remote_ocr_engine=None,
|
||||
remote_ocr_api_key=None,
|
||||
remote_ocr_endpoint=None,
|
||||
remote_ocr_mode=None,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def make_remote_ocr_config(mocker):
|
||||
def _make(app_config, **django_settings_overrides):
|
||||
mocker.patch(
|
||||
"paperless.config.BaseConfig._get_config_instance",
|
||||
return_value=app_config,
|
||||
)
|
||||
with override_settings(**django_settings_overrides):
|
||||
return RemoteOCRConfig()
|
||||
|
||||
return _make
|
||||
|
||||
|
||||
class TestRemoteOCRConfig:
|
||||
def test_falls_back_to_settings(
|
||||
self,
|
||||
make_remote_ocr_config,
|
||||
null_app_config,
|
||||
) -> None:
|
||||
cfg = make_remote_ocr_config(
|
||||
null_app_config,
|
||||
REMOTE_OCR_ENGINE="azureai",
|
||||
REMOTE_OCR_API_KEY="env-key",
|
||||
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
|
||||
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
|
||||
)
|
||||
assert cfg.remote_ocr_engine == "azureai"
|
||||
assert cfg.remote_ocr_api_key == "env-key"
|
||||
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
|
||||
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
|
||||
|
||||
def test_app_config_takes_precedence(
|
||||
self,
|
||||
make_remote_ocr_config,
|
||||
mocker,
|
||||
) -> None:
|
||||
app_config = mocker.MagicMock(
|
||||
remote_ocr_engine="azureai",
|
||||
remote_ocr_api_key="db-key",
|
||||
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
|
||||
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
|
||||
)
|
||||
cfg = make_remote_ocr_config(
|
||||
app_config,
|
||||
REMOTE_OCR_ENGINE=None,
|
||||
REMOTE_OCR_API_KEY="env-key",
|
||||
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
|
||||
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
|
||||
)
|
||||
assert cfg.remote_ocr_engine == "azureai"
|
||||
assert cfg.remote_ocr_api_key == "db-key"
|
||||
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
|
||||
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
|
||||
|
||||
def test_unset_everywhere(
|
||||
self,
|
||||
make_remote_ocr_config,
|
||||
null_app_config,
|
||||
) -> None:
|
||||
cfg = make_remote_ocr_config(
|
||||
null_app_config,
|
||||
REMOTE_OCR_ENGINE=None,
|
||||
REMOTE_OCR_API_KEY=None,
|
||||
REMOTE_OCR_ENDPOINT=None,
|
||||
)
|
||||
assert cfg.remote_ocr_engine is None
|
||||
assert cfg.remote_ocr_api_key is None
|
||||
assert cfg.remote_ocr_endpoint is None
|
||||
|
||||
|
||||
class TestRemoteOCRByDefault:
|
||||
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
|
||||
cfg = make_remote_ocr_config(
|
||||
null_app_config,
|
||||
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
|
||||
)
|
||||
|
||||
assert cfg.remote_ocr_by_default is True
|
||||
|
||||
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
|
||||
cfg = make_remote_ocr_config(
|
||||
null_app_config,
|
||||
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
|
||||
)
|
||||
|
||||
assert cfg.remote_ocr_by_default is False
|
||||
@@ -2,6 +2,8 @@ import json
|
||||
import logging
|
||||
import sys
|
||||
|
||||
from django.db.models import QuerySet
|
||||
|
||||
from documents.models import Document
|
||||
from paperless.config import AIConfig
|
||||
from paperless_ai.client import AIClient
|
||||
@@ -82,10 +84,21 @@ def _build_document_reference(
|
||||
|
||||
|
||||
def _get_document_references(
|
||||
documents: list[Document],
|
||||
documents: QuerySet[Document],
|
||||
top_nodes: list,
|
||||
) -> list[dict[str, int | str]]:
|
||||
allowed_documents = {doc.pk: doc for doc in documents}
|
||||
candidate_ids: set[int] = set()
|
||||
for node in top_nodes:
|
||||
try:
|
||||
candidate_ids.add(int(node.metadata["document_id"]))
|
||||
except (KeyError, TypeError, ValueError): # pragma: no cover
|
||||
continue
|
||||
|
||||
if not candidate_ids:
|
||||
return []
|
||||
|
||||
allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
|
||||
|
||||
references: list[dict[str, int | str]] = []
|
||||
seen_document_ids: set[int] = set()
|
||||
|
||||
@@ -119,7 +132,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
|
||||
|
||||
def stream_chat_with_documents(
|
||||
query_str: str,
|
||||
documents: list[Document],
|
||||
documents: QuerySet[Document],
|
||||
output_language: str | None = None,
|
||||
):
|
||||
try:
|
||||
@@ -135,10 +148,10 @@ def stream_chat_with_documents(
|
||||
|
||||
def _stream_chat_with_documents(
|
||||
query_str: str,
|
||||
documents: list[Document],
|
||||
documents: QuerySet[Document],
|
||||
output_language: str | None = None,
|
||||
):
|
||||
if not documents:
|
||||
if not documents.exists():
|
||||
yield CHAT_NO_CONTENT_MESSAGE
|
||||
return
|
||||
|
||||
@@ -148,7 +161,9 @@ def _stream_chat_with_documents(
|
||||
from llama_index.core.retrievers import VectorIndexRetriever
|
||||
|
||||
config = AIConfig()
|
||||
filters = _document_id_filters(str(doc.pk) for doc in documents)
|
||||
filters = _document_id_filters(
|
||||
str(pk) for pk in documents.values_list("pk", flat=True)
|
||||
)
|
||||
|
||||
# Hold the shared read lock for the whole operation: the query engine
|
||||
# retrieves from the vector store again during synthesis, so the connection
|
||||
|
||||
@@ -24,6 +24,7 @@ from paperless_ai.embedding import get_configured_model_name
|
||||
from paperless_ai.embedding import get_embedding_model
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from django.db.models import QuerySet
|
||||
from llama_index.core.schema import BaseNode
|
||||
|
||||
from paperless_ai.vector_store import PaperlessSqliteVecVectorStore
|
||||
@@ -252,6 +253,20 @@ def _safe_related_name(document: Document, field: str) -> str | None:
|
||||
return related.name if related else None
|
||||
|
||||
|
||||
def _document_index_queryset() -> "QuerySet[Document]":
|
||||
"""Document queryset with every relation build_document_node() /
|
||||
build_llm_index_text() touches -- correspondent, document_type,
|
||||
storage_path, tags, notes, custom_fields__field -- pre-loaded, so
|
||||
indexing one document costs a fixed handful of queries regardless of
|
||||
its tag or custom field count, instead of one query per related object.
|
||||
"""
|
||||
return Document.objects.select_related(
|
||||
"correspondent",
|
||||
"document_type",
|
||||
"storage_path",
|
||||
).prefetch_related("tags", "notes", "custom_fields__field")
|
||||
|
||||
|
||||
def build_document_node(
|
||||
document: Document,
|
||||
*,
|
||||
@@ -425,11 +440,7 @@ def update_llm_index(
|
||||
"Skipping LLM index update: migration check deferred; "
|
||||
"will retry next run."
|
||||
)
|
||||
documents = Document.objects.select_related(
|
||||
"correspondent",
|
||||
"document_type",
|
||||
"storage_path",
|
||||
).prefetch_related("tags", "notes", "custom_fields__field")
|
||||
documents = _document_index_queryset()
|
||||
no_documents = not documents.exists()
|
||||
|
||||
# Fast exit before touching config: nothing to index and no existing index.
|
||||
@@ -494,6 +505,21 @@ def update_llm_index(
|
||||
def llm_index_add_or_update_document(document: Document):
|
||||
"""Add or atomically replace a document's chunks in the index."""
|
||||
config = AIConfig()
|
||||
document_id = document.id
|
||||
# Re-fetch with the same select_related/prefetch_related shape as the
|
||||
# bulk path (update_llm_index()) uses: the caller's ``document`` instance
|
||||
# (e.g. straight off a signal) has none of that loaded, and
|
||||
# build_document_node()/build_llm_index_text() touch
|
||||
# correspondent/document_type/storage_path/tags/notes/custom_fields__field
|
||||
# -- without prefetching, that's one query per related object, including
|
||||
# one per custom field instance.
|
||||
document = _document_index_queryset().filter(pk=document_id).first()
|
||||
if document is None:
|
||||
logger.info(
|
||||
"Skipping LLM index update for document %s: it no longer exists.",
|
||||
document_id,
|
||||
)
|
||||
return
|
||||
new_nodes = build_document_node(
|
||||
document,
|
||||
chunk_size=config.llm_embedding_chunk_size,
|
||||
|
||||
@@ -735,8 +735,71 @@ class TestLlmIndexAddOrUpdateDocumentEmptyContent:
|
||||
)
|
||||
mock_load = mocker.patch("paperless_ai.indexing.load_or_build_index")
|
||||
|
||||
doc = MagicMock(spec=Document)
|
||||
doc.id = 42
|
||||
doc = DocumentFactory.create()
|
||||
# Must not raise
|
||||
indexing.llm_index_add_or_update_document(doc)
|
||||
|
||||
mock_load.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestLlmIndexAddOrUpdateDocumentPrefetch:
|
||||
"""llm_index_add_or_update_document must prefetch the relations
|
||||
build_document_node()/build_llm_index_text() touch, not re-query per
|
||||
tag/note/custom field.
|
||||
"""
|
||||
|
||||
def test_query_count_does_not_scale_with_custom_field_count(
|
||||
self,
|
||||
temp_llm_index_dir: Path,
|
||||
mock_embed_model: FakeEmbedding,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN a document with several custom fields, a note, and a tag
|
||||
WHEN it is incrementally indexed via llm_index_add_or_update_document
|
||||
THEN the query count stays flat instead of growing with the number
|
||||
of custom fields -- a regression here would add one query per
|
||||
custom field instance (instance.field.name unprefetched), see
|
||||
build_llm_index_text().
|
||||
"""
|
||||
doc = DocumentFactory.create()
|
||||
Note.objects.create(document=doc, note="a note")
|
||||
for i in range(5):
|
||||
field = CustomField.objects.create(
|
||||
name=f"Field {i}",
|
||||
data_type=CustomField.FieldDataType.STRING,
|
||||
)
|
||||
CustomFieldInstance.objects.create(
|
||||
document=doc,
|
||||
field=field,
|
||||
value_text="value",
|
||||
)
|
||||
|
||||
with CaptureQueriesContext(connection) as ctx:
|
||||
indexing.llm_index_add_or_update_document(doc)
|
||||
|
||||
# Flat regardless of custom field count -- an unprefetched
|
||||
# custom_fields__field would add one query per instance (5 here) on
|
||||
# top of this budget.
|
||||
assert len(ctx.captured_queries) <= 10
|
||||
|
||||
def test_skips_write_when_document_no_longer_exists(
|
||||
self,
|
||||
temp_llm_index_dir: Path,
|
||||
mock_embed_model: FakeEmbedding,
|
||||
mocker: pytest_mock.MockerFixture,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN a document that has been deleted since the caller looked it up
|
||||
(e.g. a race between a signal firing and its async task running)
|
||||
WHEN llm_index_add_or_update_document is called with that stale instance
|
||||
THEN it skips the write instead of raising Document.DoesNotExist
|
||||
"""
|
||||
doc = DocumentFactory.create()
|
||||
doc_id = doc.pk
|
||||
Document.objects.filter(pk=doc_id).delete()
|
||||
mock_load = mocker.patch("paperless_ai.indexing.load_or_build_index")
|
||||
|
||||
# Must not raise
|
||||
indexing.llm_index_add_or_update_document(doc)
|
||||
|
||||
@@ -793,8 +856,7 @@ class TestLlmIndexLocking:
|
||||
return_value=[mock_node],
|
||||
)
|
||||
|
||||
doc = MagicMock(spec=Document)
|
||||
doc.id = 1
|
||||
doc = DocumentFactory.create()
|
||||
indexing.llm_index_add_or_update_document(doc)
|
||||
|
||||
mock_store.upsert_document.assert_called_once()
|
||||
@@ -825,8 +887,7 @@ class TestLlmIndexLocking:
|
||||
return_value=[mock_node],
|
||||
)
|
||||
|
||||
doc = MagicMock(spec=Document)
|
||||
doc.id = 1
|
||||
doc = DocumentFactory.create()
|
||||
indexing.llm_index_add_or_update_document(doc)
|
||||
|
||||
mock_store.upsert_document.assert_not_called()
|
||||
@@ -863,8 +924,7 @@ class TestLlmIndexLocking:
|
||||
return_value=[mock_node],
|
||||
)
|
||||
|
||||
doc = MagicMock(spec=Document)
|
||||
doc.id = 1
|
||||
doc = DocumentFactory.create()
|
||||
indexing.llm_index_add_or_update_document(doc)
|
||||
|
||||
mock_store.upsert_document.assert_not_called()
|
||||
|
||||
@@ -3,10 +3,12 @@ from unittest.mock import MagicMock
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from django.db.models.signals import post_init
|
||||
from llama_index.core import settings as llama_settings
|
||||
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
|
||||
from llama_index.core.schema import TextNode
|
||||
|
||||
from documents.models import Document
|
||||
from documents.tests.factories import DocumentFactory
|
||||
from paperless_ai import chat
|
||||
from paperless_ai import indexing
|
||||
@@ -36,16 +38,6 @@ def patch_embed_nodes():
|
||||
yield mock_embed_nodes
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_document():
|
||||
doc = MagicMock()
|
||||
doc.pk = 1
|
||||
doc.title = "Test Document"
|
||||
doc.filename = "test_file.pdf"
|
||||
doc.content = "This is the document content."
|
||||
return doc
|
||||
|
||||
|
||||
def assert_chat_output(
|
||||
output: list[str],
|
||||
*,
|
||||
@@ -61,6 +53,13 @@ def assert_chat_output(
|
||||
}
|
||||
|
||||
|
||||
def _fake_documents_queryset(pks: list[int]) -> MagicMock:
|
||||
qs = MagicMock()
|
||||
qs.exists.return_value = bool(pks)
|
||||
qs.values_list.return_value = pks
|
||||
return qs
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("output_language", "expected_language_line"),
|
||||
[
|
||||
@@ -107,9 +106,10 @@ def test_build_refine_prompt(
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_stream_chat_with_one_document_retrieval(
|
||||
mock_document,
|
||||
patch_embed_nodes,
|
||||
) -> None:
|
||||
document = DocumentFactory.create(title="Test Document", content="ignored")
|
||||
documents = Document.objects.filter(pk=document.pk)
|
||||
with (
|
||||
patch("paperless_ai.chat.AIClient") as mock_client_cls,
|
||||
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
|
||||
@@ -124,22 +124,19 @@ def test_stream_chat_with_one_document_retrieval(
|
||||
mock_client_cls.return_value = mock_client
|
||||
mock_client.llm = MagicMock()
|
||||
|
||||
mock_node = TextNode(
|
||||
text="This is node content.",
|
||||
metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
|
||||
)
|
||||
mock_index = MagicMock()
|
||||
# Simulate get_nodes returning nodes (content exists)
|
||||
mock_index.vector_store.get_nodes.return_value = [mock_node]
|
||||
mock_index.vector_store.get_nodes.return_value = [
|
||||
TextNode(
|
||||
text="This is node content.",
|
||||
metadata={"document_id": str(document.pk), "title": "Test Document"},
|
||||
),
|
||||
]
|
||||
mock_load_index.return_value = mock_index
|
||||
|
||||
mock_retriever_instance = MagicMock()
|
||||
mock_retriever_instance.retrieve.return_value = [
|
||||
MagicMock(
|
||||
metadata={
|
||||
"document_id": str(mock_document.pk),
|
||||
"title": "Test Document",
|
||||
},
|
||||
metadata={"document_id": str(document.pk), "title": "Test Document"},
|
||||
),
|
||||
]
|
||||
|
||||
@@ -153,7 +150,7 @@ def test_stream_chat_with_one_document_retrieval(
|
||||
"llama_index.core.retrievers.VectorIndexRetriever",
|
||||
return_value=mock_retriever_instance,
|
||||
):
|
||||
output = list(stream_chat_with_documents("What is this?", [mock_document]))
|
||||
output = list(stream_chat_with_documents("What is this?", documents))
|
||||
|
||||
mock_query_engine.query.assert_called_once_with("What is this?")
|
||||
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
|
||||
@@ -166,13 +163,16 @@ def test_stream_chat_with_one_document_retrieval(
|
||||
output,
|
||||
expected_chunks=["chunk1", "chunk2"],
|
||||
expected_references=[
|
||||
{"id": mock_document.pk, "title": "Test Document"},
|
||||
{"id": document.pk, "title": "Test Document"},
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
|
||||
doc1 = DocumentFactory.create(title="Document 1", content="ignored")
|
||||
doc2 = DocumentFactory.create(title="Document 2", content="ignored")
|
||||
documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
|
||||
with (
|
||||
patch("paperless_ai.chat.AIClient") as mock_client_cls,
|
||||
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
|
||||
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
|
||||
mock_client_cls.return_value = mock_client
|
||||
mock_client.llm = MagicMock()
|
||||
|
||||
mock_node1 = TextNode(
|
||||
text="Content for doc 1.",
|
||||
metadata={"document_id": "1", "title": "Document 1"},
|
||||
)
|
||||
mock_node2 = TextNode(
|
||||
text="Content for doc 2.",
|
||||
metadata={"document_id": "2", "title": "Document 2"},
|
||||
)
|
||||
mock_index = MagicMock()
|
||||
# Simulate get_nodes returning nodes (content exists)
|
||||
mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2]
|
||||
mock_index.vector_store.get_nodes.return_value = [
|
||||
TextNode(
|
||||
text="Content for doc 1.",
|
||||
metadata={"document_id": str(doc1.pk), "title": "Document 1"},
|
||||
),
|
||||
TextNode(
|
||||
text="Content for doc 2.",
|
||||
metadata={"document_id": str(doc2.pk), "title": "Document 2"},
|
||||
),
|
||||
]
|
||||
mock_load_index.return_value = mock_index
|
||||
|
||||
mock_retriever_instance = MagicMock()
|
||||
mock_retriever_instance.retrieve.return_value = [
|
||||
MagicMock(metadata={"document_id": "1", "title": "Document 1"}),
|
||||
MagicMock(metadata={"document_id": "2", "title": "Document 2"}),
|
||||
MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
|
||||
MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
|
||||
]
|
||||
|
||||
mock_response_stream = MagicMock()
|
||||
@@ -210,14 +210,11 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
|
||||
mock_query_engine_cls.return_value = mock_query_engine
|
||||
mock_query_engine.query.return_value = mock_response_stream
|
||||
|
||||
doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
|
||||
doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
|
||||
|
||||
with patch(
|
||||
"llama_index.core.retrievers.VectorIndexRetriever",
|
||||
return_value=mock_retriever_instance,
|
||||
):
|
||||
output = list(stream_chat_with_documents("What's up?", [doc1, doc2]))
|
||||
output = list(stream_chat_with_documents("What's up?", documents))
|
||||
|
||||
mock_query_engine.query.assert_called_once_with("What's up?")
|
||||
patch_embed_nodes.assert_not_called()
|
||||
@@ -225,15 +222,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
|
||||
output,
|
||||
expected_chunks=["chunk1", "chunk2"],
|
||||
expected_references=[
|
||||
{"id": 1, "title": "Document 1"},
|
||||
{"id": 2, "title": "Document 2"},
|
||||
{"id": doc1.pk, "title": "Document 1"},
|
||||
{"id": doc2.pk, "title": "Document 2"},
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_stream_chat_empty_document_list() -> None:
|
||||
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
|
||||
output = list(stream_chat_with_documents("Any info?", []))
|
||||
output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
|
||||
mock_load_index.assert_not_called()
|
||||
assert output == ["Sorry, I couldn't find any content to answer your question."]
|
||||
|
||||
@@ -253,7 +250,9 @@ def test_stream_chat_no_matching_nodes() -> None:
|
||||
mock_index.vector_store.get_nodes.return_value = []
|
||||
mock_load_index.return_value = mock_index
|
||||
|
||||
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
|
||||
output = list(
|
||||
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
|
||||
)
|
||||
|
||||
assert output == ["Sorry, I couldn't find any content to answer your question."]
|
||||
|
||||
@@ -282,7 +281,9 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
|
||||
)
|
||||
mock_retriever_cls.return_value = mock_retriever
|
||||
|
||||
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
|
||||
output = list(
|
||||
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
|
||||
)
|
||||
|
||||
assert output == [CHAT_ERROR_MESSAGE]
|
||||
assert "Failed to stream document chat response" in caplog.text
|
||||
@@ -298,7 +299,12 @@ class TestStreamChatRetrieval:
|
||||
) -> None:
|
||||
doc = DocumentFactory.create(content="hello world")
|
||||
# Nothing indexed for this document yet.
|
||||
out = list(chat.stream_chat_with_documents("question?", [doc]))
|
||||
out = list(
|
||||
chat.stream_chat_with_documents(
|
||||
"question?",
|
||||
Document.objects.filter(pk=doc.pk),
|
||||
),
|
||||
)
|
||||
assert chat.CHAT_NO_CONTENT_MESSAGE in out
|
||||
|
||||
def test_chat_filter_contains_only_requested_document_ids(
|
||||
@@ -332,7 +338,12 @@ class TestStreamChatRetrieval:
|
||||
side_effect=capture_retriever,
|
||||
)
|
||||
|
||||
list(chat.stream_chat_with_documents("question?", [included]))
|
||||
list(
|
||||
chat.stream_chat_with_documents(
|
||||
"question?",
|
||||
Document.objects.filter(pk=included.pk),
|
||||
),
|
||||
)
|
||||
|
||||
assert captured_filters, "VectorIndexRetriever was never constructed"
|
||||
filt = captured_filters[0]
|
||||
@@ -340,3 +351,47 @@ class TestStreamChatRetrieval:
|
||||
filter_values = filt.filters[0].value
|
||||
assert str(included.pk) in filter_values
|
||||
assert str(excluded.pk) not in filter_values
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_get_document_references_only_queries_referenced_documents(
|
||||
self,
|
||||
django_assert_num_queries,
|
||||
) -> None:
|
||||
"""Building references must not hydrate every document the caller is
|
||||
permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
|
||||
the retriever actually returned nodes for.
|
||||
"""
|
||||
referenced = DocumentFactory.create(title="Referenced Document")
|
||||
# Many more documents are "accessible" but never referenced by a node.
|
||||
DocumentFactory.create_batch(200)
|
||||
|
||||
documents = Document.objects.all()
|
||||
top_nodes = [
|
||||
MagicMock(
|
||||
metadata={
|
||||
"document_id": str(referenced.pk),
|
||||
"title": "Referenced Document",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
hydrated_count = 0
|
||||
|
||||
def _count_hydration(sender, instance, **kwargs):
|
||||
nonlocal hydrated_count
|
||||
hydrated_count += 1
|
||||
|
||||
post_init.connect(_count_hydration, sender=Document)
|
||||
try:
|
||||
# One query: `documents.filter(pk__in=candidate_ids)` for the single
|
||||
# referenced id. No query should scale with the 200 unreferenced documents.
|
||||
with django_assert_num_queries(1):
|
||||
references = chat._get_document_references(documents, top_nodes)
|
||||
finally:
|
||||
post_init.disconnect(_count_hydration, sender=Document)
|
||||
|
||||
# The bug this guards against: the old code hydrated all 201 accessible
|
||||
# documents via `{doc.pk: doc for doc in documents}` before filtering by
|
||||
# top_nodes. Only the referenced document should ever be constructed.
|
||||
assert hydrated_count == 1
|
||||
assert references == [{"id": referenced.pk, "title": "Referenced Document"}]
|
||||
|
||||
Reference in New Issue
Block a user