Compare commits

..
Author SHA1 Message Date
stumpylog 03e05e0804 test(search): move schema fingerprint/version tests from PR2
Both files exercise documents.search._schema exclusively (build_schema,
schema_fingerprint, needs_rebuild, open_or_rebuild_index) with no
dependency on whoosh-compat query routing, so they belong with the
schema/field-registry work rather than PR2's query rewrite.
2026-08-28 09:46:57 -07:00
stumpylog 2f79549f92 style(search): remove em dashes from PR1 comments/docstrings 2026-08-28 09:35:24 -07:00
stumpylog 9e9f3f4e09 refactor(search): dispatch table for _public_field_descriptors
Collapse the TEXT/KEYWORD/U64/DATE/DATETIME if/elif chain into a
FieldKind -> (schema kind, tokenizer) lookup table. JSON stays an
explicit branch since it can emit a second, synthetic notes_text
descriptor.
2026-08-28 09:30:06 -07:00
stumpylog d0869bec01 feat(search): add whoosh-compat, the shared field table and the field registry 2026-08-28 08:42:14 -07:00
98 changed files with 2318 additions and 4593 deletions
+1 -2
View File
@@ -302,8 +302,7 @@ The following methods are supported:
- `delete`
- No `parameters` required
- `reprocess`
- Optional `parameters`: `{ "remote_ocr": true }` to send the documents to the
remote OCR engine, see [Remote OCR](usage.md#remote-ocr). Defaults to false.
- No `parameters` required
- `set_permissions`
- Requires `parameters`:
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
-12
View File
@@ -2066,18 +2066,6 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to None.
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
: Which documents are sent to the remote OCR engine.
- `always`: every document of a supported file type is sent to the remote
engine, bypassing the local OCR engine.
- `workflow_only`: documents are processed locally unless a workflow
explicitly enables remote OCR for them, letting you use the remote engine
selectively.
Defaults to "always".
## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
-14
View File
@@ -456,20 +456,6 @@ def score(
return 10
```
**Remote services**
If your parser sends document content to a remote service, declare it:
```python
class MyCustomParser:
uses_remote_service = True
```
Paperless-ngx excludes such parsers when the document being consumed has not
been marked for remote processing, so users can keep remote OCR off by default
and enable it selectively with a workflow. Parsers that do not declare the
attribute are treated as fully local and are always considered.
**Archive and rendition flags**
```python
+1 -51
View File
@@ -654,48 +654,6 @@ happened while it was still encrypted, that original version will likewise be mi
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
multiple PDF types with different passwords, create separate workflows for each use case.
##### Remote OCR {#workflow-action-remote-ocr}
"Remote OCR" actions send the document to the configured remote OCR engine instead of processing it
locally. To use remote OCR selectively, set the [remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE)
to `workflow_only` then add this action to a workflow that matches only the documents you
want sent to the remote engine. See [Remote OCR](#remote-ocr) for the engine setup. The action only works with
a **Consumption Started** trigger.
The action takes no options, its presence is what enables remote OCR for a matching document.
If the remote engine is not configured, or does not support the document's file type, the document is
processed locally instead and a warning is written to the log.
##### Apply AI Suggestions {#workflow-action-apply-ai-suggestions}
"Apply AI Suggestions" actions ask the configured AI service for title and metadata suggestions,
the same as the AI suggestions shown on the document detail page, except applied automatically and in bulk.
It requires [AI features](configuration.md#ai) to be enabled. You can specify:
- Which suggestions to apply: title, tags, correspondent, document type, storage path and / or created
date. Suggestions for fields you did not select are discarded.
- Whether to create missing items. By default only tags, correspondents and document types that
already exist are assigned and any other suggestion is dropped. With this enabled, suggested items
that do not exist are created. Storage paths are never created.
- Whether to overwrite existing values. By default a field is only filled in if it is currently empty.
Note that documents almost always already have a title and created date, so if you select those you
will usually want to enable this too. Tags are an exception: suggested tags are always added and
never replace the document's existing tags.
The action works with every trigger **except Consumption Started**, because suggestions are made from
the document's text, which does not exist until after the document has been processed.
Because the query to the AI service is slow, the action is queued and runs in the background rather
than as part of the workflow run itself. The document is updated once the suggestions come back.
!!! warning
Every matching document results in a query to the AI service, which may incur costs and have privacy
implications. Queries can be slow, so a workflow matching a large number of documents can occupy the
task queue, and delay consumption of new documents, etc. Consider narrowing the trigger filters,
running in small batches and / or increasing workers.
#### Workflow placeholders
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -1132,19 +1090,11 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
settings can be supplied as environment variables or via **Application Configuration**.
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
or page limitations (e.g. with a free tier).
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
unless a [remote OCR workflow action](#workflow-action-remote-ocr) enables it for them, so you can limit the remote
engine to particular documents.
Setting the mode to `workflow_only` also allows the **Reprocess** actions to selectively use remote OCR for individual documents.
## Architecture
Paperless-ngx consists of the following components:
+1
View File
@@ -77,6 +77,7 @@ dependencies = [
"torch~=2.13.0",
"watchfiles>=1.2",
"whitenoise~=6.11",
"whoosh-compat[tantivy]==0.1",
"zxing-cpp~=3.1.0",
]
[project.optional-dependencies]
+320 -484
View File
File diff suppressed because it is too large Load Diff
@@ -14,48 +14,43 @@
<a ngbNavLink>{{category}}</a>
<ng-template ngbNavContent>
<div class="p-3">
@for (section of getCategorySections(category); track section) {
@if (section) {
<h5 class="mt-4 mb-3">{{section}}</h5>
}
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category, section); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
<div class="card-title d-flex align-items-center">
<h6 class="mb-0">
{{option.title}}
</h6>
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs name="info-circle"></i-bs>
</a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
<div class="card-title d-flex align-items-center">
<h6 class="mb-0">
{{option.title}}
</h6>
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs name="info-circle"></i-bs>
</a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
}
</div>
</div>
}
</div>
}
</div>
}
</div>
</div>
</ng-template>
</li>
@@ -8,11 +8,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgSelectModule } from '@ng-select/ng-select'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of, throwError } from 'rxjs'
import {
ConfigCategory,
ConfigSection,
OutputTypeConfig,
} from 'src/app/data/paperless-config'
import { OutputTypeConfig } from 'src/app/data/paperless-config'
import { ConfigService } from 'src/app/services/config.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -162,24 +158,4 @@ describe('ConfigComponent', () => {
component.resetOption('barcodes_enabled')
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
})
it('should group options into sections within a category, or not', () => {
const sections = component.getCategorySections(ConfigCategory.OCR)
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
expect(
component
.getCategoryOptions(ConfigCategory.OCR)
.map((option) => option.key)
).toContain('output_type')
expect(
component
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
.map((option) => option.key)
).toEqual([
'remote_ocr_engine',
'remote_ocr_api_key',
'remote_ocr_endpoint',
'remote_ocr_mode',
])
})
})
@@ -74,20 +74,8 @@ export class ConfigComponent
return Object.values(ConfigCategory)
}
getCategorySections(category: string): string[] {
return [
...new Set(
PaperlessConfigOptions.filter((o) => o.category === category).map(
(o) => o.section ?? null // null means no section
)
),
]
}
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
return PaperlessConfigOptions.filter(
(o) => o.category === category && (o.section ?? null) === section
)
getCategoryOptions(category: string): ConfigOption[] {
return PaperlessConfigOptions.filter((o) => o.category === category)
}
initialConfig: PaperlessConfig
@@ -2,7 +2,7 @@
<button class="navbar-toggler d-md-none collapsed border-0" type="button" data-toggle="collapse"
data-target="#sidebarMenu" aria-controls="sidebarMenu" aria-expanded="false" aria-label="Toggle navigation"
(click)="closeMobileSearch(); toggleMenuCollapsed()">
<i-bs width="1.5em" height="1.5em" name="list"></i-bs>
<span class="navbar-toggler-icon"></span>
</button>
<a class="navbar-brand d-flex align-items-center me-0 ps-md-3 py-0 order-sm-0"
[ngClass]="{ 'slim': slimSidebarEnabled, '' : !slimSidebarEnabled }"
@@ -10,12 +10,16 @@
tourAnchor="tour.intro">
@if (!hasCustomBranding) {
<pngx-logo extra_classes="navbar-official-logo px-1" height="2.4rem"></pngx-logo>
<pngx-brand-mark class="brand-mark brand-mark-slim d-none"></pngx-brand-mark>
<svg class="brand-mark brand-mark-slim d-none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
} @else {
@if (customAppLogo) {
<img class="brand-logo" [src]="customAppLogo" alt="" />
} @else {
<pngx-brand-mark class="brand-mark"></pngx-brand-mark>
<svg class="brand-mark" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
}
<div class="brand-copy ms-2 text-truncate" [class.d-md-none]="slimSidebarEnabled">
<span class="brand-title text-truncate">{{ appTitle }}</span>
@@ -25,9 +29,9 @@
</div>
}
</a>
<div class="search-container flex-grow-1 py-2 pb-3 pb-sm-2 me-sm-auto order-3 order-sm-1"
<div class="search-container flex-grow-1 py-2 pb-3 pb-sm-2 px-3 ps-md-3 me-sm-auto order-3 order-sm-1"
[class.mobile-hidden]="mobileSearchHidden()">
<div class="col-12 header-search mx-auto">
<div class="col-12 header-search">
<pngx-global-search></pngx-global-search>
</div>
</div>
@@ -86,7 +90,7 @@
<ul class="nav flex-column">
<li class="nav-item app-link">
<a class="nav-link" routerLink="dashboard" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Dashboard" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Dashboard" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="house"></i-bs><span><ng-container i18n>Dashboard</ng-container></span>
</a>
@@ -95,7 +99,7 @@
<a class="nav-link" routerLink="documents" routerLinkActive="active"
[routerLinkActiveOptions]="{ paths: 'exact', queryParams: 'ignored', matrixParams: 'ignored', fragment: 'ignored' }"
(click)="closeMenu()"
ngbPopover="Documents" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Documents" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="files"></i-bs><span><ng-container i18n>Documents</ng-container></span>
</a>
@@ -114,7 +118,7 @@
(cdkDragEnded)="onDragEnd($event)">
<a class="nav-link" routerLink="view/{{view.id}}"
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="view.name"
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
popoverClass="popover-slim">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
@if (showSidebarCounts && !slimSidebarEnabled) {
@@ -152,7 +156,7 @@
<li class="nav-item w-100 app-link">
<a class="nav-link app-link" [class.text-truncate]="!slimSidebarEnabled" routerLink="documents/{{d.id}}"
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="d.title | documentTitle"
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
popoverClass="popover-slim">
<i-bs class="me-2" name="file-text"></i-bs><span>{{d.title | documentTitle}}</span>
<span class="close flex-column justify-content-center"
@@ -166,7 +170,7 @@
<li class="nav-item w-100 app-link">
<button type="button" class="nav-link nav-link-action app-link w-100 text-start"
[class.text-truncate]="!slimSidebarEnabled" (click)="closeAll()"
ngbPopover="Close all" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Close all" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="x"></i-bs><span><ng-container i18n>Close all</ng-container></span>
</button>
@@ -185,7 +189,7 @@
<div class="d-flex align-items-center attributes-row">
<a class="nav-link flex-fill" routerLink="attributes" routerLinkActive="active"
[routerLinkActiveOptions]="{ exact: !(slimSidebarEnabled || attributesSectionsCollapsed) }" (click)="closeMenu()"
ngbPopover="Attributes" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Attributes" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs name="stack"></i-bs><span class="ms-2"><ng-container i18n>Attributes</ng-container></span>
</a>
@@ -237,7 +241,7 @@
}
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.SavedView }">
<a class="nav-link" routerLink="savedviews" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Saved Views" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Saved Views" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="window-stack"></i-bs><span><ng-container i18n>Saved Views</ng-container></span>
</a>
@@ -246,7 +250,7 @@
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Workflow }"
tourAnchor="tour.workflows">
<a class="nav-link" routerLink="workflows" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Workflows" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Workflows" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="boxes"></i-bs><span><ng-container i18n>Workflows</ng-container></span>
</a>
@@ -254,14 +258,14 @@
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.MailAccount }"
tourAnchor="tour.mail">
<a class="nav-link" routerLink="mail" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Mail"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="envelope"></i-bs><span><ng-container i18n>Mail</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.Document }">
<a class="nav-link" routerLink="trash" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Trash"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="trash"></i-bs><span><ng-container i18n>Trash</ng-container></span>
</a>
@@ -277,21 +281,21 @@
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.UISettings }"
tourAnchor="tour.settings">
<a class="nav-link" routerLink="settings" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Settings" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Settings" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="gear"></i-bs><span><ng-container i18n>Settings</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.AppConfig }">
<a class="nav-link" routerLink="config" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Configuration" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Configuration" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="sliders2-vertical"></i-bs><span><ng-container i18n>Configuration</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.User }">
<a class="nav-link" routerLink="usersgroups" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Users & Groups" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Users & Groups" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="people"></i-bs><span><ng-container i18n>Users & Groups</ng-container></span>
</a>
@@ -300,7 +304,7 @@
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.PaperlessTask }"
tourAnchor="tour.file-tasks">
<a class="nav-link" routerLink="tasks" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Tasks" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Tasks" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="list-task"></i-bs><span><ng-container i18n>Tasks</ng-container>@if (tasksService.needsAttentionTasks.length > 0) {
<span><span class="badge bg-danger ms-2 d-inline">{{tasksService.needsAttentionTasks.length}}</span></span>
@@ -313,26 +317,26 @@
@if (permissionsService.isAdmin()) {
<li class="nav-item app-link">
<a class="nav-link" routerLink="logs" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Logs"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="text-left"></i-bs><span><ng-container i18n>Logs</ng-container></span>
</a>
</li>
}
<li class="nav-item mt-2" tourAnchor="tour.outro">
<a class="text-muted small d-flex align-items-center flex-wrap text-decoration-none nav-anchor"
<a class="px-3 py-2 text-muted small d-flex align-items-center flex-wrap text-decoration-none"
target="_blank" rel="noopener noreferrer" href="https://docs.paperless-ngx.com" ngbPopover="Documentation"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="d-flex me-2" name="question-circle"></i-bs><span><ng-container i18n>Documentation</ng-container></span>
</a>
</li>
<li class="nav-item" [class.visually-hidden]="slimSidebarEnabled">
<div class="text-muted small d-flex align-items-center flex-wrap nav-label">
<div class="px-3 py-0 text-muted small d-flex align-items-center flex-wrap">
<div class="me-3">
<a class="text-muted text-decoration-none" target="_blank" rel="noopener noreferrer"
href="https://github.com/paperless-ngx/paperless-ngx" ngbPopover="GitHub" i18n-ngbPopover
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
{{ versionString }}
</a>
@@ -67,23 +67,6 @@
main.mobile-search-hidden {
padding-top: 56px;
}
.navbar-toggler {
padding-left: calc(12px - (1.5em * 2.5 / 16));
}
}
.search-container {
padding-left: 1rem;
padding-right: 1rem;
}
// Below sm the search gets its own full-width row, so line it up with main's content edge
@media (max-width: 575.98px) {
.search-container {
padding-left: 12px;
padding-right: 12px;
}
}
main {
@@ -114,7 +97,7 @@ main {
li.nav-item span.badge {
display: inline-block;
margin-right: -4px;
margin-right: 2px;
}
}
@@ -158,11 +141,18 @@ main {
}
}
.sidebar.slim {
li.nav-item span.badge {
display: inline-block;
margin-right: 2px;
}
}
.sidebar-slim-toggler {
display: block;
position: fixed;
left: calc(var(--pngx-sidebar-width) - 12px);
bottom: 16px;
top: 72px;
z-index: 996;
--bs-btn-padding-x: 0.35rem;
--bs-btn-padding-y: 0.125rem;
@@ -237,10 +227,6 @@ main {
}
}
.sidebar .nav-anchor, .sidebar .nav-label {
padding: .25rem .7rem;
}
.attributes-row {
border-radius: .55rem;
margin: .1rem 0;
@@ -417,7 +403,7 @@ main {
:host ::ng-deep .navbar-official-logo {
.leaf {
fill: color-mix(in srgb, var(--pngx-primary-text-contrast) 85%, var(--bs-primary)) !important;
fill: color-mix(in srgb, var(--pngx-primary-text-contrast) 70%, var(--bs-primary)) !important;
}
.text {
@@ -458,7 +444,7 @@ main {
.header-search {
width: 100%;
max-width: 55rem;
max-width: 44rem;
}
@media screen and (max-width: 575.98px) {
@@ -472,7 +458,6 @@ main {
.navbar-toggler {
grid-area: toggler;
color: var(--pngx-primary-text-contrast);
text-align: left;
}
@@ -45,7 +45,6 @@ import { TasksService } from 'src/app/services/tasks.service'
import { ToastService } from 'src/app/services/toast.service'
import { environment } from 'src/environments/environment'
import { ChatComponent } from '../chat/chat/chat.component'
import { BrandMarkComponent } from '../common/logo/brand-mark/brand-mark.component'
import { LogoComponent } from '../common/logo/logo.component'
import { ProfileEditDialogComponent } from '../common/profile-edit-dialog/profile-edit-dialog.component'
import { DocumentDetailComponent } from '../document-detail/document-detail.component'
@@ -62,7 +61,6 @@ const SCROLL_THRESHOLD = 16
imports: [
GlobalSearchComponent,
LogoComponent,
BrandMarkComponent,
DocumentTitlePipe,
IfPermissionsDirective,
ToastsDropdownComponent,
@@ -280,10 +278,6 @@ export class AppFrameComponent
})
}
get slimSidebarPopoversEnabled(): boolean {
return this.slimSidebarEnabled && !this.isMobileViewport()
}
get attributesSectionsCollapsed(): boolean {
this.settingsService.trackChanges()
return this.settingsService
@@ -1,28 +0,0 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (messageBold) {
<p class="text-break"><b>{{messageBold}}</b></p>
}
@if (message) {
<p class="mb-0 text-break" [innerHTML]="message"></p>
}
@if (showRemoteOcr) {
<div class="form-check mt-3">
<input class="form-check-input" type="checkbox" id="reprocessRemoteOcr" [(ngModel)]="remoteOcr" />
<label class="form-check-label" for="reprocessRemoteOcr" i18n>Use remote OCR</label>
<div class="form-text" i18n>Sends the document to the configured remote OCR service, which may incur costs.</div>
</div>
}
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
{{btnCaption}}
</button>
</div>
@@ -1,72 +0,0 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { RemoteOCRModeConfig } from 'src/app/data/paperless-config'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { SettingsService } from 'src/app/services/settings.service'
import { ReprocessConfirmDialogComponent } from './reprocess-confirm-dialog.component'
describe('ReprocessConfirmDialogComponent', () => {
let component: ReprocessConfirmDialogComponent
let fixture: ComponentFixture<ReprocessConfirmDialogComponent>
let settingsService: SettingsService
const createComponent = (configured: boolean, mode: string) => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, configured)
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_MODE, mode)
fixture = TestBed.createComponent(ReprocessConfirmDialogComponent)
component = fixture.componentInstance
fixture.detectChanges()
}
beforeEach(async () => {
TestBed.configureTestingModule({
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
imports: [ReprocessConfirmDialogComponent],
}).compileComponents()
settingsService = TestBed.inject(SettingsService)
})
it('should not request remote OCR by default', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.remoteOcr).toBeFalsy()
})
it('should not offer remote OCR when no engine is configured', () => {
createComponent(false, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should not offer remote OCR when it already handles every document', () => {
createComponent(true, RemoteOCRModeConfig.ALWAYS)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should offer remote OCR when configured and selective', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeTruthy()
const checkbox = fixture.nativeElement.querySelector('#reprocessRemoteOcr')
expect(checkbox).not.toBeNull()
checkbox.click()
fixture.detectChanges()
expect(component.remoteOcr).toBeTruthy()
})
})
@@ -1,20 +0,0 @@
import { Component, inject } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { SettingsService } from 'src/app/services/settings.service'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-reprocess-confirm-dialog',
templateUrl: './reprocess-confirm-dialog.component.html',
imports: [FormsModule],
})
export class ReprocessConfirmDialogComponent extends ConfirmDialogComponent {
private settings = inject(SettingsService)
remoteOcr: boolean = false
public get showRemoteOcr(): boolean {
// Hidden when it is not configured, or when it already handles every document anyway.
return this.settings.remoteOCRIsSelectable
}
}
@@ -455,52 +455,6 @@
</div>
</div>
}
@case (WorkflowActionType.RemoteOcr) {
<div class="row">
<div class="col">
<p class="text-muted small" i18n>The document will be sent to the configured remote OCR service. May incur costs.</p>
</div>
</div>
}
@case (WorkflowActionType.ApplyAiSuggestions) {
<div class="row">
<div class="col">
<p class="text-muted small" i18n>The document will be sent to the configured AI service for suggestions. Consider costs and privacy.</p>
<pngx-input-select
i18n-title
title="Apply suggestions for"
[items]="aiSuggestionFieldOptions"
[multiple]="true"
formControlName="ai_suggestion_fields"
[error]="error?.actions?.[i]?.ai_suggestion_fields"
hint="Suggestions for fields that are not selected are discarded."
i18n-hint
></pngx-input-select>
</div>
</div>
<div class="row">
<div class="col-md-6">
<pngx-input-switch
[horizontal]="true"
i18n-title
title="Create missing items"
formControlName="ai_create_missing"
hint="Create suggested tags, correspondents and document types that do not exist yet."
i18n-hint
></pngx-input-switch>
</div>
<div class="col-md-6">
<pngx-input-switch
[horizontal]="true"
i18n-title
title="Overwrite existing values"
formControlName="ai_overwrite_existing"
hint="Apply suggestions even if the document already has a value. Tags are always added, never replaced."
i18n-hint
></pngx-input-switch>
</div>
</div>
}
}
</div>
</ng-template>
@@ -22,7 +22,6 @@ import {
} from 'src/app/data/matching-model'
import { Workflow } from 'src/app/data/workflow'
import {
AISuggestionField,
WorkflowAction,
WorkflowActionType,
} from 'src/app/data/workflow-action'
@@ -30,7 +29,6 @@ import {
DocumentSource,
WorkflowTriggerType,
} from 'src/app/data/workflow-trigger'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
@@ -50,7 +48,6 @@ import { TagsComponent } from '../../input/tags/tags.component'
import { TextComponent } from '../../input/text/text.component'
import { EditDialogMode } from '../edit-dialog.component'
import {
AI_SUGGESTION_FIELD_OPTIONS,
DOCUMENT_SOURCE_OPTIONS,
SCHEDULE_DATE_FIELD_OPTIONS,
TriggerFilterType,
@@ -227,12 +224,7 @@ describe('WorkflowEditDialogComponent', () => {
).toEqual('Document Added')
expect(component.getTriggerTypeOptionName(null)).toEqual('')
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
// Remote OCR is absent until the workflow has a consumption trigger
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.RemoteOcr
)
)
expect(component.actionTypeOptions).toEqual(WORKFLOW_ACTION_OPTIONS)
expect(
component.getActionTypeOptionName(WorkflowActionType.Assignment)
).toEqual('Assignment')
@@ -241,231 +233,14 @@ describe('WorkflowEditDialogComponent', () => {
SCHEDULE_DATE_FIELD_OPTIONS
)
// Email, remote OCR and AI all disabled
// Email disabled
jest.spyOn(settingsService, 'get').mockReturnValue(false)
component.ngOnInit()
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) =>
a.id !== WorkflowActionType.Email &&
a.id !== WorkflowActionType.RemoteOcr &&
a.id !== WorkflowActionType.ApplyAiSuggestions
)
WORKFLOW_ACTION_OPTIONS.filter((a) => a.id !== WorkflowActionType.Email)
)
})
it('should offer remote OCR only for consumption workflows', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// A consumption trigger makes the action reachable
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Any other trigger type runs after the document has been parsed
component.object = {
name: 'Workflow 2',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should offer remote OCR on a trigger added to a new workflow', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
component.ngOnInit()
// Nothing for the action to apply to yet
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
// addTrigger creates the form field with emitEvent false, so the options
// have to be computed on read rather than cached from valueChanges
component.addTrigger()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Switching that trigger to a type that runs after parsing removes it
component.triggerFields
.at(0)
.get('type')
.setValue(WorkflowTriggerType.DocumentAdded)
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should keep remote OCR listed when an action already uses it', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [{ type: WorkflowActionType.RemoteOcr }],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
})
it('should not offer remote OCR when no engine is configured', () => {
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should offer apply AI suggestions unless every trigger is consumption', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Consumption runs before the document has been parsed, so there would be
// no content to make suggestions from
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.ApplyAiSuggestions
)
// A second, usable trigger is enough
component.object = {
name: 'Workflow 2',
order: 0,
enabled: true,
triggers: [
{ type: WorkflowTriggerType.Consumption },
{ type: WorkflowTriggerType.DocumentAdded },
],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.ApplyAiSuggestions
)
})
it('should keep apply AI suggestions listed when an action already uses it', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [{ type: WorkflowActionType.ApplyAiSuggestions }],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.ApplyAiSuggestions
)
})
it('should not offer apply AI suggestions when AI is disabled', () => {
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.AI_ENABLED)
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.ApplyAiSuggestions
)
})
it('should create form fields for apply AI suggestions options', () => {
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [
{
type: WorkflowActionType.ApplyAiSuggestions,
ai_suggestion_fields: [
AISuggestionField.Title,
AISuggestionField.Tags,
],
ai_create_missing: true,
ai_overwrite_existing: true,
},
],
} as Workflow
component.ngOnInit()
const action = component.actionFields.at(0)
expect(action.get('ai_suggestion_fields').value).toEqual([
AISuggestionField.Title,
AISuggestionField.Tags,
])
expect(action.get('ai_create_missing').value).toBeTruthy()
expect(action.get('ai_overwrite_existing').value).toBeTruthy()
expect(component.aiSuggestionFieldOptions).toEqual(
AI_SUGGESTION_FIELD_OPTIONS
)
})
it('should default apply AI suggestions options on a new action', () => {
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [],
} as Workflow
component.addAction()
const action = component.actionFields.at(component.actionFields.length - 1)
expect(action.get('ai_suggestion_fields').value).toEqual([])
expect(action.get('ai_create_missing').value).toBeFalsy()
expect(action.get('ai_overwrite_existing').value).toBeFalsy()
})
it('should support add and remove triggers and actions', () => {
component.object = workflow
component.addTrigger()
@@ -30,7 +30,6 @@ import { StoragePath } from 'src/app/data/storage-path'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { Workflow } from 'src/app/data/workflow'
import {
AISuggestionField,
WorkflowAction,
WorkflowActionType,
} from 'src/app/data/workflow-action'
@@ -149,41 +148,6 @@ export const WORKFLOW_ACTION_OPTIONS = [
id: WorkflowActionType.MoveToTrash,
name: $localize`Move to trash`,
},
{
id: WorkflowActionType.RemoteOcr,
name: $localize`Remote OCR`,
},
{
id: WorkflowActionType.ApplyAiSuggestions,
name: $localize`Apply AI suggestions`,
},
]
export const AI_SUGGESTION_FIELD_OPTIONS = [
{
id: AISuggestionField.Title,
name: $localize`Title`,
},
{
id: AISuggestionField.Tags,
name: $localize`Tags`,
},
{
id: AISuggestionField.Correspondent,
name: $localize`Correspondent`,
},
{
id: AISuggestionField.DocumentType,
name: $localize`Document type`,
},
{
id: AISuggestionField.StoragePath,
name: $localize`Storage path`,
},
{
id: AISuggestionField.Created,
name: $localize`Created date`,
},
]
export enum TriggerFilterType {
@@ -540,6 +504,8 @@ export class WorkflowEditDialogComponent
expandedItem: number = null
readonly allowedActionTypes = signal([])
private readonly triggerFilterOptionsMap = new WeakMap<
FormArray,
TriggerFilterOption[]
@@ -582,58 +548,13 @@ export class WorkflowEditDialogComponent
this.checkRemovalActionFields.bind(this)
)
this.checkRemovalActionFields(this.objectForm.value)
}
private allowedActionTypes: typeof WORKFLOW_ACTION_OPTIONS = null
private getAllowedActionTypes() {
let allowed = WORKFLOW_ACTION_OPTIONS
if (!this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.Email)
}
// Remote OCR is decided before the document is parsed, so it is only
// offered for workflows that run at consumption.
const formWorkflow: Workflow = this.objectForm?.value
const remoteOcrUsable =
this.settingsService.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
(formWorkflow?.triggers?.some(
(trigger) => trigger.type === WorkflowTriggerType.Consumption
) ||
formWorkflow?.actions?.some(
(action) => action.type === WorkflowActionType.RemoteOcr
))
if (!remoteOcrUsable) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.RemoteOcr)
}
// Only available after consumption. Unlike remote OCR this is hidden only
// once every trigger is consumption, so it stays offered on a workflow
// that has no triggers yet.
const aiSuggestionsUsable =
this.settingsService.get(SETTINGS_KEYS.AI_ENABLED) &&
(!formWorkflow?.triggers?.length ||
formWorkflow.triggers.some(
(trigger) => trigger.type !== WorkflowTriggerType.Consumption
) ||
formWorkflow.actions?.some(
(action) => action.type === WorkflowActionType.ApplyAiSuggestions
))
if (!aiSuggestionsUsable) {
allowed = allowed.filter(
(a) => a.id !== WorkflowActionType.ApplyAiSuggestions
)
}
if (
this.allowedActionTypes?.length === allowed.length &&
this.allowedActionTypes.every((a, i) => a.id === allowed[i].id)
) {
return this.allowedActionTypes
}
this.allowedActionTypes = allowed
return allowed
this.allowedActionTypes.set(
this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)
? WORKFLOW_ACTION_OPTIONS
: WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.Email
)
)
}
private checkRemovalActionFields(formWorkflow: Workflow) {
@@ -1277,11 +1198,6 @@ export class WorkflowEditDialogComponent
passwords: new FormControl(
this.formatPasswords(action.passwords ?? [])
),
ai_suggestion_fields: new FormControl(
action.ai_suggestion_fields ?? []
),
ai_create_missing: new FormControl(!!action.ai_create_missing),
ai_overwrite_existing: new FormControl(!!action.ai_overwrite_existing),
}),
{ emitEvent }
)
@@ -1363,18 +1279,13 @@ export class WorkflowEditDialogComponent
get actionTypeOptions() {
this.settingsService.trackChanges()
// Computed on read rather than cached
return this.getAllowedActionTypes()
return this.allowedActionTypes()
}
getActionTypeOptionName(type: WorkflowActionType): string {
return this.actionTypeOptions.find((t) => t.id === type)?.name ?? ''
}
get aiSuggestionFieldOptions() {
return AI_SUGGESTION_FIELD_OPTIONS
}
addAction() {
if (!this.object) {
this.object = Object.assign({}, this.objectForm.value)
@@ -1428,9 +1339,6 @@ export class WorkflowEditDialogComponent
include_document: false,
},
passwords: [],
ai_suggestion_fields: [],
ai_create_missing: false,
ai_overwrite_existing: false,
}
this.object.actions.push(action)
this.createActionField(action)
@@ -1,3 +0,0 @@
<svg class="w-100 h-100 d-block" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" fill="currentColor" aria-hidden="true">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 545 B

@@ -1,14 +0,0 @@
import { Component, input } from '@angular/core'
@Component({
selector: 'pngx-brand-mark',
templateUrl: './brand-mark.component.html',
host: {
'[style.width]': 'width()',
'[style.height]': 'height()',
},
})
export class BrandMarkComponent {
readonly width = input<string>(null)
readonly height = input<string>(null)
}
@@ -1,6 +1,6 @@
<div class="row pt-3 pb-2 align-items-center">
<div class="row pt-3 pb-3 pb-md-2 align-items-center">
<div class="col-md text-truncate">
<h3 class="d-flex align-items-center mb-2 mb-md-1" style="line-height: 1.4">
<h3 class="d-flex align-items-center mb-1" style="line-height: 1.4">
<span class="text-truncate">{{title()}}</span>
@if (id()) {
<span class="badge bg-primary text-primary-text-contrast ms-3 small fs-normal cursor-pointer" (click)="copyID()">
@@ -12,10 +12,10 @@
</span>
}
@if (subTitle()) {
<span class="page-subtitle h6 mb-0 mt-1 fw-normal ms-md-3 text-truncate" style="line-height: 1.4">{{subTitle()}}</span>
<span class="h6 mb-0 mt-1 d-block d-md-inline fw-normal ms-md-3 text-truncate" style="line-height: 1.4">{{subTitle()}}</span>
}
@if (info()) {
<button class="btn btn-sm btn-link text-muted p-0 ms-2 p-md-2 ms-md-0" title="What's this?" i18n-title type="button" [ngbPopover]="infoPopover" [autoClose]="true">
<button class="btn btn-sm btn-link text-muted p-0 p-md-2" title="What's this?" i18n-title type="button" [ngbPopover]="infoPopover" [autoClose]="true">
<i-bs name="question-circle"></i-bs>
</button>
<ng-template #infoPopover>
@@ -1,6 +1,5 @@
h3 {
min-height: calc(1.325rem + 0.9vw);
flex-wrap: wrap;
.badge {
font-size: 0.65rem;
@@ -8,16 +7,6 @@ h3 {
}
}
// Drop the subtitle onto its own and shrink it
@media (max-width: 767.98px) {
h3 > .page-subtitle {
flex: 0 0 100%;
margin-top: 0 !important;
font-size: .7rem;
line-height: 1.3 !important;
}
}
:host {
display: block;
margin-bottom: .35rem;
@@ -1,18 +1,10 @@
<ngb-alert class="welcome-widget bg-light shadow-sm pe-3 position-relative overflow-hidden" type="primary" [dismissible]="true" (closed)="dismiss.emit(true)">
<pngx-brand-mark class="welcome-watermark d-none d-md-block position-absolute top-50 translate-middle-y text-primary pe-none" width="26rem" height="26rem"></pngx-brand-mark>
<div class="d-flex align-items-start gap-3 gap-md-4 position-relative">
<pngx-brand-mark class="d-none d-sm-block flex-shrink-0 text-primary" width="3rem" height="3rem"></pngx-brand-mark>
<div class="flex-grow-1">
<h4 class="alert-heading text-primary fw-semibold"><ng-container i18n>Paperless-ngx is running!</ng-container> 🎉</h4>
<div class="welcome-copy text-dark">
<p i18n>You're ready to start uploading documents! Explore the various features of this web app on your own, or start a quick tour using the button below.</p>
<p class="mb-0" i18n>More detail on how to use and configure Paperless-ngx is always available in the <a href="https://docs.paperless-ngx.com" target="_blank">documentation</a>.</p>
</div>
<hr class="opacity-100">
<div class="d-flex flex-column flex-sm-row align-items-sm-center gap-3">
<p class="m-0 text-secondary"><em i18n>Thanks for being a part of the Paperless-ngx community!</em></p>
<button class="btn btn-primary ms-sm-auto flex-shrink-0 align-self-start align-self-sm-auto" (click)="tourService.start()"><ng-container i18n>Start the tour</ng-container> &rarr;</button>
</div>
</div>
<ngb-alert class="pe-3" type="primary" [dismissible]="true" (closed)="dismiss.emit(true)">
<h4 class="alert-heading"><ng-container i18n>Paperless-ngx is running!</ng-container> 🎉</h4>
<p i18n>You're ready to start uploading documents! Explore the various features of this web app on your own, or start a quick tour using the button below.</p>
<p i18n>More detail on how to use and configure Paperless-ngx is always available in the <a href="https://docs.paperless-ngx.com" target="_blank">documentation</a>.</p>
<hr>
<div class="d-flex align-items-end">
<p class="lead fs-6 m-0"><em i18n>Thanks for being a part of the Paperless-ngx community!</em></p>
<button class="btn btn-primary ms-auto flex-shrink-0" (click)="tourService.start()"><ng-container i18n>Start the tour</ng-container> &rarr;</button>
</div>
</ngb-alert>
@@ -1,16 +0,0 @@
.welcome-widget {
border-color: color-mix(in srgb, var(--bs-primary) 50%, transparent);
hr {
border-top-color: color-mix(in srgb, var(--bs-primary) 25%, transparent);
}
}
.welcome-copy {
max-width: 68ch;
}
.welcome-watermark {
right: -6rem;
opacity: .07;
}
@@ -1,13 +1,12 @@
import { Component, EventEmitter, Output, inject } from '@angular/core'
import { NgbAlertModule } from '@ng-bootstrap/ng-bootstrap'
import { TourService } from 'ngx-ui-tour-ng-bootstrap'
import { BrandMarkComponent } from '../../../common/logo/brand-mark/brand-mark.component'
@Component({
selector: 'pngx-welcome-widget',
templateUrl: './welcome-widget.component.html',
styleUrls: ['./welcome-widget.component.scss'],
imports: [NgbAlertModule, BrandMarkComponent],
imports: [NgbAlertModule],
})
export class WelcomeWidgetComponent {
readonly tourService = inject(TourService)
@@ -963,24 +963,12 @@ describe('DocumentDetailComponent', () => {
component.reprocess()
const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, false)
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] })
expect(modalSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should pass remote OCR choice when reprocessing', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
reprocessSpy.mockReturnValue(of(true))
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
component.reprocess()
openModal.componentInstance.remoteOcr = true
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, true)
})
it('should show error if redo ocr call fails', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
@@ -97,7 +97,6 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
import * as UTIF from 'utif'
import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
@@ -1399,7 +1398,7 @@ export class DocumentDetailComponent
}
reprocess() {
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Reprocess confirm`
@@ -1410,10 +1409,7 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false)
this.documentsService
.reprocessDocuments(
{ documents: [this.document().id] },
modal.componentInstance.remoteOcr
)
.reprocessDocuments({ documents: [this.document().id] })
.subscribe({
next: () => {
this.toastService.showInfo(
@@ -1122,7 +1122,6 @@ describe('BulkEditorComponent', () => {
req.flush(true)
expect(req.request.body).toEqual({
documents: [3, 4],
remote_ocr: false,
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
@@ -52,7 +52,6 @@ import { flattenTags } from 'src/app/utils/flatten-tags'
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
import { MergeAsVersionsConfirmDialogComponent } from '../../common/confirm-dialog/merge-as-versions-confirm-dialog/merge-as-versions-confirm-dialog.component'
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
@@ -917,7 +916,7 @@ export class BulkEditorComponent
}
reprocessSelected() {
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Reprocess confirm`
@@ -931,10 +930,7 @@ export class BulkEditorComponent
modal.componentInstance.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.reprocessDocuments(
this.getSelectionQuery(),
modal.componentInstance.remoteOcr
)
this.documentService.reprocessDocuments(this.getSelectionQuery())
)
})
}
-55
View File
@@ -54,10 +54,6 @@ export const ConfigCategory = {
AI: $localize`AI Settings`,
}
export const ConfigSection = {
RemoteOCR: $localize`Remote OCR`,
}
export const LLMEmbeddingBackendConfig = {
OPENAI_LIKE: 'openai-like',
HUGGINGFACE: 'huggingface',
@@ -69,15 +65,6 @@ export const LLMBackendConfig = {
OLLAMA: 'ollama',
}
export const RemoteOCREngineConfig = {
AZURE_AI: 'azureai',
}
export const RemoteOCRModeConfig = {
ALWAYS: 'always',
WORKFLOW_ONLY: 'workflow_only',
}
export interface ConfigOption {
key: string
title: string
@@ -85,7 +72,6 @@ export interface ConfigOption {
choices?: Array<{ id: string; name: string }>
config_key?: string
category: string
section?: string
note?: string
}
@@ -195,43 +181,6 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_OCR_USER_ARGS',
category: ConfigCategory.OCR,
},
{
key: 'remote_ocr_engine',
title: $localize`Remote OCR Engine`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCREngineConfig),
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
},
{
key: 'remote_ocr_api_key',
title: $localize`Remote OCR API Key`,
type: ConfigOptionType.Password,
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
},
{
key: 'remote_ocr_endpoint',
title: $localize`Remote OCR Endpoint`,
type: ConfigOptionType.String,
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Required when using the Azure AI engine.`,
},
{
key: 'remote_ocr_mode',
title: $localize`Remote OCR Mode`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCRModeConfig),
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
},
{
key: 'app_logo',
title: $localize`Application Logo`,
@@ -449,10 +398,6 @@ export interface PaperlessConfig extends ObjectWithId {
barcode_enable_tag: boolean
barcode_tag_mapping: object
barcode_tag_split: boolean
remote_ocr_engine: string
remote_ocr_api_key: string
remote_ocr_endpoint: string
remote_ocr_mode: string
ai_enabled: boolean
llm_embedding_backend: string
llm_embedding_model: string
-13
View File
@@ -1,6 +1,5 @@
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
import { RemoteOCRModeConfig } from './paperless-config'
import { User } from './user'
export interface UiSettings {
@@ -95,8 +94,6 @@ export const SETTINGS_KEYS = {
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
EMAIL_ENABLED: 'email_enabled',
AI_ENABLED: 'ai_enabled',
REMOTE_OCR_CONFIGURED: 'remote_ocr:configured',
REMOTE_OCR_MODE: 'remote_ocr:mode',
}
export const SETTINGS: UiSetting[] = [
@@ -350,14 +347,4 @@ export const SETTINGS: UiSetting[] = [
type: 'string',
default: PdfEditorEditMode.Create,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_CONFIGURED,
type: 'boolean',
default: false,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_MODE,
type: 'string',
default: RemoteOCRModeConfig.ALWAYS,
},
]
-18
View File
@@ -7,18 +7,6 @@ export enum WorkflowActionType {
Webhook = 4,
PasswordRemoval = 5,
MoveToTrash = 6,
RemoteOcr = 7,
ApplyAiSuggestions = 8,
}
// see src/documents/models.py AISuggestionField
export enum AISuggestionField {
Title = 'title',
Tags = 'tags',
Correspondent = 'correspondent',
DocumentType = 'document_type',
StoragePath = 'storage_path',
Created = 'created',
}
export interface WorkflowActionEmail extends ObjectWithId {
@@ -113,10 +101,4 @@ export interface WorkflowAction extends ObjectWithId {
webhook?: WorkflowActionWebhook
passwords?: string[]
ai_suggestion_fields?: AISuggestionField[]
ai_create_missing?: boolean
ai_overwrite_existing?: boolean
}
@@ -284,21 +284,6 @@ describe(`DocumentService`, () => {
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: false,
})
})
it('should request remote OCR when reprocessing with it enabled', () => {
const ids = [1, 2, 3]
subscription = service
.reprocessDocuments({ documents: ids }, true)
.subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/reprocess/`
)
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: true,
})
})
@@ -349,13 +349,9 @@ export class DocumentService extends AbstractPaperlessService<Document> {
})
}
reprocessDocuments(
selection: DocumentSelectionQuery,
remoteOcr: boolean = false
) {
reprocessDocuments(selection: DocumentSelectionQuery) {
return this.http.post(this.getResourceUrl(null, 'reprocess'), {
...selection,
remote_ocr: remoteOcr,
})
}
@@ -13,7 +13,6 @@ import { environment } from 'src/environments/environment'
import { CustomFieldDataType } from '../data/custom-field'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { SavedView } from '../data/saved-view'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
import { PermissionsService } from './permissions.service'
import { CustomFieldsService } from './rest/custom-fields.service'
@@ -435,26 +434,4 @@ describe('SettingsService', () => {
).name
).toEqual(customFields[0].name)
})
it('should offer remote OCR only when configured and selective', () => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, false)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
// configured, but already handling every document
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, true)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.ALWAYS
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeTruthy()
})
})
@@ -19,7 +19,6 @@ import {
} from 'src/app/utils/color'
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SavedView } from '../data/saved-view'
import {
PAPERLESS_GREEN_HEX,
@@ -688,17 +687,6 @@ export class SettingsService {
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
}
/**
* Offering remote OCR as a choice only makes sense when an engine
* is configured but is not already handling every document.
*/
get remoteOCRIsSelectable(): boolean {
return (
this.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
this.get(SETTINGS_KEYS.REMOTE_OCR_MODE) !== RemoteOCRModeConfig.ALWAYS
)
}
offerTour(): boolean {
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
}
-2
View File
@@ -118,7 +118,6 @@ import {
journalBookmarkFill,
journals,
link,
list,
listNested,
listTask,
listUl,
@@ -366,7 +365,6 @@ const icons = {
journalBookmarkFill,
journals,
link,
list,
listNested,
listTask,
listUl,
-1
View File
@@ -536,7 +536,6 @@ ul.pagination {
}
.nav-underline {
gap: calc(0.5 * var(--bs-nav-underline-gap)) var(--bs-nav-underline-gap);
.nav-link {
&.active, &:hover, &:focus {
color: var(--bs-primary);
+2 -8
View File
@@ -399,16 +399,10 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
return "OK"
def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
"""
Re-run parsing for the given documents.
Consumption workflows do not run here, so ``remote_ocr`` is how the user
asks for the remote engine when it is not configured to handle everything.
"""
def reprocess(doc_ids: list[int]) -> Literal["OK"]:
for document_id in doc_ids:
update_document_content_maybe_archive_file.apply_async(
kwargs={"document_id": document_id, "remote_ocr": remote_ocr},
kwargs={"document_id": document_id},
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
)
-18
View File
@@ -53,7 +53,6 @@ from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess
from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ArchiveFileGenerationChoices
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
@@ -452,19 +451,12 @@ class ConsumerPlugin(
except Exception as e:
self.log.error(f"Error attempting to clean PDF: {e}")
# Workflows have already run at this point, so the metadata knows
# whether this document was singled out for remote OCR
allow_remote = (
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
)
# Based on the mime type, get the parser for that type
parser_class: type[ParserProtocol] | None = (
get_parser_registry().get_parser_for_file(
mime_type,
self.filename,
self.working_copy,
allow_remote=allow_remote,
)
)
if not parser_class:
@@ -473,16 +465,6 @@ class ConsumerPlugin(
f"Unsupported mime type {mime_type}",
)
if self.metadata.remote_ocr and not getattr(
parser_class,
"uses_remote_service",
False,
):
self.log.warning(
"Remote OCR was requested for this document but no remote "
"parser is available for it, processing locally instead.",
)
# Notify all listeners that we're going to do some work.
document_consumption_started.send(
-3
View File
@@ -34,7 +34,6 @@ class DocumentMetadataOverrides:
skip_asn_if_exists: bool = False
version_label: str | None = None
actor_id: int | None = None
remote_ocr: bool = False
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
"""
@@ -58,8 +57,6 @@ class DocumentMetadataOverrides:
self.actor_id = other.actor_id
if other.skip_asn_if_exists:
self.skip_asn_if_exists = True
if other.remote_ocr:
self.remote_ocr = True
if other.version_label is not None:
self.version_label = other.version_label
+17 -7
View File
@@ -12,6 +12,7 @@ from typing import TYPE_CHECKING
from typing import Any
from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import FieldError
from django.db.models import Case
from django.db.models import CharField
from django.db.models import Count
@@ -50,7 +51,6 @@ from documents.models import ShareLinkBundle
from documents.models import StoragePath
from documents.models import Tag
from documents.permissions import permitted_object_ids
from documents.versioning import ensure_effective_content
if TYPE_CHECKING:
from collections.abc import Callable
@@ -180,9 +180,14 @@ class TitleContentFilter(Filter):
logger.warning(
"Deprecated document filter parameter 'title_content' used; use `text` instead.",
)
return ensure_effective_content(qs).filter(
Q(title__icontains=value) | Q(effective_content__icontains=value),
)
try:
return qs.filter(
Q(title__icontains=value) | Q(effective_content__icontains=value),
)
except FieldError:
return qs.filter(
Q(title__icontains=value) | Q(content__icontains=value),
)
else:
return qs
@@ -193,9 +198,14 @@ class EffectiveContentFilter(Filter):
value = value.strip() if isinstance(value, str) else value
if not value:
return qs
return ensure_effective_content(qs).filter(
**{f"effective_content__{self.lookup_expr}": value},
)
try:
return qs.filter(
**{f"effective_content__{self.lookup_expr}": value},
)
except FieldError:
return qs.filter(
**{f"content__{self.lookup_expr}": value},
)
@extend_schema_field(serializers.BooleanField)
@@ -1,30 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 17:27
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0023_savedview_icon"),
]
operations = [
migrations.AlterField(
model_name="workflowaction",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Assignment"),
(2, "Removal"),
(3, "Email"),
(4, "Webhook"),
(5, "Password removal"),
(6, "Move to trash"),
(7, "Remote OCR"),
],
default=1,
verbose_name="Workflow Action Type",
),
),
]
@@ -1,84 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 18:26
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0024_alter_workflowaction_type"),
]
operations = [
migrations.AddField(
model_name="workflowaction",
name="ai_create_missing",
field=models.BooleanField(
default=False,
help_text="Create suggested tags, correspondents, document types and storage paths that do not already exist instead of skipping them.",
verbose_name="create missing objects",
),
),
migrations.AddField(
model_name="workflowaction",
name="ai_overwrite_existing",
field=models.BooleanField(
default=False,
help_text="Apply suggestions even if the document already has a value for that field. Tags are always added to, never replaced.",
verbose_name="overwrite existing values",
),
),
migrations.AddField(
model_name="workflowaction",
name="ai_suggestion_fields",
field=models.JSONField(
blank=True,
help_text="Which of the AI-suggested fields to apply to the document.",
null=True,
verbose_name="AI suggestion fields",
),
),
migrations.AlterField(
model_name="workflowaction",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Assignment"),
(2, "Removal"),
(3, "Email"),
(4, "Webhook"),
(5, "Password removal"),
(6, "Move to trash"),
(7, "Remote OCR"),
(8, "Apply AI suggestions"),
],
default=1,
verbose_name="Workflow Action Type",
),
),
migrations.AlterField(
model_name="paperlesstask",
name="task_type",
field=models.CharField(
choices=[
("consume_file", "Consume File"),
("train_classifier", "Train Classifier"),
("sanity_check", "Sanity Check"),
("index_optimize", "Index Optimize"),
("mail_fetch", "Mail Fetch"),
("llm_index", "LLM Index"),
("empty_trash", "Empty Trash"),
("check_workflows", "Check Workflows"),
("bulk_update", "Bulk Update"),
("reprocess_document", "Reprocess Document"),
("build_share_link", "Build Share Link"),
("bulk_delete", "Bulk Delete"),
("apply_ai_suggestions", "Apply AI Suggestions"),
],
db_index=True,
help_text="The kind of work being performed",
max_length=50,
verbose_name="Task Type",
),
),
]
-44
View File
@@ -766,7 +766,6 @@ class PaperlessTask(ModelWithOwner):
REPROCESS_DOCUMENT = "reprocess_document", _("Reprocess Document")
BUILD_SHARE_LINK = "build_share_link", _("Build Share Link")
BULK_DELETE = "bulk_delete", _("Bulk Delete")
APPLY_AI_SUGGESTIONS = "apply_ai_suggestions", _("Apply AI Suggestions")
COMPLETE_STATUSES = (
Status.SUCCESS,
@@ -1671,22 +1670,6 @@ class WorkflowAction(models.Model):
6,
_("Move to trash"),
)
REMOTE_OCR = (
7,
_("Remote OCR"),
)
APPLY_AI_SUGGESTIONS = (
8,
_("Apply AI suggestions"),
)
class AISuggestionField(models.TextChoices):
TITLE = ("title", _("Title"))
TAGS = ("tags", _("Tags"))
CORRESPONDENT = ("correspondent", _("Correspondent"))
DOCUMENT_TYPE = ("document_type", _("Document type"))
STORAGE_PATH = ("storage_path", _("Storage path"))
CREATED = ("created", _("Created date"))
type = models.PositiveSmallIntegerField(
_("Workflow Action Type"),
@@ -1925,33 +1908,6 @@ class WorkflowAction(models.Model):
),
)
ai_suggestion_fields = models.JSONField(
_("AI suggestion fields"),
null=True,
blank=True,
help_text=_(
"Which of the AI-suggested fields to apply to the document.",
),
)
ai_create_missing = models.BooleanField(
_("create missing objects"),
default=False,
help_text=_(
"Create suggested tags, correspondents, document types and storage "
"paths that do not already exist instead of skipping them.",
),
)
ai_overwrite_existing = models.BooleanField(
_("overwrite existing values"),
default=False,
help_text=_(
"Apply suggestions even if the document already has a value for that "
"field. Tags are always added to, never replaced.",
),
)
class Meta:
verbose_name = _("workflow action")
verbose_name_plural = _("workflow actions")
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
from whoosh_compat import FieldKind
from whoosh_compat import FieldSpec
from whoosh_compat import SubpathSpec
# Internal-only schema fields with no query-syntax meaning of their own
# (sort shadow fields, bigram CJK fields, simple_title/simple_content,
# autocomplete_word, notes_text) are NOT represented here, they are
# declared in _schema.py's field_descriptors().
#
# analyzer/pattern_normalizer are deliberately left at FieldSpec's default
# (None): they're language-specific and only meaningful to whoosh-compat's
# parser, so _registry.py attaches them per-language via dataclasses.replace()
# rather than PUBLIC_FIELDS declaring them itself. _schema.py only reads
# name/kind/fast and never sees the analyzer at all.
PUBLIC_FIELDS: tuple[FieldSpec, ...] = (
FieldSpec("title", FieldKind.TEXT),
FieldSpec("content", FieldKind.TEXT),
FieldSpec("correspondent", FieldKind.TEXT),
FieldSpec("document_type", FieldKind.TEXT, aliases=("type",)),
FieldSpec("storage_path", FieldKind.TEXT, aliases=("path",)),
FieldSpec("original_filename", FieldKind.TEXT),
FieldSpec("tag", FieldKind.TEXT, comma_values=True),
FieldSpec("checksum", FieldKind.KEYWORD),
FieldSpec("asn", FieldKind.U64, fast=True),
FieldSpec("page_count", FieldKind.U64, fast=True),
FieldSpec("num_notes", FieldKind.U64, fast=True),
FieldSpec("created", FieldKind.DATE, date_only=True, fast=True),
FieldSpec("modified", FieldKind.DATETIME, fast=True),
FieldSpec("added", FieldKind.DATETIME, fast=True),
FieldSpec(
"notes",
FieldKind.JSON,
subpaths={"user": SubpathSpec(), "note": SubpathSpec(default=True)},
),
FieldSpec(
"custom_fields",
FieldKind.JSON,
subpaths={"name": SubpathSpec(), "value": SubpathSpec(default=True)},
),
)
+91
View File
@@ -0,0 +1,91 @@
from __future__ import annotations
import dataclasses
from typing import TYPE_CHECKING
from whoosh_compat import FieldKind
from whoosh_compat import FieldRegistry
from documents.search._fields import PUBLIC_FIELDS
from documents.search._tokenizer import ascii_fold
from documents.search._tokenizer import paperless_text_analyzer
from documents.search._tokenizer import stem_pattern_text
if TYPE_CHECKING:
from whoosh_compat import PatternNormalizer
_registry_cache: dict[str | None, FieldRegistry] = {}
def _identity_analyzer(text: str) -> list[str]:
"""Analyzer for KEYWORD fields indexed with the raw tokenizer (no splitting)."""
return [text]
def _fold_normalizer(text: str) -> str:
"""Wildcard/regex literal-run normalizer for fields indexed without stemming."""
return ascii_fold(text.lower())
def _make_pattern_normalizer(language: str | None) -> PatternNormalizer:
"""Build the wildcard/regex literal-run normalizer for a search language."""
def _pattern_normalizer(text: str) -> tuple[str, ...]:
"""Normalize a literal run into the forms a term may match.
TEXT index terms go through lowercase -> ascii_fold -> stem, so a
pattern that skips stemming can never match one: "invoice*" would look
for a term starting with "invoice" while the index holds "invoic". The
run is therefore offered stemmed as well. KEYWORD fields are indexed
raw and get _fold_normalizer instead, so their patterns stay literal.
Both forms are returned, as alternatives, because neither is a prefix
of the other in general: English stemming substitutes as well as
truncates ("copy" -> "copi"), so the stem alone loses the compounds
the typed run reaches ("copyright") while the typed run alone loses
the inflections the stem reaches ("copies"). whoosh-compat ORs the
alternatives per literal run and deduplicates them, so a run the
stemmer leaves alone costs exactly the one branch it did before.
Inside a bracket class the emitter calls this once per character and
uses the answer only if it is a single one-character form; two forms
there leave the character as typed. A stemmer does not change a lone
character, so the two forms deduplicate to one and the class body is
folded as before.
"""
folded = ascii_fold(text.lower())
stemmed = stem_pattern_text(folded, language)
return (folded, stemmed)
return _pattern_normalizer
def get_field_registry(language: str | None) -> FieldRegistry:
"""Build (or return the cached) FieldRegistry for the given search language.
Cached keyed by language, rebuilt on the same trigger register_tokenizers()
uses (settings.SEARCH_LANGUAGE change). A fresh call with a new language
builds and caches a new registry rather than mutating the old one.
"""
if language in _registry_cache:
return _registry_cache[language]
text_analyzer = paperless_text_analyzer(language).analyze
pattern_normalizer = _make_pattern_normalizer(language)
specs = [
dataclasses.replace(
field,
analyzer=_identity_analyzer
if field.kind is FieldKind.KEYWORD
else text_analyzer,
pattern_normalizer=_fold_normalizer
if field.kind is FieldKind.KEYWORD
else pattern_normalizer,
)
for field in PUBLIC_FIELDS
]
registry = FieldRegistry(specs)
_registry_cache[language] = registry
return registry
+222 -83
View File
@@ -1,14 +1,19 @@
from __future__ import annotations
import hashlib
import json
import logging
import shutil
from typing import TYPE_CHECKING
from typing import Final
from typing import NamedTuple
from typing import cast
import tantivy
from django.conf import settings
from whoosh_compat import FieldKind
from documents.search._fields import PUBLIC_FIELDS
if TYPE_CHECKING:
from pathlib import Path
@@ -16,7 +21,185 @@ if TYPE_CHECKING:
logger = logging.getLogger("paperless.search")
# v1 - Initial tantivy schema format
SCHEMA_VERSION: Final[int] = 1
# v2 - build_schema() derived from PUBLIC_FIELDS, changing the field declaration
# order, and the write-only correspondent/document_type/storage_path/tag id
# columns dropped. tantivy compares schemas by ordered field list, so an
# index built by v1 rejects every write against the v2 schema.
SCHEMA_VERSION: Final[int] = 2
class FieldDescriptor(NamedTuple):
"""One tantivy field, in declaration order.
The descriptor vocabulary is paperless', not tantivy-py's: it is both the
input to the SchemaBuilder and the input to schema_fingerprint(), so the
persisted fingerprint cannot move under a tantivy-py upgrade.
"""
name: str
kind: str
stored: bool
indexed: bool
fast: bool
tokenizer: str | None
# (schema kind, tokenizer) for the FieldKind -> FieldDescriptor mapping that
# doesn't need special-casing. JSON is handled separately below since it can
# emit a second, synthetic descriptor.
_KIND_TABLE: Final[dict[FieldKind, tuple[str, str | None]]] = {
FieldKind.TEXT: ("text", "paperless_text"),
FieldKind.KEYWORD: ("text", "raw"),
FieldKind.U64: ("u64", None),
FieldKind.DATE: ("date", None),
FieldKind.DATETIME: ("date", None),
}
# Kinds whose fast-field flag follows FieldSpec.fast rather than always False.
_FAST_FROM_FIELD: Final[frozenset[FieldKind]] = frozenset(
{FieldKind.U64, FieldKind.DATE, FieldKind.DATETIME},
)
def _public_field_descriptors() -> list[FieldDescriptor]:
"""Descriptors for the query-visible fields declared in PUBLIC_FIELDS."""
descriptors: list[FieldDescriptor] = []
for field in PUBLIC_FIELDS:
if field.kind is FieldKind.JSON:
descriptors.append(
FieldDescriptor(
field.name,
"json",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
)
if field.name == "notes":
# Plain-text companion for snippet generation: tantivy's
# SnippetGenerator does not support JSON fields. Schema-only,
# no query-syntax meaning, not in PUBLIC_FIELDS.
descriptors.append(
FieldDescriptor(
"notes_text",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
)
continue
schema_kind, tokenizer = _KIND_TABLE[field.kind]
descriptors.append(
FieldDescriptor(
field.name,
schema_kind,
stored=True,
indexed=True,
fast=field.fast if field.kind in _FAST_FROM_FIELD else False,
tokenizer=tokenizer,
),
)
return descriptors
def field_descriptors() -> list[FieldDescriptor]:
"""Every field of the document index, in the order tantivy declares them.
tantivy compares schemas by *ordered* field list, so the order here is
part of the on-disk contract: schema_fingerprint() hashes it and
needs_rebuild() acts on the result.
"""
return [
FieldDescriptor(
"id",
"u64",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
*_public_field_descriptors(),
# Shadow sort fields - fast, not stored
*(
FieldDescriptor(
name,
"text",
stored=False,
indexed=True,
fast=True,
tokenizer="simple_analyzer",
)
for name in ("title_sort", "correspondent_sort", "type_sort")
),
# CJK support - not stored, indexed only
*(
FieldDescriptor(
name,
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
)
for name in (
"bigram_content",
"bigram_title",
"bigram_correspondent",
"bigram_document_type",
"bigram_tag",
)
),
# Simple substring search support for title/content - not stored,
# indexed only
*(
FieldDescriptor(
name,
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="simple_search_analyzer",
)
for name in ("simple_title", "simple_content")
),
# Autocomplete prefix scan via terms_with_prefix, which walks the
# field's term dictionary - so the field must be indexed (term dict),
# not stored. The stored value is never read back, so storing it only
# wastes space.
FieldDescriptor(
"autocomplete_word",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="raw",
),
# Permission filter columns, read by build_permission_filter.
*(
FieldDescriptor(
name,
"u64",
stored=False,
indexed=True,
fast=True,
tokenizer=None,
)
for name in ("owner_id", "viewer_id", "viewer_group_id")
),
]
def schema_fingerprint() -> str:
"""Hash of the field descriptors, stamped into .index_settings.json.
Changes whenever a field is added, removed, retyped, re-optioned or
reordered, so an index built from a different schema shape is detected
even when SCHEMA_VERSION was not bumped.
"""
payload = json.dumps([list(descriptor) for descriptor in field_descriptors()])
return hashlib.blake2b(payload.encode()).hexdigest()
def build_schema() -> tantivy.Schema:
@@ -32,85 +215,37 @@ def build_schema() -> tantivy.Schema:
"""
sb = tantivy.SchemaBuilder()
sb.add_unsigned_field("id", stored=True, indexed=True, fast=True)
sb.add_text_field("checksum", stored=True, tokenizer_name="raw")
for field in (
"title",
"correspondent",
"document_type",
"storage_path",
"original_filename",
"content",
):
sb.add_text_field(field, stored=True, tokenizer_name="paperless_text")
# Shadow sort fields - fast, not stored/indexed
for field in ("title_sort", "correspondent_sort", "type_sort"):
sb.add_text_field(
field,
stored=False,
tokenizer_name="simple_analyzer",
fast=True,
)
# CJK support - not stored, indexed only
sb.add_text_field("bigram_content", stored=False, tokenizer_name="bigram_analyzer")
sb.add_text_field("bigram_title", stored=False, tokenizer_name="bigram_analyzer")
sb.add_text_field(
"bigram_correspondent",
stored=False,
tokenizer_name="bigram_analyzer",
)
sb.add_text_field(
"bigram_document_type",
stored=False,
tokenizer_name="bigram_analyzer",
)
sb.add_text_field("bigram_tag", stored=False, tokenizer_name="bigram_analyzer")
# Simple substring search support for title/content - not stored, indexed only
sb.add_text_field(
"simple_title",
stored=False,
tokenizer_name="simple_search_analyzer",
)
sb.add_text_field(
"simple_content",
stored=False,
tokenizer_name="simple_search_analyzer",
)
# Autocomplete prefix scan via terms_with_prefix, which walks the field's
# term dictionary - so the field must be indexed (term dict), not stored.
# The stored value is never read back, so storing it only wastes space.
sb.add_text_field("autocomplete_word", stored=False, tokenizer_name="raw")
sb.add_text_field("tag", stored=True, tokenizer_name="paperless_text")
# JSON fields — structured queries: notes.user:alice, custom_fields.name:invoice
sb.add_json_field("notes", stored=True, tokenizer_name="paperless_text")
# Plain-text companion for notes — tantivy's SnippetGenerator does not support
# JSON fields, so highlights require a text field with the same content.
sb.add_text_field("notes_text", stored=True, tokenizer_name="paperless_text")
sb.add_json_field("custom_fields", stored=True, tokenizer_name="paperless_text")
for field in (
"correspondent_id",
"document_type_id",
"storage_path_id",
"tag_id",
"owner_id",
"viewer_id",
"viewer_group_id",
):
sb.add_unsigned_field(field, stored=False, indexed=True, fast=True)
for field in ("created", "modified", "added"):
sb.add_date_field(field, stored=True, indexed=True, fast=True)
for field in ("asn", "page_count", "num_notes"):
sb.add_unsigned_field(field, stored=True, indexed=True, fast=True)
for descriptor in field_descriptors():
if descriptor.kind == "text":
sb.add_text_field(
descriptor.name,
stored=descriptor.stored,
fast=descriptor.fast,
tokenizer_name=cast("str", descriptor.tokenizer),
)
elif descriptor.kind == "json":
sb.add_json_field(
descriptor.name,
stored=descriptor.stored,
fast=descriptor.fast,
tokenizer_name=cast("str", descriptor.tokenizer),
)
elif descriptor.kind == "u64":
sb.add_unsigned_field(
descriptor.name,
stored=descriptor.stored,
indexed=descriptor.indexed,
fast=descriptor.fast,
)
elif descriptor.kind == "date":
sb.add_date_field(
descriptor.name,
stored=descriptor.stored,
indexed=descriptor.indexed,
fast=descriptor.fast,
)
else:
raise ValueError(f"Unknown schema field kind: {descriptor.kind}")
return sb.build()
@@ -119,9 +254,9 @@ def needs_rebuild(index_dir: Path) -> bool:
"""
Check if the search index needs rebuilding.
Reads .index_settings.json to compare the stored schema version and
search language against the current configuration. Returns True if the
file is missing, unparsable, or either value mismatches.
Reads .index_settings.json to compare the stored schema version, search
language and schema fingerprint against the current configuration. Returns
True if the file is missing, unparsable, or any value mismatches.
Args:
index_dir: Path to the search index directory
@@ -140,6 +275,9 @@ def needs_rebuild(index_dir: Path) -> bool:
if "language" not in data or data["language"] != settings.SEARCH_LANGUAGE:
logger.info("Search index language changed - rebuilding.")
return True
if data.get("schema_fingerprint") != schema_fingerprint():
logger.info("Search index schema fingerprint mismatch - rebuilding.")
return True
except ValueError:
return True
return False
@@ -170,6 +308,7 @@ def _write_sentinels(index_dir: Path) -> None:
{
"schema_version": SCHEMA_VERSION,
"language": settings.SEARCH_LANGUAGE,
"schema_fingerprint": schema_fingerprint(),
},
),
)
+51 -2
View File
@@ -1,6 +1,7 @@
from __future__ import annotations
import logging
from functools import cache
from typing import Final
import tantivy
@@ -71,7 +72,7 @@ def register_tokenizers(index: tantivy.Index, language: str | None) -> None:
use fast=True and Tantivy requires fast-field tokenizers to exist
even for documents that omit those fields.
"""
index.register_tokenizer("paperless_text", _paperless_text(language))
index.register_tokenizer("paperless_text", paperless_text_analyzer(language))
index.register_tokenizer("simple_analyzer", _simple_analyzer())
index.register_tokenizer("bigram_analyzer", _bigram_analyzer())
index.register_tokenizer("simple_search_analyzer", _simple_search_analyzer())
@@ -79,7 +80,7 @@ def register_tokenizers(index: tantivy.Index, language: str | None) -> None:
index.register_fast_field_tokenizer("simple_analyzer", _simple_analyzer())
def _paperless_text(language: str | None) -> tantivy.TextAnalyzer:
def paperless_text_analyzer(language: str | None) -> tantivy.TextAnalyzer:
"""Main full-text tokenizer for content, title, etc: simple -> remove_long(129) -> lowercase -> ascii_fold [-> stemmer]"""
builder = (
tantivy.TextAnalyzerBuilder(tantivy.Tokenizer.simple())
@@ -100,6 +101,54 @@ def _paperless_text(language: str | None) -> tantivy.TextAnalyzer:
return builder.build()
@cache
def _pattern_stemmer(language: str | None) -> tantivy.TextAnalyzer | None:
"""The stemming tail of paperless_text_analyzer, over a whole literal run.
Same language gate and same Snowball stemmer paperless_text_analyzer
applies at index time, so query patterns follow SEARCH_LANGUAGE. Returns
None when that gate disables stemming; paperless_text_analyzer already
warns about an unsupported language, so this stays quiet.
The raw tokenizer keeps the run whole (a wildcard literal is a fragment,
not necessarily a word), and remove_long is kept so an over-long run is
treated the same way the index treats it.
"""
if not language:
return None
tantivy_lang = _LANGUAGE_MAP.get(language.lower())
if tantivy_lang is None:
return None
return (
tantivy.TextAnalyzerBuilder(tantivy.Tokenizer.raw())
.filter(tantivy.Filter.remove_long(_TOKEN_REMOVE_LONG_LIMIT))
.filter(tantivy.Filter.stemmer(tantivy_lang))
.build()
)
def stem_pattern_text(text: str, language: str | None) -> str:
"""Stem an already lowercased/ascii-folded run the way index terms are.
Returns text unchanged when stemming is disabled for language, and also
when the stem step does not yield exactly one token: remove_long drops a run
past the length limit, leaving no stem to substitute. Falling back to the
text as typed is the safe direction for a pattern prefix, since it can only
be as narrow as it was before stemming was considered.
The raw tokenizer emits one token whatever the input and the stemmer is
1-to-1, so only the zero-token case can fire today; the guard covers both
counts so a tokenizer change cannot turn this into an IndexError.
"""
analyzer = _pattern_stemmer(language)
if analyzer is None:
return text
tokens = analyzer.analyze(text)
if len(tokens) != 1:
return text
return tokens[0]
def _simple_analyzer() -> tantivy.TextAnalyzer:
"""Tokenizer for shadow sort fields (title_sort, correspondent_sort, type_sort): simple -> lowercase -> ascii_fold."""
return (
+1 -92
View File
@@ -1795,7 +1795,7 @@ class DeleteDocumentsSerializer(DocumentSelectionSerializer):
class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
remote_ocr = serializers.BooleanField(required=False, default=False)
pass
class BulkEditSerializer(
@@ -2137,13 +2137,6 @@ class BulkEditSerializer(
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
)
def _validate_parameters_reprocess(self, parameters) -> None:
if "remote_ocr" in parameters:
if not isinstance(parameters["remote_ocr"], bool):
raise serializers.ValidationError("remote_ocr must be a boolean")
else:
parameters["remote_ocr"] = False
def validate_parameters_remove_password(self, parameters):
if "password" not in parameters:
raise serializers.ValidationError("password not specified")
@@ -2208,8 +2201,6 @@ class BulkEditSerializer(
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
elif method == bulk_edit.remove_password:
self.validate_parameters_remove_password(parameters)
elif method == bulk_edit.reprocess:
self._validate_parameters_reprocess(parameters)
return attrs
@@ -3235,9 +3226,6 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
"email",
"webhook",
"passwords",
"ai_suggestion_fields",
"ai_create_missing",
"ai_overwrite_existing",
]
def validate(self, attrs):
@@ -3295,23 +3283,6 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
"Passwords are required for password removal actions",
)
if (
"type" in attrs
and attrs["type"] == WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS
):
fields = attrs.get("ai_suggestion_fields")
valid_fields = set(WorkflowAction.AISuggestionField.values)
if (
fields is None
or not isinstance(fields, list)
or len(fields) == 0
or any(field not in valid_fields for field in fields)
):
raise serializers.ValidationError(
"At least one valid field is required for apply AI "
f"suggestions actions, options are: {sorted(valid_fields)}",
)
return attrs
@@ -3332,68 +3303,6 @@ class WorkflowSerializer(serializers.ModelSerializer[Workflow]):
"actions",
]
def validate(self, attrs):
attrs = super().validate(attrs)
if "actions" in attrs:
has_remote_ocr_action = any(
action.get("type") == WorkflowAction.WorkflowActionType.REMOTE_OCR
for action in attrs["actions"]
)
has_ai_suggestions_action = any(
action.get("type")
== WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS
for action in attrs["actions"]
)
else:
has_remote_ocr_action = self.instance is not None and (
self.instance.actions.filter(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
).exists()
)
has_ai_suggestions_action = self.instance is not None and (
self.instance.actions.filter(
type=WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS,
).exists()
)
if "triggers" in attrs:
has_consumption_trigger = any(
trigger.get("type") == WorkflowTrigger.WorkflowTriggerType.CONSUMPTION
for trigger in attrs["triggers"]
)
has_non_consumption_trigger = any(
trigger.get("type") != WorkflowTrigger.WorkflowTriggerType.CONSUMPTION
for trigger in attrs["triggers"]
)
else:
has_consumption_trigger = self.instance is not None and (
self.instance.triggers.filter(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
).exists()
)
has_non_consumption_trigger = self.instance is not None and (
self.instance.triggers.exclude(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
).exists()
)
# Remote OCR can only work with consumption triggers
if has_remote_ocr_action and not has_consumption_trigger:
raise serializers.ValidationError(
"Remote OCR actions require a consumption started trigger",
)
# Suggestions are made from the document content, which does not exist
# until after consumption has finished
if has_ai_suggestions_action and not has_non_consumption_trigger:
raise serializers.ValidationError(
"Apply AI suggestions actions require a trigger other than "
"consumption started",
)
return attrs
def update_triggers_and_actions(
self,
instance: Workflow,
-40
View File
@@ -973,39 +973,6 @@ def run_workflows(
)
elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH:
has_move_to_trash_action = True
elif action.type == WorkflowAction.WorkflowActionType.REMOTE_OCR:
if use_overrides and overrides:
overrides.remote_ocr = True
else:
# If a workflow has a consumption trigger *and* another type,
# the document has already been parsed by the time the other one fires
logger.debug(
"Remote OCR action only applies to consumption "
"triggers, ignoring",
extra={"group": logging_group},
)
elif (
action.type
== WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS
):
if use_overrides:
# The document has not been parsed yet, so there is no
# content for the LLM to make suggestions from
logger.debug(
"Apply AI suggestions action does not apply to "
"consumption triggers, ignoring",
extra={"group": logging_group},
)
else:
# Queued rather than run sync
from documents.tasks import apply_ai_suggestions
# kwargs so the PaperlessTask record can note the
# document, see _extract_input_data
apply_ai_suggestions.delay(
action_id=action.pk,
document_id=document.pk,
)
if not use_overrides:
# limit title to 128 characters
@@ -1061,7 +1028,6 @@ TRACKED_TASKS: dict[str, PaperlessTask.TaskType] = {
"documents.tasks.update_document_content_maybe_archive_file": PaperlessTask.TaskType.REPROCESS_DOCUMENT,
"documents.tasks.build_share_link_bundle": PaperlessTask.TaskType.BUILD_SHARE_LINK,
"documents.bulk_edit.delete": PaperlessTask.TaskType.BULK_DELETE,
"documents.tasks.apply_ai_suggestions": PaperlessTask.TaskType.APPLY_AI_SUGGESTIONS,
}
_CELERY_STATE_TO_STATUS: dict[str, PaperlessTask.Status] = {
@@ -1115,12 +1081,6 @@ def _extract_input_data(
return {"account_ids": account_ids}
return {}
if task_type == PaperlessTask.TaskType.APPLY_AI_SUGGESTIONS:
document_id = task_kwargs.get("document_id")
if document_id is not None:
return {"document_id": document_id}
return {}
return {}
+1 -50
View File
@@ -67,11 +67,9 @@ from documents.utils import identity
from documents.versioning import annotate_effective_content
from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig
from paperless.config import RemoteOCRConfig
from paperless.logging import consume_task_id
from paperless.parsers import ParserContext
from paperless.parsers.registry import get_parser_registry
from paperless_ai.exceptions import LLMTimeoutError
from paperless_ai.indexing import llm_index_add_or_update_document
from paperless_ai.indexing import llm_index_remove_document
from paperless_ai.indexing import update_llm_index
@@ -340,17 +338,10 @@ def bulk_update_documents(document_ids) -> None:
@shared_task
def update_document_content_maybe_archive_file(
document_id,
*,
remote_ocr: bool = False,
) -> None:
def update_document_content_maybe_archive_file(document_id) -> None:
"""
Re-creates OCR content and thumbnail for a document, and archive file if
it exists.
Remote OCR is used only when the engine is configured to handle everything
or if explicitly asked for via ``remote_ocr``.
"""
document = Document.objects.get(id=document_id)
@@ -360,7 +351,6 @@ def update_document_content_maybe_archive_file(
mime_type,
document.original_filename or "",
document.source_path,
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
)
if not parser_class:
@@ -715,45 +705,6 @@ def llmindex_index(
)
@shared_task(
bind=True,
autoretry_for=(LLMTimeoutError,),
max_retries=3,
retry_backoff=60,
retry_backoff_max=600,
retry_jitter=True,
)
def apply_ai_suggestions(self, action_id: int, document_id: int) -> None:
"""
Deferred "apply AI suggestions" workflow action.
"""
from documents.models import WorkflowAction
from documents.workflows.ai import apply_ai_suggestions_to_document
try:
action = WorkflowAction.objects.get(pk=action_id)
document = Document.objects.select_related("owner").get(pk=document_id)
except (WorkflowAction.DoesNotExist, Document.DoesNotExist):
logger.warning(
"Workflow action %s or document %s no longer exists, "
"not applying AI suggestions",
action_id,
document_id,
)
return
if not apply_ai_suggestions_to_document(action, document):
return
# No document_updated signal to avoid loop
clear_document_caches(document.pk)
index_document.delay(document.pk)
ai_config = AIConfig()
if ai_config.llm_index_enabled:
update_document_in_llm_index.apply_async(kwargs={"document": document})
@shared_task
def update_document_in_llm_index(document) -> None:
llm_index_add_or_update_document(document)
+10
View File
@@ -0,0 +1,10 @@
from whoosh_compat import FieldKind
from documents.search._fields import PUBLIC_FIELDS
class TestPublicFields:
def test_json_fields_have_subpaths(self) -> None:
for field in PUBLIC_FIELDS:
if field.kind is FieldKind.JSON:
assert field.subpaths, f"{field.name} is JSON but has no subpaths"
@@ -0,0 +1,83 @@
"""Every declared JSON subpath must actually be written to the index.
PUBLIC_FIELDS declares each JSON field's subpaths (e.g. ``notes`` ->
{"user", "note"}), but nothing coupled that declaration to what
``_backend.py``'s document builder actually writes into the JSON blob at
index time. A subpath declared but never written would be
queryable-but-always-empty -- syntactically valid, silently matching
nothing -- with no test failure anywhere.
This indexes one real document carrying values for every JSON field
(a Note, a CustomFieldInstance) and inspects the document's own stored
JSON payload, rather than running field-specific queries: that way a
future JSON field's subpaths are covered automatically, without a new
per-subpath query having to be added by hand each time.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
import tantivy
from django.contrib.auth.models import User
from whoosh_compat import FieldKind
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import Note
from documents.search._fields import PUBLIC_FIELDS
if TYPE_CHECKING:
from documents.search._backend import TantivyBackend
pytestmark = [pytest.mark.search, pytest.mark.django_db]
class TestJsonSubpathsAreWrittenAtIndexTime:
def test_every_declared_json_subpath_appears_in_the_stored_document(
self,
backend: TantivyBackend,
) -> None:
user = User.objects.create_user(username="completeness-user")
field = CustomField.objects.create(
name="Completeness Field",
data_type=CustomField.FieldDataType.STRING,
)
doc = Document.objects.create(
title="Completeness doc",
content="x",
checksum="json-subpath-completeness",
)
Note.objects.create(document=doc, user=user, note="a note")
CustomFieldInstance.objects.create(
document=doc,
field=field,
value_text="a value",
)
backend.add_or_update(doc)
index = backend._index
searcher = index.searcher()
hits = searcher.search(
tantivy.Query.term_query(index.schema, "id", doc.pk),
limit=1,
).hits
assert hits, "the document was not indexed"
stored = searcher.doc(hits[0][1]).to_dict()
json_fields = [f for f in PUBLIC_FIELDS if f.kind is FieldKind.JSON]
assert json_fields, "no JSON fields declared - fixture is stale"
for field_spec in json_fields:
stored_values = stored.get(field_spec.name)
assert stored_values, (
f"{field_spec.name} was not written to the index at all"
)
written_keys = stored_values[0].keys()
for subpath in field_spec.subpaths:
assert subpath in written_keys, (
f"{field_spec.name}.{subpath} is declared in PUBLIC_FIELDS "
"but _backend.py's document builder never writes it - it "
"would be queryable but always empty"
)
@@ -0,0 +1,60 @@
"""Wildcard patterns on KEYWORD fields must stay literal.
``checksum`` is the only KEYWORD field: it is indexed with the raw tokenizer,
so its terms are never lowercased, folded or stemmed. Running its wildcard
patterns through the stemming normalizer rewrote hex prefixes ("ceded" ->
"cede") and returned documents whose checksum did not start with what the user
typed, which for an identity field is a wrong answer.
This covers only the registry-level normalizer, which is all that exists to
prove at this point in the stack: user queries are not yet routed through
whoosh-compat (that lands with the query-layer PR), so the same fact proven
end to end against real indexed documents lives in
``test_checksum_prefix_queries.py``.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from documents.search._registry import get_field_registry
if TYPE_CHECKING:
from whoosh_compat import FieldRegistry
from whoosh_compat import PatternNormalizer
pytestmark = [pytest.mark.search, pytest.mark.django_db]
def _normalizer(registry: FieldRegistry, name: str) -> PatternNormalizer:
ref = registry.make_ref(name)
assert ref is not None
resolved = registry.resolve(ref)
assert resolved is not None
assert resolved.spec.pattern_normalizer is not None
return resolved.spec.pattern_normalizer
class TestKeywordPatternNormalizer:
@pytest.mark.parametrize(
"run",
[
pytest.param("ceded", id="stems_to_cede"),
pytest.param("added", id="stems_to_ad"),
pytest.param("cafed", id="stems_to_cafe"),
],
)
def test_keyword_runs_are_folded_not_stemmed(self, run: str) -> None:
"""One form, the run as typed: a KEYWORD pattern must never be widened
to a stem, which would return checksums that do not start with what
the user typed."""
normalize = _normalizer(get_field_registry("en"), "checksum")
assert normalize(run) == run
def test_text_runs_still_offer_their_stem(self) -> None:
"""A TEXT field offers the stem alongside the typed run, so a term
matching either one is reachable."""
normalize = _normalizer(get_field_registry("en"), "title")
assert tuple(normalize("Running")) == ("running", "run")
+149
View File
@@ -0,0 +1,149 @@
from collections.abc import Sequence
import pytest
from whoosh_compat import FieldKind
from whoosh_compat import FieldRegistry
from whoosh_compat.fields import ResolvedField
from documents.search._fields import PUBLIC_FIELDS
from documents.search._registry import get_field_registry
@pytest.fixture
def registry() -> FieldRegistry:
return get_field_registry(None)
def _resolve(registry: FieldRegistry, name: str) -> ResolvedField:
ref = registry.make_ref(name)
assert ref is not None, f"{name} is not a valid field ref"
resolved = registry.resolve(ref)
assert resolved is not None, f"{name} did not resolve"
return resolved
def _distinct_forms(result: str | Sequence[str]) -> tuple[str, ...]:
"""The forms a term may match, in order, the way whoosh-compat's emitter
reads a pattern_normalizer's answer: a bare str is one form, a sequence is
several, deduplicated."""
if isinstance(result, str):
return (result,)
return tuple(dict.fromkeys(result))
class TestFieldRegistry:
def test_internal_id_fields_are_not_registered(
self,
registry: FieldRegistry,
) -> None:
for name in (
"tag_id",
"owner_id",
"viewer_id",
"correspondent_id",
"document_type_id",
"storage_path_id",
"viewer_group_id",
):
assert name not in registry
def test_no_queryable_field_name_ends_in_id(self) -> None:
# The list above names the seven that were dropped; this catches the
# eighth. Internal *_id columns are written for permission filtering
# and joins, and whoosh only exposed them as query fields by accident,
# so a new one reaching the query surface is a leak rather than a
# feature. Checked against PUBLIC_FIELDS rather than the registry so
# an internal field is caught where it is declared.
leaked = [f.name for f in PUBLIC_FIELDS if f.name.endswith("_id")]
assert not leaked, f"internal id fields reached the query surface: {leaked}"
def test_type_alias_resolves_to_document_type(
self,
registry: FieldRegistry,
) -> None:
assert _resolve(registry, "type").spec.name == "document_type"
def test_path_alias_resolves_to_storage_path(self, registry: FieldRegistry) -> None:
assert _resolve(registry, "path").spec.name == "storage_path"
def test_notes_json_subpaths_resolve(self, registry: FieldRegistry) -> None:
resolved = _resolve(registry, "notes.user")
assert resolved.spec.name == "notes"
assert resolved.json_path == "user"
assert resolved.is_subpath is True
def test_custom_fields_json_subpaths_resolve(self, registry: FieldRegistry) -> None:
for raw in ("custom_fields.name", "custom_fields.value"):
_resolve(registry, raw)
def test_unregistered_json_subpath_does_not_resolve(
self,
registry: FieldRegistry,
) -> None:
# An unregistered subpath is not even a valid FieldRef: make_ref
# returns None for a dotted name whose subpath isn't registered
# (it doesn't produce a ref for resolve() to then reject).
assert registry.make_ref("notes.bogus") is None
def test_tag_is_comma_values(self, registry: FieldRegistry) -> None:
assert _resolve(registry, "tag").spec.comma_values is True
def test_correspondent_is_not_comma_values(self, registry: FieldRegistry) -> None:
# "tag" is the only field that opts in. This is only observable here:
# end to end the two readings of "correspondent:foo,bar" agree,
# because the analyzer splits the literal value on the comma anyway,
# so a result-level test cannot tell a value list from literal text.
assert _resolve(registry, "correspondent").spec.comma_values is False
def test_created_is_date_kind(self, registry: FieldRegistry) -> None:
resolved = _resolve(registry, "created")
assert resolved.spec.kind is FieldKind.DATE
assert resolved.spec.date_only is True
def test_analyzer_lowercases_and_ascii_folds(self, registry: FieldRegistry) -> None:
# title uses the paperless_text analyzer: simple -> remove_long ->
# lowercase -> ascii_fold [-> stemmer]. With no language configured
# (None), no stemmer runs, so "Café" folds to the single token "cafe".
resolved = _resolve(registry, "title")
assert resolved.spec.analyzer is not None
assert resolved.spec.analyzer("Café") == ["cafe"]
def test_checksum_analyzer_is_identity_single_token(
self,
registry: FieldRegistry,
) -> None:
# checksum uses the raw tokenizer at index time (no splitting).
resolved = _resolve(registry, "checksum")
assert resolved.spec.analyzer is not None
assert resolved.spec.analyzer("ABC-123") == ["ABC-123"]
def test_pattern_normalizer_follows_the_registry_language(
self,
registry: FieldRegistry,
) -> None:
# Index terms are stemmed, so patterns offer their stem too, using the
# registry's own language: "Running" has to reach the indexed "run".
# Without a language the index holds surface forms, so there is no
# second form and the run is only case/accent-folded.
resolved = _resolve(registry, "title")
assert resolved.spec.pattern_normalizer is not None
assert _distinct_forms(resolved.spec.pattern_normalizer("Running")) == (
"running",
)
resolved_en = _resolve(get_field_registry("en"), "title")
assert resolved_en.spec.pattern_normalizer is not None
assert _distinct_forms(resolved_en.spec.pattern_normalizer("Running")) == (
"running",
"run",
)
def test_registry_is_cached_per_language(self) -> None:
a = get_field_registry("en")
b = get_field_registry("en")
assert a is b
def test_registry_rebuilds_on_language_change(self) -> None:
a = get_field_registry("en")
b = get_field_registry("de")
assert a is not b
+84 -1
View File
@@ -1,12 +1,20 @@
from __future__ import annotations
import json
from datetime import UTC
from datetime import datetime
from typing import TYPE_CHECKING
import pytest
import tantivy
from documents.search._fields import PUBLIC_FIELDS
from documents.search._schema import SCHEMA_VERSION
from documents.search._schema import build_schema
from documents.search._schema import field_descriptors
from documents.search._schema import needs_rebuild
from documents.search._schema import schema_fingerprint
from documents.search._tokenizer import register_tokenizers
if TYPE_CHECKING:
from pathlib import Path
@@ -29,7 +37,13 @@ class TestNeedsRebuild:
) -> None:
settings.SEARCH_LANGUAGE = "en"
(index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": SCHEMA_VERSION, "language": "en"}),
json.dumps(
{
"schema_version": SCHEMA_VERSION,
"language": "en",
"schema_fingerprint": schema_fingerprint(),
},
),
)
assert needs_rebuild(index_dir) is False
@@ -76,3 +90,72 @@ class TestNeedsRebuild:
json.dumps({"schema_version": SCHEMA_VERSION, "language": "en"}),
)
assert needs_rebuild(index_dir) is True
def _schema_fields(schema: tantivy.Schema) -> dict[str, dict]:
"""{name: field-state} for every field declared on a tantivy Schema.
tantivy-py 0.26 exposes no public introspection API on Schema (no
__iter__, get_field, to_json, etc.) -- __reduce__() (used internally for
pickling) is the only way to recover the field list, so we lean on it
here for test assertions only.
"""
state = schema.__reduce__()[1][0]
return {field["name"]: field for field in state["inner"]}
class TestSchemaMatchesPublicFields:
def test_every_public_field_is_in_the_schema(self) -> None:
schema = build_schema()
schema_field_names = set(_schema_fields(schema))
for field in PUBLIC_FIELDS:
assert field.name in schema_field_names, (
f"{field.name} is in PUBLIC_FIELDS but missing from build_schema()"
)
def test_asn_page_count_num_notes_are_fast_unsigned_fields(self) -> None:
# Spot-check kind-derived construction for the U64 fields.
schema = build_schema()
doc = tantivy.Document()
doc.add_unsigned("id", 1)
doc.add_text("checksum", "x")
doc.add_unsigned("asn", 42)
doc.add_unsigned("page_count", 3)
doc.add_unsigned("num_notes", 0)
doc.add_date("created", datetime(2020, 1, 1, tzinfo=UTC))
doc.add_date("modified", datetime(2020, 1, 1, tzinfo=UTC))
doc.add_date("added", datetime(2020, 1, 1, tzinfo=UTC))
index = tantivy.Index(schema)
register_tokenizers(index, None)
writer = index.writer()
writer.add_document(doc)
writer.commit()
index.reload()
searcher = index.searcher()
results = searcher.search(tantivy.Query.term_query(schema, "asn", 42), limit=1)
assert len(results.hits) == 1
class TestFastFlagAgreement:
def test_every_public_field_fast_flag_matches_the_built_schema(self) -> None:
# whoosh-compat's registry trusts PUBLIC_FIELDS' fast flag when resolving
# field:* existence checks (its FAST_FIELD strategy); a fast=True
# entry whose actual tantivy column is not fast would make those
# searches silently match nothing at search time. Only the U64 and
# DATE descriptors can carry the flag today, so this
# pins the agreement for EVERY kind: a future fast=True
# TEXT/KEYWORD/JSON entry the builder silently ignores fails here
# instead of at a user's query.
#
# field_descriptors() (not tantivy-py's __reduce__() pickling
# internals) is used as the probe here: it is exactly the input
# build_schema()'s SchemaBuilder consumes for the `fast` kwarg on
# every field kind, so it pins the same agreement without depending
# on a private pickled representation surviving a tantivy-py
# upgrade.
descriptor_fast = {d.name: d.fast for d in field_descriptors()}
for public_field in PUBLIC_FIELDS:
assert descriptor_fast[public_field.name] == public_field.fast, (
f"{public_field.name}: PUBLIC_FIELDS says fast={public_field.fast} but"
f" field_descriptors() says fast={descriptor_fast[public_field.name]}"
)
@@ -0,0 +1,492 @@
"""The schema fingerprint stamped into .index_settings.json.
tantivy compares schemas by *ordered* field list, and `tantivy.Index(schema,
path=...)` (what every write path does) raises on any difference. SCHEMA_VERSION
is the manual guard against that, but build_schema() is edited for *parser*
reasons - adding an alias, flipping fast=True, adding a subpath - by people not
thinking about the on-disk index, and forgetting the bump is exactly how this
branch's bug happened.
The fingerprint is the automatic guard: it hashes the field descriptor list that
build_schema() itself iterates, so any change to a field's name, kind, options
or *position* forces a rebuild on its own.
"""
from __future__ import annotations
import hashlib
import json
from typing import TYPE_CHECKING
import pytest
import tantivy
from documents.search import _schema
from documents.search._schema import SCHEMA_VERSION
from documents.search._schema import FieldDescriptor
from documents.search._schema import _write_sentinels
from documents.search._schema import build_schema
from documents.search._schema import field_descriptors
from documents.search._schema import needs_rebuild
from documents.search._schema import schema_fingerprint
if TYPE_CHECKING:
from pathlib import Path
from pytest_django.fixtures import SettingsWrapper
pytestmark = pytest.mark.search
# The on-disk field layout of a v2 index, pinned as data. Any edit here is an
# index-format change: it must come with a rebuild, which the fingerprint now
# forces automatically. Reproduced from build_schema()'s output as it stood
# before the descriptor refactor, so it also pins that the refactor changed
# nothing.
PINNED_DESCRIPTORS: tuple[FieldDescriptor, ...] = (
FieldDescriptor("id", "u64", stored=True, indexed=True, fast=True, tokenizer=None),
FieldDescriptor(
"title",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"content",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"correspondent",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"document_type",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"storage_path",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"original_filename",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"tag",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"checksum",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="raw",
),
FieldDescriptor("asn", "u64", stored=True, indexed=True, fast=True, tokenizer=None),
FieldDescriptor(
"page_count",
"u64",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"num_notes",
"u64",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"created",
"date",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"modified",
"date",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"added",
"date",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"notes",
"json",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"notes_text",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"custom_fields",
"json",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"title_sort",
"text",
stored=False,
indexed=True,
fast=True,
tokenizer="simple_analyzer",
),
FieldDescriptor(
"correspondent_sort",
"text",
stored=False,
indexed=True,
fast=True,
tokenizer="simple_analyzer",
),
FieldDescriptor(
"type_sort",
"text",
stored=False,
indexed=True,
fast=True,
tokenizer="simple_analyzer",
),
FieldDescriptor(
"bigram_content",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
),
FieldDescriptor(
"bigram_title",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
),
FieldDescriptor(
"bigram_correspondent",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
),
FieldDescriptor(
"bigram_document_type",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
),
FieldDescriptor(
"bigram_tag",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
),
FieldDescriptor(
"simple_title",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="simple_search_analyzer",
),
FieldDescriptor(
"simple_content",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="simple_search_analyzer",
),
FieldDescriptor(
"autocomplete_word",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="raw",
),
FieldDescriptor(
"owner_id",
"u64",
stored=False,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"viewer_id",
"u64",
stored=False,
indexed=True,
fast=True,
tokenizer=None,
),
FieldDescriptor(
"viewer_group_id",
"u64",
stored=False,
indexed=True,
fast=True,
tokenizer=None,
),
)
def _schema_fields(schema: tantivy.Schema) -> list[dict]:
"""The tantivy-level field list, in declaration order.
tantivy-py 0.26 exposes no public introspection API on Schema, so
__reduce__() (its pickling hook) is the only way to recover the field list.
It is used here, in a test, precisely because it is the representation the
persisted fingerprint must NOT depend on.
"""
return schema.__reduce__()[1][0]["inner"]
def _sentinels(index_dir: Path, **overrides: object) -> None:
data = {
"schema_version": SCHEMA_VERSION,
"language": None,
"schema_fingerprint": schema_fingerprint(),
}
data.update(overrides)
(index_dir / ".index_settings.json").write_text(json.dumps(data))
class TestDescriptorsDescribeTheBuiltSchema:
def test_descriptors_match_the_pinned_field_layout(self) -> None:
assert tuple(field_descriptors()) == PINNED_DESCRIPTORS
def test_built_schema_matches_the_descriptors(self) -> None:
"""The descriptors are not a parallel description - they are the input.
Reading the built schema back proves the loop honours every option, so
a descriptor edit cannot claim a shape the SchemaBuilder did not build.
"""
kinds = {"text": "text", "json": "json_object", "u64": "u64", "date": "date"}
built = [
(
field["name"],
field["type"],
field["options"]["stored"],
bool(field["options"].get("fast")),
(field["options"].get("indexing") or {}).get("tokenizer"),
)
for field in _schema_fields(build_schema())
]
expected = [
(
descriptor.name,
kinds[descriptor.kind],
descriptor.stored,
descriptor.fast,
descriptor.tokenizer,
)
for descriptor in field_descriptors()
]
assert built == expected
class TestFingerprintSensitivity:
def test_a_field_option_change_moves_the_fingerprint(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
before = schema_fingerprint()
changed = field_descriptors()
changed[1] = changed[1]._replace(fast=True)
monkeypatch.setattr(_schema, "field_descriptors", lambda: changed)
assert schema_fingerprint() != before
def test_reordering_alone_moves_the_fingerprint(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The original bug: same fields, different declaration order.
A set- or dict-based fingerprint would be blind to this, and tantivy
would reject every write against the existing index.
"""
before = schema_fingerprint()
swapped = field_descriptors()
swapped[1], swapped[2] = swapped[2], swapped[1]
monkeypatch.setattr(_schema, "field_descriptors", lambda: swapped)
assert schema_fingerprint() != before
def test_repeated_calls_agree(self) -> None:
assert schema_fingerprint() == schema_fingerprint()
class TestFingerprintIsIndependentOfTantivy:
def test_a_tantivy_option_key_addition_would_not_move_it(self) -> None:
"""A tantivy-py upgrade must not force a global reindex.
Hashing schema.__reduce__() would do exactly that: the simulated new
option key below changes that payload for every user with no schema
change at all.
"""
fields = _schema_fields(build_schema())
upgraded = [
{**field, "options": {**field["options"], "coerce": True}}
for field in fields
]
assert _hash(upgraded) != _hash(fields)
assert schema_fingerprint() == _fingerprint_of(field_descriptors())
def test_fingerprint_never_touches_the_schema_builder(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
before = schema_fingerprint()
class _RemovedSchemaBuilder:
def __init__(self) -> None:
raise AssertionError("tantivy.SchemaBuilder was consulted")
monkeypatch.setattr(tantivy, "SchemaBuilder", _RemovedSchemaBuilder)
with pytest.raises(AssertionError):
build_schema()
assert schema_fingerprint() == before
def _hash(payload: object) -> str:
return hashlib.blake2b(json.dumps(payload).encode()).hexdigest()
def _fingerprint_of(descriptors: list[FieldDescriptor]) -> str:
return _hash([list(descriptor) for descriptor in descriptors])
class TestNeedsRebuildOnFingerprint:
def test_matching_fingerprint_does_not_rebuild(
self,
index_dir: Path,
settings: SettingsWrapper,
) -> None:
settings.SEARCH_LANGUAGE = None
_sentinels(index_dir)
assert needs_rebuild(index_dir) is False
def test_stale_fingerprint_rebuilds_despite_a_matching_version(
self,
index_dir: Path,
settings: SettingsWrapper,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The failure this task exists to prevent: schema edited, version not
bumped. Without the fingerprint check, `reindex --if-needed` reports the
index up to date and every write then raises."""
settings.SEARCH_LANGUAGE = None
_sentinels(index_dir)
extended = [
*field_descriptors(),
FieldDescriptor(
"new_field",
"u64",
stored=False,
indexed=True,
fast=True,
tokenizer=None,
),
]
monkeypatch.setattr(_schema, "field_descriptors", lambda: extended)
assert needs_rebuild(index_dir) is True
def test_reordered_schema_rebuilds(
self,
index_dir: Path,
settings: SettingsWrapper,
monkeypatch: pytest.MonkeyPatch,
) -> None:
settings.SEARCH_LANGUAGE = None
_sentinels(index_dir)
reordered = field_descriptors()
reordered[1], reordered[2] = reordered[2], reordered[1]
monkeypatch.setattr(_schema, "field_descriptors", lambda: reordered)
assert needs_rebuild(index_dir) is True
def test_missing_fingerprint_rebuilds(
self,
index_dir: Path,
settings: SettingsWrapper,
) -> None:
"""No seeding: an index whose schema shape nobody recorded is rebuilt
rather than trusted."""
settings.SEARCH_LANGUAGE = None
(index_dir / ".index_settings.json").write_text(
json.dumps({"schema_version": SCHEMA_VERSION, "language": None}),
)
assert needs_rebuild(index_dir) is True
def test_written_sentinels_satisfy_the_check(
self,
index_dir: Path,
settings: SettingsWrapper,
) -> None:
settings.SEARCH_LANGUAGE = "en"
_write_sentinels(index_dir)
assert needs_rebuild(index_dir) is False
@@ -0,0 +1,164 @@
"""SCHEMA_VERSION must change whenever build_schema()'s field list or order does.
tantivy compares schemas by *ordered* field list. ``Index.open()`` loads the
schema from the index's own ``meta.json``, so reads against an index built by an
older release keep working after a field reorder. Writes do not:
``WriteBatch.__enter__`` calls ``tantivy.Index(build_schema(), path=...)``, an
open-or-create that raises ``ValueError`` on any schema difference. Nothing
catches that ValueError, so consumption, index_document and bulk edit all
hard-fail while ``/api/status/`` still reports the index healthy.
The only thing that saves such an install is ``needs_rebuild()`` noticing the
version stamped in ``.index_settings.json`` is stale.
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
import pytest
import tantivy
from django.conf import settings as django_settings
from documents.search._schema import build_schema
from documents.search._schema import needs_rebuild
from documents.search._schema import open_or_rebuild_index
if TYPE_CHECKING:
from pathlib import Path
pytestmark = [pytest.mark.search]
RELEASED_V1_SCHEMA_VERSION = 1
def _build_released_v1_schema() -> tantivy.Schema:
"""Frozen copy of build_schema() as shipped in v3.0.x (schema version 1).
Deliberately duplicated rather than imported: it must keep describing the
on-disk layout of already-deployed indexes even as build_schema() evolves.
"""
sb = tantivy.SchemaBuilder()
sb.add_unsigned_field("id", stored=True, indexed=True, fast=True)
sb.add_text_field("checksum", stored=True, tokenizer_name="raw")
for field in (
"title",
"correspondent",
"document_type",
"storage_path",
"original_filename",
"content",
):
sb.add_text_field(field, stored=True, tokenizer_name="paperless_text")
for field in ("title_sort", "correspondent_sort", "type_sort"):
sb.add_text_field(
field,
stored=False,
tokenizer_name="simple_analyzer",
fast=True,
)
for field in (
"bigram_content",
"bigram_title",
"bigram_correspondent",
"bigram_document_type",
"bigram_tag",
):
sb.add_text_field(field, stored=False, tokenizer_name="bigram_analyzer")
for field in ("simple_title", "simple_content"):
sb.add_text_field(field, stored=False, tokenizer_name="simple_search_analyzer")
sb.add_text_field("autocomplete_word", stored=False, tokenizer_name="raw")
sb.add_text_field("tag", stored=True, tokenizer_name="paperless_text")
sb.add_json_field("notes", stored=True, tokenizer_name="paperless_text")
sb.add_text_field("notes_text", stored=True, tokenizer_name="paperless_text")
sb.add_json_field("custom_fields", stored=True, tokenizer_name="paperless_text")
for field in (
"correspondent_id",
"document_type_id",
"storage_path_id",
"tag_id",
"owner_id",
"viewer_id",
"viewer_group_id",
):
sb.add_unsigned_field(field, stored=False, indexed=True, fast=True)
for field in ("created", "modified", "added"):
sb.add_date_field(field, stored=True, indexed=True, fast=True)
for field in ("asn", "page_count", "num_notes"):
sb.add_unsigned_field(field, stored=True, indexed=True, fast=True)
return sb.build()
@pytest.fixture
def released_v1_index(tmp_path: Path) -> Path:
"""An index directory as a v3.0.x install would leave it on disk."""
index_dir = tmp_path / "index"
index_dir.mkdir()
tantivy.Index(_build_released_v1_schema(), path=str(index_dir))
(index_dir / ".index_settings.json").write_text(
json.dumps(
{
"schema_version": RELEASED_V1_SCHEMA_VERSION,
"language": django_settings.SEARCH_LANGUAGE,
},
),
)
return index_dir
class TestUpgradeFromReleasedV1Index:
def test_released_v1_index_is_flagged_for_rebuild(
self,
released_v1_index: Path,
) -> None:
"""The current schema differs from v1's, so the sentinel must be stale.
If this fails, `document_index reindex --if-needed` prints "Search index
is up to date" and skips, leaving the mismatched index in place.
"""
assert needs_rebuild(released_v1_index) is True
def test_v1_index_rejects_writes_against_the_current_schema(
self,
released_v1_index: Path,
) -> None:
"""The failure mode the version bump exists to prevent.
This is exactly what WriteBatch.__enter__ does on every index write.
"""
with pytest.raises(ValueError, match="schema does not match"):
tantivy.Index(build_schema(), path=str(released_v1_index))
def test_opening_a_v1_index_leaves_it_writable(
self,
released_v1_index: Path,
) -> None:
"""End to end: open_or_rebuild_index must hand back an index that the
write path can reopen. Before the version bump, needs_rebuild() returned
False here, the stale directory survived untouched, and every subsequent
write raised the ValueError above."""
open_or_rebuild_index(released_v1_index)
tantivy.Index(build_schema(), path=str(released_v1_index))
def test_rebuilt_index_is_not_rebuilt_again(
self,
released_v1_index: Path,
) -> None:
"""The rebuild must stamp the version it actually wrote, otherwise every
startup wipes and reindexes the whole corpus."""
open_or_rebuild_index(released_v1_index)
assert needs_rebuild(released_v1_index) is False
+2 -2
View File
@@ -7,8 +7,8 @@ import pytest
import tantivy
from documents.search._tokenizer import _bigram_analyzer
from documents.search._tokenizer import _paperless_text
from documents.search._tokenizer import _simple_search_analyzer
from documents.search._tokenizer import paperless_text_analyzer
from documents.search._tokenizer import register_tokenizers
if TYPE_CHECKING:
@@ -25,7 +25,7 @@ class TestTokenizers:
sb.add_text_field("content", stored=True, tokenizer_name="paperless_text")
schema = sb.build()
idx = tantivy.Index(schema, path=None)
idx.register_tokenizer("paperless_text", _paperless_text(""))
idx.register_tokenizer("paperless_text", paperless_text_analyzer(""))
return idx
@pytest.fixture
@@ -72,10 +72,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"barcode_enable_tag": None,
"barcode_tag_mapping": None,
"barcode_tag_split": None,
"remote_ocr_engine": None,
"remote_ocr_api_key": None,
"remote_ocr_endpoint": None,
"remote_ocr_mode": None,
"ai_enabled": False,
"llm_embedding_backend": None,
"llm_embedding_model": None,
@@ -874,49 +870,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
config.refresh_from_db()
self.assertEqual(config.llm_api_key, None)
def test_update_remote_ocr_api_key(self) -> None:
"""
GIVEN:
- Existing config with remote_ocr_api_key specified
WHEN:
- API to update remote_ocr_api_key is called with all *s
- API to update remote_ocr_api_key is called with empty string
THEN:
- remote_ocr_api_key is unchanged
- remote_ocr_api_key is set to None
"""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_api_key = "1234567890"
config.save()
# Test with all *
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "*" * 32,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, "1234567890")
# Test with empty string
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, None)
def test_enable_ai_index_triggers_update(self) -> None:
"""
GIVEN:
+1 -46
View File
@@ -532,29 +532,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": False})
@mock.patch("documents.views.bulk_edit.reprocess")
def test_reprocess_documents_endpoint_remote_ocr(self, m) -> None:
"""
GIVEN:
- API data to reprocess a document with remote OCR requested
WHEN:
- API is called
THEN:
- reprocess is called with remote_ocr=True
"""
self.setup_mock(m, "reprocess")
response = self.client.post(
"/api/documents/reprocess/",
json.dumps({"documents": [self.doc1.id], "remote_ocr": True}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
self.assertEqual(len(kwargs), 0)
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None:
@@ -1575,29 +1553,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
),
)
def test_legacy_bulk_edit_reprocess_invalid_remote_ocr(self) -> None:
"""
GIVEN:
- The deprecated bulk_edit endpoint with a non-boolean remote_ocr
WHEN:
- API is called
THEN:
- The request is rejected rather than passed through to the task
"""
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc1.id],
"method": "reprocess",
"parameters": {"remote_ocr": "yes please"},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
@mock.patch("documents.views.bulk_edit.edit_pdf")
def test_edit_pdf(self, m) -> None:
self.setup_mock(m, "edit_pdf")
+23 -128
View File
@@ -2,14 +2,15 @@ from __future__ import annotations
import datetime
from typing import TYPE_CHECKING
from unittest import TestCase
from unittest import mock
from auditlog.models import LogEntry # type: ignore[import-untyped]
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import FieldError
from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import TestCase as DjangoTestCase
from django.utils import timezone
from rest_framework import status
from rest_framework.test import APITestCase
@@ -20,8 +21,6 @@ from documents.filters import TitleContentFilter
from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import read_streaming_response
from documents.versioning import annotate_effective_content
from documents.views import DocumentSelectionMixin
if TYPE_CHECKING:
from pathlib import Path
@@ -890,103 +889,32 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
)
class TestVersionAwareFilters(DjangoTestCase):
"""
The filters annotate effective_content themselves rather than relying on
the caller's queryset carrying it, so they stay version-aware on a plain
Document queryset (e.g. the bulk-edit "select all matching" path).
"""
class TestVersionAwareFilters(TestCase):
def test_title_content_filter_falls_back_to_content(self) -> None:
queryset = mock.Mock()
fallback_queryset = mock.Mock()
queryset.filter.side_effect = [FieldError("missing field"), fallback_queryset]
def setUp(self) -> None:
super().setUp()
self.root = Document.objects.create(
title="root",
checksum="root",
mime_type="application/pdf",
content="superseded-content",
)
Document.objects.create(
title="version",
checksum="version",
mime_type="application/pdf",
root_document=self.root,
version_index=1,
content="latest-content",
)
self.unversioned = Document.objects.create(
title="unversioned",
checksum="unversioned",
mime_type="application/pdf",
content="latest-content",
)
result = TitleContentFilter().filter(queryset, " latest ")
def test_title_content_filter_matches_latest_version_content(self) -> None:
result = TitleContentFilter().filter(
Document.objects.filter(root_document__isnull=True),
self.assertIs(result, fallback_queryset)
self.assertEqual(queryset.filter.call_count, 2)
def test_effective_content_filter_falls_back_to_content_lookup(self) -> None:
queryset = mock.Mock()
fallback_queryset = mock.Mock()
queryset.filter.side_effect = [FieldError("missing field"), fallback_queryset]
result = EffectiveContentFilter(lookup_expr="icontains").filter(
queryset,
" latest ",
)
self.assertCountEqual(
[doc.id for doc in result],
[self.root.id, self.unversioned.id],
)
def test_effective_content_filter_matches_latest_version_content(self) -> None:
result = EffectiveContentFilter(lookup_expr="icontains").filter(
Document.objects.filter(root_document__isnull=True),
" latest ",
)
self.assertCountEqual(
[doc.id for doc in result],
[self.root.id, self.unversioned.id],
)
def test_effective_content_filter_ignores_superseded_content(self) -> None:
result = EffectiveContentFilter(lookup_expr="icontains").filter(
Document.objects.filter(root_document__isnull=True),
"superseded",
)
self.assertEqual(list(result), [])
def test_filters_reuse_an_existing_annotation(self) -> None:
"""
Annotating twice under the same alias is an error, so an already
annotated queryset (the search path) has to be left alone.
"""
annotated = annotate_effective_content(
Document.objects.filter(root_document__isnull=True),
)
result = EffectiveContentFilter(lookup_expr="icontains").filter(
annotated,
"latest",
)
self.assertCountEqual(
[doc.id for doc in result],
[self.root.id, self.unversioned.id],
)
def test_bulk_selection_does_not_match_superseded_content(self) -> None:
"""
Bulk edit's "select all matching" builds its own queryset, so before
the filters annotated for themselves it matched the root document's
superseded content -- selecting documents the list view, filtered by
the same term, does not show.
"""
user = User.objects.create_superuser(username="bulk_selection")
selected = DocumentSelectionMixin()._resolve_document_ids(
user=user,
validated_data={
"all": True,
"filters": {"content__icontains": "superseded"},
},
)
self.assertEqual(selected, [])
self.assertIs(result, fallback_queryset)
first_kwargs = queryset.filter.call_args_list[0].kwargs
second_kwargs = queryset.filter.call_args_list[1].kwargs
self.assertEqual(first_kwargs, {"effective_content__icontains": "latest"})
self.assertEqual(second_kwargs, {"content__icontains": "latest"})
def test_effective_content_filter_returns_input_for_empty_values(self) -> None:
queryset = mock.Mock()
@@ -995,36 +923,3 @@ class TestVersionAwareFilters(DjangoTestCase):
self.assertIs(result, queryset)
queryset.filter.assert_not_called()
class TestBulkSelectionExcludesVersions(DjangoTestCase):
def test_select_all_matching_does_not_select_version_documents(self) -> None:
"""
"Select all matching" reconstructs the document list, which never
contains version documents as rows of their own.
"""
user = User.objects.create_superuser(username="bulk_versions")
root = Document.objects.create(
title="shared-title root",
checksum="bulk-root",
mime_type="application/pdf",
content="root",
)
Document.objects.create(
title="shared-title version",
checksum="bulk-version",
mime_type="application/pdf",
root_document=root,
version_index=1,
content="version",
)
selected = DocumentSelectionMixin()._resolve_document_ids(
user=user,
validated_data={
"all": True,
"filters": {"title__icontains": "shared-title"},
},
)
self.assertEqual(selected, [root.id])
-23
View File
@@ -1917,29 +1917,6 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertEqual(len(response.data["documents"]), 1)
self.assertEqual(response.data["documents"][0]["id"], title_match.id)
def test_global_search_returns_latest_version_content(self) -> None:
root = Document.objects.create(
title="bank statement",
content="superseded content",
checksum="GSV1",
pk=23,
)
Document.objects.create(
title="bank statement v2",
content="latest content",
checksum="GSV2",
pk=24,
root_document=root,
version_index=1,
)
self.client.force_authenticate(self.user)
response = self.client.get("/api/search/?query=bank&db_only=true")
self.assertEqual(response.status_code, status.HTTP_200_OK)
returned = {doc["id"]: doc["content"] for doc in response.data["documents"]}
self.assertEqual(returned.get(root.id), "latest content")
def test_global_search_filters_owned_mail_objects(self) -> None:
user1 = User.objects.create_user("mail-search-user")
user2 = User.objects.create_user("other-mail-search-user")
@@ -60,10 +60,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
},
"email_enabled": False,
"ai_enabled": False,
"remote_ocr": {
"configured": False,
"mode": "always",
},
},
)
@@ -158,50 +154,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
str(response.data["settings"]),
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="somekey",
REMOTE_OCR_ENDPOINT="https://example.cognitiveservices.azure.com",
REMOTE_OCR_MODE="workflow_only",
)
def test_settings_reports_remote_ocr_when_configured(self) -> None:
"""
GIVEN:
- A fully configured remote OCR engine in workflow_only mode
WHEN:
- The ui_settings endpoint is called
THEN:
- The UI is told remote OCR is available and selective, so it can
offer it where it would actually change something
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
response.data["settings"]["remote_ocr"],
{"configured": True, "mode": "workflow_only"},
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
def test_settings_reports_remote_ocr_incompletely_configured(self) -> None:
"""
GIVEN:
- An engine named but missing its endpoint and API key
WHEN:
- The ui_settings endpoint is called
THEN:
- It is reported as not configured, matching what the parser
registry will actually do
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertFalse(response.data["settings"]["remote_ocr"]["configured"])
@override_settings(
OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
GMAIL_OAUTH_CLIENT_ID="abc123",
-274
View File
@@ -506,280 +506,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(Workflow.objects.count(), 1)
def test_api_create_remote_ocr_action_requires_consumption_trigger(
self,
) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- No consumption started trigger, so the action could never run
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
existing_count = Workflow.objects.count()
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR too late",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Workflow.objects.count(), existing_count)
def test_api_create_remote_ocr_action_with_consumption_trigger(self) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- A consumption started trigger alongside another trigger type
WHEN:
- API is called
THEN:
- The workflow is created, the action applies to consumption only
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR on consume",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
"filter_filename": "*.pdf",
},
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
def test_api_partial_update_adds_remote_ocr_action(self) -> None:
"""
GIVEN:
- An existing workflow with a consumption started trigger
WHEN:
- A partial update adds a remote OCR action without resubmitting triggers
THEN:
- The existing trigger is considered and the update succeeds
"""
response = self.client.patch(
f"{self.ENDPOINT}{self.workflow.id}/",
json.dumps(
{
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
self.workflow.actions.get().type,
WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
def test_api_partial_update_cannot_remove_remote_ocr_trigger(self) -> None:
"""
GIVEN:
- An existing workflow with a remote OCR action
- An existing consumption started trigger
WHEN:
- A partial update replaces the trigger without resubmitting actions
THEN:
- The existing action is considered and the update is rejected
"""
self.action.type = WorkflowAction.WorkflowActionType.REMOTE_OCR
self.action.save()
response = self.client.patch(
f"{self.ENDPOINT}{self.workflow.id}/",
json.dumps(
{
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(self.workflow.triggers.get(), self.trigger)
def _post_ai_suggestions_workflow(self, *, trigger_types, action: dict):
def trigger(trigger_type):
# consumption triggers require a filter of their own
if trigger_type == WorkflowTrigger.WorkflowTriggerType.CONSUMPTION:
return {"type": trigger_type, "filter_filename": "*.pdf"}
return {"type": trigger_type}
return self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Apply AI suggestions",
"order": 1,
"triggers": [trigger(t) for t in trigger_types],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS,
**action,
},
],
},
),
content_type="application/json",
)
def test_api_create_apply_ai_suggestions_action(self) -> None:
"""
GIVEN:
- API request to create a workflow with an apply AI suggestions
action and a valid set of fields
WHEN:
- API is called
THEN:
- The workflow is created with the chosen options
"""
response = self._post_ai_suggestions_workflow(
trigger_types=[WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED],
action={
"ai_suggestion_fields": ["title", "tags", "correspondent"],
"ai_create_missing": True,
"ai_overwrite_existing": True,
},
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
action = Workflow.objects.get(name="Apply AI suggestions").actions.first()
self.assertEqual(
action.ai_suggestion_fields,
["title", "tags", "correspondent"],
)
self.assertTrue(action.ai_create_missing)
self.assertTrue(action.ai_overwrite_existing)
def test_api_create_apply_ai_suggestions_action_requires_fields(self) -> None:
"""
GIVEN:
- API request to create an apply AI suggestions action with no
fields selected, which could never do anything
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
existing_count = Workflow.objects.count()
response = self._post_ai_suggestions_workflow(
trigger_types=[WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED],
action={"ai_suggestion_fields": []},
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Workflow.objects.count(), existing_count)
def test_api_create_apply_ai_suggestions_action_rejects_unknown_field(
self,
) -> None:
"""
GIVEN:
- API request to create an apply AI suggestions action naming a
field that does not exist
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
"""
response = self._post_ai_suggestions_workflow(
trigger_types=[WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED],
action={"ai_suggestion_fields": ["title", "not_a_field"]},
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
def test_api_create_apply_ai_suggestions_action_rejects_consumption_only(
self,
) -> None:
"""
GIVEN:
- API request to create an apply AI suggestions action whose only
trigger is consumption started, so there is no document content
to make suggestions from yet
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
existing_count = Workflow.objects.count()
response = self._post_ai_suggestions_workflow(
trigger_types=[WorkflowTrigger.WorkflowTriggerType.CONSUMPTION],
action={"ai_suggestion_fields": ["title"]},
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Workflow.objects.count(), existing_count)
def test_api_create_apply_ai_suggestions_action_allows_extra_consumption_trigger(
self,
) -> None:
"""
GIVEN:
- API request to create an apply AI suggestions action with a
consumption trigger alongside a usable one
WHEN:
- API is called
THEN:
- The workflow is created, the action applies to the other trigger
"""
response = self._post_ai_suggestions_workflow(
trigger_types=[
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
],
action={"ai_suggestion_fields": ["title"]},
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
def test_api_create_workflow_trigger_action_empty_fields(self) -> None:
"""
GIVEN:
-53
View File
@@ -1782,56 +1782,3 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIn("wrong password", str(exc.exception))
self.assertIn("Error removing password from document", cm.output[0])
class TestBulkEditReprocess(DirectoriesMixin, TestCase):
def setUp(self) -> None:
super().setUp()
self.doc = Document.objects.create(
title="test",
checksum="A",
mime_type="application/pdf",
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_defaults_to_local(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that says nothing about remote OCR
WHEN:
- reprocess is called
THEN:
- The task is queued without asking for the remote engine
"""
result = bulk_edit.reprocess([self.doc.id])
self.assertEqual(result, "OK")
mock_task.apply_async.assert_called_once()
_, kwargs = mock_task.apply_async.call_args
self.assertEqual(
kwargs["kwargs"],
{"document_id": self.doc.id, "remote_ocr": False},
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_passes_remote_ocr(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that explicitly asks for remote OCR
WHEN:
- reprocess is called
THEN:
- The request is forwarded to the task for every document
"""
other = Document.objects.create(
title="test2",
checksum="B",
mime_type="application/pdf",
)
bulk_edit.reprocess([self.doc.id, other.id], remote_ocr=True)
self.assertEqual(mock_task.apply_async.call_count, 2)
for call in mock_task.apply_async.call_args_list:
self.assertTrue(call.kwargs["kwargs"]["remote_ocr"])
-80
View File
@@ -1559,72 +1559,6 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
consumer.run_post_consume_script(doc)
class TestConsumerRemoteOCR(
DirectoriesMixin,
FileSystemAssertsMixin,
GetConsumerMixin,
TestCase,
):
"""
The consumer resolves the remote OCR mode and the per-document request from
workflows into the allow_remote flag it hands to the parser registry.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.consumer.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
self.addCleanup(patcher.stop)
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
src = (
Path(__file__).parent
/ "samples"
/ "documents"
/ "originals"
/ "0000001.pdf"
)
dst = self.dirs.scratch_dir / "sample.pdf"
shutil.copy(src, dst)
with self.get_consumer(dst, overrides=overrides) as consumer:
consumer.run()
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
"""
GIVEN: Remote OCR mode is 'always'.
WHEN: A document is consumed without any workflow asking for it.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A document is consumed and nothing asked for remote OCR.
THEN: The remote parser is excluded.
"""
self.assertFalse(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A workflow set remote_ocr on the metadata overrides.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
)
class TestMetadataOverrides(TestCase):
def test_update_skip_asn_if_exists(self) -> None:
base = DocumentMetadataOverrides()
@@ -1632,20 +1566,6 @@ class TestMetadataOverrides(TestCase):
base.update(incoming)
self.assertTrue(base.skip_asn_if_exists)
def test_update_remote_ocr(self) -> None:
base = DocumentMetadataOverrides()
base.update(DocumentMetadataOverrides(remote_ocr=True))
self.assertTrue(base.remote_ocr)
def test_update_remote_ocr_is_not_unset(self) -> None:
"""
A later workflow that says nothing must not undo an earlier one that
asked for remote OCR.
"""
base = DocumentMetadataOverrides(remote_ocr=True)
base.update(DocumentMetadataOverrides())
self.assertTrue(base.remote_ocr)
def test_update_actor_and_version_label(self) -> None:
base = DocumentMetadataOverrides(
actor_id=1,
-19
View File
@@ -385,25 +385,6 @@ class TestTaskFailureHandler:
task_failure_handler(task_id=None, exception=ValueError("x"), traceback=None)
@pytest.mark.django_db
class TestApplyAiSuggestionsTracking:
def test_records_the_document_it_is_for(self) -> None:
"""
The action queues one task per document, so the tracked record notes
which document it is for -- otherwise a bulk run is an indistinguishable
wall of identical entries in the tasks list.
"""
task_id = send_publish(
"documents.tasks.apply_ai_suggestions",
(),
{"action_id": 1, "document_id": 42},
)
task = PaperlessTask.objects.get(task_id=task_id)
assert task.task_type == PaperlessTask.TaskType.APPLY_AI_SUGGESTIONS
assert task.input_data == {"document_id": 42}
@pytest.mark.django_db
class TestTaskRevokedHandler:
def test_marks_task_revoked(self, mocker: pytest_mock.MockerFixture) -> None:
-147
View File
@@ -14,7 +14,6 @@ from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.models import Tag
from documents.models import WorkflowAction
from documents.sanity_checker import SanityCheckFailedException
from documents.sanity_checker import SanityCheckMessages
from documents.tests.test_classifier import dummy_preprocess
@@ -288,45 +287,6 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
"""
Consumption workflows do not run on reprocess, so the remote parser is
used only in 'always' mode or when the caller explicitly asks for it.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.tasks.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = None
self.addCleanup(patcher.stop)
self.doc = Document.objects.create(
title="test",
content="my document",
checksum="wow",
mime_type="application/pdf",
)
def _allow_remote(self, **kwargs) -> bool:
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return call_kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
self.assertTrue(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
self.assertFalse(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
self.assertTrue(self._allow_remote(remote_ocr=True))
class TestAIIndex(DirectoriesMixin, TestCase):
@override_settings(
AI_ENABLED=True,
@@ -448,110 +408,3 @@ class TestAIIndex(DirectoriesMixin, TestCase):
rebuild=False,
document_ids=doc_ids,
)
class TestApplyAISuggestionsTask(DirectoriesMixin, TestCase):
def setUp(self) -> None:
super().setUp()
self.doc = Document.objects.create(
title="doc",
content="content",
checksum="apply-ai-suggestions",
)
self.action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS,
ai_suggestion_fields=[WorkflowAction.AISuggestionField.TITLE],
)
def test_reindexes_without_sending_document_updated(self) -> None:
"""
GIVEN:
- An apply AI suggestions action that changes the document
WHEN:
- The task runs
THEN:
- The search index and caches are refreshed directly, deliberately
not via the document_updated signal: that re-runs updated
workflows, which for this action means queueing another LLM
query for a document it just changed, forever
"""
with (
mock.patch(
"documents.workflows.ai.apply_ai_suggestions_to_document",
return_value=["title"],
),
mock.patch("documents.tasks.index_document") as index_document,
mock.patch("documents.tasks.clear_document_caches") as clear_caches,
mock.patch("documents.tasks.document_updated") as document_updated,
):
tasks.apply_ai_suggestions(self.action.pk, self.doc.pk)
index_document.delay.assert_called_once_with(self.doc.pk)
clear_caches.assert_called_once_with(self.doc.pk)
document_updated.send.assert_not_called()
def test_no_changes_skips_reindex(self) -> None:
"""
GIVEN:
- An apply AI suggestions action that changes nothing
WHEN:
- The task runs
THEN:
- No reindexing work is queued
"""
with (
mock.patch(
"documents.workflows.ai.apply_ai_suggestions_to_document",
return_value=[],
),
mock.patch("documents.tasks.index_document") as index_document,
):
tasks.apply_ai_suggestions(self.action.pk, self.doc.pk)
index_document.delay.assert_not_called()
@override_settings(AI_ENABLED=True, LLM_EMBEDDING_BACKEND="huggingface")
def test_updates_llm_index_when_enabled(self) -> None:
"""
GIVEN:
- An apply AI suggestions action that changes the document
- The LLM index is enabled
WHEN:
- The task runs
THEN:
- The document is updated in the LLM index too
"""
with (
mock.patch(
"documents.workflows.ai.apply_ai_suggestions_to_document",
return_value=["title"],
),
mock.patch("documents.tasks.index_document"),
mock.patch(
"documents.tasks.update_document_in_llm_index",
) as update_in_llm_index,
):
tasks.apply_ai_suggestions(self.action.pk, self.doc.pk)
update_in_llm_index.apply_async.assert_called_once()
def test_deleted_document_is_a_noop(self) -> None:
"""
GIVEN:
- A document that was deleted between the workflow running and the
queued task starting
WHEN:
- The task runs
THEN:
- It logs and exits rather than raising
"""
with (
mock.patch(
"documents.workflows.ai.apply_ai_suggestions_to_document",
) as apply_suggestions,
self.assertLogs("paperless.tasks", level="WARNING") as cm,
):
tasks.apply_ai_suggestions(self.action.pk, self.doc.pk + 1000)
apply_suggestions.assert_not_called()
self.assertIn("no longer exists", "".join(cm.output))
-560
View File
@@ -31,10 +31,7 @@ from documents.file_handling import create_source_path_directory
from documents.file_handling import generate_filename
from documents.file_handling import generate_unique_filename
from documents.signals.handlers import run_workflows
from documents.workflows.ai import apply_ai_suggestions_to_document
from documents.workflows.webhooks import send_webhook
from paperless_ai.base_model import ClassificationSuggestions
from paperless_ai.exceptions import LLMTimeoutError
if TYPE_CHECKING:
from django.db.models import QuerySet
@@ -5412,560 +5409,3 @@ class TestDateWorkflowLocalization(
document = Document.objects.first()
assert document is not None
assert document.title == expected_title
class TestRemoteOCRWorkflowAction(DirectoriesMixin, SampleDirMixin, APITestCase):
def _make_workflow(self, trigger_type) -> None:
trigger = WorkflowTrigger.objects.create(type=trigger_type)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
def test_consumption_trigger_requests_remote_ocr(self) -> None:
"""
GIVEN:
- A consumption workflow with a remote OCR action
WHEN:
- A matching document is consumed
THEN:
- The overrides ask for remote OCR, which is what the consumer
reads when choosing a parser
"""
self._make_workflow(WorkflowTrigger.WorkflowTriggerType.CONSUMPTION)
test_file = shutil.copy(
self.SAMPLE_DIR / "simple.pdf",
self.dirs.scratch_dir / "simple.pdf",
)
overrides = DocumentMetadataOverrides()
run_workflows(
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=test_file,
),
overrides=overrides,
)
self.assertTrue(overrides.remote_ocr)
def test_other_trigger_types_are_ignored(self) -> None:
"""
GIVEN:
- A workflow with a remote OCR action that also has a
non-consumption trigger, which is a valid combination
WHEN:
- The non-consumption trigger fires
THEN:
- The action is skipped, since the document has already been
parsed by this point
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
)
updated_trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger, updated_trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
original_filename="sample.pdf",
)
with self.assertLogs("paperless.handlers", level="DEBUG") as cm:
run_workflows(
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
doc,
)
self.assertIn("only applies to consumption triggers", "".join(cm.output))
SUGGESTIONS: ClassificationSuggestions = {
"title": "Suggested Title",
"tags": {
"existing_ids": [],
"new_names": ["Existing Tag", "Suggested Tag"],
},
"correspondents": {
"existing_ids": [],
"new_names": ["Existing Correspondent", "Suggested Correspondent"],
},
"document_types": {
"existing_ids": [],
"new_names": ["Suggested Document Type"],
},
"storage_paths": {
"existing_ids": [],
"new_names": ["Suggested Storage Path"],
},
"dates": ["2024-03-05"],
}
ALL_SUGGESTION_FIELDS = [
WorkflowAction.AISuggestionField.TITLE,
WorkflowAction.AISuggestionField.TAGS,
WorkflowAction.AISuggestionField.CORRESPONDENT,
WorkflowAction.AISuggestionField.DOCUMENT_TYPE,
WorkflowAction.AISuggestionField.STORAGE_PATH,
WorkflowAction.AISuggestionField.CREATED,
]
@override_settings(AI_ENABLED=True)
class TestApplyAISuggestionsWorkflowAction(
DirectoriesMixin,
SampleDirMixin,
APITestCase,
):
def setUp(self) -> None:
super().setUp()
self.user = User.objects.create(username="ai-user")
self.doc = Document.objects.create(
title="original.pdf",
content="the document content",
checksum="ai-suggestions-checksum",
mime_type="application/pdf",
created=datetime.date(2020, 1, 1),
owner=self.user,
)
def make_action(self, **kwargs) -> WorkflowAction:
return WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.APPLY_AI_SUGGESTIONS,
ai_suggestion_fields=kwargs.pop(
"ai_suggestion_fields",
ALL_SUGGESTION_FIELDS,
),
**kwargs,
)
def make_workflow(self, action: WorkflowAction, trigger_type) -> Workflow:
trigger = WorkflowTrigger.objects.create(type=trigger_type)
w = Workflow.objects.create(name="Apply AI suggestions", order=0)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
return w
def apply(
self,
action: WorkflowAction,
suggestions: ClassificationSuggestions = SUGGESTIONS,
) -> list[str]:
with mock.patch(
"documents.workflows.ai.get_ai_document_classification",
return_value=suggestions,
):
changed = apply_ai_suggestions_to_document(action, self.doc)
self.doc.refresh_from_db()
return changed
def test_document_added_trigger_queues_task(self) -> None:
"""
GIVEN:
- A document added workflow with an apply AI suggestions action
WHEN:
- A matching document is added
THEN:
- The work is queued rather than run inline, so a slow LLM query
cannot stall the rest of the workflow run
"""
action = self.make_action()
self.make_workflow(action, WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED)
with mock.patch("documents.tasks.apply_ai_suggestions.delay") as delay:
run_workflows(
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
self.doc,
)
delay.assert_called_once_with(action_id=action.pk, document_id=self.doc.pk)
def test_consumption_trigger_is_ignored(self) -> None:
"""
GIVEN:
- A workflow with an apply AI suggestions action and a consumption
trigger alongside a valid one
WHEN:
- The consumption trigger fires
THEN:
- The action is skipped, since the document has not been parsed
yet and so has no content to make suggestions from
"""
action = self.make_action()
w = self.make_workflow(
action,
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
)
w.triggers.add(
WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
),
)
test_file = shutil.copy(
self.SAMPLE_DIR / "simple.pdf",
self.dirs.scratch_dir / "simple.pdf",
)
with (
mock.patch("documents.tasks.apply_ai_suggestions.delay") as delay,
self.assertLogs("paperless.handlers", level="DEBUG") as cm,
):
run_workflows(
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=test_file,
),
overrides=DocumentMetadataOverrides(),
)
delay.assert_not_called()
self.assertIn("does not apply to consumption triggers", "".join(cm.output))
def test_no_selected_fields_does_nothing(self) -> None:
"""
GIVEN:
- An action with no suggestion fields selected
WHEN:
- The action is applied
THEN:
- Nothing is changed and it is logged
"""
action = self.make_action(ai_suggestion_fields=[])
with self.assertLogs("paperless.workflows.ai", level="WARNING") as cm:
changed = self.apply(action)
self.assertEqual(changed, [])
self.assertIn("no AI suggestion fields selected", "".join(cm.output))
@override_settings(AI_ENABLED=False)
def test_ai_disabled_does_nothing(self) -> None:
"""
GIVEN:
- An action on an install where AI has since been disabled
WHEN:
- The action is applied
THEN:
- Nothing is changed and it is logged
"""
action = self.make_action()
with self.assertLogs("paperless.workflows.ai", level="ERROR") as cm:
changed = self.apply(action)
self.assertEqual(changed, [])
self.assertIn("AI is not enabled", "".join(cm.output))
def test_invalid_configuration_leaves_document_untouched(self) -> None:
"""
GIVEN:
- An AI backend that is misconfigured
WHEN:
- The action is applied
THEN:
- The failure is logged and the document is left alone. It is not
re-raised, because retrying will not fix a bad configuration
"""
action = self.make_action()
with (
mock.patch(
"documents.workflows.ai.get_ai_document_classification",
side_effect=ValueError("nope"),
),
self.assertLogs("paperless.workflows.ai", level="ERROR") as cm,
):
changed = apply_ai_suggestions_to_document(action, self.doc)
self.assertEqual(changed, [])
self.doc.refresh_from_db()
self.assertEqual(self.doc.title, "original.pdf")
self.assertIn("Invalid AI configuration", "".join(cm.output))
def test_transient_llm_failure_is_raised_for_retry(self) -> None:
"""
GIVEN:
- An LLM backend that times out, or rate limits the request
WHEN:
- The action is applied
THEN:
- The error propagates so the queued task can back off and retry,
rather than silently dropping this document's suggestions
"""
action = self.make_action()
with (
mock.patch(
"documents.workflows.ai.get_ai_document_classification",
side_effect=LLMTimeoutError(),
),
self.assertRaises(LLMTimeoutError),
):
apply_ai_suggestions_to_document(action, self.doc)
self.doc.refresh_from_db()
self.assertEqual(self.doc.title, "original.pdf")
def test_only_matching_objects_are_applied(self) -> None:
"""
GIVEN:
- An action without create missing, and only some of the suggested
objects existing
WHEN:
- The action is applied
THEN:
- Only the existing objects are assigned, unmatched suggestions are
dropped rather than creating anything
"""
tag = Tag.objects.create(name="Existing Tag", owner=self.user)
correspondent = Correspondent.objects.create(
name="Existing Correspondent",
owner=self.user,
)
action = self.make_action(ai_overwrite_existing=True)
changed = self.apply(action)
self.assertEqual(self.doc.correspondent, correspondent)
self.assertEqual(list(self.doc.tags.all()), [tag])
# Nothing matched for these and create missing is off
self.assertIsNone(self.doc.document_type)
self.assertIsNone(self.doc.storage_path)
self.assertNotIn("document_type", changed)
self.assertEqual(Tag.objects.count(), 1)
self.assertEqual(Correspondent.objects.count(), 1)
def test_existing_id_suggestions_are_applied(self) -> None:
"""
GIVEN:
- AI suggestions that select existing taxonomy candidates by ID
WHEN:
- The suggestions are applied
THEN:
- Each selected object is assigned to the document
"""
tag = Tag.objects.create(name="Existing Tag", owner=self.user)
correspondent = Correspondent.objects.create(
name="Existing Correspondent",
owner=self.user,
)
document_type = DocumentType.objects.create(
name="Existing Document Type",
owner=self.user,
)
storage_path = StoragePath.objects.create(
name="Existing Storage Path",
path="{{ title }}",
owner=self.user,
)
action = self.make_action(ai_overwrite_existing=True)
suggestions: ClassificationSuggestions = {
**SUGGESTIONS,
"tags": {"existing_ids": [tag.pk], "new_names": []},
"correspondents": {
"existing_ids": [correspondent.pk],
"new_names": [],
},
"document_types": {
"existing_ids": [document_type.pk],
"new_names": [],
},
"storage_paths": {
"existing_ids": [storage_path.pk],
"new_names": [],
},
}
changed = self.apply(action, suggestions)
self.assertEqual(list(self.doc.tags.all()), [tag])
self.assertEqual(self.doc.correspondent, correspondent)
self.assertEqual(self.doc.document_type, document_type)
self.assertEqual(self.doc.storage_path, storage_path)
self.assertTrue(
{"tags", "correspondent", "document_type", "storage_path"} <= set(changed),
)
def test_create_missing_creates_objects_owned_by_document_owner(self) -> None:
"""
GIVEN:
- An action with create missing enabled
WHEN:
- The action is applied and suggestions match nothing
THEN:
- Tags, correspondents and document types are created, owned by the
document owner so they stay private to them
- Storage paths are never created, since a path template cannot be
inferred from a name
"""
action = self.make_action(
ai_create_missing=True,
ai_overwrite_existing=True,
)
changed = self.apply(action)
self.assertEqual(
sorted(t.name for t in self.doc.tags.all()),
["Existing Tag", "Suggested Tag"],
)
self.assertEqual(self.doc.correspondent.name, "Existing Correspondent")
self.assertEqual(self.doc.correspondent.owner, self.user)
self.assertEqual(self.doc.document_type.name, "Suggested Document Type")
self.assertEqual(self.doc.document_type.owner, self.user)
self.assertIsNone(self.doc.storage_path)
self.assertFalse(StoragePath.objects.exists())
self.assertNotIn("storage_path", changed)
def test_overwrite_disabled_keeps_existing_values(self) -> None:
"""
GIVEN:
- An action without overwrite existing
- A document that already has a title, created date and
correspondent
WHEN:
- The action is applied
THEN:
- The existing values are kept, only the empty document type is
filled in
"""
existing = Correspondent.objects.create(name="Mine", owner=self.user)
self.doc.correspondent = existing
self.doc.save()
action = self.make_action(ai_create_missing=True)
changed = self.apply(action)
self.assertEqual(self.doc.title, "original.pdf")
self.assertEqual(self.doc.created, datetime.date(2020, 1, 1))
self.assertEqual(self.doc.correspondent, existing)
self.assertEqual(self.doc.document_type.name, "Suggested Document Type")
self.assertNotIn("title", changed)
self.assertNotIn("correspondent", changed)
def test_overwrite_enabled_replaces_existing_values(self) -> None:
"""
GIVEN:
- An action with overwrite existing
- A document that already has a title and created date
WHEN:
- The action is applied
THEN:
- The suggested values replace them
"""
action = self.make_action(
ai_create_missing=True,
ai_overwrite_existing=True,
)
changed = self.apply(action)
self.assertEqual(self.doc.title, "Suggested Title")
self.assertEqual(self.doc.created, datetime.date(2024, 3, 5))
self.assertIn("title", changed)
self.assertIn("created", changed)
def test_tags_are_added_not_replaced(self) -> None:
"""
GIVEN:
- A document that already has a tag unrelated to the suggestions
WHEN:
- The action is applied with overwrite existing enabled
THEN:
- The existing tag is kept, since suggested tags are always
additive regardless of the overwrite setting
"""
kept = Tag.objects.create(name="Do Not Remove", owner=self.user)
self.doc.tags.add(kept)
Tag.objects.create(name="Existing Tag", owner=self.user)
action = self.make_action(ai_overwrite_existing=True)
self.apply(action)
self.assertEqual(
sorted(t.name for t in self.doc.tags.all()),
["Do Not Remove", "Existing Tag"],
)
def test_unselected_fields_are_untouched(self) -> None:
"""
GIVEN:
- An action that only selects the title
WHEN:
- The action is applied
THEN:
- Only the title changes, even though the LLM suggested everything
"""
action = self.make_action(
ai_suggestion_fields=[WorkflowAction.AISuggestionField.TITLE],
ai_create_missing=True,
ai_overwrite_existing=True,
)
changed = self.apply(action)
self.assertEqual(changed, ["title"])
self.assertEqual(self.doc.title, "Suggested Title")
self.assertEqual(self.doc.tags.count(), 0)
self.assertIsNone(self.doc.correspondent)
self.assertEqual(self.doc.created, datetime.date(2020, 1, 1))
def test_another_users_private_objects_are_not_matched(self) -> None:
"""
GIVEN:
- A suggested tag name that exists, but is owned by someone else
WHEN:
- The action is applied
THEN:
- It is not assigned, because the document owner cannot see it
"""
other = User.objects.create(username="someone-else")
Tag.objects.create(name="Existing Tag", owner=other)
action = self.make_action(
ai_suggestion_fields=[WorkflowAction.AISuggestionField.TAGS],
)
self.apply(action)
self.assertEqual(self.doc.tags.count(), 0)
def test_unparsable_dates_are_skipped(self) -> None:
"""
GIVEN:
- Suggested dates that are not all valid
WHEN:
- The action is applied
THEN:
- The first usable date is applied and the rest ignored
"""
action = self.make_action(
ai_suggestion_fields=[WorkflowAction.AISuggestionField.CREATED],
ai_overwrite_existing=True,
)
with mock.patch(
"documents.workflows.ai.get_ai_document_classification",
return_value={**SUGGESTIONS, "dates": ["not a date", "2019-07-04"]},
):
changed = apply_ai_suggestions_to_document(action, self.doc)
self.doc.refresh_from_db()
self.assertEqual(changed, ["created"])
self.assertEqual(self.doc.created, datetime.date(2019, 7, 4))
-15
View File
@@ -43,21 +43,6 @@ def annotate_effective_content(documents: QuerySet[Document]) -> QuerySet[Docume
)
def ensure_effective_content(documents: QuerySet[Document]) -> QuerySet[Document]:
"""
Annotates effective_content unless the queryset already carries it.
Lets a filter depend on effective_content without having to assume its
caller annotated one -- annotating twice under the same alias is an error,
and silently matching on the root document's own content instead is worse,
because the same filter then selects different documents depending on which
queryset it was handed.
"""
if "effective_content" in documents.query.annotations:
return documents
return annotate_effective_content(documents)
def sort_versions_newest_first(documents: list[Document]) -> list[Document]:
"""
Same sorting as versions_newest_first()
+19 -29
View File
@@ -230,7 +230,6 @@ from documents.tasks import train_classifier
from documents.tasks import update_document_parent_tags
from documents.utils import get_boolean
from documents.versioning import VersionResolutionError
from documents.versioning import annotate_effective_content
from documents.versioning import get_latest_version_for_root
from documents.versioning import get_request_version_param
from documents.versioning import get_root_document
@@ -240,15 +239,12 @@ from paperless import version
from paperless.celery import app as celery_app
from paperless.config import AIConfig
from paperless.config import GeneralConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ApplicationConfiguration
from paperless.parsers.registry import get_parser_registry
from paperless.parsers.remote import RemoteEngineConfig
from paperless.serialisers import GroupSerializer
from paperless.serialisers import UserSerializer
from paperless.views import StandardPagination
from paperless_ai.ai_classifier import get_ai_document_classification
from paperless_ai.ai_classifier import get_llm_output_language
from paperless_ai.chat import stream_chat_with_documents
from paperless_ai.exceptions import LLMTimeoutError
from paperless_ai.matching import extract_unmatched_names
@@ -667,6 +663,20 @@ class TagViewSet(PermissionsAwareDocumentCountMixin, ModelViewSet[Tag]):
update_document_parent_tags(tag, new_parent)
def _get_llm_output_language(ai_config: AIConfig, request) -> str | None:
output_language = ai_config.llm_output_language
if (
not output_language
and hasattr(request.user, "ui_settings")
and isinstance(
request.user.ui_settings.settings,
dict,
)
):
output_language = request.user.ui_settings.settings.get("language")
return output_language
@extend_schema_view(**generate_object_with_permissions_schema(DocumentTypeSerializer))
class DocumentTypeViewSet(
PermissionsAwareDocumentCountMixin,
@@ -1121,8 +1131,7 @@ class DocumentViewSet(
"custom_fields",
queryset=CustomFieldInstance.objects.select_related("field"),
),
# NotesSerializer nests the author, this avoids query per note
Prefetch("notes", queryset=Note.objects.select_related("user")),
"notes",
)
)
@@ -1530,10 +1539,7 @@ class DocumentViewSet(
if not ai_config.ai_enabled:
return HttpResponseBadRequest("AI is required for this feature")
output_language = get_llm_output_language(
ai_config=ai_config,
user=request.user,
)
output_language = _get_llm_output_language(ai_config=ai_config, request=request)
llm_cache_backend = ":".join(
part
for part in (
@@ -2322,10 +2328,7 @@ class ChatStreamingView(GenericAPIView[Any]):
id__in=permitted_document_ids(request.user),
)
output_language = get_llm_output_language(
ai_config=ai_config,
user=request.user,
)
output_language = _get_llm_output_language(ai_config=ai_config, request=request)
response = StreamingHttpResponse(
stream_chat_with_documents(
@@ -2789,11 +2792,8 @@ class DocumentSelectionMixin:
for key, value in filters.items()
if key not in _TANTIVY_SEARCH_PARAM_NAMES
}
# Operations are addressed to roots, a caller that wants
# to act on a specific version passes its id explicitly instead
permitted_documents = Document.objects.filter(
id__in=permitted_document_ids(user),
root_document__isnull=True,
)
# orm-filtered docs
filtered_documents = DocumentFilterSet(
@@ -3614,13 +3614,8 @@ class GlobalSearchView(PassUserMixin):
OBJECT_LIMIT = 3
docs = []
if request.user.has_perm("documents.view_document"):
# Never more than OBJECT_LIMIT rows come back here, so annotating
# is cheap -- and without it these results show the root
# document's superseded content.
all_docs = annotate_effective_content(
Document.objects.filter(
id__in=permitted_document_ids(request.user),
),
all_docs = Document.objects.filter(
id__in=permitted_document_ids(request.user),
)
if db_only:
docs = all_docs.filter(title__icontains=query)[:OBJECT_LIMIT]
@@ -4105,11 +4100,6 @@ class UiSettingsView(GenericAPIView[Any]):
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
ui_settings["remote_ocr"] = {
"configured": RemoteEngineConfig.from_app_config().engine_is_valid(),
"mode": RemoteOCRConfig().remote_ocr_mode,
}
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
manager = PaperlessMailOAuth2Manager()
if settings.GMAIL_OAUTH_ENABLED:
-259
View File
@@ -1,259 +0,0 @@
import logging
from datetime import date
from datetime import datetime
from typing import TypeVar
from django.contrib.auth.models import User
from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
from documents.models import WorkflowAction
from paperless.config import AIConfig
from paperless_ai.ai_classifier import get_ai_document_classification
from paperless_ai.ai_classifier import get_llm_output_language
from paperless_ai.matching import extract_unmatched_names
from paperless_ai.matching import match_correspondents_by_name
from paperless_ai.matching import match_document_types_by_name
from paperless_ai.matching import match_storage_paths_by_name
from paperless_ai.matching import match_tags_by_name
from paperless_ai.matching import resolve_correspondent_ids
from paperless_ai.matching import resolve_document_type_ids
from paperless_ai.matching import resolve_storage_path_ids
from paperless_ai.matching import resolve_tag_ids
logger = logging.getLogger("paperless.workflows.ai")
AISuggestionField = WorkflowAction.AISuggestionField
ObjT = TypeVar("ObjT", bound=MatchingModel)
# Tags use m2m relation instead
DIRECT_FIELDS: dict[str, str] = {
AISuggestionField.TITLE: "title",
AISuggestionField.CORRESPONDENT: "correspondent",
AISuggestionField.DOCUMENT_TYPE: "document_type",
AISuggestionField.STORAGE_PATH: "storage_path",
AISuggestionField.CREATED: "created",
}
def resolve_date(dates: list[str]) -> date | None:
"""
First usable date out of the suggestions, which are expected as
YYYY-MM-DD. Document.created is a DateField, so only one can be applied.
"""
for value in dates:
try:
return datetime.strptime(value, "%Y-%m-%d").date()
except (TypeError, ValueError):
logger.debug("Ignoring unparsable suggested date %s", value)
return None
def resolve_object(
model: type[ObjT],
names: list[str],
matched: list[ObjT],
*,
create_missing: bool,
owner: User | None,
) -> ObjT | None:
"""
Single object from a suggestion list. The best match if there was one, else
optionally a newly-created object. StoragePaths are excluded.
"""
if matched:
return matched[0]
if not create_missing or model is StoragePath:
return None
unmatched = extract_unmatched_names(names, matched)
if not unmatched:
return None
# (name, owner) is what MatchingModel is unique on
obj, created = model.objects.get_or_create(
name=unmatched[0][:128],
owner=owner,
)
if created:
logger.info("Created %s '%s' from AI suggestion", model.__name__, obj.name)
return obj
def resolve_tags(
names: list[str],
matched: list[Tag],
*,
create_missing: bool,
owner: User | None,
) -> list[Tag]:
"""
Matched tags, plus newly created ones if create_missing is set.
"""
tags = list(matched)
if not create_missing:
return tags
for name in extract_unmatched_names(names, matched):
tag, created = Tag.objects.get_or_create(
name=name[:128],
owner=owner,
)
if created:
logger.info("Created tag '%s' from AI suggestion", tag.name)
tags.append(tag)
return tags
def apply_ai_suggestions_to_document(
action: WorkflowAction,
document: Document,
logging_group=None,
) -> list[str]:
"""
Get suggestions about `document` and write the chosen fields.
Returns the names of the fields that were actually changed.
"""
selected = set(action.ai_suggestion_fields or [])
if not selected:
logger.warning(
"Workflow action %s has no AI suggestion fields selected, skipping",
action.pk,
extra={"group": logging_group},
)
return []
ai_config = AIConfig()
if not ai_config.ai_enabled:
logger.error(
"AI is not enabled, cannot apply AI suggestions for document %s",
document.pk,
extra={"group": logging_group},
)
return []
# Workflows run without a user, so we use the document owner
owner = document.owner
try:
suggestions = get_ai_document_classification(
document,
owner,
get_llm_output_language(ai_config, owner),
)
except ValueError:
# A bad AI config will not fix itself, so swallow it rather than
# letting the caller retry. Timeouts, rate limits, network errors etc
# propagate so the queued task can back off and try again.
logger.exception(
"Invalid AI configuration, cannot get suggestions for document %s",
document.pk,
extra={"group": logging_group},
)
return []
overwrite = action.ai_overwrite_existing
create_missing = action.ai_create_missing
updated_fields: list[str] = []
def should_set(field: str) -> bool:
# The field is selected and (overwrite or it's empty)
return field in selected and (
overwrite or getattr(document, DIRECT_FIELDS[field]) in (None, "")
)
if should_set(AISuggestionField.TITLE):
title = suggestions["title"].strip()
if title:
# title is capped at 128 characters
document.title = title[:128]
updated_fields.append("title")
if should_set(AISuggestionField.CORRESPONDENT):
choice = suggestions["correspondents"]
names = choice["new_names"]
correspondent = resolve_object(
Correspondent,
names,
resolve_correspondent_ids(choice["existing_ids"], owner)
+ match_correspondents_by_name(names, owner),
create_missing=create_missing,
owner=owner,
)
if correspondent:
document.correspondent = correspondent
updated_fields.append("correspondent")
if should_set(AISuggestionField.DOCUMENT_TYPE):
choice = suggestions["document_types"]
names = choice["new_names"]
document_type = resolve_object(
DocumentType,
names,
resolve_document_type_ids(choice["existing_ids"], owner)
+ match_document_types_by_name(names, owner),
create_missing=create_missing,
owner=owner,
)
if document_type:
document.document_type = document_type
updated_fields.append("document_type")
if should_set(AISuggestionField.STORAGE_PATH):
choice = suggestions["storage_paths"]
names = choice["new_names"]
storage_path = resolve_object(
StoragePath,
names,
resolve_storage_path_ids(choice["existing_ids"], owner)
+ match_storage_paths_by_name(names, owner),
create_missing=create_missing,
owner=owner,
)
if storage_path:
document.storage_path = storage_path
updated_fields.append("storage_path")
if should_set(AISuggestionField.CREATED):
created = resolve_date(suggestions["dates"])
if created:
document.created = created
updated_fields.append("created")
if AISuggestionField.TAGS in selected:
choice = suggestions["tags"]
names = choice["new_names"]
tags = resolve_tags(
names,
resolve_tag_ids(choice["existing_ids"], owner)
+ match_tags_by_name(names, owner),
create_missing=create_missing,
owner=owner,
)
if tags:
# Suggested tags are always added, so overwrite_existing
# does not really apply here
document.add_nested_tags(tags)
updated_fields.append("tags")
if updated_fields:
# save fields and update modified (excluding m2m tags from update_fields)
direct_updated_fields = [
field for field in updated_fields if field in DIRECT_FIELDS.values()
]
document.save(update_fields=[*direct_updated_fields, "modified"])
logger.info(
"Applied AI suggestions %s to document %s",
updated_fields or "(none)",
document.pk,
extra={"group": logging_group},
)
return updated_fields
File diff suppressed because it is too large Load Diff
+14
View File
@@ -337,6 +337,20 @@ def check_deprecated_v2_ocr_env_vars(
return warnings
@register()
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
):
return [
Error(
"Azure AI remote parser requires endpoint and API key to be configured.",
),
]
return []
def get_tesseract_langs():
proc = subprocess.run(
[shutil.which("tesseract"), "--list-langs"],
-40
View File
@@ -9,7 +9,6 @@ from paperless.models import CleanChoices
from paperless.models import ColorConvertChoices
from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices
from paperless.models import RemoteOCRMode
@dataclasses.dataclass
@@ -186,45 +185,6 @@ class GeneralConfig(BaseConfig):
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
@dataclasses.dataclass
class RemoteOCRConfig(BaseConfig):
"""
Settings for the remote (cloud) OCR parser
"""
remote_ocr_engine: str | None = dataclasses.field(init=False)
remote_ocr_api_key: str | None = dataclasses.field(init=False)
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
def __post_init__(self) -> None:
app_config = self._get_config_instance()
self.remote_ocr_engine = (
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
)
self.remote_ocr_api_key = (
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
)
self.remote_ocr_endpoint = (
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
)
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
settings.REMOTE_OCR_MODE,
)
@property
def remote_ocr_by_default(self) -> bool:
"""
Whether every supported document goes to the remote engine.
When False the remote engine is used only for documents that
explicitly asked for it, i.e. a workflow matched during consumption or
the user ticked the box when reprocessing.
"""
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
@dataclasses.dataclass
class AIConfig(BaseConfig):
"""
@@ -1,44 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 14:37
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_api_key",
field=models.CharField(
blank=True,
max_length=1024,
null=True,
verbose_name="Sets the remote OCR API key",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_endpoint",
field=models.CharField(
blank=True,
max_length=256,
null=True,
verbose_name="Sets the remote OCR endpoint",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_engine",
field=models.CharField(
blank=True,
choices=[("azureai", "Azure AI Document Intelligence")],
max_length=32,
null=True,
verbose_name="Sets the remote OCR engine",
),
),
]
@@ -1,27 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 15:43
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_mode",
field=models.CharField(
blank=True,
choices=[
("always", "All supported documents"),
("workflow_only", "Only when a workflow enables it"),
],
max_length=32,
null=True,
verbose_name="Sets which documents are sent to the remote OCR engine",
),
),
]
-55
View File
@@ -74,23 +74,6 @@ class ColorConvertChoices(models.TextChoices):
CMYK = ("CMYK", _("CMYK"))
class RemoteOCREngine(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_ENGINE
"""
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
class RemoteOCRMode(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_MODE
"""
ALWAYS = ("always", _("All supported documents"))
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
class LLMEmbeddingBackend(models.TextChoices):
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
HUGGINGFACE = ("huggingface", _("Huggingface"))
@@ -303,44 +286,6 @@ class ApplicationConfiguration(AbstractSingletonModel):
null=True,
)
"""
Settings for the remote OCR parser
"""
# PAPERLESS_REMOTE_OCR_ENGINE
remote_ocr_engine = models.CharField(
verbose_name=_("Sets the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCREngine.choices,
)
# PAPERLESS_REMOTE_OCR_API_KEY
remote_ocr_api_key = models.CharField(
verbose_name=_("Sets the remote OCR API key"),
blank=True,
null=True,
max_length=1024,
)
# PAPERLESS_REMOTE_OCR_ENDPOINT
remote_ocr_endpoint = models.CharField(
verbose_name=_("Sets the remote OCR endpoint"),
blank=True,
null=True,
max_length=256,
)
# PAPERLESS_REMOTE_OCR_MODE
remote_ocr_mode = models.CharField(
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCRMode.choices,
)
"""
AI related settings
"""
-9
View File
@@ -134,11 +134,6 @@ class ParserProtocol(Protocol):
Author or organisation name.
url : str
URL for documentation, source code, or issue tracker.
Parsers that send document content to a remote service should additionally
set ``uses_remote_service = True`` so the registry can exclude them when
remote processing has not been requested for a document. The attribute is
optional so a parser that omits it is treated as fully local.
"""
# ------------------------------------------------------------------
@@ -150,10 +145,6 @@ class ParserProtocol(Protocol):
author: str
url: str
# NOTE: uses_remote_service is not declared here, the registry reads it
# with getattr(cls, ..., False) for backwards-compatibility with existing
# parsers
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
-14
View File
@@ -334,8 +334,6 @@ class ParserRegistry:
mime_type: str,
filename: str,
path: Path | None = None,
*,
allow_remote: bool = True,
) -> type[ParserProtocol] | None:
"""Return the best parser class for the given file, or None.
@@ -361,11 +359,6 @@ class ParserRegistry:
path:
Optional filesystem path to the file. Forwarded to each
parser's score method.
allow_remote:
When False, parsers that declare ``uses_remote_service = True``
are excluded from consideration, so a document is never sent to
a remote service. Parsers that do not declare the attribute
are treated as local and are always considered.
Returns
-------
@@ -381,13 +374,6 @@ class ParserRegistry:
if mime_type not in parser_class.supported_mime_types():
continue
if not allow_remote and getattr(
parser_class,
"uses_remote_service",
False,
):
continue
score = parser_class.score(mime_type, filename, path)
if score is None:
continue
+10 -19
View File
@@ -61,18 +61,6 @@ class RemoteEngineConfig:
self.api_key = api_key
self.endpoint = endpoint
@classmethod
def from_app_config(cls) -> Self:
"""Build the config from the app config, falling back to the env."""
from paperless.config import RemoteOCRConfig
app_config = RemoteOCRConfig()
return cls(
engine=app_config.remote_ocr_engine,
api_key=app_config.remote_ocr_api_key,
endpoint=app_config.remote_ocr_endpoint,
)
def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured."""
return (
@@ -102,9 +90,6 @@ class RemoteDocumentParser:
Maintainer name.
url : str
Issue tracker / source URL.
uses_remote_service : bool
Content is sent to a remote service, True so that the registry
can skip this parser if remote processing was not requested.
"""
name: str = "Paperless-ngx Remote OCR Parser"
@@ -112,8 +97,6 @@ class RemoteDocumentParser:
author: str = "Paperless-ngx Contributors"
url: str = "https://github.com/paperless-ngx/paperless-ngx"
uses_remote_service: bool = True
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
@@ -162,7 +145,11 @@ class RemoteDocumentParser:
20 when the remote engine is configured and the MIME type is
supported, otherwise None.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
return None
if mime_type not in _SUPPORTED_MIME_TYPES:
@@ -257,7 +244,11 @@ class RemoteDocumentParser:
Whether an archive copy is wanted. For PDFs, False skips the
remote engine and uses locally-extracted text instead.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
logger.warning(
+5 -14
View File
@@ -219,13 +219,6 @@ class ApplicationConfigurationSerializer(
allow_null=True,
max_length=1024,
)
remote_ocr_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
max_length=1024,
)
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
def run_validation(self, data):
# Empty strings treated as None to avoid unexpected behavior
@@ -237,13 +230,11 @@ class ApplicationConfigurationSerializer(
data["language"] = None
if "llm_output_language" in data and data["llm_output_language"] == "":
data["llm_output_language"] = None
for field in self.OBFUSCATED_FIELDS:
if field in data and data[field] is not None:
if data[field] == "":
data[field] = None
# Not a real value, don't overwrite the stored one
elif len(data[field].replace("*", "")) == 0:
del data[field]
if "llm_api_key" in data and data["llm_api_key"] is not None:
if data["llm_api_key"] == "":
data["llm_api_key"] = None
elif len(data["llm_api_key"].replace("*", "")) == 0:
del data["llm_api_key"]
return super().run_validation(data)
def update(self, instance, validated_data):
-5
View File
@@ -1203,11 +1203,6 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
REMOTE_OCR_MODE = get_choice_from_env(
"PAPERLESS_REMOTE_OCR_MODE",
{"always", "workflow_only"},
default="always",
)
################################################################################
# AI Settings #
+2 -24
View File
@@ -114,26 +114,7 @@ def remote_parser() -> Generator[RemoteDocumentParser, None, None]:
@pytest.fixture()
def empty_remote_ocr_app_config(mocker: MockerFixture) -> MagicMock:
# empty app config without accessing db
app_config = mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
return app_config
@pytest.fixture()
def azure_settings(
settings: SettingsWrapper,
empty_remote_ocr_app_config: MagicMock,
) -> SettingsWrapper:
def azure_settings(settings: SettingsWrapper) -> SettingsWrapper:
"""Configure Django settings for a valid Azure AI OCR engine.
Sets ``REMOTE_OCR_ENGINE``, ``REMOTE_OCR_API_KEY``, and
@@ -152,10 +133,7 @@ def azure_settings(
@pytest.fixture()
def no_engine_settings(
settings: SettingsWrapper,
empty_remote_ocr_app_config: MagicMock,
) -> SettingsWrapper:
def no_engine_settings(settings: SettingsWrapper) -> SettingsWrapper:
"""Configure Django settings with no remote engine configured.
Returns
@@ -21,7 +21,6 @@ from unittest.mock import Mock
import pytest
from documents.parsers import ParseError
from paperless.models import ApplicationConfiguration
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
from paperless.parsers.remote import RemoteDocumentParser
@@ -199,21 +198,21 @@ class TestRemoteParserScore:
def test_score_returns_none_when_api_key_missing(
self,
no_engine_settings: SettingsWrapper,
settings: SettingsWrapper,
) -> None:
no_engine_settings.REMOTE_OCR_ENGINE = "azureai"
no_engine_settings.REMOTE_OCR_ENDPOINT = (
"https://test.cognitiveservices.azure.com"
)
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = None
settings.REMOTE_OCR_ENDPOINT = "https://test.cognitiveservices.azure.com"
result = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert result is None
def test_score_returns_none_when_endpoint_missing(
self,
no_engine_settings: SettingsWrapper,
settings: SettingsWrapper,
) -> None:
no_engine_settings.REMOTE_OCR_ENGINE = "azureai"
no_engine_settings.REMOTE_OCR_API_KEY = "key"
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = "key"
settings.REMOTE_OCR_ENDPOINT = None
result = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert result is None
@@ -228,24 +227,6 @@ class TestRemoteParserScore:
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert score is not None and score > 10
@pytest.mark.django_db
def test_score_uses_app_config_when_env_unset(
self,
settings: SettingsWrapper,
) -> None:
"""The app config alone is enough to activate the parser."""
settings.REMOTE_OCR_ENGINE = None
settings.REMOTE_OCR_API_KEY = None
settings.REMOTE_OCR_ENDPOINT = None
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_engine = "azureai"
config.remote_ocr_api_key = "app-config-key"
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
config.save()
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
# ---------------------------------------------------------------------------
# Properties
@@ -1277,8 +1277,6 @@ class TestParserFileTypes:
# ---------------------------------------------------------------------------
# Remote ocr config from ApplicationConfiguration needs DB access
@pytest.mark.django_db
class TestRasterisedDocumentParserRegistry:
def test_registered_in_defaults(self) -> None:
from paperless.parsers.registry import ParserRegistry
+26
View File
@@ -15,6 +15,7 @@ from paperless.checks import audit_log_check
from paperless.checks import binaries_check
from paperless.checks import check_default_language_available
from paperless.checks import check_deprecated_db_settings
from paperless.checks import check_remote_parser_configured
from paperless.checks import check_v3_minimum_upgrade_version
from paperless.checks import debug_mode_check
from paperless.checks import paths_check
@@ -630,6 +631,31 @@ class TestV3MinimumUpgradeVersionCheck:
assert check_v3_minimum_upgrade_version(None) == []
class TestRemoteParserChecks:
def test_no_engine(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
msgs = check_remote_parser_configured(None)
assert len(msgs) == 0
def test_azure_no_endpoint(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = "somekey"
settings.REMOTE_OCR_ENDPOINT = None
msgs = check_remote_parser_configured(None)
assert len(msgs) == 1
msg = msgs[0]
assert (
"Azure AI remote parser requires endpoint and API key to be configured."
in msg.msg
)
class TestTesseractChecks:
def test_default_language(self) -> None:
check_default_language_available(None)
-118
View File
@@ -468,124 +468,6 @@ class TestParserRegistryGetParserForFile:
assert result is AcceptingBuiltin
class TestParserRegistryRemoteParsers:
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
@staticmethod
def _remote_parser_cls() -> type:
class RemoteParser:
name = "remote"
version = "1.0"
author = "A"
url = "https://example.com/remote"
uses_remote_service = True
@classmethod
def supported_mime_types(cls):
return {"text/plain": ".txt"}
@classmethod
def score(cls, mime_type, filename, path=None):
return 20
return RemoteParser
def test_remote_parser_wins_when_remote_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=True.
THEN: The remote parser is returned.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=True,
)
assert result is remote_parser_cls
def test_remote_parser_skipped_when_remote_not_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: The local parser is returned despite its lower score.
"""
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
"""
GIVEN: A registry whose only candidate declares uses_remote_service.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: None is returned the remote parser is never used as a
fallback when remote processing was not requested.
"""
registry = ParserRegistry()
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is None
def test_parser_without_attribute_treated_as_local(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A third-party parser predating uses_remote_service, so it does
not declare the attribute at all.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: It is still considered, i.e. treated as fully local, rather
than raising AttributeError.
"""
assert not hasattr(dummy_parser_cls, "uses_remote_service")
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_remote_allowed_by_default(self) -> None:
"""
GIVEN: A registry containing only a remote parser.
WHEN: get_parser_for_file() is called without allow_remote.
THEN: The remote parser is returned callers that do not opt in to
the filter keep the previous behaviour.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file("text/plain", "readme.txt")
assert result is remote_parser_cls
class TestDiscover:
"""Verify entrypoint discovery in ParserRegistry.discover()."""
@@ -1,113 +0,0 @@
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from django.test import override_settings
from paperless.config import RemoteOCRConfig
from paperless.models import RemoteOCRMode
if TYPE_CHECKING:
from unittest.mock import MagicMock
@pytest.fixture()
def null_app_config(mocker) -> MagicMock:
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
return mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
@pytest.fixture()
def make_remote_ocr_config(mocker):
def _make(app_config, **django_settings_overrides):
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
with override_settings(**django_settings_overrides):
return RemoteOCRConfig()
return _make
class TestRemoteOCRConfig:
def test_falls_back_to_settings(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "env-key"
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_app_config_takes_precedence(
self,
make_remote_ocr_config,
mocker,
) -> None:
app_config = mocker.MagicMock(
remote_ocr_engine="azureai",
remote_ocr_api_key="db-key",
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
)
cfg = make_remote_ocr_config(
app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "db-key"
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_unset_everywhere(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
assert cfg.remote_ocr_engine is None
assert cfg.remote_ocr_api_key is None
assert cfg.remote_ocr_endpoint is None
class TestRemoteOCRByDefault:
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_by_default is True
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_by_default is False
-16
View File
@@ -47,22 +47,6 @@ def get_language_name(language_code: str) -> str:
return language_code
def get_llm_output_language(ai_config: AIConfig, user: User | None) -> str | None:
"""
Language to localize LLM output into: the configured language, falling back
to the user's own UI language when unset.
"""
output_language = ai_config.llm_output_language
if (
not output_language
and user is not None
and hasattr(user, "ui_settings")
and isinstance(user.ui_settings.settings, dict)
):
output_language = user.ui_settings.settings.get("language")
return output_language
def build_prompt_without_rag(
document: Document,
config: AIConfig,
+7 -17
View File
@@ -11,7 +11,7 @@ from documents.models import Correspondent
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.permissions import permitted_object_ids
from documents.permissions import get_objects_for_user_owner_aware
from documents.permissions import restrict_queryset_to_visible
MATCH_THRESHOLD = 0.8
@@ -63,40 +63,30 @@ def resolve_storage_path_ids(ids: list[int], user: User | None) -> list[StorageP
def _match_by_name(
names: list[str],
user: User | None,
user: User,
model: type[ModelT],
perm: str,
) -> list[ModelT]:
# A workflow may have no user. In that case permitted_object_ids limits
# matching to unowned objects, avoiding another user's private taxonomy.
queryset = model.objects.filter(
pk__in=permitted_object_ids(user, model, perm),
)
queryset = get_objects_for_user_owner_aware(user, [perm], model)
return _match_names_to_queryset(names, queryset)
def match_tags_by_name(names: list[str], user: User | None) -> list[Tag]:
def match_tags_by_name(names: list[str], user: User) -> list[Tag]:
return _match_by_name(names, user, Tag, "view_tag")
def match_correspondents_by_name(
names: list[str],
user: User | None,
user: User,
) -> list[Correspondent]:
return _match_by_name(names, user, Correspondent, "view_correspondent")
def match_document_types_by_name(
names: list[str],
user: User | None,
) -> list[DocumentType]:
def match_document_types_by_name(names: list[str], user: User) -> list[DocumentType]:
return _match_by_name(names, user, DocumentType, "view_documenttype")
def match_storage_paths_by_name(
names: list[str],
user: User | None,
) -> list[StoragePath]:
def match_storage_paths_by_name(names: list[str], user: User) -> list[StoragePath]:
return _match_by_name(names, user, StoragePath, "view_storagepath")
+19 -6
View File
@@ -1,4 +1,5 @@
from collections.abc import Callable
from unittest.mock import patch
import pytest
import pytest_mock
@@ -44,25 +45,33 @@ class TestAIMatching(TestCase):
self.storage_path1 = StoragePath.objects.create(name="Test Storage Path 1")
self.storage_path2 = StoragePath.objects.create(name="Test Storage Path 2")
def test_match_tags_by_name(self) -> None:
@patch("paperless_ai.matching.get_objects_for_user_owner_aware")
def test_match_tags_by_name(self, mock_get_objects) -> None:
mock_get_objects.return_value = Tag.objects.all()
names = ["Test Tag 1", "Nonexistent Tag"]
result = match_tags_by_name(names, user=None)
self.assertEqual(len(result), 1)
self.assertEqual(result[0].name, "Test Tag 1")
def test_match_correspondents_by_name(self) -> None:
@patch("paperless_ai.matching.get_objects_for_user_owner_aware")
def test_match_correspondents_by_name(self, mock_get_objects) -> None:
mock_get_objects.return_value = Correspondent.objects.all()
names = ["Test Correspondent 1", "Nonexistent Correspondent"]
result = match_correspondents_by_name(names, user=None)
self.assertEqual(len(result), 1)
self.assertEqual(result[0].name, "Test Correspondent 1")
def test_match_document_types_by_name(self) -> None:
@patch("paperless_ai.matching.get_objects_for_user_owner_aware")
def test_match_document_types_by_name(self, mock_get_objects) -> None:
mock_get_objects.return_value = DocumentType.objects.all()
names = ["Test Document Type 1", "Nonexistent Document Type"]
result = match_document_types_by_name(names, user=None)
self.assertEqual(len(result), 1)
self.assertEqual(result[0].name, "Test Document Type 1")
def test_match_storage_paths_by_name(self) -> None:
@patch("paperless_ai.matching.get_objects_for_user_owner_aware")
def test_match_storage_paths_by_name(self, mock_get_objects) -> None:
mock_get_objects.return_value = StoragePath.objects.all()
names = ["Test Storage Path 1", "Nonexistent Storage Path"]
result = match_storage_paths_by_name(names, user=None)
self.assertEqual(len(result), 1)
@@ -74,12 +83,16 @@ class TestAIMatching(TestCase):
unmatched_names = extract_unmatched_names(llm_names, matched_objects)
self.assertEqual(unmatched_names, ["Nonexistent Tag"])
def test_match_tags_by_name_with_empty_names(self) -> None:
@patch("paperless_ai.matching.get_objects_for_user_owner_aware")
def test_match_tags_by_name_with_empty_names(self, mock_get_objects) -> None:
mock_get_objects.return_value = Tag.objects.all()
names = [None, "", " "]
result = match_tags_by_name(names, user=None)
self.assertEqual(result, [])
def test_match_tags_with_fuzzy_matching(self) -> None:
@patch("paperless_ai.matching.get_objects_for_user_owner_aware")
def test_match_tags_with_fuzzy_matching(self, mock_get_objects) -> None:
mock_get_objects.return_value = Tag.objects.all()
names = ["Test Taag 1", "Teest Tag 2"]
result = match_tags_by_name(names, user=None)
self.assertEqual(len(result), 2)
-75
View File
@@ -253,81 +253,6 @@ class TestAPIMailAccounts(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["success"], True)
def test_mail_account_test_existing_no_global_perms(self) -> None:
"""
GIVEN:
- Existing account without an owner
- User without any mail account permissions
WHEN:
- API call is made to test the account by id
THEN:
- API returns forbidden
"""
account = MailAccountFactory(
username="admin",
password="secret",
imap_server="server.example.com",
imap_port=443,
owner=None,
)
user = User.objects.create_user(username="no_perms")
self.client.force_authenticate(user=user)
response = self.client.post(
f"{self.ENDPOINT}test/",
json.dumps(
{
"id": account.pk,
"imap_server": "server.example.com",
"imap_port": 443,
"imap_security": MailAccount.ImapSecurity.SSL,
"username": "admin",
"password": "******",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
self.assertEqual(response.content.decode(), "Insufficient permissions")
def test_mail_account_test_existing_object_perms_only(self) -> None:
"""
GIVEN:
- Existing account owned by another user
- User with an object level grant but no global change permission
WHEN:
- API call is made to test the account by id
THEN:
- API returns forbidden
"""
owner = User.objects.create_user(username="account_owner")
account = MailAccountFactory(
username="admin",
password="secret",
imap_server="server.example.com",
imap_port=443,
owner=owner,
)
user = User.objects.create_user(username="object_perms_only")
assign_perm("change_mailaccount", user, account)
self.client.force_authenticate(user=user)
response = self.client.post(
f"{self.ENDPOINT}test/",
json.dumps(
{
"id": account.pk,
"imap_server": "server.example.com",
"imap_port": 443,
"imap_security": MailAccount.ImapSecurity.SSL,
"username": "admin",
"password": "******",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
def test_mail_account_test_existing_nonexistent_id_forbidden(self) -> None:
response = self.client.post(
f"{self.ENDPOINT}test/",
+1 -3
View File
@@ -2195,9 +2195,7 @@ class TestMailAccountTestView(APITestCase):
password="testpassword",
)
self.user.user_permissions.add(
*Permission.objects.filter(
codename__in=["add_mailaccount", "change_mailaccount"],
),
*Permission.objects.filter(codename__in=["add_mailaccount"]),
)
self.user.save()
self.client.force_authenticate(user=self.user)
+1 -3
View File
@@ -106,9 +106,7 @@ class MailAccountViewSet(PassUserMixin, ModelViewSet[MailAccount]):
except (TypeError, ValueError, MailAccount.DoesNotExist):
return HttpResponseForbidden("Insufficient permissions")
if not request.user.has_perms(
["paperless_mail.change_mailaccount"],
) or not has_perms_owner_aware(
if not has_perms_owner_aware(
request.user,
"change_mailaccount",
existing_account,
Generated
+23 -4
View File
@@ -4,11 +4,11 @@ requires-python = ">=3.11"
resolution-markers = [
"python_full_version >= '3.15' and sys_platform == 'darwin'",
"python_full_version >= '3.15' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"(python_full_version >= '3.12' and python_full_version < '3.15' and platform_machine != 'aarch64' and platform_machine != 'x86_64' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'linux')",
"python_full_version < '3.12' and sys_platform == 'darwin'",
"python_full_version < '3.12' and sys_platform == 'linux'",
@@ -2933,6 +2933,7 @@ dependencies = [
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
{ name = "watchfiles" },
{ name = "whitenoise" },
{ name = "whoosh-compat", extra = ["tantivy"] },
{ name = "zxing-cpp" },
]
@@ -3091,6 +3092,7 @@ requires-dist = [
{ name = "torch", specifier = "~=2.13.0", index = "https://download.pytorch.org/whl/cpu" },
{ name = "watchfiles", specifier = ">=1.2" },
{ name = "whitenoise", specifier = "~=6.11" },
{ name = "whoosh-compat", extras = ["tantivy"], specifier = "==0.1.0" },
{ name = "zxing-cpp", specifier = "~=3.1.0" },
]
provides-extras = ["mariadb", "postgres", "webserver"]
@@ -5014,10 +5016,10 @@ version = "2.13.0+cpu"
source = { registry = "https://download.pytorch.org/whl/cpu" }
resolution-markers = [
"python_full_version >= '3.15' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"(python_full_version >= '3.12' and python_full_version < '3.15' and platform_machine != 'aarch64' and platform_machine != 'x86_64' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'linux')",
"python_full_version < '3.12' and sys_platform == 'linux'",
]
@@ -5652,6 +5654,23 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/db/eb/d5583a11486211f3ebd4b385545ae787f32363d453c19fffd81106c9c138/whitenoise-6.12.0-py3-none-any.whl", hash = "sha256:fc5e8c572e33ebf24795b47b6a7da8da3c00cff2349f5b04c02f28d0cc5a3cc2", size = 20302, upload-time = "2026-02-27T00:05:40.086Z" },
]
[[package]]
name = "whoosh-compat"
version = "0.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "python-dateutil" },
]
sdist = { url = "https://files.pythonhosted.org/packages/5d/f7/3e45f4a484afa174cd42e424ce2b8c514ae54f564830122656ad17b765e2/whoosh_compat-0.1.0.tar.gz", hash = "sha256:86935bdc159ed9b0a06a4661d17f1251d8280340b84e218cc73d915a2edaddf7", size = 577543, upload-time = "2026-08-25T15:22:42.316Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3f/51/9a8399d0f472814e136a2884bf910c6c59843a5a2c66afc4b5133eb531ea/whoosh_compat-0.1.0-py3-none-any.whl", hash = "sha256:3e7c5f519b4d397dbf4f8d7bbe4ca1eb6004da24c8892bc701844ed393bc97e7", size = 153875, upload-time = "2026-08-25T15:22:40.787Z" },
]
[package.optional-dependencies]
tantivy = [
{ name = "tantivy" },
]
[[package]]
name = "wrapt"
version = "2.0.1"