Compare commits

..
Author SHA1 Message Date
stumpylog 9bf63e9d06 Mark empty-pks early-return in set_permissions_for_objects as no-cover
Defensive guard for an edge case (all requested pks already gone/invalid)
rather than a path normal usage exercises; matches the existing
pragma: no cover convention elsewhere in this file.
2026-08-26 15:03:02 -07:00
stumpylog 66318f1b34 Fix: use .distinct() for existing-grant lookup, drop flaky query-count invariant tests
.distinct() lets the database dedupe identity ids server-side instead of
transferring one row per (object, grantee) match and deduping in Python --
was the dominant cost on a large selection with existing grants.

Also replaced the two query-count-equality tests (bulk_edit and the
bulk_edit_objects API path) with plain functional-correctness checks at
both batch sizes.  Hopefully stops that flake.
2026-08-26 14:48:29 -07:00
stumpylog fdb0828344 Perf: avoid unnecessary full-row fetches in batch permission assignment
set_permissions_for_objects now takes a model + pks instead of instances,
and identity filtering resolves straight to ids, so bulk-editing
permissions no longer materializes full Document/User/Group rows just to
read their pk/id. Row construction for bulk_create is also chunked to
bound peak memory for very large "apply to all" operations.
2026-08-26 14:48:06 -07:00
stumpylog 345220733a Perf: batch guardian permission assignment in bulk-edit
bulk_edit.set_permissions and BulkEditObjectPermissionsView both
looped documents/objects and called set_permissions_for_object per
object, which itself calls guardian's assign_perm/remove_perm once
per (object, user) pair -- ~10-20+ queries per object, scaling with
selection size.

Added set_permissions_for_objects, a bulk equivalent that resolves
existing permission holders once across the whole batch (not once per
object) and applies changes with a small, batch-size-independent
number of queries per action instead of one per (object, user) pair.
2026-08-26 14:47:46 -07:00
80 changed files with 1040 additions and 2840 deletions
+1 -2
View File
@@ -302,8 +302,7 @@ The following methods are supported:
- `delete`
- No `parameters` required
- `reprocess`
- Optional `parameters`: `{ "remote_ocr": true }` to send the documents to the
remote OCR engine, see [Remote OCR](usage.md#remote-ocr). Defaults to false.
- No `parameters` required
- `set_permissions`
- Requires `parameters`:
- `"set_permissions": PERMISSIONS_OBJ` (see format [above](#permissions)) and / or
-12
View File
@@ -2066,18 +2066,6 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to None.
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
: Which documents are sent to the remote OCR engine.
- `always`: every document of a supported file type is sent to the remote
engine, bypassing the local OCR engine.
- `workflow_only`: documents are processed locally unless a workflow
explicitly enables remote OCR for them, letting you use the remote engine
selectively.
Defaults to "always".
## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
-14
View File
@@ -456,20 +456,6 @@ def score(
return 10
```
**Remote services**
If your parser sends document content to a remote service, declare it:
```python
class MyCustomParser:
uses_remote_service = True
```
Paperless-ngx excludes such parsers when the document being consumed has not
been marked for remote processing, so users can keep remote OCR off by default
and enable it selectively with a workflow. Parsers that do not declare the
attribute are treated as fully local and are always considered.
**Archive and rendition flags**
```python
+1 -22
View File
@@ -654,19 +654,6 @@ happened while it was still encrypted, that original version will likewise be mi
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
multiple PDF types with different passwords, create separate workflows for each use case.
##### Remote OCR {#workflow-action-remote-ocr}
"Remote OCR" actions send the document to the configured remote OCR engine instead of processing it
locally. To use remote OCR selectively, set the [remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE)
to `workflow_only` then add this action to a workflow that matches only the documents you
want sent to the remote engine. See [Remote OCR](#remote-ocr) for the engine setup. The action only works with
a **Consumption Started** trigger.
The action takes no options, its presence is what enables remote OCR for a matching document.
If the remote engine is not configured, or does not support the document's file type, the document is
processed locally instead and a warning is written to the log.
#### Workflow placeholders
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -1103,19 +1090,11 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
settings can be supplied as environment variables or via **Application Configuration**.
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
or page limitations (e.g. with a free tier).
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
unless a [remote OCR workflow action](#workflow-action-remote-ocr) enables it for them, so you can limit the remote
engine to particular documents.
Setting the mode to `workflow_only` also allows the **Reprocess** actions to selectively use remote OCR for individual documents.
## Architecture
Paperless-ngx consists of the following components:
+279 -360
View File
File diff suppressed because it is too large Load Diff
@@ -14,48 +14,43 @@
<a ngbNavLink>{{category}}</a>
<ng-template ngbNavContent>
<div class="p-3">
@for (section of getCategorySections(category); track section) {
@if (section) {
<h5 class="mt-4 mb-3">{{section}}</h5>
}
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category, section); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
<div class="card-title d-flex align-items-center">
<h6 class="mb-0">
{{option.title}}
</h6>
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs name="info-circle"></i-bs>
</a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
<div class="card-title d-flex align-items-center">
<h6 class="mb-0">
{{option.title}}
</h6>
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs name="info-circle"></i-bs>
</a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
}
</div>
</div>
}
</div>
}
</div>
}
</div>
</div>
</ng-template>
</li>
@@ -8,11 +8,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgSelectModule } from '@ng-select/ng-select'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of, throwError } from 'rxjs'
import {
ConfigCategory,
ConfigSection,
OutputTypeConfig,
} from 'src/app/data/paperless-config'
import { OutputTypeConfig } from 'src/app/data/paperless-config'
import { ConfigService } from 'src/app/services/config.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -162,24 +158,4 @@ describe('ConfigComponent', () => {
component.resetOption('barcodes_enabled')
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
})
it('should group options into sections within a category, or not', () => {
const sections = component.getCategorySections(ConfigCategory.OCR)
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
expect(
component
.getCategoryOptions(ConfigCategory.OCR)
.map((option) => option.key)
).toContain('output_type')
expect(
component
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
.map((option) => option.key)
).toEqual([
'remote_ocr_engine',
'remote_ocr_api_key',
'remote_ocr_endpoint',
'remote_ocr_mode',
])
})
})
@@ -74,20 +74,8 @@ export class ConfigComponent
return Object.values(ConfigCategory)
}
getCategorySections(category: string): string[] {
return [
...new Set(
PaperlessConfigOptions.filter((o) => o.category === category).map(
(o) => o.section ?? null // null means no section
)
),
]
}
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
return PaperlessConfigOptions.filter(
(o) => o.category === category && (o.section ?? null) === section
)
getCategoryOptions(category: string): ConfigOption[] {
return PaperlessConfigOptions.filter((o) => o.category === category)
}
initialConfig: PaperlessConfig
@@ -2,7 +2,7 @@
<button class="navbar-toggler d-md-none collapsed border-0" type="button" data-toggle="collapse"
data-target="#sidebarMenu" aria-controls="sidebarMenu" aria-expanded="false" aria-label="Toggle navigation"
(click)="closeMobileSearch(); toggleMenuCollapsed()">
<i-bs width="1.5em" height="1.5em" name="list"></i-bs>
<span class="navbar-toggler-icon"></span>
</button>
<a class="navbar-brand d-flex align-items-center me-0 ps-md-3 py-0 order-sm-0"
[ngClass]="{ 'slim': slimSidebarEnabled, '' : !slimSidebarEnabled }"
@@ -10,12 +10,16 @@
tourAnchor="tour.intro">
@if (!hasCustomBranding) {
<pngx-logo extra_classes="navbar-official-logo px-1" height="2.4rem"></pngx-logo>
<pngx-brand-mark class="brand-mark brand-mark-slim d-none"></pngx-brand-mark>
<svg class="brand-mark brand-mark-slim d-none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
} @else {
@if (customAppLogo) {
<img class="brand-logo" [src]="customAppLogo" alt="" />
} @else {
<pngx-brand-mark class="brand-mark"></pngx-brand-mark>
<svg class="brand-mark" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
}
<div class="brand-copy ms-2 text-truncate" [class.d-md-none]="slimSidebarEnabled">
<span class="brand-title text-truncate">{{ appTitle }}</span>
@@ -25,9 +29,9 @@
</div>
}
</a>
<div class="search-container flex-grow-1 py-2 pb-3 pb-sm-2 me-sm-auto order-3 order-sm-1"
<div class="search-container flex-grow-1 py-2 pb-3 pb-sm-2 px-3 ps-md-3 me-sm-auto order-3 order-sm-1"
[class.mobile-hidden]="mobileSearchHidden()">
<div class="col-12 header-search mx-auto">
<div class="col-12 header-search">
<pngx-global-search></pngx-global-search>
</div>
</div>
@@ -86,7 +90,7 @@
<ul class="nav flex-column">
<li class="nav-item app-link">
<a class="nav-link" routerLink="dashboard" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Dashboard" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Dashboard" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="house"></i-bs><span><ng-container i18n>Dashboard</ng-container></span>
</a>
@@ -95,7 +99,7 @@
<a class="nav-link" routerLink="documents" routerLinkActive="active"
[routerLinkActiveOptions]="{ paths: 'exact', queryParams: 'ignored', matrixParams: 'ignored', fragment: 'ignored' }"
(click)="closeMenu()"
ngbPopover="Documents" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Documents" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="files"></i-bs><span><ng-container i18n>Documents</ng-container></span>
</a>
@@ -114,7 +118,7 @@
(cdkDragEnded)="onDragEnd($event)">
<a class="nav-link" routerLink="view/{{view.id}}"
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="view.name"
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
popoverClass="popover-slim">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
@if (showSidebarCounts && !slimSidebarEnabled) {
@@ -152,7 +156,7 @@
<li class="nav-item w-100 app-link">
<a class="nav-link app-link" [class.text-truncate]="!slimSidebarEnabled" routerLink="documents/{{d.id}}"
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="d.title | documentTitle"
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
popoverClass="popover-slim">
<i-bs class="me-2" name="file-text"></i-bs><span>{{d.title | documentTitle}}</span>
<span class="close flex-column justify-content-center"
@@ -166,7 +170,7 @@
<li class="nav-item w-100 app-link">
<button type="button" class="nav-link nav-link-action app-link w-100 text-start"
[class.text-truncate]="!slimSidebarEnabled" (click)="closeAll()"
ngbPopover="Close all" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Close all" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="x"></i-bs><span><ng-container i18n>Close all</ng-container></span>
</button>
@@ -185,7 +189,7 @@
<div class="d-flex align-items-center attributes-row">
<a class="nav-link flex-fill" routerLink="attributes" routerLinkActive="active"
[routerLinkActiveOptions]="{ exact: !(slimSidebarEnabled || attributesSectionsCollapsed) }" (click)="closeMenu()"
ngbPopover="Attributes" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Attributes" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs name="stack"></i-bs><span class="ms-2"><ng-container i18n>Attributes</ng-container></span>
</a>
@@ -237,7 +241,7 @@
}
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.SavedView }">
<a class="nav-link" routerLink="savedviews" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Saved Views" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Saved Views" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="window-stack"></i-bs><span><ng-container i18n>Saved Views</ng-container></span>
</a>
@@ -246,7 +250,7 @@
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Workflow }"
tourAnchor="tour.workflows">
<a class="nav-link" routerLink="workflows" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Workflows" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Workflows" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="boxes"></i-bs><span><ng-container i18n>Workflows</ng-container></span>
</a>
@@ -254,14 +258,14 @@
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.MailAccount }"
tourAnchor="tour.mail">
<a class="nav-link" routerLink="mail" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Mail"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="envelope"></i-bs><span><ng-container i18n>Mail</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.Document }">
<a class="nav-link" routerLink="trash" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Trash"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="trash"></i-bs><span><ng-container i18n>Trash</ng-container></span>
</a>
@@ -277,21 +281,21 @@
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.UISettings }"
tourAnchor="tour.settings">
<a class="nav-link" routerLink="settings" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Settings" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Settings" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="gear"></i-bs><span><ng-container i18n>Settings</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.AppConfig }">
<a class="nav-link" routerLink="config" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Configuration" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Configuration" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="sliders2-vertical"></i-bs><span><ng-container i18n>Configuration</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.User }">
<a class="nav-link" routerLink="usersgroups" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Users & Groups" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Users & Groups" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="people"></i-bs><span><ng-container i18n>Users & Groups</ng-container></span>
</a>
@@ -300,7 +304,7 @@
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.PaperlessTask }"
tourAnchor="tour.file-tasks">
<a class="nav-link" routerLink="tasks" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="Tasks" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
ngbPopover="Tasks" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="list-task"></i-bs><span><ng-container i18n>Tasks</ng-container>@if (tasksService.needsAttentionTasks.length > 0) {
<span><span class="badge bg-danger ms-2 d-inline">{{tasksService.needsAttentionTasks.length}}</span></span>
@@ -313,26 +317,26 @@
@if (permissionsService.isAdmin()) {
<li class="nav-item app-link">
<a class="nav-link" routerLink="logs" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Logs"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="text-left"></i-bs><span><ng-container i18n>Logs</ng-container></span>
</a>
</li>
}
<li class="nav-item mt-2" tourAnchor="tour.outro">
<a class="text-muted small d-flex align-items-center flex-wrap text-decoration-none nav-anchor"
<a class="px-3 py-2 text-muted small d-flex align-items-center flex-wrap text-decoration-none"
target="_blank" rel="noopener noreferrer" href="https://docs.paperless-ngx.com" ngbPopover="Documentation"
i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="d-flex me-2" name="question-circle"></i-bs><span><ng-container i18n>Documentation</ng-container></span>
</a>
</li>
<li class="nav-item" [class.visually-hidden]="slimSidebarEnabled">
<div class="text-muted small d-flex align-items-center flex-wrap nav-label">
<div class="px-3 py-0 text-muted small d-flex align-items-center flex-wrap">
<div class="me-3">
<a class="text-muted text-decoration-none" target="_blank" rel="noopener noreferrer"
href="https://github.com/paperless-ngx/paperless-ngx" ngbPopover="GitHub" i18n-ngbPopover
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body"
triggers="mouseenter:mouseleave" popoverClass="popover-slim">
{{ versionString }}
</a>
@@ -67,23 +67,6 @@
main.mobile-search-hidden {
padding-top: 56px;
}
.navbar-toggler {
padding-left: calc(12px - (1.5em * 2.5 / 16));
}
}
.search-container {
padding-left: 1rem;
padding-right: 1rem;
}
// Below sm the search gets its own full-width row, so line it up with main's content edge
@media (max-width: 575.98px) {
.search-container {
padding-left: 12px;
padding-right: 12px;
}
}
main {
@@ -114,7 +97,7 @@ main {
li.nav-item span.badge {
display: inline-block;
margin-right: -4px;
margin-right: 2px;
}
}
@@ -158,11 +141,18 @@ main {
}
}
.sidebar.slim {
li.nav-item span.badge {
display: inline-block;
margin-right: 2px;
}
}
.sidebar-slim-toggler {
display: block;
position: fixed;
left: calc(var(--pngx-sidebar-width) - 12px);
bottom: 16px;
top: 72px;
z-index: 996;
--bs-btn-padding-x: 0.35rem;
--bs-btn-padding-y: 0.125rem;
@@ -237,10 +227,6 @@ main {
}
}
.sidebar .nav-anchor, .sidebar .nav-label {
padding: .25rem .7rem;
}
.attributes-row {
border-radius: .55rem;
margin: .1rem 0;
@@ -417,7 +403,7 @@ main {
:host ::ng-deep .navbar-official-logo {
.leaf {
fill: color-mix(in srgb, var(--pngx-primary-text-contrast) 85%, var(--bs-primary)) !important;
fill: color-mix(in srgb, var(--pngx-primary-text-contrast) 70%, var(--bs-primary)) !important;
}
.text {
@@ -458,7 +444,7 @@ main {
.header-search {
width: 100%;
max-width: 55rem;
max-width: 44rem;
}
@media screen and (max-width: 575.98px) {
@@ -472,7 +458,6 @@ main {
.navbar-toggler {
grid-area: toggler;
color: var(--pngx-primary-text-contrast);
text-align: left;
}
@@ -45,7 +45,6 @@ import { TasksService } from 'src/app/services/tasks.service'
import { ToastService } from 'src/app/services/toast.service'
import { environment } from 'src/environments/environment'
import { ChatComponent } from '../chat/chat/chat.component'
import { BrandMarkComponent } from '../common/logo/brand-mark/brand-mark.component'
import { LogoComponent } from '../common/logo/logo.component'
import { ProfileEditDialogComponent } from '../common/profile-edit-dialog/profile-edit-dialog.component'
import { DocumentDetailComponent } from '../document-detail/document-detail.component'
@@ -62,7 +61,6 @@ const SCROLL_THRESHOLD = 16
imports: [
GlobalSearchComponent,
LogoComponent,
BrandMarkComponent,
DocumentTitlePipe,
IfPermissionsDirective,
ToastsDropdownComponent,
@@ -280,10 +278,6 @@ export class AppFrameComponent
})
}
get slimSidebarPopoversEnabled(): boolean {
return this.slimSidebarEnabled && !this.isMobileViewport()
}
get attributesSectionsCollapsed(): boolean {
this.settingsService.trackChanges()
return this.settingsService
@@ -1,28 +0,0 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (messageBold) {
<p class="text-break"><b>{{messageBold}}</b></p>
}
@if (message) {
<p class="mb-0 text-break" [innerHTML]="message"></p>
}
@if (showRemoteOcr) {
<div class="form-check mt-3">
<input class="form-check-input" type="checkbox" id="reprocessRemoteOcr" [(ngModel)]="remoteOcr" />
<label class="form-check-label" for="reprocessRemoteOcr" i18n>Use remote OCR</label>
<div class="form-text" i18n>Sends the document to the configured remote OCR service, which may incur costs.</div>
</div>
}
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
{{btnCaption}}
</button>
</div>
@@ -1,72 +0,0 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { RemoteOCRModeConfig } from 'src/app/data/paperless-config'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { SettingsService } from 'src/app/services/settings.service'
import { ReprocessConfirmDialogComponent } from './reprocess-confirm-dialog.component'
describe('ReprocessConfirmDialogComponent', () => {
let component: ReprocessConfirmDialogComponent
let fixture: ComponentFixture<ReprocessConfirmDialogComponent>
let settingsService: SettingsService
const createComponent = (configured: boolean, mode: string) => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, configured)
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_MODE, mode)
fixture = TestBed.createComponent(ReprocessConfirmDialogComponent)
component = fixture.componentInstance
fixture.detectChanges()
}
beforeEach(async () => {
TestBed.configureTestingModule({
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
imports: [ReprocessConfirmDialogComponent],
}).compileComponents()
settingsService = TestBed.inject(SettingsService)
})
it('should not request remote OCR by default', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.remoteOcr).toBeFalsy()
})
it('should not offer remote OCR when no engine is configured', () => {
createComponent(false, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should not offer remote OCR when it already handles every document', () => {
createComponent(true, RemoteOCRModeConfig.ALWAYS)
expect(component.showRemoteOcr).toBeFalsy()
expect(
fixture.nativeElement.querySelector('#reprocessRemoteOcr')
).toBeNull()
})
it('should offer remote OCR when configured and selective', () => {
createComponent(true, RemoteOCRModeConfig.WORKFLOW_ONLY)
expect(component.showRemoteOcr).toBeTruthy()
const checkbox = fixture.nativeElement.querySelector('#reprocessRemoteOcr')
expect(checkbox).not.toBeNull()
checkbox.click()
fixture.detectChanges()
expect(component.remoteOcr).toBeTruthy()
})
})
@@ -1,20 +0,0 @@
import { Component, inject } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { SettingsService } from 'src/app/services/settings.service'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-reprocess-confirm-dialog',
templateUrl: './reprocess-confirm-dialog.component.html',
imports: [FormsModule],
})
export class ReprocessConfirmDialogComponent extends ConfirmDialogComponent {
private settings = inject(SettingsService)
remoteOcr: boolean = false
public get showRemoteOcr(): boolean {
// Hidden when it is not configured, or when it already handles every document anyway.
return this.settings.remoteOCRIsSelectable
}
}
@@ -455,13 +455,6 @@
</div>
</div>
}
@case (WorkflowActionType.RemoteOcr) {
<div class="row">
<div class="col">
<p class="text-muted small" i18n>The document will be sent to the configured remote OCR service. May incur costs.</p>
</div>
</div>
}
}
</div>
</ng-template>
@@ -29,7 +29,6 @@ import {
DocumentSource,
WorkflowTriggerType,
} from 'src/app/data/workflow-trigger'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
@@ -225,12 +224,7 @@ describe('WorkflowEditDialogComponent', () => {
).toEqual('Document Added')
expect(component.getTriggerTypeOptionName(null)).toEqual('')
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
// Remote OCR is absent until the workflow has a consumption trigger
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.RemoteOcr
)
)
expect(component.actionTypeOptions).toEqual(WORKFLOW_ACTION_OPTIONS)
expect(
component.getActionTypeOptionName(WorkflowActionType.Assignment)
).toEqual('Assignment')
@@ -243,104 +237,7 @@ describe('WorkflowEditDialogComponent', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(false)
component.ngOnInit()
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
(a) =>
a.id !== WorkflowActionType.Email &&
a.id !== WorkflowActionType.RemoteOcr
)
)
})
it('should offer remote OCR only for consumption workflows', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// A consumption trigger makes the action reachable
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Any other trigger type runs after the document has been parsed
component.object = {
name: 'Workflow 2',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should offer remote OCR on a trigger added to a new workflow', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
component.ngOnInit()
// Nothing for the action to apply to yet
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
// addTrigger creates the form field with emitEvent false, so the options
// have to be computed on read rather than cached from valueChanges
component.addTrigger()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
// Switching that trigger to a type that runs after parsing removes it
component.triggerFields
.at(0)
.get('type')
.setValue(WorkflowTriggerType.DocumentAdded)
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
)
})
it('should keep remote OCR listed when an action already uses it', () => {
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.DocumentAdded }],
actions: [{ type: WorkflowActionType.RemoteOcr }],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).toContain(
WorkflowActionType.RemoteOcr
)
})
it('should not offer remote OCR when no engine is configured', () => {
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
component.object = {
name: 'Workflow 1',
order: 0,
enabled: true,
triggers: [{ type: WorkflowTriggerType.Consumption }],
actions: [],
} as Workflow
component.ngOnInit()
expect(component.actionTypeOptions.map((a) => a.id)).not.toContain(
WorkflowActionType.RemoteOcr
WORKFLOW_ACTION_OPTIONS.filter((a) => a.id !== WorkflowActionType.Email)
)
})
@@ -148,10 +148,6 @@ export const WORKFLOW_ACTION_OPTIONS = [
id: WorkflowActionType.MoveToTrash,
name: $localize`Move to trash`,
},
{
id: WorkflowActionType.RemoteOcr,
name: $localize`Remote OCR`,
},
]
export enum TriggerFilterType {
@@ -508,6 +504,8 @@ export class WorkflowEditDialogComponent
expandedItem: number = null
readonly allowedActionTypes = signal([])
private readonly triggerFilterOptionsMap = new WeakMap<
FormArray,
TriggerFilterOption[]
@@ -550,40 +548,13 @@ export class WorkflowEditDialogComponent
this.checkRemovalActionFields.bind(this)
)
this.checkRemovalActionFields(this.objectForm.value)
}
private allowedActionTypes: typeof WORKFLOW_ACTION_OPTIONS = null
private getAllowedActionTypes() {
let allowed = WORKFLOW_ACTION_OPTIONS
if (!this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.Email)
}
// Remote OCR is decided before the document is parsed, so it is only
// offered for workflows that run at consumption.
const formWorkflow: Workflow = this.objectForm?.value
const remoteOcrUsable =
this.settingsService.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
(formWorkflow?.triggers?.some(
(trigger) => trigger.type === WorkflowTriggerType.Consumption
) ||
formWorkflow?.actions?.some(
(action) => action.type === WorkflowActionType.RemoteOcr
))
if (!remoteOcrUsable) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.RemoteOcr)
}
if (
this.allowedActionTypes?.length === allowed.length &&
this.allowedActionTypes.every((a, i) => a.id === allowed[i].id)
) {
return this.allowedActionTypes
}
this.allowedActionTypes = allowed
return allowed
this.allowedActionTypes.set(
this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)
? WORKFLOW_ACTION_OPTIONS
: WORKFLOW_ACTION_OPTIONS.filter(
(a) => a.id !== WorkflowActionType.Email
)
)
}
private checkRemovalActionFields(formWorkflow: Workflow) {
@@ -1308,8 +1279,7 @@ export class WorkflowEditDialogComponent
get actionTypeOptions() {
this.settingsService.trackChanges()
// Computed on read rather than cached
return this.getAllowedActionTypes()
return this.allowedActionTypes()
}
getActionTypeOptionName(type: WorkflowActionType): string {
@@ -1,3 +0,0 @@
<svg class="w-100 h-100 d-block" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" fill="currentColor" aria-hidden="true">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 545 B

@@ -1,14 +0,0 @@
import { Component, input } from '@angular/core'
@Component({
selector: 'pngx-brand-mark',
templateUrl: './brand-mark.component.html',
host: {
'[style.width]': 'width()',
'[style.height]': 'height()',
},
})
export class BrandMarkComponent {
readonly width = input<string>(null)
readonly height = input<string>(null)
}
@@ -1,6 +1,6 @@
<div class="row pt-3 pb-2 align-items-center">
<div class="row pt-3 pb-3 pb-md-2 align-items-center">
<div class="col-md text-truncate">
<h3 class="d-flex align-items-center mb-2 mb-md-1" style="line-height: 1.4">
<h3 class="d-flex align-items-center mb-1" style="line-height: 1.4">
<span class="text-truncate">{{title()}}</span>
@if (id()) {
<span class="badge bg-primary text-primary-text-contrast ms-3 small fs-normal cursor-pointer" (click)="copyID()">
@@ -12,10 +12,10 @@
</span>
}
@if (subTitle()) {
<span class="page-subtitle h6 mb-0 mt-1 fw-normal ms-md-3 text-truncate" style="line-height: 1.4">{{subTitle()}}</span>
<span class="h6 mb-0 mt-1 d-block d-md-inline fw-normal ms-md-3 text-truncate" style="line-height: 1.4">{{subTitle()}}</span>
}
@if (info()) {
<button class="btn btn-sm btn-link text-muted p-0 ms-2 p-md-2 ms-md-0" title="What's this?" i18n-title type="button" [ngbPopover]="infoPopover" [autoClose]="true">
<button class="btn btn-sm btn-link text-muted p-0 p-md-2" title="What's this?" i18n-title type="button" [ngbPopover]="infoPopover" [autoClose]="true">
<i-bs name="question-circle"></i-bs>
</button>
<ng-template #infoPopover>
@@ -1,6 +1,5 @@
h3 {
min-height: calc(1.325rem + 0.9vw);
flex-wrap: wrap;
.badge {
font-size: 0.65rem;
@@ -8,16 +7,6 @@ h3 {
}
}
// Drop the subtitle onto its own and shrink it
@media (max-width: 767.98px) {
h3 > .page-subtitle {
flex: 0 0 100%;
margin-top: 0 !important;
font-size: .7rem;
line-height: 1.3 !important;
}
}
:host {
display: block;
margin-bottom: .35rem;
@@ -1,18 +1,10 @@
<ngb-alert class="welcome-widget bg-light shadow-sm pe-3 position-relative overflow-hidden" type="primary" [dismissible]="true" (closed)="dismiss.emit(true)">
<pngx-brand-mark class="welcome-watermark d-none d-md-block position-absolute top-50 translate-middle-y text-primary pe-none" width="26rem" height="26rem"></pngx-brand-mark>
<div class="d-flex align-items-start gap-3 gap-md-4 position-relative">
<pngx-brand-mark class="d-none d-sm-block flex-shrink-0 text-primary" width="3rem" height="3rem"></pngx-brand-mark>
<div class="flex-grow-1">
<h4 class="alert-heading text-primary fw-semibold"><ng-container i18n>Paperless-ngx is running!</ng-container> 🎉</h4>
<div class="welcome-copy text-dark">
<p i18n>You're ready to start uploading documents! Explore the various features of this web app on your own, or start a quick tour using the button below.</p>
<p class="mb-0" i18n>More detail on how to use and configure Paperless-ngx is always available in the <a href="https://docs.paperless-ngx.com" target="_blank">documentation</a>.</p>
</div>
<hr class="opacity-100">
<div class="d-flex flex-column flex-sm-row align-items-sm-center gap-3">
<p class="m-0 text-secondary"><em i18n>Thanks for being a part of the Paperless-ngx community!</em></p>
<button class="btn btn-primary ms-sm-auto flex-shrink-0 align-self-start align-self-sm-auto" (click)="tourService.start()"><ng-container i18n>Start the tour</ng-container> &rarr;</button>
</div>
</div>
<ngb-alert class="pe-3" type="primary" [dismissible]="true" (closed)="dismiss.emit(true)">
<h4 class="alert-heading"><ng-container i18n>Paperless-ngx is running!</ng-container> 🎉</h4>
<p i18n>You're ready to start uploading documents! Explore the various features of this web app on your own, or start a quick tour using the button below.</p>
<p i18n>More detail on how to use and configure Paperless-ngx is always available in the <a href="https://docs.paperless-ngx.com" target="_blank">documentation</a>.</p>
<hr>
<div class="d-flex align-items-end">
<p class="lead fs-6 m-0"><em i18n>Thanks for being a part of the Paperless-ngx community!</em></p>
<button class="btn btn-primary ms-auto flex-shrink-0" (click)="tourService.start()"><ng-container i18n>Start the tour</ng-container> &rarr;</button>
</div>
</ngb-alert>
@@ -1,16 +0,0 @@
.welcome-widget {
border-color: color-mix(in srgb, var(--bs-primary) 50%, transparent);
hr {
border-top-color: color-mix(in srgb, var(--bs-primary) 25%, transparent);
}
}
.welcome-copy {
max-width: 68ch;
}
.welcome-watermark {
right: -6rem;
opacity: .07;
}
@@ -1,13 +1,12 @@
import { Component, EventEmitter, Output, inject } from '@angular/core'
import { NgbAlertModule } from '@ng-bootstrap/ng-bootstrap'
import { TourService } from 'ngx-ui-tour-ng-bootstrap'
import { BrandMarkComponent } from '../../../common/logo/brand-mark/brand-mark.component'
@Component({
selector: 'pngx-welcome-widget',
templateUrl: './welcome-widget.component.html',
styleUrls: ['./welcome-widget.component.scss'],
imports: [NgbAlertModule, BrandMarkComponent],
imports: [NgbAlertModule],
})
export class WelcomeWidgetComponent {
readonly tourService = inject(TourService)
@@ -963,24 +963,12 @@ describe('DocumentDetailComponent', () => {
component.reprocess()
const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, false)
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] })
expect(modalSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should pass remote OCR choice when reprocessing', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
reprocessSpy.mockReturnValue(of(true))
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
component.reprocess()
openModal.componentInstance.remoteOcr = true
openModal.componentInstance.confirmClicked.next()
expect(reprocessSpy).toHaveBeenCalledWith({ documents: [doc.id] }, true)
})
it('should show error if redo ocr call fails', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
@@ -97,7 +97,6 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
import * as UTIF from 'utif'
import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
@@ -1399,7 +1398,7 @@ export class DocumentDetailComponent
}
reprocess() {
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Reprocess confirm`
@@ -1410,10 +1409,7 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false)
this.documentsService
.reprocessDocuments(
{ documents: [this.document().id] },
modal.componentInstance.remoteOcr
)
.reprocessDocuments({ documents: [this.document().id] })
.subscribe({
next: () => {
this.toastService.showInfo(
@@ -1122,7 +1122,6 @@ describe('BulkEditorComponent', () => {
req.flush(true)
expect(req.request.body).toEqual({
documents: [3, 4],
remote_ocr: false,
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
@@ -52,7 +52,6 @@ import { flattenTags } from 'src/app/utils/flatten-tags'
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
import { MergeAsVersionsConfirmDialogComponent } from '../../common/confirm-dialog/merge-as-versions-confirm-dialog/merge-as-versions-confirm-dialog.component'
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
@@ -917,7 +916,7 @@ export class BulkEditorComponent
}
reprocessSelected() {
let modal = this.modalService.open(ReprocessConfirmDialogComponent, {
let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Reprocess confirm`
@@ -931,10 +930,7 @@ export class BulkEditorComponent
modal.componentInstance.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.reprocessDocuments(
this.getSelectionQuery(),
modal.componentInstance.remoteOcr
)
this.documentService.reprocessDocuments(this.getSelectionQuery())
)
})
}
-55
View File
@@ -54,10 +54,6 @@ export const ConfigCategory = {
AI: $localize`AI Settings`,
}
export const ConfigSection = {
RemoteOCR: $localize`Remote OCR`,
}
export const LLMEmbeddingBackendConfig = {
OPENAI_LIKE: 'openai-like',
HUGGINGFACE: 'huggingface',
@@ -69,15 +65,6 @@ export const LLMBackendConfig = {
OLLAMA: 'ollama',
}
export const RemoteOCREngineConfig = {
AZURE_AI: 'azureai',
}
export const RemoteOCRModeConfig = {
ALWAYS: 'always',
WORKFLOW_ONLY: 'workflow_only',
}
export interface ConfigOption {
key: string
title: string
@@ -85,7 +72,6 @@ export interface ConfigOption {
choices?: Array<{ id: string; name: string }>
config_key?: string
category: string
section?: string
note?: string
}
@@ -195,43 +181,6 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_OCR_USER_ARGS',
category: ConfigCategory.OCR,
},
{
key: 'remote_ocr_engine',
title: $localize`Remote OCR Engine`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCREngineConfig),
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
},
{
key: 'remote_ocr_api_key',
title: $localize`Remote OCR API Key`,
type: ConfigOptionType.Password,
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
},
{
key: 'remote_ocr_endpoint',
title: $localize`Remote OCR Endpoint`,
type: ConfigOptionType.String,
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Required when using the Azure AI engine.`,
},
{
key: 'remote_ocr_mode',
title: $localize`Remote OCR Mode`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCRModeConfig),
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
},
{
key: 'app_logo',
title: $localize`Application Logo`,
@@ -449,10 +398,6 @@ export interface PaperlessConfig extends ObjectWithId {
barcode_enable_tag: boolean
barcode_tag_mapping: object
barcode_tag_split: boolean
remote_ocr_engine: string
remote_ocr_api_key: string
remote_ocr_endpoint: string
remote_ocr_mode: string
ai_enabled: boolean
llm_embedding_backend: string
llm_embedding_model: string
-13
View File
@@ -1,6 +1,5 @@
import { PdfEditorEditMode } from '../components/common/pdf-editor/pdf-editor-edit-mode'
import { PdfZoomScale } from '../components/common/pdf-viewer/pdf-viewer.types'
import { RemoteOCRModeConfig } from './paperless-config'
import { User } from './user'
export interface UiSettings {
@@ -95,8 +94,6 @@ export const SETTINGS_KEYS = {
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
EMAIL_ENABLED: 'email_enabled',
AI_ENABLED: 'ai_enabled',
REMOTE_OCR_CONFIGURED: 'remote_ocr:configured',
REMOTE_OCR_MODE: 'remote_ocr:mode',
}
export const SETTINGS: UiSetting[] = [
@@ -350,14 +347,4 @@ export const SETTINGS: UiSetting[] = [
type: 'string',
default: PdfEditorEditMode.Create,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_CONFIGURED,
type: 'boolean',
default: false,
},
{
key: SETTINGS_KEYS.REMOTE_OCR_MODE,
type: 'string',
default: RemoteOCRModeConfig.ALWAYS,
},
]
-1
View File
@@ -7,7 +7,6 @@ export enum WorkflowActionType {
Webhook = 4,
PasswordRemoval = 5,
MoveToTrash = 6,
RemoteOcr = 7,
}
export interface WorkflowActionEmail extends ObjectWithId {
@@ -284,21 +284,6 @@ describe(`DocumentService`, () => {
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: false,
})
})
it('should request remote OCR when reprocessing with it enabled', () => {
const ids = [1, 2, 3]
subscription = service
.reprocessDocuments({ documents: ids }, true)
.subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/reprocess/`
)
expect(req.request.body).toEqual({
documents: ids,
remote_ocr: true,
})
})
@@ -349,13 +349,9 @@ export class DocumentService extends AbstractPaperlessService<Document> {
})
}
reprocessDocuments(
selection: DocumentSelectionQuery,
remoteOcr: boolean = false
) {
reprocessDocuments(selection: DocumentSelectionQuery) {
return this.http.post(this.getResourceUrl(null, 'reprocess'), {
...selection,
remote_ocr: remoteOcr,
})
}
@@ -13,7 +13,6 @@ import { environment } from 'src/environments/environment'
import { CustomFieldDataType } from '../data/custom-field'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { SavedView } from '../data/saved-view'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SETTINGS_KEYS, UiSettings } from '../data/ui-settings'
import { PermissionsService } from './permissions.service'
import { CustomFieldsService } from './rest/custom-fields.service'
@@ -435,26 +434,4 @@ describe('SettingsService', () => {
).name
).toEqual(customFields[0].name)
})
it('should offer remote OCR only when configured and selective', () => {
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, false)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
// configured, but already handling every document
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, true)
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.ALWAYS
)
expect(settingsService.remoteOCRIsSelectable).toBeFalsy()
settingsService.set(
SETTINGS_KEYS.REMOTE_OCR_MODE,
RemoteOCRModeConfig.WORKFLOW_ONLY
)
expect(settingsService.remoteOCRIsSelectable).toBeTruthy()
})
})
@@ -19,7 +19,6 @@ import {
} from 'src/app/utils/color'
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { RemoteOCRModeConfig } from '../data/paperless-config'
import { SavedView } from '../data/saved-view'
import {
PAPERLESS_GREEN_HEX,
@@ -688,17 +687,6 @@ export class SettingsService {
return this.settingIsSet(SETTINGS_KEYS.UPDATE_CHECKING_ENABLED)
}
/**
* Offering remote OCR as a choice only makes sense when an engine
* is configured but is not already handling every document.
*/
get remoteOCRIsSelectable(): boolean {
return (
this.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
this.get(SETTINGS_KEYS.REMOTE_OCR_MODE) !== RemoteOCRModeConfig.ALWAYS
)
}
offerTour(): boolean {
return this.dashboardIsEmpty() && !this.get(SETTINGS_KEYS.TOUR_COMPLETE)
}
-2
View File
@@ -118,7 +118,6 @@ import {
journalBookmarkFill,
journals,
link,
list,
listNested,
listTask,
listUl,
@@ -366,7 +365,6 @@ const icons = {
journalBookmarkFill,
journals,
link,
list,
listNested,
listTask,
listUl,
-1
View File
@@ -536,7 +536,6 @@ ul.pagination {
}
.nav-underline {
gap: calc(0.5 * var(--bs-nav-underline-gap)) var(--bs-nav-underline-gap);
.nav-link {
&.active, &:hover, &:focus {
color: var(--bs-primary);
+9 -12
View File
@@ -27,7 +27,7 @@ from documents.models import DocumentType
from documents.models import PaperlessTask
from documents.models import StoragePath
from documents.models import Tag
from documents.permissions import set_permissions_for_object
from documents.permissions import set_permissions_for_objects
from documents.plugins.helpers import DocumentsStatusManager
from documents.tasks import bulk_update_documents
from documents.tasks import consume_file
@@ -399,16 +399,10 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
return "OK"
def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
"""
Re-run parsing for the given documents.
Consumption workflows do not run here, so ``remote_ocr`` is how the user
asks for the remote engine when it is not configured to handle everything.
"""
def reprocess(doc_ids: list[int]) -> Literal["OK"]:
for document_id in doc_ids:
update_document_content_maybe_archive_file.apply_async(
kwargs={"document_id": document_id, "remote_ocr": remote_ocr},
kwargs={"document_id": document_id},
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
)
@@ -430,10 +424,13 @@ def set_permissions(
else:
qs.update(owner=owner)
for doc in qs:
set_permissions_for_object(permissions=set_permissions, object=doc, merge=merge)
affected_docs = list(qs.values_list("pk", flat=True))
set_permissions_for_objects(
permissions=set_permissions,
model=Document,
pks=affected_docs,
merge=merge,
)
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
-18
View File
@@ -53,7 +53,6 @@ from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess
from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ArchiveFileGenerationChoices
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
@@ -452,19 +451,12 @@ class ConsumerPlugin(
except Exception as e:
self.log.error(f"Error attempting to clean PDF: {e}")
# Workflows have already run at this point, so the metadata knows
# whether this document was singled out for remote OCR
allow_remote = (
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
)
# Based on the mime type, get the parser for that type
parser_class: type[ParserProtocol] | None = (
get_parser_registry().get_parser_for_file(
mime_type,
self.filename,
self.working_copy,
allow_remote=allow_remote,
)
)
if not parser_class:
@@ -473,16 +465,6 @@ class ConsumerPlugin(
f"Unsupported mime type {mime_type}",
)
if self.metadata.remote_ocr and not getattr(
parser_class,
"uses_remote_service",
False,
):
self.log.warning(
"Remote OCR was requested for this document but no remote "
"parser is available for it, processing locally instead.",
)
# Notify all listeners that we're going to do some work.
document_consumption_started.send(
-3
View File
@@ -34,7 +34,6 @@ class DocumentMetadataOverrides:
skip_asn_if_exists: bool = False
version_label: str | None = None
actor_id: int | None = None
remote_ocr: bool = False
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
"""
@@ -58,8 +57,6 @@ class DocumentMetadataOverrides:
self.actor_id = other.actor_id
if other.skip_asn_if_exists:
self.skip_asn_if_exists = True
if other.remote_ocr:
self.remote_ocr = True
if other.version_label is not None:
self.version_label = other.version_label
@@ -1,30 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 17:27
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0023_savedview_icon"),
]
operations = [
migrations.AlterField(
model_name="workflowaction",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Assignment"),
(2, "Removal"),
(3, "Email"),
(4, "Webhook"),
(5, "Password removal"),
(6, "Move to trash"),
(7, "Remote OCR"),
],
default=1,
verbose_name="Workflow Action Type",
),
),
]
-4
View File
@@ -1670,10 +1670,6 @@ class WorkflowAction(models.Model):
6,
_("Move to trash"),
)
REMOTE_OCR = (
7,
_("Remote OCR"),
)
type = models.PositiveSmallIntegerField(
_("Workflow Action Type"),
+178
View File
@@ -173,6 +173,184 @@ def set_permissions_for_object(
)
def _resolve_permissions(codenames: set[str], ctype: ContentType) -> list[Permission]:
"""
Resolves `codenames` to Permission rows, raising like the single-object
assign_perm() this bulk path replaces does (via a `.get()` internally)
if any codename doesn't exist -- e.g. a client-supplied action name that
was never validated (BulkEditObjectsSerializer._validate_permissions
calls validate_set_permissions() only for its side-effecting id checks
and discards the filtered dict it returns, so an unrecognized action key
reaches this function as-is). A plain `.filter()` with no existence
check would otherwise silently build zero rows and no-op instead of
reporting the bad input.
"""
permission_objs = list(
Permission.objects.filter(content_type=ctype, codename__in=codenames),
)
missing = codenames - {p.codename for p in permission_objs}
if missing:
raise Permission.DoesNotExist(
f"Permission matching query does not exist for codename(s): "
f"{', '.join(sorted(missing))}",
)
return permission_objs
# Target number of permission rows to build in Python before handing them to
# bulk_create -- keeps peak memory bounded for a large "apply to all" call,
# independent of bulk_create's own batch_size (which only caps the size of
# each INSERT statement, not how many row objects exist in memory at once).
_PERMISSION_ROW_CHUNK_SIZE = 5000
def _apply_bulk_permission_entry(
*,
perm_model: type[UserObjectPermission] | type[GroupObjectPermission],
identity_model: type[User] | type[Group],
identity_field: str,
ids: list[int],
codename: str,
permission_objs: list[Permission],
ctype: ContentType,
object_pks: list[str],
merge: bool,
) -> None:
# Only the ids are needed to build permission rows (via `<field>_id=`),
# so avoid fetching full User/Group rows for identities that may not
# even end up being granted anything new.
add_ids = set(
identity_model.objects.filter(id__in=ids).values_list("id", flat=True),
)
if not merge:
existing_ids = set(
perm_model.objects.filter(
content_type=ctype,
object_pk__in=object_pks,
permission__codename=codename,
)
.values_list(f"{identity_field}_id", flat=True)
.distinct(),
)
remove_ids = existing_ids - add_ids
if remove_ids:
perm_model.objects.filter(
content_type=ctype,
object_pk__in=object_pks,
permission__codename=codename,
**{f"{identity_field}_id__in": remove_ids},
).delete()
if not add_ids:
return
rows_per_pk = len(permission_objs) * len(add_ids)
pks_per_chunk = max(1, _PERMISSION_ROW_CHUNK_SIZE // rows_per_pk)
for start in range(0, len(object_pks), pks_per_chunk):
pk_chunk = object_pks[start : start + pks_per_chunk]
rows = [
perm_model(
content_type=ctype,
object_pk=pk,
permission=permission_obj,
**{f"{identity_field}_id": identity_id},
)
for permission_obj in permission_objs
for pk in pk_chunk
for identity_id in add_ids
]
# ignore_conflicts skips only rows that already exist as an exact
# (identity, permission, object) match -- the same de-dup the
# underlying (user|group, permission, object_pk) unique constraint
# already enforces for the single-object assign_perm() this
# replaces, so it doesn't change what counts as "already granted".
# batch_size caps how many rows go into a single INSERT so a huge
# chunk doesn't build one enormous statement.
perm_model.objects.bulk_create(rows, ignore_conflicts=True, batch_size=1000)
def set_permissions_for_objects(
permissions: dict,
model: type[Model],
pks: QuerySet | list,
*,
merge: bool = False,
) -> None:
"""
Bulk equivalent of set_permissions_for_object: applies the same
permission changes to every object identified by `pks` at once.
Takes a model + pks (rather than model instances) deliberately -- the
permission rows built below only ever need `pk`, `content_type`, and
identity ids, so callers shouldn't have to fetch full rows (with every
other field) just to hand them to this function.
Deliberately does not use guardian's queryset/list-aware assign_perm:
passing a list as the object routes to bulk_assign_perm, which skips
creating a direct permission row for anyone who already has the
permission via ANY group membership (it checks
ObjectPermissionChecker.has_perm, which is group-inheritance-aware) --
unlike the single-object assign_perm this replaces, which always
ensures a direct row via get_or_create regardless of group-derived
access. Losing that guarantee would mean a later revocation of the
group's grant silently strips access an admin explicitly asked to be
direct. Bulk-creating rows straight against the permission models
instead (see _apply_bulk_permission_entry) preserves the original
always-create-a-direct-row semantics while still batching every object
and every identity into one query per action, rather than one query per
(object, user) pair.
"""
object_pks = [str(pk) for pk in pks]
if not object_pks: # pragma: no cover
return
model_name = model.__name__.lower()
ctype = ContentType.objects.get_for_model(model)
for action, entry in permissions.items():
codename = f"{action}_{model_name}"
implied_codenames = {codename}
if action == "change":
# change gives view too
implied_codenames.add(f"view_{model_name}")
# Resolved once per action (not once per users/groups branch) and
# shared between both below -- also where an unrecognized action
# name (see _resolve_permissions) is caught.
permission_objs = (
_resolve_permissions(implied_codenames, ctype)
if "users" in entry or "groups" in entry
else []
)
if "users" in entry:
_apply_bulk_permission_entry(
perm_model=UserObjectPermission,
identity_model=User,
identity_field="user",
ids=entry["users"],
codename=codename,
permission_objs=permission_objs,
ctype=ctype,
object_pks=object_pks,
merge=merge,
)
if "groups" in entry:
_apply_bulk_permission_entry(
perm_model=GroupObjectPermission,
identity_model=Group,
identity_field="group",
ids=entry["groups"],
codename=codename,
permission_objs=permission_objs,
ctype=ctype,
object_pks=object_pks,
merge=merge,
)
def permitted_object_ids(
user: User | None,
model: type[Model],
+1 -45
View File
@@ -1795,7 +1795,7 @@ class DeleteDocumentsSerializer(DocumentSelectionSerializer):
class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
remote_ocr = serializers.BooleanField(required=False, default=False)
pass
class BulkEditSerializer(
@@ -2137,13 +2137,6 @@ class BulkEditSerializer(
f"Page {op['page']} is out of bounds for document with {doc.page_count} pages.",
)
def _validate_parameters_reprocess(self, parameters) -> None:
if "remote_ocr" in parameters:
if not isinstance(parameters["remote_ocr"], bool):
raise serializers.ValidationError("remote_ocr must be a boolean")
else:
parameters["remote_ocr"] = False
def validate_parameters_remove_password(self, parameters):
if "password" not in parameters:
raise serializers.ValidationError("password not specified")
@@ -2208,8 +2201,6 @@ class BulkEditSerializer(
self._validate_parameters_edit_pdf(parameters, attrs["documents"][0])
elif method == bulk_edit.remove_password:
self.validate_parameters_remove_password(parameters)
elif method == bulk_edit.reprocess:
self._validate_parameters_reprocess(parameters)
return attrs
@@ -3312,41 +3303,6 @@ class WorkflowSerializer(serializers.ModelSerializer[Workflow]):
"actions",
]
def validate(self, attrs):
attrs = super().validate(attrs)
if "actions" in attrs:
has_remote_ocr_action = any(
action.get("type") == WorkflowAction.WorkflowActionType.REMOTE_OCR
for action in attrs["actions"]
)
else:
has_remote_ocr_action = self.instance is not None and (
self.instance.actions.filter(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
).exists()
)
if "triggers" in attrs:
has_consumption_trigger = any(
trigger.get("type") == WorkflowTrigger.WorkflowTriggerType.CONSUMPTION
for trigger in attrs["triggers"]
)
else:
has_consumption_trigger = self.instance is not None and (
self.instance.triggers.filter(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
).exists()
)
# Remote OCR can only work with consumption triggers
if has_remote_ocr_action and not has_consumption_trigger:
raise serializers.ValidationError(
"Remote OCR actions require a consumption started trigger",
)
return attrs
def update_triggers_and_actions(
self,
instance: Workflow,
-11
View File
@@ -973,17 +973,6 @@ def run_workflows(
)
elif action.type == WorkflowAction.WorkflowActionType.MOVE_TO_TRASH:
has_move_to_trash_action = True
elif action.type == WorkflowAction.WorkflowActionType.REMOTE_OCR:
if use_overrides and overrides:
overrides.remote_ocr = True
else:
# If a workflow has a consumption trigger *and* another type,
# the document has already been parsed by the time the other one fires
logger.debug(
"Remote OCR action only applies to consumption "
"triggers, ignoring",
extra={"group": logging_group},
)
if not use_overrides:
# limit title to 128 characters
+1 -10
View File
@@ -67,7 +67,6 @@ from documents.utils import identity
from documents.versioning import annotate_effective_content
from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig
from paperless.config import RemoteOCRConfig
from paperless.logging import consume_task_id
from paperless.parsers import ParserContext
from paperless.parsers.registry import get_parser_registry
@@ -339,17 +338,10 @@ def bulk_update_documents(document_ids) -> None:
@shared_task
def update_document_content_maybe_archive_file(
document_id,
*,
remote_ocr: bool = False,
) -> None:
def update_document_content_maybe_archive_file(document_id) -> None:
"""
Re-creates OCR content and thumbnail for a document, and archive file if
it exists.
Remote OCR is used only when the engine is configured to handle everything
or if explicitly asked for via ``remote_ocr``.
"""
document = Document.objects.get(id=document_id)
@@ -359,7 +351,6 @@ def update_document_content_maybe_archive_file(
mime_type,
document.original_filename or "",
document.source_path,
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
)
if not parser_class:
@@ -72,10 +72,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"barcode_enable_tag": None,
"barcode_tag_mapping": None,
"barcode_tag_split": None,
"remote_ocr_engine": None,
"remote_ocr_api_key": None,
"remote_ocr_endpoint": None,
"remote_ocr_mode": None,
"ai_enabled": False,
"llm_embedding_backend": None,
"llm_embedding_model": None,
@@ -874,49 +870,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
config.refresh_from_db()
self.assertEqual(config.llm_api_key, None)
def test_update_remote_ocr_api_key(self) -> None:
"""
GIVEN:
- Existing config with remote_ocr_api_key specified
WHEN:
- API to update remote_ocr_api_key is called with all *s
- API to update remote_ocr_api_key is called with empty string
THEN:
- remote_ocr_api_key is unchanged
- remote_ocr_api_key is set to None
"""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_api_key = "1234567890"
config.save()
# Test with all *
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "*" * 32,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, "1234567890")
# Test with empty string
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, None)
def test_enable_ai_index_triggers_update(self) -> None:
"""
GIVEN:
+1 -46
View File
@@ -532,29 +532,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": False})
@mock.patch("documents.views.bulk_edit.reprocess")
def test_reprocess_documents_endpoint_remote_ocr(self, m) -> None:
"""
GIVEN:
- API data to reprocess a document with remote OCR requested
WHEN:
- API is called
THEN:
- reprocess is called with remote_ocr=True
"""
self.setup_mock(m, "reprocess")
response = self.client.post(
"/api/documents/reprocess/",
json.dumps({"documents": [self.doc1.id], "remote_ocr": True}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
self.assertEqual(len(kwargs), 0)
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None:
@@ -1575,29 +1553,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
),
)
def test_legacy_bulk_edit_reprocess_invalid_remote_ocr(self) -> None:
"""
GIVEN:
- The deprecated bulk_edit endpoint with a non-boolean remote_ocr
WHEN:
- API is called
THEN:
- The request is rejected rather than passed through to the task
"""
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc1.id],
"method": "reprocess",
"parameters": {"remote_ocr": "yes please"},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
@mock.patch("documents.views.bulk_edit.edit_pdf")
def test_edit_pdf(self, m) -> None:
self.setup_mock(m, "edit_pdf")
+44
View File
@@ -2,10 +2,13 @@ import datetime
import json
from unittest import mock
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from guardian.shortcuts import assign_perm
from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
from rest_framework import status
from rest_framework.test import APITestCase
@@ -815,6 +818,47 @@ class TestBulkEditObjects(APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(StoragePath.objects.count(), 0)
def test_bulk_objects_set_permissions_batched_across_object_count(
self,
) -> None:
"""
GIVEN:
- Many tags are being bulk-edited to set permissions at once
WHEN:
- bulk_edit_objects API endpoint is called with set_permissions
operation over a small batch vs. a much larger one
THEN:
- Permissions are applied correctly at both scales
"""
group1 = Group.objects.create(name="perm-group")
permissions = {
"view": {"users": [self.user1.id, self.user2.id], "groups": [group1.id]},
"change": {"users": [self.user1.id], "groups": [group1.id]},
}
def run_with_n_tags(n: int) -> None:
tags = [Tag.objects.create(name=f"perm-tag-{n}-{i}") for i in range(n)]
response = self.client.post(
"/api/bulk_edit_objects/",
json.dumps(
{
"objects": [t.id for t in tags],
"object_type": "tags",
"operation": "set_permissions",
"permissions": permissions,
"merge": False,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
for tag in tags:
self.assertEqual(get_users_with_perms(tag).count(), 2)
self.assertEqual(get_groups_with_perms(tag).count(), 1)
run_with_n_tags(5)
run_with_n_tags(50)
def test_bulk_objects_delete_all_filtered(self) -> None:
"""
GIVEN:
@@ -60,10 +60,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
},
"email_enabled": False,
"ai_enabled": False,
"remote_ocr": {
"configured": False,
"mode": "always",
},
},
)
@@ -158,50 +154,6 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
str(response.data["settings"]),
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="somekey",
REMOTE_OCR_ENDPOINT="https://example.cognitiveservices.azure.com",
REMOTE_OCR_MODE="workflow_only",
)
def test_settings_reports_remote_ocr_when_configured(self) -> None:
"""
GIVEN:
- A fully configured remote OCR engine in workflow_only mode
WHEN:
- The ui_settings endpoint is called
THEN:
- The UI is told remote OCR is available and selective, so it can
offer it where it would actually change something
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
response.data["settings"]["remote_ocr"],
{"configured": True, "mode": "workflow_only"},
)
@override_settings(
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
def test_settings_reports_remote_ocr_incompletely_configured(self) -> None:
"""
GIVEN:
- An engine named but missing its endpoint and API key
WHEN:
- The ui_settings endpoint is called
THEN:
- It is reported as not configured, matching what the parser
registry will actually do
"""
response = self.client.get(self.ENDPOINT, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertFalse(response.data["settings"]["remote_ocr"]["configured"])
@override_settings(
OAUTH_CALLBACK_BASE_URL="http://localhost:8000",
GMAIL_OAUTH_CLIENT_ID="abc123",
-135
View File
@@ -506,141 +506,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(Workflow.objects.count(), 1)
def test_api_create_remote_ocr_action_requires_consumption_trigger(
self,
) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- No consumption started trigger, so the action could never run
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
existing_count = Workflow.objects.count()
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR too late",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Workflow.objects.count(), existing_count)
def test_api_create_remote_ocr_action_with_consumption_trigger(self) -> None:
"""
GIVEN:
- API request to create a workflow with a remote OCR action
- A consumption started trigger alongside another trigger type
WHEN:
- API is called
THEN:
- The workflow is created, the action applies to consumption only
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Remote OCR on consume",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
"filter_filename": "*.pdf",
},
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
},
],
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
def test_api_partial_update_adds_remote_ocr_action(self) -> None:
"""
GIVEN:
- An existing workflow with a consumption started trigger
WHEN:
- A partial update adds a remote OCR action without resubmitting triggers
THEN:
- The existing trigger is considered and the update succeeds
"""
response = self.client.patch(
f"{self.ENDPOINT}{self.workflow.id}/",
json.dumps(
{
"actions": [
{
"type": WorkflowAction.WorkflowActionType.REMOTE_OCR,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
self.workflow.actions.get().type,
WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
def test_api_partial_update_cannot_remove_remote_ocr_trigger(self) -> None:
"""
GIVEN:
- An existing workflow with a remote OCR action
- An existing consumption started trigger
WHEN:
- A partial update replaces the trigger without resubmitting actions
THEN:
- The existing action is considered and the update is rejected
"""
self.action.type = WorkflowAction.WorkflowActionType.REMOTE_OCR
self.action.save()
response = self.client.patch(
f"{self.ENDPOINT}{self.workflow.id}/",
json.dumps(
{
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(self.workflow.triggers.get(), self.trigger)
def test_api_create_workflow_trigger_action_empty_fields(self) -> None:
"""
GIVEN:
+116 -53
View File
@@ -5,6 +5,7 @@ from unittest import mock
import pikepdf
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import TestCase
from guardian.shortcuts import assign_perm
@@ -19,6 +20,7 @@ from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.permissions import set_permissions_for_objects
from documents.tests.utils import DirectoriesMixin
@@ -510,6 +512,120 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
)
self.assertEqual(groups_with_perms.count(), 2)
@mock.patch("documents.tasks.bulk_update_documents.apply_async")
def test_set_permissions_batched_across_document_count(
self,
m,
) -> None:
"""
GIVEN:
- Many documents are being bulk-edited to set permissions at once
WHEN:
- set_permissions runs over a small batch vs. a much larger one
THEN:
- Permissions are applied correctly at both scales
"""
permissions = {
"view": {
"users": [self.user1.id, self.user2.id],
"groups": [self.group2.id],
},
"change": {
"users": [self.user1.id],
"groups": [self.group2.id],
},
}
def run_with_n_documents(n: int) -> None:
docs = [
Document.objects.create(checksum=f"perm-{n}-{i}", title=f"perm-{n}-{i}")
for i in range(n)
]
bulk_edit.set_permissions(
[doc.id for doc in docs],
set_permissions=permissions,
owner=self.owner,
merge=False,
)
for doc in docs:
self.assertEqual(get_users_with_perms(doc).count(), 2)
self.assertEqual(get_groups_with_perms(doc).count(), 1)
run_with_n_documents(5)
run_with_n_documents(50)
@mock.patch("documents.tasks.bulk_update_documents.apply_async")
def test_set_permissions_grants_direct_perm_even_if_already_granted_via_group(
self,
m,
) -> None:
"""
GIVEN:
- A user already has view access to a document via group
membership, with no direct grant of their own
WHEN:
- set_permissions explicitly grants that same user direct view
access via bulk_edit
THEN:
- A direct permission grant is created for the user, not skipped
because they already have equivalent access via the group
Regression test: guardian's queryset-aware assign_perm() (routed to
when the target is a list/queryset) skips creating a direct row for
anyone whose ObjectPermissionChecker.has_perm() already returns True
-- which includes group-derived access. The single-object assign_perm
this bulk path replaces has no such check; it always ensures a
direct row via get_or_create. Losing that guarantee would mean
revoking the group's grant later silently strips access that was
supposed to be explicit.
"""
self.doc1.owner = self.user1
self.doc1.save()
self.user1.groups.add(self.group1)
assign_perm("view_document", self.group1, self.doc1)
bulk_edit.set_permissions(
[self.doc1.id],
set_permissions={
"view": {"users": [self.user1.id], "groups": []},
},
merge=True,
)
direct_users = get_users_with_perms(
self.doc1,
only_with_perms_in=["view_document"],
with_group_users=False,
)
self.assertIn(self.user1, direct_users)
def test_set_permissions_for_objects_raises_for_unknown_action(self) -> None:
"""
GIVEN:
- An unrecognized permission action name with users to grant it
to
WHEN:
- set_permissions_for_objects is called
THEN:
- Permission.DoesNotExist is raised, not a silent no-op
Regression test: the endpoint that calls this
(BulkEditObjectPermissionsView) never actually validates action
names against the raw client-supplied permissions dict --
BulkEditObjectsSerializer._validate_permissions calls
validate_set_permissions() only for its side-effecting user/group id
checks and discards the filtered dict it returns -- so a bogus
action key reaches this function as-is. Resolving the Permission via
a bare `.filter()` (which returns empty instead of raising) would
silently drop the grant and report success.
"""
with self.assertRaises(Permission.DoesNotExist):
set_permissions_for_objects(
{"not_a_real_action": {"users": [self.user1.id], "groups": []}},
Document,
[self.doc1.pk],
)
@mock.patch("documents.models.Document.delete")
def test_delete_documents_old_uuid_field(self, m) -> None:
m.side_effect = Exception("Data too long for column 'transaction_id' at row 1")
@@ -1782,56 +1898,3 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIn("wrong password", str(exc.exception))
self.assertIn("Error removing password from document", cm.output[0])
class TestBulkEditReprocess(DirectoriesMixin, TestCase):
def setUp(self) -> None:
super().setUp()
self.doc = Document.objects.create(
title="test",
checksum="A",
mime_type="application/pdf",
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_defaults_to_local(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that says nothing about remote OCR
WHEN:
- reprocess is called
THEN:
- The task is queued without asking for the remote engine
"""
result = bulk_edit.reprocess([self.doc.id])
self.assertEqual(result, "OK")
mock_task.apply_async.assert_called_once()
_, kwargs = mock_task.apply_async.call_args
self.assertEqual(
kwargs["kwargs"],
{"document_id": self.doc.id, "remote_ocr": False},
)
@mock.patch("documents.bulk_edit.update_document_content_maybe_archive_file")
def test_reprocess_passes_remote_ocr(self, mock_task: mock.Mock) -> None:
"""
GIVEN:
- A reprocess request that explicitly asks for remote OCR
WHEN:
- reprocess is called
THEN:
- The request is forwarded to the task for every document
"""
other = Document.objects.create(
title="test2",
checksum="B",
mime_type="application/pdf",
)
bulk_edit.reprocess([self.doc.id, other.id], remote_ocr=True)
self.assertEqual(mock_task.apply_async.call_count, 2)
for call in mock_task.apply_async.call_args_list:
self.assertTrue(call.kwargs["kwargs"]["remote_ocr"])
-80
View File
@@ -1559,72 +1559,6 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
consumer.run_post_consume_script(doc)
class TestConsumerRemoteOCR(
DirectoriesMixin,
FileSystemAssertsMixin,
GetConsumerMixin,
TestCase,
):
"""
The consumer resolves the remote OCR mode and the per-document request from
workflows into the allow_remote flag it hands to the parser registry.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.consumer.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
self.addCleanup(patcher.stop)
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
src = (
Path(__file__).parent
/ "samples"
/ "documents"
/ "originals"
/ "0000001.pdf"
)
dst = self.dirs.scratch_dir / "sample.pdf"
shutil.copy(src, dst)
with self.get_consumer(dst, overrides=overrides) as consumer:
consumer.run()
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
"""
GIVEN: Remote OCR mode is 'always'.
WHEN: A document is consumed without any workflow asking for it.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A document is consumed and nothing asked for remote OCR.
THEN: The remote parser is excluded.
"""
self.assertFalse(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A workflow set remote_ocr on the metadata overrides.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
)
class TestMetadataOverrides(TestCase):
def test_update_skip_asn_if_exists(self) -> None:
base = DocumentMetadataOverrides()
@@ -1632,20 +1566,6 @@ class TestMetadataOverrides(TestCase):
base.update(incoming)
self.assertTrue(base.skip_asn_if_exists)
def test_update_remote_ocr(self) -> None:
base = DocumentMetadataOverrides()
base.update(DocumentMetadataOverrides(remote_ocr=True))
self.assertTrue(base.remote_ocr)
def test_update_remote_ocr_is_not_unset(self) -> None:
"""
A later workflow that says nothing must not undo an earlier one that
asked for remote OCR.
"""
base = DocumentMetadataOverrides(remote_ocr=True)
base.update(DocumentMetadataOverrides())
self.assertTrue(base.remote_ocr)
def test_update_actor_and_version_label(self) -> None:
base = DocumentMetadataOverrides(
actor_id=1,
-39
View File
@@ -287,45 +287,6 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
"""
Consumption workflows do not run on reprocess, so the remote parser is
used only in 'always' mode or when the caller explicitly asks for it.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.tasks.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = None
self.addCleanup(patcher.stop)
self.doc = Document.objects.create(
title="test",
content="my document",
checksum="wow",
mime_type="application/pdf",
)
def _allow_remote(self, **kwargs) -> bool:
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return call_kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
self.assertTrue(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
self.assertFalse(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
self.assertTrue(self._allow_remote(remote_ocr=True))
class TestAIIndex(DirectoriesMixin, TestCase):
@override_settings(
AI_ENABLED=True,
-79
View File
@@ -5409,82 +5409,3 @@ class TestDateWorkflowLocalization(
document = Document.objects.first()
assert document is not None
assert document.title == expected_title
class TestRemoteOCRWorkflowAction(DirectoriesMixin, SampleDirMixin, APITestCase):
def _make_workflow(self, trigger_type) -> None:
trigger = WorkflowTrigger.objects.create(type=trigger_type)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
def test_consumption_trigger_requests_remote_ocr(self) -> None:
"""
GIVEN:
- A consumption workflow with a remote OCR action
WHEN:
- A matching document is consumed
THEN:
- The overrides ask for remote OCR, which is what the consumer
reads when choosing a parser
"""
self._make_workflow(WorkflowTrigger.WorkflowTriggerType.CONSUMPTION)
test_file = shutil.copy(
self.SAMPLE_DIR / "simple.pdf",
self.dirs.scratch_dir / "simple.pdf",
)
overrides = DocumentMetadataOverrides()
run_workflows(
WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
ConsumableDocument(
source=DocumentSource.ConsumeFolder,
original_file=test_file,
),
overrides=overrides,
)
self.assertTrue(overrides.remote_ocr)
def test_other_trigger_types_are_ignored(self) -> None:
"""
GIVEN:
- A workflow with a remote OCR action that also has a
non-consumption trigger, which is a valid combination
WHEN:
- The non-consumption trigger fires
THEN:
- The action is skipped, since the document has already been
parsed by this point
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
)
updated_trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.REMOTE_OCR,
)
w = Workflow.objects.create(name="Remote OCR", order=0)
w.triggers.add(trigger, updated_trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
original_filename="sample.pdf",
)
with self.assertLogs("paperless.handlers", level="DEBUG") as cm:
run_workflows(
WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
doc,
)
self.assertIn("only applies to consumption triggers", "".join(cm.output))
+8 -16
View File
@@ -178,7 +178,7 @@ from documents.permissions import has_perms_owner_aware
from documents.permissions import has_system_status_permission
from documents.permissions import permitted_document_ids
from documents.permissions import permitted_object_ids
from documents.permissions import set_permissions_for_object
from documents.permissions import set_permissions_for_objects
from documents.plugins.date_parsing import get_date_parser
from documents.schema import generate_object_with_permissions_schema
from documents.search import SearchHit
@@ -239,10 +239,8 @@ from paperless import version
from paperless.celery import app as celery_app
from paperless.config import AIConfig
from paperless.config import GeneralConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ApplicationConfiguration
from paperless.parsers.registry import get_parser_registry
from paperless.parsers.remote import RemoteEngineConfig
from paperless.serialisers import GroupSerializer
from paperless.serialisers import UserSerializer
from paperless.views import StandardPagination
@@ -1133,8 +1131,7 @@ class DocumentViewSet(
"custom_fields",
queryset=CustomFieldInstance.objects.select_related("field"),
),
# NotesSerializer nests the author, this avoids query per note
Prefetch("notes", queryset=Note.objects.select_related("user")),
"notes",
)
)
@@ -4103,11 +4100,6 @@ class UiSettingsView(GenericAPIView[Any]):
ui_settings["auditlog_enabled"] = settings.AUDIT_LOG_ENABLED
ui_settings["remote_ocr"] = {
"configured": RemoteEngineConfig.from_app_config().engine_is_valid(),
"mode": RemoteOCRConfig().remote_ocr_mode,
}
if settings.GMAIL_OAUTH_ENABLED or settings.OUTLOOK_OAUTH_ENABLED:
manager = PaperlessMailOAuth2Manager()
if settings.GMAIL_OAUTH_ENABLED:
@@ -4918,12 +4910,12 @@ class BulkEditObjectsView(PassUserMixin):
qs_owner_update.update(owner=owner)
if "permissions" in serializer.validated_data:
for obj in qs:
set_permissions_for_object(
permissions=permissions,
object=obj,
merge=merge,
)
set_permissions_for_objects(
permissions=permissions,
model=object_class,
pks=qs.values_list("pk", flat=True),
merge=merge,
)
except Exception as e:
logger.warning(
+109 -141
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-08-25 21:36+0000\n"
"POT-Creation-Date: 2026-08-19 18:18+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -89,7 +89,7 @@ msgstr ""
msgid "Automatic"
msgstr ""
#: documents/models.py:65 documents/models.py:606 documents/models.py:1924
#: documents/models.py:65 documents/models.py:606 documents/models.py:1920
#: paperless_mail/models.py:23 paperless_mail/models.py:143
msgid "name"
msgstr ""
@@ -272,7 +272,7 @@ msgid "Optional short label for a document version."
msgstr ""
#: documents/models.py:340 documents/models.py:922 documents/models.py:976
#: documents/models.py:1967
#: documents/models.py:1963
msgid "document"
msgstr ""
@@ -1385,202 +1385,198 @@ msgid "Move to trash"
msgstr ""
#: documents/models.py:1675
msgid "Remote OCR"
msgstr ""
#: documents/models.py:1679
msgid "Workflow Action Type"
msgstr ""
#: documents/models.py:1684 documents/models.py:1926
#: documents/models.py:1680 documents/models.py:1922
#: paperless_mail/models.py:145
msgid "order"
msgstr ""
#: documents/models.py:1687
#: documents/models.py:1683
msgid "assign title"
msgstr ""
#: documents/models.py:1691
#: documents/models.py:1687
msgid "Assign a document title, must be a Jinja2 template, see documentation."
msgstr ""
#: documents/models.py:1699 paperless_mail/models.py:274
#: documents/models.py:1695 paperless_mail/models.py:274
msgid "assign this tag"
msgstr ""
#: documents/models.py:1708 paperless_mail/models.py:282
#: documents/models.py:1704 paperless_mail/models.py:282
msgid "assign this document type"
msgstr ""
#: documents/models.py:1717 paperless_mail/models.py:296
#: documents/models.py:1713 paperless_mail/models.py:296
msgid "assign this correspondent"
msgstr ""
#: documents/models.py:1726
#: documents/models.py:1722
msgid "assign this storage path"
msgstr ""
#: documents/models.py:1735
#: documents/models.py:1731
msgid "assign this owner"
msgstr ""
#: documents/models.py:1742
#: documents/models.py:1738
msgid "grant view permissions to these users"
msgstr ""
#: documents/models.py:1749
#: documents/models.py:1745
msgid "grant view permissions to these groups"
msgstr ""
#: documents/models.py:1756
#: documents/models.py:1752
msgid "grant change permissions to these users"
msgstr ""
#: documents/models.py:1763
#: documents/models.py:1759
msgid "grant change permissions to these groups"
msgstr ""
#: documents/models.py:1770
#: documents/models.py:1766
msgid "assign these custom fields"
msgstr ""
#: documents/models.py:1774
#: documents/models.py:1770
msgid "custom field values"
msgstr ""
#: documents/models.py:1778
#: documents/models.py:1774
msgid "Optional values to assign to the custom fields."
msgstr ""
#: documents/models.py:1787
#: documents/models.py:1783
msgid "remove these tag(s)"
msgstr ""
#: documents/models.py:1792
#: documents/models.py:1788
msgid "remove all tags"
msgstr ""
#: documents/models.py:1799
#: documents/models.py:1795
msgid "remove these document type(s)"
msgstr ""
#: documents/models.py:1804
#: documents/models.py:1800
msgid "remove all document types"
msgstr ""
#: documents/models.py:1811
#: documents/models.py:1807
msgid "remove these correspondent(s)"
msgstr ""
#: documents/models.py:1816
#: documents/models.py:1812
msgid "remove all correspondents"
msgstr ""
#: documents/models.py:1823
#: documents/models.py:1819
msgid "remove these storage path(s)"
msgstr ""
#: documents/models.py:1828
#: documents/models.py:1824
msgid "remove all storage paths"
msgstr ""
#: documents/models.py:1835
#: documents/models.py:1831
msgid "remove these owner(s)"
msgstr ""
#: documents/models.py:1840
#: documents/models.py:1836
msgid "remove all owners"
msgstr ""
#: documents/models.py:1847
#: documents/models.py:1843
msgid "remove view permissions for these users"
msgstr ""
#: documents/models.py:1854
#: documents/models.py:1850
msgid "remove view permissions for these groups"
msgstr ""
#: documents/models.py:1861
#: documents/models.py:1857
msgid "remove change permissions for these users"
msgstr ""
#: documents/models.py:1868
#: documents/models.py:1864
msgid "remove change permissions for these groups"
msgstr ""
#: documents/models.py:1873
#: documents/models.py:1869
msgid "remove all permissions"
msgstr ""
#: documents/models.py:1880
#: documents/models.py:1876
msgid "remove these custom fields"
msgstr ""
#: documents/models.py:1885
#: documents/models.py:1881
msgid "remove all custom fields"
msgstr ""
#: documents/models.py:1894
#: documents/models.py:1890
msgid "email"
msgstr ""
#: documents/models.py:1903
#: documents/models.py:1899
msgid "webhook"
msgstr ""
#: documents/models.py:1907
#: documents/models.py:1903
msgid "passwords"
msgstr ""
#: documents/models.py:1911
#: documents/models.py:1907
msgid ""
"Passwords to try when removing PDF protection. Separate with commas or new "
"lines."
msgstr ""
#: documents/models.py:1916
#: documents/models.py:1912
msgid "workflow action"
msgstr ""
#: documents/models.py:1917
#: documents/models.py:1913
msgid "workflow actions"
msgstr ""
#: documents/models.py:1932
#: documents/models.py:1928
msgid "triggers"
msgstr ""
#: documents/models.py:1939
#: documents/models.py:1935
msgid "actions"
msgstr ""
#: documents/models.py:1942 paperless_mail/models.py:154
#: documents/models.py:1938 paperless_mail/models.py:154
msgid "enabled"
msgstr ""
#: documents/models.py:1953
#: documents/models.py:1949
msgid "workflow"
msgstr ""
#: documents/models.py:1957
#: documents/models.py:1953
msgid "workflow trigger type"
msgstr ""
#: documents/models.py:1971
#: documents/models.py:1967
msgid "date run"
msgstr ""
#: documents/models.py:1977
#: documents/models.py:1973
msgid "workflow run"
msgstr ""
#: documents/models.py:1978
#: documents/models.py:1974
msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:523 documents/serialisers.py:875
#: documents/serialisers.py:2827 documents/views.py:311 documents/views.py:2613
#: documents/serialisers.py:2818 documents/views.py:309 documents/views.py:2610
#: paperless_mail/serialisers.py:155
msgid "Insufficient permissions."
msgstr ""
@@ -1589,39 +1585,39 @@ msgstr ""
msgid "Invalid color."
msgstr ""
#: documents/serialisers.py:2304
#: documents/serialisers.py:2295
#, python-format
msgid "File type %(type)s not supported"
msgstr ""
#: documents/serialisers.py:2348
#: documents/serialisers.py:2339
#, python-format
msgid "Custom field id must be an integer: %(id)s"
msgstr ""
#: documents/serialisers.py:2355
#: documents/serialisers.py:2346
#, python-format
msgid "Custom field with id %(id)s does not exist"
msgstr ""
#: documents/serialisers.py:2372 documents/serialisers.py:2382
#: documents/serialisers.py:2363 documents/serialisers.py:2373
msgid ""
"Custom fields must be a list of integers or an object mapping ids to values."
msgstr ""
#: documents/serialisers.py:2377
#: documents/serialisers.py:2368
msgid "Some custom fields don't exist or were specified twice."
msgstr ""
#: documents/serialisers.py:2524
#: documents/serialisers.py:2515
msgid "Invalid variable detected."
msgstr ""
#: documents/serialisers.py:2883
#: documents/serialisers.py:2874
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2913 documents/views.py:4607
#: documents/serialisers.py:2904 documents/views.py:4599
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
@@ -1889,36 +1885,36 @@ msgstr ""
msgid "Unable to parse URI {value}"
msgstr ""
#: documents/views.py:304 documents/views.py:2610
#: documents/views.py:302 documents/views.py:2607
msgid "Invalid more_like_id"
msgstr ""
#: documents/views.py:1587
#: documents/views.py:1584
msgid "Invalid AI configuration."
msgstr ""
#: documents/views.py:1598
#: documents/views.py:1595
msgid "AI backend request timed out."
msgstr ""
#: documents/views.py:2435 documents/views.py:2756
#: documents/views.py:2432 documents/views.py:2753
msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr ""
#: documents/views.py:4620
#: documents/views.py:4612
#, python-format
msgid "Insufficient permissions to share document %(id)s."
msgstr ""
#: documents/views.py:4666
#: documents/views.py:4658
msgid "Bundle is already being processed."
msgstr ""
#: documents/views.py:4727
#: documents/views.py:4719
msgid "The share link bundle is still being prepared. Please try again later."
msgstr ""
#: documents/views.py:4737
#: documents/views.py:4729
msgid "The share link bundle is unavailable."
msgstr ""
@@ -2002,203 +1998,175 @@ msgstr ""
msgid "CMYK"
msgstr ""
#: paperless/models.py:82
msgid "Azure AI Document Intelligence"
msgstr ""
#: paperless/models.py:90
msgid "All supported documents"
msgstr ""
#: paperless/models.py:91
msgid "Only when a workflow enables it"
msgstr ""
#: paperless/models.py:95 paperless/models.py:105
#: paperless/models.py:78 paperless/models.py:88
msgid "OpenAI-compatible"
msgstr ""
#: paperless/models.py:96
#: paperless/models.py:79
msgid "Huggingface"
msgstr ""
#: paperless/models.py:97 paperless/models.py:106
#: paperless/models.py:80 paperless/models.py:89
msgid "Ollama"
msgstr ""
#: paperless/models.py:115
#: paperless/models.py:98
msgid "Sets the output PDF type"
msgstr ""
#: paperless/models.py:127
#: paperless/models.py:110
msgid "Do OCR from page 1 to this value"
msgstr ""
#: paperless/models.py:133
#: paperless/models.py:116
msgid "Do OCR using these languages"
msgstr ""
#: paperless/models.py:140
#: paperless/models.py:123
msgid "Sets the OCR mode"
msgstr ""
#: paperless/models.py:148
#: paperless/models.py:131
msgid "Controls archive file generation"
msgstr ""
#: paperless/models.py:156
#: paperless/models.py:139
msgid "Sets image DPI fallback value"
msgstr ""
#: paperless/models.py:163
#: paperless/models.py:146
msgid "Controls the unpaper cleaning"
msgstr ""
#: paperless/models.py:170
#: paperless/models.py:153
msgid "Enables deskew"
msgstr ""
#: paperless/models.py:173
#: paperless/models.py:156
msgid "Enables page rotation"
msgstr ""
#: paperless/models.py:178
#: paperless/models.py:161
msgid "Sets the threshold for rotation of pages"
msgstr ""
#: paperless/models.py:184
#: paperless/models.py:167
msgid "Sets the maximum image size for decompression"
msgstr ""
#: paperless/models.py:190
#: paperless/models.py:173
msgid "Sets the Ghostscript color conversion strategy"
msgstr ""
#: paperless/models.py:198
#: paperless/models.py:181
msgid "Adds additional user arguments for OCRMyPDF"
msgstr ""
#: paperless/models.py:207
#: paperless/models.py:190
msgid "Application title"
msgstr ""
#: paperless/models.py:214
#: paperless/models.py:197
msgid "Application logo"
msgstr ""
#: paperless/models.py:229
#: paperless/models.py:212
msgid "Enables barcode scanning"
msgstr ""
#: paperless/models.py:235
#: paperless/models.py:218
msgid "Enables barcode TIFF support"
msgstr ""
#: paperless/models.py:241
#: paperless/models.py:224
msgid "Sets the barcode string"
msgstr ""
#: paperless/models.py:249
#: paperless/models.py:232
msgid "Retains split pages"
msgstr ""
#: paperless/models.py:255
#: paperless/models.py:238
msgid "Enables ASN barcode"
msgstr ""
#: paperless/models.py:261
#: paperless/models.py:244
msgid "Sets the ASN barcode prefix"
msgstr ""
#: paperless/models.py:269
#: paperless/models.py:252
msgid "Sets the barcode upscale factor"
msgstr ""
#: paperless/models.py:276
#: paperless/models.py:259
msgid "Sets the barcode DPI"
msgstr ""
#: paperless/models.py:283
#: paperless/models.py:266
msgid "Sets the maximum pages for barcode"
msgstr ""
#: paperless/models.py:290
#: paperless/models.py:273
msgid "Enables tag barcode"
msgstr ""
#: paperless/models.py:296
#: paperless/models.py:279
msgid "Sets the tag barcode mapping"
msgstr ""
#: paperless/models.py:302
#: paperless/models.py:285
msgid "Enables splitting on tag barcodes"
msgstr ""
#: paperless/models.py:312
msgid "Sets the remote OCR engine"
msgstr ""
#: paperless/models.py:321
msgid "Sets the remote OCR API key"
msgstr ""
#: paperless/models.py:329
msgid "Sets the remote OCR endpoint"
msgstr ""
#: paperless/models.py:337
msgid "Sets which documents are sent to the remote OCR engine"
msgstr ""
#: paperless/models.py:349
#: paperless/models.py:294
msgid "Enables AI features"
msgstr ""
#: paperless/models.py:355
#: paperless/models.py:300
msgid "Sets the LLM embedding backend"
msgstr ""
#: paperless/models.py:363
#: paperless/models.py:308
msgid "Sets the LLM embedding model"
msgstr ""
#: paperless/models.py:370
#: paperless/models.py:315
msgid "Sets the LLM embedding endpoint, optional"
msgstr ""
#: paperless/models.py:377
#: paperless/models.py:322
msgid "Sets the LLM embedding chunk size"
msgstr ""
#: paperless/models.py:383
#: paperless/models.py:328
msgid "Sets the LLM context size"
msgstr ""
#: paperless/models.py:389
#: paperless/models.py:334
msgid "Sets the LLM backend"
msgstr ""
#: paperless/models.py:397
#: paperless/models.py:342
msgid "Sets the LLM model"
msgstr ""
#: paperless/models.py:404
#: paperless/models.py:349
msgid "Sets the LLM API key"
msgstr ""
#: paperless/models.py:411
#: paperless/models.py:356
msgid "Sets the LLM endpoint, optional"
msgstr ""
#: paperless/models.py:418
#: paperless/models.py:363
msgid "Sets the LLM output language"
msgstr ""
#: paperless/models.py:425
#: paperless/models.py:370
msgid "Sets the LLM timeout in seconds"
msgstr ""
#: paperless/models.py:431
#: paperless/models.py:376
msgid "paperless application settings"
msgstr ""
+14
View File
@@ -337,6 +337,20 @@ def check_deprecated_v2_ocr_env_vars(
return warnings
@register()
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
):
return [
Error(
"Azure AI remote parser requires endpoint and API key to be configured.",
),
]
return []
def get_tesseract_langs():
proc = subprocess.run(
[shutil.which("tesseract"), "--list-langs"],
-40
View File
@@ -9,7 +9,6 @@ from paperless.models import CleanChoices
from paperless.models import ColorConvertChoices
from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices
from paperless.models import RemoteOCRMode
@dataclasses.dataclass
@@ -186,45 +185,6 @@ class GeneralConfig(BaseConfig):
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
@dataclasses.dataclass
class RemoteOCRConfig(BaseConfig):
"""
Settings for the remote (cloud) OCR parser
"""
remote_ocr_engine: str | None = dataclasses.field(init=False)
remote_ocr_api_key: str | None = dataclasses.field(init=False)
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
def __post_init__(self) -> None:
app_config = self._get_config_instance()
self.remote_ocr_engine = (
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
)
self.remote_ocr_api_key = (
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
)
self.remote_ocr_endpoint = (
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
)
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
settings.REMOTE_OCR_MODE,
)
@property
def remote_ocr_by_default(self) -> bool:
"""
Whether every supported document goes to the remote engine.
When False the remote engine is used only for documents that
explicitly asked for it, i.e. a workflow matched during consumption or
the user ticked the box when reprocessing.
"""
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
@dataclasses.dataclass
class AIConfig(BaseConfig):
"""
@@ -1,44 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 14:37
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_api_key",
field=models.CharField(
blank=True,
max_length=1024,
null=True,
verbose_name="Sets the remote OCR API key",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_endpoint",
field=models.CharField(
blank=True,
max_length=256,
null=True,
verbose_name="Sets the remote OCR endpoint",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_engine",
field=models.CharField(
blank=True,
choices=[("azureai", "Azure AI Document Intelligence")],
max_length=32,
null=True,
verbose_name="Sets the remote OCR engine",
),
),
]
@@ -1,27 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 15:43
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_mode",
field=models.CharField(
blank=True,
choices=[
("always", "All supported documents"),
("workflow_only", "Only when a workflow enables it"),
],
max_length=32,
null=True,
verbose_name="Sets which documents are sent to the remote OCR engine",
),
),
]
-55
View File
@@ -74,23 +74,6 @@ class ColorConvertChoices(models.TextChoices):
CMYK = ("CMYK", _("CMYK"))
class RemoteOCREngine(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_ENGINE
"""
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
class RemoteOCRMode(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_MODE
"""
ALWAYS = ("always", _("All supported documents"))
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
class LLMEmbeddingBackend(models.TextChoices):
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
HUGGINGFACE = ("huggingface", _("Huggingface"))
@@ -303,44 +286,6 @@ class ApplicationConfiguration(AbstractSingletonModel):
null=True,
)
"""
Settings for the remote OCR parser
"""
# PAPERLESS_REMOTE_OCR_ENGINE
remote_ocr_engine = models.CharField(
verbose_name=_("Sets the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCREngine.choices,
)
# PAPERLESS_REMOTE_OCR_API_KEY
remote_ocr_api_key = models.CharField(
verbose_name=_("Sets the remote OCR API key"),
blank=True,
null=True,
max_length=1024,
)
# PAPERLESS_REMOTE_OCR_ENDPOINT
remote_ocr_endpoint = models.CharField(
verbose_name=_("Sets the remote OCR endpoint"),
blank=True,
null=True,
max_length=256,
)
# PAPERLESS_REMOTE_OCR_MODE
remote_ocr_mode = models.CharField(
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCRMode.choices,
)
"""
AI related settings
"""
-9
View File
@@ -134,11 +134,6 @@ class ParserProtocol(Protocol):
Author or organisation name.
url : str
URL for documentation, source code, or issue tracker.
Parsers that send document content to a remote service should additionally
set ``uses_remote_service = True`` so the registry can exclude them when
remote processing has not been requested for a document. The attribute is
optional so a parser that omits it is treated as fully local.
"""
# ------------------------------------------------------------------
@@ -150,10 +145,6 @@ class ParserProtocol(Protocol):
author: str
url: str
# NOTE: uses_remote_service is not declared here, the registry reads it
# with getattr(cls, ..., False) for backwards-compatibility with existing
# parsers
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
-14
View File
@@ -334,8 +334,6 @@ class ParserRegistry:
mime_type: str,
filename: str,
path: Path | None = None,
*,
allow_remote: bool = True,
) -> type[ParserProtocol] | None:
"""Return the best parser class for the given file, or None.
@@ -361,11 +359,6 @@ class ParserRegistry:
path:
Optional filesystem path to the file. Forwarded to each
parser's score method.
allow_remote:
When False, parsers that declare ``uses_remote_service = True``
are excluded from consideration, so a document is never sent to
a remote service. Parsers that do not declare the attribute
are treated as local and are always considered.
Returns
-------
@@ -381,13 +374,6 @@ class ParserRegistry:
if mime_type not in parser_class.supported_mime_types():
continue
if not allow_remote and getattr(
parser_class,
"uses_remote_service",
False,
):
continue
score = parser_class.score(mime_type, filename, path)
if score is None:
continue
+10 -19
View File
@@ -61,18 +61,6 @@ class RemoteEngineConfig:
self.api_key = api_key
self.endpoint = endpoint
@classmethod
def from_app_config(cls) -> Self:
"""Build the config from the app config, falling back to the env."""
from paperless.config import RemoteOCRConfig
app_config = RemoteOCRConfig()
return cls(
engine=app_config.remote_ocr_engine,
api_key=app_config.remote_ocr_api_key,
endpoint=app_config.remote_ocr_endpoint,
)
def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured."""
return (
@@ -102,9 +90,6 @@ class RemoteDocumentParser:
Maintainer name.
url : str
Issue tracker / source URL.
uses_remote_service : bool
Content is sent to a remote service, True so that the registry
can skip this parser if remote processing was not requested.
"""
name: str = "Paperless-ngx Remote OCR Parser"
@@ -112,8 +97,6 @@ class RemoteDocumentParser:
author: str = "Paperless-ngx Contributors"
url: str = "https://github.com/paperless-ngx/paperless-ngx"
uses_remote_service: bool = True
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
@@ -162,7 +145,11 @@ class RemoteDocumentParser:
20 when the remote engine is configured and the MIME type is
supported, otherwise None.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
return None
if mime_type not in _SUPPORTED_MIME_TYPES:
@@ -257,7 +244,11 @@ class RemoteDocumentParser:
Whether an archive copy is wanted. For PDFs, False skips the
remote engine and uses locally-extracted text instead.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
logger.warning(
+5 -14
View File
@@ -219,13 +219,6 @@ class ApplicationConfigurationSerializer(
allow_null=True,
max_length=1024,
)
remote_ocr_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
max_length=1024,
)
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
def run_validation(self, data):
# Empty strings treated as None to avoid unexpected behavior
@@ -237,13 +230,11 @@ class ApplicationConfigurationSerializer(
data["language"] = None
if "llm_output_language" in data and data["llm_output_language"] == "":
data["llm_output_language"] = None
for field in self.OBFUSCATED_FIELDS:
if field in data and data[field] is not None:
if data[field] == "":
data[field] = None
# Not a real value, don't overwrite the stored one
elif len(data[field].replace("*", "")) == 0:
del data[field]
if "llm_api_key" in data and data["llm_api_key"] is not None:
if data["llm_api_key"] == "":
data["llm_api_key"] = None
elif len(data["llm_api_key"].replace("*", "")) == 0:
del data["llm_api_key"]
return super().run_validation(data)
def update(self, instance, validated_data):
-5
View File
@@ -1203,11 +1203,6 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
REMOTE_OCR_MODE = get_choice_from_env(
"PAPERLESS_REMOTE_OCR_MODE",
{"always", "workflow_only"},
default="always",
)
################################################################################
# AI Settings #
+2 -24
View File
@@ -114,26 +114,7 @@ def remote_parser() -> Generator[RemoteDocumentParser, None, None]:
@pytest.fixture()
def empty_remote_ocr_app_config(mocker: MockerFixture) -> MagicMock:
# empty app config without accessing db
app_config = mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
return app_config
@pytest.fixture()
def azure_settings(
settings: SettingsWrapper,
empty_remote_ocr_app_config: MagicMock,
) -> SettingsWrapper:
def azure_settings(settings: SettingsWrapper) -> SettingsWrapper:
"""Configure Django settings for a valid Azure AI OCR engine.
Sets ``REMOTE_OCR_ENGINE``, ``REMOTE_OCR_API_KEY``, and
@@ -152,10 +133,7 @@ def azure_settings(
@pytest.fixture()
def no_engine_settings(
settings: SettingsWrapper,
empty_remote_ocr_app_config: MagicMock,
) -> SettingsWrapper:
def no_engine_settings(settings: SettingsWrapper) -> SettingsWrapper:
"""Configure Django settings with no remote engine configured.
Returns
@@ -21,7 +21,6 @@ from unittest.mock import Mock
import pytest
from documents.parsers import ParseError
from paperless.models import ApplicationConfiguration
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
from paperless.parsers.remote import RemoteDocumentParser
@@ -199,21 +198,21 @@ class TestRemoteParserScore:
def test_score_returns_none_when_api_key_missing(
self,
no_engine_settings: SettingsWrapper,
settings: SettingsWrapper,
) -> None:
no_engine_settings.REMOTE_OCR_ENGINE = "azureai"
no_engine_settings.REMOTE_OCR_ENDPOINT = (
"https://test.cognitiveservices.azure.com"
)
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = None
settings.REMOTE_OCR_ENDPOINT = "https://test.cognitiveservices.azure.com"
result = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert result is None
def test_score_returns_none_when_endpoint_missing(
self,
no_engine_settings: SettingsWrapper,
settings: SettingsWrapper,
) -> None:
no_engine_settings.REMOTE_OCR_ENGINE = "azureai"
no_engine_settings.REMOTE_OCR_API_KEY = "key"
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = "key"
settings.REMOTE_OCR_ENDPOINT = None
result = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert result is None
@@ -228,24 +227,6 @@ class TestRemoteParserScore:
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert score is not None and score > 10
@pytest.mark.django_db
def test_score_uses_app_config_when_env_unset(
self,
settings: SettingsWrapper,
) -> None:
"""The app config alone is enough to activate the parser."""
settings.REMOTE_OCR_ENGINE = None
settings.REMOTE_OCR_API_KEY = None
settings.REMOTE_OCR_ENDPOINT = None
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_engine = "azureai"
config.remote_ocr_api_key = "app-config-key"
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
config.save()
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
# ---------------------------------------------------------------------------
# Properties
@@ -1277,8 +1277,6 @@ class TestParserFileTypes:
# ---------------------------------------------------------------------------
# Remote ocr config from ApplicationConfiguration needs DB access
@pytest.mark.django_db
class TestRasterisedDocumentParserRegistry:
def test_registered_in_defaults(self) -> None:
from paperless.parsers.registry import ParserRegistry
+26
View File
@@ -15,6 +15,7 @@ from paperless.checks import audit_log_check
from paperless.checks import binaries_check
from paperless.checks import check_default_language_available
from paperless.checks import check_deprecated_db_settings
from paperless.checks import check_remote_parser_configured
from paperless.checks import check_v3_minimum_upgrade_version
from paperless.checks import debug_mode_check
from paperless.checks import paths_check
@@ -630,6 +631,31 @@ class TestV3MinimumUpgradeVersionCheck:
assert check_v3_minimum_upgrade_version(None) == []
class TestRemoteParserChecks:
def test_no_engine(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
msgs = check_remote_parser_configured(None)
assert len(msgs) == 0
def test_azure_no_endpoint(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = "somekey"
settings.REMOTE_OCR_ENDPOINT = None
msgs = check_remote_parser_configured(None)
assert len(msgs) == 1
msg = msgs[0]
assert (
"Azure AI remote parser requires endpoint and API key to be configured."
in msg.msg
)
class TestTesseractChecks:
def test_default_language(self) -> None:
check_default_language_available(None)
-118
View File
@@ -468,124 +468,6 @@ class TestParserRegistryGetParserForFile:
assert result is AcceptingBuiltin
class TestParserRegistryRemoteParsers:
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
@staticmethod
def _remote_parser_cls() -> type:
class RemoteParser:
name = "remote"
version = "1.0"
author = "A"
url = "https://example.com/remote"
uses_remote_service = True
@classmethod
def supported_mime_types(cls):
return {"text/plain": ".txt"}
@classmethod
def score(cls, mime_type, filename, path=None):
return 20
return RemoteParser
def test_remote_parser_wins_when_remote_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=True.
THEN: The remote parser is returned.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=True,
)
assert result is remote_parser_cls
def test_remote_parser_skipped_when_remote_not_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: The local parser is returned despite its lower score.
"""
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
"""
GIVEN: A registry whose only candidate declares uses_remote_service.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: None is returned the remote parser is never used as a
fallback when remote processing was not requested.
"""
registry = ParserRegistry()
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is None
def test_parser_without_attribute_treated_as_local(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A third-party parser predating uses_remote_service, so it does
not declare the attribute at all.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: It is still considered, i.e. treated as fully local, rather
than raising AttributeError.
"""
assert not hasattr(dummy_parser_cls, "uses_remote_service")
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_remote_allowed_by_default(self) -> None:
"""
GIVEN: A registry containing only a remote parser.
WHEN: get_parser_for_file() is called without allow_remote.
THEN: The remote parser is returned callers that do not opt in to
the filter keep the previous behaviour.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file("text/plain", "readme.txt")
assert result is remote_parser_cls
class TestDiscover:
"""Verify entrypoint discovery in ParserRegistry.discover()."""
@@ -1,113 +0,0 @@
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from django.test import override_settings
from paperless.config import RemoteOCRConfig
from paperless.models import RemoteOCRMode
if TYPE_CHECKING:
from unittest.mock import MagicMock
@pytest.fixture()
def null_app_config(mocker) -> MagicMock:
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
return mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
@pytest.fixture()
def make_remote_ocr_config(mocker):
def _make(app_config, **django_settings_overrides):
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
with override_settings(**django_settings_overrides):
return RemoteOCRConfig()
return _make
class TestRemoteOCRConfig:
def test_falls_back_to_settings(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "env-key"
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_app_config_takes_precedence(
self,
make_remote_ocr_config,
mocker,
) -> None:
app_config = mocker.MagicMock(
remote_ocr_engine="azureai",
remote_ocr_api_key="db-key",
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
)
cfg = make_remote_ocr_config(
app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "db-key"
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_unset_everywhere(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
assert cfg.remote_ocr_engine is None
assert cfg.remote_ocr_api_key is None
assert cfg.remote_ocr_endpoint is None
class TestRemoteOCRByDefault:
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_by_default is True
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_by_default is False
+52 -81
View File
@@ -5,12 +5,13 @@ from django.conf import settings
from django.contrib.auth.models import User
from documents.models import Document
from documents.permissions import permitted_object_ids
from documents.permissions import get_objects_for_user_owner_aware
from paperless.config import AIConfig
from paperless_ai.base_model import ClassificationSuggestions
from paperless_ai.base_model import TaxonomyChoiceDict
from paperless_ai.client import AIClient
from paperless_ai.db import db_connection_released
from paperless_ai.indexing import _node_document_ids
from paperless_ai.indexing import retrieve_similar_nodes
from paperless_ai.indexing import truncate_content
from paperless_ai.prompts.context import ClassificationPromptContext
@@ -18,9 +19,7 @@ from paperless_ai.prompts.context import LocalizationPromptContext
from paperless_ai.prompts.context import RagContextPromptContext
from paperless_ai.prompts.render import render_prompt
from paperless_ai.taxonomy import AssignedMetadata
from paperless_ai.taxonomy import SimilarDocument
from paperless_ai.taxonomy import TaxonomyCandidates
from paperless_ai.taxonomy import _node_document_weights
from paperless_ai.taxonomy import build_taxonomy_candidates
from paperless_ai.taxonomy import empty_taxonomy_candidates
from paperless_ai.taxonomy import format_taxonomy_for_prompt
@@ -40,35 +39,6 @@ logger = logging.getLogger("paperless_ai.rag_classifier")
TAXONOMY_CANDIDATE_TOP_K = 15
def _fulltext_similar_documents(
document: Document,
user: User | None,
top_k: int,
) -> list[SimilarDocument]:
"""Rank-based fallback when no embedding backend is configured. Uses
Tantivy's "More Like This" (term-overlap similarity) instead of vector
similarity - cruder, but far better than no candidates at all.
more_like_this_ids returns only a ranked ID list, no scores, so weight is
synthesized from rank (descending from top_k) rather than claiming a
similarity magnitude that doesn't exist. A superuser is normalized to
``None`` before calling, since the backend's permission filter has no
superuser short-circuit of its own.
"""
from documents.search import get_backend
search_user = None if user is not None and user.is_superuser else user
backend = get_backend()
similar_ids = backend.more_like_this_ids(
document.pk,
user=search_user,
limit=top_k,
)
return [
SimilarDocument(document_id=doc_id, weight=float(top_k - rank))
for rank, doc_id in enumerate(similar_ids)
]
def get_language_name(language_code: str) -> str:
normalized_language_code = language_code.lower()
for code, name in settings.LANGUAGES:
@@ -161,52 +131,44 @@ def get_taxonomy_context(
user: User | None = None,
max_docs: int = 5,
) -> tuple[TaxonomyCandidates, AssignedMetadata, str]:
"""One retrieval feeds both taxonomy candidates and RAG text context. Uses
vector similarity when an embedding backend is configured, otherwise
falls back to Tantivy full-text "More Like This" similarity - see
_fulltext_similar_documents. On any retrieval failure, degrades to empty
candidates/context rather than propagating the exception - neither a
vector-store outage nor a search-index issue should block classification,
only its context-assisted enrichment.
"""One retrieval feeds both taxonomy candidates and RAG text context.
On any retrieval failure, degrades to empty candidates/context rather than
propagating the exception - a vector-store outage should not block
classification, only its RAG-assisted enrichment.
"""
assigned = get_assigned_metadata(document, user)
ai_config = AIConfig()
try:
if ai_config.llm_embedding_backend:
# None means "no restriction" to retrieve_similar_nodes. A superuser
# (like no user at all) can see every document, so skip materializing
# every visible pk into a Python list and passing it through as an IN
# filter: for a large library that is a wasted quadratic scan in the
# vector store at best, and past ~32,763 documents a hard
# sqlite3.OperationalError (SQLite's bound-parameter limit) at worst.
# permitted_object_ids() has its own superuser shortcut that would
# return every Document's id anyway, so this changes nothing about
# which documents are considered -- only how we get there.
visible_document_ids = (
None
if user is None or user.is_superuser
else list(permitted_object_ids(user, Document, "view_document"))
)
nodes = retrieve_similar_nodes(
document,
top_k=TAXONOMY_CANDIDATE_TOP_K,
document_ids=visible_document_ids,
)
similar_documents = _node_document_weights(nodes)
else:
# See _fulltext_similar_documents: it applies its own permission
# filter via `user`, so no visible-document-id list is needed here.
similar_documents = _fulltext_similar_documents(
document,
user,
top_k=TAXONOMY_CANDIDATE_TOP_K,
# None means "no restriction" to retrieve_similar_nodes. A superuser
# (like no user at all) can see every document, so skip materializing
# every visible pk into a Python list and passing it through as an IN
# filter: for a large library that is a wasted quadratic scan in the
# vector store at best, and past ~32,763 documents a hard
# sqlite3.OperationalError (SQLite's bound-parameter limit) at worst.
# get_objects_for_user_owner_aware() would return every Document for a
# superuser anyway (guardian's own with_superuser shortcut), so this
# changes nothing about which documents are considered -- only how we
# get there.
visible_document_ids = (
None
if user is None or user.is_superuser
else list(
get_objects_for_user_owner_aware(
user,
"view_document",
Document,
).values_list("pk", flat=True),
)
)
nodes = retrieve_similar_nodes(
document,
top_k=TAXONOMY_CANDIDATE_TOP_K,
document_ids=visible_document_ids,
)
candidates = build_taxonomy_candidates(similar_documents, user)
candidates = build_taxonomy_candidates(nodes, user)
similar_doc_ids = [s["document_id"] for s in similar_documents]
similar_docs = list(
Document.objects.filter(pk__in=similar_doc_ids)[:max_docs],
Document.objects.filter(pk__in=_node_document_ids(nodes))[:max_docs],
)
context_blocks = []
for similar in similar_docs:
@@ -215,8 +177,8 @@ def get_taxonomy_context(
context_blocks.append(f"TITLE: {title}\n{text}")
except Exception:
logger.exception(
"Failed to retrieve similar-document context for document %s; "
"continuing without taxonomy candidates or similar-document context.",
"Failed to retrieve RAG neighbours for document %s; continuing "
"without taxonomy candidates or similar-document context.",
document.pk,
)
return empty_taxonomy_candidates(), assigned, ""
@@ -299,14 +261,23 @@ def get_ai_document_classification(
) -> ClassificationSuggestions:
ai_config = AIConfig()
candidates, assigned, context = get_taxonomy_context(document, user)
prompt = build_prompt_with_rag(
document,
ai_config,
candidates=candidates,
assigned=assigned,
context=context,
)
if ai_config.llm_embedding_backend:
candidates, assigned, context = get_taxonomy_context(document, user)
prompt = build_prompt_with_rag(
document,
ai_config,
candidates=candidates,
assigned=assigned,
context=context,
)
else:
candidates = empty_taxonomy_candidates()
prompt = build_prompt_without_rag(
document,
ai_config,
candidates=candidates,
assigned=get_assigned_metadata(document, user),
)
client = AIClient()
# Hand the pooled DB connection back while the (slow) LLM query runs so it
+14 -27
View File
@@ -33,11 +33,6 @@ class TaxonomyCandidate(TypedDict):
weight: float
class SimilarDocument(TypedDict):
document_id: int
weight: float
class TaxonomyCandidates(TypedDict):
tags: list[TaxonomyCandidate]
document_types: list[TaxonomyCandidate]
@@ -110,10 +105,10 @@ def get_assigned_metadata(document: Document, user: User | None) -> AssignedMeta
)
def _node_document_weights(nodes: list["NodeWithScore"]) -> list[SimilarDocument]:
"""Sum each node's similarity score into its document_id (a document can
appear via multiple chunks/nodes) and return one SimilarDocument per
distinct document_id."""
def _node_document_weights(nodes: list["NodeWithScore"]) -> dict[int, float]:
"""document_id -> that node's similarity score, summed if a document_id
appears more than once across the retrieved nodes (e.g. multiple chunks
of the same source document)."""
weights: dict[int, float] = defaultdict(float)
for node in nodes:
document_id = node.metadata.get("document_id")
@@ -126,10 +121,7 @@ def _node_document_weights(nodes: list["NodeWithScore"]) -> list[SimilarDocument
weights[int(document_id)] += float(node.score or 0.0)
except (TypeError, ValueError): # pragma: no cover
continue
return [
SimilarDocument(document_id=document_id, weight=weight)
for document_id, weight in weights.items()
]
return weights
def _visible_ranked_candidates(
@@ -165,25 +157,20 @@ def _visible_ranked_candidates(
def build_taxonomy_candidates(
similar_documents: list[SimilarDocument],
nodes: list["NodeWithScore"],
user: User | None,
) -> TaxonomyCandidates:
"""Resolve each similar document's id to a live Document, read its
*current* tags/type/correspondent/storage_path via the ORM (never any
possibly-stale names an adapter's source might have cached), weight each
distinct taxonomy object by aggregate similarity weight, permission-filter
"""Resolve each neighbour node's document_id to a live Document, read its
*current* tags/type/correspondent/storage_path via the ORM (never the
possibly-stale names cached in vector-index node metadata), weight each
distinct taxonomy object by aggregate neighbour similarity, permission-filter
against what ``user`` can see, and return each category ranked by weight
and capped. ``similar_documents`` may come from either the vector-RAG
adapter or the full-text fallback adapter - both produce this same shape.
and capped.
"""
if not similar_documents:
return empty_taxonomy_candidates()
# Both adapters guarantee at most one SimilarDocument per document_id, so
# this never silently drops a duplicate's weight.
document_weights: dict[int, float] = {
s["document_id"]: s["weight"] for s in similar_documents
}
document_weights = _node_document_weights(nodes)
if not document_weights:
return empty_taxonomy_candidates()
# Only .tags.all() needs prefetching (a reverse M2M, one extra query for
# the whole batch). document_type/correspondent/storage_path are read
+22 -256
View File
@@ -1,4 +1,3 @@
from collections.abc import Generator
from types import SimpleNamespace
from unittest.mock import MagicMock
from unittest.mock import patch
@@ -8,13 +7,10 @@ import pytest_mock
from django.test import override_settings
from documents.models import Document
from documents.search import TantivyBackend
from documents.tests.factories import DocumentFactory
from documents.tests.factories import TagFactory
from documents.tests.factories import UserFactory
from paperless.config import AIConfig
from paperless_ai.ai_classifier import TAXONOMY_CANDIDATE_TOP_K
from paperless_ai.ai_classifier import _fulltext_similar_documents
from paperless_ai.ai_classifier import _restrict_to_shown_candidates
from paperless_ai.ai_classifier import build_localization_prompt
from paperless_ai.ai_classifier import build_prompt_with_rag
@@ -24,7 +20,6 @@ from paperless_ai.ai_classifier import get_language_name
from paperless_ai.ai_classifier import get_taxonomy_context
from paperless_ai.base_model import ClassificationSuggestions
from paperless_ai.base_model import TaxonomyChoiceDict
from paperless_ai.taxonomy import SimilarDocument
from paperless_ai.taxonomy import TaxonomyCandidate
from paperless_ai.taxonomy import TaxonomyCandidates
from paperless_ai.taxonomy import empty_taxonomy_candidates
@@ -209,10 +204,12 @@ def test_use_rag_if_configured(
@pytest.mark.django_db
@patch("paperless_ai.client.AIClient.run_llm_query")
@patch("paperless_ai.ai_classifier.build_prompt_with_rag")
@patch("paperless_ai.ai_classifier.build_prompt_without_rag")
@patch("paperless_ai.ai_classifier.AIConfig")
@override_settings(LLM_BACKEND="ollama", LLM_MODEL="some_model")
def test_use_rag_prompt_even_without_embedding_backend(
mock_build_prompt_with_rag,
def test_use_without_rag_if_not_configured(
mock_ai_config,
mock_build_prompt_without_rag,
mock_run_llm_query,
mock_document,
):
@@ -222,13 +219,13 @@ def test_use_rag_prompt_even_without_embedding_backend(
WHEN:
- get_ai_document_classification() is called
THEN:
- The RAG-context prompt builder is still used (fed by the full-text
fallback's context/candidates instead of the vector store's)
- The non-RAG prompt builder is used
"""
mock_build_prompt_with_rag.return_value = "Prompt with RAG"
mock_ai_config.return_value.llm_embedding_backend = None
mock_build_prompt_without_rag.return_value = "Prompt without RAG"
mock_run_llm_query.return_value = NESTED_SUGGESTIONS
get_ai_document_classification(mock_document)
mock_build_prompt_with_rag.assert_called_once()
mock_build_prompt_without_rag.assert_called_once()
@pytest.mark.django_db
@@ -306,7 +303,6 @@ def test_build_localization_prompt_preserves_unicode_characters():
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
def test_get_taxonomy_context_assembles_rag_text_and_candidates():
"""
GIVEN:
@@ -348,7 +344,6 @@ def test_get_taxonomy_context_assembles_rag_text_and_candidates():
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
def test_get_taxonomy_context_no_similar_docs():
"""
GIVEN:
@@ -372,67 +367,6 @@ def test_get_taxonomy_context_no_similar_docs():
}
@pytest.mark.django_db
def test_get_taxonomy_context_uses_fulltext_fallback_when_no_embedding_backend(
mocker: pytest_mock.MockerFixture,
) -> None:
"""
GIVEN:
- No LLM embedding backend is configured (the default test settings)
WHEN:
- get_taxonomy_context() is called
THEN:
- _fulltext_similar_documents() is called with the document, the user
and TAXONOMY_CANDIDATE_TOP_K
- retrieve_similar_nodes() (the vector path) is never called
"""
document = DocumentFactory.create(content="Some content")
mock_fulltext = mocker.patch(
"paperless_ai.ai_classifier._fulltext_similar_documents",
return_value=[],
)
mock_retrieve = mocker.patch("paperless_ai.ai_classifier.retrieve_similar_nodes")
get_taxonomy_context(document, user=None)
mock_fulltext.assert_called_once_with(
document,
None,
top_k=TAXONOMY_CANDIDATE_TOP_K,
)
mock_retrieve.assert_not_called()
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
def test_get_taxonomy_context_uses_vector_path_when_embedding_backend_configured(
mocker: pytest_mock.MockerFixture,
) -> None:
"""
GIVEN:
- An LLM embedding backend is configured
WHEN:
- get_taxonomy_context() is called
THEN:
- retrieve_similar_nodes() (the vector path) is called
- _fulltext_similar_documents() (the no-embedding-backend fallback)
is never called
"""
document = DocumentFactory.create(content="Some content")
mock_retrieve = mocker.patch(
"paperless_ai.ai_classifier.retrieve_similar_nodes",
return_value=[],
)
mock_fulltext = mocker.patch(
"paperless_ai.ai_classifier._fulltext_similar_documents",
)
get_taxonomy_context(document, user=None)
mock_retrieve.assert_called_once()
mock_fulltext.assert_not_called()
class TestGetTaxonomyContextVisibility:
"""get_taxonomy_context must not materialize every visible document id
for a user who can already see the whole library: a superuser (like no
@@ -445,7 +379,6 @@ class TestGetTaxonomyContextVisibility:
"""
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
def test_skips_permission_lookup_for_superuser(
self,
mocker: pytest_mock.MockerFixture,
@@ -464,18 +397,17 @@ class TestGetTaxonomyContextVisibility:
"paperless_ai.ai_classifier.retrieve_similar_nodes",
return_value=[],
)
mock_permitted = mocker.patch(
"paperless_ai.ai_classifier.permitted_object_ids",
mock_get_objects = mocker.patch(
"paperless_ai.ai_classifier.get_objects_for_user_owner_aware",
)
user = UserFactory.create(is_superuser=True)
get_taxonomy_context(document, user)
mock_permitted.assert_not_called()
mock_get_objects.assert_not_called()
assert mock_retrieve.call_args.kwargs["document_ids"] is None
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
def test_skips_permission_lookup_when_no_user(
self,
mocker: pytest_mock.MockerFixture,
@@ -494,17 +426,16 @@ class TestGetTaxonomyContextVisibility:
"paperless_ai.ai_classifier.retrieve_similar_nodes",
return_value=[],
)
mock_permitted = mocker.patch(
"paperless_ai.ai_classifier.permitted_object_ids",
mock_get_objects = mocker.patch(
"paperless_ai.ai_classifier.get_objects_for_user_owner_aware",
)
get_taxonomy_context(document, None)
mock_permitted.assert_not_called()
mock_get_objects.assert_not_called()
assert mock_retrieve.call_args.kwargs["document_ids"] is None
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
def test_restricts_to_visible_documents_for_non_superuser(
self,
mocker: pytest_mock.MockerFixture,
@@ -515,7 +446,7 @@ class TestGetTaxonomyContextVisibility:
WHEN:
- get_taxonomy_context() is called
THEN:
- The user's permitted document ids are looked up and passed to
- The user's visible document ids are looked up and passed to
retrieve_similar_nodes() as a restriction
"""
document = DocumentFactory.create(content="Some content")
@@ -523,185 +454,21 @@ class TestGetTaxonomyContextVisibility:
"paperless_ai.ai_classifier.retrieve_similar_nodes",
return_value=[],
)
mock_permitted = mocker.patch(
"paperless_ai.ai_classifier.permitted_object_ids",
return_value=[1, 2, 3],
mock_queryset = mocker.MagicMock()
mock_queryset.values_list.return_value = [1, 2, 3]
mock_get_objects = mocker.patch(
"paperless_ai.ai_classifier.get_objects_for_user_owner_aware",
return_value=mock_queryset,
)
user = UserFactory.create(is_superuser=False)
get_taxonomy_context(document, user)
mock_permitted.assert_called_once_with(user, Document, "view_document")
mock_get_objects.assert_called_once_with(user, "view_document", Document)
assert mock_retrieve.call_args.kwargs["document_ids"] == [1, 2, 3]
@pytest.mark.django_db
class TestFulltextSimilarDocuments:
"""_fulltext_similar_documents is the no-embedding-backend fallback: it
asks the Tantivy full-text index for "More Like This" neighbours instead
of the vector store, and synthesizes a rank-based weight since Tantivy's
more_like_this_ids returns only an ordered id list, no scores.
"""
@pytest.fixture
def fulltext_backend(
self,
mocker: pytest_mock.MockerFixture,
) -> Generator[TantivyBackend, None, None]:
"""An in-memory Tantivy backend, wired up as the module-level
singleton _fulltext_similar_documents resolves via get_backend()."""
backend = TantivyBackend(path=None)
backend.open()
mocker.patch("documents.search.get_backend", return_value=backend)
try:
yield backend
finally:
backend.close()
def test_ranks_by_rank_based_weight_descending(
self,
fulltext_backend: TantivyBackend,
) -> None:
"""
GIVEN:
- A source document and two similar documents indexed in Tantivy
WHEN:
- _fulltext_similar_documents() is called
THEN:
- Each result's weight reflects its rank (first result weighted
higher than the second), not a raw similarity score
"""
source = DocumentFactory.create(content="quarterly financial report details")
first = DocumentFactory.create(content="quarterly financial report details")
second = DocumentFactory.create(content="financial report")
for doc in (source, first, second):
fulltext_backend.add_or_update(doc)
result = _fulltext_similar_documents(source, user=None, top_k=5)
assert len(result) == 2
weight_by_id = {s["document_id"]: s["weight"] for s in result}
assert weight_by_id[first.pk] > weight_by_id[second.pk]
def test_excludes_source_document(
self,
fulltext_backend: TantivyBackend,
) -> None:
"""
GIVEN:
- A source document indexed in Tantivy with no other documents
WHEN:
- _fulltext_similar_documents() is called
THEN:
- An empty list is returned - the source document is never its
own similar document
"""
source = DocumentFactory.create(content="unique unrelated content")
fulltext_backend.add_or_update(source)
result = _fulltext_similar_documents(source, user=None, top_k=5)
assert result == []
def test_empty_index_returns_empty_list(
self,
fulltext_backend: TantivyBackend,
) -> None:
"""
GIVEN:
- A document that has never been indexed (fresh/empty Tantivy index)
WHEN:
- _fulltext_similar_documents() is called
THEN:
- An empty list is returned rather than raising
"""
source = DocumentFactory.create(content="never indexed")
result = _fulltext_similar_documents(source, user=None, top_k=5)
assert result == []
def test_respects_top_k_limit(
self,
fulltext_backend: TantivyBackend,
) -> None:
"""
GIVEN:
- A source document and four similar documents indexed
WHEN:
- _fulltext_similar_documents() is called with top_k=2
THEN:
- At most 2 results are returned
"""
source = DocumentFactory.create(content="shared overlapping keyword text")
for _ in range(4):
fulltext_backend.add_or_update(
DocumentFactory.create(content="shared overlapping keyword text"),
)
result = _fulltext_similar_documents(source, user=None, top_k=2)
assert len(result) <= 2
def test_result_shape_is_similar_document(
self,
fulltext_backend: TantivyBackend,
) -> None:
"""
GIVEN:
- A source document and one similar document indexed
WHEN:
- _fulltext_similar_documents() is called
THEN:
- Each result is a SimilarDocument (document_id + weight only)
"""
source = DocumentFactory.create(content="shared content phrase")
other = DocumentFactory.create(content="shared content phrase")
fulltext_backend.add_or_update(source)
fulltext_backend.add_or_update(other)
result = _fulltext_similar_documents(source, user=None, top_k=5)
# rank 0 (the only/best result) with top_k=5 -> weight = top_k - rank = 5.0,
# per the "first result gets top_k, the last gets 1" formula.
assert result == [SimilarDocument(document_id=other.pk, weight=5.0)]
def test_superuser_sees_other_users_documents(
self,
fulltext_backend: TantivyBackend,
) -> None:
"""
GIVEN:
- A source document owned by one user and a similar document
owned by a different user, with no sharing between them
WHEN:
- _fulltext_similar_documents() is called with a superuser
THEN:
- The other user's document is still returned as a similar
document - a superuser must not be narrowed by the backend's
owner-based permission filter
"""
owner = UserFactory.create()
other_owner = UserFactory.create()
superuser = UserFactory.create(is_superuser=True)
source = DocumentFactory.create(
content="shared content phrase",
owner=owner,
)
other = DocumentFactory.create(
content="shared content phrase",
owner=other_owner,
)
fulltext_backend.add_or_update(source)
fulltext_backend.add_or_update(other)
result = _fulltext_similar_documents(source, user=superuser, top_k=5)
assert [s["document_id"] for s in result] == [other.pk]
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
@patch("paperless_ai.ai_classifier.retrieve_similar_nodes")
def test_get_taxonomy_context_retrieval_failure_degrades_to_no_hints(mock_retrieve):
"""
@@ -728,7 +495,6 @@ def test_get_taxonomy_context_retrieval_failure_degrades_to_no_hints(mock_retrie
@pytest.mark.django_db
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
@patch("paperless_ai.ai_classifier.build_taxonomy_candidates")
@patch("paperless_ai.ai_classifier.retrieve_similar_nodes")
def test_get_taxonomy_context_candidate_building_failure_degrades_to_no_hints(
+31 -33
View File
@@ -1,4 +1,5 @@
import json
from types import SimpleNamespace
import pytest
import pytest_mock
@@ -10,7 +11,6 @@ from documents.tests.factories import StoragePathFactory
from documents.tests.factories import TagFactory
from documents.tests.factories import UserFactory
from paperless_ai.taxonomy import AssignedMetadata
from paperless_ai.taxonomy import SimilarDocument
from paperless_ai.taxonomy import TaxonomyCandidates
from paperless_ai.taxonomy import build_taxonomy_candidates
from paperless_ai.taxonomy import format_taxonomy_for_prompt
@@ -132,8 +132,9 @@ class TestGetAssignedMetadata:
assert result["tags"] == ["Owned By Someone Else"]
def make_similar(document_id: int, weight: float) -> SimilarDocument:
return SimilarDocument(document_id=document_id, weight=weight)
def make_node(document_id: int, score: float) -> SimpleNamespace:
"""A stand-in for NodeWithScore: only ``.metadata``/``.score`` are read."""
return SimpleNamespace(metadata={"document_id": str(document_id)}, score=score)
@pytest.mark.django_db
@@ -169,9 +170,9 @@ class TestBuildTaxonomyCandidates:
doc_a.tags.add(tag)
doc_b = DocumentFactory.create()
doc_b.tags.add(tag)
similar_documents = [make_similar(doc_a.pk, 0.9), make_similar(doc_b.pk, 0.4)]
nodes = [make_node(doc_a.pk, 0.9), make_node(doc_b.pk, 0.4)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["tags"]) == 1
assert result["tags"][0]["id"] == tag.pk
@@ -196,9 +197,9 @@ class TestBuildTaxonomyCandidates:
document.tags.add(tag)
tag.name = "New Name"
tag.save()
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert result["tags"][0]["name"] == "New Name"
@@ -218,9 +219,9 @@ class TestBuildTaxonomyCandidates:
document = DocumentFactory.create()
document.tags.add(tag)
tag.delete()
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert result["tags"] == []
@@ -239,12 +240,9 @@ class TestBuildTaxonomyCandidates:
strong_doc.tags.add(strong_tag)
weak_doc = DocumentFactory.create()
weak_doc.tags.add(weak_tag)
similar_documents = [
make_similar(strong_doc.pk, 0.9),
make_similar(weak_doc.pk, 0.1),
]
nodes = [make_node(strong_doc.pk, 0.9), make_node(weak_doc.pk, 0.1)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert [c["name"] for c in result["tags"]] == ["Strong", "Weak"]
@@ -260,9 +258,9 @@ class TestBuildTaxonomyCandidates:
document = DocumentFactory.create()
for i in range(15):
document.tags.add(TagFactory.create(name=f"Tag{i}"))
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["tags"]) == 10
@@ -276,12 +274,12 @@ class TestBuildTaxonomyCandidates:
- Only 5 correspondents are returned
"""
correspondents = CorrespondentFactory.create_batch(7)
similar_documents = [
make_similar(DocumentFactory.create(correspondent=c).pk, 0.5)
nodes = [
make_node(DocumentFactory.create(correspondent=c).pk, 0.5)
for c in correspondents
]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["correspondents"]) == 5
@@ -296,9 +294,9 @@ class TestBuildTaxonomyCandidates:
"""
document_type = DocumentTypeFactory.create(name="Invoice")
document = DocumentFactory.create(document_type=document_type)
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["document_types"]) == 1
assert result["document_types"][0]["id"] == document_type.pk
@@ -314,12 +312,12 @@ class TestBuildTaxonomyCandidates:
- Only 5 document_types are returned
"""
document_types = DocumentTypeFactory.create_batch(7)
similar_documents = [
make_similar(DocumentFactory.create(document_type=dt).pk, 0.5)
nodes = [
make_node(DocumentFactory.create(document_type=dt).pk, 0.5)
for dt in document_types
]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["document_types"]) == 5
@@ -334,9 +332,9 @@ class TestBuildTaxonomyCandidates:
"""
storage_path = StoragePathFactory.create(name="Invoices")
document = DocumentFactory.create(storage_path=storage_path)
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["storage_paths"]) == 1
assert result["storage_paths"][0]["id"] == storage_path.pk
@@ -352,12 +350,12 @@ class TestBuildTaxonomyCandidates:
- Only 5 storage_paths are returned
"""
storage_paths = StoragePathFactory.create_batch(7)
similar_documents = [
make_similar(DocumentFactory.create(storage_path=sp).pk, 0.5)
nodes = [
make_node(DocumentFactory.create(storage_path=sp).pk, 0.5)
for sp in storage_paths
]
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert len(result["storage_paths"]) == 5
@@ -377,14 +375,14 @@ class TestBuildTaxonomyCandidates:
tag = TagFactory.create(name="Restricted")
document = DocumentFactory.create()
document.tags.add(tag)
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
user = UserFactory.create()
mocker.patch(
"documents.permissions.permitted_object_ids",
return_value=[], # user cannot see this tag
)
result = build_taxonomy_candidates(similar_documents, user=user)
result = build_taxonomy_candidates(nodes, user=user)
assert result["tags"] == []
@@ -414,10 +412,10 @@ class TestBuildTaxonomyCandidates:
tag.save()
document = DocumentFactory.create()
document.tags.add(tag)
similar_documents = [make_similar(document.pk, 0.5)]
nodes = [make_node(document.pk, 0.5)]
spy = mocker.patch("documents.permissions.permitted_object_ids")
result = build_taxonomy_candidates(similar_documents, user=None)
result = build_taxonomy_candidates(nodes, user=None)
assert result["tags"][0]["name"] == "Owned"
spy.assert_not_called()
-75
View File
@@ -253,81 +253,6 @@ class TestAPIMailAccounts(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["success"], True)
def test_mail_account_test_existing_no_global_perms(self) -> None:
"""
GIVEN:
- Existing account without an owner
- User without any mail account permissions
WHEN:
- API call is made to test the account by id
THEN:
- API returns forbidden
"""
account = MailAccountFactory(
username="admin",
password="secret",
imap_server="server.example.com",
imap_port=443,
owner=None,
)
user = User.objects.create_user(username="no_perms")
self.client.force_authenticate(user=user)
response = self.client.post(
f"{self.ENDPOINT}test/",
json.dumps(
{
"id": account.pk,
"imap_server": "server.example.com",
"imap_port": 443,
"imap_security": MailAccount.ImapSecurity.SSL,
"username": "admin",
"password": "******",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
self.assertEqual(response.content.decode(), "Insufficient permissions")
def test_mail_account_test_existing_object_perms_only(self) -> None:
"""
GIVEN:
- Existing account owned by another user
- User with an object level grant but no global change permission
WHEN:
- API call is made to test the account by id
THEN:
- API returns forbidden
"""
owner = User.objects.create_user(username="account_owner")
account = MailAccountFactory(
username="admin",
password="secret",
imap_server="server.example.com",
imap_port=443,
owner=owner,
)
user = User.objects.create_user(username="object_perms_only")
assign_perm("change_mailaccount", user, account)
self.client.force_authenticate(user=user)
response = self.client.post(
f"{self.ENDPOINT}test/",
json.dumps(
{
"id": account.pk,
"imap_server": "server.example.com",
"imap_port": 443,
"imap_security": MailAccount.ImapSecurity.SSL,
"username": "admin",
"password": "******",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
def test_mail_account_test_existing_nonexistent_id_forbidden(self) -> None:
response = self.client.post(
f"{self.ENDPOINT}test/",
+1 -3
View File
@@ -2195,9 +2195,7 @@ class TestMailAccountTestView(APITestCase):
password="testpassword",
)
self.user.user_permissions.add(
*Permission.objects.filter(
codename__in=["add_mailaccount", "change_mailaccount"],
),
*Permission.objects.filter(codename__in=["add_mailaccount"]),
)
self.user.save()
self.client.force_authenticate(user=self.user)
+1 -3
View File
@@ -106,9 +106,7 @@ class MailAccountViewSet(PassUserMixin, ModelViewSet[MailAccount]):
except (TypeError, ValueError, MailAccount.DoesNotExist):
return HttpResponseForbidden("Insufficient permissions")
if not request.user.has_perms(
["paperless_mail.change_mailaccount"],
) or not has_perms_owner_aware(
if not has_perms_owner_aware(
request.user,
"change_mailaccount",
existing_account,