Compare commits

..
Author SHA1 Message Date
stumpylog 79e8970b60 Chore: switch sqlite-vec to the paperless-ngx patched wheels
They are built from a patched fork that carries
upstream PRs including NEON on Linux arm64, memory-safety fixes and
performance improvements for both reading and cosine ranking
2026-11-12 10:32:19 -08:00
stumpylog e21a334881 Chore: restrict uv lock environments to supported architectures
Limit the lock to Linux x86_64, Linux aarch64 and macOS arm64. Locked
versions are unchanged; only the unused wheel entries are removed.
2026-11-12 10:25:12 -08:00
36 changed files with 522 additions and 2557 deletions

No files matched your search

+5 -30
View File
@@ -493,7 +493,7 @@ fields and permissions, which will be merged.
#### Types {#workflow-trigger-types}
Currently, there are five events that correspond to workflow trigger 'types':
Currently, there are four events that correspond to workflow trigger 'types':
1. **Consumption Started**: _before_ a document is consumed, so events can include filters by source (mail, consumption
folder or API), file path, file name, mail rule
@@ -505,10 +505,8 @@ Currently, there are five events that correspond to workflow trigger 'types':
4. **Scheduled**: a scheduled trigger that can be used to run workflows at a specific time. The date used can be either the document
added, created, updated date or you can specify a (date) custom field. You can also specify a day offset from the date (positive
offsets will trigger after the date, negative offsets will trigger before).
5. **Manual**: a trigger that only runs when you explicitly choose to run the workflow against one or more existing documents. See
[running a workflow manually](#running-a-workflow-manually) below.
The following flow diagram illustrates the document trigger types:
The following flow diagram illustrates the four document trigger types:
```mermaid
flowchart TD
@@ -528,10 +526,6 @@ flowchart TD
matching
trigger(s)"}
manual{"Matching
'Manual'
trigger(s)"}
A[New Document] --> consumption
consumption --> |Yes| C[Workflow Actions Run]
consumption --> |No| D
@@ -549,10 +543,6 @@ flowchart TD
scheduled --> |Yes| N[Workflow Actions Run]
scheduled --> |No| O[Document Saved]
N --> O
P[User Runs Workflow] --> manual
manual --> |Yes| Q[Workflow Actions Run]
manual --> |No| R[Document Saved]
Q --> R
```
#### Filters {#workflow-trigger-filters}
@@ -564,9 +554,9 @@ Workflows allow you to filter by:
- File path, including wildcards. Note that enabling `PAPERLESS_CONSUMER_RECURSIVE` would allow, for
example, automatically assigning documents to different owners based on the upload directory.
- Mail rule. Choosing this option will force 'mail fetch' to be the workflow source.
- Content matching (`Added`, `Updated`, `Scheduled` and `Manual` triggers only). Filter document content using the matching settings.
- Content matching (`Added`, `Updated` and `Scheduled` triggers only). Filter document content using the matching settings.
There are also 'advanced' filters available for `Added`, `Updated`, `Scheduled` and `Manual` triggers:
There are also 'advanced' filters available for `Added`, `Updated` and `Scheduled` triggers:
- Any Tags: Filter for documents with any of the specified tags.
- All Tags: Filter for documents with all of the specified tags.
@@ -579,21 +569,6 @@ There are also 'advanced' filters available for `Added`, `Updated`, `Scheduled`
- Not Storage paths: Filter documents without any of these storage paths.
- Custom field query: Filter documents with a custom field query (the same as used for the document list filters).
#### Running a workflow manually {#running-a-workflow-manually}
A workflow with a **Manual** trigger does not run automatically. Instead, it can be run on demand against existing
documents from the web UI:
- From the document list, select one or more documents and choose **Run workflow** from the "Actions" menu.
- From an individual document, choose **Run workflow** from the "Actions" menu.
You will be asked which workflow to run. By default the workflow's trigger filters still apply, so documents that do
not match are skipped. You can instead choose to **run on all selected documents**, which ignores the trigger filters
and runs the workflow's actions on every selected document.
Manual runs happen in the background. As with scheduled workflows, the document list is not refreshed automatically when
they finish, so you may need to reload the list to see the results.
### Workflow Actions
#### Types {#workflow-action-types}
@@ -674,7 +649,7 @@ new version of the same document, this time with normal OCR/text extraction appl
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
the properly processed version is created.
**Document Added**, **Document Updated**, **Scheduled**, **Manual**: these triggers run against a document that
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
already exists, so password removal happens immediately: the decrypted content is queued for
consumption as a new version right away. Note that if the document's initial consumption also
happened while it was still encrypted, that original version will likewise be missing OCR text.
+10 -3
View File
@@ -71,7 +71,7 @@ dependencies = [
"scikit-learn~=1.9.0",
"sentence-transformers>=5.6.1",
"setproctitle~=1.3.4",
"sqlite-vec==0.1.9",
"sqlite-vec==0.1.10+paperless.3",
"tantivy~=0.26.0",
"tika-client[httpx]~=1.0",
"torch>=2.13,<2.15",
@@ -151,8 +151,9 @@ typing = [
required-version = ">=0.9.0"
prerelease = "disallow"
environments = [
"sys_platform == 'darwin'",
"sys_platform == 'linux'",
"sys_platform == 'darwin' and platform_machine == 'arm64'",
"sys_platform == 'linux' and platform_machine == 'aarch64'",
"sys_platform == 'linux' and platform_machine == 'x86_64'",
]
package = false
[tool.uv.sources]
@@ -163,6 +164,12 @@ psycopg-c = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64' and python_version == '3.14'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64' and python_version == '3.14'" },
]
# Patched sqlite-vec builds
sqlite-vec = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-macosx_11_0_arm64.whl", marker = "sys_platform == 'darwin' and platform_machine == 'arm64'" },
]
torch = [
{ index = "pytorch-cpu" },
]
+211 -298
View File
File diff suppressed because it is too large. Load diff
@@ -118,7 +118,6 @@ import personFill from 'bootstrap-icons/icons/person-fill.svg'
import personLock from 'bootstrap-icons/icons/person-lock.svg'
import personSquare from 'bootstrap-icons/icons/person-square.svg'
import person from 'bootstrap-icons/icons/person.svg'
import playCircle from 'bootstrap-icons/icons/play-circle.svg'
import playFill from 'bootstrap-icons/icons/play-fill.svg'
import plusCircle from 'bootstrap-icons/icons/plus-circle.svg'
import plus from 'bootstrap-icons/icons/plus.svg'
@@ -280,7 +279,6 @@ export const icons: Record<string, string> = {
personFillLock,
personLock,
personSquare,
playCircle,
playFill,
plus,
plusCircle,
@@ -1,32 +0,0 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (message) {
<p class="text-break" [innerHTML]="message"></p>
}
<div class="mb-3">
<label class="form-label" for="runWorkflowSelect" i18n>Workflow</label>
<select class="form-select" id="runWorkflowSelect" [(ngModel)]="workflowId" (ngModelChange)="workflowSelected()">
<option [ngValue]="null" disabled i18n>Select a workflow</option>
@for (workflow of workflows; track workflow.id) {
<option [ngValue]="workflow.id">{{workflow.name}}</option>
}
</select>
</div>
<div class="form-check">
<input class="form-check-input" type="checkbox" id="runWorkflowForce" [(ngModel)]="force" />
<label class="form-check-label" for="runWorkflowForce" i18n>Run on all selected documents</label>
<div class="form-text" i18n>Ignore the workflow's trigger filters and run on every selected document.</div>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
{{btnCaption}}
</button>
</div>
@@ -1,42 +0,0 @@
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { RunWorkflowConfirmDialogComponent } from './run-workflow-confirm-dialog.component'
describe('RunWorkflowConfirmDialogComponent', () => {
let component: RunWorkflowConfirmDialogComponent
let fixture: ComponentFixture<RunWorkflowConfirmDialogComponent>
beforeEach(async () => {
TestBed.configureTestingModule({
providers: [NgbActiveModal],
imports: [RunWorkflowConfirmDialogComponent],
}).compileComponents()
fixture = TestBed.createComponent(RunWorkflowConfirmDialogComponent)
component = fixture.componentInstance
component.workflows = [{ id: 5, name: 'Manual workflow' } as any]
fixture.detectChanges()
})
it('should disable confirm until a workflow is selected and not force by default', () => {
expect(component.confirmButtonEnabled).toBeFalsy()
expect(component.force).toBeFalsy()
component.workflowId = 5
component.workflowSelected()
expect(component.confirmButtonEnabled).toBeTruthy()
component.workflowId = null
component.workflowSelected()
expect(component.confirmButtonEnabled).toBeFalsy()
})
it('should render an option per workflow', () => {
const options = fixture.nativeElement.querySelectorAll(
'#runWorkflowSelect option'
)
// placeholder option + one per workflow
expect(options).toHaveLength(2)
expect(options[1].textContent).toContain('Manual workflow')
})
})
@@ -1,24 +0,0 @@
import { Component } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { Workflow } from 'src/app/data/workflow'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-run-workflow-confirm-dialog',
templateUrl: './run-workflow-confirm-dialog.component.html',
imports: [FormsModule],
})
export class RunWorkflowConfirmDialogComponent extends ConfirmDialogComponent {
public workflows: Workflow[] = []
public workflowId: number = null
// i.e. ignore the workflow's trigger filters
public force: boolean = false
override confirmButtonEnabled = false
public workflowSelected(): void {
this.confirmButtonEnabled = !!this.workflowId
}
}
@@ -164,7 +164,7 @@
<pngx-input-text i18n-title title="Filter path" formControlName="filter_path" horizontal="true" i18n-hint hint="Apply to documents that match this path. Wildcards specified as * are allowed. Case-normalized." [error]="error?.filter_path"></pngx-input-text>
<pngx-input-select i18n-title title="Filter mail rule" [items]="mailRules()" horizontal="true" [allowNull]="true" formControlName="filter_mailrule" i18n-hint hint="Apply to documents consumed via this mail rule." [error]="error?.filter_mailrule"></pngx-input-select>
}
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled || formGroup.get('type').value === WorkflowTriggerType.Manual) {
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled) {
<pngx-input-select i18n-title title="Content matching algorithm" horizontal="true" [items]="getMatchingAlgorithms()" formControlName="matching_algorithm"></pngx-input-select>
@if (matchingPatternRequired(formGroup)) {
<pngx-input-text i18n-title title="Content matching pattern" horizontal="true" formControlName="match" [error]="error?.match"></pngx-input-text>
@@ -175,7 +175,7 @@
}
</div>
</div>
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled || formGroup.get('type').value === WorkflowTriggerType.Manual) {
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled) {
<div class="row mt-3">
<div class="col">
<div class="trigger-filters mb-3">
@@ -123,10 +123,6 @@ export const WORKFLOW_TYPE_OPTIONS = [
id: WorkflowTriggerType.Scheduled,
name: $localize`Scheduled`,
},
{
id: WorkflowTriggerType.Manual,
name: $localize`Manual`,
},
]
export const WORKFLOW_ACTION_OPTIONS = [
@@ -49,12 +49,6 @@
<pngx-bs-icon width="1em" height="1em" name="arrow-counterclockwise" class="me-2"></pngx-bs-icon><span i18n>Reprocess</span>
</button>
@if (manualWorkflows().length > 0) {
<button ngbDropdownItem (click)="runWorkflow()" [disabled]="!userCanEdit || !userIsOwner">
<pngx-bs-icon width="1em" height="1em" name="play-circle" class="me-2"></pngx-bs-icon><span i18n>Run workflow</span>
</button>
}
<button ngbDropdownItem (click)="printDocument()" [hidden]="useNativePdfViewer || isMobile">
<pngx-bs-icon width="1em" height="1em" name="printer" class="me-2"></pngx-bs-icon><span i18n>Print</span>
</button>
@@ -1030,27 +1030,6 @@ describe('DocumentDetailComponent', () => {
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should run a workflow on the document, confirm and close modal', () => {
initNormally()
const bulkEditSpy = jest.spyOn(documentService, 'bulkEdit')
bulkEditSpy.mockReturnValue(of(true))
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
const toastSpy = jest.spyOn(toastService, 'showInfo')
component.runWorkflow()
const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.workflowId = 5
openModal.componentInstance.force = true
openModal.componentInstance.confirmClicked.next()
expect(bulkEditSpy).toHaveBeenCalledWith(
{ documents: [doc.id] },
'run_workflow',
{ workflow: 5, filter_documents: false }
)
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should pass remote OCR choice when reprocessing', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
@@ -68,8 +68,6 @@ import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag'
import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { Workflow } from 'src/app/data/workflow'
import { WorkflowTriggerType } from 'src/app/data/workflow-trigger'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
import { DocumentTitlePipe } from 'src/app/pipes/document-title.pipe'
@@ -95,7 +93,6 @@ import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service'
import { WorkflowService } from 'src/app/services/rest/workflow.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { TrashService } from 'src/app/services/trash.service'
@@ -107,7 +104,6 @@ import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RunWorkflowConfirmDialogComponent } from '../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -237,7 +233,6 @@ export class DocumentDetailComponent
private componentRouterService = inject(ComponentRouterService)
private deviceDetectorService = inject(DeviceDetectorService)
private savedViewService = inject(SavedViewService)
private readonly workflowService = inject(WorkflowService)
private readonly websocketStatusService = inject(WebsocketStatusService)
private readonly useNativePdfViewerSetting = this.settings.getSignal<boolean>(
SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER
@@ -300,7 +295,6 @@ export class DocumentDetailComponent
readonly correspondents = signal<Correspondent[]>(undefined)
readonly documentTypes = signal<DocumentType[]>(undefined)
readonly storagePaths = signal<StoragePath[]>(undefined)
readonly manualWorkflows = signal<Workflow[]>([])
documentForm: FormGroup = new FormGroup({
title: new FormControl(''),
@@ -842,20 +836,6 @@ export class DocumentDetailComponent
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe((result) => this.users.set(result.results))
}
if (
this.permissionsService.currentUserCan(
PermissionAction.View,
PermissionType.Workflow
)
) {
this.workflowService
.list(1, 100000, null, null, {
trigger_type: WorkflowTriggerType.Manual,
enabled: true,
})
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe((result) => this.manualWorkflows.set(result.results))
}
this.getCustomFields()
@@ -1579,39 +1559,6 @@ export class DocumentDetailComponent
})
}
runWorkflow() {
const modal = this.modalService.open(RunWorkflowConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Run workflow`
modal.componentInstance.message = $localize`Run a workflow on "${this.document().title}".`
modal.componentInstance.btnCaption = $localize`Run`
modal.componentInstance.workflows = this.manualWorkflows()
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false)
this.documentsService
.bulkEdit({ documents: [this.document().id] }, 'run_workflow', {
workflow: modal.componentInstance.workflowId,
filter_documents: !modal.componentInstance.force,
})
.subscribe({
next: () => {
this.toastService.showInfo(
$localize`Workflow will run on "${this.document().title}" in the background.`
)
modal.close()
},
error: (error) => {
modal.componentInstance.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Error executing operation`,
error
)
},
})
})
}
private getSelectedNonLatestVersionId(): number | null {
const versions = this.document()?.versions ?? []
if (!versions.length || !this.selectedVersionId()) {
@@ -98,11 +98,6 @@
<button ngbDropdownItem (click)="mergeSelectedAsVersions()" [disabled]="!userOwnsAll || !userCanEditAll || !userCanDelete || list.allSelected || list.selectedCount < 2">
<pngx-bs-icon name="journal-bookmark-fill" class="me-2"></pngx-bs-icon><ng-container i18n>Merge as versions</ng-container>
</button>
@if (manualWorkflows.length > 0) {
<button ngbDropdownItem (click)="runWorkflowSelected()" [disabled]="!canRunWorkflow">
<pngx-bs-icon name="play-circle" class="me-2"></pngx-bs-icon><ng-container i18n>Run workflow</ng-container>
</button>
}
</div>
</div>
</div>
@@ -17,7 +17,6 @@ import { FILTER_TITLE } from 'src/app/data/filter-rule-type'
import { Results } from 'src/app/data/results'
import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag'
import { WorkflowTriggerType } from 'src/app/data/workflow-trigger'
import { FilterPipe } from 'src/app/pipes/filter.pipe'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import {
@@ -37,7 +36,6 @@ import { ShareLinkBundleService } from 'src/app/services/rest/share-link-bundle.
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service'
import { WorkflowService } from 'src/app/services/rest/workflow.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { TrashService } from 'src/app/services/trash.service'
@@ -85,7 +83,6 @@ describe('BulkEditorComponent', () => {
let customFieldsService: CustomFieldsService
let httpTestingController: HttpTestingController
let shareLinkBundleService: ShareLinkBundleService
let workflowService: WorkflowService
let router: Router
beforeEach(async () => {
@@ -176,22 +173,6 @@ describe('BulkEditorComponent', () => {
delete: jest.fn(),
},
},
{
provide: WorkflowService,
useValue: {
list: jest.fn(() =>
of({
results: [
{
id: 5,
name: 'Manual workflow',
triggers: [{ type: WorkflowTriggerType.Manual }],
},
],
})
),
},
},
{
provide: Router,
useValue: { navigate: jest.fn().mockResolvedValue(true) },
@@ -214,7 +195,6 @@ describe('BulkEditorComponent', () => {
customFieldsService = TestBed.inject(CustomFieldsService)
httpTestingController = TestBed.inject(HttpTestingController)
shareLinkBundleService = TestBed.inject(ShareLinkBundleService)
workflowService = TestBed.inject(WorkflowService)
router = TestBed.inject(Router)
fixture = TestBed.createComponent(BulkEditorComponent)
@@ -674,51 +654,6 @@ describe('BulkEditorComponent', () => {
)
})
it('should load manual workflows on init', () => {
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
fixture.detectChanges()
expect(workflowService.list).toHaveBeenCalledWith(1, 100000, null, null, {
trigger_type: WorkflowTriggerType.Manual,
enabled: true,
})
expect(component.manualWorkflows).toHaveLength(1)
expect(component.manualWorkflows[0].id).toEqual(5)
})
it('should run a workflow on the selection via the run workflow dialog', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
.spyOn(documentListViewService, 'documents', 'get')
.mockReturnValue([{ id: 3 }, { id: 4 }])
jest
.spyOn(documentListViewService, 'selected', 'get')
.mockReturnValue(new Set([3, 4]))
fixture.detectChanges()
component.runWorkflowSelected()
expect(modal).not.toBeUndefined()
expect(modal.componentInstance.workflows).toEqual(component.manualWorkflows)
modal.componentInstance.workflowId = 5
modal.componentInstance.force = true
modal.componentInstance.confirm()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
expect(req.request.body.method).toEqual('run_workflow')
expect(req.request.body.parameters).toEqual({
workflow: 5,
filter_documents: false,
})
req.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
})
it('should set modal dialog text accordingly for tag edit confirmation', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[m.length - 1]))
@@ -34,7 +34,6 @@ import { CustomField } from 'src/app/data/custom-field'
import { MatchingModel } from 'src/app/data/matching-model'
import { SelectionDataItem } from 'src/app/data/results'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { Workflow } from 'src/app/data/workflow'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import { OpenDocumentsService } from 'src/app/services/open-documents.service'
@@ -54,7 +53,6 @@ import {
} from 'src/app/services/rest/document.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { ShareLinkBundleService } from 'src/app/services/rest/share-link-bundle.service'
import { WorkflowService } from 'src/app/services/rest/workflow.service'
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
import { SettingsService } from 'src/app/services/settings.service'
@@ -66,7 +64,6 @@ import { MergeAsVersionsConfirmDialogComponent } from '../../common/confirm-dial
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { RunWorkflowConfirmDialogComponent } from '../../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -84,7 +81,6 @@ import { PermissionsDialogComponent } from '../../common/permissions-dialog/perm
import { ShareLinkBundleDialogComponent } from '../../common/share-link-bundle-dialog/share-link-bundle-dialog.component'
import { ComponentWithPermissions } from '../../with-permissions/with-permissions.component'
import { CustomFieldsBulkEditDialogComponent } from './custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component'
import { WorkflowTriggerType } from 'src/app/data/workflow-trigger'
@Component({
selector: 'pngx-bulk-editor',
@@ -118,11 +114,8 @@ export class BulkEditorComponent
public readonly permissionService = inject(PermissionsService)
private savedViewService = inject(SavedViewService)
private readonly shareLinkBundleService = inject(ShareLinkBundleService)
private readonly workflowService = inject(WorkflowService)
private readonly router = inject(Router)
public manualWorkflows: Workflow[] = []
tagSelectionModel = new FilterableDropdownSelectionModel(true)
correspondentSelectionModel = new FilterableDropdownSelectionModel()
documentTypeSelectionModel = new FilterableDropdownSelectionModel()
@@ -265,20 +258,6 @@ export class BulkEditorComponent
(result) => (this.customFieldsSelectionModel.items = result.results)
)
}
if (
this.permissionService.currentUserCan(
PermissionAction.View,
PermissionType.Workflow
)
) {
this.workflowService
.list(1, 100000, null, null, {
trigger_type: WorkflowTriggerType.Manual,
enabled: true,
})
.pipe(first())
.subscribe((result) => (this.manualWorkflows = result.results))
}
this.downloadForm
.get('downloadFileTypeArchive')
@@ -307,28 +286,6 @@ export class BulkEditorComponent
this.unsubscribeNotifier.complete()
}
get canRunWorkflow(): boolean {
return this.userCanEditAll && this.manualWorkflows.length > 0
}
runWorkflowSelected() {
const modal = this.modalService.open(RunWorkflowConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Run workflow`
modal.componentInstance.message = $localize`Run a workflow on ${this.getSelectionSize()} selected document(s).`
modal.componentInstance.btnCaption = $localize`Run`
modal.componentInstance.workflows = this.manualWorkflows
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
this.executeBulkEditMethod(modal, 'run_workflow', {
workflow: modal.componentInstance.workflowId,
filter_documents: !modal.componentInstance.force,
})
})
}
private executeBulkEditMethod(
modal: NgbModalRef,
method: DocumentBulkEditMethod,
-1
View File
@@ -12,7 +12,6 @@ export enum WorkflowTriggerType {
DocumentAdded = 2,
DocumentUpdated = 3,
Scheduled = 4,
Manual = 5,
}
export enum ScheduleDateField {
@@ -39,7 +39,6 @@ export type DocumentBulkEditMethod =
| 'modify_tags'
| 'modify_custom_fields'
| 'set_permissions'
| 'run_workflow'
export interface MergeDocumentsRequest {
metadata_document_id?: number
-20
View File
@@ -42,7 +42,6 @@ from documents.plugins.helpers import DocumentsStatusManager
from documents.tasks import bulk_update_documents
from documents.tasks import consume_file
from documents.tasks import remove_document_from_index
from documents.tasks import run_workflows_for_documents
from documents.tasks import update_document_content_maybe_archive_file
from documents.versioning import get_latest_version_for_root
from documents.versioning import get_root_document
@@ -465,25 +464,6 @@ def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
return "OK"
def run_workflow(
doc_ids: list[int],
workflow: int,
*,
filter_documents: bool = True,
) -> Literal["OK"]:
"""
Manually run a single workflow against the given documents.
Done asynchronously via a Celery task.
"""
run_workflows_for_documents.delay(
document_ids=doc_ids,
workflow_id=workflow,
filter_documents=filter_documents,
)
return "OK"
def set_permissions(
doc_ids: list[int],
set_permissions: PermissionsSpec,
-13
View File
@@ -35,7 +35,6 @@ from django_filters.rest_framework import DateTimeFilter
from django_filters.rest_framework import Filter
from django_filters.rest_framework import FilterSet
from django_filters.rest_framework import MultipleChoiceFilter
from django_filters.rest_framework import NumberFilter
from drf_spectacular.utils import extend_schema_field
from guardian.utils import get_group_obj_perms_model
from guardian.utils import get_user_obj_perms_model
@@ -53,7 +52,6 @@ from documents.models import ShareLink
from documents.models import ShareLinkBundle
from documents.models import StoragePath
from documents.models import Tag
from documents.models import Workflow
from documents.permissions import annotate_authorizing_fields
from documents.permissions import permitted_document_ids
from documents.permissions import permitted_object_ids
@@ -251,17 +249,6 @@ class CustomFieldFilterSet(FilterSet):
}
class WorkflowFilterSet(FilterSet):
trigger_type = NumberFilter(field_name="triggers__type", distinct=True)
class Meta:
model = Workflow
fields = {
"id": ID_KWARGS,
"enabled": ["exact"],
}
@extend_schema_field(serializers.CharField)
class CustomFieldsFilter(Filter):
def filter(self, qs, value):
-1
View File
@@ -684,7 +684,6 @@ def document_matches_workflow(
trigger_type == WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED
or trigger_type == WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED
or trigger_type == WorkflowTrigger.WorkflowTriggerType.SCHEDULED
or trigger_type == WorkflowTrigger.WorkflowTriggerType.MANUAL
):
trigger_matched, reason = existing_document_matches_workflow(
document,
@@ -1,43 +0,0 @@
# Generated by Django 5.2.18 on 2026-11-06 17:12
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0028_replace_tag_tree_fields"),
]
operations = [
migrations.AlterField(
model_name="workflowrun",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Consumption Started"),
(2, "Document Added"),
(3, "Document Updated"),
(4, "Scheduled"),
(5, "Manual"),
],
null=True,
verbose_name="workflow trigger type",
),
),
migrations.AlterField(
model_name="workflowtrigger",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Consumption Started"),
(2, "Document Added"),
(3, "Document Updated"),
(4, "Scheduled"),
(5, "Manual"),
],
default=1,
verbose_name="Workflow Trigger Type",
),
),
]
-1
View File
@@ -1532,7 +1532,6 @@ class WorkflowTrigger(models.Model):
DOCUMENT_ADDED = 2, _("Document Added")
DOCUMENT_UPDATED = 3, _("Document Updated")
SCHEDULED = 4, _("Scheduled")
MANUAL = 5, _("Manual")
class DocumentSourceChoices(models.IntegerChoices):
CONSUME_FOLDER = DocumentSource.ConsumeFolder.value, _("Consume Folder")
-34
View File
@@ -1829,7 +1829,6 @@ class BulkEditSerializer(
"modify_tags",
"modify_custom_fields",
"set_permissions",
"run_workflow",
*LEGACY_DOCUMENT_ACTION_METHODS,
],
label="Method",
@@ -1919,8 +1918,6 @@ class BulkEditSerializer(
return bulk_edit.reprocess
elif method == "set_permissions":
return bulk_edit.set_permissions
elif method == "run_workflow":
return bulk_edit.run_workflow
elif method == "rotate":
return bulk_edit.rotate
elif method == "merge":
@@ -2160,35 +2157,6 @@ class BulkEditSerializer(
else:
parameters["remote_ocr"] = False
def _validate_parameters_run_workflow(self, parameters) -> None:
if "workflow" not in parameters:
raise serializers.ValidationError("workflow not specified")
try:
workflow = Workflow.objects.get(id=parameters["workflow"])
except (Workflow.DoesNotExist, ValueError, TypeError):
raise serializers.ValidationError("Workflow does not exist")
request = self.context.get("request")
user = getattr(request, "user", None)
if user is not None and not user.has_perm("documents.view_workflow"):
raise PermissionDenied(
"Insufficient permissions to run this workflow",
)
if not workflow.enabled:
raise serializers.ValidationError("Workflow is not enabled")
if not workflow.triggers.filter(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
).exists():
raise serializers.ValidationError(
"Workflow does not have a manual trigger",
)
if "filter_documents" in parameters:
if not isinstance(parameters["filter_documents"], bool):
raise serializers.ValidationError(
"filter_documents must be a boolean",
)
else:
parameters["filter_documents"] = True
def validate_parameters_remove_password(self, parameters):
if "password" not in parameters:
raise serializers.ValidationError("password not specified")
@@ -2255,8 +2223,6 @@ class BulkEditSerializer(
self.validate_parameters_remove_password(parameters)
elif method == bulk_edit.reprocess:
self._validate_parameters_reprocess(parameters)
elif method == bulk_edit.run_workflow:
self._validate_parameters_run_workflow(parameters)
return attrs
+1 -11
View File
@@ -864,8 +864,6 @@ def run_workflows(
logging_group: uuid.UUID | None = None,
overrides: DocumentMetadataOverrides | None = None,
original_file: Path | None = None,
*,
skip_matching: bool = False,
) -> tuple[DocumentMetadataOverrides, str] | None:
"""
Execute workflows matching a document for the given trigger. When `overrides` is provided
@@ -876,10 +874,6 @@ def run_workflows(
`document.source_path` (Document) or `document.original_file` (ConsumableDocument).
Passing `workflow_to_run` skips the workflow query (currently only used by scheduled runs).
Passing `skip_matching` runs the workflow's actions without evaluating its trigger
filters against the document. It is only meaningful together with `workflow_to_run`
(e.g. a manual run the user has explicitly forced onto the selected documents).
"""
use_overrides = overrides is not None
@@ -929,11 +923,7 @@ def run_workflows(
)
break
if skip_matching or matching.document_matches_workflow(
document,
workflow,
trigger_type,
):
if matching.document_matches_workflow(document, workflow, trigger_type):
action: WorkflowAction
has_move_to_trash_action = False
for action in workflow.actions.order_by("order", "pk"):
-46
View File
@@ -53,7 +53,6 @@ from documents.models import ShareLink
from documents.models import ShareLinkBundle
from documents.models import StoragePath
from documents.models import Tag
from documents.models import Workflow
from documents.models import WorkflowRun
from documents.models import WorkflowTrigger
from documents.plugins.base import ConsumeTaskPlugin
@@ -692,51 +691,6 @@ def check_scheduled_workflows() -> None:
)
@shared_task
def run_workflows_for_documents(
document_ids: list[int],
workflow_id: int,
*,
filter_documents: bool = True,
) -> None:
"""
Run a single workflow manually against the given documents.
``filter_documents`` controls whether the workflow's trigger filters are applied.
When True, only documents matching the existing filters will have the workflow run.
When False, the workflow will be run on all specified documents regardless of the filters.
``
"""
try:
workflow = Workflow.objects.get(id=workflow_id)
except Workflow.DoesNotExist:
logger.warning(
f"Manual workflow run requested for missing workflow {workflow_id}",
)
return
for document in Document.objects.filter(
id__in=document_ids,
root_document__isnull=True,
):
try:
run_workflows(
trigger_type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
document=document,
workflow_to_run=workflow,
skip_matching=not filter_documents,
)
# Manual runs dont send document_updated signal, so ws here to ensure clients are refreshed
send_websocket_document_updated(
sender=None,
document=document,
)
except Exception:
logger.exception(
f"Error running workflow {workflow} on document {document.pk}",
)
def update_document_parent_tags(tag: Tag, new_parent: Tag) -> None:
"""
When a tag's parent changes, ensure all documents containing the tag also have
-109
View File
@@ -15,11 +15,7 @@ from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_all_global
from paperless_testing.permissions import grant_global
@@ -626,111 +622,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
def _make_manual_workflow(self) -> Workflow:
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
)
action = WorkflowAction.objects.create()
workflow = Workflow.objects.create(name="Manual workflow", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(action)
return workflow
def _run_workflow(self, parameters, documents=None):
return self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": documents or [self.doc1.id],
"method": "run_workflow",
"parameters": parameters,
},
),
content_type="application/json",
)
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow(self, m) -> None:
"""A valid run_workflow call passes the docs, workflow id and filter flag through."""
self.setup_mock(m, "run_workflow")
workflow = self._make_manual_workflow()
response = self._run_workflow({"workflow": workflow.id})
self.assertEqual(response.status_code, status.HTTP_200_OK)
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"workflow": workflow.id, "filter_documents": True})
self._run_workflow({"workflow": workflow.id, "filter_documents": False})
self.assertFalse(m.call_args.kwargs["filter_documents"])
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow_invalid_parameters(self, m) -> None:
"""Invalid run_workflow parameters are rejected before dispatch."""
self.setup_mock(m, "run_workflow")
manual = self._make_manual_workflow()
non_manual = Workflow.objects.create(name="Non-manual", order=0)
non_manual.triggers.add(
WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
),
)
disabled = Workflow.objects.create(
name="Disabled manual",
order=0,
enabled=False,
)
disabled.triggers.add(
WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
),
)
for label, parameters in (
("missing workflow", {}),
("nonexistent workflow", {"workflow": 9999}),
("no manual trigger", {"workflow": non_manual.id}),
("disabled workflow", {"workflow": disabled.id}),
("non-bool filter", {"workflow": manual.id, "filter_documents": "nope"}),
):
with self.subTest(label):
response = self._run_workflow(parameters)
self.assertEqual(
response.status_code,
status.HTTP_400_BAD_REQUEST,
)
m.assert_not_called()
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow_requires_ownership(self, m) -> None:
"""A non-owner without document ownership is forbidden (actions mutate docs)."""
self.setup_mock(m, "run_workflow")
user = UserFactory(username="no_perms")
grant_all_global(user)
doc = DocumentFactory(owner=UserFactory(username="other"))
workflow = self._make_manual_workflow()
self.client.force_authenticate(user=user)
response = self._run_workflow({"workflow": workflow.id}, documents=[doc.id])
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
m.assert_not_called()
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow_requires_workflow_permission(self, m) -> None:
"""Owning the documents is not enough without view_workflow permission."""
self.setup_mock(m, "run_workflow")
user = UserFactory(username="no_wf_perms")
grant_global(user, "change_document")
doc = DocumentFactory(owner=user)
workflow = self._make_manual_workflow()
self.client.force_authenticate(user=user)
response = self._run_workflow({"workflow": workflow.id}, documents=[doc.id])
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
m.assert_not_called()
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None:
"""
-41
View File
@@ -100,47 +100,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.action.assign_correspondent.pk,
)
def test_api_get_workflows_filter_trigger_type(self) -> None:
"""
GIVEN:
- A consumption workflow and a manual workflow
WHEN:
- Workflows are requested filtered by trigger_type
THEN:
- Only workflows with a matching trigger are returned
"""
manual_trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
)
manual_workflow = Workflow.objects.create(name="Manual", order=1)
manual_workflow.triggers.add(manual_trigger)
response = self.client.get(
f"{self.ENDPOINT}?trigger_type={WorkflowTrigger.WorkflowTriggerType.MANUAL}",
format="json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["results"][0]["id"], manual_workflow.id)
def test_api_get_workflows_filter_enabled(self) -> None:
"""
GIVEN:
- An enabled workflow and a disabled workflow
WHEN:
- Workflows are requested filtered by enabled
THEN:
- Only enabled workflows are returned
"""
Workflow.objects.create(name="Disabled", order=1, enabled=False)
response = self.client.get(f"{self.ENDPOINT}?enabled=true", format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["results"][0]["id"], self.workflow.id)
def test_api_get_workflow_actions_ordered(self) -> None:
"""
GIVEN:
-77
View File
@@ -2288,83 +2288,6 @@ class TestWorkflows(
doc.refresh_from_db()
self.assertEqual(doc.owner, self.user2)
def _make_manual_workflow(self, *, require_tag: Tag | None = None) -> Workflow:
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
)
if require_tag is not None:
trigger.filter_has_all_tags.add(require_tag)
action = WorkflowAction.objects.create(assign_owner=self.user2)
workflow = Workflow.objects.create(name="Manual workflow", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(action)
return workflow
@mock.patch("documents.tasks.send_websocket_document_updated")
def test_manual_workflow_run_applies_and_notifies(self, mock_ws) -> None:
"""A manual run applies the workflow's actions and pushes a websocket update."""
workflow = self._make_manual_workflow()
doc = DocumentFactory()
tasks.run_workflows_for_documents([doc.pk], workflow.pk)
doc.refresh_from_db()
self.assertEqual(doc.owner, self.user2)
self.assertEqual(mock_ws.call_args.kwargs["document"].pk, doc.pk)
def test_manual_workflow_run_respects_filters(self) -> None:
"""With filter_documents=True, a document failing the trigger filters is skipped."""
workflow = self._make_manual_workflow(require_tag=self.t1)
doc = DocumentFactory()
tasks.run_workflows_for_documents([doc.pk], workflow.pk, filter_documents=True)
doc.refresh_from_db()
self.assertIsNone(doc.owner)
def test_manual_workflow_run_force_ignores_filters(self) -> None:
"""With filter_documents=False, actions run regardless of the trigger filters."""
workflow = self._make_manual_workflow(require_tag=self.t1)
doc = DocumentFactory()
tasks.run_workflows_for_documents([doc.pk], workflow.pk, filter_documents=False)
doc.refresh_from_db()
self.assertEqual(doc.owner, self.user2)
def test_manual_workflow_run_missing_workflow(self) -> None:
"""A missing workflow id logs a warning rather than raising."""
doc = DocumentFactory()
with self.assertLogs("paperless.tasks", level="WARNING"):
tasks.run_workflows_for_documents([doc.pk], 9999)
@mock.patch("documents.tasks.send_websocket_document_updated")
@mock.patch("documents.tasks.run_workflows")
def test_manual_workflow_run_continues_after_error(
self,
mock_run_workflows,
mock_ws,
) -> None:
"""A failure on one document does not abort the run for the others."""
workflow = self._make_manual_workflow()
doc1 = DocumentFactory()
doc2 = DocumentFactory()
def side_effect(*args, document, **kwargs):
if document.pk == doc1.pk:
raise Exception("boom")
mock_run_workflows.side_effect = side_effect
with self.assertLogs("paperless.tasks", level="ERROR"):
tasks.run_workflows_for_documents([doc1.pk, doc2.pk], workflow.pk)
# Both documents were attempted; the websocket update only fired for the
# one that did not raise.
self.assertEqual(mock_run_workflows.call_count, 2)
ws_doc_ids = [c.kwargs["document"].pk for c in mock_ws.call_args_list]
self.assertEqual(ws_doc_ids, [doc2.pk])
@mock.patch("documents.tasks.send_websocket_document_updated")
def test_workflow_scheduled_trigger_sends_websocket_update(
self,
-98
View File
@@ -146,7 +146,6 @@ from documents.filters import ShareLinkFilterSet
from documents.filters import StoragePathFilterSet
from documents.filters import TagFilterSet
from documents.filters import TitleContentFilter
from documents.filters import WorkflowFilterSet
from documents.mail import EmailAttachment
from documents.mail import send_email
from documents.matching import match_correspondents
@@ -3009,100 +3008,6 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
)
return Response({"result": "OK", "task_id": task.id})
def _has_document_permissions(
self,
*,
user: User,
documents: list[int],
method,
parameters: dict[str, Any],
) -> bool:
if user.is_superuser:
return True
root_docs = {
get_root_document(doc)
for doc in Document.objects.select_related(
"owner",
"root_document__owner",
).filter(pk__in=documents)
}
user_is_owner_of_all_documents = all(
(doc.owner == user or doc.owner is None) for doc in root_docs
)
# check global and object permissions for all documents
has_perms = (
user.has_perm(
"documents.change_document",
)
and not Document.global_objects.filter(pk__in=documents)
.exclude(pk__in=permitted_document_ids(user, perm="change_document"))
.exists()
)
# check ownership for methods that change original document
if (
(
has_perms
and method
in [
bulk_edit.set_permissions,
bulk_edit.delete,
bulk_edit.rotate,
bulk_edit.delete_pages,
bulk_edit.edit_pdf,
bulk_edit.merge_as_versions,
bulk_edit.remove_password,
bulk_edit.run_workflow,
]
)
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (method == bulk_edit.edit_pdf and parameters.get("update_document"))
):
has_perms = has_perms and user_is_owner_of_all_documents
# check global add permissions for methods that create documents
if (
has_perms
and (
method in [bulk_edit.split, bulk_edit.merge]
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.add_document")
):
has_perms = False
# check global delete permissions for methods that delete documents
if (
has_perms
and (
method == bulk_edit.delete
# Sources stop being documents of their own, and removing one
# again afterwards needs delete_document
or method == bulk_edit.merge_as_versions
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and parameters.get("delete_original")
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.delete_document")
):
has_perms = False
return has_perms
def _execute_document_action(
self,
*,
@@ -3179,7 +3084,6 @@ class BulkEditView(DocumentOperationPermissionMixin):
LEGACY_MODIFIED_FIELDS = {
"delete": "deleted_at",
"reprocess": "checksum",
"remove_password": None,
}
serializer_class = BulkEditSerializer
@@ -5185,8 +5089,6 @@ class WorkflowViewSet(ModelViewSet[Workflow]):
serializer_class = WorkflowSerializer
pagination_class = StandardPagination
filter_backends = (DjangoFilterBackend,)
filterset_class = WorkflowFilterSet
model = Workflow
+160 -160
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-11-12 21:28+0000\n"
"POT-Creation-Date: 2026-11-11 23:50+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -21,43 +21,43 @@ msgstr ""
msgid "Documents"
msgstr ""
#: documents/filters.py:349
#: documents/filters.py:336
msgid "Expected a date or a relative date such as \"today\" or \"in 30 days\"."
msgstr ""
#: documents/filters.py:527
#: documents/filters.py:514
msgid "Value must be valid JSON."
msgstr ""
#: documents/filters.py:548
#: documents/filters.py:535
msgid "Invalid custom field query expression"
msgstr ""
#: documents/filters.py:558
#: documents/filters.py:545
msgid "Invalid expression list. Must be nonempty."
msgstr ""
#: documents/filters.py:579
#: documents/filters.py:566
msgid "Invalid logical operator {op!r}"
msgstr ""
#: documents/filters.py:593
#: documents/filters.py:580
msgid "Maximum number of query conditions exceeded."
msgstr ""
#: documents/filters.py:657
#: documents/filters.py:644
msgid "{name!r} is not a valid custom field."
msgstr ""
#: documents/filters.py:694
#: documents/filters.py:681
msgid "{data_type} does not support query expr {expr!r}."
msgstr ""
#: documents/filters.py:815 documents/models.py:195
#: documents/filters.py:802 documents/models.py:195
msgid "Maximum nesting depth exceeded."
msgstr ""
#: documents/filters.py:1192
#: documents/filters.py:1179
msgid "Custom field not found"
msgstr ""
@@ -93,20 +93,20 @@ msgstr ""
msgid "Automatic"
msgstr ""
#: documents/models.py:69 documents/models.py:709 documents/models.py:2150
#: documents/models.py:69 documents/models.py:709 documents/models.py:2149
#: paperless_mail/models.py:23 paperless_mail/models.py:143
msgid "name"
msgstr ""
#: documents/models.py:71 documents/models.py:1593
#: documents/models.py:71 documents/models.py:1592
msgid "match"
msgstr ""
#: documents/models.py:74 documents/models.py:1596
#: documents/models.py:74 documents/models.py:1595
msgid "matching algorithm"
msgstr ""
#: documents/models.py:79 documents/models.py:1601
#: documents/models.py:79 documents/models.py:1600
msgid "is insensitive"
msgstr ""
@@ -280,7 +280,7 @@ msgid "Optional short label for a document version."
msgstr ""
#: documents/models.py:402 documents/models.py:1027 documents/models.py:1058
#: documents/models.py:1162 documents/models.py:2193
#: documents/models.py:1162 documents/models.py:2192
msgid "document"
msgstr ""
@@ -496,7 +496,7 @@ msgstr ""
msgid "Tag"
msgstr ""
#: documents/models.py:683 documents/models.py:698 documents/models.py:1871
#: documents/models.py:683 documents/models.py:698 documents/models.py:1870
msgid "Tags"
msgstr ""
@@ -528,19 +528,19 @@ msgstr ""
msgid "Large Cards"
msgstr ""
#: documents/models.py:695 documents/models.py:1870
#: documents/models.py:695 documents/models.py:1869
msgid "Title"
msgstr ""
#: documents/models.py:696 documents/models.py:926 documents/models.py:1545
#: documents/models.py:696 documents/models.py:926 documents/models.py:1544
msgid "Created"
msgstr ""
#: documents/models.py:697 documents/models.py:1544
#: documents/models.py:697 documents/models.py:1543
msgid "Added"
msgstr ""
#: documents/models.py:699 documents/models.py:1872
#: documents/models.py:699 documents/models.py:1871
msgid "Correspondent"
msgstr ""
@@ -860,7 +860,7 @@ msgstr ""
msgid "Index Optimize"
msgstr ""
#: documents/models.py:865 documents/models.py:1540
#: documents/models.py:865 documents/models.py:1539
msgid "Mail Fetch"
msgstr ""
@@ -900,7 +900,7 @@ msgstr ""
msgid "Scheduled"
msgstr ""
#: documents/models.py:883 documents/models.py:1541
#: documents/models.py:883 documents/models.py:1540
msgid "Web UI"
msgstr ""
@@ -920,7 +920,7 @@ msgstr ""
msgid "System"
msgstr ""
#: documents/models.py:888 documents/models.py:1535
#: documents/models.py:888
msgid "Manual"
msgstr ""
@@ -1178,491 +1178,491 @@ msgstr ""
msgid "Document Updated"
msgstr ""
#: documents/models.py:1538
#: documents/models.py:1537
msgid "Consume Folder"
msgstr ""
#: documents/models.py:1539
#: documents/models.py:1538
msgid "Api Upload"
msgstr ""
#: documents/models.py:1546
#: documents/models.py:1545
msgid "Modified"
msgstr ""
#: documents/models.py:1547
#: documents/models.py:1546
msgid "Custom Field"
msgstr ""
#: documents/models.py:1550
#: documents/models.py:1549
msgid "Workflow Trigger Type"
msgstr ""
#: documents/models.py:1562
#: documents/models.py:1561
msgid "filter path"
msgstr ""
#: documents/models.py:1567
#: documents/models.py:1566
msgid ""
"Only consume documents with a path that matches this if specified. Wildcards "
"specified as * are allowed. Case insensitive."
msgstr ""
#: documents/models.py:1574
#: documents/models.py:1573
msgid "filter filename"
msgstr ""
#: documents/models.py:1579 paperless_mail/models.py:200
#: documents/models.py:1578 paperless_mail/models.py:200
msgid ""
"Only consume documents which entirely match this filename if specified. "
"Wildcards such as *.pdf or *invoice* are allowed. Case insensitive."
msgstr ""
#: documents/models.py:1590
#: documents/models.py:1589
msgid "filter documents from this mail rule"
msgstr ""
#: documents/models.py:1606
#: documents/models.py:1605
msgid "has these tag(s)"
msgstr ""
#: documents/models.py:1613
#: documents/models.py:1612
msgid "has all of these tag(s)"
msgstr ""
#: documents/models.py:1620
#: documents/models.py:1619
msgid "does not have these tag(s)"
msgstr ""
#: documents/models.py:1628
#: documents/models.py:1627
msgid "has this document type"
msgstr ""
#: documents/models.py:1635
#: documents/models.py:1634
msgid "has one of these document types"
msgstr ""
#: documents/models.py:1642
#: documents/models.py:1641
msgid "does not have these document type(s)"
msgstr ""
#: documents/models.py:1650
#: documents/models.py:1649
msgid "has this correspondent"
msgstr ""
#: documents/models.py:1657
#: documents/models.py:1656
msgid "does not have these correspondent(s)"
msgstr ""
#: documents/models.py:1664
#: documents/models.py:1663
msgid "has one of these correspondents"
msgstr ""
#: documents/models.py:1672
#: documents/models.py:1671
msgid "has this storage path"
msgstr ""
#: documents/models.py:1679
#: documents/models.py:1678
msgid "has one of these storage paths"
msgstr ""
#: documents/models.py:1686
#: documents/models.py:1685
msgid "does not have these storage path(s)"
msgstr ""
#: documents/models.py:1690
#: documents/models.py:1689
msgid "filter custom field query"
msgstr ""
#: documents/models.py:1693
#: documents/models.py:1692
msgid "JSON-encoded custom field query expression."
msgstr ""
#: documents/models.py:1697
#: documents/models.py:1696
msgid "schedule offset days"
msgstr ""
#: documents/models.py:1700
#: documents/models.py:1699
msgid "The number of days to offset the schedule trigger by."
msgstr ""
#: documents/models.py:1705
#: documents/models.py:1704
msgid "schedule is recurring"
msgstr ""
#: documents/models.py:1708
#: documents/models.py:1707
msgid "If the schedule should be recurring."
msgstr ""
#: documents/models.py:1713
#: documents/models.py:1712
msgid "schedule recurring delay in days"
msgstr ""
#: documents/models.py:1717
#: documents/models.py:1716
msgid "The number of days between recurring schedule triggers."
msgstr ""
#: documents/models.py:1722
#: documents/models.py:1721
msgid "schedule date field"
msgstr ""
#: documents/models.py:1727
#: documents/models.py:1726
msgid "The field to check for a schedule trigger."
msgstr ""
#: documents/models.py:1736
#: documents/models.py:1735
msgid "schedule date custom field"
msgstr ""
#: documents/models.py:1740
#: documents/models.py:1739
msgid "workflow trigger"
msgstr ""
#: documents/models.py:1741
#: documents/models.py:1740
msgid "workflow triggers"
msgstr ""
#: documents/models.py:1749
#: documents/models.py:1748
msgid "email subject"
msgstr ""
#: documents/models.py:1753
#: documents/models.py:1752
msgid ""
"The subject of the email, can include some placeholders, see documentation."
msgstr ""
#: documents/models.py:1759
#: documents/models.py:1758
msgid "email body"
msgstr ""
#: documents/models.py:1762
#: documents/models.py:1761
msgid ""
"The body (message) of the email, can include some placeholders, see "
"documentation."
msgstr ""
#: documents/models.py:1768
#: documents/models.py:1767
msgid "emails to"
msgstr ""
#: documents/models.py:1771
#: documents/models.py:1770
msgid "The destination email addresses, comma separated."
msgstr ""
#: documents/models.py:1777
#: documents/models.py:1776
msgid "include document in email"
msgstr ""
#: documents/models.py:1788
#: documents/models.py:1787
msgid "webhook url"
msgstr ""
#: documents/models.py:1791
#: documents/models.py:1790
msgid "The destination URL for the notification."
msgstr ""
#: documents/models.py:1796
#: documents/models.py:1795
msgid "use parameters"
msgstr ""
#: documents/models.py:1801
#: documents/models.py:1800
msgid "send as JSON"
msgstr ""
#: documents/models.py:1805
#: documents/models.py:1804
msgid "webhook parameters"
msgstr ""
#: documents/models.py:1808
#: documents/models.py:1807
msgid "The parameters to send with the webhook URL if body not used."
msgstr ""
#: documents/models.py:1812
#: documents/models.py:1811
msgid "webhook body"
msgstr ""
#: documents/models.py:1815
#: documents/models.py:1814
msgid "The body to send with the webhook URL if parameters not used."
msgstr ""
#: documents/models.py:1819
#: documents/models.py:1818
msgid "webhook headers"
msgstr ""
#: documents/models.py:1822
#: documents/models.py:1821
msgid "The headers to send with the webhook URL."
msgstr ""
#: documents/models.py:1827
#: documents/models.py:1826
msgid "include document in webhook"
msgstr ""
#: documents/models.py:1838
#: documents/models.py:1837
msgid "Assignment"
msgstr ""
#: documents/models.py:1842
#: documents/models.py:1841
msgid "Removal"
msgstr ""
#: documents/models.py:1846 documents/templates/account/password_reset.html:15
#: documents/models.py:1845 documents/templates/account/password_reset.html:15
msgid "Email"
msgstr ""
#: documents/models.py:1850
#: documents/models.py:1849
msgid "Webhook"
msgstr ""
#: documents/models.py:1854
#: documents/models.py:1853
msgid "Password removal"
msgstr ""
#: documents/models.py:1858
#: documents/models.py:1857
msgid "Move to trash"
msgstr ""
#: documents/models.py:1862
#: documents/models.py:1861
msgid "Remote OCR"
msgstr ""
#: documents/models.py:1866
#: documents/models.py:1865
msgid "Apply AI suggestions"
msgstr ""
#: documents/models.py:1873
#: documents/models.py:1872
msgid "Document type"
msgstr ""
#: documents/models.py:1874
#: documents/models.py:1873
msgid "Storage path"
msgstr ""
#: documents/models.py:1875
#: documents/models.py:1874
msgid "Created date"
msgstr ""
#: documents/models.py:1878
#: documents/models.py:1877
msgid "Workflow Action Type"
msgstr ""
#: documents/models.py:1883 documents/models.py:2152
#: documents/models.py:1882 documents/models.py:2151
#: paperless_mail/models.py:145
msgid "order"
msgstr ""
#: documents/models.py:1886
#: documents/models.py:1885
msgid "assign title"
msgstr ""
#: documents/models.py:1890
#: documents/models.py:1889
msgid "Assign a document title, must be a Jinja2 template, see documentation."
msgstr ""
#: documents/models.py:1898 paperless_mail/models.py:274
#: documents/models.py:1897 paperless_mail/models.py:274
msgid "assign this tag"
msgstr ""
#: documents/models.py:1907 paperless_mail/models.py:282
#: documents/models.py:1906 paperless_mail/models.py:282
msgid "assign this document type"
msgstr ""
#: documents/models.py:1916 paperless_mail/models.py:296
#: documents/models.py:1915 paperless_mail/models.py:296
msgid "assign this correspondent"
msgstr ""
#: documents/models.py:1925
#: documents/models.py:1924
msgid "assign this storage path"
msgstr ""
#: documents/models.py:1934
#: documents/models.py:1933
msgid "assign this owner"
msgstr ""
#: documents/models.py:1941
#: documents/models.py:1940
msgid "grant view permissions to these users"
msgstr ""
#: documents/models.py:1948
#: documents/models.py:1947
msgid "grant view permissions to these groups"
msgstr ""
#: documents/models.py:1955
#: documents/models.py:1954
msgid "grant change permissions to these users"
msgstr ""
#: documents/models.py:1962
#: documents/models.py:1961
msgid "grant change permissions to these groups"
msgstr ""
#: documents/models.py:1969
#: documents/models.py:1968
msgid "assign these custom fields"
msgstr ""
#: documents/models.py:1973
#: documents/models.py:1972
msgid "custom field values"
msgstr ""
#: documents/models.py:1977
#: documents/models.py:1976
msgid "Optional values to assign to the custom fields."
msgstr ""
#: documents/models.py:1986
#: documents/models.py:1985
msgid "remove these tag(s)"
msgstr ""
#: documents/models.py:1991
#: documents/models.py:1990
msgid "remove all tags"
msgstr ""
#: documents/models.py:1998
#: documents/models.py:1997
msgid "remove these document type(s)"
msgstr ""
#: documents/models.py:2003
#: documents/models.py:2002
msgid "remove all document types"
msgstr ""
#: documents/models.py:2010
#: documents/models.py:2009
msgid "remove these correspondent(s)"
msgstr ""
#: documents/models.py:2015
#: documents/models.py:2014
msgid "remove all correspondents"
msgstr ""
#: documents/models.py:2022
#: documents/models.py:2021
msgid "remove these storage path(s)"
msgstr ""
#: documents/models.py:2027
#: documents/models.py:2026
msgid "remove all storage paths"
msgstr ""
#: documents/models.py:2034
#: documents/models.py:2033
msgid "remove these owner(s)"
msgstr ""
#: documents/models.py:2039
#: documents/models.py:2038
msgid "remove all owners"
msgstr ""
#: documents/models.py:2046
#: documents/models.py:2045
msgid "remove view permissions for these users"
msgstr ""
#: documents/models.py:2053
#: documents/models.py:2052
msgid "remove view permissions for these groups"
msgstr ""
#: documents/models.py:2060
#: documents/models.py:2059
msgid "remove change permissions for these users"
msgstr ""
#: documents/models.py:2067
#: documents/models.py:2066
msgid "remove change permissions for these groups"
msgstr ""
#: documents/models.py:2072
#: documents/models.py:2071
msgid "remove all permissions"
msgstr ""
#: documents/models.py:2079
#: documents/models.py:2078
msgid "remove these custom fields"
msgstr ""
#: documents/models.py:2084
#: documents/models.py:2083
msgid "remove all custom fields"
msgstr ""
#: documents/models.py:2093
#: documents/models.py:2092
msgid "email"
msgstr ""
#: documents/models.py:2102
#: documents/models.py:2101
msgid "webhook"
msgstr ""
#: documents/models.py:2106
#: documents/models.py:2105
msgid "passwords"
msgstr ""
#: documents/models.py:2110
#: documents/models.py:2109
msgid ""
"Passwords to try when removing PDF protection. Separate with commas or new "
"lines."
msgstr ""
#: documents/models.py:2115
#: documents/models.py:2114
msgid "AI suggestion fields"
msgstr ""
#: documents/models.py:2119
#: documents/models.py:2118
msgid "Which of the AI-suggested fields to apply to the document."
msgstr ""
#: documents/models.py:2124
#: documents/models.py:2123
msgid "create missing objects"
msgstr ""
#: documents/models.py:2127
#: documents/models.py:2126
msgid ""
"Create suggested tags, correspondents, document types and storage paths that "
"do not already exist instead of skipping them."
msgstr ""
#: documents/models.py:2133
#: documents/models.py:2132
msgid "overwrite existing values"
msgstr ""
#: documents/models.py:2136
#: documents/models.py:2135
msgid ""
"Apply suggestions even if the document already has a value for that field. "
"Tags are always added to, never replaced."
msgstr ""
#: documents/models.py:2142
#: documents/models.py:2141
msgid "workflow action"
msgstr ""
#: documents/models.py:2143
#: documents/models.py:2142
msgid "workflow actions"
msgstr ""
#: documents/models.py:2158
#: documents/models.py:2157
msgid "triggers"
msgstr ""
#: documents/models.py:2165
#: documents/models.py:2164
msgid "actions"
msgstr ""
#: documents/models.py:2168 paperless_mail/models.py:154
#: documents/models.py:2167 paperless_mail/models.py:154
msgid "enabled"
msgstr ""
#: documents/models.py:2179
#: documents/models.py:2178
msgid "workflow"
msgstr ""
#: documents/models.py:2183
#: documents/models.py:2182
msgid "workflow trigger type"
msgstr ""
#: documents/models.py:2197
#: documents/models.py:2196
msgid "date run"
msgstr ""
#: documents/models.py:2203
#: documents/models.py:2202
msgid "workflow run"
msgstr ""
#: documents/models.py:2204
#: documents/models.py:2203
msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:483 documents/serialisers.py:844
#: documents/serialisers.py:2884 documents/views.py:355 documents/views.py:621
#: documents/views.py:2761 paperless_mail/serialisers.py:156
#: documents/serialisers.py:2850 documents/views.py:354 documents/views.py:620
#: documents/views.py:2760 paperless_mail/serialisers.py:156
msgid "Insufficient permissions."
msgstr ""
@@ -1670,39 +1670,39 @@ msgstr ""
msgid "Invalid color."
msgstr ""
#: documents/serialisers.py:2351
#: documents/serialisers.py:2317
#, python-format
msgid "File type %(type)s not supported"
msgstr ""
#: documents/serialisers.py:2395
#: documents/serialisers.py:2361
#, python-format
msgid "Custom field id must be an integer: %(id)s"
msgstr ""
#: documents/serialisers.py:2402
#: documents/serialisers.py:2368
#, python-format
msgid "Custom field with id %(id)s does not exist"
msgstr ""
#: documents/serialisers.py:2419 documents/serialisers.py:2429
#: documents/serialisers.py:2385 documents/serialisers.py:2395
msgid ""
"Custom fields must be a list of integers or an object mapping ids to values."
msgstr ""
#: documents/serialisers.py:2424
#: documents/serialisers.py:2390
msgid "Some custom fields don't exist or were specified twice."
msgstr ""
#: documents/serialisers.py:2571
#: documents/serialisers.py:2537
msgid "Invalid variable detected."
msgstr ""
#: documents/serialisers.py:2940
#: documents/serialisers.py:2906
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2970 documents/views.py:4794
#: documents/serialisers.py:2936 documents/views.py:4698
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
@@ -1953,40 +1953,40 @@ msgstr ""
msgid ", "
msgstr ""
#: documents/views.py:310
#: documents/views.py:309
msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr ""
#: documents/views.py:348 documents/views.py:2758
#: documents/views.py:347 documents/views.py:2757
msgid "Invalid more_like_id"
msgstr ""
#: documents/views.py:1685
#: documents/views.py:1684
msgid "Invalid AI configuration."
msgstr ""
#: documents/views.py:1696
#: documents/views.py:1695
msgid "AI backend request timed out."
msgstr ""
#: documents/views.py:1708
#: documents/views.py:1707
msgid "AI backend rejected the request. Check logs for details."
msgstr ""
#: documents/views.py:4810
#: documents/views.py:4714
#, python-format
msgid "Insufficient permissions to share document %(id)s."
msgstr ""
#: documents/views.py:4857
#: documents/views.py:4761
msgid "Bundle is already being processed."
msgstr ""
#: documents/views.py:4923
#: documents/views.py:4827
msgid "The share link bundle is still being prepared. Please try again later."
msgstr ""
#: documents/views.py:4937
#: documents/views.py:4841
msgid "The share link bundle is unavailable."
msgstr ""
+8 -26
View File
@@ -46,16 +46,6 @@ def _build_document_reference(
}
def _permitted_documents_filter(store, permitted_ids: set[int]):
"""Scope retrieval to ``permitted_ids`` using whichever of an inclusion or
exclusion list is shorter, since a long inclusion list is much slower for
the vector store than a long exclusion list."""
denied_ids = store.indexed_document_ids() - permitted_ids
if len(permitted_ids) <= len(denied_ids):
return document_id_filters(str(pk) for pk in permitted_ids)
return exclude_document_ids_filter(str(pk) for pk in denied_ids)
def _get_document_references(
documents: QuerySet[Document],
top_nodes: list,
@@ -138,23 +128,24 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig()
permitted_ids: set[int] | None = None
if unrestricted:
# Exclude trashed ids (usually few) rather than send every permitted
# id on each query. Trashed documents stay indexed until permanent
# deletion (post_delete, not trash), so must be excluded explicitly.
# Exclude trashed ids (usually few) instead of an IN filter over the
# full permitted set, which risks the vector store's bound parameter
# limit (_MAX_IN_VALUES) on large installs. Trashed documents stay
# indexed until permanent deletion (delete_document_from_llm_index
# hangs off post_delete, not trash), so must be excluded explicitly.
trashed_ids = Document.deleted_objects.values_list("pk", flat=True)
filters = exclude_document_ids_filter(str(pk) for pk in trashed_ids)
else:
permitted_ids = set(documents.values_list("pk", flat=True))
filters = document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
# Hold the shared read lock only while reading the index, so the compaction
# swap cannot replace the database file while this connection is open.
# Everything after retrieval works on the nodes already in memory, so a slow
# LLM stream never holds the lock.
with read_store() as store:
if permitted_ids is not None:
filters = _permitted_documents_filter(store, permitted_ids)
index = load_or_build_index(config, store)
retriever = VectorIndexRetriever(
index=index,
@@ -168,15 +159,6 @@ def _stream_chat_with_documents(
with db_connection_released():
top_nodes = retriever.retrieve(query_str)
if permitted_ids is not None:
# An exclusion filter fails open for a document indexed after the
# permitted set was read, so drop anything not explicitly permitted.
top_nodes = [
node
for node in top_nodes
if int(node.metadata["document_id"]) in permitted_ids
]
if not top_nodes:
logger.warning("No nodes found for the given documents.")
yield CHAT_NO_CONTENT_MESSAGE
-9
View File
@@ -141,15 +141,6 @@ class DocumentMetaTable:
(DocumentMetaRow(r["document_id"], r["modified"]) for r in batch),
)
@staticmethod
def all_document_ids(conn: sqlite3.Connection) -> set[int]:
"""Every indexed document_id. document_meta has one row per document,
so this is a cheap primary-key scan, unlike reading the vec0 table."""
return {
row["document_id"]
for row in conn.execute("SELECT document_id FROM document_meta")
}
@staticmethod
def all_modified_times(conn: sqlite3.Connection) -> dict[str, str]:
"""Full document_id -> modified map, for get_modified_times()'s
+4 -67
View File
@@ -13,7 +13,6 @@ from django.utils import timezone
from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode
from llama_index.core.vector_stores.types import FilterOperator
from documents.models import Document
from paperless_ai import chat
@@ -413,8 +412,10 @@ class TestStreamChatRetrieval:
WHEN:
- stream_chat_with_documents is called with unrestricted=True
THEN:
- The retriever receives a NOT IN filter excluding zero ids
rather than an IN list of every permitted id
- The retriever receives a NOT IN filter excluding zero ids, so
the whole index is effectively searched -- and no IN-list is
built from the full permitted set, which is what risks the
vector store's safety limit on large installs
"""
document = DocumentFactory.create(content="indexed document content")
indexing.llm_index_add_or_update_document(document)
@@ -467,70 +468,6 @@ class TestStreamChatRetrieval:
assert str(trashed.pk) in filter_values
assert str(kept.pk) not in filter_values
def test_restricted_chat_excludes_denied_ids_when_few_are_denied(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
captured_filters: list[Any],
) -> None:
"""
GIVEN:
- Three indexed documents, one of which the caller may not see
WHEN:
- A restricted caller chats
THEN:
- The retriever receives a NOT IN filter on the denied id, not an
IN list of the permitted ids
"""
docs = DocumentFactory.create_batch(3, content="indexed document content")
for doc in docs:
indexing.llm_index_add_or_update_document(doc)
permitted = Document.objects.filter(pk__in=[docs[0].pk, docs[1].pk])
list(chat.stream_chat_with_documents("question?", permitted))
metadata_filter = captured_filters[0].filters[0]
assert metadata_filter.operator == FilterOperator.NIN
assert metadata_filter.value == [str(docs[2].pk)]
def test_restricted_chat_drops_retrieved_nodes_of_unpermitted_documents(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
mocker: pytest_mock.MockerFixture,
) -> None:
"""
GIVEN:
- A retriever that returns a node for a document the caller may
not see, as when it is indexed after the permitted set was read
WHEN:
- A restricted caller chats
THEN:
- Only the node of the permitted document reaches the synthesizer
"""
permitted_doc = DocumentFactory.create(content="permitted content")
denied_doc = DocumentFactory.create(content="denied content")
permitted_node = MagicMock(metadata={"document_id": str(permitted_doc.pk)})
denied_node = MagicMock(metadata={"document_id": str(denied_doc.pk)})
mocker.patch("paperless_ai.chat.AIClient")
retriever = mocker.patch(
"llama_index.core.retrievers.VectorIndexRetriever",
).return_value
retriever.retrieve.return_value = [denied_node, permitted_node]
synthesizer = mocker.patch(
"llama_index.core.response_synthesizers.get_response_synthesizer",
).return_value
synthesizer.synthesize.return_value.response_gen = iter(["chunk"])
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=permitted_doc.pk),
),
)
synthesizer.synthesize.assert_called_once_with("question?", [permitted_node])
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
+21 -23
View File
@@ -18,7 +18,7 @@ from paperless_ai.migrations import Migration
from paperless_ai.migrations import m0001_v1_to_v2
from paperless_ai.tables import DocumentChunksTable
from paperless_ai.tables import DocumentMetaTable
from paperless_ai.vector_store import _MAX_INLINE_IN_VALUES
from paperless_ai.vector_store import _MAX_IN_VALUES
from paperless_ai.vector_store import DB_FILENAME
from paperless_ai.vector_store import DB_PAGE_SIZE
from paperless_ai.vector_store import DEFAULT_TABLE_NAME
@@ -354,39 +354,37 @@ class TestBuildWhere:
assert params == []
@pytest.mark.parametrize(
("build_filter", "expected_ids"),
[
pytest.param(_in_filter, ["a", "c"], id="in"),
pytest.param(_nin_filter, ["b"], id="nin"),
],
"build_filter",
[_in_filter, _nin_filter],
ids=["in", "nin"],
)
def test_query_with_oversized_filter_scopes_results(
def test_fails_closed_when_filter_exceeds_max_values(
self,
store: PaperlessSqliteVecVectorStore,
build_filter: Callable[[list[str]], MetadataFilters],
expected_ids: list[str],
caplog: pytest.LogCaptureFixture,
) -> None:
"""
GIVEN:
- A store holding documents 1, 2 and 40000
- An IN or NOT IN filter naming documents 1 and 40000, padded
past _MAX_INLINE_IN_VALUES with unindexed ids
- An IN or NOT IN filter with more values than _MAX_IN_VALUES
(SQLite's own bound-parameter limit is 32766; this guard sits
below that with headroom for the query's other bound parameters)
WHEN:
- The store is queried and get_nodes() is called with that filter
- _build_where() translates it to SQL
THEN:
- Both return exactly the documents the filter allows
- It fails closed ("1 = 0", no params) instead of building a
clause SQLite would reject, and logs a warning -- this filter
scopes document access, so refusing to build it must never
widen the scope to "everything" by accident. Failing open on
a NOT IN would surface exactly the excluded rows
"""
store.add(
[make_node("a", 1), make_node("b", 2), make_node("c", 40000)],
)
padding = range(100, 100 + _MAX_INLINE_IN_VALUES)
filters = build_filter([str(i) for i in (1, 40000, *padding)])
oversized = build_filter([str(i) for i in range(_MAX_IN_VALUES + 1)])
result = _query(store, [0.0] * DIM, top_k=5, filters=filters)
nodes = store.get_nodes(filters=filters)
with caplog.at_level("WARNING"):
where, params = _build_where(oversized)
assert sorted(result.ids) == expected_ids
assert sorted(n.node_id for n in nodes) == expected_ids
assert where == "(1 = 0)"
assert params == []
assert "document_id" in caplog.text
def test_query_with_untranslatable_filter_returns_no_rows(
self,
+25 -16
View File
@@ -89,10 +89,15 @@ class _Row(NamedTuple):
embedding: bytes
# _build_where(): the largest IN / NOT IN list bound as one parameter per value,
# leaving headroom under SQLite's 32766-variable limit for the query's other
# parameters. Longer lists are bound as a single JSON array via json_each().
_MAX_INLINE_IN_VALUES = 32700
# _build_where(): the largest IN value list translated into bound SQL
# parameters. SQLite's own hard limit (SQLITE_MAX_VARIABLE_NUMBER) is 32766
# by default; this leaves headroom below that for the query's other bound
# parameters (the embedding blob, k, and any NE clause) and for the limit
# itself to move. An IN filter this large should not happen in practice --
# callers are expected to pass None (no filter) rather than every id when
# the filter would not actually narrow anything -- so this is a guard
# against a future regression, not a normal code path.
_MAX_IN_VALUES = 32700
def _pack(embedding: Sequence[float]) -> bytes:
@@ -115,7 +120,7 @@ def _vec0_params(rows: list[_Row]) -> list[tuple[str, int, str, bytes]]:
return [(r.chunk_id, r.document_id, r.node_content, r.embedding) for r in rows]
def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int | str]]:
def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
"""Translate the EQ / IN / NIN / NE filters we use into a parameterized
SQL clause on vec0 metadata columns. Returns ("", []) when there is
nothing to filter. document_id is vec0's only filterable column and is
@@ -127,7 +132,7 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int | str]]
if filters is None or not filters.filters:
return "", []
clauses: list[str] = []
params: list[int | str] = []
params: list[int] = []
for f in filters.filters:
# filters.filters is Union[MetadataFilter, ExactMatchFilter, MetadataFilters];
# we only build MetadataFilter entries, so skip anything else at runtime.
@@ -144,11 +149,21 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int | str]]
# excludes nothing, so it matches everything.
clauses.append("1 = 0" if is_in else "1 = 1")
continue
if len(values) > _MAX_INLINE_IN_VALUES:
clauses.append(
f"{f.key} {sql_op} (SELECT value FROM json_each(?))",
if len(values) > _MAX_IN_VALUES:
# Refuse rather than risk SQLite's own bound-parameter limit
# ("too many SQL variables"): a list this large must match no
# rows, never widen the scope to "everything" -- true for
# NOT IN too, where failing open would surface every
# excluded row.
logger.warning(
"Refusing to build a %s filter on %r with %d values "
"(over the %d-value safety limit); returning no rows.",
sql_op,
f.key,
len(values),
_MAX_IN_VALUES,
)
params.append(json.dumps(values))
clauses.append("1 = 0")
continue
placeholders = ",".join("?" for _ in values)
clauses.append(f"{f.key} {sql_op} ({placeholders})")
@@ -543,12 +558,6 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
return {}
return DocumentMetaTable.all_modified_times(self._conn)
def indexed_document_ids(self) -> set[int]:
"""Return the ids of all indexed documents."""
if not self.table_exists():
return set()
return DocumentMetaTable.all_document_ids(self._conn)
@property
def _db_path(self) -> str:
return str(Path(self._uri) / DB_FILENAME)
Generated
+75 -1130
View File
File diff suppressed because it is too large. Load diff