Compare commits

...
Author SHA1 Message Date
stumpylog e02f6fff56 Bumps jbig2enc to 0.32 2026-11-12 09:20:28 -08:00
Trenton H a1d76f3a19 Fix: Enforce CRLF when writing an email message to the file (#14745) 2026-11-12 00:18:43 +00:00
GitHub Actions f5ca9bbcb6 Auto translate strings 2026-11-11 23:51:43 +00:00
shamoon 60a35edee1 Enhancement: make bulk metadata operations run async (#14711) 2026-11-11 15:50:15 -08:00
Trenton H 7cdcd71a6d Performance: load a cached Hugging Face embedding model without contacting the Hub (#14728)
Every chat question and AI suggestion request builds the embedding model, and sentence-transformers
checks the Hub for a newer revision each time. With the model already cached that check cost about
2 seconds per request (0.2 seconds without it), and it fails outright when the Hub is unreachable.

Load from the local cache first and only download when the model is missing. A full rebuild still
checks the Hub, so a new model release is picked up together with re-embedding every document
instead of silently mixing revisions.
2026-11-11 21:53:27 +00:00
shamoon 759f9cd22b Fix: restore image without archive version thumbnail generation (#14741) 2026-11-11 10:12:47 -08:00
Trenton H d05e96cda2 Fix: Blacklist certain OCR user arguments that paperless itself controls/uses (#14734) 2026-11-11 03:13:20 +00:00
shamoon fefce7be1b Fix: remove redundant ui text filters (#14736) 2026-11-10 18:06:51 -08:00
GitHub Actions cc65226d6d Auto translate strings 2026-11-11 00:55:05 +00:00
shamoon 00096e0252 Fix: fix allowed_hosts default (#14735) 2026-11-11 00:53:49 +00:00
shamoon 2829e78009 Chore: add agents.md, update contributing guidelines (#14713) 2026-11-10 23:35:41 +00:00
dependabot[bot]andstumpylog d7d4014bbd Chore(deps): Bump the pre-commit-dependencies group with 4 updates (#14636)
* Chore(deps): Bump the pre-commit-dependencies group with 4 updates

Bumps the pre-commit-dependencies group with 4 updates: [https://github.com/rbubley/mirrors-prettier](https://github.com/rbubley/mirrors-prettier), prettier, [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit) and [https://github.com/tox-dev/pyproject-fmt](https://github.com/tox-dev/pyproject-fmt).


Updates `https://github.com/rbubley/mirrors-prettier` from v3.9.9 to 3.9.10
- [Commits](https://github.com/rbubley/mirrors-prettier/compare/v3.9.9...v3.9.10)

Updates `prettier` from 3.9.9 to 3.9.10

Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.16.9 to 0.17.2
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](https://github.com/astral-sh/ruff-pre-commit/compare/v0.16.9...v0.17.2)

Updates `https://github.com/tox-dev/pyproject-fmt` from v2.29.4 to 2.30.3
- [Release notes](https://github.com/tox-dev/pyproject-fmt/releases)
- [Commits](https://github.com/tox-dev/pyproject-fmt/compare/v2.29.4...v2.30.3)

---
updated-dependencies:
- dependency-name: https://github.com/rbubley/mirrors-prettier
  dependency-version: 3.9.10
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: prettier
  dependency-version: 3.9.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.17.2
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/tox-dev/pyproject-fmt
  dependency-version: 2.30.3
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Linting fixes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
2026-11-10 17:49:06 +00:00
shamoon a139bdff46 Fixhancement: add automatic clearing of sessions (#14722) 2026-11-10 08:09:25 -08:00
GitHub Actions 936b1b3bfa Auto translate strings 2026-11-09 06:53:56 +00:00
shamoon e0c33b3d39 Tweak: chat reference tweaks (#14723) 2026-11-08 22:52:34 -08:00
58 changed files with 2082 additions and 546 deletions

No files matched your search

+1 -1
View File
@@ -8,7 +8,7 @@ ARG DEBIAN_FRONTEND=noninteractive
ARG TARGETARCH
# Can be workflow provided, defaults set for manual building
ARG JBIG2ENC_VERSION=0.30
ARG JBIG2ENC_VERSION=0.32
# Set Python environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
@@ -1,5 +1,23 @@
title: "[Feature Request] "
body:
- type: markdown
attributes:
value: |
#### Use of AI tools
If this request was generated, in whole or in part, by an AI tool or agent you **must** say so using the field below. Failing to clearly disclose this is a violation of our [Code of Conduct](https://github.com/paperless-ngx/paperless-ngx/blob/main/CODE_OF_CONDUCT.md).
The project prefers thoughtful, well-described ideas to AI-generated content. Please describe the problem, your use case, the desired behavior, and who would benefit.
- type: dropdown
id: ai-usage
attributes:
label: AI usage
description: Was any part of this request written by an AI tool or agent?
options:
- No AI tools were used
- Yes, AI tools helped write this request
validations:
required: true
- type: textarea
id: description
attributes:
+16
View File
@@ -1,5 +1,21 @@
title: "[Support] "
body:
- type: markdown
attributes:
value: |
#### Use of AI tools
If this post was generated, in whole or in part, by an AI tool or agent you **must** say so using the field below. Failing to clearly disclose this is a violation of our [Code of Conduct](https://github.com/paperless-ngx/paperless-ngx/blob/main/CODE_OF_CONDUCT.md).
- type: dropdown
id: ai-usage
attributes:
label: AI usage
description: Was any part of this post written by an AI tool or agent?
options:
- No AI tools were used
- Yes, AI tools helped write this post
validations:
required: true
- type: textarea
id: description
attributes:
+4 -4
View File
@@ -38,7 +38,7 @@ repos:
- json
# See https://github.com/prettier/prettier/issues/15742 for the fork reason
- repo: https://github.com/rbubley/mirrors-prettier
rev: 'v3.9.9'
rev: 'v3.9.10'
hooks:
- id: prettier
types_or:
@@ -46,16 +46,16 @@ repos:
- ts
- markdown
additional_dependencies:
- prettier@3.9.9
- prettier@3.9.10
- 'prettier-plugin-organize-imports@4.3.0'
# Python hooks
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.9
rev: v0.17.2
hooks:
- id: ruff-check
- id: ruff-format
- repo: https://github.com/tox-dev/pyproject-fmt
rev: "v2.29.4"
rev: "v2.30.3"
hooks:
- id: pyproject-fmt
additional_dependencies: [tomli]
+44
View File
@@ -0,0 +1,44 @@
# Guidance for AI agents
Read [CONTRIBUTING.md](CONTRIBUTING.md) before working on this repository, especially [Use of AI Tools](CONTRIBUTING.md#use-of-ai-tools). These contribution rules apply to all agent-assisted work.
## Contribution boundaries
- Do not open or submit pull requests, including "on behalf of" a human. The project does not accept PRs from AI agents. Prepare local changes for a human to review, verify, and submit personally.
- The project does not accept PRs that are entirely or mostly AI-derived. Human submission and disclosure do not override this rule.
- AI-generated content must be clearly disclosed as described in the contributing guidelines. When handing work back to a human, identify what was generated or changed with AI assistance so they can disclose it accurately.
- Trust matters. Encourage first-time contributors to discuss their proposed change before preparing a PR, especially for large changes or changes involving AI-generated code. New features and enhancements should almost always target an existing feature request with community interest and discussion.
- The project prefers thoughtful, well-described ideas to AI-generated pull requests. Help contributors describe their problem, use case, desired behavior, and who would benefit. An implementation is not required for a valuable contribution, and a flawed implementation can obscure a useful idea before it receives proper discussion.
- AI tools have lowered the effort needed to produce a PR, but reviewing and maintaining it still requires substantial work. Encourage discussion of the idea before investing in implementation and review.
- Issue reports must describe observed behavior, reproduction steps, and relevant logs. Do not add speculative root causes, code analysis, or proposed fixes to agent-generated reports.
## Security reports
- AI-assisted security reports must meet the legitimacy and verification requirements in [SECURITY.md](SECURITY.md#ai-assisted-submissions). A human must verify a real Paperless-ngx code path, concrete reproduction steps, affected versions, and realistic security impact before submission.
- Account for existing permission checks, configuration requirements, and trust boundaries. Speculation, scanner output, or concerns about third-party software or deployment choices without a demonstrated Paperless-ngx vulnerability do not meet this threshold and may be closed without extended discussion.
- Disclose AI assistance and use the private vulnerability reporting channel specified in [SECURITY.md](SECURITY.md#reporting-a-vulnerability).
## Working on changes
- Read the relevant code, tests, and documentation before editing. Paperless-ngx is complex: trace interactions between affected components rather than assuming a change is isolated.
- Keep changes focused on the requested work and follow existing patterns. Avoid unrelated refactoring, formatting, dependency updates, or generated-file changes.
- Consider effects on permissions, document processing, background tasks, storage, and the API when those areas are involved.
- Add or update meaningful tests for behavior changes and regressions. Do not weaken tests or checks to make a change pass.
- Update user or developer documentation when behavior or setup changes.
- Keep credentials and private documents out of commits, fixtures, logs, and examples.
## Repository and validation
- `src/`: Django back end.
- `src-ui/`: Angular front end, managed with `pnpm`.
- `docs/`: Documentation.
- `docker/` and `scripts/`: Deployment and development tooling.
- Base functional changes on `dev`; documentation-only changes may target `main`.
Follow [the development documentation](docs/development.md) for environment setup and testing. Common checks after setup are:
- Back end: run `uv run pytest` from `src/` (use a test path to focus on the affected area).
- Front end: run `pnpm test` and `pnpm lint` from `src-ui/`.
- Formatting and other pre-commit checks: run `uv run prek run --files <changed-files>` from the repository root.
When handing work back, summarize the changes, the checks actually run and their results, and any remaining uncertainty. Clearly state when a check could not be run; do not claim verification that did not happen. The human contributor must understand and independently verify the contribution before submitting it.
+16 -3
View File
@@ -4,6 +4,18 @@ If you feel like contributing to the project, please do! Bug fixes and improveme
⚠️ Please note: Pull requests that implement a new feature or enhancement _should almost always target an existing feature request_ with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. Pull requests that are opened without meeting this requirement may not be merged.
## Trust and collaboration
Paperless-ngx places a high value on trust between contributors and maintainers. If you have not contributed before, consider starting with a discussion before preparing a pull request, especially for a large contribution or one involving AI-generated code.
Paperless-ngx is complex, and understanding how its components interact takes time. A change that appears correct in isolation can have consequences elsewhere in the application. Contributors should understand the code they submit, explain how it fits into the project, and work with maintainers through review. Starting with smaller, focused contributions can help build that understanding and trust.
We prefer thoughtful, well-described ideas to AI-generated pull requests. You do not need to supply an implementation for your contribution to be valuable. Explain the problem you have encountered, your use case, the behavior you would like to see, and who would benefit. Concrete examples, alternatives you have considered, and relevant constraints help the community discuss and refine the idea.
AI tools have lowered the effort needed to produce a pull request, but evaluating one still requires substantial maintainer time. Maintainers still need to establish whether the implementation is correct, fits the project, and can be maintained over time.
A flawed implementation can also obscure a good idea: discussion becomes focused on problems with the generated code before the community has considered whether the underlying proposal is useful. Starting with the idea gives it a fair discussion on its own merits and lets contributors and maintainers agree on an approach before investing in implementation and review.
If you want to implement something big:
- As above, please start with a discussion! Maybe something similar is already in development and we can make it happen together.
@@ -61,9 +73,10 @@ This project does not specifically prohibit the use of AI tools or agents _durin
1. Any content — code, pull request descriptions, issue reports or comments — that was generated in whole or in part by an AI tool or agent **must be clearly disclosed as such**. Failing to clearly disclose this is considered a violation of our [Code of Conduct](CODE_OF_CONDUCT.md) and may result in your contributions being closed and further action being taken.
2. AI-generated code must not violate copyright protections.
3. We will not accept PRs that are entirely or mostly AI-derived.
4. Issue reports must describe _observed behavior only_: what happened, how to reproduce it and the relevant logs. Reports generated by an AI agent should not include code analysis, speculation about the root cause or suggest fixes.
5. Contributors are responsible for everything they submit, regardless of how it was produced. Please do not open issues or PRs you have not verified yourself.
3. We will not accept PRs opened or submitted by AI agents, including agents acting "on behalf of" a human. A human contributor must personally review, verify, and submit the contribution and take responsibility for it throughout review.
4. We will not accept PRs that are entirely or mostly AI-derived.
5. Issue reports must describe _observed behavior only_: what happened, how to reproduce it and the relevant logs. Reports generated by an AI agent should not include code analysis, speculation about the root cause or suggest fixes.
6. Contributors are responsible for everything they submit, regardless of how it was produced. Please do not open issues or PRs you have not verified yourself.
# Translating Paperless-ngx
+1 -1
View File
@@ -99,7 +99,7 @@ ARG DEBIAN_FRONTEND=noninteractive
ARG TARGETARCH
# Can be workflow provided, defaults set for manual building
ARG JBIG2ENC_VERSION=0.30
ARG JBIG2ENC_VERSION=0.32
# Set Python environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
+2
View File
@@ -565,6 +565,8 @@ Specify `rebuild` to build the index from scratch from all documents in the data
this the first time you enable the feature, or after changing the embedding backend or
model.
When using the `huggingface` embedding backend, a model that is already downloaded is loaded from the local cache without contacting the Hugging Face Hub. Only `rebuild` checks the Hub for a newer revision of the model, so a model update is picked up together with re-embedding every document. If the model is not cached yet, it is downloaded on first use.
Specify `update` to incrementally index new and changed documents. This is what the
scheduled task runs.
+22 -11
View File
@@ -501,20 +501,20 @@ See the [Django project documentation on the settings](https://docs.djangoprojec
#### [`PAPERLESS_ALLOWED_HOSTS=<comma-separated-list>`](#PAPERLESS_ALLOWED_HOSTS) {#PAPERLESS_ALLOWED_HOSTS}
: If you're planning on putting Paperless on the open internet, then
you really should set this value to the domain name you're using.
Failing to do so leaves you open to HTTP host header attacks.
: If you expose Paperless to the internet, set `PAPERLESS_URL` to its public URL
or list its public hostname in `PAPERLESS_ALLOWED_HOSTS` to help protect against
HTTP Host header attacks.
You can read more about this in [the Django project's documentation](https://docs.djangoproject.com/en/4.1/topics/security/#host-header-validation)
Just remember that this is a comma-separated list, so
"example.com" is fine, as is "example.com,www.example.com", but
NOT " example.com" or "example.com,"
Separate multiple hostnames with commas, as in `example.com,www.example.com`.
Can also be set using PAPERLESS_URL (see above).
"localhost" is always allowed for docker healthcheck
Defaults to "\*", which is all hosts.
- With neither setting configured, Paperless accepts any host.
- With only `PAPERLESS_URL` set, Paperless accepts the hostname from that URL
and `localhost` (used by the Docker health check).
- `PAPERLESS_ALLOWED_HOSTS` adds hosts to `localhost` and the hostname from
`PAPERLESS_URL`, if set. A blank list adds no hosts.
- With `PAPERLESS_ALLOWED_HOSTS=*`, Paperless accepts any host, even when
`PAPERLESS_URL` is set.
#### [`PAPERLESS_CORS_ALLOWED_HOSTS=<comma-separated-list>`](#PAPERLESS_CORS_ALLOWED_HOSTS) {#PAPERLESS_CORS_ALLOWED_HOSTS}
@@ -873,6 +873,14 @@ If both the [PAPERLESS_ACCOUNT_DEFAULT_GROUPS](#PAPERLESS_ACCOUNT_DEFAULT_GROUPS
Defaults to 1209600 (2 weeks)
#### [`PAPERLESS_SESSION_CLEANUP_TASK_CRON=<cron expression>`](#PAPERLESS_SESSION_CLEANUP_TASK_CRON) {#PAPERLESS_SESSION_CLEANUP_TASK_CRON}
: Configures the schedule to remove expired sessions from the session store.
Django does not prune these automatically, so without this they would
accumulate indefinitely.
Defaults to `0 3 * * *`, once per day at 03:00.
#### [`PAPERLESS_TOKEN_THROTTLE_RATE=<rate>`](#PAPERLESS_TOKEN_THROTTLE_RATE) {#PAPERLESS_TOKEN_THROTTLE_RATE}
: Rate limit for the API token authentication endpoint (`/api/token/`), used to mitigate brute-force login attempts.
@@ -1149,6 +1157,9 @@ they use underscores instead of dashes.
be changed by trusted users. This applies to the `AppConfig` permission as well,
which allows setting these arguments from the UI.
Arguments paperless manages itself (`input_file_or_options`, `output_file`
and `sidecar`) are rejected by the API and ignored if set here.
Specify arguments as a JSON dictionary. Keep note of lower case
booleans and double quoted parameter names and strings. Examples:
+1 -6
View File
@@ -212,12 +212,7 @@ extend-select = [
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
]
ignore = [
"DJ001",
"PLC0415",
"RUF012",
"SIM105",
]
ignore = [ "DJ001", "ICN001", "PLC0415", "RUF012", "SIM105" ]
# Migrations
per-file-ignores."*/migrations/*.py" = [
"E501",
+175 -133
View File
File diff suppressed because it is too large. Load diff
@@ -7,10 +7,15 @@ import {
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { Router } from '@angular/router'
import {
ActivatedRoute,
convertToParamMap,
ParamMap,
Router,
} from '@angular/router'
import { RouterTestingModule } from '@angular/router/testing'
import { NgbModal, NgbModalRef, NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { of, throwError } from 'rxjs'
import { BehaviorSubject, of, throwError } from 'rxjs'
import { routes } from 'src/app/app-routing.module'
import {
PaperlessTask,
@@ -159,6 +164,7 @@ const sectionCountResponse = {
}
describe('TasksComponent', () => {
let queryParams: BehaviorSubject<ParamMap>
let component: TasksComponent
let fixture: ComponentFixture<TasksComponent>
let tasksService: TasksService
@@ -169,6 +175,7 @@ describe('TasksComponent', () => {
let toastService: ToastService
beforeEach(async () => {
queryParams = new BehaviorSubject(convertToParamMap({}))
TestBed.configureTestingModule({
imports: [
NgbModule,
@@ -181,6 +188,17 @@ describe('TasksComponent', () => {
ConfirmDialogComponent,
],
providers: [
{
provide: ActivatedRoute,
useValue: {
queryParamMap: queryParams.asObservable(),
snapshot: {
get queryParamMap() {
return queryParams.value
},
},
},
},
{
provide: PermissionsService,
useValue: {
@@ -219,7 +237,11 @@ describe('TasksComponent', () => {
})
// answers task list requests with the fixture tasks matching their status filter
function flushTaskLists(pageSize: number, count?: number) {
function flushTaskLists(
pageSize: number,
count?: number,
filteredTasks: PaperlessTask[] = tasks
) {
httpTestingController
.match(
(req) =>
@@ -228,7 +250,9 @@ describe('TasksComponent', () => {
)
.forEach((req) => {
const statuses = req.request.params.getAll('status')
const results = tasks.filter((task) => statuses.includes(task.status))
const results = filteredTasks.filter((task) =>
statuses.includes(task.status)
)
req.flush({ count: count ?? results.length, results })
})
}
@@ -238,6 +262,71 @@ describe('TasksComponent', () => {
flushTaskLists(25, count)
}
it('consumes a task link on load into the normal resettable Task ID filter', () => {
jest
.spyOn(tasksService, 'list')
.mockReturnValue(of({ count: 0, results: [] }))
jest
.spyOn(tasksService, 'statusCounts')
.mockReturnValue(of(sectionCountResponse))
jest.spyOn(router, 'navigate').mockResolvedValue(true)
queryParams.next(convertToParamMap({ task_id: 'linked-task' }))
fixture.destroy()
fixture = TestBed.createComponent(TasksComponent)
component = fixture.componentInstance
fixture.detectChanges()
expect(router.navigate).toHaveBeenCalledTimes(1)
expect(router.navigate).toHaveBeenCalledWith(
[],
expect.objectContaining({
queryParams: { task_id: null },
replaceUrl: true,
})
)
expect(component.filterTargetName).toBe('Task ID')
expect(component.filterText).toBe('linked-task')
expect(component.isFiltered).toBe(true)
component.setSection(TaskSection.InProgress)
expect(tasksService.list).toHaveBeenLastCalledWith(1, 25, {
task_id: 'linked-task',
status: [PaperlessTaskStatus.Pending, PaperlessTaskStatus.Started],
})
component.resetFilters()
expect(component.filterText).toBe('')
expect(component.isFiltered).toBe(false)
expect(tasksService.list).toHaveBeenLastCalledWith(1, 25, {
acknowledged: false,
status: [PaperlessTaskStatus.Pending, PaperlessTaskStatus.Started],
})
expect(router.navigate).toHaveBeenCalledTimes(1)
})
it('supports manually filtering by Task ID and clearing the text filter', () => {
jest
.spyOn(tasksService, 'list')
.mockReturnValue(of({ count: 0, results: [] }))
jest
.spyOn(tasksService, 'statusCounts')
.mockReturnValue(of(sectionCountResponse))
component.setSection(TaskSection.InProgress)
component.setFilterTarget(TaskFilterTargetID.TaskId)
component.filterText = tasks[0].task_id
jest.advanceTimersByTime(150)
expect(tasksService.list).toHaveBeenLastCalledWith(1, 25, {
task_id: tasks[0].task_id,
status: [PaperlessTaskStatus.Pending, PaperlessTaskStatus.Started],
})
component.pagedTasks.set(tasks)
expect(component.tasksForSection(TaskSection.NeedsAttention)).toEqual([
tasks[0],
])
component.resetFilter()
expect(tasksService.list).toHaveBeenLastCalledWith(1, 25, {
acknowledged: false,
status: [PaperlessTaskStatus.Pending, PaperlessTaskStatus.Started],
})
})
it('should display task sections with counts', () => {
expect(component.selectedSection).toBe(TaskSection.All)
expect(component.selectedTaskType).toBeNull()
@@ -771,6 +860,7 @@ describe('TasksComponent', () => {
input.nativeElement.value = '191092'
input.nativeElement.dispatchEvent(new Event('input'))
jest.advanceTimersByTime(150)
flushTaskLists(10, undefined, [tasks[1]])
fixture.detectChanges()
expect(component.filterText).toEqual('191092')
expect(component.tasksForSection(TaskSection.NeedsAttention)).toHaveLength(
@@ -782,6 +872,7 @@ describe('TasksComponent', () => {
component.setSection(TaskSection.InProgress)
component.filterText = 'system'
jest.advanceTimersByTime(150)
flushTaskLists(25, undefined, [tasks[6]])
expect(component.tasksForSection(TaskSection.InProgress)).toHaveLength(1)
expect(component.tasksForSection(TaskSection.InProgress)[0].task_type).toBe(
@@ -815,6 +906,7 @@ describe('TasksComponent', () => {
input.nativeElement.value = 'duplicate'
input.nativeElement.dispatchEvent(new Event('input'))
jest.advanceTimersByTime(150)
flushTaskLists(25, undefined, [tasks[0], tasks[1]])
fixture.detectChanges()
expect(component.filterText).toEqual('duplicate')
expect(component.tasksForSection(TaskSection.NeedsAttention)).toHaveLength(
@@ -822,6 +914,53 @@ describe('TasksComponent', () => {
)
})
it('should show server-matched names when the displayed source is translated', () => {
component.setSection(TaskSection.InProgress)
flushTaskLists(25)
component.filterText = 'email'
jest.advanceTimersByTime(150)
const request = httpTestingController.expectOne(
(req) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('name') === 'email' &&
req.params.get('page_size') === '25'
)
const translatedTask = {
...tasks[5],
trigger_source_display: 'E-Mail',
}
request.flush({ count: 1, results: [translatedTask] })
fixture.detectChanges()
expect(component.totalTasks()).toBe(1)
expect(component.visibleTasks).toEqual([translatedTask])
})
it('should show server-matched duplicates when the displayed result is translated', () => {
component.setSection(TaskSection.NeedsAttention)
flushTaskLists(25)
jest
.spyOn(component, 'taskResultMessage')
.mockReturnValue('Duplikat von Document #311')
component.setFilterTarget(TaskFilterTargetID.Result)
component.filterText = 'duplicate'
jest.advanceTimersByTime(150)
const request = httpTestingController.expectOne(
(req) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('result') === 'duplicate' &&
req.params.get('page_size') === '25'
)
request.flush({ count: 1, results: [tasks[1]] })
fixture.detectChanges()
expect(component.totalTasks()).toBe(1)
expect(component.visibleTasks).toEqual([tasks[1]])
expect(fixture.nativeElement.textContent).toContain('191092.pdf')
})
it('should prefer explicit reason in the result message', () => {
expect(
component.taskResultMessage({
@@ -1,7 +1,7 @@
import { JsonPipe, NgTemplateOutlet } from '@angular/common'
import { Component, inject, OnDestroy, OnInit, signal } from '@angular/core'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { Router, RouterLink } from '@angular/router'
import { ActivatedRoute, Router, RouterLink } from '@angular/router'
import {
NgbCollapseModule,
NgbDropdownModule,
@@ -47,11 +47,13 @@ export enum TaskSection {
export enum TaskFilterTargetID {
Name,
Result,
TaskId,
}
const FILTER_TARGETS = [
{ id: TaskFilterTargetID.Name, name: $localize`Name` },
{ id: TaskFilterTargetID.Result, name: $localize`Result` },
{ id: TaskFilterTargetID.TaskId, name: $localize`Task ID` },
]
const SECTION_LABELS = {
@@ -161,6 +163,7 @@ export class TasksComponent
tasksService = inject(TasksService)
private modalService = inject(NgbModal)
private readonly router = inject(Router)
private readonly route = inject(ActivatedRoute)
private readonly toastService = inject(ToastService)
readonly TaskSection = TaskSection
@@ -275,6 +278,17 @@ export class TasksComponent
}
ngOnInit() {
const taskId = this.route.snapshot.queryParamMap.get('task_id')
if (taskId) {
this.filterTargetID = TaskFilterTargetID.TaskId
this._filterText = taskId
this.router.navigate([], {
relativeTo: this.route,
queryParams: { task_id: null },
queryParamsHandling: 'merge',
replaceUrl: true,
})
}
this.tasksService.reload()
this.reloadPage()
timer(5000, 5000)
@@ -636,17 +650,12 @@ export class TasksComponent
const query = this._filterText.toLowerCase()
if (this.filterTargetID == TaskFilterTargetID.Name) {
return [
this.taskDisplayName(task),
task.task_type_display,
task.trigger_source_display,
]
.filter(Boolean)
.some((value) => value.toLowerCase().includes(query))
if (this.filterTargetID === TaskFilterTargetID.TaskId) {
return task.task_id.toLowerCase() === query
}
return this.taskResultMessage(task)?.toLowerCase().includes(query) ?? false
// Text filtering now handled API-side
return true
}
private tasksForOptionCounts({
@@ -706,9 +715,15 @@ export class TasksComponent
}
if (this._filterText.length) {
params[
this.filterTargetID === TaskFilterTargetID.Name ? 'name' : 'result'
] = this._filterText
const parameter = {
[TaskFilterTargetID.Name]: 'name',
[TaskFilterTargetID.Result]: 'result',
[TaskFilterTargetID.TaskId]: 'task_id',
}[this.filterTargetID]
params[parameter] = this._filterText
if (this.filterTargetID === TaskFilterTargetID.TaskId) {
delete params.acknowledged
}
}
return params
@@ -11,8 +11,10 @@ import { ActivatedRoute, Router } from '@angular/router'
import { RouterTestingModule } from '@angular/router/testing'
import { NgbModal, NgbModalModule, NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { provideUiTour } from 'ngx-ui-tour-ng-bootstrap'
import { of, throwError } from 'rxjs'
import { of, Subject, throwError } from 'rxjs'
import { routes } from 'src/app/app-routing.module'
import { BulkOperationStatus } from 'src/app/data/bulk-operation'
import { PaperlessTaskStatus } from 'src/app/data/paperless-task'
import { SavedView } from 'src/app/data/saved-view'
import { HideableSidebarItemID, SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
@@ -21,10 +23,11 @@ import {
DjangoMessageLevel,
DjangoMessagesService,
} from 'src/app/services/django-messages.service'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import { OpenDocumentsService } from 'src/app/services/open-documents.service'
import {
PermissionType,
PermissionsService,
PermissionType,
} from 'src/app/services/permissions.service'
import { RemoteVersionService } from 'src/app/services/rest/remote-version.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
@@ -95,6 +98,7 @@ describe('AppFrameComponent', () => {
let savedViewSpy
let modalService: NgbModal
let maybeRefreshSpy
let bulkEvents: Subject<BulkOperationStatus>
let tasksService: TasksService
beforeEach(async () => {
@@ -182,6 +186,10 @@ describe('AppFrameComponent', () => {
savedViewSpy = jest.spyOn(savedViewService, 'reload')
maybeRefreshSpy = jest.spyOn(savedViewService, 'maybeRefreshDocumentCounts')
bulkEvents = new Subject<BulkOperationStatus>()
jest
.spyOn(TestBed.inject(WebsocketStatusService), 'onBulkOperationStatus')
.mockReturnValue(bulkEvents.asObservable())
fixture = TestBed.createComponent(AppFrameComponent)
component = fixture.componentInstance
@@ -190,6 +198,78 @@ describe('AppFrameComponent', () => {
fixture.detectChanges()
})
it('refreshes and notifies on completion without client-side task tracking', () => {
const list = TestBed.inject(DocumentListViewService)
const reload = jest.spyOn(list, 'reload').mockImplementation(() => {})
jest.spyOn(list, 'reduceSelectionToFilter').mockImplementation(() => {})
const tasksReload = jest
.spyOn(tasksService, 'reload')
.mockImplementation(() => {})
const success = jest.spyOn(toastService, 'showSuccess')
const refresh = jest
.spyOn(openDocumentsService, 'refreshDocument')
.mockImplementation(() => {})
jest
.spyOn(openDocumentsService, 'getOpenDocuments')
.mockReturnValue([document])
tasksReload.mockClear()
maybeRefreshSpy.mockClear()
bulkEvents.next({
task_id: 'bulk-task',
operation: 'set_permissions',
status: PaperlessTaskStatus.Started,
})
expect(reload).not.toHaveBeenCalled()
bulkEvents.next({
task_id: 'bulk-task',
operation: 'set_permissions',
status: PaperlessTaskStatus.Success,
})
expect(reload).toHaveBeenCalledTimes(1)
expect(reload).toHaveBeenCalledWith(null, false)
expect(refresh).toHaveBeenCalledWith(document.id)
expect(maybeRefreshSpy).toHaveBeenCalledTimes(1)
expect(tasksReload).toHaveBeenCalledTimes(1)
expect(success).toHaveBeenCalledWith('Bulk operation completed.')
fixture.destroy()
bulkEvents.next({
task_id: 'bulk-task',
operation: 'set_permissions',
status: PaperlessTaskStatus.Success,
})
expect(reload).toHaveBeenCalledTimes(1)
})
it('refreshes and reports failed or revoked bulk operations', () => {
const list = TestBed.inject(DocumentListViewService)
const reload = jest.spyOn(list, 'reload').mockImplementation(() => {})
jest.spyOn(list, 'reduceSelectionToFilter').mockImplementation(() => {})
jest.spyOn(tasksService, 'reload').mockImplementation(() => {})
const error = jest.spyOn(toastService, 'showError')
const success = jest.spyOn(toastService, 'showSuccess')
for (const status of [
PaperlessTaskStatus.Failure,
PaperlessTaskStatus.Revoked,
]) {
bulkEvents.next({
task_id: 'bulk-task',
operation: 'set_permissions',
status,
error:
status === PaperlessTaskStatus.Revoked
? 'Task expired before execution.'
: 'Could not update',
})
}
expect(reload).toHaveBeenCalledTimes(2)
expect(error).toHaveBeenCalledWith(expect.any(String), 'Could not update')
expect(error).toHaveBeenCalledWith(
'Bulk operation revoked.',
'Task expired before execution.'
)
expect(success).not.toHaveBeenCalled()
})
it('should initialize the saved view service', () => {
expect(savedViewSpy).toHaveBeenCalled()
})
@@ -8,12 +8,14 @@ import {
import { NgClass } from '@angular/common'
import {
Component,
DestroyRef,
HostListener,
inject,
OnInit,
signal,
viewChild,
} from '@angular/core'
import { takeUntilDestroyed } from '@angular/core/rxjs-interop'
import { FormsModule } from '@angular/forms'
import { ActivatedRoute, Router, RouterModule } from '@angular/router'
import {
@@ -24,10 +26,11 @@ import {
NgbPopoverModule,
} from '@ng-bootstrap/ng-bootstrap'
import { TourNgBootstrap } from 'ngx-ui-tour-ng-bootstrap'
import { Observable } from 'rxjs'
import { filter, Observable } from 'rxjs'
import { first } from 'rxjs/operators'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { Document } from 'src/app/data/document'
import { PaperlessTaskStatus } from 'src/app/data/paperless-task'
import { SavedView } from 'src/app/data/saved-view'
import {
CollapsibleSection,
@@ -41,6 +44,7 @@ import {
DjangoMessageLevel,
DjangoMessagesService,
} from 'src/app/services/django-messages.service'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import { OpenDocumentsService } from 'src/app/services/open-documents.service'
import {
PermissionAction,
@@ -101,6 +105,8 @@ export class AppFrameComponent
extends ComponentWithPermissions
implements OnInit, ComponentCanDeactivate
{
private destroyRef = inject(DestroyRef)
private documentList = inject(DocumentListViewService)
router = inject(Router)
private activatedRoute = inject(ActivatedRoute)
private openDocumentsService = inject(OpenDocumentsService)
@@ -164,6 +170,46 @@ export class AppFrameComponent
}
ngOnInit(): void {
this.websocketStatusService
.onBulkOperationStatus()
.pipe(
filter(({ status }) =>
[
PaperlessTaskStatus.Success,
PaperlessTaskStatus.Failure,
PaperlessTaskStatus.Revoked,
].includes(status)
),
takeUntilDestroyed(this.destroyRef)
)
.subscribe(({ status, error }) => {
this.documentList.reload(null, false)
this.documentList.reduceSelectionToFilter()
// maybe someday we can optimize to only refresh affected documents
this.openDocumentsService.getOpenDocuments().forEach((doc) => {
this.openDocumentsService.refreshDocument(doc.id)
})
this.savedViewService.maybeRefreshDocumentCounts()
if (
this.permissionsService.currentUserCan(
PermissionAction.View,
PermissionType.PaperlessTask
)
) {
this.tasksService.reload()
}
if (status === PaperlessTaskStatus.Success) {
this.toastService.showSuccess($localize`Bulk operation completed.`)
} else if (status === PaperlessTaskStatus.Revoked) {
this.toastService.showError($localize`Bulk operation revoked.`, error)
} else {
this.toastService.showError(
$localize`Bulk operation failed. Some changes may already have been applied.`,
error
)
}
})
this.lastScrollY = window.scrollY
this.detectClassicScrollbars()
@@ -42,11 +42,16 @@
<span class="text-break">{{ message.content }}</span>
}
@if (message.role === 'assistant' && !message.isStreaming && message.references?.length) {
<div class="chat-references list-group mt-3">
<div class="list-group-item text-muted small py-1" i18n>Sources</div>
@for (reference of message.references; track reference.id) {
<a class="list-group-item list-group-item-action d-flex align-items-center text-primary" [routerLink]="['/documents', reference.id]" [title]="reference.title">
<pngx-bs-icon width="0.9em" height="0.9em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
<div class="chat-references mt-2 pt-2 border-top">
<div class="d-flex align-items-center text-muted mb-1">
<span i18n>Sources</span>
@if (hiddenReferenceCount(message) > 0 && !expandedReferences().has(message)) {
<button type="button" class="badge rounded-pill bg-body-secondary text-body-secondary fw-normal border-0 ms-2" (click)="expandReferences(message)" i18n>+{{ hiddenReferenceCount(message) }} more</button>
}
</div>
@for (reference of visibleReferences(message); track reference.id) {
<a class="reference-link d-flex align-items-center lh-sm text-primary text-decoration-none py-1" [routerLink]="['/documents', reference.id]" [title]="reference.title">
<pngx-bs-icon width="0.8em" height="0.8em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
</a>
}
</div>
@@ -12,6 +12,10 @@
display: none;
}
.message .flex-grow-1 {
min-width: 0;
}
.dropdown-item {
white-space: initial;
}
@@ -14,6 +14,7 @@ import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon
import { MarkdownPipe } from 'src/app/pipes/markdown.pipe'
import {
ChatMessage,
ChatReference,
ChatService,
parseChatResponse,
} from 'src/app/services/chat.service'
@@ -40,8 +41,29 @@ export class ChatComponent implements OnInit {
readonly input = signal('')
readonly documentId = signal<number>(undefined)
readonly slowResponse = signal(false)
readonly expandedReferences = signal<Set<ChatMessage>>(new Set())
public popperOptions = pngxPopperOptions
static readonly COLLAPSED_REFERENCE_COUNT = 2
public visibleReferences(message: ChatMessage): ChatReference[] {
if (this.expandedReferences().has(message)) {
return message.references
}
return message.references.slice(0, ChatComponent.COLLAPSED_REFERENCE_COUNT)
}
public hiddenReferenceCount(message: ChatMessage): number {
return Math.max(
0,
message.references.length - ChatComponent.COLLAPSED_REFERENCE_COUNT
)
}
public expandReferences(message: ChatMessage): void {
this.expandedReferences.update((expanded) => new Set(expanded).add(message))
}
private chatService: ChatService = inject(ChatService)
private router: Router = inject(Router)
@@ -559,18 +559,19 @@ describe('BulkEditorComponent', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
documents: [3, 4],
method: 'modify_tags',
parameters: { add_tags: [101], remove_tags: [] },
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should execute modify tags bulk operation for all filtered documents', () => {
@@ -607,17 +608,18 @@ describe('BulkEditorComponent', () => {
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
all: true,
filters: { title_search: 'apple' },
excluded_documents: [4],
method: 'modify_tags',
parameters: { add_tags: [101], remove_tags: [] },
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
)
})
it('should execute modify tags bulk operation with confirmation dialog if enabled', () => {
@@ -644,12 +646,12 @@ describe('BulkEditorComponent', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should set modal dialog text accordingly for tag edit confirmation', () => {
@@ -729,18 +731,19 @@ describe('BulkEditorComponent', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
documents: [3, 4],
method: 'set_correspondent',
parameters: { correspondent: 101 },
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should execute modify correspondent bulk operation with confirmation dialog if enabled', () => {
@@ -767,12 +770,12 @@ describe('BulkEditorComponent', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should set modal dialog text accordingly for correspondent edit confirmation', () => {
@@ -827,18 +830,19 @@ describe('BulkEditorComponent', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
documents: [3, 4],
method: 'set_document_type',
parameters: { document_type: 101 },
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should execute modify document type bulk operation with confirmation dialog if enabled', () => {
@@ -865,12 +869,12 @@ describe('BulkEditorComponent', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should set modal dialog text accordingly for document type edit confirmation', () => {
@@ -925,18 +929,19 @@ describe('BulkEditorComponent', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
documents: [3, 4],
method: 'set_storage_path',
parameters: { storage_path: 101 },
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should execute modify storage path bulk operation with confirmation dialog if enabled', () => {
@@ -963,12 +968,12 @@ describe('BulkEditorComponent', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should set modal dialog text accordingly for storage path edit confirmation', () => {
@@ -1023,18 +1028,19 @@ describe('BulkEditorComponent', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
documents: [3, 4],
method: 'modify_custom_fields',
parameters: { add_custom_fields: [101], remove_custom_fields: [102] },
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should execute modify custom fields bulk operation with confirmation dialog if enabled', () => {
@@ -1061,12 +1067,12 @@ describe('BulkEditorComponent', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
// coverage for modal messages
component.setCustomFields({
@@ -1540,6 +1546,47 @@ describe('BulkEditorComponent', () => {
)
})
it.each([true, false])(
'shows queued feedback and gates the task link on permission (%s)',
(canViewTasks) => {
jest
.spyOn(permissionsService, 'currentUserCan')
.mockReturnValue(canViewTasks)
const queued = jest.spyOn(toastService, 'show')
let modal: NgbModalRef
modalService.activeInstances.subscribe((instances) => {
if (instances.length) modal = instances[0]
})
const reload = jest.spyOn(documentListViewService, 'reload')
component.setPermissions()
const close = jest.spyOn(modal, 'close')
modal.componentInstance.confirmClicked.emit({
permissions: {},
merge: true,
})
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush({ result: 'OK', task_id: 'queued-operation' })
expect(close).toHaveBeenCalledTimes(1)
const notification = queued.mock.calls.at(-1)[0]
expect(notification.content).toBe('Bulk operation queued.')
if (canViewTasks) {
expect(notification.actionName).toBe('View task')
notification.action()
expect(router.navigate).toHaveBeenCalledWith(['/tasks'], {
queryParams: { task_id: 'queued-operation' },
})
} else {
expect(notification.action).toBeUndefined()
expect(notification.actionName).toBeUndefined()
}
expect(reload).not.toHaveBeenCalled()
expect(documentListViewService.selected.size).toBe(0)
expect(documentListViewService.allSelected).toBe(false)
}
)
it('should support bulk permissions update', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
@@ -1572,8 +1619,9 @@ describe('BulkEditorComponent', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
req.flush(true)
req.flush({ result: 'OK', task_id: 'queued-task' })
expect(req.request.body).toEqual({
from_webui: true,
documents: [3, 4],
method: 'set_permissions',
parameters: {
@@ -1581,12 +1629,12 @@ describe('BulkEditorComponent', () => {
merge: true,
},
})
httpTestingController.match(
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should not attempt to retrieve objects if user does not have permissions', () => {
@@ -1887,15 +1935,18 @@ describe('BulkEditorComponent', () => {
expect(toastServiceShowErrorSpy).toHaveBeenCalled()
expect(listReloadSpy).not.toHaveBeenCalled()
modal.componentInstance.succeeded.emit()
expect(toastServiceShowInfoSpy).toHaveBeenCalled()
expect(listReloadSpy).toHaveBeenCalled()
httpTestingController.match(
modal.componentInstance.succeeded.emit({
result: 'OK',
task_id: 'custom-fields-task',
})
expect(toastServiceShowInfoSpy).not.toHaveBeenCalled()
expect(listReloadSpy).not.toHaveBeenCalled()
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
)
httpTestingController.expectNone(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
)
})
it('should create share link bundle and enable manage callback', () => {
@@ -104,11 +104,11 @@ export class BulkEditorComponent
private correspondentService = inject(CorrespondentService)
list = inject(DocumentListViewService)
private documentService = inject(DocumentService)
private trashService = inject(TrashService)
private modalService = inject(NgbModal)
private openDocumentService = inject(OpenDocumentsService)
private settings = inject(SettingsService)
private toastService = inject(ToastService)
private trashService = inject(TrashService)
private storagePathService = inject(StoragePathService)
private customFieldService = inject(CustomFieldsService)
public readonly permissionService = inject(PermissionsService)
@@ -299,11 +299,33 @@ export class BulkEditorComponent
.bulkEdit(overrideSelection ?? this.getSelectionQuery(), method, args)
.pipe(first())
.subscribe({
next: () => this.handleOperationSuccess(modal),
next: ({ task_id }) => this.handleOperationQueued(modal, task_id),
error: (error) => this.handleOperationError(modal, error),
})
}
private handleOperationQueued(modal: NgbModalRef, taskId: string) {
modal?.close()
this.list.selectNone()
const canViewTasks = this.permissionService.currentUserCan(
PermissionAction.View,
PermissionType.PaperlessTask
)
this.toastService.show({
content: $localize`Bulk operation queued.`,
delay: 5000,
...(canViewTasks
? {
actionName: $localize`View task`,
action: () =>
this.router.navigate(['/tasks'], {
queryParams: { task_id: taskId },
}),
}
: {}),
})
}
private executeDocumentAction(
modal: NgbModalRef,
request: Observable<any>,
@@ -1088,14 +1110,9 @@ export class BulkEditorComponent
dialog.selection = this.getSelectionQuery()
dialog.selectionCount = this.getSelectionSize()
dialog.succeeded.subscribe((result) => {
this.toastService.showSuccess($localize`Custom fields updated.`)
this.list.reload()
this.list.reduceSelectionToFilter()
this.list.selected.forEach((id) => {
this.openDocumentService.refreshDocument(id)
})
})
dialog.succeeded.subscribe(({ task_id }) =>
this.handleOperationQueued(null, task_id)
)
dialog.failed.subscribe((error) => {
this.toastService.showError(
$localize`Error updating custom fields.`,
@@ -59,7 +59,7 @@ describe('CustomFieldsBulkEditDialogComponent', () => {
it('should emit succeeded event and close modal on successful save', () => {
const editSpy = jest
.spyOn(documentService, 'bulkEdit')
.mockReturnValue(of('Success'))
.mockReturnValue(of({ result: 'OK', task_id: 'custom-fields-task' }))
const successSpy = jest.spyOn(component.succeeded, 'emit')
component.selection = [1, 2]
@@ -16,6 +16,7 @@ import { NumberComponent } from 'src/app/components/common/input/number/number.c
import { SelectComponent } from 'src/app/components/common/input/select/select.component'
import { TextComponent } from 'src/app/components/common/input/text/text.component'
import { UrlComponent } from 'src/app/components/common/input/url/url.component'
import { BulkOperationResponse } from 'src/app/data/bulk-operation'
import { CustomField, CustomFieldDataType } from 'src/app/data/custom-field'
import {
DocumentSelectionQuery,
@@ -49,7 +50,7 @@ export class CustomFieldsBulkEditDialogComponent {
CustomFieldDataType = CustomFieldDataType
@Output()
succeeded = new EventEmitter()
succeeded = new EventEmitter<BulkOperationResponse>()
@Output()
failed = new EventEmitter()
@@ -103,6 +104,8 @@ export class CustomFieldsBulkEditDialogComponent {
}
public save() {
if (this.networkActive) return
this.networkActive = true
this.documentService
.bulkEdit(this.selection, 'modify_custom_fields', {
add_custom_fields: this.form.value,
@@ -110,11 +113,12 @@ export class CustomFieldsBulkEditDialogComponent {
})
.pipe(first())
.subscribe({
next: () => {
next: (response) => {
this.activeModal.close()
this.succeeded.emit()
this.succeeded.emit(response)
},
error: (error) => {
this.networkActive = false
this.failed.emit(error)
},
})
+13
View File
@@ -0,0 +1,13 @@
import { PaperlessTaskStatus } from './paperless-task'
export interface BulkOperationResponse {
result: string
task_id: string
}
export interface BulkOperationStatus {
task_id: string
operation: string
status: PaperlessTaskStatus
error?: string
}
@@ -124,6 +124,42 @@ describe('DocumentListViewService', () => {
})
})
it.each([false, true])(
'preserves the current route on a background refresh (saved view: %s)',
(savedView) => {
if (savedView) documentListViewService.activateSavedView(view)
const navigate = jest.mocked(router.navigate)
navigate.mockClear()
documentListViewService.reload(null, false)
httpTestingController
.expectOne((req) => req.url === `${environment.apiBaseUrl}documents/`)
.flush(full_results)
expect(documentListViewService.documents).toEqual(full_results.results)
expect(navigate).not.toHaveBeenCalled()
}
)
it('preserves the route when a background refresh falls back to the first page', () => {
jest
.spyOn(documentListViewService, 'saveDocumentListView')
.mockImplementation(() => {})
documentListViewService.currentPage = 2
httpTestingController
.expectOne((req) => req.url === `${environment.apiBaseUrl}documents/`)
.flush(full_results)
const navigate = jest.mocked(router.navigate)
navigate.mockClear()
documentListViewService.reload(null, false)
httpTestingController
.expectOne((req) => req.url === `${environment.apiBaseUrl}documents/`)
.flush('Not found', { status: 404, statusText: 'Not found' })
httpTestingController
.expectOne((req) => req.url === `${environment.apiBaseUrl}documents/`)
.flush(full_results)
expect(documentListViewService.currentPage).toBe(1)
expect(navigate).not.toHaveBeenCalled()
})
it('should reload the list', () => {
expect(documentListViewService.currentPage).toEqual(1)
documentListViewService.reload()
@@ -380,7 +380,7 @@ export class DocumentListViewService {
queryParams: paramsFromViewState(activeListViewState),
replaceUrl: !this.router.routerState.snapshot.url.includes('?'), // in case navigating from params-less /documents
})
} else if (this._activeSavedViewId) {
} else if (updateQueryParams && this._activeSavedViewId) {
this.router.navigate([], {
queryParams: paramsFromViewState(activeListViewState, true),
queryParamsHandling: 'merge',
@@ -399,7 +399,7 @@ export class DocumentListViewService {
// this happens when applying a filter: the current page might not be available anymore due to the reduced result set.
activeListViewState.currentPage = 1
this.markChanged()
this.reload()
this.reload(onFinish, updateQueryParams)
} else if (
activeListViewState.sortField.indexOf('custom_field') === 0 &&
!this.settings
@@ -248,6 +248,7 @@ describe(`DocumentService`, () => {
)
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({
from_webui: true,
documents: ids,
method,
parameters,
@@ -271,6 +272,7 @@ describe(`DocumentService`, () => {
)
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({
from_webui: true,
all: true,
filters: { title__icontains: 'apple' },
excluded_documents: [2, 3],
@@ -14,6 +14,7 @@ import { DocumentSuggestions } from 'src/app/data/document-suggestions'
import { FilterRule } from 'src/app/data/filter-rule'
import { Results, SelectionData } from 'src/app/data/results'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { BulkOperationResponse } from '../../data/bulk-operation'
import { queryParamsFromFilterRules } from '../../utils/query-params'
import {
PermissionAction,
@@ -337,11 +338,15 @@ export class DocumentService extends AbstractPaperlessService<Document> {
method: DocumentBulkEditMethod,
args: any
) {
return this.http.post(this.getResourceUrl(null, 'bulk_edit'), {
...selection,
method: method,
parameters: args,
})
return this.http.post<BulkOperationResponse>(
this.getResourceUrl(null, 'bulk_edit'),
{
...selection,
method: method,
from_webui: true,
parameters: args,
}
)
}
deleteDocuments(selection: DocumentSelectionQuery) {
@@ -438,6 +438,20 @@ describe('ConsumerStatusService', () => {
expect(updated).toBeTruthy()
})
it('forwards bulk operation completion events', () => {
const received = jest.fn()
websocketStatusService.onBulkOperationStatus().subscribe(received)
const data = {
task_id: 'bulk-task',
operation: 'set_permissions',
status: 'success',
}
websocketStatusService.connect()
server.send({ type: WebsocketStatusType.BULK_OPERATION_STATUS, data })
websocketStatusService.disconnect()
expect(received).toHaveBeenCalledWith(data)
})
it('should ignore keep-alive heartbeat messages from the server', () => {
let updated = false
let deleted = false
@@ -1,6 +1,7 @@
import { Injectable, inject, signal } from '@angular/core'
import { Subject } from 'rxjs'
import { environment } from 'src/environments/environment'
import { BulkOperationStatus } from '../data/bulk-operation'
import { User } from '../data/user'
import { WebsocketDocumentUpdatedMessage } from '../data/websocket-document-updated-message'
import { WebsocketDocumentsDeletedMessage } from '../data/websocket-documents-deleted-message'
@@ -12,6 +13,7 @@ export enum WebsocketStatusType {
DOCUMENTS_DELETED = 'documents_deleted',
DOCUMENT_UPDATED = 'document_updated',
HEARTBEAT = 'heartbeat',
BULK_OPERATION_STATUS = 'bulk_operation_status',
}
// see ProgressStatusOptions in src/documents/plugins/helpers.py
@@ -116,6 +118,13 @@ export class WebsocketStatusService {
private readonly documentDeletedSubject = new Subject<boolean>()
private readonly documentUpdatedSubject =
new Subject<WebsocketDocumentUpdatedMessage>()
private readonly bulkOperationStatusSubject =
new Subject<BulkOperationStatus>()
onBulkOperationStatus() {
return this.bulkOperationStatusSubject.asObservable()
}
private readonly connectionStatusSubject = new Subject<boolean>()
private get(taskId: string, filename?: string) {
@@ -192,6 +201,7 @@ export class WebsocketStatusService {
| WebsocketProgressMessage
| WebsocketDocumentsDeletedMessage
| WebsocketDocumentUpdatedMessage
| BulkOperationStatus
} = JSON.parse(ev.data)
switch (type) {
@@ -209,6 +219,12 @@ export class WebsocketStatusService {
this.handleProgressUpdate(messageData as WebsocketProgressMessage)
break
case WebsocketStatusType.BULK_OPERATION_STATUS:
this.bulkOperationStatusSubject.next(
messageData as BulkOperationStatus
)
break
case WebsocketStatusType.HEARTBEAT:
// keep-alive from the server, see paperless.consumers.StatusConsumer
break
+53 -36
View File
@@ -48,6 +48,7 @@ from documents.versioning import get_root_document
from documents.versioning import versions_newest_first
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Mapping
from django.contrib.auth.models import User
@@ -131,9 +132,24 @@ def _scratch_path(name: str) -> Path:
return Path(tempfile.mkdtemp(dir=settings.SCRATCH_DIR)) / name
def _finish_metadata_edit(
document_ids: list[int],
update_callback: Callable[[list[int]], None] | None,
) -> None:
if update_callback is not None:
update_callback(document_ids)
else:
bulk_update_documents.apply_async(
kwargs={"document_ids": document_ids},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
def set_correspondent(
doc_ids: list[int],
correspondent: Correspondent,
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
if correspondent:
correspondent = Correspondent.objects.only("pk").get(id=correspondent)
@@ -146,15 +162,17 @@ def set_correspondent(
affected_docs = list(qs.values_list("pk", flat=True))
qs.update(correspondent=correspondent)
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
def set_storage_path(doc_ids: list[int], storage_path: StoragePath) -> Literal["OK"]:
def set_storage_path(
doc_ids: list[int],
storage_path: StoragePath,
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
if storage_path:
storage_path = StoragePath.objects.only("pk").get(id=storage_path)
@@ -168,15 +186,17 @@ def set_storage_path(doc_ids: list[int], storage_path: StoragePath) -> Literal["
affected_docs = list(qs.values_list("pk", flat=True))
qs.update(storage_path=storage_path)
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
def set_document_type(doc_ids: list[int], document_type: DocumentType) -> Literal["OK"]:
def set_document_type(
doc_ids: list[int],
document_type: DocumentType,
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
if document_type:
document_type = DocumentType.objects.only("pk").get(id=document_type)
@@ -188,15 +208,17 @@ def set_document_type(doc_ids: list[int], document_type: DocumentType) -> Litera
affected_docs = list(qs.values_list("pk", flat=True))
qs.update(document_type=document_type)
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
def add_tag(doc_ids: list[int], tag: int) -> Literal["OK"]:
def add_tag(
doc_ids: list[int],
tag: int,
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
tag_obj = Tag.objects.get(pk=tag)
tags_to_add = [tag_obj, *tag_obj.get_ancestors()]
@@ -217,15 +239,17 @@ def add_tag(doc_ids: list[int], tag: int) -> Literal["OK"]:
DocumentTagRelationship.objects.bulk_create(to_create)
if affected_docs:
bulk_update_documents.apply_async(
kwargs={"document_ids": list(affected_docs)},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(list(affected_docs), update_callback)
return "OK"
def remove_tag(doc_ids: list[int], tag: int) -> Literal["OK"]:
def remove_tag(
doc_ids: list[int],
tag: int,
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
tag_obj = Tag.objects.get(pk=tag)
tag_ids = [tag_obj.id, *tag_obj.get_descendants_pks()]
@@ -238,10 +262,7 @@ def remove_tag(doc_ids: list[int], tag: int) -> Literal["OK"]:
qs.delete()
if affected_docs:
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
@@ -250,6 +271,8 @@ def modify_tags(
doc_ids: list[int],
add_tags: list[int],
remove_tags: list[int],
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
qs = Document.objects.filter(id__in=doc_ids).only("pk")
affected_docs = list(qs.values_list("pk", flat=True))
@@ -293,10 +316,7 @@ def modify_tags(
)
if affected_docs:
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
@@ -305,6 +325,8 @@ def modify_custom_fields(
doc_ids: list[int],
add_custom_fields: list[int] | dict,
remove_custom_fields: list[int],
*,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
qs = Document.objects.filter(id__in=doc_ids).only("pk")
affected_docs = list(qs.values_list("pk", flat=True))
@@ -367,10 +389,7 @@ def modify_custom_fields(
field_id__in=remove_custom_fields,
).hard_delete()
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
@@ -451,6 +470,7 @@ def set_permissions(
*,
owner: User | None = None,
merge: bool = False,
update_callback: Callable[[list[int]], None] | None = None,
) -> Literal["OK"]:
qs = Document.objects.filter(id__in=doc_ids).select_related("owner")
@@ -468,10 +488,7 @@ def set_permissions(
merge=merge,
)
bulk_update_documents.apply_async(
kwargs={"document_ids": affected_docs},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
_finish_metadata_edit(affected_docs, update_callback)
return "OK"
+107
View File
@@ -0,0 +1,107 @@
"""Queued document metadata edits."""
from typing import Any
from auditlog.context import set_actor
from celery import shared_task
from django.conf import settings
from django.contrib.auth.models import User
from django.db import transaction
from rest_framework.exceptions import PermissionDenied
from documents import bulk_edit
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.permissions import has_document_operation_permissions
from documents.tasks import bulk_update_documents
MODIFIED_FIELDS = {
"set_correspondent": "correspondent",
"set_document_type": "document_type",
"set_storage_path": "storage_path",
"add_tag": "tags",
"remove_tag": "tags",
"modify_tags": "tags",
"modify_custom_fields": "custom_fields",
"set_permissions": None,
}
QUEUED_OPERATIONS = frozenset(MODIFIED_FIELDS)
def snapshot_field(doc_ids: list[int], field: str) -> dict[int, Any]:
"""
Returns each document's current value of field, for the audit log.
Tags and custom fields are one row per value, so they are gathered
into a sorted list of pks per document (empty when there are none).
Reading them through Document.values() instead would join those rows
and return one arbitrary value per document.
"""
if field == "tags":
rows = (
Document.tags.through.objects.filter(document_id__in=doc_ids)
.order_by("tag_id")
.values_list("document_id", "tag_id")
)
elif field == "custom_fields":
rows = (
CustomFieldInstance.objects.filter(document_id__in=doc_ids)
.order_by("pk")
.values_list("document_id", "pk")
)
else:
return dict(
Document.objects.filter(pk__in=doc_ids).values_list("pk", field),
)
values: dict[int, list[int]] = {doc_id: [] for doc_id in doc_ids}
for doc_id, pk in rows:
values[doc_id].append(pk)
return values
@shared_task
def execute_bulk_operation(
operation: str,
document_ids: list[int],
parameters: dict,
user_id: int,
) -> dict:
if operation not in QUEUED_OPERATIONS:
raise ValueError(f"Unsupported bulk operation: {operation}")
method = getattr(bulk_edit, operation)
user = User.objects.get(pk=user_id)
if not has_document_operation_permissions(
user=user,
documents=document_ids,
method=method,
parameters=parameters,
):
raise PermissionDenied("Insufficient permissions to execute bulk operation")
affected_ids: list[int] = []
modified_field = MODIFIED_FIELDS[operation]
# Keep metadata writes and their audit entries together. Follow-up processing
# runs after commit, within this same tracked task.
with set_actor(user), transaction.atomic():
old_values = (
snapshot_field(document_ids, modified_field)
if settings.AUDIT_LOG_ENABLED and modified_field
else {}
)
method(document_ids, **parameters, update_callback=affected_ids.extend)
if settings.AUDIT_LOG_ENABLED and modified_field:
from auditlog.models import LogEntry
new_values = snapshot_field(document_ids, modified_field)
for doc in Document.objects.filter(pk__in=document_ids):
LogEntry.objects.log_create(
instance=doc,
changes={modified_field: [old_values[doc.pk], new_values[doc.pk]]},
action=LogEntry.Action.UPDATE,
actor=user,
additional_data={"reason": f"Bulk edit: {operation}"},
)
if affected_ids:
bulk_update_documents(list(dict.fromkeys(affected_ids)))
return {"operation": operation, "document_count": len(document_ids)}
+2 -6
View File
@@ -631,9 +631,7 @@ class ConsumerPlugin(
pk=self.metadata.actor_id,
).first()
if actor is not None:
from auditlog.context import ( # type: ignore[import-untyped]
set_actor,
)
from auditlog.context import set_actor # type: ignore[import-untyped]
with set_actor(actor):
original_document.save()
@@ -651,9 +649,7 @@ class ConsumerPlugin(
# Create a log entry for the version addition, if enabled
if settings.AUDIT_LOG_ENABLED:
from auditlog.models import ( # type: ignore[import-untyped]
LogEntry,
)
from auditlog.models import LogEntry # type: ignore[import-untyped]
LogEntry.objects.log_create(
instance=root_doc,
+97
View File
@@ -739,3 +739,100 @@ class AcknowledgeTasksPermissions(GlobalPermissionsByMethod):
"OPTIONS": ["documents.change_paperlesstask"],
"POST": ["documents.change_paperlesstask"],
}
def has_document_operation_permissions(
*,
user: User,
documents: list[int],
method,
parameters: dict[str, Any],
) -> bool:
from documents import bulk_edit
if not user.is_active:
return False
if user.is_superuser:
return True
root_docs = {
get_root_document(doc)
for doc in Document.objects.select_related(
"owner",
"root_document__owner",
).filter(pk__in=documents)
}
user_is_owner_of_all_documents = all(
(doc.owner == user or doc.owner is None) for doc in root_docs
)
# check global and object permissions for all documents
has_perms = (
user.has_perm(
"documents.change_document",
)
and not Document.global_objects.filter(pk__in=documents)
.exclude(pk__in=permitted_document_ids(user, perm="change_document"))
.exists()
)
# check ownership for methods that change original document
if (
(
has_perms
and method
in [
bulk_edit.set_permissions,
bulk_edit.delete,
bulk_edit.rotate,
bulk_edit.delete_pages,
bulk_edit.edit_pdf,
bulk_edit.merge_as_versions,
bulk_edit.remove_password,
]
)
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (method == bulk_edit.edit_pdf and parameters.get("update_document"))
):
has_perms = has_perms and user_is_owner_of_all_documents
# check global add permissions for methods that create documents
if (
has_perms
and (
method in [bulk_edit.split, bulk_edit.merge]
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.add_document")
):
has_perms = False
# check global delete permissions for methods that delete documents
if (
has_perms
and (
method == bulk_edit.delete
# Sources stop being documents of their own, and removing one
# again afterwards needs delete_document
or method == bulk_edit.merge_as_versions
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and parameters.get("delete_original")
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.delete_document")
):
has_perms = False
return has_perms
+48 -6
View File
@@ -8,6 +8,7 @@ from pathlib import Path
from typing import TYPE_CHECKING
from typing import Any
from asgiref.sync import async_to_sync
from celery import shared_task
from celery.signals import before_task_publish
from celery.signals import task_failure
@@ -16,6 +17,7 @@ from celery.signals import task_prerun
from celery.signals import task_revoked
from celery.signals import worker_process_init
from celery.signals import worker_process_shutdown
from channels.layers import get_channel_layer
from django.conf import settings
from django.contrib.auth.models import Group
from django.contrib.auth.models import User
@@ -1051,6 +1053,8 @@ def run_workflows(
# Task tracking -- Celery signal handlers
# ---------------------------------------------------------------------------
BULK_OPERATION_TASK_NAME = "documents.bulk_operations.execute_bulk_operation"
TRACKED_TASKS: dict[str, PaperlessTask.TaskType] = {
"documents.tasks.consume_file": PaperlessTask.TaskType.CONSUME_FILE,
"documents.tasks.train_classifier": PaperlessTask.TaskType.TRAIN_CLASSIFIER,
@@ -1064,6 +1068,7 @@ TRACKED_TASKS: dict[str, PaperlessTask.TaskType] = {
"documents.tasks.build_share_link_bundle": PaperlessTask.TaskType.BUILD_SHARE_LINK,
"documents.bulk_edit.delete": PaperlessTask.TaskType.BULK_DELETE,
"documents.tasks.apply_ai_suggestions": PaperlessTask.TaskType.APPLY_AI_SUGGESTIONS,
BULK_OPERATION_TASK_NAME: PaperlessTask.TaskType.BULK_UPDATE,
}
_CELERY_STATE_TO_STATUS: dict[str, PaperlessTask.Status] = {
@@ -1081,8 +1086,7 @@ def _extract_input_data(
For consume_file tasks this includes the filename, MIME type, and any
non-null overrides from the DocumentMetadataOverrides object. For
mail_fetch tasks it captures the account_ids list. All other task
types store no input data and return {}.
mail_fetch tasks it captures the account_ids list.
"""
if task_type == PaperlessTask.TaskType.CONSUME_FILE:
input_doc = task_kwargs.get("input_doc")
@@ -1147,7 +1151,7 @@ def _extract_owner_id(
task_type: PaperlessTask.TaskType,
task_kwargs: dict,
) -> int | None:
"""Return the owner_id from consume_file overrides, or None for all other task types."""
"""Return the document owner for consumption tasks."""
if task_type != PaperlessTask.TaskType.CONSUME_FILE:
return None
overrides = task_kwargs.get("overrides")
@@ -1186,9 +1190,16 @@ def before_task_publish_handler(
_, task_kwargs, _ = body
task_id = headers["id"]
input_data = _extract_input_data(task_type, task_kwargs)
if task_name == BULK_OPERATION_TASK_NAME:
input_data = {
"operation": task_kwargs["operation"],
"document_count": len(task_kwargs["document_ids"]),
}
owner_id = task_kwargs["user_id"]
else:
input_data = _extract_input_data(task_type, task_kwargs)
owner_id = _extract_owner_id(task_type, task_kwargs)
trigger_source = _determine_trigger_source(headers)
owner_id = _extract_owner_id(task_type, task_kwargs)
# A retried task is republished with the same task_id, so this fires
# again for it; get_or_create keeps the original PENDING record
@@ -1228,6 +1239,30 @@ def task_prerun_handler(sender=None, task_id=None, task=None, **kwargs) -> None:
logger.exception("Setting PaperlessTask started failed")
def _send_bulk_operation_status(task: PaperlessTask | None) -> None:
"""Send the recorded outcome to the initiator."""
try:
if task is None or task.owner_id is None:
return
result_data = task.result_data or {}
async_to_sync(get_channel_layer().group_send)(
f"bulk_operations_{task.owner_id}",
{
"type": "bulk_operation_status",
"data": {
"task_id": task.task_id,
"status": task.status,
"operation": task.input_data["operation"],
"error": result_data.get("error_message")
or result_data.get("reason", ""),
},
},
)
except Exception:
# Just a notification failure
logger.exception("Sending bulk operation status failed")
@task_postrun.connect
def task_postrun_handler(
sender=None,
@@ -1285,6 +1320,8 @@ def task_postrun_handler(
changed_fields.append("status")
task_instance.save(update_fields=changed_fields)
if task and task.name == BULK_OPERATION_TASK_NAME:
_send_bulk_operation_status(task_instance)
except Exception: # pragma: no cover
logger.exception("Updating PaperlessTask failed")
@@ -1350,6 +1387,8 @@ def task_failure_handler(
date_started - date_created
).total_seconds()
task_qs.update(**update_fields)
if sender and sender.name == BULK_OPERATION_TASK_NAME:
_send_bulk_operation_status(task_qs.first())
except Exception: # pragma: no cover
logger.exception("Updating PaperlessTask on failure failed")
@@ -1386,13 +1425,16 @@ def task_revoked_handler(
reason = "Task expired before execution."
elif terminated:
reason = "Task terminated during execution."
PaperlessTask.objects.filter(task_id=task_id).update(
task_qs = PaperlessTask.objects.filter(task_id=task_id)
task_qs.update(
status=PaperlessTask.Status.REVOKED,
date_done=timezone.now(),
result_data={
"reason": (reason + " Check worker logs for details."),
},
)
if sender and sender.name == BULK_OPERATION_TASK_NAME:
_send_bulk_operation_status(task_qs.first())
except Exception: # pragma: no cover
logger.exception("Updating PaperlessTask on revocation failed")
+15
View File
@@ -12,6 +12,7 @@ from celery import Task
from celery import shared_task
from django.conf import settings
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from django.db import models
from django.db import transaction
from django.db.models.signals import post_save
@@ -925,3 +926,17 @@ def cleanup_expired_share_link_bundles() -> None:
)
if count:
logger.info("Deleted %s expired share link bundle(s)", count)
@shared_task
def clear_expired_sessions() -> None:
"""
Remove expired database sessions in batches to limit the size of each delete.
"""
expired = Session.objects.filter(expire_date__lt=timezone.now())
while session_keys := list(
expired.order_by("expire_date").values_list("pk", flat=True)[:500],
):
expired.filter(pk__in=session_keys).delete()
logger.info("Cleared expired sessions")
@@ -287,6 +287,28 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
self.assertEqual(config.user_args, user_args)
self.assertEqual(config.barcode_tag_mapping, barcode_tag_mapping)
def test_api_update_config_rejects_reserved_user_args(self) -> None:
"""
GIVEN:
- API request to update app config with user_args which set an
argument the parser manages itself
WHEN:
- API is called
THEN:
- Request is rejected with a 400 naming the argument
- Config is not updated
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps({"user_args": json.dumps({"sidecar": "/tmp/x.py"})}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("sidecar", str(response.data["user_args"]))
config = ApplicationConfiguration.objects.first()
assert config is not None
self.assertIsNone(config.user_args)
def test_api_update_config_rejects_invalid_json_objects(self) -> None:
"""
GIVEN:
+85 -11
View File
@@ -7,6 +7,7 @@ from django.test import override_settings
from rest_framework import status
from rest_framework.test import APITestCase
from documents.bulk_operations import execute_bulk_operation
from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
@@ -29,9 +30,18 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.user = user
self.client.force_authenticate(user=user)
patcher = mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
patcher = mock.patch("documents.bulk_operations.bulk_update_documents")
self.async_task = patcher.start()
self.addCleanup(patcher.stop)
# Exercise accepted requests and worker execution together in these
# operation/validation tests. Queue-only behavior is tested separately.
enqueue = mock.patch(
"documents.views.execute_bulk_operation.apply_async",
side_effect=lambda *, kwargs, headers: self.run_operation(kwargs),
)
enqueue.start()
self.addCleanup(enqueue.stop)
self.c1 = Correspondent.objects.create(name="c1")
self.c2 = Correspondent.objects.create(name="c2")
self.dt1 = DocumentType.objects.create(name="dt1")
@@ -61,11 +71,16 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.cf1 = CustomField.objects.create(name="cf1", data_type="string")
self.cf2 = CustomField.objects.create(name="cf2", data_type="string")
@staticmethod
def run_operation(kwargs):
execute_bulk_operation(**kwargs)
return mock.Mock(id="test-bulk-operation")
def setup_mock(self, m, method_name, return_value="OK") -> None:
m.return_value = return_value
m.__name__ = method_name
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.bulk_operations.bulk_update_documents")
def test_api_set_correspondent(self, bulk_update_task_mock) -> None:
self.assertNotEqual(self.doc1.correspondent, self.c1)
response = self.client.post(
@@ -84,11 +99,11 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(self.doc1.correspondent, self.c1)
bulk_update_task_mock.assert_called_once()
self.assertCountEqual(
bulk_update_task_mock.call_args.kwargs["kwargs"]["document_ids"],
bulk_update_task_mock.call_args.args[0],
[self.doc1.pk],
)
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.bulk_operations.bulk_update_documents")
def test_api_unset_correspondent(self, bulk_update_task_mock) -> None:
self.doc1.correspondent = self.c1
self.doc1.save()
@@ -110,7 +125,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.doc1.refresh_from_db()
self.assertIsNone(self.doc1.correspondent)
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.bulk_operations.bulk_update_documents")
def test_api_set_type(self, bulk_update_task_mock) -> None:
self.assertNotEqual(self.doc1.document_type, self.dt1)
response = self.client.post(
@@ -129,11 +144,11 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(self.doc1.document_type, self.dt1)
bulk_update_task_mock.assert_called_once()
self.assertCountEqual(
bulk_update_task_mock.call_args.kwargs["kwargs"]["document_ids"],
bulk_update_task_mock.call_args.args[0],
[self.doc1.pk],
)
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.bulk_operations.bulk_update_documents")
def test_api_unset_type(self, bulk_update_task_mock) -> None:
self.doc1.document_type = self.dt1
self.doc1.save()
@@ -154,11 +169,11 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertIsNone(self.doc1.document_type)
bulk_update_task_mock.assert_called_once()
self.assertCountEqual(
bulk_update_task_mock.call_args.kwargs["kwargs"]["document_ids"],
bulk_update_task_mock.call_args.args[0],
[self.doc1.pk],
)
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.bulk_operations.bulk_update_documents")
def test_api_add_tag(self, bulk_update_task_mock) -> None:
self.assertFalse(self.doc1.tags.filter(pk=self.t1.pk).exists())
@@ -180,11 +195,11 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
bulk_update_task_mock.assert_called_once()
self.assertCountEqual(
bulk_update_task_mock.call_args.kwargs["kwargs"]["document_ids"],
bulk_update_task_mock.call_args.args[0],
[self.doc1.pk],
)
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.bulk_operations.bulk_update_documents")
def test_api_remove_tag(self, bulk_update_task_mock) -> None:
self.doc1.tags.add(self.t1)
@@ -223,6 +238,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertListEqual(args[0], [self.doc1.id, self.doc3.id])
self.assertEqual(kwargs["add_tags"], [self.t1.id])
self.assertEqual(kwargs["remove_tags"], [self.t2.id])
@@ -277,6 +295,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertListEqual(args[0], [self.doc1.id, self.doc3.id])
self.assertEqual(kwargs["add_custom_fields"], [self.cf1.id])
self.assertEqual(kwargs["remove_custom_fields"], [self.cf2.id])
@@ -345,6 +366,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertListEqual(args[0], [self.doc1.id, self.doc3.id])
self.assertEqual(kwargs["add_custom_fields"], {self.cf1.id: "foo"})
self.assertEqual(kwargs["remove_custom_fields"], [self.cf2.id])
@@ -506,6 +530,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(len(kwargs), 0)
@@ -520,6 +547,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(len(kwargs), 0)
@@ -544,6 +574,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(len(kwargs), 0)
@@ -558,6 +591,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": False})
@@ -580,6 +616,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
@@ -609,6 +648,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertListEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs["storage_path"], self.sp1.id)
@@ -639,6 +681,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertListEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs["storage_path"], None)
@@ -761,6 +806,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertEqual(args[0], [self.doc2.id])
self.assertEqual(kwargs["storage_path"], self.sp1.id)
@@ -784,6 +832,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [self.doc1.id, self.doc3.id, self.doc5.id])
self.assertEqual(kwargs["storage_path"], self.sp1.id)
@@ -820,6 +871,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [first_duplicate.id, second_duplicate.id])
self.assertEqual(kwargs["storage_path"], self.sp1.id)
@@ -856,6 +910,11 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
get_backend.return_value.search_ids.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value
for key, value in kwargs.items()
if key != "update_callback"
}
self.assertEqual(args[0], [self.doc2.id])
self.assertEqual(kwargs["storage_path"], self.sp1.id)
@@ -1232,6 +1291,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [self.doc2.id, self.doc3.id])
self.assertEqual(len(kwargs["set_permissions"]["view"]["users"]), 2)
@@ -1644,6 +1706,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [self.doc2.id, self.doc3.id])
self.assertEqual(kwargs["degrees"], 90)
self.assertEqual(kwargs["source_mode"], "latest_version")
@@ -1930,6 +1995,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [self.doc2.id, self.doc3.id])
self.assertEqual(kwargs["metadata_document_id"], self.doc3.id)
self.assertEqual(kwargs["source_mode"], "latest_version")
@@ -2137,6 +2205,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [self.doc2.id])
self.assertEqual(kwargs["operations"], [{"page": 1}])
self.assertEqual(kwargs["source_mode"], "explicit_selection")
@@ -2482,6 +2553,9 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
kwargs = {
key: value for key, value in kwargs.items() if key != "update_callback"
}
self.assertCountEqual(args[0], [self.doc2.id])
self.assertEqual(kwargs["password"], "secret")
self.assertTrue(kwargs["update_document"])
@@ -149,10 +149,10 @@ class TestPostSelectionPathsEnforceTheCap:
assert str(_MAX_QUERY_LENGTH + 1) in message
parse_spy.assert_not_called()
@mock.patch("documents.bulk_edit.bulk_update_documents.apply_async")
@mock.patch("documents.views.execute_bulk_operation.apply_async")
def test_bulk_edit_query_at_exactly_the_cap_is_accepted(
self,
bulk_update_task_mock: mock.MagicMock,
enqueue_mock: mock.MagicMock,
admin_client: APIClient,
searchable_document: Document,
) -> None:
@@ -163,12 +163,12 @@ class TestPostSelectionPathsEnforceTheCap:
- Its `filters.query` is exactly `_MAX_QUERY_LENGTH` characters
long
THEN:
- The cap check accepts it and the request reaches the real
bulk-edit method (its Celery dispatch is mocked out here,
same as every other bulk-edit test, since nothing here is
testing that method itself)
- The cap check accepts it and queues the metadata operation.
Mock the executor dispatch so this exercises query validation
without requiring a broker.
"""
query = "a" * _MAX_QUERY_LENGTH
enqueue_mock.return_value.id = "query-cap-task"
response = admin_client.post(
"/api/documents/bulk_edit/",
@@ -183,6 +183,11 @@ class TestPostSelectionPathsEnforceTheCap:
)
assert response.status_code == status.HTTP_200_OK
assert response.data == {"result": "OK", "task_id": "query-cap-task"}
enqueue_mock.assert_called_once()
assert (
enqueue_mock.call_args.kwargs["kwargs"]["operation"] == "set_document_type"
)
def test_bulk_download_query_one_over_the_cap_is_a_400(
self,
+307
View File
@@ -0,0 +1,307 @@
"""Request acceptance, worker execution, and task tracking."""
import uuid
from unittest import mock
import pytest
from auditlog.models import LogEntry
from django.contrib.auth.models import User
from rest_framework.exceptions import PermissionDenied
from rest_framework.test import APIClient
from documents.bulk_operations import execute_bulk_operation
from documents.models import Correspondent
from documents.models import Document
from documents.models import PaperlessTask
from documents.signals.handlers import before_task_publish_handler
from documents.signals.handlers import task_failure_handler
from documents.signals.handlers import task_postrun_handler
from documents.signals.handlers import task_revoked_handler
from paperless_testing.factories import UserFactory
@pytest.fixture
def bulk_setup(paperless_dirs):
user = UserFactory(superuser=True)
client = APIClient()
client.force_authenticate(user)
doc = Document.objects.create(title="Example", checksum="bulk-op-example")
correspondent = Correspondent.objects.create(name="Example correspondent")
return user, client, doc, correspondent
@pytest.mark.django_db
@pytest.mark.parametrize(
"endpoint,operation,parameters",
[
("bulk_edit", "set_correspondent", {"correspondent": None}),
],
)
def test_request_only_queues(bulk_setup, endpoint, operation, parameters):
user, client, doc, _ = bulk_setup
task_id = str(uuid.uuid4())
data = {"documents": [doc.pk], "from_webui": True}
if endpoint == "bulk_edit":
data.update(method=operation, parameters=parameters)
with mock.patch("documents.views.execute_bulk_operation.apply_async") as enqueue:
enqueue.return_value.id = task_id
response = client.post(f"/api/documents/{endpoint}/", data, format="json")
assert response.status_code == 200
assert response.data == {"result": "OK", "task_id": task_id}
assert Document.objects.filter(pk=doc.pk).exists()
assert enqueue.call_args.kwargs["kwargs"] == {
"operation": operation,
"document_ids": [doc.pk],
"parameters": parameters,
"user_id": user.pk,
}
assert enqueue.call_args.kwargs["headers"]["trigger_source"] == "web_ui"
@pytest.mark.django_db
def test_metadata_task_includes_audit_and_followup(bulk_setup, settings):
user, _, doc, correspondent = bulk_setup
settings.AUDIT_LOG_ENABLED = True
with (
mock.patch("documents.bulk_operations.bulk_update_documents") as followup,
mock.patch(
"documents.bulk_edit.bulk_update_documents.apply_async",
) as extra_task,
):
def check_followup(ids):
doc.refresh_from_db()
assert doc.correspondent_id == correspondent.pk
assert LogEntry.objects.filter(object_pk=str(doc.pk), actor=user).exists()
assert ids == [doc.pk]
followup.side_effect = check_followup
result = execute_bulk_operation(
"set_correspondent",
[doc.pk],
{"correspondent": correspondent.pk},
user.pk,
)
followup.assert_called_once()
extra_task.assert_not_called()
assert result["document_count"] == 1
@pytest.mark.django_db
def test_permissions_rechecked_on_worker(bulk_setup):
user, _, doc, correspondent = bulk_setup
User.objects.filter(pk=user.pk).update(is_superuser=False)
with pytest.raises(PermissionDenied):
execute_bulk_operation(
"set_correspondent",
[doc.pk],
{"correspondent": correspondent.pk},
user.pk,
)
doc.refresh_from_db()
assert doc.correspondent_id is None
@pytest.mark.django_db
def test_queue_failure_does_not_change_documents(bulk_setup):
_, client, doc, correspondent = bulk_setup
with mock.patch(
"documents.views.execute_bulk_operation.apply_async",
side_effect=RuntimeError("broker unavailable"),
):
response = client.post(
"/api/documents/bulk_edit/",
{
"documents": [doc.pk],
"method": "set_correspondent",
"parameters": {"correspondent": correspondent.pk},
},
format="json",
)
assert response.status_code == 400
doc.refresh_from_db()
assert doc.correspondent_id is None
@pytest.mark.django_db
@pytest.mark.parametrize("failed", [False, True])
def test_tracked_outcome_records_initiator_and_result(bulk_setup, failed):
user, _, doc, _ = bulk_setup
task_id = str(uuid.uuid4())
before_task_publish_handler(
headers={
"task": "documents.bulk_operations.execute_bulk_operation",
"id": task_id,
},
body=(
(),
{
"operation": "set_permissions",
"document_ids": [doc.pk],
"user_id": user.pk,
},
{},
),
)
task = PaperlessTask.objects.get(task_id=task_id)
assert task.owner_id == user.pk
assert task.task_type == PaperlessTask.TaskType.BULK_UPDATE
if failed:
task_failure_handler(task_id=task_id, exception=ValueError("Could not update"))
else:
task_postrun_handler(
task_id=task_id,
state="SUCCESS",
retval={"document_count": 1},
)
task.refresh_from_db()
assert task.status == ("failure" if failed else "success")
if failed:
assert task.result_data["error_message"] == "Could not update"
else:
assert task.result_data == {"document_count": 1}
@pytest.mark.django_db
def test_large_filtered_selection_is_one_task(bulk_setup):
_, client, _, _ = bulk_setup
Document.objects.bulk_create(
[
Document(title="bulk-large-selection", checksum=f"large-{i}")
for i in range(10_000)
],
)
with mock.patch("documents.views.execute_bulk_operation.apply_async") as enqueue:
enqueue.return_value.id = str(uuid.uuid4())
response = client.post(
"/api/documents/bulk_edit/",
{
"all": True,
"filters": {"title__icontains": "bulk-large-selection"},
"method": "set_correspondent",
"parameters": {"correspondent": None},
},
format="json",
)
assert response.status_code == 200
enqueue.assert_called_once()
assert len(enqueue.call_args.kwargs["kwargs"]["document_ids"]) == 10_000
@pytest.mark.django_db
@pytest.mark.parametrize(
("outcome", "expected_status", "expected_error"),
[
("success", "success", ""),
("failure", "failure", "Could not update"),
(
"expired",
"revoked",
"Task expired before execution. Check worker logs for details.",
),
],
)
def test_tracked_outcome_notifies_only_initiator(
bulk_setup,
outcome,
expected_status,
expected_error,
):
user, _, doc, _ = bulk_setup
task_id = str(uuid.uuid4())
before_task_publish_handler(
headers={
"task": execute_bulk_operation.name,
"id": task_id,
},
body=(
(),
{
"operation": "set_permissions",
"document_ids": [doc.pk],
"user_id": user.pk,
},
{},
),
)
task = PaperlessTask.objects.get(task_id=task_id)
assert task.owner_id == user.pk
assert task.task_type == PaperlessTask.TaskType.BULK_UPDATE
with mock.patch("documents.signals.handlers.async_to_sync") as send:
if outcome == "expired":
task_revoked_handler(
sender=execute_bulk_operation,
request=mock.Mock(id=task_id),
expired=True,
)
elif outcome == "failure":
task_failure_handler(
sender=execute_bulk_operation,
task_id=task_id,
exception=ValueError("Could not update"),
)
else:
task_postrun_handler(
task=execute_bulk_operation,
task_id=task_id,
state="SUCCESS",
retval={"document_count": 1},
)
group, event = send.return_value.call_args.args
assert group == f"bulk_operations_{user.pk}"
assert event["data"]["status"] == expected_status
assert event["data"]["error"] == expected_error
assert event["data"]["task_id"] == task_id
@pytest.mark.anyio
@pytest.mark.parametrize("anyio_backend", ["asyncio"])
async def test_completion_socket_is_private(anyio_backend):
from types import SimpleNamespace
from channels.layers import get_channel_layer
from channels.testing import WebsocketCommunicator
from paperless.consumers import StatusConsumer
own = WebsocketCommunicator(StatusConsumer.as_asgi(), "/ws/status/")
other = WebsocketCommunicator(StatusConsumer.as_asgi(), "/ws/status/")
own.scope["user"] = SimpleNamespace(id=1, is_authenticated=True)
other.scope["user"] = SimpleNamespace(id=2, is_authenticated=True)
assert (await own.connect())[0]
assert (await other.connect())[0]
try:
event = {
"type": "bulk_operation_status",
"data": {"task_id": "example", "status": "success"},
}
await get_channel_layer().group_send("bulk_operations_1", event)
assert await own.receive_json_from() == event
assert await other.receive_nothing(timeout=0.02)
finally:
await own.disconnect()
await other.disconnect()
@pytest.mark.django_db
@pytest.mark.parametrize("endpoint", ["delete", "bulk_edit"])
def test_trash_remains_synchronous(bulk_setup, endpoint):
_, client, doc, _ = bulk_setup
data = {"documents": [doc.pk]}
if endpoint == "bulk_edit":
data.update(method="delete", parameters={})
with (
mock.patch("documents.views.execute_bulk_operation.apply_async") as enqueue,
mock.patch("documents.search.get_backend"),
):
response = client.post(
f"/api/documents/{endpoint}/",
data,
format="json",
HTTP_ACCEPT="application/json; version=9",
)
assert response.status_code == 200
assert response.data == {"result": "OK"}
assert not Document.objects.filter(pk=doc.pk).exists()
assert Document.deleted_objects.filter(pk=doc.pk).exists()
enqueue.assert_not_called()
+36
View File
@@ -5,8 +5,11 @@ from unittest import mock
import pytest
from django.conf import settings
from django.contrib.sessions.models import Session
from django.db import connection
from django.test import TestCase
from django.test import override_settings
from django.test.utils import CaptureQueriesContext
from django.utils import timezone
from documents import tasks
@@ -612,3 +615,36 @@ class TestApplyAISuggestionsTask(DirectoriesMixin, TestCase):
apply_suggestions.assert_not_called()
self.assertIn("no longer exists", "".join(cm.output))
@pytest.mark.django_db
class TestClearExpiredSessions:
def test_clears_only_expired_sessions_in_batches(self) -> None:
"""
GIVEN:
- More than one batch of expired sessions and one still-valid session
WHEN:
- The clear_expired_sessions task runs
THEN:
- Only expired sessions are removed, using multiple deletes
"""
now = timezone.now()
Session.objects.bulk_create(
Session(
session_key=f"expired-{i}",
session_data="",
expire_date=now - timedelta(days=1),
)
for i in range(501)
)
Session.objects.create(
session_key="valid",
session_data="",
expire_date=now + timedelta(days=1),
)
with CaptureQueriesContext(connection) as queries:
tasks.clear_expired_sessions()
assert set(Session.objects.values_list("session_key", flat=True)) == {"valid"}
assert sum(query["sql"].startswith("DELETE FROM") for query in queries) == 2
+38 -145
View File
@@ -11,6 +11,7 @@ from collections.abc import Callable
from datetime import datetime
from datetime import timedelta
from http import HTTPStatus
from inspect import signature
from itertools import islice
from pathlib import Path
from time import mktime
@@ -106,6 +107,9 @@ from documents import bulk_edit
from documents.bulk_download import ArchiveOnlyStrategy
from documents.bulk_download import OriginalAndArchiveStrategy
from documents.bulk_download import OriginalsOnlyStrategy
from documents.bulk_operations import QUEUED_OPERATIONS
from documents.bulk_operations import execute_bulk_operation
from documents.bulk_operations import snapshot_field
from documents.caching import get_llm_suggestion_cache
from documents.caching import get_metadata_cache
from documents.caching import get_suggestion_cache
@@ -173,6 +177,7 @@ from documents.permissions import ViewDocumentsPermissions
from documents.permissions import annotate_document_count_for_related_queryset
from documents.permissions import get_document_count_filter_for_user
from documents.permissions import get_permission_grantees
from documents.permissions import has_document_operation_permissions
from documents.permissions import has_global_statistics_permission
from documents.permissions import has_perms_owner_aware
from documents.permissions import has_system_status_permission
@@ -2982,98 +2987,26 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
"merge_as_versions",
}
def _has_document_permissions(
self,
*,
user: User,
documents: list[int],
method,
parameters: dict[str, Any],
) -> bool:
if user.is_superuser:
return True
root_docs = {
get_root_document(doc)
for doc in Document.objects.select_related(
"owner",
"root_document__owner",
).filter(pk__in=documents)
}
user_is_owner_of_all_documents = all(
(doc.owner == user or doc.owner is None) for doc in root_docs
def _queue_bulk_operation(self, method, documents, parameters, from_webui):
if "update_callback" in parameters:
raise ValueError("Unsupported bulk operation parameter: update_callback")
signature(method).bind(documents, **parameters)
task = execute_bulk_operation.apply_async(
kwargs={
"operation": method.__name__,
"document_ids": documents,
"parameters": parameters,
"user_id": self.request.user.pk,
},
headers={
"trigger_source": (
PaperlessTask.TriggerSource.WEB_UI
if from_webui
else PaperlessTask.TriggerSource.API_UPLOAD
),
},
)
# check global and object permissions for all documents
has_perms = (
user.has_perm(
"documents.change_document",
)
and not Document.global_objects.filter(pk__in=documents)
.exclude(pk__in=permitted_document_ids(user, perm="change_document"))
.exists()
)
# check ownership for methods that change original document
if (
(
has_perms
and method
in [
bulk_edit.set_permissions,
bulk_edit.delete,
bulk_edit.rotate,
bulk_edit.delete_pages,
bulk_edit.edit_pdf,
bulk_edit.merge_as_versions,
bulk_edit.remove_password,
]
)
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (method == bulk_edit.edit_pdf and parameters.get("update_document"))
):
has_perms = has_perms and user_is_owner_of_all_documents
# check global add permissions for methods that create documents
if (
has_perms
and (
method in [bulk_edit.split, bulk_edit.merge]
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.add_document")
):
has_perms = False
# check global delete permissions for methods that delete documents
if (
has_perms
and (
method == bulk_edit.delete
# Sources stop being documents of their own, and removing one
# again afterwards needs delete_document
or method == bulk_edit.merge_as_versions
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and parameters.get("delete_original")
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.delete_document")
):
has_perms = False
return has_perms
return Response({"result": "OK", "task_id": task.id})
def _execute_document_action(
self,
@@ -3110,7 +3043,7 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
else PaperlessTask.TriggerSource.API_UPLOAD
)
if not self._has_document_permissions(
if not has_document_operation_permissions(
user=user,
documents=documents,
method=method,
@@ -3141,67 +3074,20 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
name="BulkEditDocumentsResult",
fields={
"result": serializers.CharField(),
"task_id": serializers.CharField(required=False),
},
),
},
),
)
class BulkEditView(DocumentOperationPermissionMixin):
MODIFIED_FIELD_BY_METHOD = {
"set_correspondent": "correspondent",
"set_document_type": "document_type",
"set_storage_path": "storage_path",
"add_tag": "tags",
"remove_tag": "tags",
"modify_tags": "tags",
"modify_custom_fields": "custom_fields",
"set_permissions": None,
LEGACY_MODIFIED_FIELDS = {
"delete": "deleted_at",
# These operations create new documents/versions no longer altering
# fields on the selected document in place
"rotate": None,
"delete_pages": None,
"split": None,
"merge": None,
"edit_pdf": None,
"reprocess": "checksum",
"remove_password": None,
}
serializer_class = BulkEditSerializer
@staticmethod
def _snapshot_field(doc_ids: list[int], field: str) -> dict[int, Any]:
"""
Returns each document's current value of field, for the audit log.
Tags and custom fields are one row per value, so they are gathered
into a sorted list of pks per document (empty when there are none).
Reading them through Document.values() instead would join those rows
and return one arbitrary value per document.
"""
if field == "tags":
rows = (
Document.tags.through.objects.filter(document_id__in=doc_ids)
.order_by("tag_id")
.values_list("document_id", "tag_id")
)
elif field == "custom_fields":
rows = (
CustomFieldInstance.objects.filter(document_id__in=doc_ids)
.order_by("pk")
.values_list("document_id", "pk")
)
else:
return dict(
Document.objects.filter(pk__in=doc_ids).values_list("pk", field),
)
values: dict[int, list[int]] = {doc_id: [] for doc_id in doc_ids}
for doc_id, pk in rows:
values[doc_id].append(pk)
return values
def post(self, request, *args, **kwargs):
request_method = request.data.get("method")
api_version = int(request.version or settings.REST_FRAMEWORK["DEFAULT_VERSION"])
@@ -3237,7 +3123,7 @@ class BulkEditView(DocumentOperationPermissionMixin):
if from_webui
else PaperlessTask.TriggerSource.API_UPLOAD
)
if not self._has_document_permissions(
if not has_document_operation_permissions(
user=user,
documents=documents,
method=method,
@@ -3246,14 +3132,21 @@ class BulkEditView(DocumentOperationPermissionMixin):
return HttpResponseForbidden("Insufficient permissions")
try:
modified_field = self.MODIFIED_FIELD_BY_METHOD.get(method.__name__, None)
if method.__name__ in QUEUED_OPERATIONS:
return self._queue_bulk_operation(
method,
documents,
parameters,
from_webui,
)
modified_field = self.LEGACY_MODIFIED_FIELDS.get(method.__name__)
if settings.AUDIT_LOG_ENABLED and modified_field:
old_values = self._snapshot_field(documents, modified_field)
old_values = snapshot_field(documents, modified_field)
result = method(documents, **parameters)
if settings.AUDIT_LOG_ENABLED and modified_field:
new_values = self._snapshot_field(documents, modified_field)
new_values = snapshot_field(documents, modified_field)
for doc in Document.objects.filter(pk__in=documents):
LogEntry.objects.log_create(
instance=doc,
+50 -50
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-11-07 02:45+0000\n"
"POT-Creation-Date: 2026-11-11 23:50+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -1661,8 +1661,8 @@ msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:483 documents/serialisers.py:844
#: documents/serialisers.py:2850 documents/views.py:349 documents/views.py:615
#: documents/views.py:2755 paperless_mail/serialisers.py:156
#: documents/serialisers.py:2850 documents/views.py:354 documents/views.py:620
#: documents/views.py:2760 paperless_mail/serialisers.py:156
msgid "Insufficient permissions."
msgstr ""
@@ -1702,7 +1702,7 @@ msgstr ""
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2936 documents/views.py:4805
#: documents/serialisers.py:2936 documents/views.py:4698
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
@@ -1953,40 +1953,40 @@ msgstr ""
msgid ", "
msgstr ""
#: documents/views.py:304
#: documents/views.py:309
msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr ""
#: documents/views.py:342 documents/views.py:2752
#: documents/views.py:347 documents/views.py:2757
msgid "Invalid more_like_id"
msgstr ""
#: documents/views.py:1679
#: documents/views.py:1684
msgid "Invalid AI configuration."
msgstr ""
#: documents/views.py:1690
#: documents/views.py:1695
msgid "AI backend request timed out."
msgstr ""
#: documents/views.py:1702
#: documents/views.py:1707
msgid "AI backend rejected the request. Check logs for details."
msgstr ""
#: documents/views.py:4821
#: documents/views.py:4714
#, python-format
msgid "Insufficient permissions to share document %(id)s."
msgstr ""
#: documents/views.py:4868
#: documents/views.py:4761
msgid "Bundle is already being processed."
msgstr ""
#: documents/views.py:4934
#: documents/views.py:4827
msgid "The share link bundle is still being prepared. Please try again later."
msgstr ""
#: documents/views.py:4948
#: documents/views.py:4841
msgid "The share link bundle is unavailable."
msgstr ""
@@ -2278,151 +2278,151 @@ msgstr ""
msgid "paperless application settings"
msgstr ""
#: paperless/settings/__init__.py:570
#: paperless/settings/__init__.py:581
msgid "English (US)"
msgstr ""
#: paperless/settings/__init__.py:571
#: paperless/settings/__init__.py:582
msgid "Arabic"
msgstr ""
#: paperless/settings/__init__.py:572
#: paperless/settings/__init__.py:583
msgid "Afrikaans"
msgstr ""
#: paperless/settings/__init__.py:573
#: paperless/settings/__init__.py:584
msgid "Belarusian"
msgstr ""
#: paperless/settings/__init__.py:574
#: paperless/settings/__init__.py:585
msgid "Bulgarian"
msgstr ""
#: paperless/settings/__init__.py:575
#: paperless/settings/__init__.py:586
msgid "Catalan"
msgstr ""
#: paperless/settings/__init__.py:576
#: paperless/settings/__init__.py:587
msgid "Czech"
msgstr ""
#: paperless/settings/__init__.py:577
#: paperless/settings/__init__.py:588
msgid "Danish"
msgstr ""
#: paperless/settings/__init__.py:578
#: paperless/settings/__init__.py:589
msgid "German"
msgstr ""
#: paperless/settings/__init__.py:579
#: paperless/settings/__init__.py:590
msgid "Greek"
msgstr ""
#: paperless/settings/__init__.py:580
#: paperless/settings/__init__.py:591
msgid "English (GB)"
msgstr ""
#: paperless/settings/__init__.py:581
#: paperless/settings/__init__.py:592
msgid "Spanish"
msgstr ""
#: paperless/settings/__init__.py:582
#: paperless/settings/__init__.py:593
msgid "Persian"
msgstr ""
#: paperless/settings/__init__.py:583
#: paperless/settings/__init__.py:594
msgid "Finnish"
msgstr ""
#: paperless/settings/__init__.py:584
#: paperless/settings/__init__.py:595
msgid "French"
msgstr ""
#: paperless/settings/__init__.py:585
#: paperless/settings/__init__.py:596
msgid "Hungarian"
msgstr ""
#: paperless/settings/__init__.py:586
#: paperless/settings/__init__.py:597
msgid "Indonesian"
msgstr ""
#: paperless/settings/__init__.py:587
#: paperless/settings/__init__.py:598
msgid "Italian"
msgstr ""
#: paperless/settings/__init__.py:588
#: paperless/settings/__init__.py:599
msgid "Japanese"
msgstr ""
#: paperless/settings/__init__.py:589
#: paperless/settings/__init__.py:600
msgid "Korean"
msgstr ""
#: paperless/settings/__init__.py:590
#: paperless/settings/__init__.py:601
msgid "Luxembourgish"
msgstr ""
#: paperless/settings/__init__.py:591
#: paperless/settings/__init__.py:602
msgid "Norwegian"
msgstr ""
#: paperless/settings/__init__.py:592
#: paperless/settings/__init__.py:603
msgid "Dutch"
msgstr ""
#: paperless/settings/__init__.py:593
#: paperless/settings/__init__.py:604
msgid "Polish"
msgstr ""
#: paperless/settings/__init__.py:594
#: paperless/settings/__init__.py:605
msgid "Portuguese (Brazil)"
msgstr ""
#: paperless/settings/__init__.py:595
#: paperless/settings/__init__.py:606
msgid "Portuguese"
msgstr ""
#: paperless/settings/__init__.py:596
#: paperless/settings/__init__.py:607
msgid "Romanian"
msgstr ""
#: paperless/settings/__init__.py:597
#: paperless/settings/__init__.py:608
msgid "Russian"
msgstr ""
#: paperless/settings/__init__.py:598
#: paperless/settings/__init__.py:609
msgid "Slovak"
msgstr ""
#: paperless/settings/__init__.py:599
#: paperless/settings/__init__.py:610
msgid "Slovenian"
msgstr ""
#: paperless/settings/__init__.py:600
#: paperless/settings/__init__.py:611
msgid "Serbian"
msgstr ""
#: paperless/settings/__init__.py:601
#: paperless/settings/__init__.py:612
msgid "Swedish"
msgstr ""
#: paperless/settings/__init__.py:602
#: paperless/settings/__init__.py:613
msgid "Turkish"
msgstr ""
#: paperless/settings/__init__.py:603
#: paperless/settings/__init__.py:614
msgid "Ukrainian"
msgstr ""
#: paperless/settings/__init__.py:604
#: paperless/settings/__init__.py:615
msgid "Vietnamese"
msgstr ""
#: paperless/settings/__init__.py:605
#: paperless/settings/__init__.py:616
msgid "Chinese Simplified"
msgstr ""
#: paperless/settings/__init__.py:606
#: paperless/settings/__init__.py:617
msgid "Chinese Traditional"
msgstr ""
+11
View File
@@ -12,6 +12,17 @@ from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices
from paperless.models import RemoteOCRMode
# OCRmyPDF arguments the parser sets itself, pointing into its own temporary
# directory. The parser reads these files back after OCR, so user supplied
# values can only break consumption (or write to arbitrary paths), never help.
OCR_RESERVED_USER_ARGS: frozenset[str] = frozenset(
{
"input_file_or_options",
"output_file",
"sidecar",
},
)
@dataclasses.dataclass
class BaseConfig:
+15
View File
@@ -45,12 +45,21 @@ class StatusConsumer(AsyncWebsocketConsumer):
await self.close()
return
await self.channel_layer.group_add("status_updates", self.channel_name)
await self.channel_layer.group_add(
f"bulk_operations_{self.scope['user'].id}",
self.channel_name,
)
await self.accept()
self._start_heartbeat()
async def disconnect(self, code: int) -> None:
await self._stop_heartbeat()
await self.channel_layer.group_discard("status_updates", self.channel_name)
if self._authenticated():
await self.channel_layer.group_discard(
f"bulk_operations_{self.scope['user'].id}",
self.channel_name,
)
def _start_heartbeat(self) -> None:
self.heartbeat_task = asyncio.create_task(self._heartbeat_loop())
@@ -84,3 +93,9 @@ class StatusConsumer(AsyncWebsocketConsumer):
await self.close()
elif await self._can_view(event["data"]):
await self.send(json.dumps(event))
async def bulk_operation_status(self, event: dict) -> None:
if not self._authenticated():
await self.close()
else:
await self.send(json.dumps(event))
+22 -1
View File
@@ -17,10 +17,12 @@ from django.conf import settings
from PIL import Image
from documents.parsers import ParseError
from documents.parsers import encode_thumbnail_webp
from documents.parsers import make_thumbnail_from_pdf
from documents.utils import copy_file_with_basic_stats
from documents.utils import maybe_override_pixel_limit
from documents.utils import remove_alpha_channel
from paperless.config import OCR_RESERVED_USER_ARGS
from paperless.config import OcrConfig
from paperless.models import CleanChoices
from paperless.models import ModeChoices
@@ -161,6 +163,10 @@ class RasterisedDocumentParser:
# ------------------------------------------------------------------
def get_thumbnail(self, document_path: Path, mime_type: str) -> Path:
if self.archive_path is None and self.is_image(mime_type):
thumbnail_path = self.tempdir / "convert.webp"
encode_thumbnail_webp(Path(document_path), thumbnail_path)
return thumbnail_path
return make_thumbnail_from_pdf(
self.archive_path or Path(document_path),
self.tempdir,
@@ -351,7 +357,22 @@ class RasterisedDocumentParser:
if self.settings.user_args is not None:
try:
ocrmypdf_args = {**ocrmypdf_args, **self.settings.user_args}
reserved = sorted(
OCR_RESERVED_USER_ARGS & self.settings.user_args.keys(),
)
if reserved:
self.log.warning(
f"Ignoring OCR user arguments managed by paperless: "
f"{', '.join(reserved)}",
)
ocrmypdf_args = {
**ocrmypdf_args,
**{
k: v
for k, v in self.settings.user_args.items()
if k not in OCR_RESERVED_USER_ARGS
},
}
except Exception as e:
self.log.warning(
f"There is an issue with PAPERLESS_OCR_USER_ARGS, so "
+8
View File
@@ -19,6 +19,7 @@ from PIL import Image
from rest_framework import serializers
from rest_framework.authtoken.serializers import AuthTokenSerializer
from paperless.config import OCR_RESERVED_USER_ARGS
from paperless.models import ApplicationConfiguration
from paperless.network import validate_outbound_http_url
from paperless.validators import reject_dangerous_svg
@@ -251,6 +252,13 @@ class ApplicationConfigurationSerializer(
def validate_user_args(self, value):
self._require_json_object("user_args", value)
if value is not None:
reserved = sorted(OCR_RESERVED_USER_ARGS & value.keys())
if reserved:
raise serializers.ValidationError(
f"user_args may not set arguments managed by paperless: "
f"{', '.join(reserved)}.",
)
return value
def validate_barcode_tag_mapping(self, value):
+19 -8
View File
@@ -450,23 +450,34 @@ CORS_EXPOSE_HEADERS = [
"Content-Disposition",
]
ALLOWED_HOSTS = get_list_from_env("PAPERLESS_ALLOWED_HOSTS", default=["*"])
if ALLOWED_HOSTS != ["*"]:
# always allow localhost. Necessary e.g. for healthcheck in docker.
ALLOWED_HOSTS.append("localhost")
def _get_allowed_hosts() -> list[str]:
if "PAPERLESS_ALLOWED_HOSTS" in os.environ or os.getenv("PAPERLESS_URL"):
allowed_hosts = get_list_from_env("PAPERLESS_ALLOWED_HOSTS")
else:
allowed_hosts = ["*"]
if allowed_hosts != ["*"]:
# Always allow localhost for the Docker health check.
allowed_hosts.append("localhost")
return allowed_hosts
ALLOWED_HOSTS = _get_allowed_hosts()
def _parse_paperless_url():
global CSRF_TRUSTED_ORIGINS, CORS_ALLOWED_ORIGINS, ALLOWED_HOSTS
url = os.getenv("PAPERLESS_URL")
if url:
hostname = urlparse(url).hostname
if not hostname:
hostname = (urlparse(url).hostname or "").removesuffix(".")
if not hostname or hostname == "*" or hostname.startswith("."):
raise ImproperlyConfigured(
f"PAPERLESS_URL {url!r} has no hostname, expected a URL like https://example.com",
f"PAPERLESS_URL {url!r} has no valid hostname, expected a URL like https://example.com",
)
CSRF_TRUSTED_ORIGINS.append(url)
CORS_ALLOWED_ORIGINS.append(url)
if ":" in hostname and not hostname.startswith("["):
hostname = f"[{hostname}]" # Add brackets around IPv6 addresses
ALLOWED_HOSTS.append(hostname)
return url
@@ -690,7 +701,7 @@ logging.config.dictConfig(LOGGING)
# https://docs.celeryq.dev/en/stable/userguide/configuration.html
# Tasks outside tasks.py must be loaded even when Django skips checks
CELERY_IMPORTS = ("documents.bulk_edit",)
CELERY_IMPORTS = ("documents.bulk_edit", "documents.bulk_operations")
CELERY_BROKER_URL = _CELERY_REDIS_URL
CELERY_RESULT_BACKEND = _CELERY_REDIS_URL
+11
View File
@@ -165,6 +165,17 @@ def parse_beat_schedule() -> dict:
"expires": 23.0 * 60.0 * 60.0,
},
},
{
"name": "Clear expired sessions",
"env_key": "PAPERLESS_SESSION_CLEANUP_TASK_CRON",
# Default daily at 03:00
"env_default": "0 3 * * *",
"task": "documents.tasks.clear_expired_sessions",
"options": {
# 1 hour before default schedule sends again
"expires": 23.0 * 60.0 * 60.0,
},
},
]
for task in tasks:
# Either get the environment setting or use the default
@@ -243,6 +243,34 @@ class TestParserSettingsFromDb(DirectoriesMixin, FileSystemAssertsMixin, TestCas
"UseDeviceIndependentColor",
)
def test_ocr_user_args_cannot_override_reserved(self) -> None:
"""
GIVEN:
- User arguments which attempt to override arguments the parser
manages itself, alongside a legitimate argument
WHEN:
- OCR parameters are constructed
THEN:
- Reserved arguments keep the parser supplied values
- The legitimate argument is still applied
"""
instance = ApplicationConfiguration.objects.all().first()
assert instance is not None
instance.user_args = {
"sidecar": "/tmp/evil.py",
"output_file": "/tmp/out.pdf",
"input_file_or_options": "/etc/passwd",
"unpaper_args": "--pre-rotate 90",
}
instance.save()
params = self.get_params()
self.assertEqual(params["sidecar"], "sidecar.txt")
self.assertEqual(params["output_file"], "output.pdf")
self.assertEqual(params["input_file_or_options"], "input.pdf")
self.assertEqual(params["unpaper_args"], "--pre-rotate 90")
def test_ocr_user_args(self) -> None:
"""
GIVEN:
@@ -18,6 +18,7 @@ from ocrmypdf import SubprocessOutputError
from PIL import Image
from documents.parsers import ParseError
from documents.parsers import get_default_thumbnail
from documents.parsers import rasterize_pdf_page_to_png
from paperless.models import ModeChoices
from paperless.parsers import ParserProtocol
@@ -270,6 +271,57 @@ class TestDpiHelpers:
class TestGetThumbnail:
@pytest.mark.parametrize(
("filename", "mime_type"),
[
pytest.param("simple.jpg", "image/jpeg", id="jpeg"),
pytest.param("simple.png", "image/png", id="png"),
pytest.param("simple.tif", "image/tiff", id="tiff"),
pytest.param("simple.gif", "image/gif", id="gif"),
pytest.param("simple.bmp", "image/bmp", id="bmp"),
pytest.param("document.webp", "image/webp", id="webp"),
pytest.param("simple.heic", "image/heic", id="heic"),
],
)
def test_image_without_archive_has_image_thumbnail(
self,
mocker: MockerFixture,
tesseract_parser: RasterisedDocumentParser,
tesseract_samples_dir: Path,
filename: str,
mime_type: str,
) -> None:
original = tesseract_samples_dir / filename
render_pdf = mocker.patch("documents.parsers.rasterize_pdf_page_to_png")
thumb = tesseract_parser.get_thumbnail(original, mime_type)
render_pdf.assert_not_called()
assert thumb.read_bytes() != get_default_thumbnail().read_bytes()
with Image.open(thumb) as im:
assert im.format == "WEBP"
assert 0 < im.width <= 500
assert 0 < im.height <= 5000
def test_image_with_archive_uses_pdf_thumbnail(
self,
mocker: MockerFixture,
tesseract_parser: RasterisedDocumentParser,
tesseract_samples_dir: Path,
simple_digital_pdf_file: Path,
) -> None:
tesseract_parser.archive_path = simple_digital_pdf_file
target = "paperless.parsers.tesseract.make_thumbnail_from_pdf"
make_thumbnail = mocker.patch(target)
original = tesseract_samples_dir / "simple.jpg"
tesseract_parser.get_thumbnail(original, "image/jpeg")
make_thumbnail.assert_called_once_with(
simple_digital_pdf_file,
tesseract_parser.tempdir,
)
def test_thumbnail_is_file(
self,
tesseract_parser: RasterisedDocumentParser,
@@ -182,6 +182,7 @@ def make_expected_schedule(
workflow_expire = 59.0 * 60.0
llm_index_expire = 23.0 * 60.0 * 60.0
share_link_cleanup_expire = 23.0 * 60.0 * 60.0
session_cleanup_expire = 23.0 * 60.0 * 60.0
schedule: dict[str, Any] = {
"Check all e-mail accounts": {
@@ -248,6 +249,14 @@ def make_expected_schedule(
"headers": {"trigger_source": "scheduled"},
},
},
"Clear expired sessions": {
"task": "documents.tasks.clear_expired_sessions",
"schedule": crontab(minute=0, hour="3"),
"options": {
"expires": session_cleanup_expire,
"headers": {"trigger_source": "scheduled"},
},
},
}
overrides = overrides or {}
@@ -298,6 +307,7 @@ class TestParseBeatSchedule:
"PAPERLESS_WORKFLOW_SCHEDULED_TASK_CRON": "disable",
"PAPERLESS_LLM_INDEX_TASK_CRON": "disable",
"PAPERLESS_SHARE_LINK_BUNDLE_CLEANUP_CRON": "disable",
"PAPERLESS_SESSION_CLEANUP_TASK_CRON": "disable",
},
{},
id="all-disabled",
+89 -4
View File
@@ -3,9 +3,14 @@ from unittest import TestCase
from unittest import mock
import pytest
from django.core.exceptions import DisallowedHost
from django.core.exceptions import ImproperlyConfigured
from django.http import HttpRequest
from django.test import override_settings
import paperless.settings as paperless_settings
from paperless.settings import _get_allauth_trusted_proxy_count
from paperless.settings import _get_allowed_hosts
from paperless.settings import _get_classifier_language_setting
from paperless.settings import _get_search_language_setting
from paperless.settings import _parse_paperless_url
@@ -168,8 +173,88 @@ class TestPaperlessURLSettings(TestCase):
self.assertIn(url, settings.CORS_ALLOWED_ORIGINS)
@pytest.mark.parametrize("url", ["example.com", "https://", "/just/a/path"])
def test_paperless_url_without_hostname_is_rejected(
@pytest.mark.parametrize(
("url", "allowed_hosts", "expected", "accepted_host", "rejected_host"),
[
(None, None, ["*"], "other.example.com", None),
(None, "", ["localhost"], "localhost:8000", "other.example.com"),
(
None,
"lan.example.com",
["lan.example.com", "localhost"],
"lan.example.com",
"other.example.com",
),
(None, "*", ["*"], "other.example.com", None),
(
"https://example.com:8443",
None,
["localhost", "example.com"],
"example.com:8443",
"other.example.com",
),
(
"https://example.com.",
"",
["localhost", "example.com"],
"example.com.",
"other.example.com",
),
(
"http://[::1]:8000",
None,
["localhost", "[::1]"],
"[::1]:8000",
"other.example.com",
),
("https://example.com", "*", ["*", "example.com"], "other.example.com", None),
(
"https://example.com",
"lan.example.com",
["lan.example.com", "localhost", "example.com"],
"lan.example.com",
"other.example.com",
),
],
)
def test_allowed_hosts_at_startup(
monkeypatch: pytest.MonkeyPatch,
url: str | None,
allowed_hosts: str | None,
expected: list[str],
accepted_host: str,
rejected_host: str | None,
) -> None:
for name, value in (
("PAPERLESS_URL", url),
("PAPERLESS_ALLOWED_HOSTS", allowed_hosts),
):
if value is None:
monkeypatch.delenv(name, raising=False)
else:
monkeypatch.setenv(name, value)
monkeypatch.setattr(paperless_settings, "ALLOWED_HOSTS", _get_allowed_hosts())
monkeypatch.setattr(paperless_settings, "CSRF_TRUSTED_ORIGINS", [])
monkeypatch.setattr(paperless_settings, "CORS_ALLOWED_ORIGINS", [])
assert _parse_paperless_url() == url
assert expected == paperless_settings.ALLOWED_HOSTS
request = HttpRequest()
with override_settings(ALLOWED_HOSTS=expected):
request.META["HTTP_HOST"] = accepted_host
assert request.get_host() == accepted_host
if rejected_host:
request.META["HTTP_HOST"] = rejected_host
with pytest.raises(DisallowedHost):
request.get_host()
@pytest.mark.parametrize(
"url",
["example.com", "https://", "/just/a/path", "https://*", "https://.example.com"],
)
def test_paperless_url_without_valid_hostname_is_rejected(
monkeypatch: pytest.MonkeyPatch,
url: str,
) -> None:
@@ -179,9 +264,9 @@ def test_paperless_url_without_hostname_is_rejected(
WHEN:
- The URL is parsed
THEN:
- A configuration error is raised instead of adding None to ALLOWED_HOSTS
- A configuration error is raised for missing or wildcard hostnames
"""
monkeypatch.setenv("PAPERLESS_URL", url)
with pytest.raises(ImproperlyConfigured, match="no hostname"):
with pytest.raises(ImproperlyConfigured, match="no valid hostname"):
_parse_paperless_url()
+1 -1
View File
@@ -22,8 +22,8 @@ logger = logging.getLogger("paperless_ai.chat")
CHAT_METADATA_DELIMITER = "\n\n__PAPERLESS_CHAT_METADATA__"
CHAT_ERROR_MESSAGE = "Sorry, something went wrong while generating a response."
CHAT_NO_CONTENT_MESSAGE = "Sorry, I couldn't find any content to answer your question."
MAX_CHAT_REFERENCES = 3
CHAT_RETRIEVER_TOP_K = 5
MAX_CHAT_REFERENCES = CHAT_RETRIEVER_TOP_K
def _build_chat_prompt(output_language: str | None) -> str:
+20 -5
View File
@@ -1,4 +1,5 @@
import re
from contextlib import suppress
from typing import TYPE_CHECKING
from django.conf import settings
@@ -20,7 +21,17 @@ OCR_LEADER_REGEX = re.compile(r"[._\-\u00b7]{4,}")
HORIZONTAL_WHITESPACE_REGEX = re.compile(r"[ \t\u00a0]+")
def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
def get_embedding_model(
config: AIConfig,
*,
refresh: bool = False,
) -> "BaseEmbedding":
"""
Build the embedding model for the configured backend.
For Hugging Face, a cached model loads offline by default, saving seconds per
call. Pass ``refresh=True`` to check the Hub for a newer model revision.
"""
match config.llm_embedding_backend:
case LLMEmbeddingBackend.OPENAI_LIKE:
from llama_index.embeddings.openai_like import OpenAILikeEmbedding
@@ -52,11 +63,15 @@ def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
case LLMEmbeddingBackend.HUGGINGFACE:
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
return HuggingFaceEmbedding(
model_name=config.llm_embedding_model
hf_kwargs = {
"model_name": config.llm_embedding_model
or "sentence-transformers/all-MiniLM-L6-v2",
cache_folder=str(settings.DATA_DIR / "hf_cache"),
)
"cache_folder": str(settings.DATA_DIR / "hf_cache"),
}
if not refresh:
with suppress(OSError):
return HuggingFaceEmbedding(**hf_kwargs, local_files_only=True)
return HuggingFaceEmbedding(**hf_kwargs)
case LLMEmbeddingBackend.OLLAMA:
from llama_index.embeddings.ollama import OllamaEmbedding
from ollama import AsyncClient
+1 -1
View File
@@ -470,7 +470,7 @@ def update_llm_index(
logger.warning("No documents found to index.")
chunk_size = config.llm_embedding_chunk_size
embed_model = get_embedding_model(config)
embed_model = get_embedding_model(config, refresh=rebuild)
with write_store(embed_model_name=model_name) as store:
if rebuild or not store.table_exists():
@@ -212,6 +212,10 @@ def test_update_llm_index(
ai_config.assert_called_once()
build_document_node.assert_called_once_with(real_document, chunk_size=512)
indexing.get_embedding_model.assert_called_once_with(
mock_config,
refresh=True,
)
@pytest.mark.django_db
@@ -336,6 +340,8 @@ def test_update_llm_index_partial_update(
indexing.update_llm_index(rebuild=False)
assert indexing.get_embedding_model.call_args.kwargs == {"refresh": False}
with indexing.get_vector_store() as store:
assert store.table_exists(), (
"Expected the vector store table to exist after incremental update"
+36
View File
@@ -1,5 +1,6 @@
from unittest.mock import ANY
from unittest.mock import MagicMock
from unittest.mock import call
from unittest.mock import patch
import pytest
@@ -167,10 +168,45 @@ def test_get_embedding_model_huggingface(mock_ai_config):
MockHuggingFaceEmbedding.assert_called_once_with(
model_name="sentence-transformers/all-MiniLM-L6-v2",
cache_folder=str(settings.DATA_DIR / "hf_cache"),
local_files_only=True,
)
assert model == MockHuggingFaceEmbedding.return_value
def test_get_embedding_model_huggingface_downloads_when_not_cached(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.HUGGINGFACE
mock_ai_config.return_value.llm_embedding_model = "some/model"
downloaded = MagicMock()
with patch(
"llama_index.embeddings.huggingface.HuggingFaceEmbedding",
side_effect=[OSError("not cached"), downloaded],
) as MockHuggingFaceEmbedding:
model = get_embedding_model(mock_ai_config.return_value)
cache_folder = str(settings.DATA_DIR / "hf_cache")
assert MockHuggingFaceEmbedding.call_args_list == [
call(model_name="some/model", cache_folder=cache_folder, local_files_only=True),
call(model_name="some/model", cache_folder=cache_folder),
]
assert model is downloaded
def test_get_embedding_model_huggingface_refresh_checks_the_hub(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.HUGGINGFACE
mock_ai_config.return_value.llm_embedding_model = "some/model"
with patch(
"llama_index.embeddings.huggingface.HuggingFaceEmbedding",
) as MockHuggingFaceEmbedding:
get_embedding_model(mock_ai_config.return_value, refresh=True)
MockHuggingFaceEmbedding.assert_called_once_with(
model_name="some/model",
cache_folder=str(settings.DATA_DIR / "hf_cache"),
)
def test_get_embedding_model_ollama(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.OLLAMA
mock_ai_config.return_value.llm_embedding_model = "embeddinggemma"
+7 -1
View File
@@ -1159,7 +1159,13 @@ class MailAccountHandler(LoggingMixin):
new_headers += message.obj._headers
message.obj._headers = new_headers
f.write(message.obj.as_bytes())
# RFC 5322 requires CRLF; the default compat32 policy writes bare LF,
# which Outlook misreads as literal "=" in quoted-printable soft breaks
f.write(
message.obj.as_bytes(
policy=message.obj.policy.clone(linesep="\r\n"),
),
)
correspondent = self._get_correspondent(message, rule)