Compare commits

...
Author SHA1 Message Date
stumpylog b347022c4c docs: add views/serialisers split spec and implementation plan
Specifies splitting documents/views.py and documents/serialisers.py
(5,395 and 3,532 lines) into per-domain-area packages, pure move/reorganize
with no behavior change. Scoped as its own branch against dev, to land
after the in-flight feature-ai-taxonomy-hints-v2 work merges rather than
layering on top of it.
2026-08-13 13:35:06 -07:00
GitHub Actions 01c12d9ea4 Auto translate strings 2026-08-13 19:48:40 +00:00
Max TruxaandGitHub f5c0d118f7 Fix: fix validation of workflow title assignment (#13659) 2026-08-13 12:46:57 -07:00
ff13847d0a Feature: Allow selection of compression type and and level during export (#13661)
* Feature: Allow configuring the compression type and compression levels during export

Building on the zip export improvements, this now allows users to further configure the
zip to fit their needs.  A simple stored zip for speed, or a high compression zstd for
the smallest archive.  Full validation of the method and levels at the command line

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-08-13 18:29:33 +00:00
shamoonandGitHub 634f803872 Fix: dont clip search dropdown on mobile (#13675) 2026-08-13 10:57:16 -07:00
GitHub Actions 639d566a7c Auto translate strings 2026-08-13 16:50:00 +00:00
shamoonandGitHub 0a94f8f0d4 Fix: use proper signal for modal buttonsEnabled (#13672) 2026-08-13 09:48:23 -07:00
shamoonandGitHub 2a8579f610 Fix: include sharelink bundle perms in WebUI (#13664) 2026-08-12 14:19:21 -07:00
GitHub Actions 4789fe9a52 Auto translate strings 2026-08-12 19:04:59 +00:00
shamoonandGitHub e150c8c7c0 Enhancement: customizable icons for saved views (#13388) 2026-08-12 19:03:24 +00:00
JaydenandGitHub 879cd4a30a Enhancement: Add --url argument to document_fuzzy_match to improve output (#13123)
Added a new --url argument to specify the base URL of the Paperless instance, allowing matched documents to be displayed as clickable links. Updated the logic to fetch document titles based on the presence of the base URL.
2026-08-12 08:23:05 -07:00
Trenton HandGitHub 6a02b87dde Feature: Updates remote OCR parser to respect the OCR mode setting (#13408)
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks

* Updates the documentation to be correct about the respecting now

* merge conflict fixing
2026-08-11 19:23:46 +00:00
Trenton HandGitHub 59a2651804 Fix: pass document chat queries as a QuerySet instead of a materialized list (#13638)
In tracemalloc based profiling, not materializing the whole Document list
reduced memory to approximately 20% of the baseline, with a peak memory
that scaled with the library size.  Now, the lazt queryset is used and only
the needed pk value is actually contributing to memory
2026-08-11 15:25:08 +00:00
GitHub Actions a99f63e059 Auto translate strings 2026-08-11 14:11:20 +00:00
shamoonandGitHub 939cb52f6e Fix: add pagination to saved views management page (#13646) 2026-08-11 07:08:30 -07:00
shamoonandGitHub 855669ddf9 Fix: fixes for workflow assign custom field values (#13630) 2026-08-10 07:38:07 -07:00
79 changed files with 3903 additions and 1030 deletions
+16
View File
@@ -299,6 +299,8 @@ optional arguments:
-sm, --split-manifest
-z, --zip
-zn, --zip-name
--zip-compression
--zip-compression-level
--data-only
--no-progress-bar
--passphrase
@@ -361,6 +363,19 @@ If `-z` or `--zip` is provided, the export will be a zip file
in the target directory, named according to the current local date or the
value set in `-zn` or `--zip-name`.
The compression method for the zip can be set with `--zip-compression`
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
`--zip-compression-level` (deflated: 09, bzip2: 19, zstd: -2222; ignored
for `stored` and `lzma`). Both options require `--zip`.
!!! warning
`zstd` compression requires Python 3.14 or newer on **both** the machine
creating the export and any machine importing it. An archive compressed with
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
imported on a runtime that lacks the codec; the importer will refuse it with
a clear error. The default `deflated` is universally readable.
If `--data-only` is provided, only the database will be exported. This option is intended
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
@@ -699,6 +714,7 @@ document_fuzzy_match [--ratio] [--processes N]
| --ratio | No | 85.0 | a number between 0 and 100, setting how similar a document must be for it to be reported. Higher numbers mean more similarity. |
| --processes | No | 1/4 of system cores | Number of processes to use for matching. Setting 1 disables multiple processes |
| --delete | No | False | If provided, one document of a matched pair above the ratio will be deleted. |
| --url | No | blank | If an instance URL is provided, the output table will show URLs to each documents instead of the document ID and name. |
!!! warning
+5 -4
View File
@@ -948,10 +948,11 @@ for display in the web interface.
!!! note
The **remote OCR parser** (Azure AI) always produces a searchable
PDF and stores it as the archive copy, regardless of this setting.
`ARCHIVE_FILE_GENERATION=never` has no effect when the remote
parser handles a document.
The **remote OCR parser** (Azure AI) also honors this setting: when
no archive is requested (`never`, or `auto` with a born-digital PDF),
the remote engine is skipped entirely and locally-extracted text is
used instead, avoiding an unnecessary API call and a duplicate text
layer.
#### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
+5 -4
View File
@@ -187,10 +187,11 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
### Remote OCR parser
If you use the **remote OCR parser** (Azure AI), note that it always produces a
searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
has no effect for documents handled by the remote parser - the archive is produced
unconditionally by the remote engine.
If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
honored the same way as for the local engine: when no archive is requested
(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
entirely and locally-extracted text is used instead, avoiding an unnecessary
API call and a duplicate text layer.
## Search Index (Whoosh -> Tantivy)
@@ -0,0 +1,428 @@
# Split views.py and serialisers.py Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Split `src/documents/views.py` (5,395 lines) and `src/documents/serialisers.py` (3,532 lines) into domain-based module packages, with zero behavior change.
**Architecture:** Both files become packages (`documents/views/`, `documents/serialisers/`), one module per domain area. Serialisers split first (views depend on serialisers, never the reverse), then views, then the three external call sites (`paperless/urls.py`, `paperless_mail/views.py`, `paperless_mail/serialisers.py`) are pointed at the new submodules. No `__init__.py` re-exports in either package — every internal and external consumer imports the exact submodule.
**Tech Stack:** Django REST Framework (viewsets/serializers), ruff (lint/format), pytest via the project's VM test runner.
**Spec:** `docs/superpowers/specs/2026-08-13-views-serialisers-split-design.md`
## Global Constraints
- No behavior change: class/function bodies, names, and public API responses are unchanged — pure move/reorganize. (spec: Non-goals)
- Domain module names are identical across both packages (`bulk_edit.py` exists in both, etc.). (spec: Import direction)
- Import direction is one-way: `documents/views/*` may import from `documents/serialisers/*`; `documents/serialisers/*` must never import from `documents/views/*`. (spec: Import direction)
- Neither package's `__init__.py` re-exports submodule contents — every consumer, internal or external, imports the specific submodule (e.g. `from documents.views.workflows import WorkflowViewSet`). (spec: Architecture)
- `src/documents/tests/test_views.py` and `src/documents/tests/test_api_documents.py` are not modified — they must pass unchanged, proving the move didn't alter behavior. (spec: Non-goals, Testing)
- This branch targets `dev` and is separate from `feature-ai-taxonomy-hints-v2`. (spec: Non-goals)
- Backend tests run on the Linux VM via the helper script, never locally: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "<pytest targets>"`. `ruff check` / `ruff format` run locally (global binary, not `uv run ruff`).
---
## Reference: symbol-to-module maps
These tables (from the spec) are the authoritative source for which class/function goes to which new file. Copy them exactly — do not improvise groupings.
### `documents/serialisers/` map
| Module | Symbols |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `base.py` | `DynamicFieldsModelSerializer`, `DocumentUpdateFieldsModelSerializer`, `MatchingModelSerializer`, `SetPermissionsMixin`, `SerializerWithPerms`, `SetPermissionsSerializer`, `OwnedObjectSerializer`, `OwnedObjectListSerializer`, `ReadWriteSerializerMethodField`, `DocumentListSerializer`, `DocumentSelectionSerializer`, `SourceModeValidationMixin`, `BasicUserSerializer`, `NotesSerializer` |
| `metadata.py` | `CorrespondentSerializer`, `DocumentTypeSerializer`, `DeprecatedColors`, `ColorField`, `TagSerializer`, `CorrespondentField`, `TagsField`, `DocumentTypeField`, `StoragePathField`, `StoragePathSerializer`, `StoragePathTestSerializer`, `CustomFieldSerializer`, `CustomFieldInstanceSerializer`, `validate_documentlink_targets` |
| `documents.py` | `DocumentSerializer`, `SearchResultListSerializer`, `SearchResultSerializer`, `DuplicateDocumentSummarySerializer`, `_DocumentVersionInfo`, `DocumentVersionInfoSerializer`, `DocumentVersionSerializer`, `DocumentVersionLabelSerializer`, `_get_viewable_duplicates` |
| `upload.py` | `PostDocumentSerializer` |
| `saved_views.py` | `SavedViewFilterRuleSerializer`, `SavedViewSerializer` |
| `bulk_edit.py` | `RotateDocumentsSerializer`, `MergeDocumentsSerializer`, `EditPdfDocumentsSerializer`, `RemovePasswordDocumentsSerializer`, `DeleteDocumentsSerializer`, `ReprocessDocumentsSerializer`, `BulkEditSerializer`, `BulkDownloadSerializer`, `BulkEditObjectsSerializer` |
| `sharing.py` | `EmailSerializer`, `ShareLinkSerializer`, `ShareLinkBundleSerializer` |
| `tasks.py` | `TaskSerializerV10`, `TaskSerializerV9`, `TaskSummarySerializer`, `RunTaskSerializer`, `AcknowledgeTasksViewSerializer` |
| `workflows.py` | `WorkflowTriggerSerializer`, `WorkflowActionEmailSerializer`, `WorkflowActionWebhookSerializer`, `WorkflowActionSerializer`, `WorkflowSerializer` |
| `system.py` | `UiSettingsViewSerializer`, `TrashSerializer` |
Extraction order matters (later modules reference earlier ones): `base``metadata``documents``upload``saved_views``bulk_edit``sharing``tasks``workflows``system`.
### `documents/views/` map
| Module | Symbols |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `base.py` | `PassUserMixin`, `BulkPermissionMixin`, `PermissionsAwareDocumentCountMixin`, `DocumentSelectionMixin`, `DocumentOperationPermissionMixin`, `SearchParams`, `SearchResultPage`, `ResolvedRequestDocs`, `_get_tantivy_query_and_mode`, `_get_more_like_id`, `serve_file` |
| `index.py` | `IndexView`, `serve_logo` |
| `metadata.py` | `CorrespondentViewSet`, `TagViewSet`, `DocumentTypeViewSet`, `StoragePathViewSet`, `CustomFieldViewSet`, `_get_llm_output_language` |
| `documents.py` | `EmailDocumentDetailSchema`, `DocumentViewSet`, `UnifiedSearchViewSet` |
| `upload.py` | `PostDocumentView` |
| `chat.py` | `ChatStreamingSerializer`, `ChatStreamingView` |
| `search.py` | `SearchAutoCompleteView`, `GlobalSearchView`, `SelectionDataView`, `StatisticsView` |
| `bulk_edit.py` | `BulkEditView`, `RotateDocumentsView`, `MergeDocumentsView`, `DeleteDocumentsView`, `ReprocessDocumentsView`, `EditPdfDocumentsView`, `RemovePasswordDocumentsView`, `BulkEditObjectsView`, `BulkDownloadView` |
| `sharing.py` | `ShareLinkViewSet`, `ShareLinkBundleViewSet`, `SharedLinkView` |
| `saved_views.py` | `SavedViewViewSet` |
| `tasks.py` | `_TasksViewSetSchema`, `TasksViewSet` |
| `workflows.py` | `WorkflowTriggerViewSet`, `WorkflowActionViewSet`, `WorkflowViewSet` |
| `system.py` | `UiSettingsView`, `RemoteVersionView`, `SystemStatusView`, `TrashView` |
| `logs.py` | `LogViewSet` |
Extraction order: `base``index``metadata``documents``upload``chat``search``bulk_edit``sharing``saved_views``tasks``workflows``system``logs`. Note `ChatStreamingSerializer` is defined in `views.py` today, directly above `ChatStreamingView` — it moves with it into `views/chat.py`, not into the serialisers package.
### Mechanical extraction recipe (applies to every task below)
For each module being created:
1. `grep -n "^class |^def " src/documents/<serialisers|views>.py` to get current line numbers for every symbol still in the monolith (numbers shift as earlier modules are extracted, so re-run this each time, don't reuse stale numbers).
2. Create the new file. Start it by copying the **entire top-of-file import block** from the monolith verbatim, plus a relative `from .base import ...` line if the module isn't `base.py` itself.
3. Cut each listed symbol (including any decorators/comments immediately above it) from the monolith and paste it into the new file, preserving original order.
4. Remove the cut symbols from the monolith.
5. Run `ruff check --select F401,F811,F821 <new file> <monolith file>` and fix everything reported:
- `F401` (unused import) → delete the import line.
- `F821` (undefined name) → the symbol lives in a sibling module already extracted; add `from .<sibling> import <Symbol>`. If it hasn't been extracted yet, that's an ordering bug — stop and re-check the extraction order table.
- `F811` (redefinition) → duplicate import, delete one.
6. Run `ruff format <new file> <monolith file>`.
## Task 1: Scaffold `documents/serialisers/` and extract `base.py`
**Agent:** django-expert — **Model:** sonnet (mechanical extraction, but sets the foundation every later serialiser module imports from — get the base set right or every later task inherits the mistake)
**Files:**
- Create: `src/documents/serialisers/__init__.py` (empty — no re-exports, per Global Constraints)
- Create: `src/documents/serialisers/base.py`
- Modify: `src/documents/serialisers.py` (shrinks; stays in place as the monolith for the remaining tasks in this phase — it is only deleted in Task 2 once empty)
- Test: `src/documents/tests/` (full app suite), `src/paperless_mail/tests/`
**Interfaces:**
- Produces: `documents.serialisers.base` exporting `DynamicFieldsModelSerializer`, `DocumentUpdateFieldsModelSerializer`, `MatchingModelSerializer`, `SetPermissionsMixin`, `SerializerWithPerms`, `SetPermissionsSerializer`, `OwnedObjectSerializer`, `OwnedObjectListSerializer`, `ReadWriteSerializerMethodField`, `DocumentListSerializer`, `DocumentSelectionSerializer`, `SourceModeValidationMixin`, `BasicUserSerializer`, `NotesSerializer` — every later serialiser/view module that needs one of these imports `from documents.serialisers.base import <Symbol>`.
- [ ] **Step 1: Create the package directory and empty `__init__.py`**
```bash
mkdir -p src/documents/serialisers
touch src/documents/serialisers/__init__.py
```
- [ ] **Step 2: Extract `base.py` per the mechanical extraction recipe above**
Move exactly these 14 symbols (in their current relative order) out of `src/documents/serialisers.py` into `src/documents/serialisers/base.py`: `DynamicFieldsModelSerializer`, `DocumentUpdateFieldsModelSerializer`, `MatchingModelSerializer`, `SetPermissionsMixin`, `SerializerWithPerms`, `SetPermissionsSerializer`, `OwnedObjectSerializer`, `OwnedObjectListSerializer`, `ReadWriteSerializerMethodField`, `DocumentListSerializer`, `DocumentSelectionSerializer`, `SourceModeValidationMixin`, `BasicUserSerializer`, `NotesSerializer`.
Run the ruff fix-up (`ruff check --select F401,F811,F821 src/documents/serialisers/base.py src/documents/serialisers.py` then `ruff format` both files) as described in the recipe.
- [ ] **Step 3: Verify `documents.serialisers` (the monolith module, still at `src/documents/serialisers.py`) still imports cleanly and the app still boots**
Note: at this point Python resolves `documents.serialisers` to the package `src/documents/serialisers/__init__.py` (empty), **not** to `src/documents/serialisers.py` — having both a `serialisers.py` file and a `serialisers/` directory in the same parent package is invalid and Python will pick the package. So before running anything, rename the monolith out of the way so it's importable as a submodule of the new package for the rest of Phase A:
```bash
git mv src/documents/serialisers.py src/documents/serialisers/_monolith.py
```
Everywhere else in this phase, "the monolith file" now means `src/documents/serialisers/_monolith.py`. Because nothing outside this package imports the monolith directly by its old dotted path (`documents.serialisers` resolved to the file before; now it's the package), you must update every consumer of `documents.serialisers` symbols still owned by the monolith to import from `documents.serialisers._monolith` for the remainder of this phase. Concretely, in `src/documents/views.py`, change every `from documents.serialisers import <Symbol>` line for a symbol _not yet extracted_ (i.e., not one of the 14 `base.py` symbols) to `from documents.serialisers._monolith import <Symbol>`, and change the 14 now-extracted symbols' import lines to `from documents.serialisers.base import <Symbol>`. Do the same in `src/paperless_mail/serialisers.py` for `OwnedObjectSerializer` (→ `documents.serialisers.base`); its other three imports (`CorrespondentField`, `DocumentTypeField`, `TagsField`) stay pointed at `documents.serialisers._monolith` until Task 2 moves them into `metadata.py`.
This `_monolith` re-pointing is scaffolding only — Task 2 finishes emptying and deletes `_monolith.py`, and every import that currently says `._monolith` gets its final home then.
- [ ] **Step 4: Run the full test suite for this app boundary**
```bash
bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests src/paperless_mail/tests -v"
```
Expected: PASS, no collection errors (a collection error here almost always means a missed import update in `views.py` or `paperless_mail/serialisers.py`).
- [ ] **Step 5: Commit**
```bash
git add src/documents/serialisers src/documents/views.py src/paperless_mail/serialisers.py
git commit -m "refactor: extract documents/serialisers/base.py from the serialisers monolith"
```
## Task 2: Extract the remaining 8 serialiser domain modules and delete the monolith
**Agent:** django-expert — **Model:** sonnet (repetitive but each of the 8 modules needs its own cross-reference check against `base.py` and previously-extracted siblings; DocumentSerializer in particular is large and central)
**Files:**
- Create: `src/documents/serialisers/metadata.py`, `src/documents/serialisers/documents.py`, `src/documents/serialisers/upload.py`, `src/documents/serialisers/saved_views.py`, `src/documents/serialisers/bulk_edit.py`, `src/documents/serialisers/sharing.py`, `src/documents/serialisers/tasks.py`, `src/documents/serialisers/workflows.py`, `src/documents/serialisers/system.py`
- Delete: `src/documents/serialisers/_monolith.py` (once empty)
- Modify: `src/documents/views.py` (finish re-pointing every `from documents.serialisers._monolith import X` line at the correct new submodule), `src/paperless_mail/serialisers.py` (re-point `CorrespondentField`, `DocumentTypeField`, `TagsField` at `documents.serialisers.metadata`)
- Test: `src/documents/tests/` (full app suite), `src/paperless_mail/tests/`
**Interfaces:**
- Consumes: `documents.serialisers.base` from Task 1 (relative import `.base` within the package).
- Produces: the full `documents/serialisers/` package as specified in the Reference map above — this is what Task 3/4 (views split) and Task 5 (external call sites) import from.
- [ ] **Step 1: Extract the 8 remaining domain modules in order**
Following the mechanical extraction recipe, and in this exact order (each may depend on symbols extracted earlier in this same order, plus anything in `base.py`):
1. `metadata.py``CorrespondentSerializer`, `DocumentTypeSerializer`, `DeprecatedColors`, `ColorField`, `TagSerializer`, `CorrespondentField`, `TagsField`, `DocumentTypeField`, `StoragePathField`, `StoragePathSerializer`, `StoragePathTestSerializer`, `CustomFieldSerializer`, `CustomFieldInstanceSerializer`, `validate_documentlink_targets`
2. `documents.py``DocumentSerializer`, `SearchResultListSerializer`, `SearchResultSerializer`, `DuplicateDocumentSummarySerializer`, `_DocumentVersionInfo`, `DocumentVersionInfoSerializer`, `DocumentVersionSerializer`, `DocumentVersionLabelSerializer`, `_get_viewable_duplicates`
3. `upload.py``PostDocumentSerializer`
4. `saved_views.py``SavedViewFilterRuleSerializer`, `SavedViewSerializer`
5. `bulk_edit.py``RotateDocumentsSerializer`, `MergeDocumentsSerializer`, `EditPdfDocumentsSerializer`, `RemovePasswordDocumentsSerializer`, `DeleteDocumentsSerializer`, `ReprocessDocumentsSerializer`, `BulkEditSerializer`, `BulkDownloadSerializer`, `BulkEditObjectsSerializer`
6. `sharing.py``EmailSerializer`, `ShareLinkSerializer`, `ShareLinkBundleSerializer`
7. `tasks.py``TaskSerializerV10`, `TaskSerializerV9`, `TaskSummarySerializer`, `RunTaskSerializer`, `AcknowledgeTasksViewSerializer`
8. `workflows.py``WorkflowTriggerSerializer`, `WorkflowActionEmailSerializer`, `WorkflowActionWebhookSerializer`, `WorkflowActionSerializer`, `WorkflowSerializer`
9. `system.py``UiSettingsViewSerializer`, `TrashSerializer`
After each individual module extraction, run the ruff fix-up from the recipe against that new file and `_monolith.py` before moving to the next module (don't batch all 8 and fix imports once at the end — F821 errors compound and get harder to attribute to the right module).
- [ ] **Step 2: Confirm the monolith is empty and delete it**
```bash
grep -n "^class |^def " src/documents/serialisers/_monolith.py
```
Expected: no output. If anything remains, it wasn't in the Reference map — stop and reconcile with the spec rather than deleting a symbol.
```bash
git rm src/documents/serialisers/_monolith.py
```
- [ ] **Step 3: Re-point every remaining `._monolith` import**
Search for any import left pointing at the now-deleted module:
```bash
grep -rn "serialisers\._monolith\|serialisers/_monolith" src/
```
Expected: no output. Fix any that remain by pointing them at the correct submodule per the Reference map (e.g. `from documents.serialisers._monolith import DocumentSerializer``from documents.serialisers.documents import DocumentSerializer`).
- [ ] **Step 4: Update `paperless_mail/serialisers.py`'s remaining imports**
```python
# was: from documents.serialisers import CorrespondentField, DocumentTypeField, OwnedObjectSerializer, TagsField
from documents.serialisers.base import OwnedObjectSerializer
from documents.serialisers.metadata import CorrespondentField, DocumentTypeField, TagsField
```
- [ ] **Step 5: Ruff and full test suite**
```bash
ruff check src/documents/serialisers src/documents/views.py src/paperless_mail/serialisers.py
ruff format src/documents/serialisers src/documents/views.py src/paperless_mail/serialisers.py
```
```bash
bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests src/paperless_mail/tests -v"
```
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add src/documents/serialisers src/documents/views.py src/paperless_mail/serialisers.py
git commit -m "refactor: finish splitting serialisers.py into documents/serialisers/"
```
## Task 3: Scaffold `documents/views/` and extract `base.py`
**Agent:** django-expert — **Model:** sonnet (same shape as Task 1, one level up — views/base.py is imported by every other view module)
**Files:**
- Create: `src/documents/views/__init__.py` (empty), `src/documents/views/base.py`
- Modify: `src/documents/views.py``src/documents/views/_monolith.py` (renamed, same reasoning as Task 1 Step 3)
- Modify: `src/paperless/urls.py`, `src/paperless_mail/views.py` (re-point the 1 symbol each currently pulls from `documents.views` that now lives in `base.py`, if any — see step 3)
- Test: `src/documents/tests/` (full app suite, includes URL-resolution-dependent tests), `src/paperless_mail/tests/`
**Interfaces:**
- Consumes: `documents.serialisers.*` submodules from Tasks 12 (already at final locations — import these directly, e.g. `from documents.serialisers.documents import DocumentSerializer`, never through a monolith or shim).
- Produces: `documents.views.base` exporting `PassUserMixin`, `BulkPermissionMixin`, `PermissionsAwareDocumentCountMixin`, `DocumentSelectionMixin`, `DocumentOperationPermissionMixin`, `SearchParams`, `SearchResultPage`, `ResolvedRequestDocs`, `_get_tantivy_query_and_mode`, `_get_more_like_id`, `serve_file`.
- [ ] **Step 1: Create the package directory, empty `__init__.py`, and rename the monolith**
```bash
mkdir -p src/documents/views
touch src/documents/views/__init__.py
git mv src/documents/views.py src/documents/views/_monolith.py
```
- [ ] **Step 2: Extract `base.py` per the mechanical extraction recipe**
Move exactly these 11 symbols out of `_monolith.py` into `views/base.py`: `PassUserMixin`, `BulkPermissionMixin`, `PermissionsAwareDocumentCountMixin`, `DocumentSelectionMixin`, `DocumentOperationPermissionMixin`, `SearchParams`, `SearchResultPage`, `ResolvedRequestDocs`, `_get_tantivy_query_and_mode`, `_get_more_like_id`, `serve_file`.
Within `_monolith.py`, every reference to these 11 symbols needs `from .base import <Symbol>` added (they're used throughout the rest of the file by the not-yet-extracted viewsets).
- [ ] **Step 3: Re-point external consumers of the now-moved symbol**
```bash
grep -n "from documents.views import PassUserMixin" src/paperless_mail/views.py
```
Update it to `from documents.views.base import PassUserMixin`.
`paperless/urls.py` doesn't import any of the 11 `base.py` symbols directly (it only imports viewsets/views, which are all still in `_monolith.py` at this point) — confirm with:
```bash
grep -nE "from documents\.views import (PassUserMixin|BulkPermissionMixin|PermissionsAwareDocumentCountMixin|DocumentSelectionMixin|DocumentOperationPermissionMixin|serve_file)" src/paperless/urls.py
```
Expected: no output. If something does match, re-point it at `documents.views.base` the same way.
- [ ] **Step 4: Ruff and test**
```bash
ruff check src/documents/views src/paperless_mail/views.py
ruff format src/documents/views src/paperless_mail/views.py
bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests src/paperless_mail/tests -v"
```
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add src/documents/views src/paperless_mail/views.py
git commit -m "refactor: extract documents/views/base.py from the views monolith"
```
## Task 4: Extract the remaining 13 view domain modules and delete the monolith
**Agent:** django-expert — **Model:** opus (highest blast radius in the plan — `DocumentViewSet` alone is ~1,300 lines and central to the whole API; this task also rewires `paperless/urls.py`'s ~34 import lines that drive URL routing for the entire backend, where a mistake breaks the app at startup, not just in one test)
**Files:**
- Create: `src/documents/views/index.py`, `src/documents/views/metadata.py`, `src/documents/views/documents.py`, `src/documents/views/upload.py`, `src/documents/views/chat.py`, `src/documents/views/search.py`, `src/documents/views/bulk_edit.py`, `src/documents/views/sharing.py`, `src/documents/views/saved_views.py`, `src/documents/views/tasks.py`, `src/documents/views/workflows.py`, `src/documents/views/system.py`, `src/documents/views/logs.py`
- Delete: `src/documents/views/_monolith.py` (once empty)
- Modify: `src/paperless/urls.py` (all ~34 `from documents.views import X` lines)
- Test: `src/documents/tests/` (full app suite — includes `test_views.py`, `test_api_documents.py`), `src/paperless_mail/tests/`
**Interfaces:**
- Consumes: `documents.serialisers.*` (Tasks 12) and `documents.views.base` (Task 3).
- Produces: the full `documents/views/` package as specified in the Reference map above.
- [ ] **Step 1: Extract the 13 remaining domain modules in order**
Following the mechanical extraction recipe, in this exact order:
1. `index.py``IndexView`, `serve_logo`
2. `metadata.py``CorrespondentViewSet`, `TagViewSet`, `DocumentTypeViewSet`, `StoragePathViewSet`, `CustomFieldViewSet`, `_get_llm_output_language`
3. `documents.py``EmailDocumentDetailSchema`, `DocumentViewSet`, `UnifiedSearchViewSet`
4. `upload.py``PostDocumentView`
5. `chat.py``ChatStreamingSerializer`, `ChatStreamingView`
6. `search.py``SearchAutoCompleteView`, `GlobalSearchView`, `SelectionDataView`, `StatisticsView`
7. `bulk_edit.py``BulkEditView`, `RotateDocumentsView`, `MergeDocumentsView`, `DeleteDocumentsView`, `ReprocessDocumentsView`, `EditPdfDocumentsView`, `RemovePasswordDocumentsView`, `BulkEditObjectsView`, `BulkDownloadView`
8. `sharing.py``ShareLinkViewSet`, `ShareLinkBundleViewSet`, `SharedLinkView`
9. `saved_views.py``SavedViewViewSet`
10. `tasks.py``_TasksViewSetSchema`, `TasksViewSet`
11. `workflows.py``WorkflowTriggerViewSet`, `WorkflowActionViewSet`, `WorkflowViewSet`
12. `system.py``UiSettingsView`, `RemoteVersionView`, `SystemStatusView`, `TrashView`
13. `logs.py``LogViewSet`
After each module, run the ruff fix-up from the recipe before continuing to the next (same rationale as Task 2 Step 1 — attribute F821s to the right module while context is fresh). `documents.py` is the biggest single extraction in this whole plan (`DocumentViewSet` is ~1,300 lines) — expect the most F821 fix-ups here, mostly resolved by adding `from documents.serialisers.documents import ...`, `from documents.serialisers.metadata import ...`, and `from .base import ...` as needed.
- [ ] **Step 2: Confirm the monolith is empty and delete it**
```bash
grep -n "^class |^def " src/documents/views/_monolith.py
```
Expected: no output.
```bash
git rm src/documents/views/_monolith.py
```
- [ ] **Step 3: Re-point every remaining `._monolith` import**
```bash
grep -rn "views\._monolith\|views/_monolith" src/
```
Expected: no output. Fix any stragglers per the Reference map.
- [ ] **Step 4: Update `paperless/urls.py`**
Replace each of the ~34 `from documents.views import X` lines with `from documents.views.<domain> import X` per the Reference map. For example:
```python
# was:
from documents.views import CorrespondentViewSet
from documents.views import WorkflowViewSet
from documents.views import serve_logo
# becomes:
from documents.views.metadata import CorrespondentViewSet
from documents.views.workflows import WorkflowViewSet
from documents.views.index import serve_logo
```
Do this for every import in that block — check off against the full symbol list in the Reference map above so none are missed.
- [ ] **Step 5: Ruff and test**
```bash
ruff check src/documents/views src/paperless/urls.py
ruff format src/documents/views src/paperless/urls.py
bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests src/paperless_mail/tests -v"
```
Expected: PASS, including `test_views.py` and `test_api_documents.py` — these exercise URL routing end-to-end, so a broken `urls.py` import shows up here as a collection error.
- [ ] **Step 6: Commit**
```bash
git add src/documents/views src/paperless/urls.py
git commit -m "refactor: finish splitting views.py into documents/views/"
```
## Task 5: Repo-wide verification sweep
**Agent:** general-purpose — **Model:** sonnet (an audit/verification pass: run targeted checks, read the output, fix anything found — moderate judgment, not novel design work)
**Files:**
- Modify: any file a grep in this task turns up beyond the ones already handled in Tasks 14 (expected: none, per the spec's stated blast radius of exactly `paperless/urls.py`, `paperless_mail/views.py`, `paperless_mail/serialisers.py` — this task exists to confirm that, not to find new work)
- Test: full backend suite (all apps, not just `documents`/`paperless_mail`)
**Interfaces:**
- Consumes: the finished `documents/views/` and `documents/serialisers/` packages from Tasks 14.
- [ ] **Step 1: Grep the whole repo for any remaining bare-module reference**
```bash
grep -rn "from documents\.views import\|from documents\.serialisers import\|documents\.views\.\_monolith\|documents\.serialisers\.\_monolith\|import documents\.views$\|import documents\.serialisers$" src/
```
Expected: no output. `documents/views/__init__.py` and `documents/serialisers/__init__.py` should still be empty (`0` bytes or a single blank line) — confirm with:
```bash
wc -l src/documents/views/__init__.py src/documents/serialisers/__init__.py
```
- [ ] **Step 2: Confirm import direction was never violated**
```bash
grep -rln "from documents\.views" src/documents/serialisers/
```
Expected: no output (no file in `serialisers/` imports from `views/`).
- [ ] **Step 3: Full ruff pass**
```bash
ruff check src/documents/views src/documents/serialisers src/paperless/urls.py src/paperless_mail/views.py src/paperless_mail/serialisers.py
ruff format --check src/documents/views src/documents/serialisers src/paperless/urls.py src/paperless_mail/views.py src/paperless_mail/serialisers.py
```
Expected: clean.
- [ ] **Step 4: Full backend test suite**
```bash
bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "-v"
```
(No path filter — this runs the whole backend suite, confirming nothing outside `documents`/`paperless_mail` was quietly relying on the old module shape, e.g. a management command or a script under `scripts/`.)
Expected: PASS.
- [ ] **Step 5: If Steps 14 found nothing to fix, commit is a no-op — skip it. If they found strays, fix and commit**
```bash
git add -A
git commit -m "refactor: fix stray documents.views/serialisers references found in repo sweep"
```
@@ -0,0 +1,158 @@
# Split `documents/views.py` and `documents/serialisers.py` into modules
## Problem
`src/documents/views.py` (5,395 lines) and `src/documents/serialisers.py`
(3,532 lines) have grown into monolithic files covering every REST resource
in the `documents` app: correspondents, tags, document types, storage paths,
custom fields, the core document viewset and search, chat, bulk-edit
operations, sharing, saved views, tasks, workflows, and system/UI settings.
Their size makes them hard to navigate, hard to review incrementally, and
increases the chance of unrelated changes colliding in the same file.
This document specifies splitting both files into packages, one module per
domain area, with no behavior change.
## Non-goals
- No behavior change. Class names, method bodies, and public API responses
are unchanged — this is a pure move/reorganize.
- No change to `test_views.py` or `test_api_documents.py`. They exercise the
moved classes via imports or via the live API; class names and behavior
don't change, so they need no edits. Splitting those test files is a
separate, later task if desired.
- No change to the frontend, migrations, or any other app beyond the three
files that import from `documents.views` / `documents.serialisers`
(`paperless/urls.py`, `paperless_mail/views.py`,
`paperless_mail/serialisers.py`).
- This work happens as its own branch/PR against `dev`, after the in-flight
`feature-ai-taxonomy-hints-v2` work merges — not layered on top of it.
## Architecture
`documents/views.py` becomes the package `documents/views/`, and
`documents/serialisers.py` becomes `documents/serialisers/`. Each gets one
module per domain area (table below). Neither package's `__init__.py`
re-exports its submodules' contents — it stays empty (or a short docstring
only). The three external call sites that currently do
`from documents.views import X` / `from documents.serialisers import X` are
updated to import from the specific submodule instead
(`from documents.views.workflows import WorkflowViewSet`, etc.). This avoids
adding an indirection layer that could quietly regrow into a second dumping
ground, at the cost of touching those three files.
### Import direction
`views/*` modules may import from `serialisers/*` modules; `serialisers/*`
modules never import from `views/*`. This keeps the dependency graph acyclic
by construction — there is no case in the current code where a serializer
needs a view.
Domain module names are the same across both packages (e.g. `bulk_edit.py`
exists in both), which makes the natural import `from documents.serialisers.bulk_edit import BulkEditSerializer`
inside `documents/views/bulk_edit.py` easy to find, but a view is free to
import a serializer from a different domain module when needed (e.g. a
`documents.py` view using a `metadata.py` field serializer) — that's a plain
cross-module import, not a cycle risk, since the reverse direction never
happens.
## Module breakdown — `documents/views/`
| Module | Contents |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `base.py` | Shared mixins/helpers: `PassUserMixin`, `BulkPermissionMixin`, `PermissionsAwareDocumentCountMixin`, `DocumentSelectionMixin`, `DocumentOperationPermissionMixin`, `SearchParams`/`SearchResultPage`/`ResolvedRequestDocs`, `_get_tantivy_query_and_mode`, `_get_more_like_id`, `serve_file` |
| `index.py` | `IndexView`, `serve_logo` |
| `metadata.py` | `CorrespondentViewSet`, `TagViewSet`, `DocumentTypeViewSet`, `StoragePathViewSet`, `CustomFieldViewSet`, `_get_llm_output_language` |
| `documents.py` | `EmailDocumentDetailSchema`, `DocumentViewSet`, `UnifiedSearchViewSet` |
| `upload.py` | `PostDocumentView` |
| `chat.py` | `ChatStreamingSerializer`, `ChatStreamingView` |
| `search.py` | `SearchAutoCompleteView`, `GlobalSearchView`, `SelectionDataView`, `StatisticsView` |
| `bulk_edit.py` | `BulkEditView`, `RotateDocumentsView`, `MergeDocumentsView`, `DeleteDocumentsView`, `ReprocessDocumentsView`, `EditPdfDocumentsView`, `RemovePasswordDocumentsView`, `BulkEditObjectsView`, `BulkDownloadView` |
| `sharing.py` | `ShareLinkViewSet`, `ShareLinkBundleViewSet`, `SharedLinkView` |
| `saved_views.py` | `SavedViewViewSet` |
| `tasks.py` | `_TasksViewSetSchema`, `TasksViewSet` |
| `workflows.py` | `WorkflowTriggerViewSet`, `WorkflowActionViewSet`, `WorkflowViewSet` |
| `system.py` | `UiSettingsView`, `RemoteVersionView`, `SystemStatusView`, `TrashView` |
| `logs.py` | `LogViewSet` |
`documents.py` remains the largest module at roughly 1,600 lines
(`DocumentViewSet` alone is ~1,300 lines in the current file); every other
module is well under 500 lines.
## Module breakdown — `documents/serialisers/`
| Module | Contents |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `base.py` | `DynamicFieldsModelSerializer`, `DocumentUpdateFieldsModelSerializer`, `MatchingModelSerializer`, `SetPermissionsMixin`, `SerializerWithPerms`, `SetPermissionsSerializer`, `OwnedObjectSerializer`, `OwnedObjectListSerializer`, `ReadWriteSerializerMethodField`, `DocumentListSerializer`, `DocumentSelectionSerializer`, `SourceModeValidationMixin`, `BasicUserSerializer`, `NotesSerializer` |
| `metadata.py` | `CorrespondentSerializer`, `DocumentTypeSerializer`, `DeprecatedColors`, `ColorField`, `TagSerializer`, `CorrespondentField`, `TagsField`, `DocumentTypeField`, `StoragePathField`, `StoragePathSerializer`, `StoragePathTestSerializer`, `CustomFieldSerializer`, `CustomFieldInstanceSerializer`, `validate_documentlink_targets` |
| `documents.py` | `DocumentSerializer`, `SearchResultListSerializer`, `SearchResultSerializer`, `DuplicateDocumentSummarySerializer`, `_DocumentVersionInfo`, `DocumentVersionInfoSerializer`, `DocumentVersionSerializer`, `DocumentVersionLabelSerializer`, `_get_viewable_duplicates` |
| `upload.py` | `PostDocumentSerializer` |
| `saved_views.py` | `SavedViewFilterRuleSerializer`, `SavedViewSerializer` |
| `bulk_edit.py` | `RotateDocumentsSerializer`, `MergeDocumentsSerializer`, `EditPdfDocumentsSerializer`, `RemovePasswordDocumentsSerializer`, `DeleteDocumentsSerializer`, `ReprocessDocumentsSerializer`, `BulkEditSerializer`, `BulkDownloadSerializer`, `BulkEditObjectsSerializer` |
| `sharing.py` | `EmailSerializer`, `ShareLinkSerializer`, `ShareLinkBundleSerializer` |
| `tasks.py` | `TaskSerializerV10`, `TaskSerializerV9`, `TaskSummarySerializer`, `RunTaskSerializer`, `AcknowledgeTasksViewSerializer` |
| `workflows.py` | `WorkflowTriggerSerializer`, `WorkflowActionEmailSerializer`, `WorkflowActionWebhookSerializer`, `WorkflowActionSerializer`, `WorkflowSerializer` |
| `system.py` | `UiSettingsViewSerializer`, `TrashSerializer` |
Note: `ChatStreamingSerializer` is defined in `views.py` today (not
`serialisers.py`), directly above `ChatStreamingView`. It moves with
`ChatStreamingView` into `documents/views/chat.py` rather than into the
serialisers package, preserving its current co-location.
## External call sites to update
Only three files import from these two modules today, and all move to
importing from the specific new submodule:
- `src/paperless/urls.py` — ~34 `from documents.views import X` lines, one
per viewset/view used in URL routing. Each becomes
`from documents.views.<domain> import X`.
- `src/paperless_mail/views.py``from documents.views import PassUserMixin`
becomes `from documents.views.base import PassUserMixin`.
- `src/paperless_mail/serialisers.py``CorrespondentField`,
`DocumentTypeField`, `OwnedObjectSerializer`, `TagsField` move to
`from documents.serialisers.metadata import CorrespondentField, DocumentTypeField, TagsField`
and `from documents.serialisers.base import OwnedObjectSerializer`.
## Migration order
1. Split `serialisers.py` into `documents/serialisers/` first — serializers
have no dependency on views, so this half can be verified in isolation.
Run the full backend test suite after this step.
2. Split `views.py` into `documents/views/`, importing from the new
`documents/serialisers/*` modules per the table above. Run the full
backend test suite.
3. Update the three external call sites (`paperless/urls.py`,
`paperless_mail/views.py`, `paperless_mail/serialisers.py`).
4. Run `ruff check` / `ruff format` and the full backend test suite once
more end to end.
Splitting serialisers before views (rather than in parallel) means step 2
can immediately import finished, correctly-located serializer modules
instead of guessing at not-yet-final paths.
## Risks / error handling
- **Circular imports**: prevented by construction (serialisers never import
from views — see Import direction above). If a genuine cross-domain need
is discovered during implementation that seems to require a
views→views import cycle (e.g. `UnifiedSearchViewSet` extending
`DocumentViewSet` from a different module — both already live in
`documents.py` so this doesn't arise), resolve it by moving the shared
piece to `base.py` rather than introducing a cycle.
- **Missed re-export consumers**: verified via a full-repo grep for
`from documents.views import` / `from documents.serialisers import` /
`documents.views.` / `documents.serialisers.` before considering the split
complete, in case something beyond the three known call sites appears
(e.g. in a management command or a rarely-run script).
- **Silent behavior drift during move**: since this is a pure reorganization,
the full test suite passing after each step (rather than only at the end)
is the primary safety net; no new tests are required for this refactor
itself.
## Testing
No new tests. Existing coverage (`test_views.py`, `test_api_documents.py`,
and the rest of the `documents` test suite) is run after each migration step
per the ordering above, and must pass unchanged — a failure indicates the
move altered behavior, not that new coverage is needed.
+3 -1
View File
@@ -576,7 +576,9 @@ The following workflow action types are available:
- Tags, correspondent, document type and storage path
- Document owner
- View and / or edit permissions to users or groups
- Custom fields. Note that no value for the field will be set
- Custom fields, optionally with a value. If no value is set, the field is only added to the
document and any value it may already have is left untouched. If a value is set, it will
overwrite an existing value of that field on the document.
##### Removal {#workflow-action-removal}
+544 -152
View File
File diff suppressed because it is too large Load Diff
@@ -576,7 +576,7 @@ describe('TasksComponent', () => {
expect(dismissSpy).toHaveBeenCalledWith(new Set([tasks[0].id, tasks[1].id]))
expect(toastSpy).toHaveBeenCalledWith('Error dismissing tasks', error)
expect(modal.componentInstance.buttonsEnabled).toBe(true)
expect(modal.componentInstance.buttonsEnabled()).toBe(true)
expect(component.selectedTasks.size).toBe(0)
})
@@ -642,7 +642,7 @@ describe('TasksComponent', () => {
expect(dismissSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalledWith('Error dismissing tasks', error)
expect(modal.componentInstance.buttonsEnabled).toBe(true)
expect(modal.componentInstance.buttonsEnabled()).toBe(true)
})
it('should dismiss the currently visible scoped and filtered tasks', () => {
@@ -316,7 +316,7 @@ export class TasksComponent
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Dismiss`
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
this.tasksService.dismissTasks(tasks).subscribe({
next: () => {
@@ -324,7 +324,7 @@ export class TasksComponent
},
error: (e) => {
this.toastService.showError($localize`Error dismissing tasks`, e)
modal.componentInstance.buttonsEnabled = true
modal.componentInstance.buttonsEnabled.set(true)
},
})
this.clearSelection()
@@ -350,7 +350,7 @@ export class TasksComponent
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Dismiss`
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
this.tasksService.dismissAllTasks().subscribe({
next: () => {
@@ -358,7 +358,7 @@ export class TasksComponent
},
error: (e) => {
this.toastService.showError($localize`Error dismissing tasks`, e)
modal.componentInstance.buttonsEnabled = true
modal.componentInstance.buttonsEnabled.set(true)
},
})
this.clearSelection()
@@ -82,7 +82,7 @@ export class TrashComponent
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.trashService.emptyTrash([document.id]).subscribe({
next: () => {
this.toastService.showInfo(
@@ -146,7 +146,7 @@ export class UsersAndGroupsComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.usersService.delete(user).subscribe({
next: () => {
modal.close()
@@ -199,7 +199,7 @@ export class UsersAndGroupsComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.groupsService.delete(group).subscribe({
next: () => {
modal.close()
@@ -111,7 +111,7 @@
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="view.name"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
popoverClass="popover-slim">
<i-bs class="me-2" name="funnel"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
@if (showSidebarCounts && !slimSidebarEnabled) {
<span class="badge bg-info text-dark ms-2 d-inline">{{ savedViewService.getDocumentCount(view) }}</span>
}
@@ -47,11 +47,12 @@
.search-container {
max-height: 4.5rem;
overflow: hidden;
overflow: visible;
transition: max-height .2s ease, opacity .2s ease, padding-top .2s ease, padding-bottom .2s ease;
&.mobile-hidden {
max-height: 0;
overflow: hidden;
opacity: 0;
padding-top: 0 !important;
padding-bottom: 0 !important;
@@ -12,10 +12,10 @@
}
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled()">
<span>
{{btnCaption}}
<span class="visually-hidden">{{ seconds | number: '1.0-0' }} seconds</span>
@@ -25,7 +25,7 @@
}
</button>
@if (alternativeBtnCaption) {
<button type="button" class="btn" [class]="alternativeBtnClass" (click)="alternative()" [disabled]="!alternativeButtonEnabled || !buttonsEnabled">
<button type="button" class="btn" [class]="alternativeBtnClass" (click)="alternative()" [disabled]="!alternativeButtonEnabled || !buttonsEnabled()">
{{alternativeBtnCaption}}
</button>
}
@@ -64,6 +64,22 @@ describe('ConfirmDialogComponent', () => {
expect(confirmSubjectResult).toBeTruthy()
})
it('should re-render the buttons when they are toggled from outside', async () => {
const confirmButton: HTMLButtonElement =
fixture.nativeElement.querySelectorAll('.modal-footer button')[1]
expect(confirmButton.disabled).toBeFalsy()
// Deliberately no detectChanges: a request callback toggling this is all
// that happens, and nothing else schedules a render for the modal
component.buttonsEnabled.set(false)
await fixture.whenStable()
expect(confirmButton.disabled).toBeTruthy()
component.buttonsEnabled.set(true)
await fixture.whenStable()
expect(confirmButton.disabled).toBeFalsy()
})
it('should support cancel & close modal', () => {
let confirmSubjectResult
const closeModalSpy = jest.spyOn(modal, 'close')
@@ -1,5 +1,12 @@
import { DecimalPipe } from '@angular/common'
import { Component, EventEmitter, Input, Output, inject } from '@angular/core'
import {
Component,
EventEmitter,
Input,
Output,
inject,
signal,
} from '@angular/core'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { Subject } from 'rxjs'
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
@@ -46,8 +53,7 @@ export class ConfirmDialogComponent extends LoadingComponentWithPermissions {
@Input()
cancelBtnCaption = $localize`Cancel`
@Input()
buttonsEnabled = true
readonly buttonsEnabled = signal(true)
confirmButtonEnabled = true
alternativeButtonEnabled = true
@@ -56,10 +56,10 @@
}
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled()">
{{btnCaption}}
</button>
</div>
@@ -57,7 +57,7 @@
class="btn"
[class]="cancelBtnClass"
(click)="cancel()"
[disabled]="!buttonsEnabled"
[disabled]="!buttonsEnabled()"
>
<span class="d-inline-block" style="padding-bottom: 1px;">
{{cancelBtnCaption}}
@@ -68,7 +68,7 @@
class="btn"
[class]="btnClass"
(click)="confirm()"
[disabled]="!confirmButtonEnabled || !buttonsEnabled"
[disabled]="!confirmButtonEnabled || !buttonsEnabled()"
>
{{btnCaption}}
</button>
@@ -34,10 +34,10 @@
<p class="mb-0 small"><b>{{messageBold}}</b></p>
}
</div>
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled || degrees === 0">
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled() || degrees === 0">
{{btnCaption}}
@if (!confirmButtonEnabled) {
<ngb-progressbar style="height: 1px;" type="dark" [max]="secondsTotal" [value]="seconds"></ngb-progressbar>
@@ -52,10 +52,10 @@ describe('CustomFieldsValuesComponent', () => {
})
it('should set selectedFields and map values correctly', () => {
component.value = { 1: 'value1' }
component.selectedFields = [1, 2]
expect(component.selectedFields).toEqual([1, 2])
expect(component.value).toEqual({ 1: 'value1', 2: null })
component.value = { 1: 'value1', 3: 0, 4: false }
component.selectedFields = [1, 2, 3, 4]
expect(component.selectedFields).toEqual([1, 2, 3, 4])
expect(component.value).toEqual({ 1: 'value1', 2: null, 3: 0, 4: false })
})
it('should return the correct custom field by id', () => {
@@ -77,7 +77,7 @@ export class CustomFieldsValuesComponent extends AbstractInputComponent<Object>
this._selectedFields = newFields
// map the selected fields to an object with field_id as key and value as value
this.value = newFields.reduce((acc, fieldId) => {
acc[fieldId] = this.value?.[fieldId] || null
acc[fieldId] = this.value?.[fieldId] ?? null
return acc
}, {})
this.onChange(this.value)
@@ -36,7 +36,16 @@
(focus)="clearLastSearchTerm()"
(clear)="clearLastSearchTerm()"
(blur)="onBlur()">
<ng-template ng-label-tmp let-item="item">
@if (iconField && item[iconField]) {
<i-bs class="me-2" [name]="item[iconField]"></i-bs>
}
<span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
</ng-template>
<ng-template ng-option-tmp let-item="item">
@if (iconField && item[iconField]) {
<i-bs class="me-2" [name]="item[iconField]"></i-bs>
}
<span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
</ng-template>
</ng-select>
@@ -35,7 +35,7 @@ import { AbstractInputComponent } from '../abstract-input'
NgxBootstrapIconsModule,
],
})
export class SelectComponent extends AbstractInputComponent<number> {
export class SelectComponent extends AbstractInputComponent<number | string> {
constructor() {
super()
this.addItemRef = this.addItem.bind(this)
@@ -100,6 +100,9 @@ export class SelectComponent extends AbstractInputComponent<number> {
@Input()
bindLabel: string = 'name'
@Input()
iconField: string
public searchFn = (term: string, item: any): boolean =>
matchesSearchText(item?.[this.bindLabel], term)
@@ -100,7 +100,7 @@
</div>
}
<div class="form-group ms-md-auto">
<button type="button" class="btn me-2" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">{{ cancelBtnCaption }}</button>
<button type="button" class="btn me-2" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">{{ cancelBtnCaption }}</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="pages.length === 0">{{ btnCaption }}</button>
</div>
</div>
@@ -17,6 +17,10 @@ const permissions = [
'view_document',
'change_document',
'delete_document',
'add_sharelinkbundle',
'view_sharelinkbundle',
'change_sharelinkbundle',
'delete_sharelinkbundle',
'change_tag',
'view_documenttype',
]
@@ -75,6 +79,7 @@ describe('PermissionsSelectComponent', () => {
component.ngOnInit()
component.writeValue(permissions)
expect(component.typesWithAllActions).toContain('Document')
expect(component.typesWithAllActions).toContain('ShareLinkBundle')
})
it('should update checkboxes on permissions set', () => {
@@ -85,6 +90,10 @@ describe('PermissionsSelectComponent', () => {
expect(input1.nativeElement.checked).toBeTruthy()
const input2 = fixture.debugElement.query(By.css('input#Tag_Change'))
expect(input2.nativeElement.checked).toBeTruthy()
const bundleInput = fixture.debugElement.query(
By.css('input#ShareLinkBundle_Add')
)
expect(bundleInput.nativeElement.checked).toBeTruthy()
})
it('disable checkboxes when permissions are inherited', () => {
@@ -119,7 +119,7 @@
type="button"
class="btn btn-primary btn-sm d-inline-flex align-items-center gap-2 text-nowrap"
(click)="submit()"
[disabled]="loading() || !buttonsEnabled">
[disabled]="loading() || !buttonsEnabled()">
@if (loading()) {
<span class="spinner-border spinner-border-sm" role="status" aria-hidden="true"></span>
}
@@ -69,7 +69,7 @@ describe('ShareLinkBundleDialogComponent', () => {
file_version: FileVersion.Original,
expiration_days: 3,
})
expect(component.buttonsEnabled).toBe(false)
expect(component.buttonsEnabled()).toBe(false)
expect(confirmSpy).toHaveBeenCalled()
component.form.setValue({
@@ -78,7 +78,7 @@ export class ShareLinkBundleDialogComponent extends ConfirmDialogComponent {
: FileVersion.Original,
expiration_days: this.form.value.expirationDays,
}
this.buttonsEnabled = false
this.buttonsEnabled.set(false)
super.confirm()
}
@@ -1,6 +1,7 @@
<pngx-widget-frame
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Document }"
[title]="savedView.name"
[titleIcon]="savedView.icon || 'funnel'"
[loading]="false"
[draggable]="savedView"
>
@@ -8,7 +8,12 @@
<i-bs name="grip-vertical"></i-bs>
</div>
}
<h6 class="card-title mb-0">{{title()}}</h6>
<h6 class="card-title mb-0">
@if (titleIcon()) {
<i-bs class="me-2" [name]="titleIcon()"></i-bs>
}
{{title()}}
</h6>
<ng-content select="[title-badge]"></ng-content>
@if (badge() !== null && badge() !== undefined) {
<span class="badge bg-info text-dark ms-2">{{badge()}}</span>
@@ -16,6 +16,8 @@ export class WidgetFrameComponent implements AfterViewInit {
title = input<string>()
titleIcon = input<string>()
draggable = input<any>()
cardless = input(false)
@@ -1564,7 +1564,7 @@ describe('DocumentDetailComponent', () => {
dialog.confirmClicked.next()
await openModal.result
expect(dialog.buttonsEnabled).toBe(false)
expect(dialog.buttonsEnabled()).toBe(false)
expect(reloadSpy).toHaveBeenCalled()
expect((component as any).incomingUpdateModal).toBeNull()
})
@@ -1789,7 +1789,7 @@ describe('DocumentDetailComponent', () => {
expect(errorSpy).toHaveBeenCalled()
expect(component.networkActive()).toBe(false)
expect(dialog.buttonsEnabled).toBe(true)
expect(dialog.buttonsEnabled()).toBe(true)
})
it('should refresh the document when removing password in update mode', () => {
@@ -659,7 +659,7 @@ export class DocumentDetailComponent
modal.componentInstance.cancelBtnCaption = $localize`Dismiss`
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
this.reloadRemoteVersion()
})
@@ -1374,7 +1374,7 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked
.pipe(
switchMap(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
return this.documentsService.delete(this.document())
})
)
@@ -1386,7 +1386,7 @@ export class DocumentDetailComponent
},
error: (error) => {
this.toastService.showError($localize`Error deleting document`, error)
modal.componentInstance.buttonsEnabled = true
modal.componentInstance.buttonsEnabled.set(true)
this.subscribeModalDelete(modal)
},
})
@@ -1411,7 +1411,7 @@ export class DocumentDetailComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.documentsService
.reprocessDocuments({ documents: [this.document().id] })
.subscribe({
@@ -1425,7 +1425,7 @@ export class DocumentDetailComponent
},
error: (error) => {
if (modal) {
modal.componentInstance.buttonsEnabled = true
modal.componentInstance.buttonsEnabled.set(true)
}
this.toastService.showError(
$localize`Error executing operation`,
@@ -1798,7 +1798,7 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.documentsService
.editPdfDocuments([sourceDocumentId], {
operations: modal.componentInstance.getOperations(),
@@ -1821,7 +1821,7 @@ export class DocumentDetailComponent
},
error: (error) => {
if (modal) {
modal.componentInstance.buttonsEnabled = true
modal.componentInstance.buttonsEnabled.set(true)
}
this.toastService.showError(
$localize`Error executing PDF edit operation`,
@@ -1855,7 +1855,7 @@ export class DocumentDetailComponent
const sourceDocumentId = this.selectedVersionId() ?? this.document().id
const dialog =
modal.componentInstance as PasswordRemovalConfirmDialogComponent
dialog.buttonsEnabled = false
dialog.buttonsEnabled.set(false)
this.networkActive.set(true)
this.documentsService
.removePasswordDocuments([sourceDocumentId], {
@@ -1880,7 +1880,7 @@ export class DocumentDetailComponent
}
},
error: (error) => {
dialog.buttonsEnabled = true
dialog.buttonsEnabled.set(true)
this.networkActive.set(false)
this.toastService.showError(
$localize`Error executing password removal operation`,
@@ -1683,7 +1683,7 @@ describe('BulkEditorComponent', () => {
expiration_days: 7,
},
loading: signal(false),
buttonsEnabled: true,
buttonsEnabled: signal(true),
copied: signal(false),
},
}
@@ -1715,7 +1715,7 @@ describe('BulkEditorComponent', () => {
expiration_days: 7,
})
expect(dialogInstance.loading()).toBe(false)
expect(dialogInstance.buttonsEnabled).toBe(false)
expect(dialogInstance.buttonsEnabled()).toBe(false)
expect(dialogInstance.createdBundle).toEqual({ id: 42 })
expect(typeof dialogInstance.onOpenManage).toBe('function')
expect(toastInfoSpy).toHaveBeenCalledWith(
@@ -1755,7 +1755,7 @@ describe('BulkEditorComponent', () => {
expiration_days: null,
},
loading: signal(false),
buttonsEnabled: true,
buttonsEnabled: signal(true),
},
}
@@ -1777,7 +1777,7 @@ describe('BulkEditorComponent', () => {
expect.any(Error)
)
expect(dialogInstance.loading()).toBe(false)
expect(dialogInstance.buttonsEnabled).toBe(true)
expect(dialogInstance.buttonsEnabled()).toBe(true)
openSpy.mockRestore()
})
@@ -273,7 +273,7 @@ export class BulkEditorComponent
overrideSelection?: DocumentSelectionQuery
) {
if (modal) {
this.setModalButtonsEnabled(modal, false)
modal.componentInstance.buttonsEnabled.set(false)
}
this.documentService
.bulkEdit(overrideSelection ?? this.getSelectionQuery(), method, args)
@@ -290,7 +290,7 @@ export class BulkEditorComponent
options: { deleteOriginals?: boolean } = {}
) {
if (modal) {
this.setModalButtonsEnabled(modal, false)
modal.componentInstance.buttonsEnabled.set(false)
}
request.pipe(first()).subscribe({
next: () => {
@@ -320,7 +320,7 @@ export class BulkEditorComponent
private handleOperationError(modal: NgbModalRef, error: any) {
if (modal) {
this.setModalButtonsEnabled(modal, true)
modal.componentInstance.buttonsEnabled.set(true)
}
this.toastService.showError(
$localize`Error executing bulk operation`,
@@ -328,15 +328,6 @@ export class BulkEditorComponent
)
}
private setModalButtonsEnabled(modal: NgbModalRef, enabled: boolean) {
const buttonsEnabled = modal.componentInstance.buttonsEnabled
if (typeof buttonsEnabled?.set === 'function') {
buttonsEnabled.set(enabled)
} else {
modal.componentInstance.buttonsEnabled = enabled
}
}
private applySelectionData(
items: SelectionDataItem[],
selectionModel: FilterableDropdownSelectionModel
@@ -872,7 +863,7 @@ export class BulkEditorComponent
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.deleteDocuments(this.getSelectionQuery())
@@ -920,7 +911,7 @@ export class BulkEditorComponent
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.reprocessDocuments(this.getSelectionQuery())
@@ -957,7 +948,7 @@ export class BulkEditorComponent
rotateDialog.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
rotateDialog.buttonsEnabled = false
rotateDialog.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.rotateDocuments(
@@ -990,7 +981,7 @@ export class BulkEditorComponent
if (mergeDialog.archiveFallback()) {
args.archive_fallback = true
}
mergeDialog.buttonsEnabled = false
mergeDialog.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.mergeDocuments(mergeDialog.documentIDs(), args),
@@ -1063,14 +1054,14 @@ export class BulkEditorComponent
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
dialog.loading.set(true)
dialog.buttonsEnabled = false
dialog.buttonsEnabled.set(false)
this.shareLinkBundleService
.createBundle(dialog.payload)
.pipe(first())
.subscribe({
next: (result) => {
dialog.loading.set(false)
dialog.buttonsEnabled = false
dialog.buttonsEnabled.set(false)
dialog.createdBundle = result
dialog.copied.set(false)
dialog.payload = null
@@ -1084,7 +1075,7 @@ export class BulkEditorComponent
},
error: (error) => {
dialog.loading.set(false)
dialog.buttonsEnabled = true
dialog.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Share link bundle creation is not available yet.`,
error
@@ -97,7 +97,9 @@
<div class="dropdown-menu shadow dropdown-menu-right" ngbDropdownMenu>
@if (!list.activeSavedViewId) {
@for (view of savedViewService.allViews; track view) {
<button ngbDropdownItem (click)="loadViewConfig(view.id)">{{view.name}}</button>
<button ngbDropdownItem (click)="loadViewConfig(view.id)">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs>{{view.name}}
</button>
}
@if (savedViewService.allViews.length > 0) {
<div class="dropdown-divider"></div>
@@ -457,6 +457,7 @@ export class DocumentListComponent
modal.componentInstance.buttonsEnabled.set(false)
let savedView: SavedView = {
name: formValue.name,
icon: formValue.icon,
filter_rules: this.list.filterRules,
sort_reverse: this.list.sortReverse,
sort_field: this.list.sortField,
@@ -6,6 +6,14 @@
</div>
<div class="modal-body">
<pngx-input-text i18n-title title="Name" formControlName="name" [error]="error()?.name" autocomplete="off"></pngx-input-text>
<pngx-input-select
i18n-title
title="Icon"
formControlName="icon"
[items]="savedViewIcons"
iconField="icon"
[error]="error()?.icon">
</pngx-input-select>
<pngx-input-check i18n-title title="Show in sidebar" formControlName="showInSideBar"></pngx-input-check>
<pngx-input-check i18n-title title="Show on dashboard" formControlName="showOnDashboard"></pngx-input-check>
<pngx-permissions-form accordion="true" formControlName="permissions_form"></pngx-permissions-form>
@@ -9,6 +9,7 @@ import { CheckComponent } from '../../common/input/check/check.component'
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
import { PermissionsGroupComponent } from '../../common/input/permissions/permissions-group/permissions-group.component'
import { PermissionsUserComponent } from '../../common/input/permissions/permissions-user/permissions-user.component'
import { SelectComponent } from '../../common/input/select/select.component'
import { TextComponent } from '../../common/input/text/text.component'
import { SaveViewConfigDialogComponent } from './save-view-config-dialog.component'
@@ -40,6 +41,7 @@ describe('SaveViewConfigDialogComponent', () => {
ReactiveFormsModule,
SaveViewConfigDialogComponent,
TextComponent,
SelectComponent,
CheckComponent,
PermissionsFormComponent,
PermissionsUserComponent,
@@ -63,6 +65,7 @@ describe('SaveViewConfigDialogComponent', () => {
expect(component.defaultName()).toEqual(name)
expect(result).toEqual({
name,
icon: 'funnel',
showInSideBar: false,
showOnDashboard: false,
})
@@ -94,6 +97,7 @@ describe('SaveViewConfigDialogComponent', () => {
component.save()
expect(result).toEqual({
name,
icon: 'funnel',
showInSideBar: true,
showOnDashboard: true,
})
@@ -113,6 +117,7 @@ describe('SaveViewConfigDialogComponent', () => {
component.save()
expect(result).toEqual({
name: '',
icon: 'funnel',
showInSideBar: false,
showOnDashboard: false,
permissions_form: permissions,
@@ -13,9 +13,14 @@ import {
ReactiveFormsModule,
} from '@angular/forms'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import {
DEFAULT_SAVED_VIEW_ICON,
SAVED_VIEW_ICONS,
} from 'src/app/data/saved-view-icons'
import { User } from 'src/app/data/user'
import { CheckComponent } from '../../common/input/check/check.component'
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
import { SelectComponent } from '../../common/input/select/select.component'
import { TextComponent } from '../../common/input/text/text.component'
@Component({
@@ -24,6 +29,7 @@ import { TextComponent } from '../../common/input/text/text.component'
styleUrls: ['./save-view-config-dialog.component.scss'],
imports: [
CheckComponent,
SelectComponent,
TextComponent,
PermissionsFormComponent,
FormsModule,
@@ -41,6 +47,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
public saveClicked = new EventEmitter()
users: User[]
readonly savedViewIcons = SAVED_VIEW_ICONS
setDefaultName(value: string) {
this.defaultName.set(value)
@@ -49,6 +56,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
saveViewConfigForm = new FormGroup({
name: new FormControl(''),
icon: new FormControl(DEFAULT_SAVED_VIEW_ICON),
showInSideBar: new FormControl(false),
showOnDashboard: new FormControl(false),
permissions_form: new FormControl(null),
@@ -65,6 +73,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
const formValue = this.saveViewConfigForm.value
const saveViewConfig = {
name: formValue.name,
icon: formValue.icon,
showInSideBar: formValue.showInSideBar,
showOnDashboard: formValue.showOnDashboard,
}
@@ -105,7 +105,7 @@ export class CustomFieldsComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.customFieldsService.delete(field).subscribe({
next: () => {
modal.close()
@@ -274,7 +274,7 @@ export abstract class ManagementListComponent<T extends MatchingModel>
activeModal.componentInstance.btnClass = 'btn-danger'
activeModal.componentInstance.btnCaption = $localize`Delete`
activeModal.componentInstance.confirmClicked.subscribe(() => {
activeModal.componentInstance.buttonsEnabled = false
activeModal.componentInstance.buttonsEnabled.set(false)
this.service
.delete(object)
.pipe(takeUntil(this.unsubscribeNotifier))
@@ -284,7 +284,7 @@ export abstract class ManagementListComponent<T extends MatchingModel>
this.reloadData()
},
error: (error) => {
activeModal.componentInstance.buttonsEnabled = true
activeModal.componentInstance.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Error while deleting element`,
error
@@ -455,7 +455,7 @@ export abstract class ManagementListComponent<T extends MatchingModel>
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.service
.bulk_edit_objects(
this.allSelectionActive ? [] : Array.from(this.selectedObjects),
@@ -472,7 +472,7 @@ export abstract class ManagementListComponent<T extends MatchingModel>
this.reloadData()
},
error: (error) => {
modal.componentInstance.buttonsEnabled = true
modal.componentInstance.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Error deleting objects`,
error
@@ -196,7 +196,7 @@ export class MailComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.mailAccountService.delete(account).subscribe({
next: () => {
modal.close()
@@ -298,7 +298,7 @@ export class MailComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.mailRuleService.delete(rule).subscribe({
next: () => {
modal.close()
@@ -7,15 +7,24 @@
</pngx-page-header>
<form [formGroup]="savedViewsForm" (ngSubmit)="save()">
<ul class="list-group mb-3" formGroupName="savedViews">
@for (view of savedViews(); track view) {
@for (view of pagedSavedViews(); track view) {
<li class="list-group-item py-3">
<div [formGroupName]="view.id">
<div class="row">
<div class="col">
<div class="col-md">
<pngx-input-text title="Name" formControlName="name"></pngx-input-text>
</div>
<div class="col-md">
<pngx-input-select
i18n-title
title="Icon"
formControlName="icon"
[items]="savedViewIcons"
iconField="icon">
</pngx-input-select>
</div>
@if (canSaveSettings) {
<div class="col">
<div class="col-md">
<div class="form-check form-switch mt-3">
<input type="checkbox" class="form-check-input" id="show_on_dashboard_{{view.id}}" formControlName="show_on_dashboard">
<label class="form-check-label" for="show_on_dashboard_{{view.id}}" i18n>Show on dashboard</label>
@@ -81,6 +90,11 @@
}
</ul>
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
<div class="d-flex align-items-center mb-3">
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
@if (savedViews()?.length > pageSize) {
<ngb-pagination class="ms-auto" [pageSize]="pageSize" [collectionSize]="savedViews().length" [page]="page()" [maxSize]="5" (pageChange)="page.set($event)" size="sm" aria-label="Pagination"></ngb-pagination>
}
</div>
</form>
@@ -4,6 +4,7 @@ import { provideHttpClientTesting } from '@angular/common/http/testing'
import { signal } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { NgbModal, NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { Subject, of, throwError } from 'rxjs'
@@ -25,8 +26,20 @@ import { PageHeaderComponent } from '../../common/page-header/page-header.compon
import { SavedViewsComponent } from './saved-views.component'
const savedViews = [
{ id: 1, name: 'view1', show_in_sidebar: true, show_on_dashboard: true },
{ id: 2, name: 'view2', show_in_sidebar: false, show_on_dashboard: false },
{
id: 1,
name: 'view1',
icon: 'archive',
show_in_sidebar: true,
show_on_dashboard: true,
},
{
id: 2,
name: 'view2',
icon: 'funnel',
show_in_sidebar: false,
show_on_dashboard: false,
},
]
describe('SavedViewsComponent', () => {
@@ -157,6 +170,24 @@ describe('SavedViewsComponent', () => {
expect(patchBody.show_in_sidebar).toBeUndefined()
})
it('should persist a changed icon', () => {
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
const view = savedViews[0]
const iconControl = component.savedViewsForm
.get('savedViews')
.get(view.id.toString())
.get('icon')
iconControl.setValue('bell')
iconControl.markAsDirty()
component.save()
expect(patchSpy.mock.calls[0][0][0]).toMatchObject({
id: view.id,
icon: 'bell',
})
})
it('should persist visibility changes to user settings', () => {
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
const updateVisibilitySpy = jest
@@ -222,6 +253,44 @@ describe('SavedViewsComponent', () => {
).toEqual(view.show_on_dashboard)
})
it('should page saved views, clamp the page if views are removed', () => {
const manyViews = Array.from({ length: 30 }, (_, i) => ({
id: i + 1,
name: `view${i + 1}`,
})) as SavedView[]
const listSpy = jest.spyOn(savedViewService, 'list').mockReturnValue(
of({
all: manyViews.map((v) => v.id),
count: manyViews.length,
results: manyViews.concat([]),
})
)
component.ngOnInit()
fixture.detectChanges()
expect(listSpy).toHaveBeenCalledWith(1, 100000, null, false, {
full_perms: true,
})
expect(component.pagedSavedViews()).toHaveLength(25)
expect(fixture.debugElement.query(By.css('ngb-pagination'))).not.toBeNull()
// all views have controls, not just the current page
expect(
Object.keys(component.savedViewsForm.get('savedViews').value)
).toHaveLength(30)
component.page.set(2)
expect(component.pagedSavedViews()).toHaveLength(5)
listSpy.mockReturnValue(
of({
all: manyViews.slice(0, 25).map((v) => v.id),
count: 25,
results: manyViews.slice(0, 25),
})
)
component.ngOnInit()
expect(component.page()).toEqual(1)
})
it('should support editing permissions', () => {
const confirmClicked = new Subject<any>()
const modalRef = {
@@ -1,18 +1,29 @@
import { AsyncPipe } from '@angular/common'
import { Component, OnDestroy, OnInit, inject, signal } from '@angular/core'
import {
Component,
OnDestroy,
OnInit,
computed,
inject,
signal,
} from '@angular/core'
import {
FormControl,
FormGroup,
FormsModule,
ReactiveFormsModule,
} from '@angular/forms'
import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { NgbModal, NgbPaginationModule } from '@ng-bootstrap/ng-bootstrap'
import { dirtyCheck } from '@ngneat/dirty-check-forms'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { BehaviorSubject, Observable, of, switchMap, takeUntil } from 'rxjs'
import { PermissionsDialogComponent } from 'src/app/components/common/permissions-dialog/permissions-dialog.component'
import { DisplayMode } from 'src/app/data/document'
import { SavedView } from 'src/app/data/saved-view'
import {
DEFAULT_SAVED_VIEW_ICON,
SAVED_VIEW_ICONS,
} from 'src/app/data/saved-view-icons'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import {
PermissionAction,
@@ -25,6 +36,7 @@ import { ToastService } from 'src/app/services/toast.service'
import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-button.component'
import { DragDropSelectComponent } from '../../common/input/drag-drop-select/drag-drop-select.component'
import { NumberComponent } from '../../common/input/number/number.component'
import { SelectComponent } from '../../common/input/select/select.component'
import { TextComponent } from '../../common/input/text/text.component'
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
@@ -36,12 +48,14 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
PageHeaderComponent,
ConfirmButtonComponent,
NumberComponent,
SelectComponent,
TextComponent,
IfPermissionsDirective,
DragDropSelectComponent,
FormsModule,
ReactiveFormsModule,
AsyncPipe,
NgbPaginationModule,
NgxBootstrapIconsModule,
],
})
@@ -56,8 +70,17 @@ export class SavedViewsComponent
private readonly modalService = inject(NgbModal)
DisplayMode = DisplayMode
readonly savedViewIcons = SAVED_VIEW_ICONS
readonly savedViews = signal<SavedView[]>(undefined)
readonly page = signal(1)
public readonly pageSize = 25
// All views are loaded at init, so paging is only for display
readonly pagedSavedViews = computed(() => {
const start = (this.page() - 1) * this.pageSize
return this.savedViews()?.slice(start, start + this.pageSize)
})
private savedViewsGroup = new FormGroup({})
public savedViewsForm: FormGroup = new FormGroup({
savedViews: this.savedViewsGroup,
@@ -84,9 +107,11 @@ export class SavedViewsComponent
private reloadViews(): void {
this.loading.set(true)
this.savedViewService
.list(null, null, null, false, { full_perms: true })
.list(1, 100000, null, false, { full_perms: true })
.subscribe((r) => {
this.savedViews.set(r.results)
const pageCount = Math.ceil(r.results.length / this.pageSize)
this.page.update((page) => Math.min(page, Math.max(1, pageCount)))
this.initialize()
})
}
@@ -110,6 +135,7 @@ export class SavedViewsComponent
storeData.savedViews[view.id.toString()] = {
id: view.id,
name: view.name,
icon: view.icon ?? DEFAULT_SAVED_VIEW_ICON,
show_on_dashboard: view.show_on_dashboard,
show_in_sidebar: view.show_in_sidebar,
page_size: view.page_size,
@@ -122,6 +148,7 @@ export class SavedViewsComponent
new FormGroup({
id: new FormControl({ value: null, disabled: !canEdit }),
name: new FormControl({ value: null, disabled: !canEdit }),
icon: new FormControl({ value: null, disabled: !canEdit }),
show_on_dashboard: new FormControl({
value: null,
disabled: false,
@@ -200,6 +227,7 @@ export class SavedViewsComponent
const modelFieldsChanged =
group.get('name')?.dirty ||
group.get('icon')?.dirty ||
group.get('page_size')?.dirty ||
group.get('display_mode')?.dirty ||
group.get('display_fields')?.dirty
@@ -134,7 +134,7 @@ export class WorkflowsComponent
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
this.workflowService.delete(workflow).subscribe({
next: () => {
modal.close()
+89
View File
@@ -0,0 +1,89 @@
export const DEFAULT_SAVED_VIEW_ICON = 'funnel'
export const SAVED_VIEW_ICONS = [
{ id: 'archive', name: $localize`Archive`, icon: 'archive' },
{ id: 'bank', name: $localize`Bank`, icon: 'bank' },
{ id: 'basket', name: $localize`Basket`, icon: 'basket' },
{ id: 'bell', name: $localize`Bell`, icon: 'bell' },
{ id: 'bookmark', name: $localize`Bookmark`, icon: 'bookmark' },
{ id: 'boxes', name: $localize`Boxes`, icon: 'boxes' },
{ id: 'briefcase', name: $localize`Briefcase`, icon: 'briefcase' },
{ id: 'building', name: $localize`Building`, icon: 'building' },
{ id: 'calculator', name: $localize`Calculator`, icon: 'calculator' },
{ id: 'calendar', name: $localize`Calendar`, icon: 'calendar' },
{ id: 'camera', name: $localize`Camera`, icon: 'camera' },
{
id: 'card-checklist',
name: $localize`Checklist`,
icon: 'card-checklist',
},
{ id: 'cash', name: $localize`Cash`, icon: 'cash' },
{ id: 'chat-left-text', name: $localize`Chat`, icon: 'chat-left-text' },
{ id: 'check-circle', name: $localize`Check`, icon: 'check-circle' },
{ id: 'clipboard', name: $localize`Clipboard`, icon: 'clipboard' },
{ id: 'clock-history', name: $localize`Clock`, icon: 'clock-history' },
{ id: 'credit-card', name: $localize`Credit card`, icon: 'credit-card' },
{ id: 'download', name: $localize`Download`, icon: 'download' },
{ id: 'envelope', name: $localize`Envelope`, icon: 'envelope' },
{
id: 'exclamation-triangle',
name: $localize`Warning`,
icon: 'exclamation-triangle',
},
{ id: 'file-earmark', name: $localize`File`, icon: 'file-earmark' },
{
id: 'file-earmark-check',
name: $localize`Checked file`,
icon: 'file-earmark-check',
},
{
id: 'file-earmark-lock',
name: $localize`Locked file`,
icon: 'file-earmark-lock',
},
{
id: 'file-earmark-medical',
name: $localize`Medical file`,
icon: 'file-earmark-medical',
},
{
id: 'file-earmark-person',
name: $localize`Person file`,
icon: 'file-earmark-person',
},
{
id: 'file-earmark-spreadsheet',
name: $localize`Spreadsheet`,
icon: 'file-earmark-spreadsheet',
},
{ id: 'file-text', name: $localize`Text file`, icon: 'file-text' },
{ id: 'files', name: $localize`Files`, icon: 'files' },
{ id: 'folder', name: $localize`Folder`, icon: 'folder' },
{ id: 'funnel', name: $localize`Filter`, icon: 'funnel' },
{ id: 'gear', name: $localize`Gear`, icon: 'gear' },
{ id: 'globe2', name: $localize`Globe`, icon: 'globe2' },
{ id: 'hash', name: $localize`Hash`, icon: 'hash' },
{ id: 'heart', name: $localize`Heart`, icon: 'heart' },
{ id: 'house', name: $localize`House`, icon: 'house' },
{ id: 'inbox', name: $localize`Inbox`, icon: 'inbox' },
{ id: 'journals', name: $localize`Journals`, icon: 'journals' },
{ id: 'list-task', name: $localize`Task list`, icon: 'list-task' },
{ id: 'newspaper', name: $localize`Newspaper`, icon: 'newspaper' },
{ id: 'paperclip', name: $localize`Attachment`, icon: 'paperclip' },
{ id: 'people', name: $localize`People`, icon: 'people' },
{ id: 'person', name: $localize`Person`, icon: 'person' },
{ id: 'printer', name: $localize`Printer`, icon: 'printer' },
{ id: 'receipt', name: $localize`Receipt`, icon: 'receipt' },
{ id: 'safe', name: $localize`Safe`, icon: 'safe' },
{ id: 'search', name: $localize`Search`, icon: 'search' },
{ id: 'send', name: $localize`Send`, icon: 'send' },
{ id: 'shop', name: $localize`Shop`, icon: 'shop' },
{ id: 'stack', name: $localize`Stack`, icon: 'stack' },
{ id: 'stars', name: $localize`Stars`, icon: 'stars' },
{ id: 'tag', name: $localize`Tag`, icon: 'tag' },
{ id: 'tags', name: $localize`Tags`, icon: 'tags' },
{ id: 'telephone', name: $localize`Telephone`, icon: 'telephone' },
{ id: 'truck', name: $localize`Truck`, icon: 'truck' },
{ id: 'upc-scan', name: $localize`Barcode`, icon: 'upc-scan' },
{ id: 'wallet2', name: $localize`Wallet`, icon: 'wallet2' },
]
+2
View File
@@ -5,6 +5,8 @@ import { ObjectWithPermissions } from './object-with-permissions'
export interface SavedView extends ObjectWithPermissions {
name?: string
icon?: string
show_on_dashboard?: boolean
show_in_sidebar?: boolean
+1 -1
View File
@@ -18,7 +18,7 @@ export class DirtyFormGuard extends DirtyCheckGuard {
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Leave page`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
})
const subject = new Subject<boolean>()
@@ -36,12 +36,12 @@ export class DirtySavedViewGuard {
modal.componentInstance.alternativeBtnClass = 'btn-primary'
modal.componentInstance.alternativeBtnCaption = $localize`Save and close`
modal.componentInstance.alternativeClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
component.saveViewConfig()
modal.close()
})
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
})
@@ -142,7 +142,7 @@ export class OpenDocumentsService {
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Close document`
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
this.openDocuments.splice(index, 1)
this.dirtyDocuments.delete(doc.id)
@@ -165,7 +165,7 @@ export class OpenDocumentsService {
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Close documents`
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.componentInstance.buttonsEnabled.set(false)
modal.close()
this.openDocuments.splice(0, this.openDocuments.length)
this.dirtyDocuments.clear()
@@ -120,6 +120,12 @@ describe('PermissionsService', () => {
actionKey: 'View', // PermissionAction.View
typeKey: 'SystemMonitoring', // PermissionType.SystemMonitoring
})
expect(permissionsService.getPermissionKeys('add_sharelinkbundle')).toEqual(
{
actionKey: 'Add', // PermissionAction.Add
typeKey: 'ShareLinkBundle', // PermissionType.ShareLinkBundle
}
)
})
it('correctly checks explicit global permissions', () => {
@@ -269,6 +275,10 @@ describe('PermissionsService', () => {
'view_sharelink',
'change_sharelink',
'delete_sharelink',
'add_sharelinkbundle',
'view_sharelinkbundle',
'change_sharelinkbundle',
'delete_sharelinkbundle',
'add_workflow',
'view_workflow',
'change_workflow',
@@ -26,6 +26,7 @@ export enum PermissionType {
User = '%s_user',
Group = '%s_group',
ShareLink = '%s_sharelink',
ShareLinkBundle = '%s_sharelinkbundle',
CustomField = '%s_customfield',
Workflow = '%s_workflow',
ProcessedMail = '%s_processedmail',
+46
View File
@@ -35,19 +35,27 @@ import {
arrowRightShort,
arrowUpRight,
asterisk,
bank,
basket,
bell,
bodyText,
bookmark,
boxArrowUp,
boxArrowUpRight,
boxes,
braces,
briefcase,
building,
calculator,
calendar,
calendarEvent,
calendarEventFill,
camera,
cardChecklist,
cardHeading,
caretDown,
caretUp,
cash,
chatLeftText,
chatSquareDots,
check,
@@ -65,6 +73,7 @@ import {
clipboardCheckFill,
clipboardFill,
clockHistory,
creditCard,
dash,
dashCircle,
diagram3,
@@ -83,9 +92,12 @@ import {
fileEarmarkDiff,
fileEarmarkFill,
fileEarmarkLock,
fileEarmarkMedical,
fileEarmarkMinus,
fileEarmarkPerson,
fileEarmarkPlus,
fileEarmarkRichtext,
fileEarmarkSpreadsheet,
fileText,
files,
filter,
@@ -93,12 +105,15 @@ import {
folderFill,
funnel,
gear,
globe2,
google,
grid,
gripVertical,
hash,
hddStack,
heart,
house,
inbox,
infoCircle,
journals,
link,
@@ -106,7 +121,9 @@ import {
listTask,
listUl,
microsoft,
newspaper,
nodePlus,
paperclip,
pencil,
people,
peopleFill,
@@ -121,9 +138,12 @@ import {
plusCircle,
printer,
questionCircle,
receipt,
safe,
scissors,
search,
send,
shop,
slashCircle,
sliders2Vertical,
sortAlphaDown,
@@ -133,14 +153,17 @@ import {
tag,
tagFill,
tags,
telephone,
textIndentLeft,
textLeft,
threeDots,
threeDotsVertical,
trash,
truck,
uiRadios,
unlock,
upcScan,
wallet2,
windowStack,
x,
xCircle,
@@ -258,15 +281,22 @@ const icons = {
arrowRightShort,
arrowUpRight,
asterisk,
bank,
basket,
bell,
braces,
bodyText,
bookmark,
boxArrowUp,
boxArrowUpRight,
boxes,
briefcase,
building,
calculator,
calendar,
calendarEvent,
calendarEventFill,
camera,
cardChecklist,
cardHeading,
caretDown,
@@ -288,6 +318,8 @@ const icons = {
clipboardCheckFill,
clipboardFill,
clockHistory,
cash,
creditCard,
dash,
dashCircle,
diagram3,
@@ -306,9 +338,12 @@ const icons = {
fileEarmarkDiff,
fileEarmarkFill,
fileEarmarkLock,
fileEarmarkMedical,
fileEarmarkMinus,
fileEarmarkPerson,
fileEarmarkPlus,
fileEarmarkRichtext,
fileEarmarkSpreadsheet,
files,
fileText,
filter,
@@ -316,12 +351,15 @@ const icons = {
folderFill,
funnel,
gear,
globe2,
google,
grid,
gripVertical,
hash,
hddStack,
heart,
house,
inbox,
infoCircle,
journals,
link,
@@ -329,8 +367,10 @@ const icons = {
listTask,
listUl,
microsoft,
newspaper,
nodePlus,
pencil,
paperclip,
people,
peopleFill,
person,
@@ -344,10 +384,13 @@ const icons = {
plusCircle,
printer,
questionCircle,
receipt,
safe,
scissors,
search,
send,
slashCircle,
shop,
sliders2Vertical,
sortAlphaDown,
sortAlphaUpAlt,
@@ -358,12 +401,15 @@ const icons = {
tags,
textIndentLeft,
textLeft,
telephone,
threeDots,
threeDotsVertical,
trash,
truck,
uiRadios,
unlock,
upcScan,
wallet2,
windowStack,
x,
xCircle,
+106
View File
@@ -0,0 +1,106 @@
from __future__ import annotations
import importlib
import zipfile
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
# availability is enforced separately in compression_available().
COMPRESSION_CHOICES: tuple[str, ...] = (
"stored",
"deflated",
"bzip2",
"lzma",
"zstd",
)
# Method name -> zipfile compression constant (zstd only when supported).
COMPRESSION_METHODS: dict[str, int] = {
"stored": zipfile.ZIP_STORED,
"deflated": zipfile.ZIP_DEFLATED,
"bzip2": zipfile.ZIP_BZIP2,
"lzma": zipfile.ZIP_LZMA,
}
if ZSTD is not None:
COMPRESSION_METHODS["zstd"] = ZSTD
# Inclusive (min, max) level bounds per method; None => level not applicable.
# Verified on CPython 3.14.3.
#
# zstd's raw library bounds are (-131072, 22)
# (compression.zstd.CompressionParameter.compression_level.bounds()) — the
# minimum is an internal implementation constant (-ZSTD_TARGETLENGTH_MAX),
# not a meaningful distinct "level"; deeper negative values than -22 buy
# nothing over -22 in practice. We expose the conventional zstd CLI range
# instead of the raw library bounds.
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
"stored": None,
"deflated": (0, 9),
"bzip2": (1, 9),
"lzma": None,
"zstd": (-22, 22),
}
# zipfile compress_type id -> method name.
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
zipfile.ZIP_STORED: "stored",
zipfile.ZIP_DEFLATED: "deflated",
zipfile.ZIP_BZIP2: "bzip2",
zipfile.ZIP_LZMA: "lzma",
93: "zstd",
}
def compression_available(method: str) -> bool:
"""Whether the running interpreter can actually use the given method."""
if method in ("stored", "deflated"):
# zlib is a hard CPython dependency; stored needs nothing.
return True
if method == "bzip2":
return _module_importable("bz2")
if method == "lzma":
return _module_importable("lzma")
if method == "zstd":
return ZSTD is not None and _module_importable("compression.zstd")
return False # pragma: no cover -- method is always one of COMPRESSION_CHOICES
def _module_importable(name: str) -> bool:
try:
importlib.import_module(name)
except ImportError:
return False
return True
def level_error(method: str, level: int | None) -> str | None:
"""Return a human message if (method, level) is invalid, else None."""
if level is None:
return None
bounds = LEVEL_BOUNDS[method]
if bounds is None:
return f"--zip-compression-level has no effect for '{method}'"
low, high = bounds
if not (low <= level <= high):
return (
f"--zip-compression-level for '{method}' must be between {low} and {high}"
)
return None
def compress_type_readable(compress_type: int) -> bool:
"""Whether this interpreter can decompress an entry of the given type."""
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
if method is None:
return False
return compression_available(method)
def unreadable_method_names(compress_types: set[int]) -> set[str]:
"""Map a set of compress_type ids to human method names for error messages."""
names: set[str] = set()
for ct in compress_types:
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
return names
+13 -2
View File
@@ -243,11 +243,21 @@ class ZipExportSink(ExportSink):
added as an entry at finalize (a zip entry cannot be interleaved with others).
"""
def __init__(self, target: Path, zip_name: str, *, delete: bool = False) -> None:
def __init__(
self,
target: Path,
zip_name: str,
*,
delete: bool = False,
compression: int = zipfile.ZIP_DEFLATED,
compresslevel: int | None = None,
) -> None:
self._target = target.resolve()
self._zip_path = (self._target / zip_name).with_suffix(".zip")
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
self._delete = delete
self._compression = compression
self._compresslevel = compresslevel
self._zip: zipfile.ZipFile | None = None
self._dirs: set[str] = set()
self._pending_manifest: tuple[Path, str] | None = None
@@ -258,7 +268,8 @@ class ZipExportSink(ExportSink):
self._zip = zipfile.ZipFile(
self._tmp_path,
"w",
compression=zipfile.ZIP_DEFLATED,
compression=self._compression,
compresslevel=self._compresslevel,
allowZip64=True,
)
@@ -29,6 +29,11 @@ if TYPE_CHECKING:
if settings.AUDIT_LOG_ENABLED:
from auditlog.models import LogEntry
from documents.export.compression import COMPRESSION_CHOICES
from documents.export.compression import COMPRESSION_METHODS
from documents.export.compression import ZSTD
from documents.export.compression import compression_available
from documents.export.compression import level_error
from documents.export.sinks import DirectoryExportSink
from documents.export.sinks import ExportSink
from documents.export.sinks import StreamingManifestWriter
@@ -192,6 +197,28 @@ class Command(CryptMixin, PaperlessCommand):
help="Sets the export zip file name",
)
parser.add_argument(
"--zip-compression",
choices=COMPRESSION_CHOICES,
default=None,
help=(
"Compression method for the export zip (requires --zip). "
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
"exporting and importing machine."
),
)
parser.add_argument(
"--zip-compression-level",
type=int,
default=None,
help=(
"Compression level for the export zip (requires --zip). "
"deflated: 0-9, bzip2: 1-9, zstd: -22..22; ignored for "
"stored/lzma."
),
)
parser.add_argument(
"--data-only",
default=False,
@@ -247,12 +274,39 @@ class Command(CryptMixin, PaperlessCommand):
if not os.access(self.target, os.W_OK):
raise CommandError("That path doesn't appear to be writable")
zip_compression: str | None = options["zip_compression"]
zip_compression_level: int | None = options["zip_compression_level"]
if not self.zip_export and (
zip_compression is not None or zip_compression_level is not None
):
raise CommandError(
"--zip-compression and --zip-compression-level require --zip",
)
compression_method = zip_compression or "deflated"
if self.zip_export:
if not compression_available(compression_method):
if compression_method == "zstd" and ZSTD is None:
raise CommandError(
"zstd compression requires Python 3.14 or newer",
)
raise CommandError(
f"Compression method '{compression_method}' is not "
f"available on this Python runtime",
)
level_msg = level_error(compression_method, zip_compression_level)
if level_msg is not None:
raise CommandError(level_msg)
sink: ExportSink
if self.zip_export:
sink = ZipExportSink(
self.target,
options["zip_name"],
delete=self.delete,
compression=COMPRESSION_METHODS[compression_method],
compresslevel=zip_compression_level,
)
else:
sink = DirectoryExportSink(
@@ -55,7 +55,7 @@ class Command(PaperlessCommand):
"--ratio",
default=85.0,
type=float,
help="Ratio to consider documents a match",
help="Ratio to consider documents a match (0.0 - 100.0)",
)
parser.add_argument(
"--delete",
@@ -69,6 +69,17 @@ class Command(PaperlessCommand):
action="store_true",
help="Skip the confirmation prompt when used with --delete",
)
parser.add_argument(
"--url",
default=None,
type=str,
help=(
"Base URL of the Paperless instance (e.g. "
"http://localhost:8000 or https://paperless.local). If set, matched "
"documents are shown as clickable (usually ctrl+click) links to "
"<url>/documents/<id>/details instead of by title."
),
)
def _render_results(
self,
@@ -76,6 +87,7 @@ class Command(PaperlessCommand):
*,
opt_ratio: float,
do_delete: bool,
base_url: str | None = None,
) -> list[int]:
"""Render match results as a Rich table. Returns list of PKs to delete."""
if not matches:
@@ -88,13 +100,22 @@ class Command(PaperlessCommand):
)
return []
# Fetch titles for matched documents in a single query.
all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
titles: dict[int, str] = dict(
Document.objects.filter(pk__in=all_pks)
.only("pk", "title")
.values_list("pk", "title"),
)
# Fetch titles for matched documents in a single query, unless we're
# going to show URLs instead.
titles: dict[int, str] = {}
if not base_url:
all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
titles = dict(
Document.objects.filter(pk__in=all_pks)
.only("pk", "title")
.values_list("pk", "title"),
)
def _cell(pk: int) -> str:
if base_url:
doc_url = f"{base_url.rstrip('/')}/documents/{pk}/details"
return f"[link={doc_url}]{doc_url}[/link]"
return f"[dim]#{pk}[/dim] {titles.get(pk, 'Unknown')}"
table = Table(
title=f"Fuzzy Matches (threshold: {opt_ratio:.1f}%)",
@@ -124,8 +145,8 @@ class Command(PaperlessCommand):
table.add_row(
str(i),
f"[dim]#{pk_a}[/dim] {titles.get(pk_a, 'Unknown')}",
f"[dim]#{pk_b}[/dim] {titles.get(pk_b, 'Unknown')}",
_cell(pk_a),
_cell(pk_b),
Text(f"{ratio:.1f}%", style=ratio_style),
)
maybe_delete_ids.append(pk_b)
@@ -208,6 +229,7 @@ class Command(PaperlessCommand):
matches,
opt_ratio=opt_ratio,
do_delete=options["delete"],
base_url=options["url"],
)
if options["delete"] and maybe_delete_ids:
@@ -32,6 +32,8 @@ from django.db.models.signals import post_save
from filelock import FileLock
from guardian.shortcuts import clear_ct_cache
from documents.export.compression import compress_type_readable
from documents.export.compression import unreadable_method_names
from documents.file_handling import create_source_path_directory
from documents.management.commands.base import PaperlessCommand
from documents.management.commands.mixins import CryptMixin
@@ -460,6 +462,20 @@ class Command(CryptMixin, PaperlessCommand):
with tempfile.TemporaryDirectory() as tmp_dir:
if is_zipfile(self.source):
with ZipFile(self.source) as zf:
unsupported = {
info.compress_type
for info in zf.infolist()
if not compress_type_readable(info.compress_type)
}
if unsupported:
names = sorted(unreadable_method_names(unsupported))
message = (
f"This archive uses compression this Python version cannot "
f"read ({', '.join(names)})."
)
if "zstd" in names:
message += " zstd archives require Python 3.14+."
raise CommandError(message)
zf.extractall(tmp_dir)
self.source = Path(tmp_dir)
self._run_import()
@@ -0,0 +1,79 @@
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0022_add_perf_indexes"),
]
operations = [
migrations.AddField(
model_name="savedview",
name="icon",
field=models.CharField(
choices=[
("archive", "Archive"),
("bank", "Bank"),
("basket", "Basket"),
("bell", "Bell"),
("bookmark", "Bookmark"),
("boxes", "Boxes"),
("briefcase", "Briefcase"),
("building", "Building"),
("calculator", "Calculator"),
("calendar", "Calendar"),
("camera", "Camera"),
("card-checklist", "Checklist"),
("cash", "Cash"),
("chat-left-text", "Chat"),
("check-circle", "Check"),
("clipboard", "Clipboard"),
("clock-history", "Clock"),
("credit-card", "Credit card"),
("download", "Download"),
("envelope", "Envelope"),
("exclamation-triangle", "Warning"),
("file-earmark", "File"),
("file-earmark-check", "Checked file"),
("file-earmark-lock", "Locked file"),
("file-earmark-medical", "Medical file"),
("file-earmark-person", "Person file"),
("file-earmark-spreadsheet", "Spreadsheet"),
("file-text", "Text file"),
("files", "Files"),
("folder", "Folder"),
("funnel", "Filter"),
("gear", "Gear"),
("globe2", "Globe"),
("hash", "Hash"),
("heart", "Heart"),
("house", "House"),
("inbox", "Inbox"),
("journals", "Journals"),
("list-task", "Task list"),
("newspaper", "Newspaper"),
("paperclip", "Attachment"),
("people", "People"),
("person", "Person"),
("printer", "Printer"),
("receipt", "Receipt"),
("safe", "Safe"),
("search", "Search"),
("send", "Send"),
("shop", "Shop"),
("stack", "Stack"),
("stars", "Stars"),
("tag", "Tag"),
("tags", "Tags"),
("telephone", "Telephone"),
("truck", "Truck"),
("upc-scan", "Barcode"),
("wallet2", "Wallet"),
],
default="funnel",
max_length=64,
verbose_name="icon",
),
),
]
+69
View File
@@ -519,6 +519,68 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
class SavedView(ModelWithOwner):
class Icon(models.TextChoices):
ARCHIVE = ("archive", _("Archive"))
BANK = ("bank", _("Bank"))
BASKET = ("basket", _("Basket"))
BELL = ("bell", _("Bell"))
BOOKMARK = ("bookmark", _("Bookmark"))
BOXES = ("boxes", _("Boxes"))
BRIEFCASE = ("briefcase", _("Briefcase"))
BUILDING = ("building", _("Building"))
CALCULATOR = ("calculator", _("Calculator"))
CALENDAR = ("calendar", _("Calendar"))
CAMERA = ("camera", _("Camera"))
CARD_CHECKLIST = ("card-checklist", _("Checklist"))
CASH = ("cash", _("Cash"))
CHAT_LEFT_TEXT = ("chat-left-text", _("Chat"))
CHECK_CIRCLE = ("check-circle", _("Check"))
CLIPBOARD = ("clipboard", _("Clipboard"))
CLOCK_HISTORY = ("clock-history", _("Clock"))
CREDIT_CARD = ("credit-card", _("Credit card"))
DOWNLOAD = ("download", _("Download"))
ENVELOPE = ("envelope", _("Envelope"))
EXCLAMATION_TRIANGLE = ("exclamation-triangle", _("Warning"))
FILE_EARMARK = ("file-earmark", _("File"))
FILE_EARMARK_CHECK = ("file-earmark-check", _("Checked file"))
FILE_EARMARK_LOCK = ("file-earmark-lock", _("Locked file"))
FILE_EARMARK_MEDICAL = ("file-earmark-medical", _("Medical file"))
FILE_EARMARK_PERSON = ("file-earmark-person", _("Person file"))
FILE_EARMARK_SPREADSHEET = (
"file-earmark-spreadsheet",
_("Spreadsheet"),
)
FILE_TEXT = ("file-text", _("Text file"))
FILES = ("files", _("Files"))
FOLDER = ("folder", _("Folder"))
FUNNEL = ("funnel", _("Filter"))
GEAR = ("gear", _("Gear"))
GLOBE = ("globe2", _("Globe"))
HASH = ("hash", _("Hash"))
HEART = ("heart", _("Heart"))
HOUSE = ("house", _("House"))
INBOX = ("inbox", _("Inbox"))
JOURNALS = ("journals", _("Journals"))
LIST_TASK = ("list-task", _("Task list"))
NEWSPAPER = ("newspaper", _("Newspaper"))
PAPERCLIP = ("paperclip", _("Attachment"))
PEOPLE = ("people", _("People"))
PERSON = ("person", _("Person"))
PRINTER = ("printer", _("Printer"))
RECEIPT = ("receipt", _("Receipt"))
SAFE = ("safe", _("Safe"))
SEARCH = ("search", _("Search"))
SEND = ("send", _("Send"))
SHOP = ("shop", _("Shop"))
STACK = ("stack", _("Stack"))
STARS = ("stars", _("Stars"))
TAG = ("tag", _("Tag"))
TAGS = ("tags", _("Tags"))
TELEPHONE = ("telephone", _("Telephone"))
TRUCK = ("truck", _("Truck"))
UPC_SCAN = ("upc-scan", _("Barcode"))
WALLET = ("wallet2", _("Wallet"))
class DisplayMode(models.TextChoices):
TABLE = ("table", _("Table"))
SMALL_CARDS = ("smallCards", _("Small Cards"))
@@ -541,6 +603,13 @@ class SavedView(ModelWithOwner):
name = models.CharField(_("name"), max_length=128)
icon = models.CharField(
_("icon"),
max_length=64,
choices=Icon.choices,
default=Icon.FUNNEL,
)
sort_field = models.CharField(
_("sort field"),
max_length=128,
+11 -25
View File
@@ -85,6 +85,7 @@ from documents.permissions import set_permissions_for_object
from documents.regex import validate_regex_pattern
from documents.templating.filepath import validate_filepath_template_and_render
from documents.templating.utils import convert_format_str_to_template_format
from documents.templating.workflows import validate_workflow_template
from documents.validators import uri_validator
from documents.validators import url_validator
@@ -1383,6 +1384,7 @@ class SavedViewSerializer(OwnedObjectSerializer):
fields = [
"id",
"name",
"icon",
"sort_field",
"sort_reverse",
"filter_rules",
@@ -3184,35 +3186,19 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
attrs["assign_title"] = None
else:
try:
# test against all placeholders, see consumer.py `parse_doc_title_w_placeholders`
attrs["assign_title"].format(
correspondent="",
document_type="",
added="",
added_year="",
added_year_short="",
added_month="",
added_month_name="",
added_month_name_short="",
added_day="",
added_time="",
owner_username="",
original_filename="",
filename="",
created="",
created_year="",
created_year_short="",
created_month="",
created_month_name="",
created_month_name_short="",
created_day="",
created_time="",
)
validate_workflow_template(attrs["assign_title"])
except (ValueError, KeyError) as e:
raise serializers.ValidationError(
{"assign_title": f'Invalid f-string detected: "{e.args[0]}"'},
{"assign_title": f"{e.args[0]}"},
)
if attrs.get("assign_custom_fields_values"):
# Empty strings treated as None to avoid unexpected behavior
attrs["assign_custom_fields_values"] = {
field_id: (None if value == "" else value)
for field_id, value in attrs["assign_custom_fields_values"].items()
}
if (
"type" in attrs
and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL
+45
View File
@@ -6,9 +6,11 @@ from pathlib import Path
from django.utils.text import slugify as django_slugify
from jinja2 import StrictUndefined
from jinja2 import Template
from jinja2 import TemplateAssertionError
from jinja2 import TemplateSyntaxError
from jinja2 import UndefinedError
from jinja2 import make_logging_undefined
from jinja2.meta import find_undeclared_variables
from jinja2.sandbox import SecurityError
from documents.templating.environment import _template_environment
@@ -29,6 +31,49 @@ _template_environment.filters["slugify"] = django_slugify
_template_environment.filters["localize_date"] = localize_date
_known_placeholder_names = {
"correspondent",
"document_type",
"added",
"added_year",
"added_year_short",
"added_month",
"added_month_name",
"added_month_name_short",
"added_day",
"added_time",
"owner_username",
"original_filename",
"filename",
"created",
"created_year",
"created_year_short",
"created_month",
"created_month_name",
"created_month_name_short",
"created_day",
"created_time",
"doc_title",
"doc_url",
"doc_id",
}
def validate_workflow_template(text: str) -> None:
try:
ast = _template_environment.parse(text)
undeclared_vars = find_undeclared_variables(ast)
except TemplateAssertionError as e:
raise ValueError(f"Template assertion error: {e}")
except TemplateSyntaxError as e:
raise ValueError(f"Template syntax error: {e}")
unknown_vars = undeclared_vars - _known_placeholder_names
if unknown_vars:
raise KeyError(
f"Template references unknown placeholders: {', '.join(unknown_vars)}",
)
def parse_w_workflow_placeholders(
text: str,
correspondent_name: str,
@@ -0,0 +1,208 @@
import sys
import zipfile
import pytest
import pytest_mock
from documents.export import compression
class TestCompressionMethods:
def test_choices_always_include_zstd(self) -> None:
"""
GIVEN:
- The compression policy module's CLI choices list
WHEN:
- Read on any runtime
THEN:
- zstd is always present; availability is checked separately so
argparse never hides it based on the current Python version
"""
assert compression.COMPRESSION_CHOICES == (
"stored",
"deflated",
"bzip2",
"lzma",
"zstd",
)
@pytest.mark.parametrize(
("name", "constant"),
[
("stored", zipfile.ZIP_STORED),
("deflated", zipfile.ZIP_DEFLATED),
("bzip2", zipfile.ZIP_BZIP2),
("lzma", zipfile.ZIP_LZMA),
],
)
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
"""
GIVEN:
- A compression method name
WHEN:
- Looked up in COMPRESSION_METHODS
THEN:
- It maps to the matching zipfile compression constant
"""
assert compression.COMPRESSION_METHODS[name] == constant
def test_stored_and_deflated_always_available(self) -> None:
"""
GIVEN:
- The stored and deflated compression methods
WHEN:
- Checked with compression_available()
THEN:
- Both are always available (zlib is a hard CPython dependency)
"""
assert compression.compression_available("stored")
assert compression.compression_available("deflated")
def test_zstd_availability_tracks_runtime(self) -> None:
"""
GIVEN:
- The zstd compression method
WHEN:
- Checked with compression_available() on this runtime
THEN:
- Availability matches whether Python is 3.14+
"""
expected: bool = sys.version_info >= (3, 14)
assert compression.compression_available("zstd") == expected
def test_unimportable_module_reports_unavailable(
self,
mocker: pytest_mock.MockerFixture,
) -> None:
"""
GIVEN:
- A compression method whose backing module fails to import
(e.g. a minimal Python build without bz2/lzma compiled in)
WHEN:
- Checked with compression_available()
THEN:
- False is returned rather than the ImportError propagating
"""
mocker.patch(
"documents.export.compression.importlib.import_module",
side_effect=ImportError,
)
assert not compression.compression_available("bzip2")
class TestLevelError:
@pytest.mark.parametrize(
("method", "level"),
[
("deflated", 0),
("deflated", 9),
("bzip2", 1),
("bzip2", 9),
("zstd", -22),
("zstd", 22),
("deflated", None),
("stored", None),
],
)
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
"""
GIVEN:
- A method and a level within its valid bounds (or no level)
WHEN:
- Checked with level_error()
THEN:
- No error message is returned
"""
assert compression.level_error(method, level) is None
@pytest.mark.parametrize(
("method", "level"),
[
("deflated", 10),
("deflated", -1),
("bzip2", 0),
("bzip2", 10),
("zstd", -23),
("zstd", 23),
],
)
def test_out_of_range_levels_return_message(
self,
method: str,
level: int,
) -> None:
"""
GIVEN:
- A method and a level outside its valid bounds
WHEN:
- Checked with level_error()
THEN:
- An error message naming the valid range is returned
"""
msg: str | None = compression.level_error(method, level)
assert msg is not None
assert "between" in msg
@pytest.mark.parametrize("method", ["stored", "lzma"])
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
"""
GIVEN:
- A method that ignores compression level (stored, lzma)
WHEN:
- A level is passed to level_error() anyway
THEN:
- An error message noting the level has no effect is returned
"""
msg: str | None = compression.level_error(method, 5)
assert msg is not None
assert "no effect" in msg
class TestCompressTypeReadable:
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
"""
GIVEN:
- A stored or deflated compress_type id
WHEN:
- Checked with compress_type_readable()
THEN:
- It is always readable
"""
assert compression.compress_type_readable(ct)
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
"""
GIVEN:
- The zstd compress_type id (93, ZIP_ZSTANDARD)
WHEN:
- Checked with compress_type_readable() on this runtime
THEN:
- Readability matches whether Python is 3.14+
"""
expected: bool = sys.version_info >= (3, 14)
assert compression.compress_type_readable(93) == expected
def test_unknown_compress_type_is_unreadable(self) -> None:
"""
GIVEN:
- An unrecognized compress_type id
WHEN:
- Checked with compress_type_readable()
THEN:
- It is reported as unreadable
"""
assert not compression.compress_type_readable(9999)
def test_unreadable_method_names_lists_methods(self) -> None:
"""
GIVEN:
- A set containing an unknown compress_type id
WHEN:
- Passed to unreadable_method_names()
THEN:
- It is reported generically as "method <id>"
"""
# An unknown method id maps to no name and is reported generically.
names: set[str] = compression.unreadable_method_names({9999})
assert names == {"method 9999"}
+43
View File
@@ -5,6 +5,7 @@ import zipfile
from pathlib import Path
import pytest
import pytest_mock
from pytest_django.fixtures import SettingsWrapper
from documents.export.sinks import DirectoryExportSink
@@ -305,6 +306,48 @@ class TestZipExportSink:
assert not (target / "export.zip").exists()
class TestZipExportSinkCompression:
@pytest.mark.parametrize(
("method", "constant"),
[
("stored", zipfile.ZIP_STORED),
("deflated", zipfile.ZIP_DEFLATED),
("bzip2", zipfile.ZIP_BZIP2),
("lzma", zipfile.ZIP_LZMA),
],
)
def test_compression_and_level_forwarded_to_zipfile(
self,
mocker: pytest_mock.MockerFixture,
tmp_path: Path,
method: str,
constant: int,
) -> None:
"""
GIVEN:
- A ZipExportSink constructed with a compression method and level
WHEN:
- The sink is opened
THEN:
- zipfile.ZipFile is constructed with those values forwarded
unchanged (whether ZipFile actually compresses is Python's own
contract, not ours, so this checks the call args, not a real
archive)
"""
target: Path = tmp_path / "out"
target.mkdir()
zip_cls = mocker.patch("documents.export.sinks.zipfile.ZipFile")
sink = ZipExportSink(target, "export", compression=constant, compresslevel=5)
sink._open()
zip_cls.assert_called_once_with(
mocker.ANY,
"w",
compression=constant,
compresslevel=5,
allowZip64=True,
)
class TestStreamContract:
@pytest.fixture(params=["dir", "zip"])
def sink(self, request: pytest.FixtureRequest, tmp_path: Path) -> ExportSink:
+10 -1
View File
@@ -2905,18 +2905,20 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
v1 = SavedView.objects.get(name="test")
self.assertEqual(v1.sort_field, "created2")
self.assertEqual(v1.icon, SavedView.Icon.FUNNEL)
self.assertEqual(v1.filter_rules.count(), 1)
self.assertEqual(v1.owner, self.user)
response = self.client.patch(
f"/api/saved_views/{v1.id}/",
{"sort_reverse": True},
{"sort_reverse": True, "icon": SavedView.Icon.RECEIPT},
format="json",
)
v1 = SavedView.objects.get(id=v1.id)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertTrue(v1.sort_reverse)
self.assertEqual(v1.icon, SavedView.Icon.RECEIPT)
self.assertEqual(v1.filter_rules.count(), 1)
view["filter_rules"] = [{"rule_type": 12, "value": "secret"}]
@@ -2936,6 +2938,13 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
v1 = SavedView.objects.get(id=v1.id)
self.assertEqual(v1.filter_rules.count(), 0)
response = self.client.patch(
f"/api/saved_views/{v1.id}/",
{"icon": "not-an-icon"},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
def test_saved_view_display_options(self) -> None:
"""
GIVEN:
+130 -5
View File
@@ -351,11 +351,45 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(WorkflowTrigger.objects.count(), 1)
def test_api_create_invalid_assign_title(self) -> None:
def test_api_create_complex_assign_title(self) -> None:
"""
GIVEN:
- API request to create a workflow
- Invalid f-string for assign_title
- Template using Jinja flow control statements
WHEN:
- API is called
THEN:
- Workflow is created
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Workflow 2",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
},
],
"actions": [
{
"assign_title": '{# this is a comment #}foo{% if created_year < 2000 %}bar{% endif %}{{ "{:04d}".format(42) }}',
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
self.assertEqual(Workflow.objects.count(), 2)
def test_api_create_invalid_assign_title_syntax_error(self) -> None:
"""
GIVEN:
- API request to create a workflow
- Invalid template for assign_title
WHEN:
- API is called
THEN:
@@ -366,7 +400,7 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.ENDPOINT,
json.dumps(
{
"name": "Workflow 1",
"name": "Workflow 2",
"order": 1,
"triggers": [
{
@@ -375,7 +409,7 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
],
"actions": [
{
"assign_title": "{created_year]",
"assign_title": "{{created_year}",
},
],
},
@@ -384,7 +418,89 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn(
"Invalid f-string detected",
"Template syntax error",
response.data["actions"][0]["assign_title"][0],
)
self.assertEqual(Workflow.objects.count(), 1)
def test_api_create_invalid_assign_title_assertion_error(self) -> None:
"""
GIVEN:
- API request to create a workflow
- Template using unknown filters for assign_title
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Workflow 2",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
},
],
"actions": [
{
"assign_title": "{{ created_year | foo }}",
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn(
"Template assertion error",
response.data["actions"][0]["assign_title"][0],
)
self.assertEqual(Workflow.objects.count(), 1)
def test_api_create_invalid_assign_title_unknown_placeholder(self) -> None:
"""
GIVEN:
- API request to create a workflow
- Template with unknown placeholders for assign_title
WHEN:
- API is called
THEN:
- Correct HTTP 400 response
- No objects are created
"""
response = self.client.post(
self.ENDPOINT,
json.dumps(
{
"name": "Workflow 2",
"order": 1,
"triggers": [
{
"type": WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
},
],
"actions": [
{
"assign_title": "{{creation_year}}",
},
],
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn(
"Template references unknown placeholders",
response.data["actions"][0]["assign_title"][0],
)
self.assertIn(
"creation_year",
response.data["actions"][0]["assign_title"][0],
)
@@ -422,6 +538,11 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
json.dumps(
{
"assign_title": "",
"assign_custom_fields": [self.cf1.id, self.cf2.id],
"assign_custom_fields_values": {
str(self.cf1.id): "",
str(self.cf2.id): 0,
},
},
),
content_type="application/json",
@@ -429,6 +550,10 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
action = WorkflowAction.objects.get(id=response.data["id"])
self.assertIsNone(action.assign_title)
self.assertEqual(
action.assign_custom_fields_values,
{str(self.cf1.id): None, str(self.cf2.id): 0},
)
response = self.client.post(
self.ENDPOINT_TRIGGERS,
@@ -6,6 +6,8 @@ from datetime import timedelta
from io import StringIO
from pathlib import Path
from unittest import mock
from zipfile import ZIP_DEFLATED
from zipfile import ZIP_LZMA
from zipfile import ZipFile
import pytest
@@ -1078,6 +1080,197 @@ class TestExportImport(
skip_checks=True,
)
def test_compression_flags_require_zip(self) -> None:
"""
GIVEN:
- A request to export without --zip
WHEN:
- --zip-compression or --zip-compression-level is passed anyway
THEN:
- A CommandError is raised (the flags are meaningless without --zip)
"""
cases = {
"zip-compression": ["--zip-compression", "lzma"],
"zip-compression-level": ["--zip-compression-level", "5"],
}
for case_id, args in cases.items():
with self.subTest(case_id), self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
*args,
skip_checks=True,
)
def test_zip_compression_level_out_of_range_raises(self) -> None:
"""
GIVEN:
- A request to export to a zip file
WHEN:
- --zip-compression-level is outside the chosen method's valid range
THEN:
- A CommandError is raised
"""
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"deflated",
"--zip-compression-level",
"99",
skip_checks=True,
)
def test_zip_compression_level_rejected_for_levelless_method(self) -> None:
"""
GIVEN:
- A request to export to a zip file with a compression method
that ignores level entirely (stored, lzma)
WHEN:
- --zip-compression-level is also passed
THEN:
- A CommandError is raised
"""
for method in ("stored", "lzma"):
with self.subTest(method), self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
method,
"--zip-compression-level",
"5",
skip_checks=True,
)
def test_zstd_unavailable_raises_friendly_error(self) -> None:
"""
GIVEN:
- A Python runtime without zstd support (< 3.14)
WHEN:
- --zip-compression zstd is requested
THEN:
- A CommandError naming the Python version requirement is raised
zstd availability is mocked rather than relying on the actual
runtime: on a Python 3.14+ CI leg, ZSTD is not None, so without the
mock this check is skipped and the command falls through into the
real export, which fails on missing document files instead of
raising the expected CommandError.
"""
with (
mock.patch(
"documents.management.commands.document_exporter.ZSTD",
None,
),
mock.patch(
"documents.management.commands.document_exporter.compression_available",
return_value=False,
),
self.assertRaises(CommandError) as e,
):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"zstd",
skip_checks=True,
)
self.assertIn("3.14", str(e.exception))
def test_non_zstd_unavailable_raises_generic_error(self) -> None:
"""
GIVEN:
- A Python runtime missing the module backing a non-zstd method
(e.g. bz2/lzma not compiled in on a minimal build)
WHEN:
- That method is requested via --zip-compression
THEN:
- A CommandError is raised naming the method, not the
zstd-specific "requires 3.14" message
"""
with (
mock.patch(
"documents.management.commands.document_exporter.compression_available",
return_value=False,
),
self.assertRaises(CommandError) as e,
):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"bzip2",
skip_checks=True,
)
self.assertIn("bzip2", str(e.exception))
self.assertNotIn("3.14", str(e.exception))
def test_zip_compression_flag_resolves_to_sink_constant(self) -> None:
"""
GIVEN:
- A request to export to a zip file with --zip-compression lzma
WHEN:
- The export runs
THEN:
- ZipExportSink is constructed with the resolved ZIP_LZMA constant
(whether zipfile actually compresses with the chosen method is
Python's own contract, and ZipExportSink's own tests already
cover the forwarding; what this command owns is resolving the
CLI string to the right constant, so assert that resolution
directly)
"""
with mock.patch(
"documents.management.commands.document_exporter.ZipExportSink",
) as sink_cls:
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"lzma",
skip_checks=True,
)
sink_cls.assert_called_once_with(
mock.ANY,
mock.ANY,
delete=False,
compression=ZIP_LZMA,
compresslevel=None,
)
def test_default_zip_compression_resolves_to_deflate(self) -> None:
"""
GIVEN:
- A request to export to a zip file with no --zip-compression flag
WHEN:
- The export runs
THEN:
- ZipExportSink is constructed with the default ZIP_DEFLATED
constant and compresslevel=None, matching pre-existing behavior
"""
with mock.patch(
"documents.management.commands.document_exporter.ZipExportSink",
) as sink_cls:
call_command(
"document_exporter",
self.target,
"--zip",
skip_checks=True,
)
sink_cls.assert_called_once_with(
mock.ANY,
mock.ANY,
delete=False,
compression=ZIP_DEFLATED,
compresslevel=None,
)
@pytest.mark.management
class TestCryptExportImport(
+41 -1
View File
@@ -1,3 +1,4 @@
import os
from io import StringIO
from unittest.mock import patch
@@ -41,7 +42,7 @@ class TestFuzzyMatchCommand(TestCase):
def test_invalid_ratio_upper_limit(self) -> None:
"""
GIVEN:s
GIVEN:
- Invalid ratio above upper
WHEN:
- Command is called
@@ -108,6 +109,45 @@ class TestFuzzyMatchCommand(TestCase):
stdout, _ = self.call_command("--processes", "1")
self.assertIn("Found 1 matching pair(s)", stdout)
def test_with_matches_and_url(self) -> None:
"""
GIVEN:
- 2 documents exist
- Similarity between content is 86.667
- --url is provided
WHEN:
- Command is called with --url
THEN:
- 1 match is returned from doc 1 to doc 2
- No match from doc 2 to doc 1 reported
- Output contains clickable links to the documents instead of titles
"""
# Content similarity is 86.667
Document.objects.create(
checksum="BEEFCAFE",
title="A",
content="first document scanned by bob",
mime_type="application/pdf",
filename="test.pdf",
)
Document.objects.create(
checksum="DEADBEAF",
title="A",
content="first document scanned by alice",
mime_type="application/pdf",
filename="other_test.pdf",
)
with patch.dict(os.environ, {"COLUMNS": "200"}):
stdout, _ = self.call_command(
"--processes",
"1",
"--url",
"http://localhost:8000",
)
self.assertIn("Found 1 matching pair(s)", stdout)
self.assertIn("http://localhost:8000/documents/1/details", stdout)
self.assertIn("http://localhost:8000/documents/2/details", stdout)
def test_with_3_matches(self) -> None:
"""
GIVEN:
@@ -525,6 +525,71 @@ class TestCommandImport(
self.assertEqual(doc.tags.count(), 1)
self.assertEqual(doc.tags.first().name, "batch-flush-tag")
def test_import_rejects_unreadable_compression(self) -> None:
"""
GIVEN:
- A zip archive with an entry whose compression this Python can't read
WHEN:
- Import is attempted
THEN:
- A CommandError naming the issue is raised, before extraction
"""
import zipfile
from unittest import mock
archive = Path(self.dirs.scratch_dir) / "export.zip"
with zipfile.ZipFile(archive, "w") as zf:
zf.writestr("manifest.json", "[]")
with mock.patch(
"documents.management.commands.document_importer.compress_type_readable",
return_value=False,
):
with self.assertRaises(CommandError) as e:
call_command(
"document_importer",
str(archive),
"--no-progress-bar",
skip_checks=True,
)
self.assertIn("compression", str(e.exception))
def test_import_rejects_unreadable_zstd_with_version_hint(self) -> None:
"""
GIVEN:
- A zip archive with an entry compressed with zstd
WHEN:
- Import is attempted on a Python runtime that can't read zstd
THEN:
- The CommandError names the 3.14+ requirement, not just the
generic "can't read" message
"""
import zipfile
from unittest import mock
archive = Path(self.dirs.scratch_dir) / "export.zip"
with zipfile.ZipFile(archive, "w") as zf:
zf.writestr("manifest.json", "[]")
with (
mock.patch(
"documents.management.commands.document_importer.compress_type_readable",
return_value=False,
),
mock.patch(
"documents.management.commands.document_importer.unreadable_method_names",
return_value={"zstd"},
),
):
with self.assertRaises(CommandError) as e:
call_command(
"document_importer",
str(archive),
"--no-progress-bar",
skip_checks=True,
)
self.assertIn("3.14", str(e.exception))
@pytest.mark.management
@pytest.mark.django_db
+49
View File
@@ -2000,6 +2000,55 @@ class TestWorkflows(
r"Doc added in \w{3,}",
) # Match any 3-letter month name
def test_document_updated_workflow_existing_custom_field_empty_value(self) -> None:
"""
GIVEN:
- Existing workflow with UPDATED trigger and action that assigns a custom field
with an empty value
WHEN:
- Document is updated that already contains the field with a value
THEN:
- The existing value is left untouched, see GH #13627
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
filter_has_document_type=self.dt,
)
action = WorkflowAction.objects.create()
action.assign_custom_fields.add(self.cf1)
action.assign_custom_fields_values = {self.cf1.pk: ""}
action.save()
w = Workflow.objects.create(
name="Workflow 1",
order=0,
)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
correspondent=self.c,
original_filename="sample.pdf",
)
CustomFieldInstance.objects.create(
document=doc,
field=self.cf1,
value_text="existing value",
)
superuser = User.objects.create_superuser("superuser")
self.client.force_authenticate(user=superuser)
self.client.patch(
f"/api/documents/{doc.id}/",
{"document_type": self.dt.id},
format="json",
)
doc.refresh_from_db()
self.assertEqual(doc.custom_fields.get(field=self.cf1).value, "existing value")
def test_document_updated_workflow_existing_custom_field(self) -> None:
"""
GIVEN:
+1 -1
View File
@@ -2267,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions")
documents = [document]
documents = Document.objects.filter(pk=document.pk)
else:
documents = Document.objects.filter(
id__in=permitted_document_ids(request.user),
+2 -1
View File
@@ -105,7 +105,8 @@ def apply_assignment_to_document(
field=field,
document=document,
).first()
if instance and args[value_field_name] is not None:
# empty string is indistinguishable from no value in the UI
if instance and args[value_field_name] not in (None, ""):
setattr(instance, value_field_name, args[value_field_name])
instance.save()
elif not instance:
File diff suppressed because it is too large Load Diff
+33 -7
View File
@@ -3,7 +3,9 @@ Built-in remote-OCR document parser.
Handles documents by sending them to a configured remote OCR engine
(currently Azure AI Vision / Document Intelligence) and retrieving both
the extracted text and a searchable PDF with an embedded text layer.
the extracted text and a searchable PDF with an embedded text layer. For
born-digital PDFs that need no archive copy, the remote call is skipped
entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
When no engine is configured, ``score()`` returns ``None`` so the parser
is effectively invisible to the registry the tesseract parser handles
@@ -22,6 +24,8 @@ from typing import Self
from django.conf import settings
from documents.parsers import ParseError
from paperless.parsers.utils import extract_pdf_text
from paperless.parsers.utils import post_process_text
from paperless.version import __full_version_str__
if TYPE_CHECKING:
@@ -70,8 +74,11 @@ class RemoteDocumentParser:
"""Parse documents via a remote OCR API (currently Azure AI Vision).
This parser sends documents to a remote engine that returns both
extracted text and a searchable PDF with an embedded text layer.
It does not depend on Tesseract or ocrmypdf.
extracted text and a searchable PDF with an embedded text layer,
except when ``parse()`` is called with ``produce_archive=False`` for
a PDF, in which case the remote call is skipped and only locally
extracted text is returned (no archive). It does not depend on
Tesseract or ocrmypdf.
Class attributes
----------------
@@ -160,8 +167,11 @@ class RemoteDocumentParser:
Returns
-------
bool
Always True the remote engine always returns a PDF with an
embedded text layer that serves as the archive copy.
Always True the remote engine is capable of returning a PDF
with an embedded text layer to serve as the archive copy.
Whether it actually does so for a given document depends on
``produce_archive`` passed to :meth:`parse` (see there for when
the remote engine call, and thus archive generation, is skipped).
"""
return True
@@ -218,6 +228,12 @@ class RemoteDocumentParser:
) -> None:
"""Send the document to the remote engine and store results.
When *produce_archive* is False for a PDF, the caller (via
``documents.consumer.should_produce_archive``) has already determined
that the document is born-digital and needs no archive skip the
remote engine entirely rather than re-OCRing it and creating a
duplicate text layer.
Parameters
----------
document_path:
@@ -225,8 +241,8 @@ class RemoteDocumentParser:
mime_type:
Detected MIME type of the document.
produce_archive:
Ignored the remote engine always returns a searchable PDF,
which is stored as the archive copy regardless of this flag.
Whether an archive copy is wanted. For PDFs, False skips the
remote engine and uses locally-extracted text instead.
"""
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
@@ -241,6 +257,16 @@ class RemoteDocumentParser:
self._text = ""
return
if not produce_archive and mime_type == "application/pdf":
logger.debug(
"Remote OCR: skipped — no archive requested, "
"using locally-extracted text",
)
self._text = (
post_process_text(extract_pdf_text(document_path, log=logger)) or ""
)
return
if config.engine == "azureai":
self._text = self._azure_ai_vision_parse(document_path, config)
@@ -337,6 +337,117 @@ class TestRemoteParserParse:
assert remote_parser.get_date() is None
# ---------------------------------------------------------------------------
# parse() — produce_archive=False skips the remote engine (PDFs only)
# ---------------------------------------------------------------------------
class TestRemoteParserSkipsWhenNoArchiveWanted:
"""When the caller has already decided no archive is needed for a PDF
(documents.consumer.should_produce_archive), the remote engine call is
skipped entirely in favor of locally-extracted text.
"""
def test_pdf_skips_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: Azure is never invoked, no archive is produced, and text
comes from local pdftotext extraction
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_not_called()
assert remote_parser.get_archive_path() is None
assert remote_parser.get_text() != ""
def test_pdf_no_archive_requested_text_matches_local_extraction(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
mocker: MockerFixture,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: the returned text is exactly the locally-extracted text,
not anything from the (unused) Azure mock
"""
mocker.patch(
"paperless.parsers.remote.extract_pdf_text",
return_value="Local digital text.",
)
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == "Local digital text."
def test_pdf_no_archive_requested_closes_no_client(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.close.assert_not_called()
def test_non_pdf_still_calls_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
Images have no local-text fallback, so produce_archive=False does
not skip the remote engine for non-PDF MIME types.
"""
remote_parser.parse(
simple_digital_pdf_file,
"image/png",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_called_once()
assert remote_parser.get_text() == _DEFAULT_TEXT
@pytest.mark.usefixtures("no_engine_settings")
def test_unconfigured_engine_takes_precedence_over_skip(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
) -> None:
"""An unconfigured engine still short-circuits before the
produce_archive check, returning empty text as before.
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == ""
assert remote_parser.get_archive_path() is None
# ---------------------------------------------------------------------------
# parse() — Azure failure path
# ---------------------------------------------------------------------------
+21 -6
View File
@@ -2,6 +2,8 @@ import json
import logging
import sys
from django.db.models import QuerySet
from documents.models import Document
from paperless.config import AIConfig
from paperless_ai.client import AIClient
@@ -82,10 +84,21 @@ def _build_document_reference(
def _get_document_references(
documents: list[Document],
documents: QuerySet[Document],
top_nodes: list,
) -> list[dict[str, int | str]]:
allowed_documents = {doc.pk: doc for doc in documents}
candidate_ids: set[int] = set()
for node in top_nodes:
try:
candidate_ids.add(int(node.metadata["document_id"]))
except (KeyError, TypeError, ValueError): # pragma: no cover
continue
if not candidate_ids:
return []
allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
references: list[dict[str, int | str]] = []
seen_document_ids: set[int] = set()
@@ -119,7 +132,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
def stream_chat_with_documents(
query_str: str,
documents: list[Document],
documents: QuerySet[Document],
output_language: str | None = None,
):
try:
@@ -135,10 +148,10 @@ def stream_chat_with_documents(
def _stream_chat_with_documents(
query_str: str,
documents: list[Document],
documents: QuerySet[Document],
output_language: str | None = None,
):
if not documents:
if not documents.exists():
yield CHAT_NO_CONTENT_MESSAGE
return
@@ -148,7 +161,9 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig()
filters = _document_id_filters(str(doc.pk) for doc in documents)
filters = _document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
# Hold the shared read lock for the whole operation: the query engine
# retrieves from the vector store again during synthesis, so the connection
+101 -46
View File
@@ -3,10 +3,12 @@ from unittest.mock import MagicMock
from unittest.mock import patch
import pytest
from django.db.models.signals import post_init
from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode
from documents.models import Document
from documents.tests.factories import DocumentFactory
from paperless_ai import chat
from paperless_ai import indexing
@@ -36,16 +38,6 @@ def patch_embed_nodes():
yield mock_embed_nodes
@pytest.fixture
def mock_document():
doc = MagicMock()
doc.pk = 1
doc.title = "Test Document"
doc.filename = "test_file.pdf"
doc.content = "This is the document content."
return doc
def assert_chat_output(
output: list[str],
*,
@@ -61,6 +53,13 @@ def assert_chat_output(
}
def _fake_documents_queryset(pks: list[int]) -> MagicMock:
qs = MagicMock()
qs.exists.return_value = bool(pks)
qs.values_list.return_value = pks
return qs
@pytest.mark.parametrize(
("output_language", "expected_language_line"),
[
@@ -107,9 +106,10 @@ def test_build_refine_prompt(
@pytest.mark.django_db
def test_stream_chat_with_one_document_retrieval(
mock_document,
patch_embed_nodes,
) -> None:
document = DocumentFactory.create(title="Test Document", content="ignored")
documents = Document.objects.filter(pk=document.pk)
with (
patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -124,22 +124,19 @@ def test_stream_chat_with_one_document_retrieval(
mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock()
mock_node = TextNode(
text="This is node content.",
metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
)
mock_index = MagicMock()
# Simulate get_nodes returning nodes (content exists)
mock_index.vector_store.get_nodes.return_value = [mock_node]
mock_index.vector_store.get_nodes.return_value = [
TextNode(
text="This is node content.",
metadata={"document_id": str(document.pk), "title": "Test Document"},
),
]
mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [
MagicMock(
metadata={
"document_id": str(mock_document.pk),
"title": "Test Document",
},
metadata={"document_id": str(document.pk), "title": "Test Document"},
),
]
@@ -153,7 +150,7 @@ def test_stream_chat_with_one_document_retrieval(
"llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance,
):
output = list(stream_chat_with_documents("What is this?", [mock_document]))
output = list(stream_chat_with_documents("What is this?", documents))
mock_query_engine.query.assert_called_once_with("What is this?")
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
@@ -166,13 +163,16 @@ def test_stream_chat_with_one_document_retrieval(
output,
expected_chunks=["chunk1", "chunk2"],
expected_references=[
{"id": mock_document.pk, "title": "Test Document"},
{"id": document.pk, "title": "Test Document"},
],
)
@pytest.mark.django_db
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
doc1 = DocumentFactory.create(title="Document 1", content="ignored")
doc2 = DocumentFactory.create(title="Document 2", content="ignored")
documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
with (
patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock()
mock_node1 = TextNode(
text="Content for doc 1.",
metadata={"document_id": "1", "title": "Document 1"},
)
mock_node2 = TextNode(
text="Content for doc 2.",
metadata={"document_id": "2", "title": "Document 2"},
)
mock_index = MagicMock()
# Simulate get_nodes returning nodes (content exists)
mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2]
mock_index.vector_store.get_nodes.return_value = [
TextNode(
text="Content for doc 1.",
metadata={"document_id": str(doc1.pk), "title": "Document 1"},
),
TextNode(
text="Content for doc 2.",
metadata={"document_id": str(doc2.pk), "title": "Document 2"},
),
]
mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [
MagicMock(metadata={"document_id": "1", "title": "Document 1"}),
MagicMock(metadata={"document_id": "2", "title": "Document 2"}),
MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
]
mock_response_stream = MagicMock()
@@ -210,14 +210,11 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_query_engine_cls.return_value = mock_query_engine
mock_query_engine.query.return_value = mock_response_stream
doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
with patch(
"llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance,
):
output = list(stream_chat_with_documents("What's up?", [doc1, doc2]))
output = list(stream_chat_with_documents("What's up?", documents))
mock_query_engine.query.assert_called_once_with("What's up?")
patch_embed_nodes.assert_not_called()
@@ -225,15 +222,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
output,
expected_chunks=["chunk1", "chunk2"],
expected_references=[
{"id": 1, "title": "Document 1"},
{"id": 2, "title": "Document 2"},
{"id": doc1.pk, "title": "Document 1"},
{"id": doc2.pk, "title": "Document 2"},
],
)
def test_stream_chat_empty_document_list() -> None:
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
output = list(stream_chat_with_documents("Any info?", []))
output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
mock_load_index.assert_not_called()
assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -253,7 +250,9 @@ def test_stream_chat_no_matching_nodes() -> None:
mock_index.vector_store.get_nodes.return_value = []
mock_load_index.return_value = mock_index
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -282,7 +281,9 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
)
mock_retriever_cls.return_value = mock_retriever
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
assert output == [CHAT_ERROR_MESSAGE]
assert "Failed to stream document chat response" in caplog.text
@@ -298,7 +299,12 @@ class TestStreamChatRetrieval:
) -> None:
doc = DocumentFactory.create(content="hello world")
# Nothing indexed for this document yet.
out = list(chat.stream_chat_with_documents("question?", [doc]))
out = list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=doc.pk),
),
)
assert chat.CHAT_NO_CONTENT_MESSAGE in out
def test_chat_filter_contains_only_requested_document_ids(
@@ -332,7 +338,12 @@ class TestStreamChatRetrieval:
side_effect=capture_retriever,
)
list(chat.stream_chat_with_documents("question?", [included]))
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=included.pk),
),
)
assert captured_filters, "VectorIndexRetriever was never constructed"
filt = captured_filters[0]
@@ -340,3 +351,47 @@ class TestStreamChatRetrieval:
filter_values = filt.filters[0].value
assert str(included.pk) in filter_values
assert str(excluded.pk) not in filter_values
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
django_assert_num_queries,
) -> None:
"""Building references must not hydrate every document the caller is
permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
the retriever actually returned nodes for.
"""
referenced = DocumentFactory.create(title="Referenced Document")
# Many more documents are "accessible" but never referenced by a node.
DocumentFactory.create_batch(200)
documents = Document.objects.all()
top_nodes = [
MagicMock(
metadata={
"document_id": str(referenced.pk),
"title": "Referenced Document",
},
),
]
hydrated_count = 0
def _count_hydration(sender, instance, **kwargs):
nonlocal hydrated_count
hydrated_count += 1
post_init.connect(_count_hydration, sender=Document)
try:
# One query: `documents.filter(pk__in=candidate_ids)` for the single
# referenced id. No query should scale with the 200 unreferenced documents.
with django_assert_num_queries(1):
references = chat._get_document_references(documents, top_nodes)
finally:
post_init.disconnect(_count_hydration, sender=Document)
# The bug this guards against: the old code hydrated all 201 accessible
# documents via `{doc.pk: doc for doc in documents}` before filtering by
# top_nodes. Only the referenced document should ever be constructed.
assert hydrated_count == 1
assert references == [{"id": referenced.pk, "title": "Referenced Document"}]