mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-02 00:52:20 +00:00
Compare commits
116
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
82ae872b60 | ||
|
|
c05d069e97 | ||
|
|
f5e47a86c3 | ||
|
|
fe196eaf0e | ||
|
|
7fda0e877b | ||
|
|
5b8f4db1ca | ||
|
|
08ab98f3fc | ||
|
|
8fb73b2709 | ||
|
|
b8ab8327b0 | ||
|
|
3a0ca03544 | ||
|
|
882dfa78d2 | ||
|
|
a5645392a5 | ||
|
|
bbdf2cbd06 | ||
|
|
0bbe180f63 | ||
|
|
e35452beeb | ||
|
|
68a769b40c | ||
|
|
e5b27df99c | ||
|
|
95a48dd685 | ||
|
|
aa938de747 | ||
|
|
486b4babac | ||
|
|
adf1dd4e05 | ||
|
|
3d57f4396f | ||
|
|
752c949009 | ||
|
|
e37ef5aa71 | ||
|
|
919ffdd794 | ||
|
|
30fe172847 | ||
|
|
e79f0d4106 | ||
|
|
c29c9178f0 | ||
|
|
baae99eb49 | ||
|
|
f50a021ef6 | ||
|
|
c8e00dca04 | ||
|
|
0a56018d6b | ||
|
|
350684cd6b | ||
|
|
668fa77428 | ||
|
|
5bd72014a6 | ||
|
|
bbb9c86ba4 | ||
|
|
09c9fa03cf | ||
|
|
04779d72bb | ||
|
|
1b32b9d678 | ||
|
|
12d318deff | ||
|
|
9c98f30d3f | ||
|
|
6a1d7b1bca | ||
|
|
de9f520143 | ||
|
|
a68c487d60 | ||
|
|
217421bceb | ||
|
|
3bc03bbaec | ||
|
|
aaee24ac0b | ||
|
|
674dab29df | ||
|
|
b2b2f7b488 | ||
|
|
0b588c2d31 | ||
|
|
dcff067dc1 | ||
|
|
7eee8538ec | ||
|
|
98b66fdf24 | ||
|
|
693a111c5a | ||
|
|
b19edd0b74 | ||
|
|
0e98a7f1ce | ||
|
|
14517062c4 | ||
|
|
0c0faf7f80 | ||
|
|
eda79603fe | ||
|
|
8f017942d9 | ||
|
|
f4e7a2e9e4 | ||
|
|
317e534aa0 | ||
|
|
3ba6d0325a | ||
|
|
318520a0f1 | ||
|
|
ea1f3653a9 | ||
|
|
423d4b9f2d | ||
|
|
8bc4e5ee94 | ||
|
|
65fe6aacc7 | ||
|
|
b5a23cff2f | ||
|
|
aa90f79c21 | ||
|
|
e704bf88dc | ||
|
|
1646756c6c | ||
|
|
48d6ff13f2 | ||
|
|
bfe2d92619 | ||
|
|
4f0845b094 | ||
|
|
dee568e2a6 | ||
|
|
f60a2af841 | ||
|
|
33cd48a17f | ||
|
|
d76dbf5366 | ||
|
|
b66182cd7b | ||
|
|
3fe6562c57 | ||
|
|
737d568f80 | ||
|
|
b992f9fc05 | ||
|
|
ed714b8a5a | ||
|
|
e0ab6a2e75 | ||
|
|
6d249b4932 | ||
|
|
90ea4f27c4 | ||
|
|
371c4f57d3 | ||
|
|
5569447ea0 | ||
|
|
17796357ee | ||
|
|
b47b9800a1 | ||
|
|
0be884d441 | ||
|
|
fce2fbb8f6 | ||
|
|
fa01396dfd | ||
|
|
c9443e890f | ||
|
|
21e4721a82 | ||
|
|
1985da14f8 | ||
|
|
59a737f669 | ||
|
|
90bee4285f | ||
|
|
7dca0bfa6a | ||
|
|
3a34bdc7b0 | ||
|
|
c7899a9a84 | ||
|
|
82fc1c6cee | ||
|
|
cd3c162e6e | ||
|
|
50c88dab7a | ||
|
|
5744a75e6a | ||
|
|
02c6b5f3e4 | ||
|
|
b566232542 | ||
|
|
be40356d2e | ||
|
|
5ea8cf2335 | ||
|
|
84a04301d1 | ||
|
|
3edda48324 | ||
|
|
f86bc57880 | ||
|
|
97662b6c5c | ||
|
|
6e052d5507 | ||
|
|
afb19fe637 |
@@ -191,7 +191,7 @@ jobs:
|
||||
runs-on: ubuntu-24.04
|
||||
permissions:
|
||||
contents: read
|
||||
container: mcr.microsoft.com/playwright:v1.61.1-noble
|
||||
container: mcr.microsoft.com/playwright:v1.62.0-noble
|
||||
env:
|
||||
PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
|
||||
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
|
||||
|
||||
@@ -86,7 +86,7 @@ jobs:
|
||||
# ---- Prepare Release ----
|
||||
- name: Generate requirements file
|
||||
run: |
|
||||
uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt
|
||||
uv export --quiet --no-dev --all-extras --emit-index-url --format requirements-txt --output-file requirements.txt
|
||||
- name: Compile messages
|
||||
env:
|
||||
PAPERLESS_SECRET_KEY: "ci-release-not-a-real-secret"
|
||||
|
||||
@@ -56,7 +56,7 @@ services:
|
||||
environment:
|
||||
PAPERLESS_REDIS: redis://broker:6379
|
||||
PAPERLESS_DBHOST: db
|
||||
PAPERLESS_DBENGINE: postgres
|
||||
PAPERLESS_DBENGINE: postgresql
|
||||
env_file:
|
||||
- stack.env
|
||||
volumes:
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/command/with-contenv /usr/bin/bash
|
||||
# shellcheck shell=bash
|
||||
|
||||
declare -r log_prefix="[init-llmindex-migrate]"
|
||||
|
||||
echo "${log_prefix} Checking for pending LLM index migrations..."
|
||||
cd "${PAPERLESS_SRC_DIR}"
|
||||
if [[ -n "${USER_IS_NON_ROOT}" ]]; then
|
||||
python3 manage.py document_llmindex migrate
|
||||
else
|
||||
s6-setuidgid paperless python3 manage.py document_llmindex migrate
|
||||
fi
|
||||
@@ -0,0 +1 @@
|
||||
oneshot
|
||||
@@ -0,0 +1 @@
|
||||
/etc/s6-overlay/s6-rc.d/init-llmindex-migrate/run
|
||||
+28
-16
@@ -151,6 +151,23 @@ the background.
|
||||
|
||||
### Bare Metal Route {#bare-metal-updating}
|
||||
|
||||
!!! warning
|
||||
|
||||
Extracting a new release archive on top of an existing installation
|
||||
(e.g. `tar -xf paperless-ngx-vX.Y.Z.tar.xz -C /opt/paperless`) only adds
|
||||
and overwrites files -- it does not remove files that were deleted in
|
||||
the new release. Leftover files from an old version, such as
|
||||
superseded database migrations, can remain on disk and cause errors
|
||||
like `NodeNotFoundError` during `manage.py migrate`.
|
||||
|
||||
Before unpacking a new release over an existing bare-metal
|
||||
installation, remove the previous source tree first (everything
|
||||
except your `media`, `data`, and `consume` directories and your
|
||||
`paperless.conf`/`.env`), or unpack into a fresh directory and move
|
||||
your persistent data and configuration over. Installations updated via
|
||||
`git pull` on a clean checkout are not affected, since Git removes
|
||||
files that no longer exist upstream.
|
||||
|
||||
After grabbing the new release and unpacking the contents, do the
|
||||
following:
|
||||
|
||||
@@ -195,6 +212,16 @@ following:
|
||||
This is a no-op if the index is already up to date, so it is safe to
|
||||
run on every upgrade.
|
||||
|
||||
5. Migrate the LLM index if needed.
|
||||
|
||||
```shell-session
|
||||
cd src
|
||||
python3 manage.py document_llmindex migrate
|
||||
```
|
||||
|
||||
This is a no-op if the index schema is already current, so it is safe
|
||||
to run on every upgrade.
|
||||
|
||||
### Database Upgrades
|
||||
|
||||
Paperless-ngx is compatible with Django-supported versions of PostgreSQL and MariaDB and it is generally
|
||||
@@ -272,8 +299,6 @@ optional arguments:
|
||||
-sm, --split-manifest
|
||||
-z, --zip
|
||||
-zn, --zip-name
|
||||
--zip-compression
|
||||
--zip-compression-level
|
||||
--data-only
|
||||
--no-progress-bar
|
||||
--passphrase
|
||||
@@ -336,19 +361,6 @@ If `-z` or `--zip` is provided, the export will be a zip file
|
||||
in the target directory, named according to the current local date or the
|
||||
value set in `-zn` or `--zip-name`.
|
||||
|
||||
The compression method for the zip can be set with `--zip-compression`
|
||||
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
|
||||
`--zip-compression-level` (deflated: 0–9, bzip2: 1–9, zstd: -22–22; ignored
|
||||
for `stored` and `lzma`). Both options require `--zip`.
|
||||
|
||||
!!! warning
|
||||
|
||||
`zstd` compression requires Python 3.14 or newer on **both** the machine
|
||||
creating the export and any machine importing it. An archive compressed with
|
||||
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
|
||||
imported on a runtime that lacks the codec; the importer will refuse it with
|
||||
a clear error. The default `deflated` is universally readable.
|
||||
|
||||
If `--data-only` is provided, only the database will be exported. This option is intended
|
||||
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
|
||||
|
||||
@@ -530,7 +542,7 @@ index is updated automatically on the schedule set by
|
||||
can manage it manually:
|
||||
|
||||
```
|
||||
document_llmindex {rebuild,update,compact}
|
||||
document_llmindex {rebuild,update,compact,migrate}
|
||||
```
|
||||
|
||||
Specify `rebuild` to build the index from scratch from all documents in the database. Use
|
||||
|
||||
@@ -1,5 +1,204 @@
|
||||
# Changelog
|
||||
|
||||
## paperless-ngx 3.0.5
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix: accept Whoosh-era abbreviated relative-date units (yrs, mos, wks, etc) in search queries [@stumpylog](https://github.com/stumpylog) ([#13486](https://github.com/paperless-ngx/paperless-ngx/pull/13486))
|
||||
- Fix: fix edit dialog error change detection [@shamoon](https://github.com/shamoon) ([#13483](https://github.com/paperless-ngx/paperless-ngx/pull/13483))
|
||||
- Fix: key the AI suggestion cache by model and endpoint [@lunetics](https://github.com/lunetics) ([#13449](https://github.com/paperless-ngx/paperless-ngx/pull/13449))
|
||||
- Fix: validate custom field values in bulk operations [@shamoon](https://github.com/shamoon) ([#13457](https://github.com/paperless-ngx/paperless-ngx/pull/13457))
|
||||
- Fixhancement: better handle empty fields from AI suggestions [@shamoon](https://github.com/shamoon) ([#13454](https://github.com/paperless-ngx/paperless-ngx/pull/13454))
|
||||
- Fix: normalize monetary decimal symbol by locale [@shamoon](https://github.com/shamoon) ([#13427](https://github.com/paperless-ngx/paperless-ngx/pull/13427))
|
||||
- Fix: fold overflowing path text in sanity checker [@stumpylog](https://github.com/stumpylog) ([#13426](https://github.com/paperless-ngx/paperless-ngx/pull/13426))
|
||||
- Fix: correct delayed add field button for custom fields [@shamoon](https://github.com/shamoon) ([#13424](https://github.com/paperless-ngx/paperless-ngx/pull/13424))
|
||||
- Fix: hide attributes collapse / show button without UI settings [@shamoon](https://github.com/shamoon) ([#13425](https://github.com/paperless-ngx/paperless-ngx/pull/13425))
|
||||
- Fix: consolidate born-digital PDF detection between archive decision and OCR [@stumpylog](https://github.com/stumpylog) ([#13409](https://github.com/paperless-ngx/paperless-ngx/pull/13409))
|
||||
- Fix: prevent scale vs page loop in pngx PDF viewer [@shamoon](https://github.com/shamoon) ([#13406](https://github.com/paperless-ngx/paperless-ngx/pull/13406))
|
||||
- Fix: handle hidden line breaks in email subjects when sending email [@shamoon](https://github.com/shamoon) ([#13402](https://github.com/paperless-ngx/paperless-ngx/pull/13402))
|
||||
- Fix: avoid NotSupportedError from document\_importer on MariaDB [@stumpylog](https://github.com/stumpylog) ([#13400](https://github.com/paperless-ngx/paperless-ngx/pull/13400))
|
||||
- Fix: exclude next-period start from relative date-range filters [@stumpylog](https://github.com/stumpylog) ([#13381](https://github.com/paperless-ngx/paperless-ngx/pull/13381))
|
||||
- Fix: close non-atomic db connections in before\_task\_publish [@shamoon](https://github.com/shamoon) ([#13366](https://github.com/paperless-ngx/paperless-ngx/pull/13366))
|
||||
- Chore/fix: refactor frontend task service [@shamoon](https://github.com/shamoon) ([#13365](https://github.com/paperless-ngx/paperless-ngx/pull/13365))
|
||||
- Fix: fix Enter selection in search autocomplete [@shamoon](https://github.com/shamoon) ([#13361](https://github.com/paperless-ngx/paperless-ngx/pull/13361))
|
||||
|
||||
### Documentation
|
||||
|
||||
- Documentation: PAPERLESS\_CONSUMER\_IGNORE\_PATTERNS clarifications [@shamoon](https://github.com/shamoon) ([#13489](https://github.com/paperless-ngx/paperless-ngx/pull/13489))
|
||||
- Documentation: Add Password Removal workflow action documentation [@stumpylog](https://github.com/stumpylog) ([#13377](https://github.com/paperless-ngx/paperless-ngx/pull/13377))
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Chore(deps): Bump pymdown-extensions from 10.21.3 to 11.0 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13378](https://github.com/paperless-ngx/paperless-ngx/pull/13378))
|
||||
|
||||
### All App Changes
|
||||
|
||||
<details>
|
||||
<summary>21 changes</summary>
|
||||
|
||||
- Fix: accept Whoosh-era abbreviated relative-date units (yrs, mos, wks, etc) in search queries [@stumpylog](https://github.com/stumpylog) ([#13486](https://github.com/paperless-ngx/paperless-ngx/pull/13486))
|
||||
- Fix: fix edit dialog error change detection [@shamoon](https://github.com/shamoon) ([#13483](https://github.com/paperless-ngx/paperless-ngx/pull/13483))
|
||||
- Performance: sqlite-vec point-delete for document chunks [@stumpylog](https://github.com/stumpylog) ([#13438](https://github.com/paperless-ngx/paperless-ngx/pull/13438))
|
||||
- Fix: key the AI suggestion cache by model and endpoint [@lunetics](https://github.com/lunetics) ([#13449](https://github.com/paperless-ngx/paperless-ngx/pull/13449))
|
||||
- Fix: validate custom field values in bulk operations [@shamoon](https://github.com/shamoon) ([#13457](https://github.com/paperless-ngx/paperless-ngx/pull/13457))
|
||||
- Fixhancement: better handle empty fields from AI suggestions [@shamoon](https://github.com/shamoon) ([#13454](https://github.com/paperless-ngx/paperless-ngx/pull/13454))
|
||||
- Performance: Use server side iterators during LLM index updating [@stumpylog](https://github.com/stumpylog) ([#13430](https://github.com/paperless-ngx/paperless-ngx/pull/13430))
|
||||
- Fix: normalize monetary decimal symbol by locale [@shamoon](https://github.com/shamoon) ([#13427](https://github.com/paperless-ngx/paperless-ngx/pull/13427))
|
||||
- Fix: fold overflowing path text in sanity checker [@stumpylog](https://github.com/stumpylog) ([#13426](https://github.com/paperless-ngx/paperless-ngx/pull/13426))
|
||||
- Fix: correct delayed add field button for custom fields [@shamoon](https://github.com/shamoon) ([#13424](https://github.com/paperless-ngx/paperless-ngx/pull/13424))
|
||||
- Fix: hide attributes collapse / show button without UI settings [@shamoon](https://github.com/shamoon) ([#13425](https://github.com/paperless-ngx/paperless-ngx/pull/13425))
|
||||
- Fix: consolidate born-digital PDF detection between archive decision and OCR [@stumpylog](https://github.com/stumpylog) ([#13409](https://github.com/paperless-ngx/paperless-ngx/pull/13409))
|
||||
- Fix: prevent scale vs page loop in pngx PDF viewer [@shamoon](https://github.com/shamoon) ([#13406](https://github.com/paperless-ngx/paperless-ngx/pull/13406))
|
||||
- Fix: handle hidden line breaks in email subjects when sending email [@shamoon](https://github.com/shamoon) ([#13402](https://github.com/paperless-ngx/paperless-ngx/pull/13402))
|
||||
- Fix: avoid NotSupportedError from document\_importer on MariaDB [@stumpylog](https://github.com/stumpylog) ([#13400](https://github.com/paperless-ngx/paperless-ngx/pull/13400))
|
||||
- Tweak: adjust doc details button toolbar flow [@shamoon](https://github.com/shamoon) ([#13382](https://github.com/paperless-ngx/paperless-ngx/pull/13382))
|
||||
- Fix: exclude next-period start from relative date-range filters [@stumpylog](https://github.com/stumpylog) ([#13381](https://github.com/paperless-ngx/paperless-ngx/pull/13381))
|
||||
- Chore(deps): Bump pymdown-extensions from 10.21.3 to 11.0 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13378](https://github.com/paperless-ngx/paperless-ngx/pull/13378))
|
||||
- Fix: close non-atomic db connections in before\_task\_publish [@shamoon](https://github.com/shamoon) ([#13366](https://github.com/paperless-ngx/paperless-ngx/pull/13366))
|
||||
- Chore/fix: refactor frontend task service [@shamoon](https://github.com/shamoon) ([#13365](https://github.com/paperless-ngx/paperless-ngx/pull/13365))
|
||||
- Fix: fix Enter selection in search autocomplete [@shamoon](https://github.com/shamoon) ([#13361](https://github.com/paperless-ngx/paperless-ngx/pull/13361))
|
||||
|
||||
</details>
|
||||
|
||||
## paperless-ngx 3.0.4
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
|
||||
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
|
||||
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
|
||||
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
|
||||
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
|
||||
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
|
||||
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
|
||||
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
|
||||
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
|
||||
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
|
||||
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
|
||||
|
||||
### Documentation
|
||||
|
||||
- Documentation: fix PAPERLESS\_AI\_LLM\_OUTPUT\_LANGUAGE heading level [@SuperSandro2000](https://github.com/SuperSandro2000) ([#13341](https://github.com/paperless-ngx/paperless-ngx/pull/13341))
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
|
||||
|
||||
### All App Changes
|
||||
|
||||
<details>
|
||||
<summary>14 changes</summary>
|
||||
|
||||
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
|
||||
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
|
||||
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
|
||||
- Performance: prefetch notes and custom fields for LLM index text building [@stumpylog](https://github.com/stumpylog) ([#13350](https://github.com/paperless-ngx/paperless-ngx/pull/13350))
|
||||
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
|
||||
- Performance: Scope llm index updates to actually modified documents [@stumpylog](https://github.com/stumpylog) ([#13322](https://github.com/paperless-ngx/paperless-ngx/pull/13322))
|
||||
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
|
||||
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
|
||||
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
|
||||
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
|
||||
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
|
||||
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
|
||||
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
|
||||
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
|
||||
|
||||
</details>
|
||||
|
||||
## paperless-ngx 3.0.3
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fixhancement: PAPERLESS\_ALLAUTH\_TRUSTED\_PROXY\_COUNT [@shamoon](https://github.com/shamoon) ([#13281](https://github.com/paperless-ngx/paperless-ngx/pull/13281))
|
||||
- Fix: preserve document fields during Gotenberg conversion to PDF [@stumpylog](https://github.com/stumpylog) ([#13271](https://github.com/paperless-ngx/paperless-ngx/pull/13271))
|
||||
- Fix: Makes the email date aware as soon as possible during parsing [@stumpylog](https://github.com/stumpylog) ([#13266](https://github.com/paperless-ngx/paperless-ngx/pull/13266))
|
||||
- Fix: Handle a plain string as Celery sometimes provides for the traceback [@stumpylog](https://github.com/stumpylog) ([#13267](https://github.com/paperless-ngx/paperless-ngx/pull/13267))
|
||||
- Fix: Emit the torch index into the requirements.txt for people still using it [@stumpylog](https://github.com/stumpylog) ([#13265](https://github.com/paperless-ngx/paperless-ngx/pull/13265))
|
||||
- Fix: handle notes without a user when building the search index [@matthiasmast](https://github.com/matthiasmast) ([#13260](https://github.com/paperless-ngx/paperless-ngx/pull/13260))
|
||||
|
||||
### Documentation
|
||||
|
||||
- Docs: warn bare-metal users about stale files when upgrading [@stumpylog](https://github.com/stumpylog) ([#13296](https://github.com/paperless-ngx/paperless-ngx/pull/13296))
|
||||
- Documentation: Add the NumPy CPU baseline increase to the migration guide [@stumpylog](https://github.com/stumpylog) ([#13269](https://github.com/paperless-ngx/paperless-ngx/pull/13269))
|
||||
|
||||
### Maintenance
|
||||
|
||||
- Fix: Emit the torch index into the requirements.txt for people still using it [@stumpylog](https://github.com/stumpylog) ([#13265](https://github.com/paperless-ngx/paperless-ngx/pull/13265))
|
||||
|
||||
### All App Changes
|
||||
|
||||
<details>
|
||||
<summary>6 changes</summary>
|
||||
|
||||
- Fixhancement: PAPERLESS\_ALLAUTH\_TRUSTED\_PROXY\_COUNT [@shamoon](https://github.com/shamoon) ([#13281](https://github.com/paperless-ngx/paperless-ngx/pull/13281))
|
||||
- Tweak: adjust top navbar wrapping when AI chat button visible [@shamoon](https://github.com/shamoon) ([#13280](https://github.com/paperless-ngx/paperless-ngx/pull/13280))
|
||||
- Fix: preserve document fields during Gotenberg conversion to PDF [@stumpylog](https://github.com/stumpylog) ([#13271](https://github.com/paperless-ngx/paperless-ngx/pull/13271))
|
||||
- Fix: Makes the email date aware as soon as possible during parsing [@stumpylog](https://github.com/stumpylog) ([#13266](https://github.com/paperless-ngx/paperless-ngx/pull/13266))
|
||||
- Fix: Handle a plain string as Celery sometimes provides for the traceback [@stumpylog](https://github.com/stumpylog) ([#13267](https://github.com/paperless-ngx/paperless-ngx/pull/13267))
|
||||
- Fix: handle notes without a user when building the search index [@matthiasmast](https://github.com/matthiasmast) ([#13260](https://github.com/paperless-ngx/paperless-ngx/pull/13260))
|
||||
|
||||
</details>
|
||||
|
||||
## paperless-ngx 3.0.2
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix: fix broken migration in 3.0.1 [@shamoon](https://github.com/shamoon) ([#13242](https://github.com/paperless-ngx/paperless-ngx/pull/13242))
|
||||
|
||||
### All App Changes
|
||||
|
||||
- Fix: fix broken migration in 3.0.1 [@shamoon](https://github.com/shamoon) ([#13242](https://github.com/paperless-ngx/paperless-ngx/pull/13242))
|
||||
|
||||
## paperless-ngx 3.0.1
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix: selection\_data re-derives the filtered document set 5 times over [@stumpylog](https://github.com/stumpylog) ([#13229](https://github.com/paperless-ngx/paperless-ngx/pull/13229))
|
||||
- Fix: exclude source document from its own RAG similarity results [@stumpylog](https://github.com/stumpylog) ([#13233](https://github.com/paperless-ngx/paperless-ngx/pull/13233))
|
||||
- Fix: split mailrule maximum\_age clamp into its own migration [@stumpylog](https://github.com/stumpylog) ([#13231](https://github.com/paperless-ngx/paperless-ngx/pull/13231))
|
||||
- Chore: mark yes in confirm button for translation [@shamoon](https://github.com/shamoon) ([#13225](https://github.com/paperless-ngx/paperless-ngx/pull/13225))
|
||||
- Fix: correct URL for W001 check [@shamoon](https://github.com/shamoon) ([#13220](https://github.com/paperless-ngx/paperless-ngx/pull/13220))
|
||||
- Fix: ensure create dialog uses correct attribute type [@shamoon](https://github.com/shamoon) ([#13221](https://github.com/paperless-ngx/paperless-ngx/pull/13221))
|
||||
- Fix: correct database engine from postgres to postgresql [@OberstVonGatow](https://github.com/OberstVonGatow) ([#13213](https://github.com/paperless-ngx/paperless-ngx/pull/13213))
|
||||
- Fix: remove unnecessary .distinct() dominating document list queries [@stumpylog](https://github.com/stumpylog) ([#13205](https://github.com/paperless-ngx/paperless-ngx/pull/13205))
|
||||
- Fix: also wrap non-breaking words in chat messages, handle whitespace [@shamoon](https://github.com/shamoon) ([#13211](https://github.com/paperless-ngx/paperless-ngx/pull/13211))
|
||||
- Fix: tag/custom-field document\_count scales badly with tag count [@stumpylog](https://github.com/stumpylog) ([#13203](https://github.com/paperless-ngx/paperless-ngx/pull/13203))
|
||||
- Fix: batch document user\_can\_change checks to avoid per-row N+1 [@stumpylog](https://github.com/stumpylog) ([#13204](https://github.com/paperless-ngx/paperless-ngx/pull/13204))
|
||||
- Fix: fix app title restoration [@shamoon](https://github.com/shamoon) ([#13208](https://github.com/paperless-ngx/paperless-ngx/pull/13208))
|
||||
- Fix: handle long wrapping titles in AI chat document list [@shamoon](https://github.com/shamoon) ([#13206](https://github.com/paperless-ngx/paperless-ngx/pull/13206))
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Chore(deps-dev): Bump postcss from 8.5.6 to 8.5.22 in /src/paperless\_mail/templates in the npm\_and\_yarn group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13236](https://github.com/paperless-ngx/paperless-ngx/pull/13236))
|
||||
- Chore(deps): Bump pyasn1 from 0.6.3 to 0.6.4 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13237](https://github.com/paperless-ngx/paperless-ngx/pull/13237))
|
||||
- Chore(deps): Bump the uv group across 1 directory with 8 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#13189](https://github.com/paperless-ngx/paperless-ngx/pull/13189))
|
||||
|
||||
### All App Changes
|
||||
|
||||
<details>
|
||||
<summary>17 changes</summary>
|
||||
|
||||
- Chore(deps-dev): Bump postcss from 8.5.6 to 8.5.22 in /src/paperless\_mail/templates in the npm\_and\_yarn group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13236](https://github.com/paperless-ngx/paperless-ngx/pull/13236))
|
||||
- Fix: selection\_data re-derives the filtered document set 5 times over [@stumpylog](https://github.com/stumpylog) ([#13229](https://github.com/paperless-ngx/paperless-ngx/pull/13229))
|
||||
- Chore: reword "Settings > Default permissions" description for clarity [@NotaInutilis](https://github.com/NotaInutilis) ([#13232](https://github.com/paperless-ngx/paperless-ngx/pull/13232))
|
||||
- Chore(deps): Bump pyasn1 from 0.6.3 to 0.6.4 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13237](https://github.com/paperless-ngx/paperless-ngx/pull/13237))
|
||||
- Chore(deps): Bump the uv group across 1 directory with 8 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#13189](https://github.com/paperless-ngx/paperless-ngx/pull/13189))
|
||||
- Fix: exclude source document from its own RAG similarity results [@stumpylog](https://github.com/stumpylog) ([#13233](https://github.com/paperless-ngx/paperless-ngx/pull/13233))
|
||||
- Fix: split mailrule maximum\_age clamp into its own migration [@stumpylog](https://github.com/stumpylog) ([#13231](https://github.com/paperless-ngx/paperless-ngx/pull/13231))
|
||||
- Chore: mark yes in confirm button for translation [@shamoon](https://github.com/shamoon) ([#13225](https://github.com/paperless-ngx/paperless-ngx/pull/13225))
|
||||
- Fix: correct URL for W001 check [@shamoon](https://github.com/shamoon) ([#13220](https://github.com/paperless-ngx/paperless-ngx/pull/13220))
|
||||
- Fix: ensure create dialog uses correct attribute type [@shamoon](https://github.com/shamoon) ([#13221](https://github.com/paperless-ngx/paperless-ngx/pull/13221))
|
||||
- Fix: correct database engine from postgres to postgresql [@OberstVonGatow](https://github.com/OberstVonGatow) ([#13213](https://github.com/paperless-ngx/paperless-ngx/pull/13213))
|
||||
- Fix: remove unnecessary .distinct() dominating document list queries [@stumpylog](https://github.com/stumpylog) ([#13205](https://github.com/paperless-ngx/paperless-ngx/pull/13205))
|
||||
- Fix: also wrap non-breaking words in chat messages, handle whitespace [@shamoon](https://github.com/shamoon) ([#13211](https://github.com/paperless-ngx/paperless-ngx/pull/13211))
|
||||
- Fix: tag/custom-field document\_count scales badly with tag count [@stumpylog](https://github.com/stumpylog) ([#13203](https://github.com/paperless-ngx/paperless-ngx/pull/13203))
|
||||
- Fix: batch document user\_can\_change checks to avoid per-row N+1 [@stumpylog](https://github.com/stumpylog) ([#13204](https://github.com/paperless-ngx/paperless-ngx/pull/13204))
|
||||
- Fix: fix app title restoration [@shamoon](https://github.com/shamoon) ([#13208](https://github.com/paperless-ngx/paperless-ngx/pull/13208))
|
||||
- Fix: handle long wrapping titles in AI chat document list [@shamoon](https://github.com/shamoon) ([#13206](https://github.com/paperless-ngx/paperless-ngx/pull/13206))
|
||||
|
||||
</details>
|
||||
|
||||
## paperless-ngx 3.0.0
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
+33
-12
@@ -522,22 +522,36 @@ do CORS calls. Set this to your public domain name.
|
||||
fail2ban with log entries for failed authorization attempts. Value should be
|
||||
IP address(es).
|
||||
|
||||
This setting also controls allauth's
|
||||
[`ALLAUTH_TRUSTED_PROXY_COUNT`](https://docs.allauth.org/en/latest/account/configuration.html),
|
||||
which is set to the number of proxies listed here. Without this,
|
||||
allauth cannot determine the client IP address for rate limiting when
|
||||
running behind a reverse proxy, resulting in a `403 Forbidden` on login.
|
||||
By default, this setting also controls allauth's trusted proxy count,
|
||||
which is set to the number of proxies listed here. Override that default
|
||||
with [`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT`](#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT)
|
||||
when the list length does not match the number of proxy hops.
|
||||
|
||||
Defaults to empty string.
|
||||
|
||||
#### [`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT=<integer>`](#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT) {#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT}
|
||||
|
||||
: Sets allauth's
|
||||
[`ALLAUTH_TRUSTED_PROXY_COUNT`](https://docs.allauth.org/en/latest/common/rate_limits.html#configuration).
|
||||
This is the number of trusted proxy **hops** represented in each
|
||||
`X-Forwarded-For` header, not the number of IP addresses through which those
|
||||
proxies may be reached. For example, a single dual-stack proxy is one hop even
|
||||
when its IPv4 and IPv6 addresses are both listed in
|
||||
[`PAPERLESS_TRUSTED_PROXIES`](#PAPERLESS_TRUSTED_PROXIES).
|
||||
|
||||
Only trust `X-Forwarded-For` when untrusted clients cannot connect directly
|
||||
to Paperless-ngx.
|
||||
|
||||
Defaults to the number of entries in `PAPERLESS_TRUSTED_PROXIES`.
|
||||
|
||||
#### [`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=<header-name>`](#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER) {#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER}
|
||||
|
||||
: Sets allauth's
|
||||
[`ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](https://docs.allauth.org/en/latest/account/configuration.html).
|
||||
[`ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](https://docs.allauth.org/en/latest/common/rate_limits.html#configuration).
|
||||
Use this when your reverse proxy sets a dedicated header for the real
|
||||
client IP instead of `X-Forwarded-For`, for example `X-Real-IP` (nginx)
|
||||
or `CF-Connecting-IP` (Cloudflare). When set, this takes precedence over
|
||||
[`PAPERLESS_TRUSTED_PROXIES`](#PAPERLESS_TRUSTED_PROXIES).
|
||||
`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT`.
|
||||
|
||||
Defaults to none.
|
||||
|
||||
@@ -918,8 +932,8 @@ for display in the web interface.
|
||||
| Document type | `never` | `auto` (default) | `always` |
|
||||
| -------------------------- | ------- | -------------------------- | -------- |
|
||||
| Scanned image (TIFF, JPEG) | No | **Yes** | Yes |
|
||||
| Image-based PDF | No | **Yes** (short/no text, untagged) | Yes |
|
||||
| Born-digital PDF | No | No (tagged or has embedded text) | Yes |
|
||||
| Image-based PDF | No | **Yes** (no embedded text) | Yes |
|
||||
| Born-digital PDF | No | No (has embedded text, optionally confirmed by tag) | Yes |
|
||||
| Plain text, email, HTML | No | No | No |
|
||||
| DOCX / ODT (via Tika) | Yes\* | Yes\* | Yes\* |
|
||||
|
||||
@@ -1346,13 +1360,16 @@ don't exist yet.
|
||||
#### [`PAPERLESS_CONSUMER_IGNORE_PATTERNS=<json>`](#PAPERLESS_CONSUMER_IGNORE_PATTERNS) {#PAPERLESS_CONSUMER_IGNORE_PATTERNS}
|
||||
|
||||
: Additional regex patterns for files to ignore in the consumption directory. Patterns are matched against filenames only (not full paths)
|
||||
using Python's `re.match()`, which anchors at the start of the filename.
|
||||
using Python's `re.search()`. Use `^` to anchor a pattern to the start of the filename and `$` to anchor it to the end.
|
||||
|
||||
See the [watchfiles documentation](https://watchfiles.helpmanual.io/api/filters/#watchfiles.BaseFilter.ignore_entity_patterns)
|
||||
|
||||
This setting is for additional patterns beyond the built-in defaults. Common system files and directories are already ignored automatically.
|
||||
The patterns will be compiled via Python's standard `re` module.
|
||||
|
||||
These are regular expressions, not glob patterns. For example, the glob pattern `._*` does not mean "starts with `._`" when used as a
|
||||
regular expression; it matches nearly any non-empty filename. Use `^\._.*` for that behavior instead.
|
||||
|
||||
Example custom patterns:
|
||||
|
||||
```json
|
||||
@@ -1367,7 +1384,11 @@ using Python's `re.match()`, which anchors at the start of the filename.
|
||||
|
||||
Defaults to `[]` (empty list, uses only built-in defaults).
|
||||
|
||||
The default ignores are `[.DS_Store, .DS_STORE, ._*, desktop.ini, Thumbs.db]` and cannot be overridden.
|
||||
The built-in file patterns are equivalent to the following regular expressions and cannot be overridden:
|
||||
|
||||
```json
|
||||
["^\\.DS_Store$", "^\\.DS_STORE$", "^\\._.*", "^desktop\\.ini$", "^Thumbs\\.db$"]
|
||||
```
|
||||
|
||||
#### [`PAPERLESS_CONSUMER_IGNORE_DIRS=<json>`](#PAPERLESS_CONSUMER_IGNORE_DIRS) {#PAPERLESS_CONSUMER_IGNORE_DIRS}
|
||||
|
||||
@@ -2121,7 +2142,7 @@ used with the OpenAI-compatible backend to target a custom provider or local gat
|
||||
|
||||
Defaults to None.
|
||||
|
||||
### [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=<str>`](#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE) {#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE}
|
||||
#### [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=<str>`](#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE) {#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE}
|
||||
|
||||
: The language to use for AI suggestions (results may vary by LLM model). If not supplied, defaults to the user's UI language setting or None.
|
||||
|
||||
|
||||
+65
-4
@@ -133,7 +133,7 @@ PAPERLESS_DB_OPTIONS="sslmode=require,sslrootcert=/certs/ca.pem,pool.max_size=10
|
||||
|
||||
## OCR and Archive File Generation Settings
|
||||
|
||||
The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** — old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
|
||||
The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** - old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
|
||||
|
||||
### Removed settings
|
||||
|
||||
@@ -166,7 +166,7 @@ Remove any `PAPERLESS_OCR_SKIP_ARCHIVE_FILE` variable from your environment. If
|
||||
# v2: skip OCR when text present, always archive
|
||||
PAPERLESS_OCR_MODE=skip
|
||||
# v3: equivalent (auto is the new default)
|
||||
# No change needed — auto is the default
|
||||
# No change needed - auto is the default
|
||||
|
||||
# v2: skip OCR when text present, skip archive too
|
||||
PAPERLESS_OCR_MODE=skip_noarchive
|
||||
@@ -189,7 +189,7 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
|
||||
|
||||
If you use the **remote OCR parser** (Azure AI), note that it always produces a
|
||||
searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
|
||||
has no effect for documents handled by the remote parser — the archive is produced
|
||||
has no effect for documents handled by the remote parser - the archive is produced
|
||||
unconditionally by the remote engine.
|
||||
|
||||
## Search Index (Whoosh -> Tantivy)
|
||||
@@ -324,8 +324,69 @@ option since v1.8.0.
|
||||
Allauth changed how it determines the client IP address for login rate limiting. Users running
|
||||
behind a reverse proxy may need to set
|
||||
[`PAPERLESS_TRUSTED_PROXIES`](configuration.md#PAPERLESS_TRUSTED_PROXIES),
|
||||
[`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT),
|
||||
[`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER),
|
||||
or both, to avoid `403 Forbidden` errors on login.
|
||||
or a combination of these settings to avoid `403 Forbidden` errors on login.
|
||||
The proxy count is the number of proxy hops in `X-Forwarded-For`, which may
|
||||
differ from the number of configured proxy IP addresses.
|
||||
|
||||
## Minimum CPU Requirements (NumPy Baseline)
|
||||
|
||||
Starting with NumPy 2.4.0, official `manylinux` x86_64 wheels are compiled with a minimum
|
||||
CPU baseline of `x86-64-v2`, which requires SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, and
|
||||
CMPXCHG16B. This is a [deliberate upstream change](https://github.com/numpy/numpy/issues/27851)
|
||||
citing that these instructions have been present in over 99.7% of CPUs since 2008
|
||||
([Intel](<https://en.wikipedia.org/wiki/Nehalem_(microarchitecture)>)) or 2011
|
||||
([AMD](<https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)>)).
|
||||
|
||||
NumPy is a dependency of the document classifier (via scikit-learn), so any CPU that
|
||||
predates SSE4.2 support will crash with `SIGILL` (illegal instruction) when the classifier
|
||||
is loaded or trained, regardless of whether AI features are enabled.
|
||||
|
||||
This differs from NumPy's optional SIMD dispatch (e.g. AVX2, AVX512), which is detected and
|
||||
selected safely at runtime - the `x86-64-v2` requirement above is a hard floor baked into the
|
||||
wheel, with no runtime fallback.
|
||||
|
||||
### Affected hardware
|
||||
|
||||
CPUs older than roughly 2008 (Intel) or 2011 (AMD) that lack SSE4.2 support. This is more
|
||||
likely to affect low-power or embedded hardware - e.g. early Atom, Celeron, or pre-Bulldozer
|
||||
AMD chips - than typical desktop or server hardware from the last decade.
|
||||
|
||||
Check for SSE4.2 support with:
|
||||
|
||||
```bash
|
||||
grep -o -m1 sse4_2 /proc/cpuinfo
|
||||
```
|
||||
|
||||
If this prints nothing, your CPU is affected.
|
||||
|
||||
### Symptoms
|
||||
|
||||
The Celery worker (and potentially the web server) repeatedly crashes and restarts with a
|
||||
`SIGILL` error, typically visible in `dmesg`/`journalctl` as a `trap invalid opcode` inside
|
||||
`_multiarray_umath...so`. Because the classifier is trained on a periodic schedule
|
||||
(hourly, by default), affected instances see intermittent, hard-to-reproduce document
|
||||
consumption failures whenever that scheduled task runs and takes down the worker process
|
||||
mid-task.
|
||||
|
||||
### Action Required (for affected hardware only)
|
||||
|
||||
There is no way to make the classifier itself work on such CPUs - it requires an unofficial
|
||||
NumPy build with `cpu-baseline=none`, which is not something we can ship. The practical
|
||||
path forward is to stop the classifier from ever loading or training, which avoids
|
||||
importing NumPy at all:
|
||||
|
||||
```bash
|
||||
PAPERLESS_TRAIN_TASK_CRON=disable
|
||||
```
|
||||
|
||||
This disables the periodic classifier training task (see
|
||||
[`PAPERLESS_TRAIN_TASK_CRON`](configuration.md#PAPERLESS_TRAIN_TASK_CRON)). Automatic
|
||||
matching based on the classifier (suggested correspondents, document types, tags, and
|
||||
storage paths from trained rules) will no longer be available, but rule-based matching is
|
||||
unaffected, and document consumption itself will no longer be at risk of crashing the
|
||||
worker.
|
||||
|
||||
## Database Migrations
|
||||
|
||||
|
||||
@@ -262,6 +262,14 @@ to a positive number to enable polling and disable native filesystem notificatio
|
||||
and copy the contents to the home directory of the user you created
|
||||
earlier (`/opt/paperless`).
|
||||
|
||||
!!! note
|
||||
|
||||
If you are updating an existing bare-metal installation rather than
|
||||
installing for the first time, see the
|
||||
[bare metal update instructions](administration.md#bare-metal-updating)
|
||||
instead -- extracting a new release on top of an old one can leave
|
||||
behind stale files from the previous version.
|
||||
|
||||
Optional: If you cloned the Git repository, you will need to
|
||||
compile the frontend yourself. See [here](development.md#front-end-development)
|
||||
and use the `build` step, not `serve`.
|
||||
|
||||
@@ -1,839 +0,0 @@
|
||||
# Export Zip Compression Control Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add `--zip-compression {stored,deflated,bzip2,lzma,zstd}` and `--zip-compression-level N` flags to `document_exporter`, threaded into `ZipExportSink`, with import-side safety for codecs the running Python can't read.
|
||||
|
||||
**Architecture:** A new pure-data module `documents/export/compression.py` owns the method↔constant map, per-method level bounds, the runtime availability probe, and a compress-type readability check. `ZipExportSink` gains `compression`/`compresslevel` constructor params. The command validates flags up front (fail-fast `CommandError`) and constructs the sink; the importer pre-checks entry compress types before extracting.
|
||||
|
||||
**Tech Stack:** Python ≥3.11 (zstd only on 3.14+), `zipfile`, `compression.zstd` (PEP 784), pytest + pytest-mock + factory-boy. Backend tests run on the Linux VM (Python 3.11 — zstd positive tests are `skipif`-guarded); `ruff` runs locally.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-16-export-zip-compression-design.md`
|
||||
|
||||
**PREREQUISITE:** The base refactor `docs/superpowers/plans/2026-06-16-export-sink-architecture.md` MUST be merged first. This plan assumes `src/documents/export/sinks.py` exists with `ZipExportSink(target, zip_name, *, delete=False)` opening its `ZipFile` in `_open()`.
|
||||
|
||||
---
|
||||
|
||||
## Verified facts (CPython 3.14.3, via `uv run --python 3.14 --no-project`)
|
||||
|
||||
- Constants: `ZIP_STORED=0`, `ZIP_DEFLATED=8`, `ZIP_BZIP2=12`, `ZIP_LZMA=14`, `ZIP_ZSTANDARD=93` (zstd added 3.14; absent on < 3.14).
|
||||
- `ZipFile(file, "w", compression=…, compresslevel=…)` applies both as the default for every `write`/`writestr` — no per-entry args needed (verified).
|
||||
- Level bounds: `deflated` 0–9, `bzip2` 1–9, `lzma`/`stored` ignore level, `zstd` -131072…22 (`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`).
|
||||
- An invalid level fails at the **first write** (`ValueError: Invalid initialization option` / `compresslevel must be between 1 and 9`), plus GC-time `AttributeError` noise on close — hence up-front validation.
|
||||
- zstd is backed by `compression.zstd`; `zipfile` raises `RuntimeError` if it's unavailable.
|
||||
|
||||
## Conventions for every task
|
||||
|
||||
- **Run backend tests on the VM:** `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "<targets>"` (never locally).
|
||||
- **Lint locally:** `ruff check <paths> && ruff format <paths>` (global ruff, not `uv run`).
|
||||
- **Tests are pytest-style:** classes, `@pytest.mark.django_db` on the class only where DB is needed (the `compression.py` and sink tests need no DB), factory-boy, `mocker`, `parametrize`, full type annotations.
|
||||
- The VM runs Python 3.11, so **zstd positive tests must be `@pytest.mark.skipif(...)`-guarded**; they will simply not run there. zstd _rejection_ tests (the < 3.14 path) DO run on the VM.
|
||||
|
||||
## File structure
|
||||
|
||||
- **Create** `src/documents/export/compression.py` — method map, CLI choices, level bounds, `compression_available()`, `level_error()`, `compress_type_readable()`, `unreadable_method_names()`. Pure, no Django.
|
||||
- **Create** `src/documents/tests/export/test_compression.py` — unit tests for the above.
|
||||
- **Modify** `src/documents/export/sinks.py` — `ZipExportSink.__init__` gains `compression`/`compresslevel`; `_open()` passes them to `ZipFile`.
|
||||
- **Modify** `src/documents/tests/export/test_sinks.py` — assert the chosen `compress_type` is applied.
|
||||
- **Modify** `src/documents/management/commands/document_exporter.py` — add the two CLI flags, up-front validation, and pass resolved values to `ZipExportSink`.
|
||||
- **Modify** `src/documents/tests/test_management_exporter.py` — flag validation + default-unchanged tests.
|
||||
- **Modify** `src/documents/management/commands/document_importer.py` — pre-extract compress-type check.
|
||||
- **Modify** `src/documents/tests/test_management_importer.py` — unsupported-codec → `CommandError`.
|
||||
- **Modify** `docs/administration.md` — document both flags + zstd portability caveat.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: `documents/export/compression.py` (pure compression policy)
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `src/documents/export/compression.py`
|
||||
- Test: `src/documents/tests/export/test_compression.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Create `src/documents/tests/export/test_compression.py`:
|
||||
|
||||
```python
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
import pytest
|
||||
|
||||
from documents.export import compression
|
||||
|
||||
|
||||
class TestCompressionMethods:
|
||||
def test_choices_always_include_zstd(self) -> None:
|
||||
# zstd is offered regardless of runtime; availability is checked separately
|
||||
assert compression.COMPRESSION_CHOICES == (
|
||||
"stored",
|
||||
"deflated",
|
||||
"bzip2",
|
||||
"lzma",
|
||||
"zstd",
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "constant"),
|
||||
[
|
||||
("stored", zipfile.ZIP_STORED),
|
||||
("deflated", zipfile.ZIP_DEFLATED),
|
||||
("bzip2", zipfile.ZIP_BZIP2),
|
||||
("lzma", zipfile.ZIP_LZMA),
|
||||
],
|
||||
)
|
||||
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
|
||||
assert compression.COMPRESSION_METHODS[name] == constant
|
||||
|
||||
def test_stored_and_deflated_always_available(self) -> None:
|
||||
assert compression.compression_available("stored")
|
||||
assert compression.compression_available("deflated")
|
||||
|
||||
def test_zstd_availability_tracks_runtime(self) -> None:
|
||||
expected: bool = sys.version_info >= (3, 14)
|
||||
assert compression.compression_available("zstd") == expected
|
||||
|
||||
|
||||
class TestLevelError:
|
||||
@pytest.mark.parametrize(
|
||||
("method", "level"),
|
||||
[
|
||||
("deflated", 0),
|
||||
("deflated", 9),
|
||||
("bzip2", 1),
|
||||
("bzip2", 9),
|
||||
("deflated", None),
|
||||
("stored", None),
|
||||
],
|
||||
)
|
||||
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
|
||||
assert compression.level_error(method, level) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "level"),
|
||||
[
|
||||
("deflated", 10),
|
||||
("deflated", -1),
|
||||
("bzip2", 0),
|
||||
("bzip2", 10),
|
||||
],
|
||||
)
|
||||
def test_out_of_range_levels_return_message(
|
||||
self,
|
||||
method: str,
|
||||
level: int,
|
||||
) -> None:
|
||||
msg: str | None = compression.level_error(method, level)
|
||||
assert msg is not None
|
||||
assert "between" in msg
|
||||
|
||||
@pytest.mark.parametrize("method", ["stored", "lzma"])
|
||||
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
|
||||
msg: str | None = compression.level_error(method, 5)
|
||||
assert msg is not None
|
||||
assert "no effect" in msg
|
||||
|
||||
|
||||
class TestCompressTypeReadable:
|
||||
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
|
||||
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
|
||||
assert compression.compress_type_readable(ct)
|
||||
|
||||
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
|
||||
# 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
|
||||
expected: bool = sys.version_info >= (3, 14)
|
||||
assert compression.compress_type_readable(93) == expected
|
||||
assert compression.compress_type_readable(20) == expected
|
||||
|
||||
def test_unknown_compress_type_is_unreadable(self) -> None:
|
||||
assert not compression.compress_type_readable(9999)
|
||||
|
||||
def test_unreadable_method_names_lists_methods(self) -> None:
|
||||
# An unknown method id maps to no name and is reported generically.
|
||||
names: set[str] = compression.unreadable_method_names({9999})
|
||||
assert names == {"method 9999"}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_compression.py -v"`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'documents.export.compression'`.
|
||||
|
||||
- [ ] **Step 3: Implement `compression.py`**
|
||||
|
||||
Create `src/documents/export/compression.py`:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import zipfile
|
||||
|
||||
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
|
||||
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
|
||||
|
||||
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
|
||||
# availability is enforced separately in compression_available().
|
||||
COMPRESSION_CHOICES: tuple[str, ...] = (
|
||||
"stored",
|
||||
"deflated",
|
||||
"bzip2",
|
||||
"lzma",
|
||||
"zstd",
|
||||
)
|
||||
|
||||
# Method name -> zipfile compression constant (zstd only when supported).
|
||||
COMPRESSION_METHODS: dict[str, int] = {
|
||||
"stored": zipfile.ZIP_STORED,
|
||||
"deflated": zipfile.ZIP_DEFLATED,
|
||||
"bzip2": zipfile.ZIP_BZIP2,
|
||||
"lzma": zipfile.ZIP_LZMA,
|
||||
}
|
||||
if ZSTD is not None:
|
||||
COMPRESSION_METHODS["zstd"] = ZSTD
|
||||
|
||||
# Inclusive (min, max) level bounds per method; None => level not applicable.
|
||||
# Verified on CPython 3.14.3.
|
||||
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
|
||||
"stored": None,
|
||||
"deflated": (0, 9),
|
||||
"bzip2": (1, 9),
|
||||
"lzma": None,
|
||||
"zstd": (-131072, 22),
|
||||
}
|
||||
|
||||
# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
|
||||
# zstd id that zipfile can still read.
|
||||
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
|
||||
zipfile.ZIP_STORED: "stored",
|
||||
zipfile.ZIP_DEFLATED: "deflated",
|
||||
zipfile.ZIP_BZIP2: "bzip2",
|
||||
zipfile.ZIP_LZMA: "lzma",
|
||||
93: "zstd",
|
||||
20: "zstd",
|
||||
}
|
||||
|
||||
|
||||
def compression_available(method: str) -> bool:
|
||||
"""Whether the running interpreter can actually use the given method."""
|
||||
if method in ("stored", "deflated"):
|
||||
# zlib is a hard CPython dependency; stored needs nothing.
|
||||
return True
|
||||
if method == "bzip2":
|
||||
return _module_importable("bz2")
|
||||
if method == "lzma":
|
||||
return _module_importable("lzma")
|
||||
if method == "zstd":
|
||||
return ZSTD is not None and _module_importable("compression.zstd")
|
||||
return False
|
||||
|
||||
|
||||
def _module_importable(name: str) -> bool:
|
||||
try:
|
||||
importlib.import_module(name)
|
||||
except ImportError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def level_error(method: str, level: int | None) -> str | None:
|
||||
"""Return a human message if (method, level) is invalid, else None."""
|
||||
if level is None:
|
||||
return None
|
||||
bounds = LEVEL_BOUNDS[method]
|
||||
if bounds is None:
|
||||
return f"--zip-compression-level has no effect for '{method}'"
|
||||
low, high = bounds
|
||||
if not (low <= level <= high):
|
||||
return (
|
||||
f"--zip-compression-level for '{method}' must be between "
|
||||
f"{low} and {high}"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def compress_type_readable(compress_type: int) -> bool:
|
||||
"""Whether this interpreter can decompress an entry of the given type."""
|
||||
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
|
||||
if method is None:
|
||||
return False
|
||||
return compression_available(method)
|
||||
|
||||
|
||||
def unreadable_method_names(compress_types: set[int]) -> set[str]:
|
||||
"""Map a set of compress_type ids to human method names for error messages."""
|
||||
names: set[str] = set()
|
||||
for ct in compress_types:
|
||||
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
|
||||
return names
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_compression.py -v"`
|
||||
Expected: PASS (on the 3.11 VM, `test_zstd_availability_tracks_runtime` and `test_zstd_compress_type_readability_tracks_runtime` assert `False`).
|
||||
|
||||
- [ ] **Step 5: Lint**
|
||||
|
||||
Run: `ruff check src/documents/export/compression.py src/documents/tests/export/test_compression.py && ruff format src/documents/export/compression.py src/documents/tests/export/test_compression.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/export/compression.py src/documents/tests/export/test_compression.py
|
||||
git commit -m "Feature: add export compression policy module"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `ZipExportSink` accepts compression method + level
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/export/sinks.py`
|
||||
- Test: `src/documents/tests/export/test_sinks.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Append to `src/documents/tests/export/test_sinks.py` (the top-of-file block already imports `zipfile`, `Path`, `pytest`, `ZipExportSink`, `StreamingManifestWriter` from the base-refactor plan):
|
||||
|
||||
```python
|
||||
class TestZipExportSinkCompression:
|
||||
@pytest.fixture()
|
||||
def source_file(self, tmp_path: Path) -> Path:
|
||||
src: Path = tmp_path / "src" / "doc.pdf"
|
||||
src.parent.mkdir(parents=True)
|
||||
src.write_bytes(b"PDF-CONTENT" * 100)
|
||||
return src
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "constant"),
|
||||
[
|
||||
("stored", zipfile.ZIP_STORED),
|
||||
("deflated", zipfile.ZIP_DEFLATED),
|
||||
("bzip2", zipfile.ZIP_BZIP2),
|
||||
("lzma", zipfile.ZIP_LZMA),
|
||||
],
|
||||
)
|
||||
def test_compression_method_is_applied_to_file_entries(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
source_file: Path,
|
||||
method: str,
|
||||
constant: int,
|
||||
) -> None:
|
||||
target: Path = tmp_path / "out"
|
||||
target.mkdir()
|
||||
with ZipExportSink(
|
||||
target,
|
||||
"export",
|
||||
delete=False,
|
||||
compression=constant,
|
||||
) as sink:
|
||||
sink.add_file(source_file, "doc.pdf")
|
||||
with zipfile.ZipFile(target / "export.zip") as zf:
|
||||
info = zf.getinfo("doc.pdf")
|
||||
assert info.compress_type == constant
|
||||
|
||||
def test_compressing_method_beats_stored(
|
||||
self,
|
||||
tmp_path: Path,
|
||||
source_file: Path,
|
||||
) -> None:
|
||||
# Robust size invariant: a compressing method must be <= stored on
|
||||
# compressible content (avoids flaky level-9-vs-level-1 comparisons).
|
||||
sizes: dict[str, int] = {}
|
||||
for name, constant in (("stored", zipfile.ZIP_STORED), ("deflated", zipfile.ZIP_DEFLATED)):
|
||||
target: Path = tmp_path / name
|
||||
target.mkdir()
|
||||
with ZipExportSink(target, "export", delete=False, compression=constant) as sink:
|
||||
sink.add_file(source_file, "doc.pdf")
|
||||
sizes[name] = (target / "export.zip").stat().st_size
|
||||
assert sizes["deflated"] <= sizes["stored"]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_sinks.py::TestZipExportSinkCompression -v"`
|
||||
Expected: FAIL with `TypeError: __init__() got an unexpected keyword argument 'compression'`.
|
||||
|
||||
- [ ] **Step 3: Add the params to `ZipExportSink`**
|
||||
|
||||
In `src/documents/export/sinks.py`, change `ZipExportSink.__init__` to accept the new keyword-only params and store them, and pass them in `_open()`:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
target: Path,
|
||||
zip_name: str,
|
||||
*,
|
||||
delete: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED,
|
||||
compresslevel: int | None = None,
|
||||
) -> None:
|
||||
self._target = target.resolve()
|
||||
self._zip_path = (self._target / zip_name).with_suffix(".zip")
|
||||
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
|
||||
self._delete = delete
|
||||
self._compression = compression
|
||||
self._compresslevel = compresslevel
|
||||
self._zip: zipfile.ZipFile | None = None
|
||||
self._dirs: set[str] = set()
|
||||
self._pending_manifest: tuple[Path, str] | None = None
|
||||
self._stream_open = False
|
||||
```
|
||||
|
||||
And in `_open()`:
|
||||
|
||||
```python
|
||||
def _open(self) -> None:
|
||||
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
|
||||
self._zip = zipfile.ZipFile(
|
||||
self._tmp_path,
|
||||
"w",
|
||||
compression=self._compression,
|
||||
compresslevel=self._compresslevel,
|
||||
allowZip64=True,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_sinks.py -v"`
|
||||
Expected: PASS (all sink tests, including the four method params and the size invariant). `bzip2`/`lzma` are present on the VM's CPython, so those params pass.
|
||||
|
||||
- [ ] **Step 5: Lint**
|
||||
|
||||
Run: `ruff check src/documents/export/sinks.py && ruff format src/documents/export/sinks.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/export/sinks.py src/documents/tests/export/test_sinks.py
|
||||
git commit -m "Feature: ZipExportSink accepts compression method and level"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Wire CLI flags + validation into `document_exporter`
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/management/commands/document_exporter.py`
|
||||
- Test: `src/documents/tests/test_management_exporter.py`
|
||||
|
||||
- [ ] **Step 1: Add the argparse flags**
|
||||
|
||||
In `document_exporter.py`, add the import near the other `documents.export` import:
|
||||
|
||||
```python
|
||||
from documents.export.compression import COMPRESSION_CHOICES
|
||||
from documents.export.compression import COMPRESSION_METHODS
|
||||
from documents.export.compression import compression_available
|
||||
from documents.export.compression import level_error
|
||||
from documents.export.compression import ZSTD
|
||||
```
|
||||
|
||||
In `add_arguments`, after the `--zip-name` argument, add:
|
||||
|
||||
```python
|
||||
parser.add_argument(
|
||||
"--zip-compression",
|
||||
choices=COMPRESSION_CHOICES,
|
||||
default=None,
|
||||
help=(
|
||||
"Compression method for the export zip (requires --zip). "
|
||||
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
|
||||
"exporting and importing machine."
|
||||
),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--zip-compression-level",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Compression level for the export zip (requires --zip). "
|
||||
"deflated: 0-9, bzip2: 1-9, zstd: -131072..22; ignored for "
|
||||
"stored/lzma."
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Read + validate the flags in `handle()`**
|
||||
|
||||
In `handle()`, after the existing `--compare-*` + `--zip` guard, add the compression flag handling. Insert before the sink construction:
|
||||
|
||||
```python
|
||||
zip_compression: str | None = options["zip_compression"]
|
||||
zip_compression_level: int | None = options["zip_compression_level"]
|
||||
|
||||
if not self.zip_export and (
|
||||
zip_compression is not None or zip_compression_level is not None
|
||||
):
|
||||
raise CommandError(
|
||||
"--zip-compression and --zip-compression-level require --zip",
|
||||
)
|
||||
|
||||
compression_method = zip_compression or "deflated"
|
||||
if self.zip_export:
|
||||
if not compression_available(compression_method):
|
||||
if compression_method == "zstd" and ZSTD is None:
|
||||
raise CommandError(
|
||||
"zstd compression requires Python 3.14 or newer",
|
||||
)
|
||||
raise CommandError(
|
||||
f"Compression method '{compression_method}' is not "
|
||||
f"available on this Python runtime",
|
||||
)
|
||||
level_msg = level_error(compression_method, zip_compression_level)
|
||||
if level_msg is not None:
|
||||
raise CommandError(level_msg)
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Pass the resolved values into `ZipExportSink`**
|
||||
|
||||
Change the `ZipExportSink(...)` construction in `handle()` to:
|
||||
|
||||
```python
|
||||
if self.zip_export:
|
||||
sink = ZipExportSink(
|
||||
self.target,
|
||||
options["zip_name"],
|
||||
delete=self.delete,
|
||||
compression=COMPRESSION_METHODS[compression_method],
|
||||
compresslevel=zip_compression_level,
|
||||
)
|
||||
else:
|
||||
sink = DirectoryExportSink(
|
||||
self.target,
|
||||
compare_checksums=self.compare_checksums,
|
||||
compare_json=self.compare_json,
|
||||
delete=self.delete,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Write the command-level tests**
|
||||
|
||||
Add to the `TestExportImport` class in `src/documents/tests/test_management_exporter.py` (imports `call_command`, `CommandError`, `ZipFile`, `timezone` already present):
|
||||
|
||||
```python
|
||||
def test_compression_flags_require_zip(self) -> None:
|
||||
for args in (
|
||||
["--zip-compression", "lzma"],
|
||||
["--zip-compression-level", "5"],
|
||||
):
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
*args,
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_out_of_range_raises(self) -> None:
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"deflated",
|
||||
"--zip-compression-level",
|
||||
"99",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_compression_level_rejected_for_stored(self) -> None:
|
||||
with self.assertRaises(CommandError):
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"stored",
|
||||
"--zip-compression-level",
|
||||
"5",
|
||||
skip_checks=True,
|
||||
)
|
||||
|
||||
def test_zip_lzma_compression_round_trips(self) -> None:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
"--zip-compression",
|
||||
"lzma",
|
||||
skip_checks=True,
|
||||
)
|
||||
expected = str(
|
||||
self.target / f"export-{timezone.localdate().isoformat()}.zip",
|
||||
)
|
||||
self.assertIsFile(expected)
|
||||
with ZipFile(expected) as zip_file:
|
||||
info = zip_file.getinfo("manifest.json")
|
||||
# manifest.json carries the chosen method; deflated is the default
|
||||
self.assertEqual(info.compress_type, 14) # ZIP_LZMA
|
||||
|
||||
def test_default_zip_uses_deflate(self) -> None:
|
||||
call_command(
|
||||
"document_exporter",
|
||||
self.target,
|
||||
"--zip",
|
||||
skip_checks=True,
|
||||
)
|
||||
expected = str(
|
||||
self.target / f"export-{timezone.localdate().isoformat()}.zip",
|
||||
)
|
||||
with ZipFile(expected) as zip_file:
|
||||
info = zip_file.getinfo("manifest.json")
|
||||
self.assertEqual(info.compress_type, 8) # ZIP_DEFLATED
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the tests**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_exporter.py -v"`
|
||||
Expected: PASS — the new tests plus all existing exporter tests stay green.
|
||||
|
||||
- [ ] **Step 6: Lint**
|
||||
|
||||
Run: `ruff check src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py && ruff format src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py
|
||||
git commit -m "Feature: add --zip-compression and --zip-compression-level flags"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Importer pre-check for unreadable codecs
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/management/commands/document_importer.py`
|
||||
- Test: `src/documents/tests/test_management_importer.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
The importer test file `src/documents/tests/test_management_importer.py` is
|
||||
`TestCase`-style (`class TestCommandImport(... TestCase)`, `self.assertRaises`,
|
||||
`DirectoriesMixin` gives `self.dirs.scratch_dir`). Match that style. Add this
|
||||
method to `TestCommandImport`. It builds a valid zip and patches the readability
|
||||
probe so the check fires deterministically on any runtime:
|
||||
|
||||
```python
|
||||
def test_import_rejects_unreadable_compression(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A zip archive with an entry whose compression this Python can't read
|
||||
WHEN:
|
||||
- Import is attempted
|
||||
THEN:
|
||||
- A CommandError naming the issue is raised, before extraction
|
||||
"""
|
||||
import zipfile
|
||||
from unittest import mock
|
||||
|
||||
archive = Path(self.dirs.scratch_dir) / "export.zip"
|
||||
with zipfile.ZipFile(archive, "w") as zf:
|
||||
zf.writestr("manifest.json", "[]")
|
||||
|
||||
with mock.patch(
|
||||
"documents.management.commands.document_importer.compress_type_readable",
|
||||
return_value=False,
|
||||
):
|
||||
with self.assertRaises(CommandError) as e:
|
||||
call_command(
|
||||
"document_importer",
|
||||
str(archive),
|
||||
"--no-progress-bar",
|
||||
skip_checks=True,
|
||||
)
|
||||
self.assertIn("compression", str(e.exception))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_importer.py -k unreadable_compression -v"`
|
||||
Expected: FAIL — no pre-check exists yet, so the import proceeds (or fails with a different error).
|
||||
|
||||
- [ ] **Step 3: Implement the pre-check**
|
||||
|
||||
In `document_importer.py`, add the import:
|
||||
|
||||
```python
|
||||
from documents.export.compression import compress_type_readable
|
||||
from documents.export.compression import unreadable_method_names
|
||||
```
|
||||
|
||||
Find the zip-handling block (around `document_importer.py:453`):
|
||||
|
||||
```python
|
||||
with ZipFile(self.source) as zf:
|
||||
zf.extractall(tmp_dir)
|
||||
```
|
||||
|
||||
Replace it with a pre-check before extraction:
|
||||
|
||||
```python
|
||||
with ZipFile(self.source) as zf:
|
||||
unsupported = {
|
||||
info.compress_type
|
||||
for info in zf.infolist()
|
||||
if not compress_type_readable(info.compress_type)
|
||||
}
|
||||
if unsupported:
|
||||
names = ", ".join(sorted(unreadable_method_names(unsupported)))
|
||||
raise CommandError(
|
||||
f"This archive uses compression this Python cannot "
|
||||
f"read ({names}). zstd archives require Python 3.14+.",
|
||||
)
|
||||
zf.extractall(tmp_dir)
|
||||
```
|
||||
|
||||
Confirm `CommandError` is imported in `document_importer.py` (it is used elsewhere; if not, add `from django.core.management.base import CommandError`).
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_importer.py -v"`
|
||||
Expected: PASS — the new test plus all existing importer tests (normal deflated/stored archives still import).
|
||||
|
||||
- [ ] **Step 5: Lint**
|
||||
|
||||
Run: `ruff check src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py && ruff format src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py
|
||||
git commit -m "Feature: importer rejects archives with unreadable compression"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Document the flags
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `docs/administration.md`
|
||||
|
||||
- [ ] **Step 1: Add the flags to the option list**
|
||||
|
||||
In `docs/administration.md`, update the usage block (around line 257) to include the new flags:
|
||||
|
||||
```
|
||||
document_exporter target [-c] [-d] [-f] [-na] [-nt] [-p] [-sm] [-z]
|
||||
|
||||
optional arguments:
|
||||
-c, --compare-checksums
|
||||
-cj, --compare-json
|
||||
-d, --delete
|
||||
-f, --use-filename-format
|
||||
-na, --no-archive
|
||||
-nt, --no-thumbnail
|
||||
-p, --use-folder-prefix
|
||||
-sm, --split-manifest
|
||||
-z, --zip
|
||||
-zn, --zip-name
|
||||
--zip-compression
|
||||
--zip-compression-level
|
||||
--data-only
|
||||
--no-progress-bar
|
||||
--passphrase
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the prose**
|
||||
|
||||
After the `-z`/`--zip` paragraph (around line 330), add:
|
||||
|
||||
```markdown
|
||||
The compression method for the zip can be set with `--zip-compression`
|
||||
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
|
||||
`--zip-compression-level` (deflated: 0–9, bzip2: 1–9, zstd: -131072–22; ignored
|
||||
for `stored` and `lzma`). Both options require `--zip`.
|
||||
|
||||
!!! warning
|
||||
|
||||
`zstd` compression requires Python 3.14 or newer on **both** the machine
|
||||
creating the export and any machine importing it. An archive compressed with
|
||||
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
|
||||
imported on a runtime that lacks the codec; the importer will refuse it with
|
||||
a clear error. The default `deflated` is universally readable.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify the docs build is not broken (lint markdown)**
|
||||
|
||||
Run: `ruff check docs/ 2>/dev/null; echo "docs are markdown; rely on prettier pre-commit"`
|
||||
(No code to test. The prettier pre-commit hook will reformat on commit.)
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/administration.md
|
||||
git commit -m "Docs: document --zip-compression and --zip-compression-level"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Final verification
|
||||
|
||||
**Files:** none (verification only).
|
||||
|
||||
- [ ] **Step 1: Full backend suites on the VM**
|
||||
|
||||
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/ src/documents/tests/test_management_exporter.py src/documents/tests/test_management_importer.py -v"`
|
||||
Expected: PASS, no failures.
|
||||
|
||||
- [ ] **Step 2: Spot-check the zstd happy path on Python 3.14 (cannot run under Django on the 3.11 VM)**
|
||||
|
||||
The zstd positive round-trip can't run in the 3.11 test env. Confirm the policy module behaves on a real 3.14 interpreter with a standalone check (no Django needed):
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run --python 3.14 --no-project python -c "import sys; sys.path.insert(0,'src'); import django; print('skip')" 2>/dev/null || \
|
||||
uv run --python 3.14 --no-project python -c "
|
||||
import zipfile, io
|
||||
from compression.zstd import CompressionParameter as CP
|
||||
print('zstd const', zipfile.ZIP_ZSTANDARD, 'bounds', CP.compression_level.bounds())
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf,'w',compression=zipfile.ZIP_ZSTANDARD,compresslevel=19) as zf:
|
||||
zf.writestr('a.txt','x'*1000)
|
||||
with zipfile.ZipFile(buf) as zf:
|
||||
assert zf.getinfo('a.txt').compress_type == zipfile.ZIP_ZSTANDARD
|
||||
assert zf.read('a.txt') == b'x'*1000
|
||||
print('zstd round-trip OK')
|
||||
"
|
||||
```
|
||||
|
||||
Expected: prints `zstd const 93 bounds (-131072, 22)` and `zstd round-trip OK`. This validates the constant, bounds, and that a zstd archive round-trips — the parts the 3.11 CI cannot exercise.
|
||||
|
||||
- [ ] **Step 3: Type-check on the VM (pyrefly)**
|
||||
|
||||
```bash
|
||||
tar czf - src pyproject.toml uv.lock .pyrefly-baseline.json | ssh -o BatchMode=yes -p 2244 trenton@localhost 'tar xzf - -C ~/projects/paperless-ngx'
|
||||
ssh -o BatchMode=yes -p 2244 trenton@localhost 'bash -lc "cd ~/projects/paperless-ngx && uv run pyrefly check"'
|
||||
```
|
||||
|
||||
Expected: no new type errors beyond the baseline. (Note: `import compression.zstd` is guarded behind `importlib.import_module`, so it is never statically resolved on the 3.11 baseline.)
|
||||
|
||||
- [ ] **Step 4: Final lint**
|
||||
|
||||
Run: `ruff check src/documents/export/ src/documents/management/commands/document_exporter.py src/documents/management/commands/document_importer.py && ruff format --check src/documents/export/ src/documents/management/commands/document_exporter.py src/documents/management/commands/document_importer.py`
|
||||
Expected: clean.
|
||||
|
||||
---
|
||||
|
||||
## Notes for the implementer
|
||||
|
||||
- **Default behavior is unchanged:** with no flags, the sink is constructed with `compression=ZIP_DEFLATED, compresslevel=None` — byte-method-identical to today (`shutil.make_archive` used `ZIP_DEFLATED` with no level). `test_default_zip_uses_deflate` pins this.
|
||||
- **zstd availability is gated three ways and never imported statically:** the constant via `getattr`, the codec via `importlib.import_module("compression.zstd")`, and the CLI value rejected with a friendly message on < 3.14. The choices list always contains `zstd` so argparse doesn't hide it.
|
||||
- **The importer pre-check is the safety net** for portability foot-guns — without it an unreadable entry raises a bare `NotImplementedError` mid-`extractall`. The check runs on `infolist()` (metadata only) before any extraction.
|
||||
- **Why `--zip-compression` defaults to `None`, not `"deflated"`:** so `handle()` can detect "user passed it without `--zip`" and fail fast. The effective default is resolved as `zip_compression or "deflated"`.
|
||||
@@ -1,236 +0,0 @@
|
||||
# Export Zip Compression Control — Design
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Branch base:** `dev`
|
||||
**Status:** Design complete (zstd facts verified on CPython 3.14.3) — **depends on**
|
||||
`2026-06-16-export-sink-architecture-design.md` being implemented first.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
This builds directly on the export sink refactor. It assumes `ZipExportSink`
|
||||
already exists and is the single place that owns `zipfile.ZipFile` creation and
|
||||
entry writes. Do not start this until that refactor has landed; without it, the
|
||||
change would have to touch the command's zip branches again.
|
||||
|
||||
## Problem
|
||||
|
||||
Zip export is hardwired to `ZIP_DEFLATED` at the library default level. Users
|
||||
have no way to trade speed against archive size — a fast `ZIP_STORED` pass for a
|
||||
quick local copy, or a maximal `ZIP_LZMA` pass for the smallest off-site backup.
|
||||
The sink refactor turns "which compression" into a single constructor argument,
|
||||
so exposing it is now a small, isolated change.
|
||||
|
||||
## Goal
|
||||
|
||||
Let the operator choose the zip compression method and level from the CLI, with
|
||||
behavior identical to today when the flags are omitted. All knowledge of
|
||||
compression stays inside `ZipExportSink`; the command only parses flags and maps
|
||||
them to sink arguments.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- `ZipExportSink` gains `compression: int` and `compresslevel: int | None`
|
||||
constructor parameters (default `ZIP_DEFLATED`, `None` → library default),
|
||||
passed straight to `zipfile.ZipFile(...)`.
|
||||
- New `document_exporter` flags: `--zip-compression` and
|
||||
`--zip-compression-level`, valid only with `--zip`.
|
||||
- Validation: method availability, level range per method, and the
|
||||
requires-`--zip` guard.
|
||||
- Import-side: a pre-extract support check in `document_importer` that turns an
|
||||
unsupported codec into a clear `CommandError` (the importer otherwise decompresses
|
||||
transparently via `ZipFile.extractall`).
|
||||
- Docs: add both flags and the zstd-portability caveat to `docs/administration.md`
|
||||
(the `document_exporter` option list, lines ~257-270 and the `-z`/`-zn` section,
|
||||
lines ~328-330). New flags are long-form only (`--zip-compression`,
|
||||
`--zip-compression-level`) — no short aliases, to avoid `-zc`/`-zl` collisions
|
||||
with the existing `-z`/`-zn`.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Compression for any non-zip sink (folder has none; a future S3 sink would
|
||||
handle its own object storage compression separately).
|
||||
- Changing the default. Omitting the flags must produce a byte-compatible-method
|
||||
archive to today's (`ZIP_DEFLATED`, default level).
|
||||
|
||||
## Design
|
||||
|
||||
### `ZipExportSink` changes
|
||||
|
||||
The base sink's signature is `ZipExportSink(target, zip_name, *, delete)`; this
|
||||
adds two keyword-only params after `delete`:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
target: Path,
|
||||
zip_name: str,
|
||||
*,
|
||||
delete: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED,
|
||||
compresslevel: int | None = None,
|
||||
) -> None:
|
||||
...
|
||||
# opened in __enter__:
|
||||
self._zip = zipfile.ZipFile(
|
||||
self._tmp_path,
|
||||
"w",
|
||||
compression=compression,
|
||||
compresslevel=compresslevel,
|
||||
allowZip64=True,
|
||||
)
|
||||
```
|
||||
|
||||
`ZipFile` applies `compression`/`compresslevel` as the default for every
|
||||
`write`/`writestr` (verified: a `ZipFile(..., compression=ZIP_BZIP2)` yields
|
||||
entries with `compress_type == ZIP_BZIP2` without per-call args), so `add_file` /
|
||||
`add_json` / the manifest entry need no changes. Directory marker entries are
|
||||
empty so their compressed payload is zero, but they are still _tagged_ with the
|
||||
chosen `compress_type` — harmless, but tests that read `infolist()` should filter
|
||||
or account for marker entries (see Testing).
|
||||
|
||||
### CLI flags (`document_exporter`)
|
||||
|
||||
- `--zip-compression {stored,deflated,bzip2,lzma}` — and `zstd` **when the
|
||||
runtime supports it** (see below). Maps to the matching `zipfile.ZIP_*`
|
||||
constant. Default `deflated`.
|
||||
- `--zip-compression-level N` — integer. Per-method accepted ranges (verified
|
||||
against the [3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html#zipfile.ZipFile)):
|
||||
- `deflated`: **0–9** (`zlib` also accepts `-1` = "default", identical to
|
||||
omitting the flag / `compresslevel=None`).
|
||||
- `bzip2`: **1–9** (`0` is invalid for bzip2).
|
||||
- `lzma`, `stored`: level has **no effect** — passing `--zip-compression-level`
|
||||
with either is a `CommandError`, not a silent accept (consistent with the
|
||||
base refactor's fail-fast posture).
|
||||
- `zstd`: **-131072 … 22** (the documented commonly-accepted range; the
|
||||
authoritative bounds are
|
||||
`compression.zstd.CompressionParameter.compression_level.bounds()`).
|
||||
|
||||
Default: unset → library default (`compresslevel=None`).
|
||||
|
||||
Both flags require `--zip`; passing either without `--zip` raises a
|
||||
`CommandError`, matching the incremental-flag rule from the base refactor.
|
||||
|
||||
**Why validate up front (not let `zipfile` raise) — verified on 3.14.3:** an
|
||||
invalid level does _not_ fail at `ZipFile(...)` construction — it fails at the
|
||||
**first `write`/`writestr` call**, with an opaque message
|
||||
(`ValueError: Invalid initialization option` for deflated > 9, or
|
||||
`ValueError: compresslevel must be between 1 and 9` for bzip2). Worse, on context
|
||||
exit the half-initialized write handle emits a secondary
|
||||
`AttributeError: '_ZipWriteFile' object has no attribute '_compressor'` during GC
|
||||
finalization, so the user sees stack-trace noise unrelated to the real cause.
|
||||
Up-front validation turns all of that into a single clean `CommandError`.
|
||||
|
||||
### Validation (in `handle()`, before constructing the sink)
|
||||
|
||||
1. **Requires `--zip`.** Either flag without `--zip` → `CommandError`.
|
||||
2. **Method availability — via a named, patchable seam.** Expose a module-level
|
||||
helper `compression_available(method: str) -> bool` that does
|
||||
`try: import bz2 / import lzma / from compression import zstd except ImportError:
|
||||
return False` — **not** `importlib.util.find_spec`, which can report a stdlib
|
||||
C-extension as present when importing it actually fails. `stored`/`deflated`
|
||||
are always available (`zlib` is a hard CPython dependency). For `zstd` the probe
|
||||
must import `compression.zstd` (3.14+), not merely check that
|
||||
`zipfile.ZIP_ZSTANDARD` exists. Making this a named function is also what lets
|
||||
the test patch "method unavailable" with `mocker`. If the chosen method is
|
||||
unavailable, raise a `CommandError` naming the missing capability — `zipfile`
|
||||
itself would otherwise raise a bare `RuntimeError`
|
||||
("Compression requires the (missing) … module").
|
||||
3. **Level range.** Reject an out-of-range `--zip-compression-level` for the
|
||||
chosen method with a clear `CommandError`; reject the flag entirely for
|
||||
`stored`/`lzma` (see above).
|
||||
|
||||
### zstd (Python 3.14+)
|
||||
|
||||
**Verified empirically on CPython 3.14.3** (via `uv run --python 3.14 --no-project`)
|
||||
and against [PEP 784](https://peps.python.org/pep-0784/) +
|
||||
[the 3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html):
|
||||
|
||||
- The compression-method constant is **`zipfile.ZIP_ZSTANDARD`** (added 3.14; its
|
||||
numeric value is `93`). It does **not** exist on < 3.14.
|
||||
- It is backed by the new **`compression.zstd`** stdlib module (PEP 784 added a
|
||||
`compression` namespace package; legacy `bz2`/`lzma`/`zlib` imports are
|
||||
unchanged). `zipfile` raises `RuntimeError` if `compression.zstd` is
|
||||
unavailable when zstd is requested.
|
||||
- Accepted `compresslevel` is **`-131072 … 22`**, confirmed at runtime via
|
||||
`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`.
|
||||
|
||||
Gate everything zstd-related at runtime so nothing is imported or referenced on
|
||||
< 3.14 (the project targets Python ≥ 3.11):
|
||||
|
||||
```python
|
||||
_ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None) # None before 3.14
|
||||
```
|
||||
|
||||
Presence of the _constant_ does not guarantee the _codec_ is usable, so the
|
||||
availability probe (validation step 2) imports `compression.zstd`, not merely
|
||||
checks the constant.
|
||||
|
||||
Keep `zstd` in the `--zip-compression` `choices` **always** (even on < 3.14), and
|
||||
reject it in validation with a friendly "zstd requires Python 3.14+" message. If
|
||||
it were dropped from `choices` on older runtimes, argparse would emit a generic
|
||||
"invalid choice" that reads as though the option never existed — worse UX.
|
||||
|
||||
### Import-side compatibility
|
||||
|
||||
`document_importer` reads zips with `ZipFile(self.source).extractall(...)`
|
||||
(`document_importer.py:453`), which decompresses each entry transparently using
|
||||
whatever method it was stored with — **provided the matching module exists on the
|
||||
importing machine.**
|
||||
|
||||
The failure mode when it doesn't is unfriendly and must be handled: a zstd (or
|
||||
otherwise unsupported) entry raises a bare `NotImplementedError` **per-entry,
|
||||
during `extractall`** — _not_ at `ZipFile(self.source)` open, and `is_zipfile()`
|
||||
still returns true (a zstd archive is a valid zip container). So the importer
|
||||
enters the zip branch, creates its temp dir, may partially extract other entries,
|
||||
then blows up mid-extract with no context. **Mitigation (in scope here):** before
|
||||
extracting, inspect `ZipFile(self.source).infolist()` compress types and, if any
|
||||
is unsupported on this runtime, raise a `CommandError` naming the method and the
|
||||
requirement (e.g. "this archive uses zstd, which needs Python 3.14+") instead of
|
||||
letting `NotImplementedError` escape.
|
||||
|
||||
Per-method summary (document in help text + `administration.md`):
|
||||
|
||||
- `deflated`/`stored`: universally importable.
|
||||
- `bzip2`/`lzma`: importable wherever the `bz2`/`lzma` modules are present
|
||||
(essentially always).
|
||||
- `zstd`: importable only on Python 3.14+. An archive compressed with `zstd` is
|
||||
**not** importable on older runtimes.
|
||||
|
||||
## Testing
|
||||
|
||||
New cases in the sink tests and an export→import round-trip
|
||||
(pytest classes, factory-boy, `mocker`, `parametrize`, typed; run on the VM):
|
||||
|
||||
- **Round-trip per method.** Parametrize over the available methods (skip `zstd`
|
||||
below 3.14, skip `bzip2`/`lzma` if the module is somehow absent): export a
|
||||
small library, import it back, assert documents/manifest match.
|
||||
- **Method is applied.** Assert each written _file_ entry's `compress_type`
|
||||
equals the requested method (read back via `ZipFile.infolist()`), filtering out
|
||||
directory marker entries (which are tagged but empty).
|
||||
- **Level affects size — robustly.** Do **not** compare deflate level 9 vs 1
|
||||
(on small or incompressible fixtures level 9 can equal or slightly exceed level
|
||||
1, causing flaky CI). Instead assert that a compressing method on a
|
||||
moderately-compressible fixture yields a total smaller than `stored`
|
||||
(`ZIP_STORED`), which is a stable invariant.
|
||||
- **Validation.** Each flag without `--zip` → `CommandError`; out-of-range level
|
||||
(`--zip-compression-level 99`) → a clean `CommandError` from validation
|
||||
(asserting we never reach the `writestr` that would raise the masked
|
||||
`ValueError`); `--zip-compression-level` with `stored`/`lzma` → `CommandError`;
|
||||
unavailable method (patch the named availability seam with `mocker`) →
|
||||
`CommandError`; on < 3.14, `--zip-compression zstd` → the friendly
|
||||
"requires 3.14+" `CommandError`.
|
||||
- **Import pre-check.** An archive containing an unsupported compress type
|
||||
produces a `CommandError` from the importer naming the method, not a raw
|
||||
`NotImplementedError` (simulate by patching the importer's support probe).
|
||||
- **Default unchanged.** Omitting both flags yields file entries with
|
||||
`compress_type == ZIP_DEFLATED`, identical to pre-feature behavior.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Foot-gun archives.** A user could produce a `zstd`/`lzma` archive their
|
||||
import target can't read. Mitigation: explicit help text and the import-side
|
||||
notes above; the default stays the universally-readable `deflated`.
|
||||
- **Optional-module assumptions.** Don't assume `bz2`/`lzma` are always compiled
|
||||
in; probe and error clearly. Mitigation: the availability validation step.
|
||||
+38
-9
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
|
||||
If a "Move to Trash" action is executed in a consume pipeline, the consumption
|
||||
will be aborted and the file will be deleted.
|
||||
|
||||
##### Password Removal {#workflow-action-password-removal}
|
||||
|
||||
"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
|
||||
|
||||
- One or more passwords to try, separated by commas or new lines
|
||||
- Each password is tried in order until one successfully unlocks the document
|
||||
|
||||
Password removal never modifies a file in place. Instead, once a working password is found, the
|
||||
decrypted content is consumed as a new [document version](#document-file-versions), leaving the
|
||||
original (still encrypted) version in the document's version history.
|
||||
|
||||
**Consumption Started**: because this trigger fires before the document exists yet, the password
|
||||
removal itself is deferred until after the initial consumption of the encrypted file has completed.
|
||||
OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
|
||||
extracted text (unless the file already contained extractable text outside of OCR). Immediately
|
||||
afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
|
||||
new version of the same document, this time with normal OCR/text extraction applied. In other words,
|
||||
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
|
||||
the properly processed version is created.
|
||||
|
||||
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
|
||||
already exists, so password removal happens immediately: the decrypted content is queued for
|
||||
consumption as a new version right away. Note that if the document's initial consumption also
|
||||
happened while it was still encrypted, that original version will likewise be missing OCR text.
|
||||
|
||||
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
|
||||
multiple PDF types with different passwords, create separate workflows for each use case.
|
||||
|
||||
#### Workflow placeholders
|
||||
|
||||
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
|
||||
@@ -833,10 +861,10 @@ contract you signed 8 years ago).
|
||||
|
||||
When you search paperless for a document, it tries to match this query
|
||||
against your documents. Paperless will look for matching documents by
|
||||
inspecting their content, title, correspondent, type, tags, notes, and
|
||||
custom field values. Paperless returns a scored list of results, so that
|
||||
documents matching your query better will appear further up in the search
|
||||
results.
|
||||
inspecting their content, title, correspondent, type, and tags. Paperless
|
||||
returns a scored list of results, so that documents matching your query
|
||||
better will appear further up in the search results. Notes and custom field
|
||||
values can be searched using the advanced search syntax described below.
|
||||
|
||||
By default, paperless returns only documents which contain all words
|
||||
typed in the search bar. A few things to know about how matching works:
|
||||
@@ -891,9 +919,9 @@ Supported date keywords: `today`, `yesterday`, `previous week`,
|
||||
|
||||
#### Searching custom fields
|
||||
|
||||
Custom field values are included in the full-text index, so a plain search
|
||||
already matches documents whose custom field values contain your search terms.
|
||||
To narrow by field name or value specifically:
|
||||
Custom field names and values are included in the full-text index, but they
|
||||
are not searched by a plain, unqualified query. Use the advanced search syntax
|
||||
to search by field name or value:
|
||||
|
||||
```
|
||||
custom_fields.value:policy
|
||||
@@ -921,8 +949,9 @@ custom_fields.name:"Contract Number" custom_fields.value:1312
|
||||
|
||||
#### Searching notes
|
||||
|
||||
Notes content is included in full-text search automatically. To search
|
||||
by note author or content specifically:
|
||||
Notes are included in the full-text index, but they are not searched by a
|
||||
plain, unqualified query. Use the advanced search syntax to search by note
|
||||
author or content:
|
||||
|
||||
```
|
||||
notes.user:alice
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "paperless-ngx"
|
||||
version = "3.0.0"
|
||||
version = "3.0.5"
|
||||
description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
@@ -75,7 +75,7 @@ dependencies = [
|
||||
"sqlite-vec==0.1.9",
|
||||
"tantivy~=0.26.0",
|
||||
"tika-client~=0.11.0",
|
||||
"torch~=2.12.0",
|
||||
"torch~=2.13.0",
|
||||
"watchfiles>=1.1.1",
|
||||
"whitenoise~=6.11",
|
||||
"zxing-cpp~=3.0.0",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { expect, test } from '@playwright/test'
|
||||
import { expect, test, type WebSocketRoute } from '@playwright/test'
|
||||
import path from 'node:path'
|
||||
|
||||
const REQUESTS_HAR = path.join(__dirname, 'requests/api-document-detail.har')
|
||||
@@ -95,3 +95,60 @@ test('should support quick filters', async ({ page }) => {
|
||||
.click()
|
||||
await expect(page).toHaveURL(/tags__id__all=4&sort=created&reverse=1&page=1/)
|
||||
})
|
||||
|
||||
test('should finish reloading the preview after a remote document update', async ({
|
||||
page,
|
||||
}) => {
|
||||
let resolveStatusSocket: (socket: WebSocketRoute) => void
|
||||
const statusSocketReady = new Promise<WebSocketRoute>((resolve) => {
|
||||
resolveStatusSocket = resolve
|
||||
})
|
||||
await page.routeWebSocket(/\/ws\/status\/$/, (socket) => {
|
||||
resolveStatusSocket(socket)
|
||||
})
|
||||
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
|
||||
let previewRequestCount = 0
|
||||
page.on('request', (request) => {
|
||||
if (request.url().includes('/api/documents/175/preview/')) {
|
||||
previewRequestCount++
|
||||
}
|
||||
})
|
||||
await page.goto('/documents/175/details')
|
||||
|
||||
await page.locator('pngx-document-detail').waitFor()
|
||||
await expect(page.getByTitle('Storage path', { exact: true })).toHaveText(
|
||||
/\w+/
|
||||
)
|
||||
const previewWasLoaded = await page.evaluate(() => {
|
||||
const detail = document.querySelector('pngx-document-detail')
|
||||
const component = (window as any).ng.getComponent(detail)
|
||||
component.pdfPreviewLoaded({ numPages: 1 })
|
||||
return component.previewLoaded()
|
||||
})
|
||||
expect(previewWasLoaded).toBe(true)
|
||||
const previewRequestsBeforeReload = previewRequestCount
|
||||
|
||||
const statusSocket = await statusSocketReady
|
||||
const documentReloaded = page.waitForResponse(
|
||||
(response) =>
|
||||
response.url().includes('/api/documents/175/?full_perms=true') &&
|
||||
response.request().method() === 'GET'
|
||||
)
|
||||
statusSocket.send(
|
||||
JSON.stringify({
|
||||
type: 'document_updated',
|
||||
data: {
|
||||
document_id: 175,
|
||||
modified: '2026-07-26T20:00:00Z',
|
||||
},
|
||||
})
|
||||
)
|
||||
await documentReloaded
|
||||
|
||||
await expect(
|
||||
page.getByText('Document reloaded with latest changes.').first()
|
||||
).toBeVisible()
|
||||
await expect
|
||||
.poll(() => previewRequestCount)
|
||||
.toBeGreaterThan(previewRequestsBeforeReload + 1)
|
||||
})
|
||||
|
||||
+152
-148
File diff suppressed because it is too large
Load Diff
+25
-25
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "paperless-ngx-ui",
|
||||
"version": "3.0.0",
|
||||
"version": "3.0.5",
|
||||
"scripts": {
|
||||
"preinstall": "npx only-allow pnpm",
|
||||
"ng": "ng",
|
||||
@@ -11,16 +11,16 @@
|
||||
},
|
||||
"private": true,
|
||||
"dependencies": {
|
||||
"@angular/cdk": "^22.0.3",
|
||||
"@angular/common": "~22.0.5",
|
||||
"@angular/compiler": "~22.0.5",
|
||||
"@angular/core": "~22.0.5",
|
||||
"@angular/forms": "~22.0.5",
|
||||
"@angular/localize": "~22.0.5",
|
||||
"@angular/platform-browser": "~22.0.5",
|
||||
"@angular/router": "~22.0.5",
|
||||
"@angular/cdk": "^22.0.6",
|
||||
"@angular/common": "~22.0.8",
|
||||
"@angular/compiler": "~22.0.8",
|
||||
"@angular/core": "~22.0.8",
|
||||
"@angular/forms": "~22.0.8",
|
||||
"@angular/localize": "~22.0.8",
|
||||
"@angular/platform-browser": "~22.0.8",
|
||||
"@angular/router": "~22.0.8",
|
||||
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
|
||||
"@ng-select/ng-select": "^23.2.0",
|
||||
"@ng-select/ng-select": "^23.5.0",
|
||||
"@ngneat/dirty-check-forms": "^3.0.3",
|
||||
"@popperjs/core": "^2.11.8",
|
||||
"bootstrap": "^5.3.8",
|
||||
@@ -41,24 +41,24 @@
|
||||
"devDependencies": {
|
||||
"@angular-builders/custom-webpack": "^22.0.1",
|
||||
"@angular-builders/jest": "^22.0.1",
|
||||
"@angular-devkit/core": "^22.0.5",
|
||||
"@angular-devkit/schematics": "^22.0.5",
|
||||
"@angular-eslint/builder": "22.0.0",
|
||||
"@angular-eslint/eslint-plugin": "22.0.0",
|
||||
"@angular-eslint/eslint-plugin-template": "22.0.0",
|
||||
"@angular-eslint/schematics": "22.0.0",
|
||||
"@angular-eslint/template-parser": "22.0.0",
|
||||
"@angular/build": "^22.0.5",
|
||||
"@angular-devkit/core": "^22.0.8",
|
||||
"@angular-devkit/schematics": "^22.0.8",
|
||||
"@angular-eslint/builder": "22.1.0",
|
||||
"@angular-eslint/eslint-plugin": "22.1.0",
|
||||
"@angular-eslint/eslint-plugin-template": "22.1.0",
|
||||
"@angular-eslint/schematics": "22.1.0",
|
||||
"@angular-eslint/template-parser": "22.1.0",
|
||||
"@angular/build": "^22.0.8",
|
||||
"@angular/cli": "~22.0.5",
|
||||
"@angular/compiler-cli": "~22.0.5",
|
||||
"@angular/compiler-cli": "~22.0.8",
|
||||
"@codecov/webpack-plugin": "^2.0.1",
|
||||
"@playwright/test": "^1.61.1",
|
||||
"@playwright/test": "^1.62.0",
|
||||
"@types/jest": "^30.0.0",
|
||||
"@types/node": "^26.0.0",
|
||||
"@typescript-eslint/eslint-plugin": "^8.62.0",
|
||||
"@typescript-eslint/parser": "^8.62.0",
|
||||
"@typescript-eslint/utils": "^8.62.0",
|
||||
"eslint": "^10.5.0",
|
||||
"@types/node": "^26.1.1",
|
||||
"@typescript-eslint/eslint-plugin": "^8.65.0",
|
||||
"@typescript-eslint/parser": "^8.65.0",
|
||||
"@typescript-eslint/utils": "^8.65.0",
|
||||
"eslint": "^10.8.0",
|
||||
"jest": "30.4.2",
|
||||
"jest-environment-jsdom": "^30.4.1",
|
||||
"jest-junit": "^17.0.0",
|
||||
|
||||
Generated
+793
-610
File diff suppressed because it is too large
Load Diff
@@ -2,6 +2,9 @@ packages:
|
||||
- "."
|
||||
minimumReleaseAge: 10080
|
||||
trustPolicy: no-downgrade
|
||||
trustPolicyExclude:
|
||||
- "chokidar@4.0.3"
|
||||
- "semver@6.3.1 || 5.7.2"
|
||||
allowBuilds:
|
||||
"@parcel/watcher": true
|
||||
canvas: true
|
||||
|
||||
@@ -304,7 +304,7 @@
|
||||
<div class="row mb-3">
|
||||
<div class="col">
|
||||
<p i18n>
|
||||
Settings apply to this user account for objects (Tags, Mail Rules, etc. but not documents) created via the web UI.
|
||||
Settings apply to this user account for objects (Tags, Mail Rules, etc.) created via the web UI. These settings do not apply to documents.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
</div>
|
||||
</pngx-page-header>
|
||||
|
||||
@if (!tasksService.completedFileTasks && tasksService.loading) {
|
||||
@if (loading() && pagedTasks().length === 0) {
|
||||
<div class="spinner-border spinner-border-sm fw-normal ms-2 me-auto" role="status"></div>
|
||||
<div class="visually-hidden" i18n>Loading...</div>
|
||||
}
|
||||
|
||||
@@ -182,7 +182,7 @@
|
||||
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
|
||||
<i-bs class="me-2" name="stack"></i-bs><span><ng-container i18n>Attributes</ng-container></span>
|
||||
</a>
|
||||
@if (!slimSidebarEnabled) {
|
||||
@if (!slimSidebarEnabled && canSaveSettings) {
|
||||
<button
|
||||
type="button"
|
||||
class="btn btn-link btn-sm text-muted p-0 me-3 attributes-expand-btn"
|
||||
|
||||
@@ -280,10 +280,52 @@ main {
|
||||
}
|
||||
}
|
||||
|
||||
@media screen and (min-width: 376px) and (max-width: 768px) {
|
||||
@media screen and (max-width: 575.98px) {
|
||||
.navbar {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr auto 1fr;
|
||||
grid-template-areas:
|
||||
"toggler brand actions"
|
||||
"search search search";
|
||||
}
|
||||
|
||||
.navbar-toggler {
|
||||
// compensate for 2 buttons on the right
|
||||
margin-right: 45px;
|
||||
grid-area: toggler;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.navbar-brand {
|
||||
grid-area: brand;
|
||||
min-width: 0;
|
||||
max-width: 100%;
|
||||
justify-self: center;
|
||||
overflow: hidden;
|
||||
|
||||
> div,
|
||||
.custom-title {
|
||||
min-width: 0;
|
||||
max-width: 100%;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.title,
|
||||
.byline {
|
||||
max-width: 100%;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
.search-container {
|
||||
grid-area: search;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.navbar > ul {
|
||||
grid-area: actions;
|
||||
justify-self: end;
|
||||
flex-wrap: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -293,13 +335,6 @@ main {
|
||||
}
|
||||
}
|
||||
|
||||
@media screen and (max-width: 345px) {
|
||||
.custom-title {
|
||||
max-width: 110px;
|
||||
overflow: hidden;
|
||||
}
|
||||
}
|
||||
|
||||
:host ::ng-deep .dropdown.show .dropdown-toggle,
|
||||
:host ::ng-deep .dropdown-toggle:hover {
|
||||
opacity: 0.7;
|
||||
|
||||
@@ -9,14 +9,14 @@
|
||||
@for (message of messages(); track message) {
|
||||
<div class="message d-flex flex-row small" [class.justify-content-end]="message.role === 'user'">
|
||||
<div class="p-2 m-2" [class.bg-body]="message.role === 'user'">
|
||||
<span>
|
||||
<span class="text-break">
|
||||
{{ message.content }}
|
||||
@if (message.isStreaming) { <span class="blinking-cursor">|</span> }
|
||||
</span>
|
||||
@if (message.role === 'assistant' && message.references?.length) {
|
||||
<div class="chat-references list-group mt-3">
|
||||
@for (reference of message.references; track reference.id) {
|
||||
<a class="list-group-item list-group-item-action text-primary" [routerLink]="['/documents', reference.id]">
|
||||
<a class="list-group-item list-group-item-action d-flex text-primary text-break" [routerLink]="['/documents', reference.id]">
|
||||
<i-bs width="0.9em" height="0.9em" name="file-text" class="me-1"></i-bs><span>{{ reference.title }}</span>
|
||||
</a>
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
.chat-messages {
|
||||
max-height: 350px;
|
||||
overflow-y: auto;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.chat-references {
|
||||
|
||||
@@ -17,6 +17,6 @@
|
||||
|
||||
<ng-template #popoverContent>
|
||||
<div>
|
||||
{{confirmMessage}} <button class="btn btn-link btn-sm text-danger p-0 m-0 lh-1" type="button" (click)="onConfirm($event)">Yes</button>
|
||||
{{confirmMessage}} <button class="btn btn-link btn-sm text-danger p-0 m-0 lh-1" type="button" (click)="onConfirm($event)" i18n>Yes</button>
|
||||
</div>
|
||||
</ng-template>
|
||||
|
||||
+5
@@ -68,6 +68,11 @@
|
||||
></ng-select>
|
||||
} @else if (getCustomFieldByID(atom.field)?.data_type === CustomFieldDataType.DocumentLink) {
|
||||
<pngx-input-document-link [(ngModel)]="atom.value" class="w-25 form-select doc-link-select p-0" placeholder="Search docs..." i18n-placeholder [minimal]="true"></pngx-input-document-link>
|
||||
} @else if (getCustomFieldByID(atom.field)?.data_type === CustomFieldDataType.Monetary) {
|
||||
<input class="w-25 form-control rounded-end" type="text" inputmode="decimal"
|
||||
[ngModel]="atom.value"
|
||||
(ngModelChange)="setMonetaryValue(atom, $event)"
|
||||
[disabled]="disabled">
|
||||
} @else {
|
||||
<input class="w-25 form-control rounded-end" type="text" [(ngModel)]="atom.value" [disabled]="disabled">
|
||||
}
|
||||
|
||||
+24
@@ -1,5 +1,6 @@
|
||||
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
|
||||
import { provideHttpClientTesting } from '@angular/common/http/testing'
|
||||
import { LOCALE_ID } from '@angular/core'
|
||||
import { ComponentFixture, TestBed } from '@angular/core/testing'
|
||||
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
|
||||
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
|
||||
@@ -41,6 +42,12 @@ const customFields = [
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
id: 3,
|
||||
name: 'Test Monetary Field',
|
||||
data_type: CustomFieldDataType.Monetary,
|
||||
extra_data: { default_currency: 'EUR' },
|
||||
},
|
||||
]
|
||||
|
||||
describe('CustomFieldsQueryDropdownComponent', () => {
|
||||
@@ -61,6 +68,7 @@ describe('CustomFieldsQueryDropdownComponent', () => {
|
||||
providers: [
|
||||
provideHttpClient(withInterceptorsFromDi()),
|
||||
provideHttpClientTesting(),
|
||||
{ provide: LOCALE_ID, useValue: 'de' },
|
||||
],
|
||||
}).compileComponents()
|
||||
|
||||
@@ -150,6 +158,22 @@ describe('CustomFieldsQueryDropdownComponent', () => {
|
||||
expect(options2).toEqual([])
|
||||
})
|
||||
|
||||
it('should normalize localized monetary comparison values', () => {
|
||||
const atom = new CustomFieldQueryAtom([3, 'exact', null])
|
||||
|
||||
component.setMonetaryValue(atom, '1.234,56')
|
||||
|
||||
expect(atom.value).toEqual('1234.56')
|
||||
})
|
||||
|
||||
it('should preserve API-formatted monetary comparison values', () => {
|
||||
const atom = new CustomFieldQueryAtom([3, 'exact', null])
|
||||
|
||||
component.setMonetaryValue(atom, '1234.56')
|
||||
|
||||
expect(atom.value).toEqual('1234.56')
|
||||
})
|
||||
|
||||
it('should remove an element from the selection model', () => {
|
||||
const expression = new CustomFieldQueryExpression()
|
||||
const atom = new CustomFieldQueryAtom()
|
||||
|
||||
+21
-1
@@ -1,9 +1,14 @@
|
||||
import { NgTemplateOutlet } from '@angular/common'
|
||||
import {
|
||||
getLocaleNumberSymbol,
|
||||
NgTemplateOutlet,
|
||||
NumberSymbol,
|
||||
} from '@angular/common'
|
||||
import {
|
||||
Component,
|
||||
EventEmitter,
|
||||
inject,
|
||||
Input,
|
||||
LOCALE_ID,
|
||||
Output,
|
||||
QueryList,
|
||||
signal,
|
||||
@@ -212,6 +217,7 @@ export class CustomFieldQueriesModel {
|
||||
})
|
||||
export class CustomFieldsQueryDropdownComponent extends LoadingComponentWithPermissions {
|
||||
protected customFieldsService = inject(CustomFieldsService)
|
||||
private readonly locale = inject(LOCALE_ID)
|
||||
|
||||
public CustomFieldQueryComponentType = CustomFieldQueryElementType
|
||||
public CustomFieldQueryOperator = CustomFieldQueryOperator
|
||||
@@ -376,4 +382,18 @@ export class CustomFieldsQueryDropdownComponent extends LoadingComponentWithPerm
|
||||
}
|
||||
return []
|
||||
}
|
||||
|
||||
setMonetaryValue(atom: CustomFieldQueryAtom, value: string) {
|
||||
// Normalize the decimal symbol e.g. . vs , by locale
|
||||
const decimalSymbol = getLocaleNumberSymbol(
|
||||
this.locale,
|
||||
NumberSymbol.Decimal
|
||||
)
|
||||
if (decimalSymbol !== '.' && value.includes(decimalSymbol)) {
|
||||
const groupSymbol = getLocaleNumberSymbol(this.locale, NumberSymbol.Group)
|
||||
value = value.split(groupSymbol).join('').split(decimalSymbol).join('.')
|
||||
}
|
||||
|
||||
atom.value = value
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,6 +32,8 @@ import { EditDialogComponent, EditDialogMode } from './edit-dialog.component'
|
||||
template: `
|
||||
<div>
|
||||
<h4 class="modal-title" id="modal-basic-title">{{ getTitle() }}</h4>
|
||||
<span class="error">{{ error?.name }}</span>
|
||||
<button [disabled]="networkActive" (click)="save()">Save</button>
|
||||
</div>
|
||||
`,
|
||||
imports: [FormsModule, ReactiveFormsModule],
|
||||
@@ -276,4 +278,22 @@ describe('EditDialogComponent', () => {
|
||||
expect(failedSpy).toHaveBeenCalled()
|
||||
expect(component.error).toEqual('error')
|
||||
})
|
||||
|
||||
it('should update the view after a failed save', async () => {
|
||||
const button: HTMLButtonElement =
|
||||
fixture.nativeElement.querySelector('button')
|
||||
button.click()
|
||||
await fixture.whenStable()
|
||||
expect(button.disabled).toBe(true)
|
||||
|
||||
httpTestingController
|
||||
.expectOne(`${environment.apiBaseUrl}tags/`)
|
||||
.flush({ name: ['Name is required.'] }, { status: 400, statusText: '' })
|
||||
await fixture.whenStable()
|
||||
|
||||
expect(button.disabled).toBe(false)
|
||||
expect(fixture.nativeElement.querySelector('.error').textContent).toContain(
|
||||
'Name is required.'
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import {
|
||||
ChangeDetectorRef,
|
||||
Directive,
|
||||
EventEmitter,
|
||||
Input,
|
||||
@@ -45,6 +46,7 @@ export abstract class EditDialogComponent<
|
||||
protected userService = inject(UserService)
|
||||
protected settingsService = inject(SettingsService)
|
||||
protected permissionsService = inject(PermissionsService)
|
||||
protected changeDetector = inject(ChangeDetectorRef)
|
||||
|
||||
dialogMode = model(EditDialogMode.CREATE)
|
||||
|
||||
@@ -108,10 +110,12 @@ export abstract class EditDialogComponent<
|
||||
// wait to enable close button so it doesn't steal focus from input since its the first clickable element in the DOM
|
||||
setTimeout(() => {
|
||||
this.closeEnabled = true
|
||||
this.changeDetector.markForCheck()
|
||||
})
|
||||
|
||||
this.userService.listAll().subscribe((r) => {
|
||||
this.users = r.results
|
||||
this.changeDetector.markForCheck()
|
||||
})
|
||||
}
|
||||
|
||||
@@ -191,6 +195,7 @@ export abstract class EditDialogComponent<
|
||||
this.error = error.error
|
||||
this.networkActive = false
|
||||
this.failed.next(error)
|
||||
this.changeDetector.markForCheck()
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
|
||||
;(component as any).applyScale()
|
||||
expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
|
||||
expect(viewer.currentScale).toBe(2)
|
||||
})
|
||||
|
||||
it('does not reapply scale for page-only changes', async () => {
|
||||
await initComponent()
|
||||
|
||||
const pdf = (component as any).pdf as { numPages: number }
|
||||
pdf.numPages = 3
|
||||
const viewer = (component as any).pdfViewer as PDFViewer
|
||||
viewer.setDocument(pdf)
|
||||
const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
|
||||
component.page = 2
|
||||
;(component as any).lastViewerPage = 2
|
||||
;(component as any).applyViewerState()
|
||||
|
||||
component.ngOnChanges({
|
||||
page: new SimpleChange(1, 2, false),
|
||||
})
|
||||
|
||||
expect(viewer.currentPageNumber).toBe(2)
|
||||
expect((component as any).lastViewerPage).toBeUndefined()
|
||||
expect(applyScaleSpy).toHaveBeenCalled()
|
||||
expect(applyScaleSpy).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('does not reset the viewer when it is already on the requested page', async () => {
|
||||
@@ -283,6 +295,22 @@ describe('PngxPdfViewerComponent', () => {
|
||||
expect(mockViewer.currentPageNumber).toBe(1)
|
||||
})
|
||||
|
||||
it('reloads when the source revision changes', () => {
|
||||
const resetSpy = jest.spyOn(component as any, 'resetViewerState')
|
||||
const loadSpy = jest
|
||||
.spyOn(component as any, 'loadDocument')
|
||||
.mockImplementation(() => {})
|
||||
component.src = 'test.pdf'
|
||||
component.sourceRevision = 1
|
||||
|
||||
component.ngOnChanges({
|
||||
sourceRevision: new SimpleChange(0, 1, false),
|
||||
})
|
||||
|
||||
expect(resetSpy).toHaveBeenCalled()
|
||||
expect(loadSpy).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('applies viewer state after view init when already loaded', () => {
|
||||
const applySpy = jest.spyOn(component as any, 'applyViewerState')
|
||||
;(component as any).hasLoaded = true
|
||||
|
||||
@@ -43,6 +43,7 @@ export class PngxPdfViewerComponent
|
||||
private readonly document = inject<Document>(DOCUMENT)
|
||||
|
||||
@Input() src!: string
|
||||
@Input() sourceRevision = 0
|
||||
@Input() password?: string
|
||||
@Input() page?: number
|
||||
@Output() pageChange = new EventEmitter<number>()
|
||||
@@ -93,7 +94,7 @@ export class PngxPdfViewerComponent
|
||||
}
|
||||
|
||||
ngOnChanges(changes: SimpleChanges): void {
|
||||
if (changes['src'] || changes['password']) {
|
||||
if (changes['src'] || changes['sourceRevision'] || changes['password']) {
|
||||
this.resetViewerState()
|
||||
if (this.src) {
|
||||
this.loadDocument()
|
||||
@@ -115,7 +116,10 @@ export class PngxPdfViewerComponent
|
||||
changes['zoomScale'] ||
|
||||
changes['rotation']
|
||||
) {
|
||||
this.applyViewerState()
|
||||
// Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
|
||||
this.applyViewerState(
|
||||
!!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
|
||||
)
|
||||
}
|
||||
|
||||
if (changes['searchQuery']) {
|
||||
@@ -239,7 +243,7 @@ export class PngxPdfViewerComponent
|
||||
}
|
||||
}
|
||||
|
||||
private applyViewerState(): void {
|
||||
private applyViewerState(applyScale = true): void {
|
||||
if (!this.pdfViewer) {
|
||||
return
|
||||
}
|
||||
@@ -263,7 +267,7 @@ export class PngxPdfViewerComponent
|
||||
if (this.page === this.lastViewerPage) {
|
||||
this.lastViewerPage = undefined
|
||||
}
|
||||
if (hasPages) {
|
||||
if (hasPages && applyScale) {
|
||||
this.applyScale()
|
||||
}
|
||||
this.dispatchFindIfReady()
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
<a [href]="link ?? previewUrl" class="{{linkClasses}}" [target]="linkTarget" [title]="linkTitle"
|
||||
[ngbPopover]="previewContent" [popoverTitle]="document.title | documentTitle" container="body"
|
||||
autoClose="true" [popoverClass]="popoverClass()" (mouseenter)="mouseEnterPreview()" (mouseleave)="mouseLeavePreview()" #popover="ngbPopover">
|
||||
autoClose="true" [popoverClass]="popoverClass()" [popperOptions]="popperOptions"
|
||||
(mouseenter)="mouseEnterPreview()" (mouseleave)="mouseLeavePreview()" #popover="ngbPopover">
|
||||
<ng-content></ng-content>
|
||||
</a>
|
||||
<ng-template #previewContent>
|
||||
|
||||
@@ -67,6 +67,10 @@ describe('PreviewPopupComponent', () => {
|
||||
expect(component.useNativePdfViewer).toBeTruthy()
|
||||
})
|
||||
|
||||
it('should use fixed positioning for the preview popover', () => {
|
||||
expect(component.popperOptions({}).strategy).toEqual('fixed')
|
||||
})
|
||||
|
||||
it('should render object if native PDF viewer enabled', () => {
|
||||
settingsService.set(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER, true)
|
||||
component.popover.open()
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
ViewChild,
|
||||
} from '@angular/core'
|
||||
import { NgbPopover, NgbPopoverModule } from '@ng-bootstrap/ng-bootstrap'
|
||||
import { Options } from '@popperjs/core'
|
||||
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
||||
import { first, Subject, takeUntil } from 'rxjs'
|
||||
import { Document } from 'src/app/data/document'
|
||||
@@ -74,6 +75,11 @@ export class PreviewPopupComponent implements OnDestroy {
|
||||
|
||||
readonly popoverClass = signal('shadow popover-preview')
|
||||
|
||||
readonly popperOptions = (options: Partial<Options>): Partial<Options> => ({
|
||||
...options,
|
||||
strategy: 'fixed',
|
||||
})
|
||||
|
||||
get renderAsObject(): boolean {
|
||||
return (this.isPdf && this.useNativePdfViewer) || !this.isPdf
|
||||
}
|
||||
|
||||
@@ -113,8 +113,8 @@
|
||||
|
||||
<form [formGroup]='documentForm' (ngSubmit)="save()">
|
||||
|
||||
<div class="btn-toolbar mb-1 border-bottom">
|
||||
<div class="btn-group pb-3">
|
||||
<div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
|
||||
<div class="btn-group me-auto">
|
||||
<button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
|
||||
<i-bs width="1.2em" height="1.2em" name="x"></i-bs>
|
||||
</button>
|
||||
@@ -127,32 +127,36 @@
|
||||
</div>
|
||||
|
||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||
<div class="btn-group pb-3 ms-auto">
|
||||
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
||||
[disabled]="!userCanEdit || suggestionsLoading()"
|
||||
[loading]="suggestionsLoading()"
|
||||
[suggestions]="suggestions()"
|
||||
[aiEnabled]="aiEnabled"
|
||||
(getSuggestions)="getSuggestions()"
|
||||
(addTag)="createTag($event)"
|
||||
(addDocumentType)="createDocumentType($event)"
|
||||
(addCorrespondent)="createCorrespondent($event)">
|
||||
</pngx-suggestions-dropdown>
|
||||
<div class="d-flex gap-2">
|
||||
<div class="btn-group">
|
||||
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
||||
[disabled]="!userCanEdit || suggestionsLoading()"
|
||||
[loading]="suggestionsLoading()"
|
||||
[suggestions]="suggestions()"
|
||||
[aiEnabled]="aiEnabled"
|
||||
(getSuggestions)="getSuggestions()"
|
||||
(addTag)="createTag($event)"
|
||||
(addDocumentType)="createDocumentType($event)"
|
||||
(addCorrespondent)="createCorrespondent($event)">
|
||||
</pngx-suggestions-dropdown>
|
||||
</div>
|
||||
|
||||
<div class="btn-group">
|
||||
<pngx-custom-fields-dropdown
|
||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
||||
[documentId]="documentId()"
|
||||
[disabled]="!userCanEdit"
|
||||
[existingFields]="document()?.custom_fields"
|
||||
(created)="refreshCustomFields()"
|
||||
(added)="addField($event)">
|
||||
</pngx-custom-fields-dropdown>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="btn-group pb-3 ms-2">
|
||||
<pngx-custom-fields-dropdown
|
||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
||||
[documentId]="documentId()"
|
||||
[disabled]="!userCanEdit"
|
||||
[existingFields]="document()?.custom_fields"
|
||||
(created)="refreshCustomFields()"
|
||||
(added)="addField($event)">
|
||||
</pngx-custom-fields-dropdown>
|
||||
<div class="ps-3">
|
||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||
</div>
|
||||
</ng-container>
|
||||
|
||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||
</div>
|
||||
|
||||
<ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
|
||||
@@ -276,7 +280,7 @@
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="d-flex border-top pt-3">
|
||||
<div class="d-flex justify-content-end border-top pt-3">
|
||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||
</div>
|
||||
</ng-template>
|
||||
@@ -440,7 +444,7 @@
|
||||
</div>
|
||||
|
||||
<ng-template #saveButtons>
|
||||
<div class="btn-group pb-3 ms-4">
|
||||
<div class="btn-group">
|
||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
|
||||
@if (hasNext()) {
|
||||
@@ -472,6 +476,7 @@
|
||||
<div class="preview-sticky pdf-viewer-container">
|
||||
<pngx-pdf-viewer
|
||||
[src]="pdfSource()"
|
||||
[sourceRevision]="previewRevision()"
|
||||
[password]="pdfPassword()"
|
||||
[renderMode]="PdfRenderMode.All"
|
||||
[page]="previewCurrentPage()" (pageChange)="previewCurrentPage.set($event)"
|
||||
|
||||
@@ -1602,6 +1602,7 @@ describe('DocumentDetailComponent', () => {
|
||||
expect(openDoc.__changedFields).toEqual([])
|
||||
expect(setDirtySpy).toHaveBeenCalledWith(openDoc, false)
|
||||
expect(saveSpy).toHaveBeenCalled()
|
||||
expect(component.previewRevision()).toBe(1)
|
||||
})
|
||||
|
||||
it('should ignore incoming update for a different document id', () => {
|
||||
|
||||
@@ -249,6 +249,7 @@ export class DocumentDetailComponent
|
||||
titleSubject: Subject<string> = new Subject()
|
||||
readonly previewUrl = signal<string>(undefined)
|
||||
readonly pdfSource = signal<string>(undefined)
|
||||
readonly previewRevision = signal(0)
|
||||
readonly pdfPassword = signal<string>(undefined)
|
||||
readonly thumbUrl = signal<string>(undefined)
|
||||
readonly previewText = signal<string>(undefined)
|
||||
@@ -609,6 +610,9 @@ export class DocumentDetailComponent
|
||||
.subscribe()
|
||||
}
|
||||
this.updateComponent(useDoc)
|
||||
if (forceRemote) {
|
||||
this.previewRevision.update((revision) => revision + 1)
|
||||
}
|
||||
this.titleSubject
|
||||
.pipe(
|
||||
debounceTime(1000),
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
<input #textFilterInput class="form-control form-control-sm" type="text"
|
||||
[disabled]="textFilterModifierIsNull"
|
||||
[(ngModel)]="textFilter"
|
||||
(keyup)="textFilterKeyup($event)"
|
||||
(keydown)="textFilterKeydown($event)"
|
||||
[ngbTypeahead]="searchAutoComplete"
|
||||
(selectItem)="itemSelected($event)"
|
||||
[readonly]="textFilterTarget === 'fulltext-morelike'">
|
||||
|
||||
+33
-3
@@ -2180,17 +2180,17 @@ describe('FilterEditorComponent', () => {
|
||||
it('should support Enter / Esc key on text field', () => {
|
||||
component.textFilterInput.nativeElement.value = 'foo'
|
||||
component.textFilterInput.nativeElement.dispatchEvent(
|
||||
new KeyboardEvent('keyup', { key: 'Enter' })
|
||||
new KeyboardEvent('keydown', { key: 'Enter' })
|
||||
)
|
||||
expect(component.textFilter).toEqual('foo')
|
||||
component.textFilterInput.nativeElement.value = 'foo bar'
|
||||
component.textFilterInput.nativeElement.dispatchEvent(
|
||||
new KeyboardEvent('keyup', { key: 'Escape' })
|
||||
new KeyboardEvent('keydown', { key: 'Escape' })
|
||||
)
|
||||
expect(component.textFilter).toEqual('')
|
||||
const blurSpy = jest.spyOn(component.textFilterInput.nativeElement, 'blur')
|
||||
component.textFilterInput.nativeElement.dispatchEvent(
|
||||
new KeyboardEvent('keyup', { key: 'Escape' })
|
||||
new KeyboardEvent('keydown', { key: 'Escape' })
|
||||
)
|
||||
expect(blurSpy).toHaveBeenCalled()
|
||||
})
|
||||
@@ -2269,4 +2269,34 @@ describe('FilterEditorComponent', () => {
|
||||
component.itemSelected({ item: 'world', preventDefault: () => true })
|
||||
expect(component.textFilter).toEqual('hello world ')
|
||||
})
|
||||
|
||||
it('should choose the active autocomplete item with Enter', () => {
|
||||
component.textFilterTarget = 'fulltext-query'
|
||||
jest
|
||||
.spyOn(searchService, 'autocomplete')
|
||||
.mockReturnValue(of(['hello', 'help']))
|
||||
|
||||
const input = component.textFilterInput.nativeElement as HTMLInputElement
|
||||
input.value = 'he'
|
||||
input.dispatchEvent(new Event('input'))
|
||||
tick(250)
|
||||
|
||||
input.dispatchEvent(
|
||||
new KeyboardEvent('keydown', {
|
||||
key: 'ArrowDown',
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
})
|
||||
)
|
||||
input.dispatchEvent(
|
||||
new KeyboardEvent('keydown', {
|
||||
key: 'Enter',
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
})
|
||||
)
|
||||
fixture.detectChanges()
|
||||
|
||||
expect(component.textFilter).toEqual('help ')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1312,8 +1312,12 @@ export class FilterEditorComponent
|
||||
}
|
||||
}
|
||||
|
||||
textFilterKeyup(event: KeyboardEvent) {
|
||||
textFilterKeydown(event: KeyboardEvent) {
|
||||
if (event.key == 'Enter') {
|
||||
if (event.defaultPrevented) {
|
||||
// NgbTypeahead calls preventDefault, so use that to detect if the Enter key was for the dropdown
|
||||
return
|
||||
}
|
||||
const filterString = (
|
||||
this.textFilterInput.nativeElement as HTMLInputElement
|
||||
).value
|
||||
|
||||
+2
-2
@@ -51,8 +51,8 @@
|
||||
*pngxIfPermissions="{ action: PermissionAction.Add, type: activeManagementList.permissionType }">
|
||||
<i-bs name="plus-circle" class="me-1"></i-bs><ng-container i18n>Create</ng-container>
|
||||
</button>
|
||||
} @else if (activeCustomFields) {
|
||||
<button type="button" class="btn btn-sm btn-outline-primary" (click)="activeCustomFields.editField()"
|
||||
} @else if (customFieldsActive) {
|
||||
<button type="button" class="btn btn-sm btn-outline-primary" (click)="addCustomField()"
|
||||
*pngxIfPermissions="{ action: PermissionAction.Add, type: PermissionType.CustomField }">
|
||||
<i-bs name="plus-circle" class="me-1"></i-bs><ng-container i18n>Add Field</ng-container>
|
||||
</button>
|
||||
|
||||
+49
-2
@@ -18,6 +18,8 @@ import {
|
||||
DocumentAttributesComponent,
|
||||
DocumentAttributesSectionKind,
|
||||
} from './document-attributes.component'
|
||||
import { CustomFieldsComponent } from './custom-fields/custom-fields.component'
|
||||
import { ManagementListComponent } from './management-list/management-list.component'
|
||||
|
||||
@Component({
|
||||
selector: 'pngx-dummy-section',
|
||||
@@ -170,10 +172,32 @@ describe('DocumentAttributesComponent', () => {
|
||||
expect(component.activeManagementList).toBeNull()
|
||||
|
||||
component.activeNavID.set(1)
|
||||
const managementList = Object.create(ManagementListComponent.prototype)
|
||||
component.activeOutlet = {
|
||||
componentInstance: managementList,
|
||||
} as any
|
||||
expect(component.activeSection.kind).toBe(
|
||||
DocumentAttributesSectionKind.ManagementList
|
||||
)
|
||||
expect(component.activeManagementList).toBeDefined()
|
||||
expect(component.activeManagementList).toBe(managementList)
|
||||
})
|
||||
|
||||
it('should use the current component instance when the outlet is reused', () => {
|
||||
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
|
||||
component.activeNavID.set(1)
|
||||
const firstManagementList = Object.create(ManagementListComponent.prototype)
|
||||
const secondManagementList = Object.create(
|
||||
ManagementListComponent.prototype
|
||||
)
|
||||
component.activeOutlet = {
|
||||
componentInstance: firstManagementList,
|
||||
} as any
|
||||
|
||||
expect(component.activeManagementList).toBe(firstManagementList)
|
||||
|
||||
component.activeOutlet.componentInstance = secondManagementList
|
||||
|
||||
expect(component.activeManagementList).toBe(secondManagementList)
|
||||
})
|
||||
|
||||
it('should return activeCustomFields correctly', () => {
|
||||
@@ -184,6 +208,29 @@ describe('DocumentAttributesComponent', () => {
|
||||
expect(component.activeSection.kind).toBe(
|
||||
DocumentAttributesSectionKind.CustomFields
|
||||
)
|
||||
expect(component.activeCustomFields).toBeDefined()
|
||||
const customFields = Object.create(CustomFieldsComponent.prototype)
|
||||
customFields.editField = jest.fn()
|
||||
component.activeOutlet = {
|
||||
componentInstance: customFields,
|
||||
} as any
|
||||
expect(component.activeCustomFields).toBe(customFields)
|
||||
|
||||
component.addCustomField()
|
||||
expect(customFields.editField).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('should show the add field button before the custom fields instance is available', async () => {
|
||||
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
|
||||
|
||||
fixture.detectChanges()
|
||||
component.activeNavID.set(2)
|
||||
await fixture.whenStable()
|
||||
|
||||
expect(component.activeCustomFields).toBeNull()
|
||||
expect(
|
||||
fixture.nativeElement.querySelector(
|
||||
'pngx-page-header .btn-outline-primary'
|
||||
)?.textContent
|
||||
).toContain('Add Field')
|
||||
})
|
||||
})
|
||||
|
||||
+14
-8
@@ -132,11 +132,8 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
|
||||
]
|
||||
|
||||
@ViewChild('activeOutlet', { read: NgComponentOutlet })
|
||||
set activeOutlet(outlet: NgComponentOutlet | undefined) {
|
||||
this.activeComponent.set(outlet?.componentInstance ?? null)
|
||||
}
|
||||
activeOutlet: NgComponentOutlet
|
||||
|
||||
readonly activeComponent = signal<unknown>(null)
|
||||
readonly activeNavID = signal<number>(null)
|
||||
|
||||
get visibleSections(): DocumentAttributesSection[] {
|
||||
@@ -161,17 +158,22 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
|
||||
this.activeSection?.kind !== DocumentAttributesSectionKind.ManagementList
|
||||
)
|
||||
return null
|
||||
const instance = this.activeComponent()
|
||||
const instance = this.activeOutlet?.componentInstance
|
||||
return instance instanceof ManagementListComponent ? instance : null
|
||||
}
|
||||
|
||||
get activeCustomFields(): CustomFieldsComponent | null {
|
||||
if (this.activeSection?.kind !== DocumentAttributesSectionKind.CustomFields)
|
||||
return null
|
||||
const instance = this.activeComponent()
|
||||
if (!this.customFieldsActive) return null
|
||||
const instance = this.activeOutlet?.componentInstance
|
||||
return instance instanceof CustomFieldsComponent ? instance : null
|
||||
}
|
||||
|
||||
get customFieldsActive(): boolean {
|
||||
return (
|
||||
this.activeSection?.kind === DocumentAttributesSectionKind.CustomFields
|
||||
)
|
||||
}
|
||||
|
||||
get activeTabLabel(): string {
|
||||
return this.activeSection?.label ?? ''
|
||||
}
|
||||
@@ -227,6 +229,10 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
|
||||
this.router.navigate(['attributes', nextSection])
|
||||
}
|
||||
|
||||
addCustomField(): void {
|
||||
this.activeCustomFields?.editField(null)
|
||||
}
|
||||
|
||||
private getDefaultNavID(): DocumentAttributesNavIDs | null {
|
||||
return this.visibleSections[0]?.id ?? null
|
||||
}
|
||||
|
||||
@@ -108,6 +108,18 @@ describe('PermissionsService', () => {
|
||||
actionKey: 'View', // PermissionAction.View
|
||||
typeKey: 'Document', // PermissionType.Document
|
||||
})
|
||||
expect(
|
||||
permissionsService.getPermissionKeys('view_global_statistics')
|
||||
).toEqual({
|
||||
actionKey: 'View', // PermissionAction.View
|
||||
typeKey: 'GlobalStatistics', // PermissionType.GlobalStatistics
|
||||
})
|
||||
expect(
|
||||
permissionsService.getPermissionKeys('view_system_monitoring')
|
||||
).toEqual({
|
||||
actionKey: 'View', // PermissionAction.View
|
||||
typeKey: 'SystemMonitoring', // PermissionType.SystemMonitoring
|
||||
})
|
||||
})
|
||||
|
||||
it('correctly checks explicit global permissions', () => {
|
||||
|
||||
@@ -110,19 +110,16 @@ export class PermissionsService {
|
||||
actionKey: string
|
||||
typeKey: string
|
||||
} {
|
||||
const matches = permissionStr.match(/(.+)_/)
|
||||
let typeKey
|
||||
let actionKey
|
||||
if (matches?.length > 0) {
|
||||
const action = matches[1]
|
||||
const actionIndex = Object.values(PermissionAction).indexOf(
|
||||
action as PermissionAction
|
||||
)
|
||||
if (actionIndex > -1) {
|
||||
actionKey = Object.keys(PermissionAction)[actionIndex]
|
||||
}
|
||||
const actionIndex = Object.values(PermissionAction).findIndex((action) =>
|
||||
permissionStr.startsWith(`${action}_`)
|
||||
)
|
||||
if (actionIndex > -1) {
|
||||
const action = Object.values(PermissionAction)[actionIndex]
|
||||
actionKey = Object.keys(PermissionAction)[actionIndex]
|
||||
const typeIndex = Object.values(PermissionType).indexOf(
|
||||
permissionStr.replace(action, '%s') as PermissionType
|
||||
permissionStr.replace(`${action}_`, '%s_') as PermissionType
|
||||
)
|
||||
if (typeIndex > -1) {
|
||||
typeKey = Object.keys(PermissionType)[typeIndex]
|
||||
|
||||
@@ -17,7 +17,7 @@ import {
|
||||
estimateBrightnessForColor,
|
||||
hexToHsl,
|
||||
} from 'src/app/utils/color'
|
||||
import { environment } from 'src/environments/environment'
|
||||
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
|
||||
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
|
||||
import { SavedView } from '../data/saved-view'
|
||||
import {
|
||||
@@ -359,9 +359,8 @@ export class SettingsService {
|
||||
}),
|
||||
tap((uisettings) => {
|
||||
this.assignSafeSettings(uisettings.settings)
|
||||
if (this.get(SETTINGS_KEYS.APP_TITLE)?.length) {
|
||||
environment.appTitle = this.get(SETTINGS_KEYS.APP_TITLE)
|
||||
}
|
||||
environment.appTitle =
|
||||
this.get(SETTINGS_KEYS.APP_TITLE) || DEFAULT_APP_TITLE
|
||||
this.maybeMigrateSettings()
|
||||
// to update lang cookie
|
||||
if (this.settings['language']?.length)
|
||||
|
||||
@@ -50,13 +50,66 @@ describe('TasksService', () => {
|
||||
req.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('does not call tasks api endpoint on reload if already loading', () => {
|
||||
tasksService.loading = true
|
||||
it('cancels an in-progress reload when reloading again', () => {
|
||||
tasksService.reload()
|
||||
httpTestingController.expectNone(
|
||||
const staleReload = httpTestingController.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
tasksService.reload()
|
||||
|
||||
expect(staleReload.cancelled).toBe(true)
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('continues reloading after a reload request fails', () => {
|
||||
tasksService.reload()
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
.flush('error', { status: 500, statusText: 'error' })
|
||||
|
||||
expect(tasksService.loading).toBe(false)
|
||||
|
||||
tasksService.reload()
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('reloads after dismissing a task while a reload is already in progress', () => {
|
||||
tasksService.reload()
|
||||
const staleReload = httpTestingController.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||
req.params.get('acknowledged') === 'false'
|
||||
)
|
||||
|
||||
tasksService.dismissTasks(new Set([1])).subscribe()
|
||||
httpTestingController
|
||||
.expectOne(`${environment.apiBaseUrl}tasks/acknowledge/`)
|
||||
.flush([])
|
||||
|
||||
expect(staleReload.cancelled).toBe(true)
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||
req.params.get('acknowledged') === 'false'
|
||||
)
|
||||
.flush({ count: 0, results: [] })
|
||||
|
||||
expect(tasksService.needsAttentionTasks).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('calls acknowledge_tasks api endpoint on dismiss and reloads', () => {
|
||||
@@ -101,79 +154,6 @@ describe('TasksService', () => {
|
||||
.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('groups mixed task types by status when reloading', () => {
|
||||
expect(tasksService.total).toEqual(0)
|
||||
const mockTasks = [
|
||||
{
|
||||
task_type: PaperlessTaskType.ConsumeFile,
|
||||
trigger_source: PaperlessTaskTriggerSource.FolderConsume,
|
||||
status: PaperlessTaskStatus.Success,
|
||||
acknowledged: false,
|
||||
task_id: '1234',
|
||||
input_data: { filename: 'file1.pdf' },
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.SanityCheck,
|
||||
trigger_source: PaperlessTaskTriggerSource.System,
|
||||
status: PaperlessTaskStatus.Failure,
|
||||
acknowledged: false,
|
||||
task_id: '1235',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.MailFetch,
|
||||
trigger_source: PaperlessTaskTriggerSource.Scheduled,
|
||||
status: PaperlessTaskStatus.Pending,
|
||||
acknowledged: false,
|
||||
task_id: '1236',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.LlmIndex,
|
||||
trigger_source: PaperlessTaskTriggerSource.WebUI,
|
||||
status: PaperlessTaskStatus.Started,
|
||||
acknowledged: false,
|
||||
task_id: '1237',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.EmptyTrash,
|
||||
trigger_source: PaperlessTaskTriggerSource.Manual,
|
||||
status: PaperlessTaskStatus.Success,
|
||||
acknowledged: false,
|
||||
task_id: '1238',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
]
|
||||
|
||||
tasksService.reload()
|
||||
|
||||
const req = httpTestingController.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||
req.params.get('acknowledged') === 'false' &&
|
||||
req.params.get('page_size') === '1000'
|
||||
)
|
||||
|
||||
req.flush({ count: mockTasks.length, results: mockTasks })
|
||||
|
||||
expect(tasksService.allFileTasks).toHaveLength(5)
|
||||
expect(tasksService.completedFileTasks).toHaveLength(2)
|
||||
expect(tasksService.failedFileTasks).toHaveLength(1)
|
||||
expect(tasksService.queuedFileTasks).toHaveLength(1)
|
||||
expect(tasksService.startedFileTasks).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('includes revoked tasks in needs attention', () => {
|
||||
const mockTasks = [
|
||||
{
|
||||
|
||||
@@ -1,7 +1,15 @@
|
||||
import { HttpClient } from '@angular/common/http'
|
||||
import { Injectable, inject, signal } from '@angular/core'
|
||||
import { Observable, Subject } from 'rxjs'
|
||||
import { first, map, takeUntil, tap } from 'rxjs/operators'
|
||||
import { EMPTY, Observable, Subject } from 'rxjs'
|
||||
import {
|
||||
catchError,
|
||||
finalize,
|
||||
first,
|
||||
map,
|
||||
switchMap,
|
||||
takeUntil,
|
||||
tap,
|
||||
} from 'rxjs/operators'
|
||||
import {
|
||||
PaperlessTask,
|
||||
PaperlessTaskStatus,
|
||||
@@ -23,44 +31,40 @@ export class TasksService {
|
||||
|
||||
public loading: boolean = false
|
||||
|
||||
private readonly fileTasks = signal<PaperlessTask[]>([])
|
||||
private readonly tasks = signal<PaperlessTask[]>([])
|
||||
private readonly reloadNotifier = new Subject<void>()
|
||||
|
||||
private unsubscribeNotifer: Subject<any> = new Subject()
|
||||
|
||||
public get total(): number {
|
||||
return this.fileTasks().length
|
||||
}
|
||||
|
||||
public get allFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().slice(0)
|
||||
}
|
||||
|
||||
public get queuedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Pending
|
||||
)
|
||||
}
|
||||
|
||||
public get startedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Started
|
||||
)
|
||||
}
|
||||
|
||||
public get completedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Success
|
||||
)
|
||||
}
|
||||
|
||||
public get failedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Failure
|
||||
)
|
||||
constructor() {
|
||||
this.reloadNotifier
|
||||
.pipe(
|
||||
switchMap(() => {
|
||||
this.loading = true
|
||||
return this.http
|
||||
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
|
||||
params: {
|
||||
acknowledged: 'false',
|
||||
page_size: this.defaultReloadPageSize,
|
||||
},
|
||||
})
|
||||
.pipe(
|
||||
map((response) => response.results),
|
||||
takeUntil(this.unsubscribeNotifer),
|
||||
catchError(() => EMPTY),
|
||||
finalize(() => {
|
||||
this.loading = false
|
||||
})
|
||||
)
|
||||
})
|
||||
)
|
||||
.subscribe((tasks) => {
|
||||
this.tasks.set(tasks)
|
||||
})
|
||||
}
|
||||
|
||||
public get needsAttentionTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter((t) =>
|
||||
return this.tasks().filter((t) =>
|
||||
[PaperlessTaskStatus.Failure, PaperlessTaskStatus.Revoked].includes(
|
||||
t.status
|
||||
)
|
||||
@@ -68,22 +72,7 @@ export class TasksService {
|
||||
}
|
||||
|
||||
public reload() {
|
||||
if (this.loading) return
|
||||
this.loading = true
|
||||
|
||||
this.http
|
||||
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
|
||||
params: {
|
||||
acknowledged: 'false',
|
||||
page_size: this.defaultReloadPageSize,
|
||||
},
|
||||
})
|
||||
.pipe(map((r) => r.results))
|
||||
.pipe(takeUntil(this.unsubscribeNotifer), first())
|
||||
.subscribe((r) => {
|
||||
this.fileTasks.set(r)
|
||||
this.loading = false
|
||||
})
|
||||
this.reloadNotifier.next()
|
||||
}
|
||||
|
||||
public list(
|
||||
|
||||
@@ -14,4 +14,20 @@ describe('text search utilities', () => {
|
||||
expect(matchesSearchText('Tax\u00e9s 2026', 'taxe 26')).toBeTruthy()
|
||||
expect(matchesSearchText('taxes 2026', 'tax receipt')).toBeFalsy()
|
||||
})
|
||||
|
||||
it('matches a large set of tag names without blocking input', () => {
|
||||
const tagNames = Array.from(
|
||||
{ length: 1280 },
|
||||
(_, index) =>
|
||||
`Customer Party ${index} Jos\u00e9 M\u00fcller \u00c5ngstr\u00f6m`
|
||||
)
|
||||
|
||||
const start = performance.now()
|
||||
const matches = tagNames.filter((name) => matchesSearchText(name, 'party'))
|
||||
const duration = performance.now() - start
|
||||
|
||||
expect(matches).toHaveLength(tagNames.length)
|
||||
// The previous implementation took roughly 500 ms for this workload.
|
||||
expect(duration).toBeLessThan(250)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,10 +1,17 @@
|
||||
import { normalizeSync } from 'normalize-diacritics'
|
||||
import { diacritics } from 'normalize-diacritics/diacritics'
|
||||
|
||||
export type SearchTextValue =
|
||||
string | number | boolean | bigint | null | undefined
|
||||
|
||||
export function normalizeSearchText(value: SearchTextValue): string {
|
||||
return normalizeSync(String(value ?? '')).toLocaleLowerCase()
|
||||
const normalized = diacritics.reduce(
|
||||
(text, replacement) => {
|
||||
return text.replace(replacement.diacritics, replacement.letter)
|
||||
},
|
||||
String(value ?? '')
|
||||
)
|
||||
|
||||
return normalized.toLocaleLowerCase()
|
||||
}
|
||||
|
||||
export function matchesSearchText(
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
const base_url = new URL(document.baseURI)
|
||||
|
||||
export const DEFAULT_APP_TITLE = 'Paperless-ngx'
|
||||
|
||||
export const environment = {
|
||||
production: true,
|
||||
apiBaseUrl: document.baseURI + 'api/',
|
||||
apiVersion: '10', // match src/paperless/settings.py
|
||||
appTitle: 'Paperless-ngx',
|
||||
appTitle: DEFAULT_APP_TITLE,
|
||||
tag: 'prod',
|
||||
version: '3.0.0',
|
||||
version: '3.0.5',
|
||||
webSocketHost: window.location.host,
|
||||
webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
|
||||
webSocketBaseUrl: base_url.pathname + 'ws/',
|
||||
|
||||
@@ -2,11 +2,13 @@
|
||||
// `ng build --configuration production` replaces `environment.ts` with `environment.prod.ts`.
|
||||
// The list of file replacements can be found in `angular.json`.
|
||||
|
||||
export const DEFAULT_APP_TITLE = 'Paperless-ngx'
|
||||
|
||||
export const environment = {
|
||||
production: false,
|
||||
apiBaseUrl: 'http://localhost:8000/api/',
|
||||
apiVersion: '10',
|
||||
appTitle: 'Paperless-ngx',
|
||||
appTitle: DEFAULT_APP_TITLE,
|
||||
tag: 'dev',
|
||||
version: 'DEVELOPMENT',
|
||||
webSocketHost: 'localhost:8000',
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user