Compare commits

...
Author SHA1 Message Date
stumpylog 2a8df9dca0 Chore: Trim thumbnail comments and docstrings
The comments and docstrings around PDF thumbnail generation were wordier than the code they describe. This cuts them back to the essential facts.
2026-10-07 14:56:47 -07:00
stumpylog f1e85fff3e Chore: Simplify the PDF thumbnail helpers and their tests
The qpdf repair step in the thumbnail fallback is now its own helper, so the fallback has a single ParseError handler instead of a nested try. The fallback also stops re-wrapping temp_dir in Path and annotating obvious locals.

Parameters no caller used are removed: the cropbox toggle on the rasterizer, which now always passes -cropbox, and the width and height limits on the WebP encoder, which uses the module constants directly. The comments around the 300 DPI cap are consolidated so the purpose is stated once.

In the tests, the clamp and supersample encoder cases are merged into one parametrized test, the PDF thumbnail tests share a work directory fixture and a blank PDF helper, redundant assertions are dropped, and the tesseract fallback test uses a plain import of the rasterizer. A garbled docstring is rewritten.
2026-10-07 14:51:09 -07:00
stumpylog f549201b7c Chore: Tighten thumbnail supersampling edge cases
Pages so large that the computed DPI hit the floor of 1 were still supersampled to 2 DPI, quadrupling the pixel count pdftoppm had to render for an already oversized page. Supersampling is now skipped at that floor.

The downsample test also did not prove the factor was applied, because the final 500x5000 clamp hid it in most cases. A 900x1200 render now has to come out at 450x600, which only happens when the downsample runs. The comments on the DPI cap and the clamp are reworded to describe the supersampled pipeline accurately.
2026-10-07 14:32:54 -07:00
stumpylog 26efad77ac Chore: Supersample the PDF thumbnail render 2x
Rendering page 1 straight at the thumbnail density left text slightly soft, since pdftoppm antialiases at the final size.

The page is now rendered at twice the computed DPI and downsampled by that factor with Lanczos before the existing 500x5000 clamp, which gives crisper text at about the same file size for a negligible memory cost. When the page geometry cannot be read, the fixed 150 DPI fallback is kept as a single unsupersampled render, because doubling a render that is not bounded by the thumbnail size would quadruple its pixel count. The qpdf repair path shares the same render step and gets the same behavior.
2026-10-07 14:30:50 -07:00
stumpylog 901dd4be93 Chore: Drop system check for deprecated convert variables
A dedicated system check for the two deprecated ImageMagick convert variables is more machinery than a simple deprecation needs.

Remove the check and its tests. The settings removal, the deprecation notes in the configuration docs, and the example configuration cleanup stay as they were.
2026-10-07 13:47:50 -07:00
stumpylog 8f73514fbb Chore: Deprecate PAPERLESS_CONVERT_MEMORY_LIMIT and PAPERLESS_CONVERT_TMPDIR
These two options set ImageMagick's memory limit and scratch directory, but their only reader was the PDF thumbnail conversion helper. PDF thumbnails no longer use ImageMagick, so the settings did nothing while still being documented and advertised in the example configuration.

Remove the unused settings, add system warnings for anyone who still has either variable set, mark both options as deprecated and without effect in the configuration docs, and drop them from paperless.conf.example. The manual setup guide is also corrected so it no longer claims ImageMagick is needed for PDF conversion, and the ImageMagick policy step now only covers hardening.
2026-10-07 13:46:40 -07:00
stumpylog f30a65f440 Fix: Correct ImageMagick PDF policy note in setup docs
The note claimed that steps still relying on ImageMagick, such as TIFF to
PDF conversion, could fail without the PDF policy change. The remaining
convert calls only strip alpha from raster images, and the TIFF to PDF
step uses img2pdf, so the PDF coder policy no longer matters.

The passage now says Paperless-ngx no longer passes PDF documents to
ImageMagick, so enabling PDF processing is not required, while keeping the
policy hardening guidance in place.
2026-10-07 12:59:05 -07:00
stumpylog 2136659e2b Fix: Harden PDF thumbnail encoding and align CI and docs
Pillow raises DecompressionBombError, which is not an OSError, so an
enormous render from the unknown-geometry fallback DPI skipped the
default thumbnail fallback. It is now converted to a ParseError like other
encoding failures, with a test covering both error types.

CI did not install qpdf, which the new thumbnail repair tests need, so it
is added to the backend package list. The setup docs now list poppler-utils
as used for thumbnail generation and no longer claim PDF thumbnails fall
back to Ghostscript when the ImageMagick PDF policy is not enabled.
2026-10-07 12:58:40 -07:00
stumpylog 26baf52a78 Chore: Remove the unused run_convert helper
run_convert wrapped ImageMagick for the PDF thumbnail path, but thumbnails are now produced with pdftoppm and Pillow, and the remaining convert callers invoke the binary directly through run_subprocess. The helper had no callers left, so it and its now unused os import are removed. CONVERT_BINARY is still used by those callers and stays.
2026-10-07 12:54:40 -07:00
stumpylog 3143d1936a Chore: Cover the qpdf repair and double failure thumbnail paths
The existing fallback test fails the first rasterization artificially and only proves the retry plumbing. Nothing showed that a PDF which pdftoppm genuinely cannot read is repaired by qpdf into a real rendered thumbnail, and nothing covered the case where both the render and the repair fail.

Add a test which builds a PDF with its cross reference table and trailer cut off, confirms pdftoppm rejects it, and checks that the thumbnail produced via the qpdf copy is a real 500px page while the original file is untouched. Add a parametrized test for the double failure, with qpdf itself failing and with the repaired copy still unrenderable, asserting the result is a copy of the default thumbnail.
2026-10-07 12:51:21 -07:00
stumpylog 1a9b08394c Chore: Keep PDF thumbnails at 500px wide for small and odd-sized pages
The new pdftoppm thumbnail path capped its computed DPI at 72, on the
assumption that the old "-scale 500x5000>" never enlarged a page past its
natural size. The old pipeline actually rendered at 300 DPI before
shrinking, so any page wider than about 120pt used to produce a 500px wide
thumbnail, while the 72 DPI cap made A5 pages, receipts and other small
documents come out noticeably narrower. Rounding the DPI to the nearest
integer could also land a few pixels short of 500px (a landscape Letter page
rendered 495px wide at 45 DPI), and the Pillow clamp only ever shrinks.

The DPI is now capped at 300 to match the old render density, and rounded
up so the render always lands at or just above the target, letting the
shrink-only clamp trim it to exactly 500px. The page size helper also no
longer logs a full traceback for every encrypted or corrupt PDF, matching
the page count helper.
2026-10-07 12:46:49 -07:00
stumpylog ae3d8680fa Chore: Generate PDF thumbnails with pdftoppm and Pillow instead of ImageMagick
PDF thumbnails were produced by handing the original PDF to ImageMagick's
convert, which delegates PDF handling to Ghostscript. When that failed, a
fallback invoked gs directly and then ran convert a second time just to
encode the WebP, so a single thumbnail could take three subprocess calls
through two general purpose tools for what is only "render page one small".

The first page is now rasterized with Poppler's pdftoppm, which is already
installed for pdftotext, at a DPI computed from the page's own CropBox and
effective rotation as read by pikepdf. The DPI is chosen so the page fits
500x5000 pixels in a single render and is capped at 72 so small pages are
never enlarged, matching the previous shrink-only scale. Pillow then
flattens any alpha onto white, applies a no-enlarge safety clamp and saves
the WebP in process. If the geometry cannot be read, a fixed 150 DPI is
used and the clamp keeps the output in bounds.

The Ghostscript fallback is replaced with a qpdf repair and retry: the PDF
is copied, repaired in place with qpdf (treating its "repaired with
warnings" exit status as success), and rasterized again. If that also
fails, the default thumbnail is used as before.
2026-10-07 12:43:18 -07:00
GitHub Actions ee34a6598e Auto translate strings 2026-10-06 15:13:08 +00:00
Trenton H 3a3b3ef66a Chore: Upgrade runners to Ubuntu 26.04 (#14347)
* Moves runners to 26.04 and a few jobs to -slim variant

* Probably fixing the imagemagik 7 problems and maybe the frontend playwright thing?

* Compare thumbnails, but allow a little difference in the perceptual hash
2026-10-06 08:11:44 -07:00
github-actions[bot]andCrowdin Bot 662a5237a3 New Crowdin translations by GitHub Action (#14361)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-10-06 03:29:47 +00:00
GitHub Actions 562c2d0985 Auto translate strings 2026-10-06 03:19:38 +00:00
shamoon 6cdb93cd59 Fix: dont overcount same title or date suggestions 2026-10-05 20:18:09 -07:00
GitHub Actions 85fcf66691 Auto translate strings 2026-10-06 03:05:53 +00:00
shamoon 73b2321e87 Fix create after ML-only 2026-10-05 20:04:35 -07:00
GitHub Actions 5eecd6971e Auto translate strings 2026-10-06 02:34:22 +00:00
shamoon bc92e4ff98 Fix suggestions dropdown counts and empty state 2026-10-05 19:32:56 -07:00
github-actions[bot]andCrowdin Bot dacc213917 New Crowdin translations by GitHub Action (#14214)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-10-06 02:09:40 +00:00
GitHub Actions c780586c72 Auto translate strings 2026-10-05 16:26:46 +00:00
jurassicparkicecreamandshamoon 62cc31fbdb Feature: store barcode contents, list and search them (#14276)
* Feature: store barcode contents, list and search them

New setting PAPERLESS_CONSUMER_STORE_BARCODE_VALUES (off by default)
stores all barcodes found during consumption with the document: page,
type and content. They are listed on the metadata tab with a copy
button, returned by the documents API and searchable with barcodes:
in the advanced search. Versions keep their own barcodes, reprocessing
reads them again. Refs #9898

* Tests: cover the remaining barcode branches

Covers unchanged and failing barcode reads on reprocessing, unsupported files and DocumentBarcode.__str__, and uses toHaveLength in the barcode list spec as suggested by SonarCloud.

* Address review: keep barcodes when they can't be read, format choices

- Reprocessing and new versions keep the stored barcodes when the file
  can't be scanned or the scan fails, and replace them atomically.
- Format is a TextChoices of the zxing-cpp formats, with a test.
- Shared scan code, latest_version helper, TypedDict, serializer reuse.
- Barcodes in the split manifest, export/import tests, pytest-style tests.

* Barcode tests: TIFF reprocess, format check both ways, fixtures

- Reprocessing a TIFF with TIFF support off keeps the stored barcodes.
- The format test also fails when zxing-cpp drops a format.
- Fixtures in place of the sample dir mixin, plugin-level disabled test.
- Format labels aren't translated, OpenAPI enum named BarcodeFormatEnum.

* Review: module-level zxing reader, shorter barcode docs

- read_barcodes_zxing is a module-level function used by scan_pdf.
- Drop the trivial __str__ test, mark it no cover.
- Shorten the barcode docs and remove the duplicate in configuration.md.

* Fix header

* Use utility class

* Return barcodes from the metadata endpoint only

Drop the barcodes field and its prefetches from the document serializer,
as agreed in the review. Also remove the now empty component stylesheet.

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-10-05 16:25:19 +00:00
Trenton H 312684aef3 Fix: Set the ProcessedMail owner based on the rule owner in all cases (#14356) 2026-10-05 05:57:41 -07:00
dependabot[bot]andTrenton H 1d42d372a2 Chore(deps): Bump django-filter from 25.2 to 26.1 (#14337)
* Chore(deps): Bump django-filter from 25.2 to 26.1

Bumps [django-filter](https://github.com/carltongibson/django-filter) from 25.2 to 26.1.
- [Release notes](https://github.com/carltongibson/django-filter/releases)
- [Changelog](https://github.com/carltongibson/django-filter/blob/main/CHANGES.rst)
- [Commits](https://github.com/carltongibson/django-filter/compare/25.2...26.1)

---
updated-dependencies:
- dependency-name: django-filter
  dependency-version: '26.1'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Linting

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
2026-10-04 22:24:13 +00:00
dependabot[bot] 08c4f3d3aa Chore(deps): Bump the utilities-patch group across 1 directory with 2 updates (#14338)
Bumps the utilities-patch group with 2 updates in the / directory: [psycopg-pool](https://github.com/psycopg/psycopg) and [ruff](https://github.com/astral-sh/ruff).


Updates `psycopg-pool` from 3.3.2 to 3.3.3
- [Changelog](https://github.com/psycopg/psycopg/blob/master/docs/news.rst)
- [Commits](https://github.com/psycopg/psycopg/compare/3.3.2...3.3.3)

Updates `ruff` from 0.16.8 to 0.16.9
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.16.8...0.16.9)

---
updated-dependencies:
- dependency-name: psycopg-pool
  dependency-version: 3.3.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: ruff
  dependency-version: 0.16.9
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-04 13:54:08 -07:00
163 changed files with 43549 additions and 34213 deletions

No files matched your search

+4 -4
View File
@@ -80,7 +80,7 @@ jobs:
needs: changes
if: needs.changes.outputs.backend_changed == 'true'
name: "Python ${{ matrix.python-version }}"
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
strategy:
@@ -111,10 +111,10 @@ jobs:
timeout-minutes: 12
uses: $/.github/actions/apt-install
with:
packages: unpaper tesseract-ocr imagemagick ghostscript poppler-utils
packages: unpaper tesseract-ocr imagemagick ghostscript poppler-utils qpdf
- name: Configure ImageMagick
run: |
sudo cp docker/rootfs/etc/ImageMagick-6/paperless-policy.xml /etc/ImageMagick-6/policy.xml
sudo cp docker/rootfs/etc/ImageMagick-6/paperless-policy.xml /etc/ImageMagick-7/policy.xml
- name: Install Python dependencies
env:
PYTHON_VERSION: ${{ steps.setup-python.outputs.python-version }}
@@ -158,7 +158,7 @@ jobs:
needs: changes
if: needs.changes.outputs.backend_changed == 'true'
name: Check project typing
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
env:
+3 -3
View File
@@ -24,10 +24,10 @@ jobs:
fail-fast: false
matrix:
include:
- runner: ubuntu-24.04
- runner: ubuntu-26.04
arch: amd64
platform: linux/amd64
- runner: ubuntu-24.04-arm
- runner: ubuntu-26.04-arm
arch: arm64
platform: linux/arm64
runs-on: ${{ matrix.runner }}
@@ -163,7 +163,7 @@ jobs:
archive: false
merge-and-push:
name: Merge and Push Manifest
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
needs: build-arch
if: needs.build-arch.outputs.should-push == 'true'
environment: image-publishing
+2 -2
View File
@@ -65,7 +65,7 @@ jobs:
needs: changes
if: needs.changes.outputs.docs_changed == 'true'
name: Build Documentation
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
steps:
- uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
- name: Checkout
@@ -102,7 +102,7 @@ jobs:
name: Deploy Documentation
needs: [changes, build]
if: github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.changes.outputs.docs_changed == 'true'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
pages: write
id-token: write
+6 -6
View File
@@ -72,7 +72,7 @@ jobs:
needs: changes
if: needs.changes.outputs.frontend_changed == 'true'
name: Install Dependencies
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
steps:
@@ -104,7 +104,7 @@ jobs:
name: Lint
needs: [changes, install-dependencies]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
steps:
@@ -137,7 +137,7 @@ jobs:
name: "Unit Tests (${{ matrix.shard-index }}/${{ matrix.shard-count }})"
needs: [changes, install-dependencies]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
strategy:
@@ -188,10 +188,10 @@ jobs:
name: E2E Tests
needs: [changes, install-dependencies]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
container: mcr.microsoft.com/playwright:v1.62.1-noble
container: mcr.microsoft.com/playwright:v1.62.1-resolute
env:
PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
@@ -246,7 +246,7 @@ jobs:
name: Frontend Build
needs: [changes, unit-tests, e2e-tests]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
steps:
+5 -5
View File
@@ -14,7 +14,7 @@ permissions: {}
jobs:
wait-for-docker:
name: Wait for Docker Build
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
checks: read
statuses: read
@@ -30,7 +30,7 @@ jobs:
build-release:
name: Build Release
needs: wait-for-docker
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
steps:
@@ -73,7 +73,7 @@ jobs:
timeout-minutes: 12
uses: $/.github/actions/apt-install
with:
packages: gettext liblept5
packages: gettext libleptonica6
# ---- Build Documentation ----
- name: Build documentation
env:
@@ -145,7 +145,7 @@ jobs:
publish-release:
name: Publish Release
needs: build-release
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: write
pull-requests: write
@@ -197,7 +197,7 @@ jobs:
name: Append Changelog
needs: publish-release
if: needs.publish-release.outputs.prerelease == 'false'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: write
pull-requests: write
+2 -2
View File
@@ -15,7 +15,7 @@ permissions:
jobs:
zizmor:
name: Run zizmor
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
contents: read
actions: read
@@ -29,7 +29,7 @@ jobs:
uses: zizmorcore/zizmor-action@cc914d7f3750a2d13d75c7f184a1060aa0e9d482 # v0.6.4
semgrep:
name: Semgrep CE
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
container:
image: semgrep/semgrep:1.155.0@sha256:cc869c685dcc0fe497c86258da9f205397d8108e56d21a86082ea4886e52784d
if: github.actor != 'dependabot[bot]'
+2 -2
View File
@@ -17,7 +17,7 @@ jobs:
cleanup-images:
name: Cleanup Image Tags for ${{ matrix.primary-name }}
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
environment: registry-maintenance
strategy:
fail-fast: false
@@ -42,7 +42,7 @@ jobs:
cleanup-untagged-images:
name: Cleanup Untagged Images Tags for ${{ matrix.primary-name }}
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
needs:
- cleanup-images
environment: registry-maintenance
+1 -1
View File
@@ -21,7 +21,7 @@ on:
jobs:
analyze:
name: Analyze
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
actions: read
contents: read
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
synchronize-with-crowdin:
name: Crowdin Sync
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
environment: translation-sync
steps:
- name: Checkout
+1 -1
View File
@@ -7,7 +7,7 @@ jobs:
# Note: peakoss/anti-slop does not support the `issues` event yet (all of its
# issue inputs are still commented out upstream), so the checks that the PR Bot
# workflow gets from the action are implemented manually here.
runs-on: ubuntu-latest
runs-on: ubuntu-slim
permissions:
issues: write
steps:
+2 -2
View File
@@ -4,7 +4,7 @@ on:
types: [opened]
jobs:
Anti-slop:
runs-on: ubuntu-latest
runs-on: ubuntu-slim
permissions:
contents: read
issues: read
@@ -24,7 +24,7 @@ jobs:
ASLOP-PR-VERIFY
pr-bot:
name: Automated PR Bot
runs-on: ubuntu-latest
runs-on: ubuntu-slim
# Runs after Anti-slop so the welcome comment can see whether the PR was closed
# instead of racing it. Still runs if that job fails, so labeling is not lost.
needs: Anti-slop
+1 -1
View File
@@ -12,7 +12,7 @@ permissions:
jobs:
pr_opened_or_reopened:
name: pr_opened_or_reopened
runs-on: ubuntu-24.04
runs-on: ubuntu-slim
permissions:
# write permission is required for autolabeler
pull-requests: write
+5 -5
View File
@@ -9,7 +9,7 @@ jobs:
stale:
name: 'Stale'
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
issues: write
pull-requests: write
@@ -34,7 +34,7 @@ jobs:
lock-threads:
name: 'Lock Old Threads'
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-26.04
permissions:
issues: write
pull-requests: write
@@ -58,7 +58,7 @@ jobs:
close-answered-discussions:
name: 'Close Answered Discussions'
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-slim
permissions:
discussions: write
steps:
@@ -117,7 +117,7 @@ jobs:
close-outdated-discussions:
name: 'Close Outdated Discussions'
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-slim
permissions:
discussions: write
steps:
@@ -211,7 +211,7 @@ jobs:
close-unsupported-feature-requests:
name: 'Close Unsupported Feature Requests'
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
runs-on: ubuntu-slim
permissions:
discussions: write
steps:
+1 -1
View File
@@ -8,7 +8,7 @@ env:
jobs:
generate-translate-strings:
name: Generate Translation Strings
runs-on: ubuntu-latest
runs-on: ubuntu-26.04
environment: translation-sync
permissions:
contents: write
+13
View File
@@ -1010,6 +1010,19 @@ documents to both separate and categorize them in a single operation.
**Example:** A 6-page scan with TAG:invoice on page 3 and TAG:receipt on page 5 will create
three documents: pages 1-2 (no tags), pages 3-4 (tagged "invoice"), and pages 5-6 (tagged "receipt").
### Barcode Contents {#barcode-contents}
By default, Paperless only uses barcodes for splitting, ASNs and tags. With
[`PAPERLESS_CONSUMER_STORE_BARCODE_VALUES`](configuration.md#PAPERLESS_CONSUMER_STORE_BARCODE_VALUES)
enabled, it stores the content of every barcode with the document, e.g. payment codes or QR codes.
- Barcodes are listed on the **Metadata** tab with page, type and content, and can be copied.
- The API returns them in the `barcodes` field of `/api/documents/{id}/metadata/`.
- They can be [searched](usage.md#searching-barcodes), e.g. `barcodes:DE89370400440532013000`.
- Only the first [`PAPERLESS_CONSUMER_BARCODE_MAX_PAGES`](configuration.md#PAPERLESS_CONSUMER_BARCODE_MAX_PAGES)
pages are scanned. Reprocessing reads the barcodes of existing documents.
- Each version keeps its own barcodes, and the newest version's are shown and searched.
## Automatic collation of double-sided documents {#collate}
!!! note
+13 -18
View File
@@ -1315,29 +1315,17 @@ valid crontab(5) expression describing when to run.
#### [`PAPERLESS_CONVERT_MEMORY_LIMIT=<num>`](#PAPERLESS_CONVERT_MEMORY_LIMIT) {#PAPERLESS_CONVERT_MEMORY_LIMIT}
: On smaller systems, or even in the case of Very Large Documents, the
consumer may explode, complaining about how it's "unable to extend
pixel cache". In such cases, try setting this to a reasonably low
value, like 32. The default is to use whatever is necessary to do
everything without writing to disk, and units are in megabytes.
!!! warning
For more information on how to use this value, you should search the
web for "MAGICK_MEMORY_LIMIT".
Defaults to 0, which disables the limit.
Deprecated and has no effect, since PDF thumbnails no longer use
ImageMagick. It will be removed in a future release.
#### [`PAPERLESS_CONVERT_TMPDIR=<path>`](#PAPERLESS_CONVERT_TMPDIR) {#PAPERLESS_CONVERT_TMPDIR}
: Similar to the memory limit, if you've got a small system and your
OS mounts /tmp as tmpfs, you should set this to a path that's on a
physical disk, like /home/your_user/tmp or something. ImageMagick
will use this as scratch space when crunching through very large
documents.
!!! warning
For more information on how to use this value, you should search the
web for "MAGICK_TMPDIR".
Default is none, which disables the temporary directory.
Deprecated and has no effect, since PDF thumbnails no longer use
ImageMagick. It will be removed in a future release.
#### [`PAPERLESS_APPS=<string>`](#PAPERLESS_APPS) {#PAPERLESS_APPS}
@@ -1796,6 +1784,13 @@ assigns or creates tags if a properly formatted barcode is detected.
Defaults to false.
#### [`PAPERLESS_CONSUMER_STORE_BARCODE_VALUES=<bool>`](#PAPERLESS_CONSUMER_STORE_BARCODE_VALUES) {#PAPERLESS_CONSUMER_STORE_BARCODE_VALUES}
: Stores the content of every barcode found during consumption, see
[Barcode Contents](advanced_usage.md#barcode-contents).
Defaults to false.
## Audit Trail
#### [`PAPERLESS_AUDIT_LOG_ENABLED=<bool>`](#PAPERLESS_AUDIT_LOG_ENABLED) {#PAPERLESS_AUDIT_LOG_ENABLED}
+4 -7
View File
@@ -177,12 +177,12 @@ to a positive number to enable polling and disable native filesystem notificatio
- `pkg-config` for mysqlclient (python dependency)
- `fonts-liberation` for generating thumbnails for plain text
files
- `imagemagick` >= 6 for PDF conversion
- `imagemagick` >= 6 for image alpha handling
- `gnupg` for decrypting GPG-encrypted email
- `libpq-dev` for PostgreSQL
- `libmagic-dev` for mime type detection
- `mariadb-client` for MariaDB compile time
- `poppler-utils` for barcode detection
- `poppler-utils` for thumbnail generation and barcode detection
Use this list for your preferred package management:
@@ -416,11 +416,8 @@ to a positive number to enable polling and disable native filesystem notificatio
You may need to change the path in the files. Example:
`ExecStart=/opt/paperless/.local/bin/celery --app paperless worker --loglevel INFO`
12. Configure ImageMagick to allow processing of PDF documents and disable
formats that Paperless-ngx does not use. Most distributions disable PDF
processing by default, since PDF documents can contain malware. If you
don't enable it, Paperless-ngx will fall back to Ghostscript for certain
steps such as thumbnail generation.
12. Harden ImageMagick by disabling formats that Paperless-ngx does not use.
PDF processing is not needed and should stay disabled.
Configure the active ImageMagick policy file (commonly
`/etc/ImageMagick-6/policy.xml` or `/etc/ImageMagick-7/policy.xml`) and
+14
View File
@@ -1061,6 +1061,20 @@ notes.user:alice notes.note:insurance
The bare `notes:` prefix is shorthand for `notes.note:`.
#### Searching barcodes
If [barcode contents are stored](advanced_usage.md#barcode-contents), they can be searched by
content or type, but only with a field name:
```
barcodes.value:DE89370400440532013000
barcodes.format:qrcode
barcodes:wifi barcodes:guest
```
`barcodes:` is shorthand for `barcodes.value:`. Separators are stripped, so each part of e.g.
`WIFI:S:Guest;P:secret;;` can be searched on its own.
All of these can be combined. Syntax not described here may not work as expected, and an unknown field name is searched as ordinary text.
!!! note
-2
View File
@@ -50,8 +50,6 @@ PAPERLESS_SECRET_KEY=change-me
#PAPERLESS_OCR_ROTATE_PAGES=true
#PAPERLESS_OCR_ROTATE_PAGES_THRESHOLD=12.0
#PAPERLESS_OCR_USER_ARGS={}
#PAPERLESS_CONVERT_MEMORY_LIMIT=0
#PAPERLESS_CONVERT_TMPDIR=/var/tmp/paperless
# Software tweaks
+2 -2
View File
@@ -33,7 +33,7 @@ dependencies = [
"django-compression-middleware~=0.5.0",
"django-cors-headers~=4.9.0",
"django-extensions~=4.1",
"django-filter~=25.1",
"django-filter>=25.1,<27",
"django-guardian>=3.3.3,<3.6",
"django-multiselectfield~=1.0.1",
"django-rich~=2.2.0",
@@ -90,7 +90,7 @@ postgres = [
"psycopg[c,pool]==3.3.4",
# Direct dependency for proper resolution of the pre-built wheels
"psycopg-c==3.3.4",
"psycopg-pool==3.3.2",
"psycopg-pool==3.3.3",
]
webserver = [
"granian[uvloop]>=2.7,<2.9",
+150 -113
View File
@@ -698,7 +698,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">457,458</context>
<context context-type="linenumber">463,464</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
@@ -870,7 +870,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">476</context>
<context context-type="linenumber">482</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -1373,7 +1373,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1901</context>
<context context-type="linenumber">1907</context>
</context-group>
</trans-unit>
<trans-unit id="1577733187050997705" datatype="html">
@@ -1451,7 +1451,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">402,403</context>
<context context-type="linenumber">408,409</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1700,7 +1700,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">177</context>
<context context-type="linenumber">179</context>
</context-group>
</trans-unit>
<trans-unit id="2691296884221415710" datatype="html">
@@ -1715,7 +1715,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">182</context>
<context context-type="linenumber">184</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1750,7 +1750,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">186</context>
<context context-type="linenumber">188</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1785,7 +1785,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">190</context>
<context context-type="linenumber">192</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -2478,7 +2478,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">711</context>
<context context-type="linenumber">713</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html</context>
@@ -3392,11 +3392,11 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1515</context>
<context context-type="linenumber">1521</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1902</context>
<context context-type="linenumber">1908</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -4012,7 +4012,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1468</context>
<context context-type="linenumber">1474</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -4156,7 +4156,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1955</context>
<context context-type="linenumber">1961</context>
</context-group>
</trans-unit>
<trans-unit id="6661109599266152398" datatype="html">
@@ -4167,7 +4167,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1956</context>
<context context-type="linenumber">1962</context>
</context-group>
</trans-unit>
<trans-unit id="5162686434580248853" datatype="html">
@@ -4178,7 +4178,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1957</context>
<context context-type="linenumber">1963</context>
</context-group>
</trans-unit>
<trans-unit id="6665634854532231106" datatype="html">
@@ -5430,7 +5430,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">368,369</context>
<context context-type="linenumber">374,375</context>
</context-group>
</trans-unit>
<trans-unit id="8057014866157903311" datatype="html">
@@ -6311,7 +6311,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1472</context>
<context context-type="linenumber">1478</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -6344,7 +6344,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">175</context>
<context context-type="linenumber">177</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -7176,6 +7176,10 @@
<context context-type="sourcefile">src/app/components/common/system-status-dialog/system-status-dialog.component.html</context>
<context context-type="linenumber">56,57</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-barcodes/document-barcodes.component.html</context>
<context context-type="linenumber">5,6</context>
</context-group>
</trans-unit>
<trans-unit id="1230154438678955604" datatype="html">
<source>Change</source>
@@ -7748,8 +7752,8 @@
<context context-type="linenumber">120</context>
</context-group>
</trans-unit>
<trans-unit id="5700628356844396417" datatype="html">
<source>{VAR_PLURAL, plural, =1 {1 existing value suggested below} other {<x id="INTERPOLATION"/> existing values suggested below}}</source>
<trans-unit id="5377184518735933155" datatype="html">
<source>{VAR_PLURAL, plural, =1 {1 suggestion available below} other {<x id="INTERPOLATION"/> suggestions available below}}</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">54</context>
@@ -8333,6 +8337,10 @@
</trans-unit>
<trans-unit id="1407560924967345762" datatype="html">
<source>Page</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-barcodes/document-barcodes.component.html</context>
<context context-type="linenumber">4,5</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">5,6</context>
@@ -8346,6 +8354,28 @@
<context context-type="linenumber">34</context>
</context-group>
</trans-unit>
<trans-unit id="6205355627445317276" datatype="html">
<source>Content</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-barcodes/document-barcodes.component.html</context>
<context context-type="linenumber">6,7</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">300,301</context>
</context-group>
</trans-unit>
<trans-unit id="3846359579066496296" datatype="html">
<source>Copy content</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-barcodes/document-barcodes.component.html</context>
<context context-type="linenumber">32,33</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-barcodes/document-barcodes.component.html</context>
<context context-type="linenumber">40,41</context>
</context-group>
</trans-unit>
<trans-unit id="2266163016683537825" datatype="html">
<source>of <x id="INTERPOLATION" equiv-text="{{previewNumPages()}}"/></source>
<context-group purpose="location">
@@ -8477,39 +8507,32 @@
<source>Details</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">172,173</context>
<context context-type="linenumber">174,175</context>
</context-group>
</trans-unit>
<trans-unit id="5114742157723900905" datatype="html">
<source>Date created</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">179</context>
<context context-type="linenumber">181</context>
</context-group>
</trans-unit>
<trans-unit id="5607669932062416162" datatype="html">
<source>Default</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">191</context>
<context context-type="linenumber">193</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
<context context-type="linenumber">71</context>
</context-group>
</trans-unit>
<trans-unit id="6205355627445317276" datatype="html">
<source>Content</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">298,299</context>
</context-group>
</trans-unit>
<trans-unit id="218403386307979629" datatype="html">
<source>Metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">307,308</context>
<context context-type="linenumber">309,310</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
@@ -8520,228 +8543,235 @@
<source>Date modified</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">314,315</context>
<context context-type="linenumber">316,317</context>
</context-group>
</trans-unit>
<trans-unit id="6392918669949841614" datatype="html">
<source>Date added</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">318,319</context>
<context context-type="linenumber">320,321</context>
</context-group>
</trans-unit>
<trans-unit id="146828917013192897" datatype="html">
<source>Media filename</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">322,323</context>
<context context-type="linenumber">324,325</context>
</context-group>
</trans-unit>
<trans-unit id="4500855521601039868" datatype="html">
<source>Original filename</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">326,327</context>
<context context-type="linenumber">328,329</context>
</context-group>
</trans-unit>
<trans-unit id="2659735245739197634" datatype="html">
<source>Original SHA256 checksum</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">330,331</context>
<context context-type="linenumber">332,333</context>
</context-group>
</trans-unit>
<trans-unit id="5888243105821763422" datatype="html">
<source>Original file size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">334,335</context>
<context context-type="linenumber">336,337</context>
</context-group>
</trans-unit>
<trans-unit id="2696647325713149563" datatype="html">
<source>Original mime type</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">338,339</context>
<context context-type="linenumber">340,341</context>
</context-group>
</trans-unit>
<trans-unit id="6714358112223607756" datatype="html">
<source>Archive SHA256 checksum</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">343,344</context>
<context context-type="linenumber">345,346</context>
</context-group>
</trans-unit>
<trans-unit id="6033581412811562084" datatype="html">
<source>Archive file size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">349,350</context>
<context context-type="linenumber">351,352</context>
</context-group>
</trans-unit>
<trans-unit id="8459338343197260257" datatype="html">
<source>Barcodes</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">360,361</context>
</context-group>
</trans-unit>
<trans-unit id="6992781481378431874" datatype="html">
<source>Original document metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">358</context>
<context context-type="linenumber">364</context>
</context-group>
</trans-unit>
<trans-unit id="2846565152091361585" datatype="html">
<source>Archived document metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">361</context>
<context context-type="linenumber">367</context>
</context-group>
</trans-unit>
<trans-unit id="7206723502037428235" datatype="html">
<source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="{{document().notes.length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span" equiv-text="&lt;/span&gt;"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">380,383</context>
<context context-type="linenumber">386,389</context>
</context-group>
</trans-unit>
<trans-unit id="186236568870281953" datatype="html">
<source>History</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">391,392</context>
<context context-type="linenumber">397,398</context>
</context-group>
</trans-unit>
<trans-unit id="8236092845697214347" datatype="html">
<source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="{{ document().duplicate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span" equiv-text="&lt;/span&gt;"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">414,417</context>
<context context-type="linenumber">420,423</context>
</context-group>
</trans-unit>
<trans-unit id="6449374629822973702" datatype="html">
<source>Duplicate documents detected:</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">419,420</context>
<context context-type="linenumber">425,426</context>
</context-group>
</trans-unit>
<trans-unit id="14058600336670816" datatype="html">
<source>In trash</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">430,431</context>
<context context-type="linenumber">436,437</context>
</context-group>
</trans-unit>
<trans-unit id="5129524307369213584" datatype="html">
<source>Save &amp; next</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">459</context>
<context context-type="linenumber">465</context>
</context-group>
</trans-unit>
<trans-unit id="4910102545766233758" datatype="html">
<source>Save &amp; close</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">461</context>
<context context-type="linenumber">467</context>
</context-group>
</trans-unit>
<trans-unit id="3823219296477075982" datatype="html">
<source>Discard</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">463,464</context>
<context context-type="linenumber">469,470</context>
</context-group>
</trans-unit>
<trans-unit id="1309556917227148591" datatype="html">
<source>Document loading...</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">471</context>
<context context-type="linenumber">477</context>
</context-group>
</trans-unit>
<trans-unit id="8191371354890763172" datatype="html">
<source>Enter Password</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">525</context>
<context context-type="linenumber">531</context>
</context-group>
</trans-unit>
<trans-unit id="5758784066858623886" datatype="html">
<source>Error retrieving metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">461</context>
<context context-type="linenumber">463</context>
</context-group>
</trans-unit>
<trans-unit id="2218903673684131427" datatype="html">
<source>An error occurred loading content: <x id="PH" equiv-text="err.message ?? err.toString()"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">564,566</context>
<context context-type="linenumber">566,568</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1045,1047</context>
<context context-type="linenumber">1047,1049</context>
</context-group>
</trans-unit>
<trans-unit id="6357361810318120957" datatype="html">
<source>Document was updated</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">706</context>
<context context-type="linenumber">708</context>
</context-group>
</trans-unit>
<trans-unit id="5154064822428631306" datatype="html">
<source>Document was updated at <x id="PH" equiv-text="formattedModified"/>.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">707</context>
<context context-type="linenumber">709</context>
</context-group>
</trans-unit>
<trans-unit id="8462497568316256794" datatype="html">
<source>Reload to discard your local unsaved edits and load the latest remote version.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">708</context>
<context context-type="linenumber">710</context>
</context-group>
</trans-unit>
<trans-unit id="7967484035994732534" datatype="html">
<source>Reload</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">710</context>
<context context-type="linenumber">712</context>
</context-group>
</trans-unit>
<trans-unit id="2907037627372942104" datatype="html">
<source>Document reloaded with latest changes.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">766</context>
<context context-type="linenumber">768</context>
</context-group>
</trans-unit>
<trans-unit id="6435639868943916539" datatype="html">
<source>Document reloaded.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">777</context>
<context context-type="linenumber">779</context>
</context-group>
</trans-unit>
<trans-unit id="6142395741265832184" datatype="html">
<source>Next document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">879</context>
<context context-type="linenumber">881</context>
</context-group>
</trans-unit>
<trans-unit id="651985345816518480" datatype="html">
<source>Previous document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">889</context>
<context context-type="linenumber">891</context>
</context-group>
</trans-unit>
<trans-unit id="2885986061416655600" datatype="html">
<source>Close document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">897</context>
<context context-type="linenumber">899</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
@@ -8752,28 +8782,28 @@
<source>Save document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">904</context>
<context context-type="linenumber">906</context>
</context-group>
</trans-unit>
<trans-unit id="1784543155727940353" datatype="html">
<source>Save and close / next</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">913</context>
<context context-type="linenumber">915</context>
</context-group>
</trans-unit>
<trans-unit id="7427704425579737895" datatype="html">
<source>Error retrieving version content</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1027</context>
<context context-type="linenumber">1029</context>
</context-group>
</trans-unit>
<trans-unit id="159901853873315050" datatype="html">
<source>Unsaved Changes</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1071</context>
<context context-type="linenumber">1073</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/guards/dirty-form.guard.ts</context>
@@ -8796,74 +8826,74 @@
<source>You have unsaved changes to the content of this version.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1072</context>
<context context-type="linenumber">1074</context>
</context-group>
</trans-unit>
<trans-unit id="85184271222513014" datatype="html">
<source>Switching versions will discard them.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1073</context>
<context context-type="linenumber">1075</context>
</context-group>
</trans-unit>
<trans-unit id="2565707334844767610" datatype="html">
<source>Discard and switch</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1075</context>
<context context-type="linenumber">1077</context>
</context-group>
</trans-unit>
<trans-unit id="2109314380040637387" datatype="html">
<source>Save and switch</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1077</context>
<context context-type="linenumber">1079</context>
</context-group>
</trans-unit>
<trans-unit id="3456881259945295697" datatype="html">
<source>Error retrieving suggestions.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1120</context>
<context context-type="linenumber">1122</context>
</context-group>
</trans-unit>
<trans-unit id="2194092841814123758" datatype="html">
<source>Document &quot;<x id="PH" equiv-text="newValues.title"/>&quot; saved successfully.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1347</context>
<context context-type="linenumber">1353</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1375</context>
<context context-type="linenumber">1381</context>
</context-group>
</trans-unit>
<trans-unit id="6626387786259219838" datatype="html">
<source>Error saving document &quot;<x id="PH" equiv-text="this.document().title"/>&quot;</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1381</context>
<context context-type="linenumber">1387</context>
</context-group>
</trans-unit>
<trans-unit id="448882439049417053" datatype="html">
<source>Error saving document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1436</context>
<context context-type="linenumber">1442</context>
</context-group>
</trans-unit>
<trans-unit id="8410796510716511826" datatype="html">
<source>Do you really want to move the document &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash?</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1469</context>
<context context-type="linenumber">1475</context>
</context-group>
</trans-unit>
<trans-unit id="282586936710748252" datatype="html">
<source>Documents can be restored prior to permanent deletion.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1470</context>
<context context-type="linenumber">1476</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8874,14 +8904,14 @@
<source>Error deleting document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1491</context>
<context context-type="linenumber">1497</context>
</context-group>
</trans-unit>
<trans-unit id="619486176823357521" datatype="html">
<source>Reprocess confirm</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1511</context>
<context context-type="linenumber">1517</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8892,102 +8922,102 @@
<source>This operation will permanently recreate the archive file for this document.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1512</context>
<context context-type="linenumber">1518</context>
</context-group>
</trans-unit>
<trans-unit id="302054111564709516" datatype="html">
<source>The archive file will be re-generated with the current settings.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1513</context>
<context context-type="linenumber">1519</context>
</context-group>
</trans-unit>
<trans-unit id="4700389117298802932" datatype="html">
<source>Reprocess operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1526</context>
<context context-type="linenumber">1532</context>
</context-group>
</trans-unit>
<trans-unit id="4409560272830824468" datatype="html">
<source>Error executing operation</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1537</context>
<context context-type="linenumber">1543</context>
</context-group>
</trans-unit>
<trans-unit id="6030453331794586802" datatype="html">
<source>Error downloading document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1601</context>
<context context-type="linenumber">1607</context>
</context-group>
</trans-unit>
<trans-unit id="4458954481601077369" datatype="html">
<source>Page Fit</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1679</context>
<context context-type="linenumber">1685</context>
</context-group>
</trans-unit>
<trans-unit id="4663705961777238777" datatype="html">
<source>PDF edit operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1922</context>
<context context-type="linenumber">1928</context>
</context-group>
</trans-unit>
<trans-unit id="9043972994040261999" datatype="html">
<source>Error executing PDF edit operation</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1934</context>
<context context-type="linenumber">1940</context>
</context-group>
</trans-unit>
<trans-unit id="6172690334763056188" datatype="html">
<source>Please enter the current password before attempting to remove it.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1945</context>
<context context-type="linenumber">1951</context>
</context-group>
</trans-unit>
<trans-unit id="968660764814228922" datatype="html">
<source>Password removal operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1979</context>
<context context-type="linenumber">1985</context>
</context-group>
</trans-unit>
<trans-unit id="2282118435712883014" datatype="html">
<source>Error executing password removal operation</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1993</context>
<context context-type="linenumber">1999</context>
</context-group>
</trans-unit>
<trans-unit id="3740891324955700797" datatype="html">
<source>Print failed.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2043</context>
<context context-type="linenumber">2049</context>
</context-group>
</trans-unit>
<trans-unit id="6457245677384603573" datatype="html">
<source>Error loading document for printing.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2052</context>
<context context-type="linenumber">2058</context>
</context-group>
</trans-unit>
<trans-unit id="6085793215710522488" datatype="html">
<source>An error occurred loading tiff: <x id="PH" equiv-text="err.toString()"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2135</context>
<context context-type="linenumber">2141</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2141</context>
<context context-type="linenumber">2147</context>
</context-group>
</trans-unit>
<trans-unit id="4958946940233632319" datatype="html">
@@ -12053,123 +12083,130 @@
<context context-type="linenumber">328</context>
</context-group>
</trans-unit>
<trans-unit id="906679600349814230" datatype="html">
<source>Store Barcode Contents</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">335</context>
</context-group>
</trans-unit>
<trans-unit id="7011909364081812031" datatype="html">
<source>AI Enabled</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">335</context>
<context context-type="linenumber">342</context>
</context-group>
</trans-unit>
<trans-unit id="8028880048909383956" datatype="html">
<source>Consider privacy implications when enabling AI features, especially if using a remote model.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">339</context>
<context context-type="linenumber">346</context>
</context-group>
</trans-unit>
<trans-unit id="8131374115579345652" datatype="html">
<source>LLM Embedding Backend</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">343</context>
<context context-type="linenumber">350</context>
</context-group>
</trans-unit>
<trans-unit id="6647708571891295756" datatype="html">
<source>LLM Embedding Model</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">351</context>
<context context-type="linenumber">358</context>
</context-group>
</trans-unit>
<trans-unit id="861068592166833023" datatype="html">
<source>LLM Embedding API Key</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">358</context>
<context context-type="linenumber">365</context>
</context-group>
</trans-unit>
<trans-unit id="2929108042259892948" datatype="html">
<source>Used for embeddings when set, otherwise LLM API key is used.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">360</context>
<context context-type="linenumber">367</context>
</context-group>
</trans-unit>
<trans-unit id="3554114880473286122" datatype="html">
<source>LLM Embedding Endpoint</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">366</context>
<context context-type="linenumber">373</context>
</context-group>
</trans-unit>
<trans-unit id="1044242175651289991" datatype="html">
<source>LLM Embedding Chunk Size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">373</context>
<context context-type="linenumber">380</context>
</context-group>
</trans-unit>
<trans-unit id="7218245223139363113" datatype="html">
<source>LLM Context Size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">380</context>
<context context-type="linenumber">387</context>
</context-group>
</trans-unit>
<trans-unit id="4234495692726214397" datatype="html">
<source>LLM Backend</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">387</context>
<context context-type="linenumber">394</context>
</context-group>
</trans-unit>
<trans-unit id="7935234833834000002" datatype="html">
<source>LLM Model</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">395</context>
<context context-type="linenumber">402</context>
</context-group>
</trans-unit>
<trans-unit id="1980550530387803165" datatype="html">
<source>LLM API Key</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">402</context>
<context context-type="linenumber">409</context>
</context-group>
</trans-unit>
<trans-unit id="6126617860376156501" datatype="html">
<source>LLM Endpoint</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">409</context>
<context context-type="linenumber">416</context>
</context-group>
</trans-unit>
<trans-unit id="6572826277249350975" datatype="html">
<source>LLM Output Language</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">416</context>
<context context-type="linenumber">423</context>
</context-group>
</trans-unit>
<trans-unit id="3284403507172415792" datatype="html">
<source>Language to use for generated AI suggestions. When unset, AI suggestions use the user&apos;s display language if explicitly set.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">420</context>
<context context-type="linenumber">427</context>
</context-group>
</trans-unit>
<trans-unit id="4493921125434706859" datatype="html">
<source>LLM Request Timeout</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">424</context>
<context context-type="linenumber">431</context>
</context-group>
</trans-unit>
<trans-unit id="483994032066441287" datatype="html">
<source>Timeout in seconds for LLM requests.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">428</context>
<context context-type="linenumber">435</context>
</context-group>
</trans-unit>
<trans-unit id="1055686627716339120" datatype="html">
@@ -26,7 +26,7 @@
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
<div class="list-group list-group-flush small pb-0">
@if (novelSuggestions === 0 && reusableSuggestions === 0) {
@if (novelSuggestions === 0 && fieldSuggestions === 0) {
<div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>No novel suggestions</small>
</div>
@@ -49,9 +49,9 @@
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addCorrespondent.emit(correspondent)">{{ correspondent }}</button>
}
}
@if (reusableSuggestions > 0) {
@if (fieldSuggestions > 0) {
<div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>{reusableSuggestions, plural, =1 {1 existing value suggested below} other {{{reusableSuggestions}} existing values suggested below}}</small>
<small class="text-muted small fst-italic" i18n>{fieldSuggestions, plural, =1 {1 suggestion available below} other {{{fieldSuggestions}} suggestions available below}}</small>
</div>
}
</div>
@@ -84,7 +84,7 @@ describe('SuggestionsDropdownComponent', () => {
fixture.detectChanges()
expect(fixture.nativeElement.textContent).toContain(
'2 existing values suggested below'
'2 suggestions available below'
)
expect(fixture.nativeElement.textContent).not.toContain(
'No novel suggestions'
@@ -108,7 +108,7 @@ describe('SuggestionsDropdownComponent', () => {
expect(component.totalSuggestions).toBe(4)
expect(fixture.nativeElement.textContent).toContain('Arbitration')
expect(fixture.nativeElement.textContent).toContain(
'2 existing values suggested below'
'2 suggestions available below'
)
})
@@ -125,10 +125,46 @@ describe('SuggestionsDropdownComponent', () => {
})
expect(component.novelSuggestions).toBe(0)
expect(component.totalSuggestions).toBe(5)
expect(component.totalSuggestions).toBe(6)
fixture.componentRef.setInput('appliedStoragePath', 7)
expect(component.totalSuggestions).toBe(4)
expect(component.totalSuggestions).toBe(5)
})
it('should count title and dates as field suggestions without calling them existing values', () => {
fixture.componentRef.setInput('aiEnabled', true)
fixture.componentRef.setInput('fetchedSources', [SuggestionSource.ML])
fixture.componentRef.setInput('suggestions', {
title: 'Suggested title',
dates: ['2026-01-04', '2026-02-01', '2026-03-01'],
correspondents: [1, 2, 3, 4],
document_types: [1, 2, 3, 4],
tags: [1, 2, 3, 4, 5, 6, 7, 8],
})
fixture.detectChanges()
component.clickSuggest()
fixture.detectChanges()
expect(component.reusableSuggestions).toBe(16)
expect(component.fieldSuggestions).toBe(20)
expect(component.totalSuggestions).toBe(20)
expect(fixture.nativeElement.textContent).toContain(
'20 suggestions available below'
)
expect(fixture.nativeElement.textContent).not.toContain('existing value')
})
it('should not count title or date suggestions matching the current values', () => {
fixture.componentRef.setInput('suggestions', {
title: 'Current title',
dates: ['2026-01-04', '2026-02-01'],
})
expect(component.fieldSuggestions).toBe(3)
fixture.componentRef.setInput('appliedTitle', 'Current title')
fixture.componentRef.setInput('appliedCreated', '2026-01-04')
expect(component.fieldSuggestions).toBe(1)
expect(component.totalSuggestions).toBe(1)
})
it('should show when a completed request returned no suggestions', () => {
@@ -145,6 +181,28 @@ describe('SuggestionsDropdownComponent', () => {
expect(fixture.nativeElement.textContent).toContain('No suggestions')
})
it('should wait for all pending responses before showing the empty state', () => {
fixture.componentRef.setInput('aiEnabled', true)
fixture.componentRef.setInput('source', SuggestionSource.Both)
fixture.componentRef.setInput('fetchedSources', [SuggestionSource.ML])
fixture.componentRef.setInput('suggestions', { tags: [] })
fixture.componentRef.setInput('loading', true)
fixture.detectChanges()
expect(component.noSuggestions).toBeFalsy()
expect(fixture.nativeElement.textContent).not.toContain('No suggestions')
expect(
fixture.nativeElement.querySelector('[role="status"]')
).not.toBeNull()
fixture.componentRef.setInput('loading', false)
fixture.detectChanges()
expect(component.noSuggestions).toBeTruthy()
expect(fixture.nativeElement.textContent).toContain('No suggestions')
expect(fixture.nativeElement.querySelector('[role="status"]')).toBeNull()
})
it('should not show the empty state before a request or with suggestions', () => {
expect(component.noSuggestions).toBeFalsy()
@@ -34,6 +34,8 @@ export class SuggestionsDropdownComponent {
readonly appliedCorrespondent = input<number>(null)
readonly appliedDocumentType = input<number>(null)
readonly appliedStoragePath = input<number>(null)
readonly appliedTitle = input<string>(null)
readonly appliedCreated = input<string>(null)
@Output()
getSuggestions: EventEmitter<SuggestionSource> = new EventEmitter()
@@ -128,7 +130,28 @@ export class SuggestionsDropdownComponent {
}
get totalSuggestions(): number {
return this.novelSuggestions + this.reusableSuggestions
return this.novelSuggestions + this.fieldSuggestions
}
get fieldSuggestions(): number {
return (
this.reusableSuggestions +
this.unappliedTitleSuggestions +
this.unappliedDateSuggestions
)
}
// hide a title or date suggestion equal to the current value
private get unappliedTitleSuggestions(): number {
const title = this.suggestions()?.title
return title && title !== this.appliedTitle() ? 1 : 0
}
private get unappliedDateSuggestions(): number {
const created = this.appliedCreated()
return (this.suggestions()?.dates ?? []).filter(
(date) => !created || date !== created
).length
}
private countUnapplied(suggested: number[], applied: number[]): number {
@@ -139,6 +162,7 @@ export class SuggestionsDropdownComponent {
get noSuggestions(): boolean {
const suggestions = this.suggestions()
return (
!this.loading() &&
suggestions != null &&
!suggestions.title &&
!suggestions.tags?.length &&
@@ -0,0 +1,46 @@
<table class="table table-borderless align-baseline">
<thead>
<tr>
<th i18n>Page</th>
<th i18n>Type</th>
<th i18n>Content</th>
<th></th>
</tr>
</thead>
<tbody>
@for (barcode of barcodes(); track $index) {
<tr>
<td>{{ barcode.page }}</td>
<td class="text-nowrap">{{ barcode.format }}</td>
<td class="text-break">
@if (isLink(barcode.value)) {
<a
[href]="barcode.value"
target="_blank"
rel="noopener noreferrer nofollow"
>{{ barcode.value }}</a
>
} @else {
{{ barcode.value }}
}
</td>
<td class="text-end">
<button
type="button"
class="btn btn-sm btn-outline-primary"
(click)="copy($index)"
title="Copy content"
i18n-title
>
@if (copiedIndex() === $index) {
<i-bs name="clipboard-check"></i-bs>
} @else {
<i-bs name="clipboard"></i-bs>
}
<span class="visually-hidden" i18n>Copy content</span>
</button>
</td>
</tr>
}
</tbody>
</table>
@@ -0,0 +1,73 @@
import { Clipboard } from '@angular/cdk/clipboard'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { DocumentBarcodesComponent } from './document-barcodes.component'
const barcodes = [
{ page: 1, value: 'ASN00123', format: 'Code128' },
{ page: 2, value: 'https://example.com/invoice/4711', format: 'QRCode' },
{ page: 2, value: 'javascript:alert(1)', format: 'QRCode' },
]
describe('DocumentBarcodesComponent', () => {
let component: DocumentBarcodesComponent
let fixture: ComponentFixture<DocumentBarcodesComponent>
let clipboard: Clipboard
beforeEach(async () => {
TestBed.configureTestingModule({
imports: [
DocumentBarcodesComponent,
NgxBootstrapIconsModule.pick(allIcons),
],
}).compileComponents()
fixture = TestBed.createComponent(DocumentBarcodesComponent)
component = fixture.componentInstance
clipboard = TestBed.inject(Clipboard)
fixture.componentRef.setInput('barcodes', barcodes)
fixture.detectChanges()
})
it('should display all barcodes', () => {
const rows = fixture.nativeElement.querySelectorAll('tbody tr')
expect(rows).toHaveLength(3)
expect(rows[0].textContent).toContain('ASN00123')
expect(rows[0].textContent).toContain('Code128')
})
it('should only link http(s) values', () => {
const links = fixture.nativeElement.querySelectorAll('tbody a')
expect(links).toHaveLength(1)
expect(links[0].getAttribute('href')).toEqual(
'https://example.com/invoice/4711'
)
expect(links[0].getAttribute('target')).toEqual('_blank')
})
it('should copy a value and show feedback', () => {
jest.useFakeTimers()
const copySpy = jest.spyOn(clipboard, 'copy').mockReturnValue(true)
const buttons = fixture.nativeElement.querySelectorAll('tbody button')
buttons[0].click()
fixture.detectChanges()
expect(copySpy).toHaveBeenCalledWith('ASN00123')
expect(component.copiedIndex()).toEqual(0)
expect(buttons[0].querySelector('i-bs').getAttribute('name')).toEqual(
'clipboard-check'
)
jest.advanceTimersByTime(3000)
fixture.detectChanges()
expect(component.copiedIndex()).toBeNull()
expect(buttons[0].querySelector('i-bs').getAttribute('name')).toEqual(
'clipboard'
)
jest.useRealTimers()
})
it('should not show feedback if copying failed', () => {
jest.spyOn(clipboard, 'copy').mockReturnValue(false)
component.copy(1)
expect(component.copiedIndex()).toBeNull()
})
})
@@ -0,0 +1,38 @@
import { Clipboard } from '@angular/cdk/clipboard'
import { Component, inject, input, OnDestroy, signal } from '@angular/core'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { DocumentBarcode } from 'src/app/data/document-barcode'
@Component({
selector: 'pngx-document-barcodes',
templateUrl: './document-barcodes.component.html',
imports: [NgxBootstrapIconsModule],
})
export class DocumentBarcodesComponent implements OnDestroy {
private readonly clipboard = inject(Clipboard)
readonly barcodes = input<DocumentBarcode[]>([])
readonly copiedIndex = signal<number>(null)
private copyTimeout: ReturnType<typeof setTimeout>
public isLink(value: string): boolean {
try {
const url = new URL(value.trim())
return ['http:', 'https:'].includes(url.protocol) && !!url.host
} catch {
return false
}
}
public copy(index: number) {
if (!this.clipboard.copy(this.barcodes()[index].value)) return
this.copiedIndex.set(index)
clearTimeout(this.copyTimeout)
this.copyTimeout = setTimeout(() => this.copiedIndex.set(null), 3000)
}
ngOnDestroy(): void {
clearTimeout(this.copyTimeout)
}
}
@@ -141,6 +141,8 @@
[appliedCorrespondent]="documentForm.value.correspondent"
[appliedDocumentType]="documentForm.value.document_type"
[appliedStoragePath]="documentForm.value.storage_path"
[appliedTitle]="documentForm.value.title"
[appliedCreated]="documentForm.value.created"
(getSuggestions)="getSuggestions($event)"
(sourceChange)="suggestionSourceOverride.set($event)"
(addTag)="createTag($event)"
@@ -354,6 +356,10 @@
</table>
}
@if (metadata()?.barcodes?.length > 0) {
<h6 i18n>Barcodes</h6>
<pngx-document-barcodes [barcodes]="metadata().barcodes"></pngx-document-barcodes>
}
@if (metadata()?.original_metadata?.length > 0) {
<pngx-metadata-collapse i18n-title title="Original document metadata" [metadata]="metadata()?.original_metadata"></pngx-metadata-collapse>
}
@@ -662,6 +662,45 @@ describe('DocumentDetailComponent', () => {
)
})
it.each([
['tag', 'createTag', 'tags', 'suggested_tags'],
[
'document type',
'createDocumentType',
'document_type',
'suggested_document_types',
],
[
'correspondent',
'createCorrespondent',
'correspondent',
'suggested_correspondents',
],
])(
'should create a %s after ML-only suggestions',
(_, method, field, suggestedField) => {
initNormally()
component.suggestions.set({ tags: [1] })
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
component[method]('New value')
openModal.componentInstance.succeeded.next({
id: 12,
name: 'New value',
is_inbox_tag: false,
color: '#ff0000',
text_color: '#000000',
})
if (field === 'tags') {
expect(component.tagsInput.value).toContain(12)
} else {
expect(component.documentForm.get(field).value).toBe(12)
}
expect(component.suggestions()[suggestedField]).toEqual([])
}
)
it('should support creating storage path', () => {
initNormally()
let openModal: NgbModalRef
@@ -135,6 +135,7 @@ import { ShareLinksDialogComponent } from '../common/share-links-dialog/share-li
import { SuggestionsDropdownComponent } from '../common/suggestions-dropdown/suggestions-dropdown.component'
import { DocumentNotesComponent } from '../document-notes/document-notes.component'
import { ComponentWithPermissions } from '../with-permissions/with-permissions.component'
import { DocumentBarcodesComponent } from './document-barcodes/document-barcodes.component'
import { DocumentHistoryComponent } from './document-history/document-history.component'
import { DocumentVersionDropdownComponent } from './document-version-dropdown/document-version-dropdown.component'
import { MetadataCollapseComponent } from './metadata-collapse/metadata-collapse.component'
@@ -177,6 +178,7 @@ interface IncomingDocumentUpdate {
DateComponent,
DocumentLinkComponent,
MetadataCollapseComponent,
DocumentBarcodesComponent,
PermissionsFormComponent,
SelectComponent,
TagsComponent,
@@ -1152,7 +1154,7 @@ export class DocumentDetailComponent
if (this.suggestions()) {
this.suggestions.set({
...this.suggestions(),
suggested_tags: this.suggestions().suggested_tags.filter(
suggested_tags: (this.suggestions().suggested_tags ?? []).filter(
(tag) => tag !== newTag.name
),
})
@@ -1191,10 +1193,12 @@ export class DocumentDetailComponent
this.documentForm.get('document_type').setValue(newDocumentType.id)
this.documentForm.get('document_type').markAsDirty()
if (this.suggestions()) {
this.suggestions().suggested_document_types =
this.suggestions().suggested_document_types.filter(
(dt) => dt !== newName
)
this.suggestions.set({
...this.suggestions(),
suggested_document_types: (
this.suggestions().suggested_document_types ?? []
).filter((dt) => dt !== newName),
})
}
})
}
@@ -1221,10 +1225,12 @@ export class DocumentDetailComponent
this.documentForm.get('correspondent').setValue(newCorrespondent.id)
this.documentForm.get('correspondent').markAsDirty()
if (this.suggestions()) {
this.suggestions().suggested_correspondents =
this.suggestions().suggested_correspondents.filter(
(c) => c !== newName
)
this.suggestions.set({
...this.suggestions(),
suggested_correspondents: (
this.suggestions().suggested_correspondents ?? []
).filter((c) => c !== newName),
})
}
})
}
+7
View File
@@ -0,0 +1,7 @@
export interface DocumentBarcode {
page: number
value: string
format: string
}
+4
View File
@@ -1,3 +1,5 @@
import { DocumentBarcode } from './document-barcode'
export interface DocumentMetadata {
original_checksum?: string
@@ -12,4 +14,6 @@ export interface DocumentMetadata {
has_archive_version?: boolean
lang?: string
barcodes?: DocumentBarcode[]
}
+8
View File
@@ -330,6 +330,13 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_CONSUMER_TAG_BARCODE_SPLIT',
category: ConfigCategory.Barcode,
},
{
key: 'barcode_store_values',
title: $localize`Store Barcode Contents`,
type: ConfigOptionType.Boolean,
config_key: 'PAPERLESS_CONSUMER_STORE_BARCODE_VALUES',
category: ConfigCategory.Barcode,
},
{
key: 'ai_enabled',
title: $localize`AI Enabled`,
@@ -458,6 +465,7 @@ export interface PaperlessConfig extends ObjectWithId {
barcode_enable_tag: boolean
barcode_tag_mapping: object
barcode_tag_split: boolean
barcode_store_values: boolean
remote_ocr_engine: string
remote_ocr_api_key: string
remote_ocr_endpoint: string
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+126 -74
View File
@@ -18,6 +18,7 @@ from documents.converters import convert_from_tiff_to_pdf
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource
from documents.data_models import StoredBarcode
from documents.models import Document
from documents.models import PaperlessTask
from documents.models import Tag
@@ -47,6 +48,7 @@ class Barcode:
page: int
value: str
settings: BarcodeConfig
format: str = ""
@property
def is_separator(self) -> bool:
@@ -78,6 +80,12 @@ class Barcode:
return True
return False
def stored(self) -> StoredBarcode:
"""
The barcode as it is stored with a document, page 1-indexed
"""
return {"page": self.page + 1, "value": self.value, "format": self.format}
class BarcodePlugin(ConsumeTaskPlugin):
NAME: str = "BarcodePlugin"
@@ -89,16 +97,12 @@ class BarcodePlugin(ConsumeTaskPlugin):
- ASN from barcode detection is enabled or
- Barcode support is enabled and the mime type is supported
"""
if self.settings.barcode_enable_tiff_support:
supported_mimes: set[str] = {"application/pdf", "image/tiff"}
else:
supported_mimes = {"application/pdf"}
return (
self.settings.barcode_enable_asn
or self.settings.barcodes_enabled
or self.settings.barcode_enable_tag
) and self.input_doc.mime_type in supported_mimes
or self.settings.barcode_store_values
) and self.input_doc.mime_type in scannable_mime_types(self.settings)
def get_settings(self) -> BarcodeConfig:
"""
@@ -244,6 +248,10 @@ class BarcodePlugin(ConsumeTaskPlugin):
if self.settings.barcode_enable_asn and (located_asn := self.asn) is not None:
self._apply_detected_asn(located_asn)
# After splitting too, so each split document keeps its own barcodes
if self.settings.barcode_store_values:
self.metadata.barcodes = [x.stored() for x in self.barcodes] or None
def cleanup(self) -> None:
self.temp_dir.cleanup()
@@ -262,22 +270,6 @@ class BarcodePlugin(ConsumeTaskPlugin):
)
self._tiff_conversion_done = True
@staticmethod
def read_barcodes_zxing(image: Image.Image) -> list[str]:
barcodes = []
import zxingcpp
detected_barcodes = zxingcpp.read_barcodes(image)
for barcode in detected_barcodes:
if barcode.text:
barcodes.append(barcode.text)
logger.debug(
f"Barcode of type {barcode.format} found: {barcode.text}",
)
return barcodes
def detect(self) -> None:
"""
Scan all pages of the PDF as images, updating barcodes and the pages
@@ -291,60 +283,12 @@ class BarcodePlugin(ConsumeTaskPlugin):
self.convert_from_tiff_to_pdf()
try:
# Read number of pages from pdf
with Pdf.open(self.pdf_file) as pdf:
num_of_pages = len(pdf.pages)
logger.debug(f"PDF has {num_of_pages} pages")
# Get limit from configuration
barcode_max_pages: int = (
num_of_pages
if self.settings.barcode_max_pages == 0
else self.settings.barcode_max_pages
self.barcodes = scan_pdf(
self.pdf_file,
self.settings,
Path(self.temp_dir.name),
)
if barcode_max_pages < num_of_pages: # pragma: no cover
logger.debug(
f"Barcodes detection will be limited to the first {barcode_max_pages} pages",
)
# Loop al page
for current_page_number in range(min(num_of_pages, barcode_max_pages)):
logger.debug(f"Processing page {current_page_number}")
# Convert page to image
page = convert_from_path(
self.pdf_file,
dpi=self.settings.barcode_dpi,
output_folder=self.temp_dir.name,
first_page=current_page_number + 1,
last_page=current_page_number + 1,
)[0]
# Remember filename, since it is lost by upscaling
page_filepath = Path(page.filename)
logger.debug(f"Image is at {page_filepath}")
# Upscale image if configured
factor = self.settings.barcode_upscale
if factor > 1.0:
logger.debug(
f"Upscaling image by {factor} for better barcode detection",
)
x, y = page.size
page = page.resize(
(round(x * factor), (round(y * factor))),
)
# Detect barcodes
for barcode_value in self.read_barcodes_zxing(page):
self.barcodes.append(
Barcode(current_page_number, barcode_value, self.settings),
)
# Delete temporary image file
page_filepath.unlink()
# Password protected files can't be checked
# This is the exception raised for those
except PasswordError as e:
@@ -534,3 +478,111 @@ class BarcodePlugin(ConsumeTaskPlugin):
document_paths.append(savepath)
return document_paths
def scannable_mime_types(settings: BarcodeConfig) -> set[str]:
"""
The file types the barcode scan supports with the current settings
"""
if settings.barcode_enable_tiff_support:
return {"application/pdf", "image/tiff"}
return {"application/pdf"}
def read_barcodes_zxing(image: Image.Image) -> list[tuple[str, str]]:
"""
Returns the text and format (zxing enum name) of each barcode found in
the image
"""
barcodes = []
import zxingcpp
detected_barcodes = zxingcpp.read_barcodes(image)
for barcode in detected_barcodes:
if barcode.text:
barcodes.append((barcode.text, barcode.format.name))
logger.debug(
f"Barcode of type {barcode.format} found: {barcode.text}",
)
return barcodes
def scan_pdf(pdf_path: Path, settings: BarcodeConfig, work_dir: Path) -> list[Barcode]:
"""
Scans the pages of a PDF as images for barcodes. Errors are not caught,
so callers can tell a failed scan from one that found nothing.
"""
barcodes: list[Barcode] = []
with Pdf.open(pdf_path) as pdf:
num_of_pages = len(pdf.pages)
logger.debug(f"PDF has {num_of_pages} pages")
# Get limit from configuration
barcode_max_pages: int = (
num_of_pages if settings.barcode_max_pages == 0 else settings.barcode_max_pages
)
if barcode_max_pages < num_of_pages: # pragma: no cover
logger.debug(
f"Barcodes detection will be limited to the first {barcode_max_pages} pages",
)
for current_page_number in range(min(num_of_pages, barcode_max_pages)):
logger.debug(f"Processing page {current_page_number}")
# Convert page to image
page = convert_from_path(
pdf_path,
dpi=settings.barcode_dpi,
output_folder=work_dir,
first_page=current_page_number + 1,
last_page=current_page_number + 1,
)[0]
# Remember filename, since it is lost by upscaling
page_filepath = Path(page.filename)
logger.debug(f"Image is at {page_filepath}")
# Upscale image if configured
factor = settings.barcode_upscale
if factor > 1.0:
logger.debug(
f"Upscaling image by {factor} for better barcode detection",
)
x, y = page.size
page = page.resize(
(round(x * factor), (round(y * factor))),
)
for barcode_value, barcode_format in read_barcodes_zxing(page):
barcodes.append(
Barcode(current_page_number, barcode_value, settings, barcode_format),
)
# Delete temporary image file
page_filepath.unlink()
return barcodes
def read_barcode_values(
path: Path,
mime_type: str,
settings: BarcodeConfig,
work_dir: Path,
) -> list[StoredBarcode] | None:
"""
Reads the barcodes of a file outside of the consumption plugins: for new
versions, which skip the barcode plugin, and when reprocessing.
Returns None if the file can't be scanned with the current settings.
Errors while scanning are raised.
"""
if mime_type not in scannable_mime_types(settings):
return None
if mime_type == "image/tiff":
path = convert_from_tiff_to_pdf(path, work_dir)
return [x.stored() for x in scan_pdf(path, settings, work_dir)]
+38
View File
@@ -18,6 +18,7 @@ from django.utils import timezone
from filelock import FileLock
from rest_framework.reverse import reverse
from documents.barcodes import read_barcode_values
from documents.classifier import load_classifier
from documents.data_models import ConsumableDocument
from documents.data_models import ConsumeFileSuccessResult
@@ -31,6 +32,7 @@ from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import DocumentBarcode
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
@@ -53,6 +55,7 @@ from documents.utils import compute_checksum
from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess
from paperless.config import BarcodeConfig
from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ArchiveFileGenerationChoices
@@ -504,6 +507,13 @@ class ConsumerPlugin(
f"Parser: {document_parser.name} v{document_parser.version}",
)
# New versions skip the barcode plugin, so read their barcodes here
if (
self.input_doc.root_document_id is not None
and self.metadata.barcodes is None
):
self._read_version_barcodes(mime_type, Path(tmpdir))
# Parse the document. This may take some time.
text = None
@@ -631,6 +641,8 @@ class ConsumerPlugin(
else:
original_document.save()
self._store_barcodes(original_document)
# Adding a version changes the effective document, so update root modified
Document.objects.filter(pk=root_doc.pk).update(
modified=timezone.now(),
@@ -962,6 +974,32 @@ class ConsumerPlugin(
}
CustomFieldInstance.objects.create(**args) # adds to document
self._store_barcodes(document)
def _read_version_barcodes(self, mime_type: str, work_dir: Path) -> None:
barcode_settings = BarcodeConfig()
if not barcode_settings.barcode_store_values:
return
try:
self.metadata.barcodes = (
read_barcode_values(
self.working_copy,
mime_type,
barcode_settings,
work_dir,
)
or None
)
except Exception as e:
self.log.warning(f"Could not read barcodes of {self.filename}: {e}")
def _store_barcodes(self, document: Document) -> None:
if self.metadata.barcodes:
DocumentBarcode.objects.bulk_create(
DocumentBarcode(document=document, **barcode)
for barcode in self.metadata.barcodes
)
def _write(self, source, target) -> None:
with (
Path(source).open("rb") as read_file,
+11
View File
@@ -9,6 +9,16 @@ from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
class StoredBarcode(TypedDict):
"""
A detected barcode as it is stored with a document
"""
page: int # 1-indexed
value: str
format: str # a DocumentBarcode.Format value
@dataclasses.dataclass
class DocumentMetadataOverrides:
"""
@@ -35,6 +45,7 @@ class DocumentMetadataOverrides:
version_label: str | None = None
actor_id: int | None = None
remote_ocr: bool = False
barcodes: list[StoredBarcode] | None = None
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
"""
@@ -45,6 +45,7 @@ from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import DocumentBarcode
from documents.models import DocumentType
from documents.models import Note
from documents.models import SavedView
@@ -353,6 +354,7 @@ class Command(CryptMixin, PaperlessCommand):
"workflows": Workflow.objects.all(),
"custom_fields": CustomField.objects.all(),
"custom_field_instances": CustomFieldInstance.global_objects.all(),
"document_barcodes": DocumentBarcode.objects.all(),
"app_configs": ApplicationConfiguration.objects.all(),
"notes": Note.global_objects.all(),
"documents": Document.global_objects.order_by("id").all(),
@@ -411,6 +413,7 @@ class Command(CryptMixin, PaperlessCommand):
elif self.split_manifest and key in (
"notes",
"custom_field_instances",
"document_barcodes",
):
# Written per-document in _write_split_manifest
pass
@@ -651,6 +654,12 @@ class Command(CryptMixin, PaperlessCommand):
CustomFieldInstance.global_objects.filter(document=document),
),
)
content.extend(
serializers.serialize(
"python",
DocumentBarcode.objects.filter(document=document),
),
)
manifest_name = base_name.with_name(f"{base_name.stem}-manifest.json")
if self.use_folder_prefix:
manifest_name = Path("json") / manifest_name
@@ -77,6 +77,8 @@ class Command(PaperlessCommand):
"notes__user",
"custom_fields__field",
"versions",
"barcodes",
"versions__barcodes",
)
total = documents.count()
rebuild_kwargs = {}
@@ -0,0 +1,102 @@
# Generated by Django 5.2.16 on 2026-09-30 23:01
import django.db.models.deletion
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0026_alter_document_archive_checksum_and_more"),
]
operations = [
migrations.CreateModel(
name="DocumentBarcode",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"page",
models.PositiveIntegerField(
help_text="Page of the original file, starting at 1",
verbose_name="page",
),
),
("value", models.TextField(verbose_name="value")),
(
"format",
models.CharField(
choices=[
("Codabar", "Codabar"),
("Code39", "Code 39"),
("Code39Std", "Code 39 Standard"),
("Code39Ext", "Code 39 Extended"),
("Code32", "Code 32"),
("PZN", "Pharmazentralnummer"),
("Code93", "Code 93"),
("Code128", "Code 128"),
("ITF", "ITF"),
("ITF14", "ITF-14"),
("DataBar", "DataBar"),
("DataBarOmni", "DataBar Omni"),
("DataBarStk", "DataBar Stacked"),
("DataBarStkOmni", "DataBar Stacked Omni"),
("DataBarLtd", "DataBar Limited"),
("DataBarExp", "DataBar Expanded"),
("DataBarExpStk", "DataBar Expanded Stacked"),
("EANUPC", "EAN/UPC"),
("EAN13", "EAN-13"),
("EAN8", "EAN-8"),
("EAN5", "EAN-5"),
("EAN2", "EAN-2"),
("ISBN", "ISBN"),
("UPCA", "UPC-A"),
("UPCE", "UPC-E"),
("Telepen", "Telepen"),
("TelepenAlpha", "Telepen Alpha"),
("TelepenNumeric", "Telepen Numeric"),
("OtherBarcode", "Other barcode"),
("DXFilmEdge", "DX Film Edge"),
("PDF417", "PDF417"),
("CompactPDF417", "Compact PDF417"),
("MicroPDF417", "MicroPDF417"),
("Aztec", "Aztec"),
("AztecCode", "Aztec Code"),
("AztecRune", "Aztec Rune"),
("QRCode", "QR Code"),
("QRCodeModel1", "QR Code Model 1"),
("QRCodeModel2", "QR Code Model 2"),
("MicroQRCode", "Micro QR Code"),
("RMQRCode", "rMQR Code"),
("DataMatrix", "Data Matrix"),
("MaxiCode", "MaxiCode"),
],
max_length=32,
verbose_name="format",
),
),
(
"document",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="barcodes",
to="documents.document",
verbose_name="document",
),
),
],
options={
"verbose_name": "document barcode",
"verbose_name_plural": "document barcodes",
"ordering": ("page", "id"),
},
),
]
+91
View File
@@ -366,6 +366,16 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
res += f" {self.title}"
return res
def get_effective_barcodes(self) -> list["DocumentBarcode"]:
"""
Returns the stored barcodes for the document, like
get_effective_content(): for root documents those of the latest
version when there is one, as that is the file users see.
"""
from documents.versioning import latest_version
return list(latest_version(self).barcodes.all())
def get_effective_content(self) -> str | None:
"""
Returns the effective content for the document.
@@ -969,6 +979,87 @@ class Note(SoftDeleteModel):
return self.note
class DocumentBarcode(models.Model):
"""
A barcode found in a document during consumption, kept so its content
can be shown and copied
"""
document = models.ForeignKey(
Document,
related_name="barcodes",
on_delete=models.CASCADE,
verbose_name=_("document"),
)
page = models.PositiveIntegerField(
_("page"),
help_text=_("Page of the original file, starting at 1"),
)
value = models.TextField(_("value"))
class Format(models.TextChoices):
"""
The concrete barcode formats of zxing-cpp, keyed on the enum name.
The labels are symbology names and aren't translated.
"""
CODABAR = "Codabar", "Codabar"
CODE39 = "Code39", "Code 39"
CODE39_STD = "Code39Std", "Code 39 Standard"
CODE39_EXT = "Code39Ext", "Code 39 Extended"
CODE32 = "Code32", "Code 32"
PZN = "PZN", "Pharmazentralnummer"
CODE93 = "Code93", "Code 93"
CODE128 = "Code128", "Code 128"
ITF = "ITF", "ITF"
ITF14 = "ITF14", "ITF-14"
DATA_BAR = "DataBar", "DataBar"
DATA_BAR_OMNI = "DataBarOmni", "DataBar Omni"
DATA_BAR_STK = "DataBarStk", "DataBar Stacked"
DATA_BAR_STK_OMNI = "DataBarStkOmni", "DataBar Stacked Omni"
DATA_BAR_LTD = "DataBarLtd", "DataBar Limited"
DATA_BAR_EXP = "DataBarExp", "DataBar Expanded"
DATA_BAR_EXP_STK = "DataBarExpStk", "DataBar Expanded Stacked"
EANUPC = "EANUPC", "EAN/UPC"
EAN13 = "EAN13", "EAN-13"
EAN8 = "EAN8", "EAN-8"
EAN5 = "EAN5", "EAN-5"
EAN2 = "EAN2", "EAN-2"
ISBN = "ISBN", "ISBN"
UPCA = "UPCA", "UPC-A"
UPCE = "UPCE", "UPC-E"
TELEPEN = "Telepen", "Telepen"
TELEPEN_ALPHA = "TelepenAlpha", "Telepen Alpha"
TELEPEN_NUMERIC = "TelepenNumeric", "Telepen Numeric"
OTHER_BARCODE = "OtherBarcode", "Other barcode"
DX_FILM_EDGE = "DXFilmEdge", "DX Film Edge"
PDF417 = "PDF417", "PDF417"
COMPACT_PDF417 = "CompactPDF417", "Compact PDF417"
MICRO_PDF417 = "MicroPDF417", "MicroPDF417"
AZTEC = "Aztec", "Aztec"
AZTEC_CODE = "AztecCode", "Aztec Code"
AZTEC_RUNE = "AztecRune", "Aztec Rune"
QR_CODE = "QRCode", "QR Code"
QR_CODE_MODEL1 = "QRCodeModel1", "QR Code Model 1"
QR_CODE_MODEL2 = "QRCodeModel2", "QR Code Model 2"
MICRO_QR_CODE = "MicroQRCode", "Micro QR Code"
RMQR_CODE = "RMQRCode", "rMQR Code"
DATA_MATRIX = "DataMatrix", "Data Matrix"
MAXI_CODE = "MaxiCode", "MaxiCode"
format = models.CharField(_("format"), max_length=32, choices=Format.choices)
class Meta:
ordering = ("page", "id")
verbose_name = _("document barcode")
verbose_name_plural = _("document barcodes")
def __str__(self) -> str: # pragma: no cover
return self.value
class ShareLink(SoftDeleteModel):
class FileVersion(models.TextChoices):
ARCHIVE = ("archive", _("Archive"))
+161 -98
View File
@@ -1,8 +1,8 @@
from __future__ import annotations
import logging
import math
import mimetypes
import os
import shutil
import subprocess
import tempfile
@@ -68,58 +68,6 @@ def get_supported_file_extensions() -> set[str]:
return extensions
def run_convert(
input_file,
output_file,
*,
density=None,
scale=None,
alpha=None,
strip=False,
trim=False,
type=None,
depth=None,
auto_orient=False,
use_cropbox=False,
extra=None,
logging_group=None,
) -> None:
environment = os.environ.copy()
if settings.CONVERT_MEMORY_LIMIT:
# MAGICK_MEMORY_LIMIT sets the maximum amount of RAM the pixel cache can use.
# MAGICK_MAP_LIMIT sets the maximum amount of memory-mapped I/O allowed.
#
# For large-format documents ImageMagick will hit the RAM limit and
# immediately try to "map" the remaining data. If MAGICK_MAP_LIMIT isn't
# also set, the process may trigger an OOM kill because the default
# system/policy map limit is often too restrictive for these massive bitmaps.
environment["MAGICK_MEMORY_LIMIT"] = settings.CONVERT_MEMORY_LIMIT
environment["MAGICK_MAP_LIMIT"] = settings.CONVERT_MEMORY_LIMIT
if settings.CONVERT_TMPDIR:
environment["MAGICK_TMPDIR"] = settings.CONVERT_TMPDIR
args = [settings.CONVERT_BINARY]
args += ["-density", str(density)] if density else []
args += ["-scale", str(scale)] if scale else []
args += ["-alpha", str(alpha)] if alpha else []
args += ["-strip"] if strip else []
args += ["-trim"] if trim else []
args += ["-type", str(type)] if type else []
args += ["-depth", str(depth)] if depth else []
args += ["-auto-orient"] if auto_orient else []
args += ["-define", "pdf:use-cropbox=true"] if use_cropbox else []
args += [str(input_file), str(output_file)]
logger.debug("Execute: " + " ".join(args), extra={"group": logging_group})
try:
run_subprocess(args, environment, logger)
except subprocess.CalledProcessError as e:
raise ParseError(f"Convert failed at {args}") from e
except Exception as e: # pragma: no cover
raise ParseError("Unknown error running convert") from e
def get_default_thumbnail() -> Path:
"""
Returns the path to a generic thumbnail
@@ -127,46 +75,168 @@ def get_default_thumbnail() -> Path:
return (Path(__file__).parent / "resources" / "document.webp").resolve()
def make_thumbnail_from_pdf_gs_fallback(in_path, temp_dir, logging_group=None) -> Path:
out_path: Path = Path(temp_dir) / "convert_gs.webp"
_THUMBNAIL_MAX_WIDTH = 500
_THUMBNAIL_MAX_HEIGHT = 5000
# Used only when the page geometry cannot be read
_THUMBNAIL_FALLBACK_DPI = 150
# Applied before supersampling, so tiny pages are not enlarged
_THUMBNAIL_MAX_DPI = 300
# Rendering at a multiple and downsampling keeps text crisper
_THUMBNAIL_SUPERSAMPLE = 2
# if convert fails, fall back to extracting
# the first PDF page as a PNG using Ghostscript
logger.warning(
"Thumbnail generation with ImageMagick failed, falling back "
"to ghostscript. Check your /etc/ImageMagick-x/policy.xml!",
extra={"group": logging_group},
)
# Ghostscript doesn't handle WebP outputs
gs_out_path: Path = Path(temp_dir) / "gs_out.png"
cmd = [settings.GS_BINARY, "-q", "-sDEVICE=pngalpha", "-o", gs_out_path, in_path]
def rasterize_pdf_page_to_png(
in_path: Path,
out_path: Path,
*,
dpi: int,
logging_group=None,
) -> None:
"""
Rasterizes the first page of a PDF to a PNG with pdftoppm.
"""
# -singlefile drops the page number and -png appends ".png", so pass the
# path without its suffix
args = [
"pdftoppm",
"-f",
"1",
"-l",
"1",
"-r",
str(dpi),
"-png",
"-singlefile",
"-cropbox",
str(in_path),
str(out_path.with_suffix("")),
]
logger.debug("Execute: " + " ".join(args), extra={"group": logging_group})
try:
try:
run_subprocess(cmd, logger=logger)
except subprocess.CalledProcessError as e:
raise ParseError(f"Thumbnail (gs) failed at {cmd}") from e
# then run convert on the output from gs to make WebP
run_convert(
density=300,
scale="500x5000>",
alpha="remove",
strip=True,
trim=False,
auto_orient=True,
input_file=gs_out_path,
output_file=out_path,
logging_group=logging_group,
)
run_subprocess(args, logger=logger)
except subprocess.CalledProcessError as e:
raise ParseError(f"pdftoppm failed at {args}") from e
except Exception as e: # pragma: no cover
raise ParseError("Unknown error running pdftoppm") from e
def encode_thumbnail_webp(
png_path: Path,
out_path: Path,
*,
supersample: int = 1,
) -> None:
"""
Flattens alpha onto white, undoes supersampling, shrinks to fit and saves as WebP.
"""
from PIL import Image
try:
with Image.open(png_path) as im:
if im.mode in ("RGBA", "LA"):
flattened = Image.new("RGB", im.size, (255, 255, 255))
flattened.paste(im, mask=im.split()[-1])
else:
flattened = im.convert("RGB")
if supersample > 1:
flattened = flattened.resize(
(
max(1, round(flattened.width / supersample)),
max(1, round(flattened.height / supersample)),
),
Image.Resampling.LANCZOS,
)
flattened.thumbnail((_THUMBNAIL_MAX_WIDTH, _THUMBNAIL_MAX_HEIGHT))
flattened.save(out_path, format="WEBP")
except (OSError, Image.DecompressionBombError) as e:
raise ParseError(f"Unable to encode thumbnail from {png_path}") from e
def _compute_thumbnail_dpi(in_path: Path, logging_group=None) -> tuple[int, int]:
"""
Returns (dpi, supersample). Unknown geometry is not supersampled, since
the render size cannot be bounded.
"""
from paperless.parsers.utils import get_pdf_first_page_size_points
size = get_pdf_first_page_size_points(in_path)
if size is None:
logger.debug(
"Could not read PDF page size, using fallback DPI",
extra={"group": logging_group},
)
return _THUMBNAIL_FALLBACK_DPI, 1
width_pts, height_pts = size
dpi_for_width = _THUMBNAIL_MAX_WIDTH * 72 / width_pts
dpi_for_height = _THUMBNAIL_MAX_HEIGHT * 72 / height_pts
# Round up so the downsampled render is never a few pixels short of the
# target; the shrink-only clamp in encode_thumbnail_webp trims the excess.
dpi = max(
1,
math.ceil(min(_THUMBNAIL_MAX_DPI, dpi_for_width, dpi_for_height)),
)
# At the 1 DPI floor the page is already oversized, so do not supersample
return dpi, 1 if dpi == 1 else _THUMBNAIL_SUPERSAMPLE
def _render_pdf_thumbnail(
in_path: Path,
png_path: Path,
out_path: Path,
logging_group=None,
) -> None:
dpi, supersample = _compute_thumbnail_dpi(in_path, logging_group=logging_group)
rasterize_pdf_page_to_png(
in_path,
png_path,
dpi=dpi * supersample,
logging_group=logging_group,
)
encode_thumbnail_webp(png_path, out_path, supersample=supersample)
def _repair_pdf_with_qpdf(in_path: Path, out_path: Path) -> None:
# qpdf exits 3 after a repair; --warning-exit-0 keeps that from failing
try:
shutil.copy(in_path, out_path)
run_subprocess(
["qpdf", "--warning-exit-0", "--replace-input", str(out_path)],
logger=logger,
)
except (subprocess.CalledProcessError, OSError) as e:
raise ParseError(f"qpdf repair failed for {in_path}") from e
def make_thumbnail_from_pdf_qpdf_fallback(
in_path: Path,
temp_dir: Path,
logging_group=None,
) -> Path:
png_path = temp_dir / "page1_repaired.png"
out_path = temp_dir / "convert_qpdf.webp"
repaired_path = temp_dir / "repaired.pdf"
logger.warning(
"Thumbnail generation with pdftoppm failed, attempting qpdf repair and retry.",
extra={"group": logging_group},
)
try:
_repair_pdf_with_qpdf(in_path, repaired_path)
_render_pdf_thumbnail(repaired_path, png_path, out_path, logging_group)
return out_path
except ParseError as e:
logger.error(f"Unable to make thumbnail with Ghostscript: {e}")
logger.error(f"Unable to make thumbnail after qpdf repair: {e}")
# The caller might expect a generated thumbnail that can be moved,
# so we need to copy it before it gets moved.
# https://github.com/paperless-ngx/paperless-ngx/issues/3631
default_thumbnail_path: Path = Path(temp_dir) / "document.webp"
default_thumbnail_path = temp_dir / "document.webp"
copy_file_with_basic_stats(get_default_thumbnail(), default_thumbnail_path)
return default_thumbnail_path
@@ -175,25 +245,18 @@ def make_thumbnail_from_pdf(in_path: Path, temp_dir: Path, logging_group=None) -
"""
The thumbnail of a PDF is just a 500px wide image of the first page.
"""
png_path: Path = temp_dir / "page1.png"
out_path: Path = temp_dir / "convert.webp"
# Run convert to get a decent thumbnail
try:
run_convert(
density=300,
scale="500x5000>",
alpha="remove",
strip=True,
trim=False,
auto_orient=True,
use_cropbox=True,
input_file=f"{in_path}[0]",
output_file=str(out_path),
logging_group=logging_group,
)
_render_pdf_thumbnail(in_path, png_path, out_path, logging_group)
except ParseError as e:
logger.error(f"Unable to make thumbnail with convert: {e}")
out_path = make_thumbnail_from_pdf_gs_fallback(in_path, temp_dir, logging_group)
logger.error(f"Unable to make thumbnail with pdftoppm: {e}")
out_path = make_thumbnail_from_pdf_qpdf_fallback(
in_path,
temp_dir,
logging_group,
)
return out_path
+17 -1
View File
@@ -311,7 +311,13 @@ class WriteBatch:
queryset = annotate_effective_content(
Document.objects.filter(pk__in=ids)
.select_related("correspondent", "document_type", "storage_path", "owner")
.prefetch_related("tags", "notes__user", "custom_fields__field"),
.prefetch_related(
"tags",
"notes__user",
"custom_fields__field",
"barcodes",
"versions__barcodes",
),
)
for document, grant in _DocumentViewerStream(queryset, chunk_size=1000):
self.remove(document.pk)
@@ -604,6 +610,16 @@ class TantivyBackend:
},
)
# Barcodes: JSON field like custom_fields, only filled when stored
for barcode in document.get_effective_barcodes():
doc.add_json(
"barcodes",
{
"value": normalize_search_text(barcode.value),
"format": normalize_search_text(barcode.format),
},
)
# Dates
created_date = datetime(
document.created.year,
+5
View File
@@ -39,4 +39,9 @@ PUBLIC_FIELDS: tuple[FieldSpec, ...] = (
FieldKind.JSON,
subpaths={"name": SubpathSpec(), "value": SubpathSpec(default=True)},
),
FieldSpec(
"barcodes",
FieldKind.JSON,
subpaths={"value": SubpathSpec(default=True), "format": SubpathSpec()},
),
)
+2 -1
View File
@@ -25,7 +25,8 @@ logger = logging.getLogger("paperless.search")
# order, and the write-only correspondent/document_type/storage_path/tag id
# columns dropped. tantivy compares schemas by ordered field list, so an
# index built by v1 rejects every write against the v2 schema.
SCHEMA_VERSION: Final[int] = 2
# v3 - barcodes JSON field for stored barcode contents
SCHEMA_VERSION: Final[int] = 3
class FieldDescriptor(NamedTuple):
+7
View File
@@ -60,6 +60,7 @@ from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import DocumentBarcode
from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import Note
@@ -987,6 +988,12 @@ class BasicUserSerializer(serializers.ModelSerializer[User]):
fields = ["id", "username", "first_name", "last_name"]
class DocumentBarcodeSerializer(serializers.ModelSerializer[DocumentBarcode]):
class Meta:
model = DocumentBarcode
fields = ["page", "value", "format"]
class NotesSerializer(serializers.ModelSerializer[Note]):
user = BasicUserSerializer(read_only=True)
+40
View File
@@ -20,6 +20,7 @@ from filelock import FileLock
from documents import sanity_checker
from documents.barcodes import BarcodePlugin
from documents.barcodes import read_barcode_values
from documents.bulk_download import ArchiveOnlyStrategy
from documents.bulk_download import OriginalsOnlyStrategy
from documents.caching import clear_document_caches
@@ -36,6 +37,7 @@ from documents.data_models import ConsumeFileDuplicateResult
from documents.data_models import ConsumeFileStoppedResult
from documents.data_models import ConsumeFileSuccessResult
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import StoredBarcode
from documents.double_sided import CollatePlugin
from documents.file_handling import create_source_path_directory
from documents.file_handling import generate_unique_filename
@@ -43,6 +45,7 @@ from documents.matching import prefilter_documents_by_workflowtrigger
from documents.models import Correspondent
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import DocumentBarcode
from documents.models import DocumentType
from documents.models import PaperlessTask
from documents.models import ShareLink
@@ -67,6 +70,7 @@ from documents.utils import identity
from documents.versioning import annotate_effective_content
from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig
from paperless.config import BarcodeConfig
from paperless.config import RemoteOCRConfig
from paperless.logging import consume_task_id
from paperless.parsers import ParserContext
@@ -341,6 +345,29 @@ def bulk_update_documents(document_ids) -> None:
)
def _read_barcodes_for_reprocess(document: Document) -> list[StoredBarcode] | None:
"""
Reads the barcodes of the original again, e.g. for documents consumed
before storing them was enabled. Returns None if they should be left as
they are: storing is off, the file can't be scanned with the current
settings, or the scan failed.
"""
barcode_settings = BarcodeConfig()
if not barcode_settings.barcode_store_values:
return None
try:
with TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmpdir:
return read_barcode_values(
document.source_path,
document.mime_type,
barcode_settings,
Path(tmpdir),
)
except Exception as e:
logger.warning(f"Could not read barcodes of document {document}: {e}")
return None
@shared_task
def update_document_content_maybe_archive_file(
document_id,
@@ -387,6 +414,8 @@ def update_document_content_maybe_archive_file(
produce_archive=produce_archive,
)
barcodes = _read_barcodes_for_reprocess(document)
thumbnail = parser.get_thumbnail(document.source_path, mime_type)
with transaction.atomic():
@@ -443,6 +472,17 @@ def update_document_content_maybe_archive_file(
action=LogEntry.Action.UPDATE,
)
if barcodes is not None:
document.barcodes.all().delete()
DocumentBarcode.objects.bulk_create(
DocumentBarcode(document=document, **barcode)
for barcode in barcodes
)
# metadata_etag includes modified
Document.objects.filter(pk=document.pk).update(
modified=timezone.now(),
)
with FileLock(settings.MEDIA_LOCK):
if parser.get_archive_path():
create_source_path_directory(document.archive_path)
@@ -0,0 +1,95 @@
"""Stored barcode contents in the search index.
Barcodes are a JSON field like notes and custom fields: barcodes: resolves to
barcodes.value:, and a plain query without the prefix does not look at them.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from documents.models import DocumentBarcode
from paperless_testing.factories import DocumentBarcodeFactory
from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING:
from collections.abc import Callable
from documents.models import Document
from documents.search._backend import TantivyBackend
pytestmark = [pytest.mark.search, pytest.mark.django_db]
class TestBarcodeSearch:
@pytest.fixture
def with_barcodes(self, backend: TantivyBackend) -> Document:
document = DocumentFactory(title="Letter", content="x")
DocumentBarcodeFactory(
document=document,
value="WIFI:T:WPA;S:Guest-WLAN;P:crocodile123;;",
)
DocumentBarcodeFactory(
document=document,
page=2,
value="DE89370400440532013000",
format=DocumentBarcode.Format.CODE128,
)
backend.add_or_update(document)
return document
def test_bare_barcodes_prefix_searches_values(
self,
with_barcodes: Document,
index_document: Callable[..., Document],
matched_ids: Callable[[str], set[int]],
) -> None:
"""
GIVEN:
- A document with stored barcodes, and a decoy document whose
content (not a barcode) contains the same word
WHEN:
- A bare "barcodes:" prefix query is run
THEN:
- Only the document whose barcode matches is returned, also for a
part of a barcode between separators
"""
index_document(title="Decoy", content="crocodile123 in the text")
assert matched_ids("barcodes:crocodile123") == {with_barcodes.pk}
assert matched_ids("barcodes.value:DE89370400440532013000") == {
with_barcodes.pk,
}
def test_barcodes_format_subpath(
self,
with_barcodes: Document,
matched_ids: Callable[[str], set[int]],
) -> None:
"""
GIVEN:
- A document with a QR code and a Code 128 barcode
WHEN:
- The format subpath is queried
THEN:
- The document is found by its barcode formats
"""
assert matched_ids("barcodes.format:qrcode") == {with_barcodes.pk}
assert matched_ids("barcodes.format:aztec") == set()
def test_plain_query_ignores_barcodes(
self,
with_barcodes: Document,
matched_ids: Callable[[str], set[int]],
) -> None:
"""
GIVEN:
- A document with a barcode value not found in its text
WHEN:
- The value is searched without a field prefix
THEN:
- Nothing is found, as with notes and custom fields
"""
assert matched_ids("crocodile123") == set()
@@ -8,7 +8,7 @@ queryable-but-always-empty -- syntactically valid, silently matching
nothing -- with no test failure anywhere.
This indexes one real document carrying values for every JSON field
(a Note, a CustomFieldInstance) and inspects the document's own stored
(a Note, a CustomFieldInstance, a DocumentBarcode) and inspects the document's own stored
JSON payload, rather than running field-specific queries: that way a
future JSON field's subpaths are covered automatically, without a new
per-subpath query having to be added by hand each time.
@@ -27,6 +27,7 @@ from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import Note
from documents.search._fields import PUBLIC_FIELDS
from paperless_testing.factories import DocumentBarcodeFactory
from paperless_testing.factories import UserFactory
if TYPE_CHECKING:
@@ -42,11 +43,12 @@ class TestJsonSubpathsAreWrittenAtIndexTime:
) -> None:
"""
GIVEN:
- A document with a Note and a CustomFieldInstance attached
- A document with a Note, a CustomFieldInstance and a
DocumentBarcode attached
WHEN:
- The document is indexed via TantivyBackend.add_or_update
THEN:
- Every subpath PUBLIC_FIELDS declares for notes/custom_fields
- Every subpath PUBLIC_FIELDS declares for notes/custom_fields/barcodes
is present as a key in the document's stored JSON payload
"""
user = UserFactory(username="completeness-user")
@@ -65,6 +67,7 @@ class TestJsonSubpathsAreWrittenAtIndexTime:
field=field,
value_text="a value",
)
DocumentBarcodeFactory(document=doc, value="a barcode")
backend.add_or_update(doc)
index = backend._index
@@ -175,6 +175,14 @@ PINNED_DESCRIPTORS: tuple[FieldDescriptor, ...] = (
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"barcodes",
"json",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
FieldDescriptor(
"title_sort",
"text",
@@ -74,6 +74,7 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"barcode_enable_tag": None,
"barcode_tag_mapping": None,
"barcode_tag_split": None,
"barcode_store_values": None,
"remote_ocr_engine": None,
"remote_ocr_api_key": None,
"remote_ocr_endpoint": None,
+316 -1
View File
@@ -1,29 +1,46 @@
from __future__ import annotations
import shutil
from collections.abc import Generator
from contextlib import contextmanager
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
import zxingcpp
from django.conf import settings
from django.test import TestCase
from django.test import override_settings
from rest_framework import status
from documents import tasks
from documents.barcodes import BarcodePlugin
from documents.barcodes import read_barcode_values
from documents.consumer import ConsumerError
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource
from documents.models import Document
from documents.models import DocumentBarcode
from documents.models import Tag
from documents.plugins.base import StopConsumeTaskError
from documents.tests.utils import ConsumeTaskMixin
from documents.tests.utils import SampleDirMixin
from paperless.config import BarcodeConfig
from paperless.models import ApplicationConfiguration
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.fakes.progress import FakeProgressManager
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Generator
from pytest_django.fixtures import Settings
from pytest_mock import MockerFixture
from rest_framework.test import APIClient
from paperless_testing.dirs import PaperlessDirs
class GetReaderPluginMixin:
@contextmanager
@@ -1127,3 +1144,301 @@ class TestTagBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes
document_list = reader.separate_pages(separator_pages)
self.assertEqual(len(document_list), 3)
SAMPLE_VALUES = [
{"page": 1, "value": "javascript:alert(1)", "format": "QRCode"},
{"page": 2, "value": "https://example.com/invoice/4711", "format": "QRCode"},
]
@pytest.fixture
def samples_dir() -> Path:
return Path(__file__).parent / "samples"
@pytest.fixture
def barcode_samples_dir(samples_dir: Path) -> Path:
return samples_dir / "barcodes"
@pytest.fixture
def store_barcodes(settings: Settings) -> None:
settings.CONSUMER_STORE_BARCODE_VALUES = True
@pytest.fixture
def barcode_reader(
paperless_dirs: PaperlessDirs,
) -> Generator[Callable[[Path], BarcodePlugin], None, None]:
readers: list[BarcodePlugin] = []
def make(path: Path) -> BarcodePlugin:
reader = BarcodePlugin(
ConsumableDocument(DocumentSource.ConsumeFolder, original_file=path),
DocumentMetadataOverrides(),
FakeProgressManager(path.name, None),
paperless_dirs.scratch_dir,
"task-id",
)
reader.setup()
readers.append(reader)
return reader
yield make
for reader in readers:
reader.cleanup()
@pytest.fixture
def consume_sample(
paperless_dirs: PaperlessDirs,
barcode_samples_dir: Path,
fake_progress_manager: type[FakeProgressManager],
settings: Settings,
) -> Callable[..., Document]:
settings.CELERY_TASK_ALWAYS_EAGER = True
settings.OCR_MODE = "auto"
def consume(name: str, *, root_document_id: int | None = None) -> Document:
dst = paperless_dirs.scratch_dir / name
shutil.copy(barcode_samples_dir / name, dst)
tasks.consume_file(
ConsumableDocument(
source=DocumentSource.ApiUpload
if root_document_id
else DocumentSource.ConsumeFolder,
original_file=dst,
root_document_id=root_document_id,
),
None,
)
return Document.objects.latest("id")
return consume
def _stored(document: Document) -> list[dict]:
return list(document.barcodes.values("page", "value", "format"))
def test_formats_cover_zxing() -> None:
"""
DocumentBarcode.Format matches the concrete formats of zxing-cpp, so a
zxing-cpp update that adds or removes one fails here
"""
members = zxingcpp.BarcodeFormat.__members__
# skip NONE and the groups like AllLinear, including their aliases
seen = {int(m) for n, m in members.items() if n == "NONE" or n.startswith("All")}
concrete = set()
for name, member in members.items():
# aliases such as DataBarExpanded come after the name zxing reports
if int(member) not in seen:
seen.add(int(member))
concrete.add(name)
assert set(DocumentBarcode.Format.values) == concrete
@pytest.mark.django_db
class TestBarcodeValues:
@pytest.mark.parametrize(
("filename", "mime_type", "tiff_support", "expected"),
[
pytest.param("simple.jpg", "image/jpeg", True, None, id="jpeg"),
pytest.param("simple.tiff", "image/tiff", False, None, id="tiff-off"),
pytest.param("simple.tiff", "image/tiff", True, [], id="tiff-on"),
],
)
def test_read_values_file_types(
self,
paperless_dirs: PaperlessDirs,
samples_dir: Path,
settings: Settings,
filename: str,
mime_type: str,
tiff_support: bool, # noqa: FBT001
expected: list | None,
) -> None:
"""
Files the scan doesn't support return None, so stored barcodes are
kept instead of being replaced with an empty list
"""
settings.CONSUMER_BARCODE_TIFF_SUPPORT = tiff_support
values = read_barcode_values(
samples_dir / filename,
mime_type,
BarcodeConfig(),
paperless_dirs.scratch_dir,
)
assert values == expected
def test_values_detected(
self,
barcode_reader: Callable[[Path], BarcodePlugin],
barcode_samples_dir: Path,
store_barcodes: None,
) -> None:
reader = barcode_reader(barcode_samples_dir / "barcode-qr-url.pdf")
assert reader.able_to_run
reader.run()
assert reader.metadata.barcodes == SAMPLE_VALUES
def test_values_disabled(
self,
barcode_reader: Callable[[Path], BarcodePlugin],
barcode_samples_dir: Path,
settings: Settings,
) -> None:
settings.CONSUMER_ENABLE_ASN_BARCODE = True
reader = barcode_reader(barcode_samples_dir / "barcode-qr-url.pdf")
reader.run()
assert reader.metadata.barcodes is None
def test_consume_and_reprocess(
self,
consume_sample: Callable[..., Document],
admin_client: APIClient,
store_barcodes: None,
) -> None:
"""
GIVEN:
- PDF with a QR code on each of its two pages
WHEN:
- File is consumed, the values are lost, and the document is reprocessed
THEN:
- The barcodes are stored, shown in the API and searchable
- Reprocessing reads them again
"""
document = consume_sample("barcode-qr-url.pdf")
assert _stored(document) == SAMPLE_VALUES
response = admin_client.get(f"/api/documents/{document.pk}/metadata/")
assert response.status_code == status.HTTP_200_OK
assert response.data["barcodes"] == SAMPLE_VALUES
response = admin_client.get(f"/api/documents/{document.pk}/")
assert "barcodes" not in response.data
response = admin_client.get("/api/documents/?query=barcodes:invoice")
assert [x["id"] for x in response.data["results"]] == [document.pk]
response = admin_client.get("/api/documents/?query=invoice")
assert response.data["results"] == []
document.barcodes.all().delete()
modified = Document.objects.get(pk=document.pk).modified
tasks.update_document_content_maybe_archive_file(document.pk)
assert _stored(document) == SAMPLE_VALUES
assert Document.objects.get(pk=document.pk).modified > modified
@pytest.mark.parametrize(
"read_values",
[
pytest.param({"side_effect": RuntimeError("broken")}, id="scan-fails"),
pytest.param({"return_value": None}, id="not-scannable"),
],
)
def test_reprocess_keeps_values(
self,
consume_sample: Callable[..., Document],
mocker: MockerFixture,
store_barcodes: None,
read_values: dict,
) -> None:
"""
GIVEN:
- A document with stored barcodes
WHEN:
- It is reprocessed, but the barcodes can't be read
THEN:
- The stored barcodes are kept
"""
document = consume_sample("barcode-qr-url.pdf")
mocker.patch("documents.tasks.read_barcode_values", **read_values)
tasks.update_document_content_maybe_archive_file(document.pk)
assert _stored(document) == SAMPLE_VALUES
def test_reprocess_tiff_support_off_keeps_values(
self,
consume_sample: Callable[..., Document],
settings: Settings,
store_barcodes: None,
) -> None:
"""
GIVEN:
- A TIFF document with barcodes stored while TIFF support was on
WHEN:
- TIFF support is turned off and the document is reprocessed
THEN:
- The stored barcodes are kept
"""
settings.CONSUMER_BARCODE_TIFF_SUPPORT = True
document = consume_sample("patch-code-t-middle.tiff")
stored = _stored(document)
assert stored
settings.CONSUMER_BARCODE_TIFF_SUPPORT = False
tasks.update_document_content_maybe_archive_file(document.pk)
assert _stored(document) == stored
def test_reprocess_values_disabled(
self,
consume_sample: Callable[..., Document],
settings: Settings,
store_barcodes: None,
) -> None:
document = consume_sample("barcode-qr-url.pdf")
settings.CONSUMER_STORE_BARCODE_VALUES = False
assert tasks._read_barcodes_for_reprocess(document) is None
def test_consume_version_stores_own_values(
self,
consume_sample: Callable[..., Document],
admin_client: APIClient,
store_barcodes: None,
) -> None:
"""
GIVEN:
- A document with stored barcodes
WHEN:
- A new version with a different barcode is consumed, like after
rotating or removing pages
THEN:
- The version keeps its own barcodes, the original ones are kept
- The metadata and the search use those of the newest version
"""
root = consume_sample("barcode-qr-url.pdf")
version = consume_sample("barcode-128-custom.pdf", root_document_id=root.pk)
assert version.root_document == root
assert _stored(version) == [
{"page": 1, "value": "CUSTOM BARCODE", "format": "Code128"},
]
assert root.barcodes.count() == 2
assert [x.value for x in root.get_effective_barcodes()] == ["CUSTOM BARCODE"]
response = admin_client.get(f"/api/documents/{root.pk}/metadata/")
assert [x["value"] for x in response.data["barcodes"]] == ["CUSTOM BARCODE"]
response = admin_client.get('/api/documents/?query=barcodes:"custom barcode"')
assert [x["id"] for x in response.data["results"]] == [root.pk]
response = admin_client.get("/api/documents/?query=barcodes:invoice")
assert response.data["results"] == []
def test_consume_version_scan_fails(
self,
consume_sample: Callable[..., Document],
mocker: MockerFixture,
store_barcodes: None,
) -> None:
"""
A failed scan of a new version is logged and doesn't stop consumption
"""
root = consume_sample("barcode-qr-url.pdf")
mocker.patch(
"documents.consumer.read_barcode_values",
side_effect=RuntimeError("broken"),
)
version = consume_sample("barcode-128-custom.pdf", root_document_id=root.pk)
assert version.root_document == root
assert not version.barcodes.exists()
@@ -32,6 +32,7 @@ from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import DocumentBarcode
from documents.models import DocumentType
from documents.models import Note
from documents.models import ShareLink
@@ -50,6 +51,7 @@ from paperless_mail.models import MailAccount
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.dirs import paperless_environment
from paperless_testing.factories import DocumentBarcodeFactory
from paperless_testing.permissions import grant_object
@@ -856,6 +858,37 @@ class TestExportImport(
self.assertEqual(Document.objects.count(), 4)
self.assertEqual(CustomFieldInstance.objects.count(), 1)
def _export_import_barcodes(self, *, split_manifest: bool) -> None:
shutil.rmtree(Path(self.dirs.media_dir) / "documents")
shutil.copytree(
Path(__file__).parent / "samples" / "documents",
Path(self.dirs.media_dir) / "documents",
)
DocumentBarcodeFactory(document=self.d1, value="https://example.com")
DocumentBarcodeFactory(document=self.d2, page=2, value="DE8937")
self._do_export(split_manifest=split_manifest)
with paperless_environment():
Document.objects.all().delete()
self.assertEqual(DocumentBarcode.objects.count(), 0)
call_command(
"document_importer",
"--no-progress-bar",
self.target,
skip_checks=True,
)
self.assertEqual(
set(DocumentBarcode.objects.values_list("document", "page", "value")),
{(self.d1.pk, 1, "https://example.com"), (self.d2.pk, 2, "DE8937")},
)
def test_export_import_barcodes(self) -> None:
self._export_import_barcodes(split_manifest=False)
def test_export_import_barcodes_split_manifest(self) -> None:
self._export_import_barcodes(split_manifest=True)
def test_folder_prefix(self) -> None:
"""
GIVEN:
Loaded 100 of 163 files, more files were not shown because too many files have changed in this diff. Show more