* Mark the thumbnail constants Final
* Comment each pdftoppm option
* Feature/Chore: Replaces all usages of ImageMagik's convert binary with a simple Pillow based helper
This removes ImageMagik as a dependency entirly and all documentation is cleaned up from it.
We no longer need to install it, have a policy for it to allow PDFs and no longer shell
out to it either.
* Fix: Save WebP images losslessly when removing the alpha channel
Pillow's default WebP save is lossy, so a WebP with an alpha channel came
back from remove_alpha_channel with slightly shifted pixel values before
it reached OCR. Save WebP lossless; other formats are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* Chore: Generate PDF thumbnails with pdftoppm and Pillow instead of ImageMagick
Replace the ImageMagick convert based PDF thumbnail path with pdftoppm and
Pillow. Keeps thumbnails at 500px wide for small and odd sized pages, covers
the qpdf repair and double failure paths, removes the unused run_convert
helper, hardens thumbnail encoding, deprecates PAPERLESS_CONVERT_MEMORY_LIMIT
and PAPERLESS_CONVERT_TMPDIR, and corrects the related docs and CI.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Chore: Supersample the PDF thumbnail render 2x
Render the PDF page at twice the target size and downsample, with edge case
handling for small and odd sized pages, simplified helpers and tests, and
trimmed comments.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Fix: use a fractional thumbnail DPI so oversized pages render a bounded raster
* Deprecate the unused PAPERLESS_GS_BINARY setting too
* Warn at startup when pdftoppm is missing
* Resample PDF thumbnails once instead of twice with LANCZOS, still never enlarging
* Read PDF page size from pdfinfo so it matches pdftoppm
* Mark the thumbnail constants Final
* Comment each pdftoppm option
scikit-learn treats a single-class fit as binary, so predict_proba returns
[1 - p, p] for one class and argmax could index past classes_. Use only
column 0, the real class, so the confidence threshold still applies.
get_public_filename relied on sanitize_filename to truncate, which cuts
from the end and so dropped the counter and extension of long titles.
Multi-byte titles could also exceed the 255 byte filesystem limit.
Shorten the title by UTF-8 bytes before appending the counter and
extension. The exporter reserves room for its longer derived thumbnail
and archive suffixes.
get_users_with_perms includes group members by default, so PDF actions such
as merge and split gave them direct grants on the new documents, which
outlived their group membership.
The trash list needs to show a version to whoever owns its root document, but
it deliberately shows only owned and unowned documents and ignores explicit
shares, so it cannot use permitted_object_ids. #14384 gave it its own
filter_queryset with a hand-built condition through the root, which describes
the same rule as the annotations in permitted_object_ids a second time.
Move those annotations into annotate_authorizing_fields. PermittedObjectsFilter
gains a parent_field attribute that its owner-only and granted paths both use,
and the trash filter shrinks to setting include_granted and parent_field.
* Fix: index document versions as their root document
The search index and the LLM index hold root documents only, a root being
indexed with its newest version's content. Every write path therefore had to
remember to hand them the root. Several did not.
WriteBatch.add_or_update, WriteBatch.add_or_update_ids and
llm_index_add_or_update_document now resolve a version to its root themselves.
The reindex command and update_llm_index only walk root documents, and an
incremental LLM update is scoped to the roots of the given ids through
versioning.root_document_ids, which add_or_update_ids shares. more_like_id
returns the root's id for a version. The reprocess task no longer picks the root
for the indexes and only still clears the caches of both documents.
* Fix: rebuild old search indexes and look versions up as their root for similar documents
An index built before versions were indexed as their root can still hold
version entries under their own id and metadata, and only a rebuild clears
them. Bump the schema version so the index is rebuilt on upgrade.
more_like_this_ids looked the given id up in the index as is, so asking for
documents like a version, as the AI suggestions endpoint can, silently found
nothing. It now looks the root up instead.
* Fix: authorize document versions by their root in the single-object permission check
has_perms_owner_aware judged a document by its own owner and grants, so each
endpoint that fetches a document itself had to remember to map a version to
its root document first, and one that forgot, like the more-like-this search
filter, authorized by a stale version owner.
The check now maps a Document to its root before looking at the owner and the
guardian grants, matching what permitted_document_ids does for id sets. The
eight call sites that mapped the document themselves pass it straight through.
The DRF object permission class needs no change because the document viewset
only ever serves root documents.
* Fix: authorize document versions by their root and speed up permission id sets
permitted_document_ids judged a version by its own owner and grants, so a
version whose owner had drifted from its root's was visible to the wrong
people and hidden from the right ones. Callers patched this individually by
mapping each document to its root first. The query itself was also slow on
MariaDB: the guardian grants were a UNION cast to integers and tested with
IN inside an OR with the owner checks, which MariaDB cannot materialize, so it
re-scans the user's grants for every document. At 20k documents that took
seconds for a user with a couple of hundred grants.
permitted_object_ids now casts the row key to a string, as guardian stores
object_pk, and tests it against a single uncorrelated UNION ALL of the user's
and groups' grants. No integer cast is needed and the user's groups are
matched with an IN subquery rather than a join through the membership table.
Postgres and SQLite build the grant set once, and MariaDB probes guardian's
unique indexes per row, which is cheap. A correlated EXISTS per grant also
fixed MariaDB but was up to 3x slower than before on Postgres and SQLite.
permitted_object_ids also takes an optional parent_field naming a
self-referencing foreign key whose target authorizes the row, and
permitted_document_ids passes root_document, so a version is visible exactly
when its root is. The helper that mapped documents to their roots at the call
sites is no longer needed, so the email, selection data, share link bundle,
trash, bulk download and bulk edit checks use the id set directly.
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
An interrupted rebuild left an empty index stamped as current, so the next
start reported it as up to date. Mark the rebuild as in progress and only
clear the marker once it completes.
* Feature: store barcode contents, list and search them
New setting PAPERLESS_CONSUMER_STORE_BARCODE_VALUES (off by default)
stores all barcodes found during consumption with the document: page,
type and content. They are listed on the metadata tab with a copy
button, returned by the documents API and searchable with barcodes:
in the advanced search. Versions keep their own barcodes, reprocessing
reads them again. Refs #9898
* Tests: cover the remaining barcode branches
Covers unchanged and failing barcode reads on reprocessing, unsupported files and DocumentBarcode.__str__, and uses toHaveLength in the barcode list spec as suggested by SonarCloud.
* Address review: keep barcodes when they can't be read, format choices
- Reprocessing and new versions keep the stored barcodes when the file
can't be scanned or the scan fails, and replace them atomically.
- Format is a TextChoices of the zxing-cpp formats, with a test.
- Shared scan code, latest_version helper, TypedDict, serializer reuse.
- Barcodes in the split manifest, export/import tests, pytest-style tests.
* Barcode tests: TIFF reprocess, format check both ways, fixtures
- Reprocessing a TIFF with TIFF support off keeps the stored barcodes.
- The format test also fails when zxing-cpp drops a format.
- Fixtures in place of the sample dir mixin, plugin-level disabled test.
- Format labels aren't translated, OpenAPI enum named BarcodeFormatEnum.
* Review: module-level zxing reader, shorter barcode docs
- read_barcodes_zxing is a module-level function used by scan_pdf.
- Drop the trivial __str__ test, mark it no cover.
- Shorten the barcode docs and remove the duplicate in configuration.md.
* Fix header
* Use utility class
* Return barcodes from the metadata endpoint only
Drop the barcodes field and its prefetches from the document serializer,
as agreed in the review. Also remove the now empty component stylesheet.
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
The `created` date fallback derived a naive datetime from a file's
mtime using the OS-local zone, then labeled it as the configured
TIME_ZONE without converting. When the OS-local zone and TIME_ZONE
disagree, or when the C library can't resolve zoneinfo at all (as in
some sandboxed environments, where it silently falls back to UTC),
the resulting date can land on the wrong calendar day.
Convert the timestamp directly into the target zone with `tz=` on
fromtimestamp() instead of a naive conversion plus make_aware().
* Fix: redirect SHARE_LINK_BUNDLE_DIR to the test temp layout instead of the real media directory
* Fix: include f_to in test_filters subTest labels so each of the 8 cases reports distinctly
* Fix: run the post_consume error-log assertion after the raising call and match the actual paperless_mail logger name
* Fix: rename the blank-password workflow test to match its behavior and add a real wrong-password-fails test
* Fix: use the created social account's actual pk and remove an accidental tuple wrapping the mock provider
* Fix: assert against the created documents' actual pks instead of hardcoded 1 and 2
* Fix: assert test_compression actually produces a valid LZMA-compressed zip
* Fix: clear os.environ when patching PAPERLESS_ADMIN_* vars so a host-set value can't leak into the no-user test
* Fix: restore MIDDLEWARE, AUTHENTICATION_BACKENDS and REST_FRAMEWORK auth classes after each remote-user settings test instead of leaking the mutation into later tests
* Fix: use a guaranteed-nonexistent temp path instead of hardcoded /tmp/foo/bar in test_export_target_not_exists
Snapshot only the edited field, before and after the operation, gathering tags
and custom field instances into sorted id lists per document (empty when there
are none).
shamoon
Trenton HandClaude Sonnet 5.5
jurassicparkicecreamandshamoon
