* Fix: match Apple Mail tag colors case-insensitively
* Fix: reject OAuth callbacks with an unrecognized scope
* Fix: refuse to refresh OAuth tokens for non-OAuth accounts or missing refresh tokens
* Fix: parse datetime strings in the format_datetime filter and reject invalid ones
* Fix: reject a PAPERLESS_URL without a hostname
* Fix: report a database error in the system status instead of failing with a 500
* Fix: surface the database error when refreshing a document before moving its files
* Fix: report no unapplied migrations when the applied migrations cannot be read
* Fix: return the parsed value directly in format_datetime for typing
* Chore: annotate the paperless_dirs fixture in the refresh failure test
* Allow null for the latest_migration system status type
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Chore: Import dateparser lazily so processes that never parse dates skip it
* Chore: Import turbohtml lazily in the mail parser
* Chore: Import lxml lazily in the SVG validator
* Chore: Import paperless_ai.indexing lazily in document tasks
* Chore: Import paperless_ai helpers lazily in views
* Chore: Read the OCRmyPDF sRGB profile on first use instead of at import
* Chore: Import pikepdf and img2pdf lazily so startup does not load them
* Chore: Reuse the initialized Django ASGI application for HTTP routing
* Chore: drop get_groups_with_only_permission
It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.
* Chore: consolidate direct grant lookups into get_permission_grantees
Six places listed who holds a permission on an object, each with its own
query and result shape, and one had already drifted into expanding group
members.
* Chore: drop the unreachable empty viewer grant default
get_permission_grantees returns an entry for every requested document.
Drops get_objects_for_user_owner_aware. Callers now use restrict_queryset_to_visible.
The same question was answered two ways, and the guardian path did not deny
deactivated users the way permitted_object_ids does.
The trash list needs to show a version to whoever owns its root document, but
it deliberately shows only owned and unowned documents and ignores explicit
shares, so it cannot use permitted_object_ids. #14384 gave it its own
filter_queryset with a hand-built condition through the root, which describes
the same rule as the annotations in permitted_object_ids a second time.
Move those annotations into annotate_authorizing_fields. PermittedObjectsFilter
gains a parent_field attribute that its owner-only and granted paths both use,
and the trash filter shrinks to setting include_granted and parent_field.
* Fix: index document versions as their root document
The search index and the LLM index hold root documents only, a root being
indexed with its newest version's content. Every write path therefore had to
remember to hand them the root. Several did not.
WriteBatch.add_or_update, WriteBatch.add_or_update_ids and
llm_index_add_or_update_document now resolve a version to its root themselves.
The reindex command and update_llm_index only walk root documents, and an
incremental LLM update is scoped to the roots of the given ids through
versioning.root_document_ids, which add_or_update_ids shares. more_like_id
returns the root's id for a version. The reprocess task no longer picks the root
for the indexes and only still clears the caches of both documents.
* Fix: rebuild old search indexes and look versions up as their root for similar documents
An index built before versions were indexed as their root can still hold
version entries under their own id and metadata, and only a rebuild clears
them. Bump the schema version so the index is rebuilt on upgrade.
more_like_this_ids looked the given id up in the index as is, so asking for
documents like a version, as the AI suggestions endpoint can, silently found
nothing. It now looks the root up instead.
* Fix: authorize document versions by their root in the single-object permission check
has_perms_owner_aware judged a document by its own owner and grants, so each
endpoint that fetches a document itself had to remember to map a version to
its root document first, and one that forgot, like the more-like-this search
filter, authorized by a stale version owner.
The check now maps a Document to its root before looking at the owner and the
guardian grants, matching what permitted_document_ids does for id sets. The
eight call sites that mapped the document themselves pass it straight through.
The DRF object permission class needs no change because the document viewset
only ever serves root documents.
* Fix: authorize document versions by their root and speed up permission id sets
permitted_document_ids judged a version by its own owner and grants, so a
version whose owner had drifted from its root's was visible to the wrong
people and hidden from the right ones. Callers patched this individually by
mapping each document to its root first. The query itself was also slow on
MariaDB: the guardian grants were a UNION cast to integers and tested with
IN inside an OR with the owner checks, which MariaDB cannot materialize, so it
re-scans the user's grants for every document. At 20k documents that took
seconds for a user with a couple of hundred grants.
permitted_object_ids now casts the row key to a string, as guardian stores
object_pk, and tests it against a single uncorrelated UNION ALL of the user's
and groups' grants. No integer cast is needed and the user's groups are
matched with an IN subquery rather than a join through the membership table.
Postgres and SQLite build the grant set once, and MariaDB probes guardian's
unique indexes per row, which is cheap. A correlated EXISTS per grant also
fixed MariaDB but was up to 3x slower than before on Postgres and SQLite.
permitted_object_ids also takes an optional parent_field naming a
self-referencing foreign key whose target authorizes the row, and
permitted_document_ids passes root_document, so a version is visible exactly
when its root is. The helper that mapped documents to their roots at the call
sites is no longer needed, so the email, selection data, share link bundle,
trash, bulk download and bulk edit checks use the id set directly.
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* Feature: store barcode contents, list and search them
New setting PAPERLESS_CONSUMER_STORE_BARCODE_VALUES (off by default)
stores all barcodes found during consumption with the document: page,
type and content. They are listed on the metadata tab with a copy
button, returned by the documents API and searchable with barcodes:
in the advanced search. Versions keep their own barcodes, reprocessing
reads them again. Refs #9898
* Tests: cover the remaining barcode branches
Covers unchanged and failing barcode reads on reprocessing, unsupported files and DocumentBarcode.__str__, and uses toHaveLength in the barcode list spec as suggested by SonarCloud.
* Address review: keep barcodes when they can't be read, format choices
- Reprocessing and new versions keep the stored barcodes when the file
can't be scanned or the scan fails, and replace them atomically.
- Format is a TextChoices of the zxing-cpp formats, with a test.
- Shared scan code, latest_version helper, TypedDict, serializer reuse.
- Barcodes in the split manifest, export/import tests, pytest-style tests.
* Barcode tests: TIFF reprocess, format check both ways, fixtures
- Reprocessing a TIFF with TIFF support off keeps the stored barcodes.
- The format test also fails when zxing-cpp drops a format.
- Fixtures in place of the sample dir mixin, plugin-level disabled test.
- Format labels aren't translated, OpenAPI enum named BarcodeFormatEnum.
* Review: module-level zxing reader, shorter barcode docs
- read_barcodes_zxing is a module-level function used by scan_pdf.
- Drop the trivial __str__ test, mark it no cover.
- Shorten the barcode docs and remove the duplicate in configuration.md.
* Fix header
* Use utility class
* Return barcodes from the metadata endpoint only
Drop the barcodes field and its prefetches from the document serializer,
as agreed in the review. Also remove the now empty component stylesheet.
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Snapshot only the edited field, before and after the operation, gathering tags
and custom field instances into sorted id lists per document (empty when there
are none).
* Feature: parse advanced search with whoosh-compat and delete the hand-written translator
* Don't cover these, they exist for defensive, but there's no other current diagnostic
* A mix of more no cover and tests
* Route SCHEMA_FIELD_MISSING into an error, not an HTTP 400