Template and document references gave 400 for a missing id but 403 for a
hidden one, and a template could point at a hidden sample document. Narrow
the querysets, keeping a template's stored sample document, and drop the
view checks they replace.
OCR dropped currency codes and stored amounts the API would reject. Keep an
ISO code or locale currency symbol, read parentheses as negative, and validate
with the same rule as the API.
* Chore: drop get_groups_with_only_permission
It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.
* Chore: consolidate direct grant lookups into get_permission_grantees
Six places listed who holds a permission on an object, each with its own
query and result shape, and one had already drifted into expanding group
members.
* Chore: drop the unreachable empty viewer grant default
get_permission_grantees returns an entry for every requested document.
* Chore: drop set_permissions_for_object in favor of the bulk helper
The same grant rules were implemented twice, and only the bulk version
validated every action before writing, so the single-object one could leave
a partial change behind.
* Chore: type the set-permissions dict
Callers passed whatever they had, including querysets of User and Group
objects that only worked because id__in accepts a subquery.
* Refactor: add pure pdf_ops module with real-PDF tests
* Drops tests we don't relly need + adds the typing annotation
* Now that I'm sure it works, re-work these tests to mock pdf_ops, rather than pikepdf
* Validates pages ahead of time in build_pdfs rather than during the building
* Adds the given/when/then and some additional cleanups of pytest style
* This is redundant now
* Feature: store barcode contents, list and search them
New setting PAPERLESS_CONSUMER_STORE_BARCODE_VALUES (off by default)
stores all barcodes found during consumption with the document: page,
type and content. They are listed on the metadata tab with a copy
button, returned by the documents API and searchable with barcodes:
in the advanced search. Versions keep their own barcodes, reprocessing
reads them again. Refs #9898
* Tests: cover the remaining barcode branches
Covers unchanged and failing barcode reads on reprocessing, unsupported files and DocumentBarcode.__str__, and uses toHaveLength in the barcode list spec as suggested by SonarCloud.
* Address review: keep barcodes when they can't be read, format choices
- Reprocessing and new versions keep the stored barcodes when the file
can't be scanned or the scan fails, and replace them atomically.
- Format is a TextChoices of the zxing-cpp formats, with a test.
- Shared scan code, latest_version helper, TypedDict, serializer reuse.
- Barcodes in the split manifest, export/import tests, pytest-style tests.
* Barcode tests: TIFF reprocess, format check both ways, fixtures
- Reprocessing a TIFF with TIFF support off keeps the stored barcodes.
- The format test also fails when zxing-cpp drops a format.
- Fixtures in place of the sample dir mixin, plugin-level disabled test.
- Format labels aren't translated, OpenAPI enum named BarcodeFormatEnum.
* Review: module-level zxing reader, shorter barcode docs
- read_barcodes_zxing is a module-level function used by scan_pdf.
- Drop the trivial __str__ test, mark it no cover.
- Shorten the barcode docs and remove the duplicate in configuration.md.
* Fix header
* Use utility class
* Return barcodes from the metadata endpoint only
Drop the barcodes field and its prefetches from the document serializer,
as agreed in the review. Also remove the now empty component stylesheet.
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
BulkEditSerializer's hand-parsed validators only caught the exceptions
their happy paths raised, so wrong-typed input produced a 500 or was
passed through to the task:
- owner: a nonexistent or wrong-typed id raised an uncaught error, and a
boolean was accepted (Django coerces True to pk 1). It is now validated
with PrimaryKeyRelatedField and the validated pk is passed on.
- rotate: null raised TypeError, while true, "90" and 45 were accepted and
failed later in QPDF. Degrees are now an IntegerField plus a
multiple-of-90 check, passing an int on. The dedicated rotate endpoint
and edit_pdf operations share the same multiple-of-90 check.
- split: null raised AttributeError, "0" silently became the last page,
"3-1" gave an empty group that crashed the task, and a range like
"1-5000000" was expanded into a list during the request with no upper
bound. Each range is now checked against 1 <= start <= end <= page_count
before it is built.
SetPermissionsSerializer was a bare DictField, and the legacy bulk_edit
set_permissions parameter bypassed even that by hand-calling
validate_set_permissions() on an unchecked dict. A bool or a non-list in
place of users/groups crashed with a raw TypeError instead of a 400.
It is now a nested serializer (view/change, each with users/groups as
lists of integers), used for owned-object create/update, the legacy
bulk_edit set_permissions parameter, and bulk_edit_objects permissions.
Unknown action keys are rejected: previously a typo like "veiw" was
silently dropped, leaving an empty permission set that could clear
existing grants. An explicit set_permissions null (an owner-only change)
remains a no-op, and an empty bulk_edit_objects permissions dict is still
rejected.
operations was a ListField with no child, so both EditPdfDocumentsSerializer
and BulkEditSerializer._validate_parameters_edit_pdf hand-checked each entry
with isinstance(). That accepted booleans (isinstance(True, int) is true) and
passed through any extra keys.
Adds PdfEditOperationSerializer (page >= 1, doc >= 0, rotate a multiple of
90) and uses it as the ListField child on both paths, dropping the manual
type checks and the now-redundant negative/page < 1 bounds checks. Rotations
that are not a multiple of 90 previously passed validation and then failed
inside the task when QPDF refused them.
* perf: skip effective_content annotation on document list unless filtered on
DocumentViewSet.get_queryset() always attached a correlated subquery
resolving each document's latest version content, even though it's only
needed for the deprecated search/title_content/content__* filter params.
Evaluated for every candidate row before pagination's LIMIT, this is
pathological on MariaDB: its default cardinality estimate for the mostly-
NULL root_document_id self-join drives it to a near-full-table scan per
row instead of using the FK index, turning a normal filtered list request
into a multi-second query (root cause of paperless-ngx#13778's report).
Only attach the annotation when a request actually filters on it. The
common case now relies on Document.get_effective_content()'s existing
prefetch-based fallback instead (extended the "versions" prefetch to
include content), which DocumentSerializer.to_representation() now calls
directly instead of checking for the annotation via hasattr().
* fix: address review feedback on effective_content annotation skip
- _needs_effective_content_annotation() now checks for a non-blank,
stripped param value rather than mere key presence, matching how
SearchFilter/TitleContentFilter/EffectiveContentFilter themselves
no-op on a blank value. An empty ?search= or a saved view with a
cleared text filter no longer re-triggers the annotation.
- The "versions" prefetch on DocumentViewSet no longer carries content
for every historical version of every document -- that's unused
bloat for version-heavy documents. Added
latest_version_content_prefetch() (versioning.py), a separate,
windowed prefetch scoped to just the newest version's content per
root, and taught Document.get_effective_content() to check it first.
- DocumentSerializer.to_representation() no longer unconditionally
calls get_effective_content(). Added has_prefetched_effective_content()
(versioning.py) as a cheap upfront check: only resolve version-aware
content when an SQL annotation or a versions prefetch is already on
the instance. TrashView and GlobalSearchView build their own
querysets independently of DocumentViewSet and never display
document content at all (checked both frontend components), so they
now keep showing the document's own, unresolved content with zero
extra queries -- the same behavior as before effective_content
resolution existed, just generalized past the narrow hasattr() check
it replaced.
* Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it
* Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even
* CLean up the new test with the docstrings, handle the fields in one place
* Fun with contenttype and caching. Compare only the
queries spent on the documents themselves or else
TagSerializer.get_children() built a full nested TagSerializer(many=True)
for every tag, even when it had zero children, likely the common case for
most tags and maybe even most installs. Constructing a DRF ModelSerializer isn't
free (field introspection, deepcopy of declared fields, i18n lookups
all re-run per instantiation), so this scaled GET /api/tags/ linearly
with tag count in pure Python overhead, unrelated to SQL query count.
* Ok! Backend stuff for the remote ocr workflow
* Frotnend workflow stuff
* And docs
* Fix dynamic action fields thing
* Actually, fix the action dropdown thing
* Fix this validation thing, and we have to check existing actions
* Fix migration