Commit Graph
377 Commits
Author SHA1 Message Date
stumpylog f81ab41c1b Fix: reject zone date formats that cannot read a full date
A bad format silently fell back to auto-detection, and one without a year
read every date as 1900.
2026-11-19 09:09:40 -08:00
stumpylog ee83aff920 Fix: require unique zone names and known combine format keys
Combine tokens refer to zones by name, and formats for fields no zone uses
were silently ignored.
2026-11-19 09:06:09 -08:00
stumpylog 6eb0e4e6b3 Fix: reject a custom field on zones with a built-in target
Title, ASN and created zones stored a custom field that was then ignored.
2026-11-19 09:02:40 -08:00
stumpylog 884fbf7fb2 Fix: reject invalid zone validation regexes on save
A bad pattern made every extraction fail the regex check.
2026-11-19 09:00:05 -08:00
stumpylog 0117fed721 Fix: limit OCR template lookups to objects the user can see
Template and document references gave 400 for a missing id but 403 for a
hidden one, and a template could point at a hidden sample document. Narrow
the querysets, keeping a template's stored sample document, and drop the
view checks they replace.
2026-11-19 08:40:49 -08:00
stumpylog 6fc4526947 Fix: keep the currency and validate monetary zone values
OCR dropped currency codes and stored amounts the API would reject. Keep an
ISO code or locale currency symbol, read parentheses as negative, and validate
with the same rule as the API.
2026-11-19 07:34:16 -08:00
shamoon 855b161407 Move run_zone_ocr to the new bulk operations pathway 2026-11-17 13:43:13 -08:00
shamoon 3153b118bd Merge branch 'dev' into feature/ocr-templates 2026-11-17 13:30:29 -08:00
shamoon de75249e51 Chorehancement: move all bulk operations to async hand-off (#14773) 2026-11-17 00:23:44 +00:00
shamoon 8c50a5ee7b Merge branch 'dev' into feature/ocr-templates 2026-11-13 09:09:39 -08:00
Johannes Hagerandshamoon b63ea77543 Enhancement: render document notes as markdown (#14626)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-11-13 09:08:20 -08:00
shamoon 3dbeca603e Feature: manual workflow trigger (#14704) 2026-11-12 13:28:10 -08:00
shamoon 5c28d8282d Merge branch 'dev' into feature/ocr-templates 2026-11-09 08:40:39 -08:00
shamoonandCodex 8aa99a381a Performance: replace django-treenode with adjacency-list tags (#14612)
---------

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-11-06 20:18:59 +00:00
shamoon 800748d5ab Merge branch 'dev' into feature/ocr-templates 2026-10-25 10:29:32 -07:00
Trenton H 4299e9b12e Chore: consolidate direct permission grant lookups into one helper (#14456)
* Chore: drop get_groups_with_only_permission

It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.

* Chore: consolidate direct grant lookups into get_permission_grantees

Six places listed who holds a permission on an object, each with its own
query and result shape, and one had already drifted into expanding group
members.

* Chore: drop the unreachable empty viewer grant default

get_permission_grantees returns an entry for every requested document.
2026-10-24 13:28:25 -07:00
Trenton H 10a1a9eae8 Chore: drop get_groups_with_only_permission (#14454)
It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.
2026-10-24 13:28:25 -07:00
Trenton H e7d70497c2 Chore: Consolidate to a single way to set permission on an object (#14453)
* Chore: drop set_permissions_for_object in favor of the bulk helper

The same grant rules were implemented twice, and only the bulk version
validated every action before writing, so the single-object one could leave
a partial change behind.

* Chore: type the set-permissions dict

Callers passed whatever they had, including querysets of User and Group
objects that only worked because id__in accepts a subquery.
2026-10-24 13:28:25 -07:00
shamoon 182b0b4712 Merge branch 'dev' into feature/ocr-templates 2026-10-22 15:39:03 -07:00
Trenton H 7bdbcca051 Chore: Refactor bulk edit to remove PDF knowledge (#14445)
* Refactor: add pure pdf_ops module with real-PDF tests
* Drops tests we don't relly need + adds the typing annotation
* Now that I'm sure it works, re-work these tests to mock pdf_ops, rather than pikepdf
* Validates pages ahead of time in build_pdfs rather than during the building
* Adds the given/when/then and some additional cleanups of pytest style
* This is redundant now
2026-10-22 19:25:39 +00:00
shamoon 87913411ef Update serialisers.py 2026-10-20 16:49:35 -07:00
shamoon 95c8bf2058 Cleanup the serializers and views 2026-10-20 16:45:04 -07:00
shamoon bbdc016a32 Backend for manual tempalte choice 2026-10-20 13:51:54 -07:00
shamoon bd73fe1e80 Update the serializer 2026-10-20 13:31:08 -07:00
shamoon 4ff2cb9c72 Make them owned, duh
Update 0028_ocrtemplate_ocrtemplatezone.py
2026-10-20 11:15:30 -07:00
shamoon 8bb2f88015 Backend work for proper bulk zone ocr 2026-10-20 10:20:28 -07:00
shamoon cf6d16edcd Merge branch 'dev' into feature/ocr-templates 2026-10-19 21:37:23 -07:00
jurassicparkicecreamandshamoon 62cc31fbdb Feature: store barcode contents, list and search them (#14276)
* Feature: store barcode contents, list and search them

New setting PAPERLESS_CONSUMER_STORE_BARCODE_VALUES (off by default)
stores all barcodes found during consumption with the document: page,
type and content. They are listed on the metadata tab with a copy
button, returned by the documents API and searchable with barcodes:
in the advanced search. Versions keep their own barcodes, reprocessing
reads them again. Refs #9898

* Tests: cover the remaining barcode branches

Covers unchanged and failing barcode reads on reprocessing, unsupported files and DocumentBarcode.__str__, and uses toHaveLength in the barcode list spec as suggested by SonarCloud.

* Address review: keep barcodes when they can't be read, format choices

- Reprocessing and new versions keep the stored barcodes when the file
  can't be scanned or the scan fails, and replace them atomically.
- Format is a TextChoices of the zxing-cpp formats, with a test.
- Shared scan code, latest_version helper, TypedDict, serializer reuse.
- Barcodes in the split manifest, export/import tests, pytest-style tests.

* Barcode tests: TIFF reprocess, format check both ways, fixtures

- Reprocessing a TIFF with TIFF support off keeps the stored barcodes.
- The format test also fails when zxing-cpp drops a format.
- Fixtures in place of the sample dir mixin, plugin-level disabled test.
- Format labels aren't translated, OpenAPI enum named BarcodeFormatEnum.

* Review: module-level zxing reader, shorter barcode docs

- read_barcodes_zxing is a module-level function used by scan_pdf.
- Drop the trivial __str__ test, mark it no cover.
- Shorten the barcode docs and remove the duplicate in configuration.md.

* Fix header

* Use utility class

* Return barcodes from the metadata endpoint only

Drop the barcodes field and its prefetches from the document serializer,
as agreed in the review. Also remove the now empty component stylesheet.

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-10-05 16:25:19 +00:00
shamoon a59edf3e5b Fix: use version page_count for versioned document (#14280) 2026-09-27 23:26:00 -07:00
shamoon 452ed005bd Fix: handle legacy bulk edit page range with missing page_count (#14212) 2026-09-21 19:00:14 +00:00
shamoon f4ff87e69b Enhancement (QoL): support deselecting single items from "select all" (#14117) 2026-09-18 01:28:55 +00:00
shamoon c97c6923ea Enhancement: centralized management of share links + bundles (#14115) 2026-09-16 15:45:49 +00:00
Trenton H 683f2ac250 Fix: validate legacy bulk_edit owner/rotate/split parameters (#14120)
BulkEditSerializer's hand-parsed validators only caught the exceptions
their happy paths raised, so wrong-typed input produced a 500 or was
passed through to the task:

- owner: a nonexistent or wrong-typed id raised an uncaught error, and a
  boolean was accepted (Django coerces True to pk 1). It is now validated
  with PrimaryKeyRelatedField and the validated pk is passed on.
- rotate: null raised TypeError, while true, "90" and 45 were accepted and
  failed later in QPDF. Degrees are now an IntegerField plus a
  multiple-of-90 check, passing an int on. The dedicated rotate endpoint
  and edit_pdf operations share the same multiple-of-90 check.
- split: null raised AttributeError, "0" silently became the last page,
  "3-1" gave an empty group that crashed the task, and a range like
  "1-5000000" was expanded into a list during the request with no upper
  bound. Each range is now checked against 1 <= start <= end <= page_count
  before it is built.
2026-09-15 19:23:37 +00:00
Trenton H ef26bc1570 Fix: enforce set_permissions shape via a nested DRF serializer (#14119)
SetPermissionsSerializer was a bare DictField, and the legacy bulk_edit
set_permissions parameter bypassed even that by hand-calling
validate_set_permissions() on an unchecked dict. A bool or a non-list in
place of users/groups crashed with a raw TypeError instead of a 400.

It is now a nested serializer (view/change, each with users/groups as
lists of integers), used for owned-object create/update, the legacy
bulk_edit set_permissions parameter, and bulk_edit_objects permissions.
Unknown action keys are rejected: previously a typo like "veiw" was
silently dropped, leaving an empty permission set that could clear
existing grants. An explicit set_permissions null (an owner-only change)
remains a no-op, and an empty bulk_edit_objects permissions dict is still
rejected.
2026-09-15 18:46:32 +00:00
Trenton H 970eb55a70 Fix: type edit_pdf operations via a nested serializer (#14116)
operations was a ListField with no child, so both EditPdfDocumentsSerializer
and BulkEditSerializer._validate_parameters_edit_pdf hand-checked each entry
with isinstance(). That accepted booleans (isinstance(True, int) is true) and
passed through any extra keys.

Adds PdfEditOperationSerializer (page >= 1, doc >= 0, rotate a multiple of
90) and uses it as the ListField child on both paths, dropping the manual
type checks and the now-redundant negative/page < 1 bounds checks. Rotations
that are not a multiple of 90 previously passed validation and then failed
inside the task when QPDF refused them.
2026-09-15 15:43:21 +00:00
shamoon 5293194551 Fix: validate PDF output doc indexes in bulk edit (#14083) 2026-09-13 22:12:10 +00:00
shamoon 4421d4fe58 Fix: update some api global perms inconsistencies (#14086) 2026-09-12 16:17:48 -07:00
shamoon 26094bc863 Fix: ignore nested action IDs on WF create (#14084) 2026-09-12 16:14:17 -07:00
shamoon 9dbad4de09 Chore: read-only deleted_at 2026-09-12 16:13:58 -07:00
Trenton H d5f9605daf Performance: skip effective_content annotation on document list unless required (#13789)
* perf: skip effective_content annotation on document list unless filtered on

DocumentViewSet.get_queryset() always attached a correlated subquery
resolving each document's latest version content, even though it's only
needed for the deprecated search/title_content/content__* filter params.
Evaluated for every candidate row before pagination's LIMIT, this is
pathological on MariaDB: its default cardinality estimate for the mostly-
NULL root_document_id self-join drives it to a near-full-table scan per
row instead of using the FK index, turning a normal filtered list request
into a multi-second query (root cause of paperless-ngx#13778's report).

Only attach the annotation when a request actually filters on it. The
common case now relies on Document.get_effective_content()'s existing
prefetch-based fallback instead (extended the "versions" prefetch to
include content), which DocumentSerializer.to_representation() now calls
directly instead of checking for the annotation via hasattr().

* fix: address review feedback on effective_content annotation skip

- _needs_effective_content_annotation() now checks for a non-blank,
  stripped param value rather than mere key presence, matching how
  SearchFilter/TitleContentFilter/EffectiveContentFilter themselves
  no-op on a blank value. An empty ?search= or a saved view with a
  cleared text filter no longer re-triggers the annotation.

- The "versions" prefetch on DocumentViewSet no longer carries content
  for every historical version of every document -- that's unused
  bloat for version-heavy documents. Added
  latest_version_content_prefetch() (versioning.py), a separate,
  windowed prefetch scoped to just the newest version's content per
  root, and taught Document.get_effective_content() to check it first.

- DocumentSerializer.to_representation() no longer unconditionally
  calls get_effective_content(). Added has_prefetched_effective_content()
  (versioning.py) as a cheap upfront check: only resolve version-aware
  content when an SQL annotation or a versions prefetch is already on
  the instance. TrashView and GlobalSearchView build their own
  querysets independently of DocumentViewSet and never display
  document content at all (checked both frontend components), so they
  now keep showing the document's own, unresolved content with zero
  extra queries -- the same behavior as before effective_content
  resolution existed, just generalized past the narrow hasattr() check
  it replaced.

* Perf: derive _CONTENT_FILTER_PARAMS from DocumentFilterSet and search_fields instead of hand-maintaining it

* Fixes the new test failure and restricts doing the annotation even further, so content must have been requested to annotate even

* CLean up the new test with the docstrings, handle the fields in one place

* Fun with contenttype and caching. Compare only the
queries spent on the documents themselves or else
2026-09-10 12:38:41 -07:00
shamoon ca512af5ec Fix: ensure remove inbox tag children on remove_inbox_tags (#14050) 2026-09-10 16:03:47 +00:00
15c95c251d Feature: OCR Templates (#13043)
[skip ci]

Signed-off-by: dependabot[bot] <support@github.com>
Co-Authored-By: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-Authored-By: stumpylog <797416+stumpylog@users.noreply.github.com>
Co-Authored-By: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-Authored-By: shamoon <4887959+shamoon@users.noreply.github.com>
2026-09-09 12:43:09 -07:00
Trenton H 73e777a48c Fix: skip nested TagSerializer construction when a tag has no children (#14039)
TagSerializer.get_children() built a full nested TagSerializer(many=True)
for every tag, even when it had zero children, likely the common case for
most tags and maybe even most installs. Constructing a DRF ModelSerializer isn't
free (field introspection, deepcopy of declared fields, i18n lookups
all re-run per instantiation), so this scaled GET /api/tags/ linearly
with tag count in pure Python overhead, unrelated to SQL query count.
2026-09-08 15:30:58 +00:00
shamoon 3f4d2a4b5a And the backend 2026-09-04 07:14:58 -07:00
shamoon bfe8213b78 Fix: re-use permitted_object_ids 2026-09-01 07:45:47 -07:00
shamoon 549afde1fd Enhancement: Apply AI suggestions workflow action (#13639) 2026-08-26 16:41:21 +00:00
shamoon b90ccf910f Finally, the remote ocr workflow (#13637)
* Ok! Backend stuff for the remote ocr workflow

* Frotnend workflow stuff

* And docs

* Fix dynamic action fields thing

* Actually, fix the action dropdown thing

* Fix this validation thing, and we have to check existing actions

* Fix migration
2026-08-24 14:43:05 -07:00
shamoon c93c996edf Remote ocr reprocess (#13636)
* Backend stuff for remote ocr reprocess, add to bulk edit pass in from ui settings

* Ok, frontend reprocess remote option

* Docs
2026-08-24 14:43:05 -07:00
shamoon e389298aab Enhancement: merge documents as versions (#13515) 2026-08-19 07:20:14 -07:00
Max Truxa f5c0d118f7 Fix: fix validation of workflow title assignment (#13659) 2026-08-13 12:46:57 -07:00