Compare commits

..
Author SHA1 Message Date
stumpylog 3a2f003c44 Chore: remove the unused _process_template wrapper 2026-11-20 15:00:01 -08:00
stumpylog 612f8eb451 Chore: test that bulk edit rejects an update_callback parameter 2026-11-20 14:34:17 -08:00
stumpylog c1fa4eaf88 Chore: restore the fullmatch invalid pattern row 2026-11-20 14:34:16 -08:00
stumpylog bda4959f93 Chore: test combine formats on a template created without zones 2026-11-20 14:34:16 -08:00
stumpylog ab86b17f17 Chore: cover the zone OCR precision, no-template and trailing token paths 2026-11-20 14:34:15 -08:00
stumpylog 952b808a76 Chore: exclude two unreachable zone OCR handlers from coverage 2026-11-20 14:34:12 -08:00
stumpylog 9c396da87a Chore: exclude unreachable and defensive zone OCR code from coverage 2026-11-20 14:25:13 -08:00
stumpylog 19ee159a8a Chore: trim the OCR template export test to counts and restore 2026-11-20 14:09:20 -08:00
stumpylog aa1d41f7c5 Chore: trim the pdftoppm argument test to timeout and resolution 2026-11-20 14:09:19 -08:00
stumpylog 6d1f7a7f2d Chore: drop redundant fullmatch regex rows 2026-11-20 14:09:19 -08:00
stumpylog b341e15fbe Chore: drop mock-only bulk zone OCR queue tests 2026-11-20 14:09:18 -08:00
stumpylog f4da7a0ca5 Chore: drop bulk zone OCR tests covered by bulk_edit 2026-11-20 14:09:18 -08:00
stumpylog 3383f6a38d Chore: drop the consumer duplicate ASN test 2026-11-20 14:09:17 -08:00
stumpylog ce4bd6bc8b Chore: drop redundant OCR template API tests and rows 2026-11-20 14:09:17 -08:00
stumpylog 436dd22aa3 Chore: drop redundant zone OCR tests and rows 2026-11-20 14:09:16 -08:00
stumpylog 12fa07cb12 Chore: drop monetary validator rows that test Django 2026-11-20 13:45:41 -08:00
stumpylog 71dc5888e2 Chore: drop OCR template API tests of DRF behavior 2026-11-20 13:45:40 -08:00
stumpylog af378f8a8c Chore: drop zone OCR tests of framework and library behavior 2026-11-20 13:45:40 -08:00
stumpylog d54e78cdd0 Chore: drop zone OCR tests that cannot fail 2026-11-20 13:38:34 -08:00
stumpylog 3bb48fdc7a Chore: make the page image range test fail without the range check 2026-11-20 13:36:56 -08:00
stumpylog 0ed6d6a4f2 Chore: make the consumer write failure test reach the consumer's handler 2026-11-20 13:36:55 -08:00
stumpylog 133a4595bf Chore: make the zone date tests catch an ignored format and the 1900 boundary 2026-11-20 13:35:45 -08:00
stumpylog 9cc322455f Fix: keep exception text out of zone template validation errors
CodeQL flags py/stack-trace-exposure. Log the detail and return a generic message, as MatchRuleSerializer does.
2026-11-20 13:07:18 -08:00
stumpylog d6ae3e1ec6 Chore: log a warning when a zone title is cut to 128 characters 2026-11-20 12:54:28 -08:00
stumpylog a7d4c04a4b Chore: test the bulk zone OCR permission checks 2026-11-20 12:51:02 -08:00
stumpylog dcb0d2014a Chore: test that OCR templates cannot be deleted without permission 2026-11-20 12:49:40 -08:00
stumpylog 499ed8d442 Chore: keep the OCR field type list next to the converter 2026-11-20 11:54:30 -08:00
stumpylog 2d5e419850 Chore: match a root's own content without re-fetching the document 2026-11-20 11:46:02 -08:00
stumpylog 4d714f263b Chore: drop dead code left in zone OCR 2026-11-20 11:39:08 -08:00
stumpylog 17d5cb8855 Fix: let the first matching OCR template win a field
Later templates no longer overwrite it. The skipped zones report superseded and a warning is logged.
2026-11-20 11:30:31 -08:00
stumpylog a2c18123fa Fix: convert crops PNG cannot store before handing them to tesseract 2026-11-20 11:24:06 -08:00
stumpylog ad5a7b365e Fix: apply EXIF orientation when cropping zone images 2026-11-20 11:20:10 -08:00
stumpylog a54141058d Chore: number the editor page image URL from 1 like zone pages 2026-11-20 10:05:14 -08:00
stumpylog e81bdff6c2 Fix: report a last-page zone as missing when the page count is unknown
An image counts as one page and a PDF without a stored count is counted from the file.
2026-11-20 09:59:52 -08:00
stumpylog 6b8b946567 Chore: document the page numbering in zone OCR 2026-11-20 09:52:58 -08:00
stumpylog ad64f2a1b1 Chore: import dateparser and zxingcpp lazily in zone OCR 2026-11-20 09:51:57 -08:00
stumpylog 0348dc532f Fix: create the scratch dir before zone OCR renders into it 2026-11-20 09:38:50 -08:00
stumpylog 5182b719c5 Chore: resolve the zone OCR source file in one place 2026-11-20 09:34:24 -08:00
stumpylog 36de14d2c5 Fix: read the effective document version in the zone editor endpoints
The page image and test-zone endpoints now resolve the version like zone OCR runs do, including ?version=.
2026-11-20 09:28:36 -08:00
stumpylog f0c91bbd59 Chore: pass is_image to zone OCR instead of sniffing the file type 2026-11-20 09:20:25 -08:00
stumpylog ccbff94208 Fix: make bulk zone OCR survive a missing template or failing document
Inline extract_zones into run_zone_ocr, load documents and the template once, and process each root once. A failing document is logged and the task fails after the rest finish.
2026-11-20 09:09:28 -08:00
shamoon a08918eb9c Update frontend to use the bulk operation queuing pathway 2026-11-19 22:25:07 -08:00
stumpylog c5fdba73a1 Fix: queue single document zone OCR runs instead of running them in the request
Rendering and OCR ran inside the HTTP request and could block other
requests for minutes. The action now validates, then queues the bulk
zone OCR operation for the selected file and returns its task id.
2026-11-19 15:25:48 -08:00
stumpylog a6b81b2d64 Fix: read zones before the consume transaction and write them before workflows
Zone OCR ran inside the consume transaction, read the working copy instead
of the archive, and ran after the document added workflows. Zones are now
read from the archive before the transaction and written just before
those workflows, keeping values supplied at consume time.
2026-11-19 15:09:41 -08:00
stumpylog 54be9dc9c5 Fix: finish manual and bulk zone OCR runs like other bulk edits
Runs reindexed each document separately, never sent document_updated
and did not bump modified for custom field writes. They now finish
through bulk_update_documents for the documents that got a value.
2026-11-19 14:18:30 -08:00
stumpylog 0bca50588d Fix: drop a deleted custom field's combine format from OCR templates
The template kept the stale key, so saving it again failed validation.
2026-11-19 14:05:39 -08:00
stumpylog 246f4e53b1 Chore: describe the zone OCR responses in the API schema
The run status and skip reason were string literals, and the test zone
and 404 bodies had no schema. Add enums and response serializers for
them. The JSON is unchanged.
2026-11-19 13:54:59 -08:00
stumpylog 8f1a6d21ce Chore: annotate the zone OCR views, signal and helpers
The OCR views, signal handler and regex helpers were untyped, which hid
dead path checks in the views. Annotate them and drop the dead checks.
2026-11-19 13:46:52 -08:00
stumpylog d4f435de7d Chore: type the OCR template serializers
Validated zone data and the serializer methods were untyped, so the
checkers could not follow zones through validation and saving. Add a
ZoneData TypedDict, annotate the methods and add serializer generics.
2026-11-19 13:36:48 -08:00
stumpylog a9ad9d9d65 Chore: share one combine format key between OCR runs and validation
The run and the serializer each built the key and could drift. Both now
use field_key from zone_ocr.
2026-11-19 13:21:03 -08:00
stumpylog ff6b5f3820 Chore: dispatch zone OCR on the target, transform and data type enums
The string if/elif chains ended in catch-alls. Match on the enums with
assert_never instead. A stored value outside the choices now raises,
failing only that template.
2026-11-19 12:41:40 -08:00
stumpylog f35a2a4343 Chore: give zone OCR values a real type
Values were typed as object, and a result's value doubled as the
extracted text. Add a ZoneValue alias, carry the text separately, and
make the result dataclasses keyword-only.
2026-11-19 12:26:16 -08:00
stumpylog a490d9d348 Chore: type zone OCR results and reasons
Reasons become a StrEnum, the run response has a serializer instead of an
untyped dict list, and the zone preview returns its dataclass. The JSON is
unchanged.
2026-11-19 10:15:43 -08:00
stumpylog 4c2df48795 Chore: give OCR templates their own matching fields
Templates are never assigned to documents, so the classifier-based Automatic
matching inherited from MatchingModel could never work. Declare the matching
fields with the workflow trigger choices instead. Also keep zones when their
custom field is deleted, allow longer OCR language lists, fix help texts,
and regenerate the unreleased migration.
2026-11-19 10:04:31 -08:00
stumpylog 8e07452604 Fix: reject malformed zone OCR language codes
Garbage values went straight to tesseract and looked like an empty read.
2026-11-19 09:12:37 -08:00
stumpylog f81ab41c1b Fix: reject zone date formats that cannot read a full date
A bad format silently fell back to auto-detection, and one without a year
read every date as 1900.
2026-11-19 09:09:40 -08:00
stumpylog ee83aff920 Fix: require unique zone names and known combine format keys
Combine tokens refer to zones by name, and formats for fields no zone uses
were silently ignored.
2026-11-19 09:06:09 -08:00
stumpylog 6eb0e4e6b3 Fix: reject a custom field on zones with a built-in target
Title, ASN and created zones stored a custom field that was then ignored.
2026-11-19 09:02:40 -08:00
stumpylog 884fbf7fb2 Fix: reject invalid zone validation regexes on save
A bad pattern made every extraction fail the regex check.
2026-11-19 09:00:05 -08:00
stumpylog 0f4a71ed88 Fix: let documents without an owner use every OCR template
Unowned documents only matched unowned templates, so they never matched
templates created in the UI. Consider every template, as matching does.
2026-11-19 08:43:29 -08:00
stumpylog 0117fed721 Fix: limit OCR template lookups to objects the user can see
Template and document references gave 400 for a missing id but 403 for a
hidden one, and a template could point at a hidden sample document. Narrow
the querysets, keeping a template's stored sample document, and drop the
view checks they replace.
2026-11-19 08:40:49 -08:00
stumpylog f401267c4b Fix: let template viewers preview zones
The zone preview is a POST, so it required permission to add templates. Both
preview endpoints now require viewing templates and documents.
2026-11-19 08:27:51 -08:00
stumpylog 0e7cbda3d9 Fix: don't pass raw text on when a zone transform fails
A date or numeric transform that found nothing returned the OCR text, so the
field got junk. Return nothing and report transform_failed, which also blocks
a combined field.
2026-11-19 08:19:07 -08:00
stumpylog fcc465ec2e Fix: reject over-long text zone values instead of truncating
A truncated URL is a different URL. Let field validation reject string and URL
values that are too long.
2026-11-19 08:10:35 -08:00
stumpylog 7b25551e1e Fix: drop only the ASN when a zone's ASN is already in use
A duplicate ASN failed the save and rolled back every write of the template.
Check first and report the zone as duplicate_asn instead.
2026-11-19 08:06:08 -08:00
stumpylog 3504ad07c6 Fix: keep zone number parsing linear and cap its input
Two patterns backtracked quadratically on long OCR text, taking seconds on a
large zone. Rewrite both, skip numbers and amounts longer than 256 characters,
and search only the first 256 characters for a date.
2026-11-19 08:02:51 -08:00
stumpylog 6fc4526947 Fix: keep the currency and validate monetary zone values
OCR dropped currency codes and stored amounts the API would reject. Keep an
ISO code or locale currency symbol, read parentheses as negative, and validate
with the same rule as the API.
2026-11-19 07:34:16 -08:00
stumpylog 218122452c Chore: test zone OCR outcomes instead of mocking internals
Run real pdftoppm and tesseract on a sample page, and stop patching
_process_template in the extraction pipeline tests.
2026-11-19 07:21:28 -08:00
shamoon 4e2bda3a35 Merge branch 'dev' into feature/ocr-templates 2026-11-18 16:30:55 -08:00
shamoon 61c8eec856 Fix page count, picker corners 2026-11-18 15:11:13 -08:00
stumpylog 409539d4d1 Chore: render zone OCR pages through the shared pdftoppm helper
pdftoppm was called in three places with different options. Missing-binary handling is dropped since pdftoppm and tesseract are required at startup.
2026-11-18 14:51:40 -08:00
stumpylog c1bb327f1f Fix: keep OCR values intact when combining zones into one field
Separator cleanup ran over the extracted values too, so a negative amount lost its sign. A zone failing validation also no longer lets a partial value through.
2026-11-18 14:20:11 -08:00
stumpylog 0e6d06c255 Feature: find dates after a label in OCR zones
A crop may include a label, such as "Datum: 13.04.2026", and those dates never parsed unless they were ISO.
2026-11-18 13:56:25 -08:00
stumpylog acf7ba346c Fix: parse OCR zone dates strictly using the zone's language
Dates ignored the document's day/month order, and words like "now" or a bare "5" were accepted as dates.
2026-11-18 13:46:21 -08:00
stumpylog 63a87dd11b Fix: parse OCR numbers with Babel using the zone's language
Separators were guessed from the text, so "12.50" became 1250 for integer fields and the thousands and decimal separators were often swapped.
2026-11-18 13:34:26 -08:00
stumpylog 1f41e63af1 Chore: share compile and timeout handling across the safe regex helpers 2026-11-18 13:09:37 -08:00
stumpylog dc16714e19 Chore: align zone OCR test naming, URLs and docstrings 2026-11-18 13:03:41 -08:00
stumpylog 044da9a627 Chore: move misplaced zone OCR tests 2026-11-18 12:55:53 -08:00
stumpylog 093c18e68b Chore: parametrize near-duplicate zone OCR tests 2026-11-18 12:52:03 -08:00
stumpylog d706904a18 Chore: remove duplicate zone OCR tests and assert on untested ones 2026-11-18 12:48:20 -08:00
stumpylog 1c6742723f Chore: use shared user and client fixtures in zone OCR tests 2026-11-18 12:45:11 -08:00
stumpylog e03faab730 Chore: share zone OCR mock fixtures and helpers 2026-11-18 12:41:46 -08:00
stumpylog 9ff48ecfce Chore: add CustomFieldFactory 2026-11-18 12:35:38 -08:00
stumpylog 65fdfcdea7 Chore: default OcrTemplateFactory to MATCH_ANY and drop redundant test arguments
Match the model's default matching, so tests stop restating it along with other factory defaults.
2026-11-18 12:33:49 -08:00
stumpylog bb78373ad1 Chore: move zone OCR bulk edit API tests to a pytest class 2026-11-18 08:35:33 -08:00
stumpylog 73e0390a84 Chore: group zone OCR signal tests 2026-11-18 08:33:13 -08:00
stumpylog 6009027bf7 Chore: group zone OCR bulk operation tests and use mocker 2026-11-18 08:31:24 -08:00
stumpylog 17cdcda76b Chore: convert OCR template preview API tests to pytest 2026-11-18 08:28:50 -08:00
stumpylog 88d0f30c1a Chore: convert run zone OCR API tests to pytest 2026-11-18 08:23:48 -08:00
stumpylog e3904fe376 Chore: convert OCR template CRUD API tests to pytest 2026-11-18 08:19:10 -08:00
stumpylog 9d5c7abfaa Chore: convert zone OCR database tests to pytest
TestRunZoneExtraction used the real scratch dir, so it failed when run on its own.
2026-11-18 08:10:27 -08:00
stumpylog 1c685eb85a Chore: convert zone OCR extraction and render tests to pytest 2026-11-18 08:02:23 -08:00
stumpylog 44bbfd8932 Chore: convert zone OCR value function tests to pytest 2026-11-18 07:57:14 -08:00
stumpylog 4788ac602b Fix: limit manually run OCR templates to those the requester can view
A user editing someone else's document could run the owner's private
templates and see their names. Manual runs now use templates both can view.
2026-11-18 07:55:52 -08:00
stumpylog 0c90c57e3e Fix: include OCR templates in the document exporter
Templates and their zones were missing from the export manifest, so an
import lost them.
2026-11-18 07:55:52 -08:00
shamoon 855b161407 Move run_zone_ocr to the new bulk operations pathway 2026-11-17 13:43:13 -08:00
shamoon 3153b118bd Merge branch 'dev' into feature/ocr-templates 2026-11-17 13:30:29 -08:00
shamoon 30e95a6d28 Fix migration 2026-11-13 09:20:47 -08:00
shamoon 8c50a5ee7b Merge branch 'dev' into feature/ocr-templates 2026-11-13 09:09:39 -08:00
shamoon 886761a53b sonar 2026-11-09 09:32:40 -08:00
shamoon 198b165f26 Dont confuse image archive pdf with actual image 2026-11-09 09:21:47 -08:00
shamoon e63f27a712 Crap missed versioning stuff, default to latest but allow specific 2026-11-09 09:19:30 -08:00
shamoon 3ce5f8bc69 Ensure reindex happens after extraction on manual runs 2026-11-09 09:01:20 -08:00
shamoon 58cdd4f19a fix migration 2026-11-09 08:44:09 -08:00
shamoon 5c8c0db08f preview should also require global view doc 2026-11-09 08:43:34 -08:00
shamoon 5c28d8282d Merge branch 'dev' into feature/ocr-templates 2026-11-09 08:40:39 -08:00
shamoon aede2e6a5a Merge branch 'dev' into feature/ocr-templates 2026-11-06 08:46:46 -08:00
shamoon fa792d33c2 Fix icon imports 2026-11-06 08:46:42 -08:00
shamoon ade5901a4d Update for new icons 2026-11-02 21:46:45 -08:00
shamoon 2b620a2823 Merge branch 'dev' into feature/ocr-templates 2026-11-02 21:44:11 -08:00
shamoon 45b8977623 Merge branch 'dev' into feature/ocr-templates 2026-10-29 15:46:31 -07:00
shamoon 800748d5ab Merge branch 'dev' into feature/ocr-templates 2026-10-25 10:29:32 -07:00
shamoon 182b0b4712 Merge branch 'dev' into feature/ocr-templates 2026-10-22 15:39:03 -07:00
shamoon c718636f79 Update usage.md 2026-10-20 17:07:09 -07:00
shamoonandOpenAI Codex 54f7f43f35 Reuse OCR page renders and simplify value conversion
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 17:07:00 -07:00
shamoonandOpenAI Codex 94da3d4c57 Report actual OCR extraction outcomes instead of existing field values
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 17:02:23 -07:00
shamoonandOpenAI Codex 453e89d6dd Prepare and atomically apply OCR template field updates
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 17:00:03 -07:00
shamoonandOpenAI Codex 6562415894 Unify OCR zone extraction for preview and saved templates
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 16:58:10 -07:00
shamoon 257c33453d Update test_api_ocr_templates.py 2026-10-20 16:49:39 -07:00
shamoon 87913411ef Update serialisers.py 2026-10-20 16:49:35 -07:00
shamoon cec393dcae Update views.py 2026-10-20 16:49:32 -07:00
shamoon f5526d5700 Improve backend tests 2026-10-20 16:47:48 -07:00
shamoon 2534d3c16b Absurd comments 2026-10-20 16:46:21 -07:00
shamoon 95c8bf2058 Cleanup the serializers and views 2026-10-20 16:45:04 -07:00
shamoon 930d23956b Up frontend coverage 2026-10-20 16:34:22 -07:00
shamoon d2e6834602 Make sort order stable 2026-10-20 16:28:31 -07:00
shamoon a7bf04ddee Update spacing 2026-10-20 16:17:56 -07:00
shamoon 8ab2bf63aa Merge branch 'dev' into feature/ocr-templates 2026-10-20 16:16:31 -07:00
shamoon c63a7e25e7 Update docs 2026-10-20 15:37:08 -07:00
shamoon 9171a1aa6a Changeup the icon 2026-10-20 15:33:01 -07:00
shamoon 55c5d9121f Yes, cute little split button that opens the dialog 2026-10-20 15:31:48 -07:00
shamoon e4efb62667 And bulk edit 2026-10-20 14:02:04 -07:00
shamoon f0597a7c54 doc detail use it this way 2026-10-20 14:00:25 -07:00
shamoon bbdc016a32 Backend for manual tempalte choice 2026-10-20 13:51:54 -07:00
shamoon eae392469f Update the frontend for the matching stuff 2026-10-20 13:41:55 -07:00
shamoon bedefd85f2 Remove the handlers, make the matching work 2026-10-20 13:39:09 -07:00
shamoon bd73fe1e80 Update the serializer 2026-10-20 13:31:08 -07:00
shamoon 8b77a924dc Make it a matchingmodel 2026-10-20 13:29:42 -07:00
shamoon 1bdeb7ebcd Mas sonar 2026-10-20 13:15:17 -07:00
shamoon fa3acfe13f Sure sonar 2026-10-20 13:12:26 -07:00
shamoon 2bc22b14bd Make perms accordion 2026-10-20 13:03:04 -07:00
shamoon dbce8c1409 User perms on frontend 2026-10-20 11:25:48 -07:00
shamoon e976b2ec7a Tests 2026-10-20 11:23:44 -07:00
shamoon 05da0ba5a3 Apply only visible templates 2026-10-20 11:15:31 -07:00
shamoon 4ff2cb9c72 Make them owned, duh
Update 0028_ocrtemplate_ocrtemplatezone.py
2026-10-20 11:15:30 -07:00
shamoon 77a7fb40ec safe regex 2026-10-20 11:05:49 -07:00
shamoon 8100a8e1eb Lol and run zone ocr 2026-10-20 11:04:17 -07:00
shamoon 1e334903f3 And test zone 2026-10-20 11:01:11 -07:00
shamoon a958d89bf4 Add perms to document_page_image 2026-10-20 10:58:54 -07:00
shamoon 8b3295fce0 Docs 2026-10-20 10:42:16 -07:00
shamoon 9783c4995f Fix the tests for now 2026-10-20 10:42:12 -07:00
shamoon e89495dc02 Frontend use the bulk edit 2026-10-20 10:32:48 -07:00
shamoon 8bb2f88015 Backend work for proper bulk zone ocr 2026-10-20 10:20:28 -07:00
shamoon 7d9b8c770d Cleanup doc detail run zone ocr 2026-10-20 10:16:58 -07:00
shamoon 3296fd7cb5 Formatting, remove hard-coded alpha stuff 2026-10-20 10:14:03 -07:00
shamoon bff41907c2 Move back controls, drop (lol) dnd 2026-10-20 10:08:43 -07:00
shamoon 592f882394 Get rid of the x for the picker 2026-10-20 10:06:40 -07:00
shamoon 29ea9cec66 Gosh so much cleanup 2026-10-20 10:03:55 -07:00
shamoon 123df2e826 Pngx conventions 2026-10-20 09:50:29 -07:00
shamoon 639827ec65 Lol, no not deu+eng default 2026-10-20 09:45:30 -07:00
shamoon b3106e408c Lots of enum cleanup 2026-10-20 09:44:37 -07:00
shamoon ea31b6a043 Make it close not leave page 2026-10-20 09:38:29 -07:00
shamoon 0572b54579 Proper zoneless stuff 2026-10-20 09:35:39 -07:00
shamoon 2afec3e0b7 Lots of dead code 2026-10-20 09:31:59 -07:00
shamoon 57019bccbe Cleanup save now that we're modal 2026-10-20 09:30:12 -07:00
shamoon 26b3964a20 Cleanup the geometry stuff 2026-10-20 09:26:42 -07:00
shamoon a2eeadca7f Fix the delete handler 2026-10-20 09:24:26 -07:00
shamoon ba5474ba81 Lol, just one icon 2026-10-20 09:20:06 -07:00
shamoon e3592804bd Embiggen 2026-10-20 09:18:36 -07:00
shamoon 2d54927be8 Fuck it, make it a modal like literally everything else, drop the routes 2026-10-20 09:14:52 -07:00
shamoon 1d2832ac30 Update ocr-template-editor.component.ts 2026-10-20 09:05:24 -07:00
shamoon 3321ba46d6 Automatically update the combine string when changing zone name 2026-10-20 09:00:17 -07:00
shamoon bf316cba8a Dont fail if one bulk fails, collect results separately 2026-10-20 08:57:24 -07:00
shamoon 36c31b3a73 Also prevent editing template while saving 2026-10-20 08:47:26 -07:00
shamoon d10d303980 Warn about losing changes, disable while zone ocr running 2026-10-20 08:36:22 -07:00
shamoon a51e6a71f4 Fix reload after running zone ocr 2026-10-20 08:20:44 -07:00
shamoon ebd8dba3bb Fix the no zones note 2026-10-20 08:20:25 -07:00
shamoon 3ed9586347 Remove the create link from doc details 2026-10-20 08:11:40 -07:00
shamoon 7055f7a745 Fix perms test 2026-10-20 08:07:46 -07:00
shamoon 8f7500ba64 Fix the signal thing 2026-10-20 08:03:33 -07:00
shamoon 816774815c Add a little warning for empty zones 2026-10-20 07:39:47 -07:00
shamoon c5e6748771 Use braces icon instead 2026-10-20 07:18:01 -07:00
shamoon 9b42170554 Make the floating controls movable 2026-10-20 07:15:18 -07:00
shamoon d2ed0c55a4 dirtycheck 2026-10-19 23:12:09 -07:00
shamoon 4d31063082 Default to empty field 2026-10-19 23:07:12 -07:00
shamoon a4fb9bf2b0 Drop the silly mode toggle 2026-10-19 23:04:35 -07:00
shamoon e028257da5 Cleanup the zone combine stuff 2026-10-19 22:55:23 -07:00
shamoon 68eb4fe96d Make the zone list more useful, show colors 2026-10-19 22:43:12 -07:00
shamoon d2d434535b Remove the slop quick create CF stuff 2026-10-19 22:36:17 -07:00
shamoon 1d75c0c99b Use pngx inputs, move test button, generally make less terrible 2026-10-19 22:31:33 -07:00
shamoon 1de1261728 Ok thats a better start, two tabs 2026-10-19 22:23:50 -07:00
shamoon f5f33e0744 make the ui usable 2026-10-19 22:06:11 -07:00
shamoon c9d62afa50 Make it hideable too 2026-10-19 21:43:10 -07:00
shamoon b5a9353780 Move migration 2026-10-19 21:39:26 -07:00
shamoon cf6d16edcd Merge branch 'dev' into feature/ocr-templates 2026-10-19 21:37:23 -07:00
shamoon 551445773a Use SVG, get rid of all the manual redraw stuff 2026-09-09 12:43:11 -07:00
shamoon c54cb4a84b Use a real canvas interaction model
[skip ci]
2026-09-09 12:43:11 -07:00
shamoon af61044b63 Fix some permissions stuff, show error on toggle 2026-09-09 12:43:11 -07:00
shamoon 1ef5a6d3cf Get rid of absurd alert 2026-09-09 12:43:11 -07:00
shamoon 4da311f91f Proper data types
[skip ci]
2026-09-09 12:43:10 -07:00
shamoon cbb84649c7 Pull out the zone list into a sub-component 2026-09-09 12:43:10 -07:00
shamoon 8da734a495 Extract the geometry stuff a bit 2026-09-09 12:43:10 -07:00
shamoon 351671011d Remove afterviewinit, fix page change, custom date format per zone 2026-09-09 12:43:09 -07:00
15c95c251d Feature: OCR Templates (#13043)
[skip ci]

Signed-off-by: dependabot[bot] <support@github.com>
Co-Authored-By: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-Authored-By: stumpylog <797416+stumpylog@users.noreply.github.com>
Co-Authored-By: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-Authored-By: shamoon <4887959+shamoon@users.noreply.github.com>
2026-09-09 12:43:09 -07:00
74 changed files with 15675 additions and 212 deletions

No files matched your search

-14
View File
@@ -896,13 +896,6 @@
"concise_description": "Unexpected keyword argument `fast` in function `tantivy.tantivy.SchemaBuilder.add_json_field`", "concise_description": "Unexpected keyword argument `fast` in function `tantivy.tantivy.SchemaBuilder.add_json_field`",
"severity": "error" "severity": "error"
}, },
{
"column": 9,
"path": "src/documents/serialisers.py",
"name": "bad-override-param-name",
"concise_description": "Class member `MatchingModelSerializer.validate` overrides parent class `ModelSerializer` in an inconsistent manner",
"severity": "error"
},
{ {
"column": 62, "column": 62,
"path": "src/documents/serialisers.py", "path": "src/documents/serialisers.py",
@@ -952,13 +945,6 @@
"concise_description": "Cannot set item in `Mapping[str, Any]`", "concise_description": "Cannot set item in `Mapping[str, Any]`",
"severity": "error" "severity": "error"
}, },
{
"column": 9,
"path": "src/documents/serialisers.py",
"name": "bad-override-param-name",
"concise_description": "Class member `TagSerializer.validate` overrides parent class `MatchingModelSerializer` in an inconsistent manner",
"severity": "error"
},
{ {
"column": 17, "column": 17,
"path": "src/documents/serialisers.py", "path": "src/documents/serialisers.py",
+94 -1
View File
@@ -109,7 +109,7 @@ Think of versions as **file history** for a document.
Paperless-ngx includes management lists for tags, correspondents, document types Paperless-ngx includes management lists for tags, correspondents, document types
and more. These areas allow you to view, add, edit, delete and manage permissions and more. These areas allow you to view, add, edit, delete and manage permissions
for these objects. You can also manage saved views, mail accounts, mail rules, for these objects. You can also manage saved views, mail accounts, mail rules,
workflows and more from the management sections. workflows, [OCR templates](#ocr-templates) and more from the management sections.
### Nested Tags ### Nested Tags
@@ -821,6 +821,99 @@ The following custom field types are supported:
- `Document Link`: reference(s) to other document(s) displayed as links, automatically creates a symmetrical link in reverse - `Document Link`: reference(s) to other document(s) displayed as links, automatically creates a symmetrical link in reverse
- `Select`: a pre-defined list of strings from which the user can choose - `Select`: a pre-defined list of strings from which the user can choose
## OCR Templates {#ocr-templates}
OCR templates read text from fixed areas ("zones") of a document and write it to a
[custom field](#custom-fields), or to the document's title, archive serial number or created date. They are
useful for documents that always share the same layout, such as bills from a particular utility. They allow you
to draw a zone around the account number once, and every new bill gets its account number filled in.
Like tags or document types, each template has a [matching algorithm](advanced_usage.md#matching) that
decides which documents it automatically applies to (or not), based on their content.
### Creating a template
1. Open **OCR Templates** under "Manage" and choose **Add Template**.
2. Give the template a name and set up its matching, for example **Any word** with the utility's name. Choose
**None** for a template that should only ever be run manually.
3. Search for a sample document with the layout the template is for. Its pages are shown next to the template
settings.
4. Draw zones on the document. Click a zone to select and edit it.
5. Choose the field each zone fills.
6. Use **Test** to run OCR on the selected zone of the sample document and see the text it finds and the value
that would be written.
Zones are positioned relative to the sample document's page, so templates work best when the documents they
match have an identical layout. Keep the matching specific enough that it only catches documents with that
layout. When more than one template applies to a document, the first one by name fills a field and later
templates do not overwrite it.
### Zone settings
- **Page**: the page the zone is on. Turn on **Last page** for zones that are always on the final page,
regardless of how many pages a document has. If the number of pages cannot be determined, the zone is skipped.
- **OCR language**: the Tesseract languages used to read the zone. Languages other than the defaults must be
installed, see [`PAPERLESS_OCR_LANGUAGES`](configuration.md#PAPERLESS_OCR_LANGUAGES).
- **Transform**: post-processing applied to the text, such as stripping whitespace, keeping only numbers,
parsing a date or reading a QR code or barcode instead of text.
- **Date format**: for the "Parse date" transform, the format of the date using
[Python date codes](https://docs.python.org/3/library/datetime.html#format-codes), e.g. `%d.%m.%Y`.
Leave it empty to detect the format automatically.
- **Validation regex**: a regular expression the whole value must match. Values that do not match are not
written.
Zones can fill custom fields of type text, long text, URL, date, integer, number, monetary or boolean.
Document link and select fields are not supported.
### Combining zones
When several zones fill the same field, a **Combined value** setting appears for that field. Use it to arrange
the zones' values with `{Zone name}` tokens and any text in between, for example `{Start} - {End}`. If it
is left empty, the values are joined with spaces in zone order.
### When templates run
When a document is added, every enabled template that matches its content runs automatically, reading the
zones from the archived file, or from the original PDF or image when there is no archived file. A template is
only used if the document's owner can view it, so users' templates apply to their own documents, and unowned
templates apply to everyone's. The owner here is the one set by the upload, the mail rule or a consumption
workflow. A document without an owner at that point, for example from the consume folder or a mail rule
without an owner, considers every enabled template. An owner assigned later by a "Document Added" workflow
does not change which templates ran. Templates run in name order, so if two templates fill the same field, the
later one wins.
Zone OCR keeps values supplied when the document is consumed: an uploaded title, ASN, created date or custom
field value, a mail rule's title, a barcode ASN, and values assigned by a consumption workflow. Zones fill the
remaining fields before "Document Added" workflows run, so those workflows can use or overwrite the values.
Turn off **Enabled** to pause a template without deleting it. Disabled templates never run automatically, but
can still be run manually.
### Running templates manually
To run templates on documents that are already in Paperless-ngx, open the **Actions** menu on the document
details page:
- **Run Zone OCR** runs every enabled template that matches the document.
- The button next to it opens a dialog where you can choose one template to run
instead. A template you choose runs even if it does not match the document or is disabled.
Both actions use the file version selected in document details. The latest version is selected by default.
Extracted values update the document's metadata, which is shared across its versions. If no template
applies to the document, nothing is run. Otherwise the run is processed in the background and appears in the
task list. The document shows the extracted values once the task finishes.
The bulk editor's **Run Zone OCR** action opens the same dialog for the selected documents. Bulk runs use
the latest version of each document and are processed in the background.
!!! warning
Running zone OCR manually or in bulk overwrites the fields it fills. A zone that finds no value leaves its
field unchanged.
Managing OCR templates requires the OCR template permissions. Running zone OCR on documents requires
permission to change them, and choosing a template requires permission to view it.
## PDF Actions ## PDF Actions
Paperless-ngx supports basic editing operations for PDFs (these operations currently cannot be performed on non-PDF files). When viewing an individual document you can Paperless-ngx supports basic editing operations for PDFs (these operations currently cannot be performed on non-PDF files). When viewing an individual document you can
+1 -7
View File
@@ -71,7 +71,7 @@ dependencies = [
"scikit-learn~=1.9.0", "scikit-learn~=1.9.0",
"sentence-transformers>=5.6.1", "sentence-transformers>=5.6.1",
"setproctitle~=1.3.4", "setproctitle~=1.3.4",
"sqlite-vec==0.1.10+paperless.3", "sqlite-vec==0.1.9",
"tantivy~=0.26.0", "tantivy~=0.26.0",
"tika-client[httpx]~=1.0", "tika-client[httpx]~=1.0",
"torch>=2.13,<2.15", "torch>=2.13,<2.15",
@@ -164,12 +164,6 @@ psycopg-c = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64' and python_version == '3.14'" }, { url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64' and python_version == '3.14'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64' and python_version == '3.14'" }, { url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64' and python_version == '3.14'" },
] ]
# Patched sqlite-vec builds
sqlite-vec = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-macosx_11_0_arm64.whl", marker = "sys_platform == 'darwin' and platform_machine == 'arm64'" },
]
torch = [ torch = [
{ index = "pytorch-cpu" }, { index = "pytorch-cpu" },
] ]
+13
View File
@@ -13,6 +13,7 @@ import { DocumentDetailComponent } from './components/document-detail/document-d
import { DocumentListComponent } from './components/document-list/document-list.component' import { DocumentListComponent } from './components/document-list/document-list.component'
import { DocumentAttributesComponent } from './components/manage/document-attributes/document-attributes.component' import { DocumentAttributesComponent } from './components/manage/document-attributes/document-attributes.component'
import { MailComponent } from './components/manage/mail/mail.component' import { MailComponent } from './components/manage/mail/mail.component'
import { OcrTemplatesComponent } from './components/manage/ocr-templates/ocr-templates.component'
import { SavedViewsComponent } from './components/manage/saved-views/saved-views.component' import { SavedViewsComponent } from './components/manage/saved-views/saved-views.component'
import { ShareLinksComponent } from './components/manage/share-links/share-links.component' import { ShareLinksComponent } from './components/manage/share-links/share-links.component'
import { WorkflowsComponent } from './components/manage/workflows/workflows.component' import { WorkflowsComponent } from './components/manage/workflows/workflows.component'
@@ -275,6 +276,18 @@ export const routes: Routes = [
componentName: 'WorkflowsComponent', componentName: 'WorkflowsComponent',
}, },
}, },
{
path: 'ocr-templates',
component: OcrTemplatesComponent,
canActivate: [PermissionsGuard],
data: {
requiredPermission: {
action: PermissionAction.View,
type: PermissionType.OcrTemplate,
},
componentName: 'OcrTemplatesComponent',
},
},
{ {
path: 'mail', path: 'mail',
component: MailComponent, component: MailComponent,
@@ -113,6 +113,7 @@ const sidebarItemLabels: Record<HideableSidebarItemID, string> = {
[HideableSidebarItemID.SavedViews]: $localize`Saved Views`, [HideableSidebarItemID.SavedViews]: $localize`Saved Views`,
[HideableSidebarItemID.ShareLinks]: $localize`Share Links`, [HideableSidebarItemID.ShareLinks]: $localize`Share Links`,
[HideableSidebarItemID.Workflows]: $localize`Workflows`, [HideableSidebarItemID.Workflows]: $localize`Workflows`,
[HideableSidebarItemID.OcrTemplates]: $localize`OCR Templates`,
[HideableSidebarItemID.Mail]: $localize`Mail`, [HideableSidebarItemID.Mail]: $localize`Mail`,
[HideableSidebarItemID.Documentation]: $localize`Documentation`, [HideableSidebarItemID.Documentation]: $localize`Documentation`,
} }
@@ -289,6 +289,17 @@
<pngx-input-switch class="position-absolute top-50 end-0 translate-middle-y me-1" [class.d-none]="slimSidebarEnabled || slimSidebarAnimating()" [compact]="true" title="Workflows" i18n-title [ngModel]="!settingsService.sidebarItemIsHidden(HideableSidebarItemID.Workflows)" (ngModelChange)="toggleSidebarItem(HideableSidebarItemID.Workflows, $event)"></pngx-input-switch> <pngx-input-switch class="position-absolute top-50 end-0 translate-middle-y me-1" [class.d-none]="slimSidebarEnabled || slimSidebarAnimating()" [compact]="true" title="Workflows" i18n-title [ngModel]="!settingsService.sidebarItemIsHidden(HideableSidebarItemID.Workflows)" (ngModelChange)="toggleSidebarItem(HideableSidebarItemID.Workflows, $event)"></pngx-input-switch>
} }
</li> </li>
<li class="nav-item app-link position-relative" [class.d-none]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.OcrTemplates) && !settingsService.organizingSidebarItems()"
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.OcrTemplate }">
<a class="nav-link" [class.opacity-50]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.OcrTemplates)" [class.pe-5]="settingsService.organizingSidebarItems() && !slimSidebarEnabled && !slimSidebarAnimating()" routerLink="ocr-templates" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="OCR Templates" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<pngx-bs-icon class="me-2" name="bounding-box-circles"></pngx-bs-icon><span class="nav-link-label"><ng-container i18n>OCR Templates</ng-container></span>
</a>
@if (settingsService.organizingSidebarItems()) {
<pngx-input-switch class="position-absolute top-50 end-0 translate-middle-y me-1" [class.d-none]="slimSidebarEnabled || slimSidebarAnimating()" [compact]="true" title="OCR Templates" i18n-title [ngModel]="!settingsService.sidebarItemIsHidden(HideableSidebarItemID.OcrTemplates)" (ngModelChange)="toggleSidebarItem(HideableSidebarItemID.OcrTemplates, $event)"></pngx-input-switch>
}
</li>
<li class="nav-item app-link position-relative" [class.d-none]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail) && !settingsService.organizingSidebarItems()" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.MailAccount }" <li class="nav-item app-link position-relative" [class.d-none]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail) && !settingsService.organizingSidebarItems()" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.MailAccount }"
tourAnchor="tour.mail"> tourAnchor="tour.mail">
<a class="nav-link" [class.opacity-50]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail)" [class.pe-5]="settingsService.organizingSidebarItems() && !slimSidebarEnabled && !slimSidebarAnimating()" routerLink="mail" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Mail" <a class="nav-link" [class.opacity-50]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail)" [class.pe-5]="settingsService.organizingSidebarItems() && !slimSidebarEnabled && !slimSidebarAnimating()" routerLink="mail" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Mail"
@@ -404,6 +404,7 @@ describe('AppFrameComponent', () => {
HideableSidebarItemID.Dashboard, HideableSidebarItemID.Dashboard,
HideableSidebarItemID.Workflows, HideableSidebarItemID.Workflows,
HideableSidebarItemID.ShareLinks, HideableSidebarItemID.ShareLinks,
HideableSidebarItemID.OcrTemplates,
]) ])
fixture.detectChanges() fixture.detectChanges()
@@ -419,6 +420,10 @@ describe('AppFrameComponent', () => {
fixture.nativeElement.querySelector('[routerLink="share-links"]') fixture.nativeElement.querySelector('[routerLink="share-links"]')
.parentElement.classList .parentElement.classList
).toContain('d-none') ).toContain('d-none')
expect(
fixture.nativeElement.querySelector('[routerLink="ocr-templates"]')
.parentElement.classList
).toContain('d-none')
expect( expect(
fixture.nativeElement.querySelector('[routerLink="mail"]').parentElement fixture.nativeElement.querySelector('[routerLink="mail"]').parentElement
.classList .classList
@@ -431,12 +436,21 @@ describe('AppFrameComponent', () => {
]) ])
settingsService.sidebarHiddenItemsEditing.set([ settingsService.sidebarHiddenItemsEditing.set([
HideableSidebarItemID.Dashboard, HideableSidebarItemID.Dashboard,
HideableSidebarItemID.OcrTemplates,
]) ])
fixture.detectChanges() fixture.detectChanges()
expect( expect(
fixture.nativeElement.querySelectorAll('pngx-input-switch').length fixture.nativeElement.querySelectorAll('pngx-input-switch').length
).toBe(6) ).toBe(7)
const ocrTemplatesLink = fixture.nativeElement.querySelector(
'[routerLink="ocr-templates"]'
)
expect(ocrTemplatesLink.parentElement.classList).not.toContain('d-none')
expect(ocrTemplatesLink.classList).toContain('opacity-50')
expect(
ocrTemplatesLink.parentElement.querySelector('pngx-input-switch')
).not.toBeNull()
expect( expect(
fixture.nativeElement.querySelector('[routerLink="dashboard"]') fixture.nativeElement.querySelector('[routerLink="dashboard"]')
.parentElement.classList .parentElement.classList
@@ -16,6 +16,7 @@ import basket from 'bootstrap-icons/icons/basket.svg'
import bell from 'bootstrap-icons/icons/bell.svg' import bell from 'bootstrap-icons/icons/bell.svg'
import bodyText from 'bootstrap-icons/icons/body-text.svg' import bodyText from 'bootstrap-icons/icons/body-text.svg'
import bookmark from 'bootstrap-icons/icons/bookmark.svg' import bookmark from 'bootstrap-icons/icons/bookmark.svg'
import boundingBoxCircles from 'bootstrap-icons/icons/bounding-box-circles.svg'
import boxArrowInRight from 'bootstrap-icons/icons/box-arrow-in-right.svg' import boxArrowInRight from 'bootstrap-icons/icons/box-arrow-in-right.svg'
import boxArrowUpRight from 'bootstrap-icons/icons/box-arrow-up-right.svg' import boxArrowUpRight from 'bootstrap-icons/icons/box-arrow-up-right.svg'
import boxArrowUp from 'bootstrap-icons/icons/box-arrow-up.svg' import boxArrowUp from 'bootstrap-icons/icons/box-arrow-up.svg'
@@ -77,6 +78,7 @@ import fileEarmarkMinus from 'bootstrap-icons/icons/file-earmark-minus.svg'
import fileEarmarkPerson from 'bootstrap-icons/icons/file-earmark-person.svg' import fileEarmarkPerson from 'bootstrap-icons/icons/file-earmark-person.svg'
import fileEarmarkPlus from 'bootstrap-icons/icons/file-earmark-plus.svg' import fileEarmarkPlus from 'bootstrap-icons/icons/file-earmark-plus.svg'
import fileEarmarkRichtext from 'bootstrap-icons/icons/file-earmark-richtext.svg' import fileEarmarkRichtext from 'bootstrap-icons/icons/file-earmark-richtext.svg'
import fileEarmarkRuled from 'bootstrap-icons/icons/file-earmark-ruled.svg'
import fileEarmarkSpreadsheet from 'bootstrap-icons/icons/file-earmark-spreadsheet.svg' import fileEarmarkSpreadsheet from 'bootstrap-icons/icons/file-earmark-spreadsheet.svg'
import fileEarmark from 'bootstrap-icons/icons/file-earmark.svg' import fileEarmark from 'bootstrap-icons/icons/file-earmark.svg'
import fileText from 'bootstrap-icons/icons/file-text.svg' import fileText from 'bootstrap-icons/icons/file-text.svg'
@@ -131,6 +133,7 @@ import search from 'bootstrap-icons/icons/search.svg'
import send from 'bootstrap-icons/icons/send.svg' import send from 'bootstrap-icons/icons/send.svg'
import shop from 'bootstrap-icons/icons/shop.svg' import shop from 'bootstrap-icons/icons/shop.svg'
import slashCircle from 'bootstrap-icons/icons/slash-circle.svg' import slashCircle from 'bootstrap-icons/icons/slash-circle.svg'
import sliders2 from 'bootstrap-icons/icons/sliders2.svg'
import sliders2Vertical from 'bootstrap-icons/icons/sliders2-vertical.svg' import sliders2Vertical from 'bootstrap-icons/icons/sliders2-vertical.svg'
import sortAlphaDown from 'bootstrap-icons/icons/sort-alpha-down.svg' import sortAlphaDown from 'bootstrap-icons/icons/sort-alpha-down.svg'
import sortAlphaUpAlt from 'bootstrap-icons/icons/sort-alpha-up-alt.svg' import sortAlphaUpAlt from 'bootstrap-icons/icons/sort-alpha-up-alt.svg'
@@ -179,6 +182,7 @@ export const icons: Record<string, string> = {
braces, braces,
bodyText, bodyText,
bookmark, bookmark,
boundingBoxCircles,
boxArrowInRight, boxArrowInRight,
boxArrowUp, boxArrowUp,
boxArrowUpRight, boxArrowUpRight,
@@ -240,6 +244,7 @@ export const icons: Record<string, string> = {
fileEarmarkPerson, fileEarmarkPerson,
fileEarmarkPlus, fileEarmarkPlus,
fileEarmarkRichtext, fileEarmarkRichtext,
fileEarmarkRuled,
fileEarmarkSpreadsheet, fileEarmarkSpreadsheet,
files, files,
fileText, fileText,
@@ -292,6 +297,7 @@ export const icons: Record<string, string> = {
search, search,
send, send,
slashCircle, slashCircle,
sliders2,
shop, shop,
sliders2Vertical, sliders2Vertical,
sortAlphaDown, sortAlphaDown,
@@ -0,0 +1,30 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (messageBold) {
<p class="text-break"><b>{{messageBold}}</b></p>
}
@if (message) {
<p class="text-break" [innerHTML]="message"></p>
}
<pngx-input-select
[items]="templates()"
[(ngModel)]="template"
[allowNull]="true"
title="OCR template"
i18n-title
placeholder="All matching templates"
i18n-placeholder
></pngx-input-select>
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled()">
{{btnCaption}}
</button>
</div>
@@ -0,0 +1,52 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { of } from 'rxjs'
import { MATCH_NONE } from 'src/app/data/matching-model'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { ZoneOcrConfirmDialogComponent } from './zone-ocr-confirm-dialog.component'
const templates: OcrTemplate[] = [
{
id: 1,
name: 'Invoices',
match: '',
matching_algorithm: MATCH_NONE,
sample_document: null,
source_width: 0,
source_height: 0,
enabled: true,
zones: [],
},
]
describe('ZoneOcrConfirmDialogComponent', () => {
let component: ZoneOcrConfirmDialogComponent
let fixture: ComponentFixture<ZoneOcrConfirmDialogComponent>
beforeEach(() => {
TestBed.configureTestingModule({
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
imports: [ZoneOcrConfirmDialogComponent],
})
jest
.spyOn(TestBed.inject(OcrTemplateService), 'listAll')
.mockReturnValue(of({ count: 1, all: [1], results: templates }))
fixture = TestBed.createComponent(ZoneOcrConfirmDialogComponent)
component = fixture.componentInstance
fixture.detectChanges()
})
it('should offer templates and default to all matching templates', () => {
expect(component.templates()).toEqual(templates)
expect(component.template).toBeNull()
expect(fixture.nativeElement.textContent).toContain('OCR template')
})
})
@@ -0,0 +1,31 @@
import { Component, OnInit, inject, signal } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { takeUntil } from 'rxjs'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { SelectComponent } from '../../input/select/select.component'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-zone-ocr-confirm-dialog',
templateUrl: './zone-ocr-confirm-dialog.component.html',
imports: [FormsModule, SelectComponent],
})
export class ZoneOcrConfirmDialogComponent
extends ConfirmDialogComponent
implements OnInit
{
private readonly ocrTemplateService = inject(OcrTemplateService)
public readonly templates = signal<OcrTemplate[]>([])
/** The template to run, or null for every template that matches. */
template: number | null = null
ngOnInit() {
this.ocrTemplateService
.listAll()
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((r) => this.templates.set(r.results))
}
}
@@ -40,7 +40,7 @@
(change)="onChange(selectedDocumentIDs)"> (change)="onChange(selectedDocumentIDs)">
<ng-template ng-label-tmp let-document="item"> <ng-template ng-label-tmp let-document="item">
<div class="d-flex align-items-center"> <div class="d-flex align-items-center">
@if (!disabled) { @if (!disabled && multiple) {
<button class="btn p-0 lh-1" (click)="unselect(document)" (mousedown)="$event.stopImmediatePropagation()" type="button" title="Remove link" i18n-title><pngx-bs-icon name="x"></pngx-bs-icon></button> <button class="btn p-0 lh-1" (click)="unselect(document)" (mousedown)="$event.stopImmediatePropagation()" type="button" title="Remove link" i18n-title><pngx-bs-icon name="x"></pngx-bs-icon></button>
} }
@if (document.title) { @if (document.title) {
@@ -81,6 +81,9 @@ export class DocumentLinkComponent
@Input() @Input()
placeholder: string = $localize`Search for documents` placeholder: string = $localize`Search for documents`
@Input()
multiple: boolean = true
/** /**
* Parent for ng-select dropdown, needed to prevent close on click. * Parent for ng-select dropdown, needed to prevent close on click.
*/ */
@@ -67,6 +67,22 @@
<pngx-bs-icon name="pencil" class="me-2"></pngx-bs-icon><ng-container i18n>PDF Editor</ng-container> <pngx-bs-icon name="pencil" class="me-2"></pngx-bs-icon><ng-container i18n>PDF Editor</ng-container>
</button> </button>
<div class="d-flex" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.OcrTemplate }">
<button ngbDropdownItem (click)="runZoneOcr()" [disabled]="!userCanEdit || networkActive()">
<pngx-bs-icon width="1em" height="1em" name="bounding-box-circles" class="me-2"></pngx-bs-icon><span i18n>Run Zone OCR</span>
</button>
<button
ngbDropdownItem
class="w-auto flex-shrink-0 px-2"
(click)="runZoneOcrWithOptions()"
[disabled]="!userCanEdit || networkActive()"
title="Choose OCR template"
i18n-title
>
<pngx-bs-icon width="1em" height="1em" name="sliders2"></pngx-bs-icon>
</button>
</div>
@if (userIsOwner && (requiresPassword || password)) { @if (userIsOwner && (requiresPassword || password)) {
<button ngbDropdownItem (click)="removePassword()" [disabled]="!password"> <button ngbDropdownItem (click)="removePassword()" [disabled]="!password">
<pngx-bs-icon name="unlock" class="me-2"></pngx-bs-icon><ng-container i18n>Remove Password</ng-container> <pngx-bs-icon name="unlock" class="me-2"></pngx-bs-icon><ng-container i18n>Remove Password</ng-container>
@@ -96,7 +112,7 @@
<div class="row"> <div class="row">
<div class="col-md-6 col-xl-5 mb-4"> <div class="col-md-6 col-xl-5 mb-4">
<form [formGroup]='documentForm' (ngSubmit)="save()"> <form [formGroup]='documentForm' (ngSubmit)="save()" [attr.inert]="zoneOcrRunning() ? '' : null" [attr.aria-busy]="zoneOcrRunning()">
<div class="btn-toolbar mb-1 pb-3 gap-2 row-gap-2 border-bottom"> <div class="btn-toolbar mb-1 pb-3 gap-2 row-gap-2 border-bottom">
<div class="btn-group"> <div class="btn-group">
@@ -41,6 +41,7 @@ import {
FILTER_STORAGE_PATH, FILTER_STORAGE_PATH,
} from 'src/app/data/filter-rule-type' } from 'src/app/data/filter-rule-type'
import { StoragePath } from 'src/app/data/storage-path' import { StoragePath } from 'src/app/data/storage-path'
import { PaperlessTaskStatus } from 'src/app/data/paperless-task'
import { Tag } from 'src/app/data/tag' import { Tag } from 'src/app/data/tag'
import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings' import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings'
import { PermissionsGuard } from 'src/app/guards/permissions.guard' import { PermissionsGuard } from 'src/app/guards/permissions.guard'
@@ -59,6 +60,7 @@ import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service' import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service' import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service' import { ToastService } from 'src/app/services/toast.service'
import { TaskNotificationService } from 'src/app/services/task-notification.service'
import { TrashService } from 'src/app/services/trash.service' import { TrashService } from 'src/app/services/trash.service'
import { WebsocketStatusService } from 'src/app/services/websocket-status.service' import { WebsocketStatusService } from 'src/app/services/websocket-status.service'
import { environment } from 'src/environments/environment' import { environment } from 'src/environments/environment'
@@ -1965,6 +1967,282 @@ describe('DocumentDetailComponent', () => {
expect(toastSpy).toHaveBeenCalledWith('Document reloaded.') expect(toastSpy).toHaveBeenCalledWith('Document reloaded.')
}) })
it('should queue a chosen zone OCR template on the selected version', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, versions: docVersions })
component.selectedVersionId.set(10)
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(of({ result: 'OK', task_id: 'zone-ocr-task' }))
const loadSpy = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
const notifySpy = jest.spyOn(component.docChangeNotifier, 'next')
const queued = jest.spyOn(
TestBed.inject(TaskNotificationService),
'showQueued'
)
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcrWithOptions()
const dialog = modalSpy.mock.results[0].value.componentInstance
expect(dialog.message).toBe(
'Fields filled by the template will be overwritten.'
)
dialog.template = 3
dialog.confirmClicked.emit()
expect(documentService.runZoneOcr).toHaveBeenCalledWith(doc.id, 3, 10)
expect(queued).toHaveBeenCalledWith('zone-ocr-task', 'Zone OCR queued.', {
content: 'Zone OCR completed.',
})
expect(notifySpy).not.toHaveBeenCalled()
expect(loadSpy).not.toHaveBeenCalled()
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
})
it('should queue matching templates on the latest version without a dialog', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, versions: docVersions })
component.selectedVersionId.set(12)
jest.spyOn(component as any, 'loadDocument').mockImplementation(() => {})
const runSpy = jest.spyOn(documentService, 'runZoneOcr')
const queued = jest.spyOn(
TestBed.inject(TaskNotificationService),
'showQueued'
)
const modalSpy = jest.spyOn(modalService, 'open')
runSpy.mockReturnValueOnce(
of({ result: 'OK', task_id: 'matching-zone-ocr' })
)
component.runZoneOcr()
expect(modalSpy).not.toHaveBeenCalled()
expect(runSpy).toHaveBeenCalledWith(doc.id, null, null)
expect(queued).toHaveBeenCalledWith(
'matching-zone-ocr',
'Zone OCR queued.',
{ content: 'Zone OCR completed.' }
)
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
})
it('should discard confirmed local edits after OCR is queued', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
component.documentForm.patchValue({ title: 'Unsaved title' })
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(of({ result: 'OK', task_id: 'zone-ocr-task' }))
const modalSpy = jest.spyOn(modalService, 'open')
const load = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
component.runZoneOcr()
modalSpy.mock.results[0].value.componentInstance.confirm()
expect(load).toHaveBeenCalledWith(doc.id, true, expect.any(Function))
load.mock.calls[0][2]()
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
})
it('should handle task completion before the queue response', () => {
jest.useFakeTimers()
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: [] })
const pending = new Subject<any>()
jest.spyOn(documentService, 'runZoneOcr').mockReturnValue(pending)
const notifications = TestBed.inject(TaskNotificationService)
const toast = jest.spyOn(toastService, 'show')
component.runZoneOcr()
notifications.complete({
task_id: 'fast-zone-ocr',
operation: 'run_zone_ocr',
status: PaperlessTaskStatus.Success,
})
pending.next({ result: 'OK', task_id: 'fast-zone-ocr' })
expect(toast).toHaveBeenLastCalledWith(
expect.objectContaining({
id: 'task-fast-zone-ocr',
content: 'Zone OCR completed.',
})
)
expect(component.networkActive()).toBe(false)
pending.complete()
})
it('should refresh queued OCR changes through document update events', () => {
initNormally()
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(of({ result: 'OK', task_id: 'zone-ocr-task' }))
const load = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
component.runZoneOcr()
expect(load).not.toHaveBeenCalled()
expect(component.networkActive()).toBe(false)
TestBed.inject(WebsocketStatusService).onDocumentUpdated().next({
document_id: doc.id,
modified: '2026-02-17T00:00:00Z',
})
expect(load).toHaveBeenCalledWith(doc.id, true)
})
it('should protect edits made after OCR was queued from its document update', () => {
initNormally()
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(of({ result: 'OK', task_id: 'zone-ocr-task' }))
component.runZoneOcr()
component.documentForm.patchValue({ title: 'New local edit' })
jest.spyOn(openDocumentsService, 'isDirty').mockReturnValue(true)
const conflict = jest
.spyOn(component as any, 'showIncomingUpdateModal')
.mockImplementation(() => {})
const load = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
TestBed.inject(WebsocketStatusService).onDocumentUpdated().next({
document_id: doc.id,
modified: '2026-02-17T00:00:00Z',
})
expect(conflict).toHaveBeenCalled()
expect(load).not.toHaveBeenCalled()
expect(component.documentForm.get('title').value).toBe('New local edit')
})
it('should confirm before running zone OCR over unsaved changes', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
component.documentForm.patchValue({ title: 'Unsaved title' })
const runSpy = jest.spyOn(documentService, 'runZoneOcr')
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcr()
const modal = modalSpy.mock.results[0].value
expect(modal.componentInstance.message).toBe(
'Running Zone OCR will discard them.'
)
expect(runSpy).not.toHaveBeenCalled()
modal.componentInstance.cancel()
expect(runSpy).not.toHaveBeenCalled()
expect(component.documentForm.get('title').value).toBe('Unsaved title')
expect(component.zoneOcrRunning()).toBe(false)
})
it('should warn about unsaved changes in the zone OCR options dialog', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
const runSpy = jest.spyOn(documentService, 'runZoneOcr')
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcrWithOptions()
const dialog = modalSpy.mock.results[0].value.componentInstance
expect(dialog.message).toBe('Running Zone OCR will discard them.')
expect(dialog.btnClass).toBe('btn-warning')
dialog.cancel()
expect(runSpy).not.toHaveBeenCalled()
component.networkActive.set(true)
component.runZoneOcrWithOptions()
expect(modalSpy).toHaveBeenCalledTimes(1)
})
it('should not allow saving or a second zone OCR request while queueing', () => {
component.documentId.set(doc.id)
component.document.set({
...doc,
versions: docVersions,
__changedFields: ['title'],
})
component.selectedVersionId.set(doc.id)
const pending = new Subject<any>()
const runSpy = jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(pending)
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcr()
modalSpy.mock.results[0].value.componentInstance.confirmClicked.emit()
expect(runSpy).toHaveBeenCalledWith(doc.id, null, doc.id)
expect(component.zoneOcrRunning()).toBe(true)
expect(component.networkActive()).toBe(true)
const saveSpy = jest.spyOn(documentService, 'patch')
component.save()
expect(saveSpy).not.toHaveBeenCalled()
component.runZoneOcr()
expect(modalSpy).toHaveBeenCalledTimes(1)
expect(runSpy).toHaveBeenCalledTimes(1)
pending.error(new Error('failed'))
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
expect(component.document().__changedFields).toEqual(['title'])
})
it('should not reload after zone OCR if another document is open', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: [] })
const pending = new Subject<any>()
jest.spyOn(documentService, 'runZoneOcr').mockReturnValue(pending)
const loadSpy = jest.spyOn(component as any, 'loadDocument')
component.runZoneOcr()
component.documentId.set(99)
pending.next({ result: 'OK', task_id: 'zone-ocr-task' })
expect(loadSpy).not.toHaveBeenCalled()
expect(component.zoneOcrRunning()).toBe(false)
pending.complete()
})
it('should keep unsaved changes when zone OCR is skipped', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
component.documentForm.patchValue({ title: 'Unsaved title' })
const modalSpy = jest.spyOn(modalService, 'open')
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(of({ result: 'skipped', reason: 'no_template' }))
const loadSpy = jest.spyOn(component as any, 'loadDocument')
const infoSpy = jest.spyOn(toastService, 'showInfo')
component.runZoneOcr()
modalSpy.mock.results[0].value.componentInstance.confirm()
expect(loadSpy).not.toHaveBeenCalled()
expect(component.documentForm.get('title').value).toBe('Unsaved title')
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
expect(infoSpy).toHaveBeenCalledWith(
'No OCR template applies to this document.'
)
})
it('should show an error when zone OCR fails', () => {
component.document.set(doc)
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(throwError(() => new Error('nope')))
const loadSpy = jest.spyOn(component as any, 'loadDocument')
const errorSpy = jest.spyOn(toastService, 'showError')
component.runZoneOcr()
expect(errorSpy).toHaveBeenCalledWith(
'Error running zone OCR.',
expect.any(Error)
)
expect(loadSpy).not.toHaveBeenCalled()
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
})
it('should change preview element by render type', () => { it('should change preview element by render type', () => {
initNormally() initNormally()
component.document.update((document) => ({ component.document.update((document) => ({
@@ -109,6 +109,7 @@ import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component' import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component' import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RunWorkflowConfirmDialogComponent } from '../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component' import { RunWorkflowConfirmDialogComponent } from '../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component'
import { ZoneOcrConfirmDialogComponent } from '../common/confirm-dialog/zone-ocr-confirm-dialog/zone-ocr-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component' import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component' import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component' import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -273,6 +274,7 @@ export class DocumentDetailComponent
readonly error = signal<any>(undefined) readonly error = signal<any>(undefined)
readonly networkActive = signal(false) readonly networkActive = signal(false)
readonly zoneOcrRunning = signal(false)
readonly documentId = signal<number>(undefined) readonly documentId = signal<number>(undefined)
readonly document = signal<Document>(undefined) readonly document = signal<Document>(undefined)
@@ -551,7 +553,11 @@ export class DocumentDetailComponent
) )
} }
private loadDocument(documentId: number, forceRemote: boolean = false): void { private loadDocument(
documentId: number,
forceRemote: boolean = false,
onComplete?: () => void
): void {
let redirectedToRoot = false let redirectedToRoot = false
this.closeIncomingUpdateModal() this.closeIncomingUpdateModal()
this.pendingIncomingUpdate = null this.pendingIncomingUpdate = null
@@ -608,7 +614,8 @@ export class DocumentDetailComponent
}), }),
first(), first(),
takeUntil(this.unsubscribeNotifier), takeUntil(this.unsubscribeNotifier),
takeUntil(this.docChangeNotifier) takeUntil(this.docChangeNotifier),
finalize(() => onComplete?.())
) )
.subscribe({ .subscribe({
next: (doc) => { next: (doc) => {
@@ -1356,6 +1363,7 @@ export class DocumentDetailComponent
} }
save(close: boolean = false, savedCallback: () => void = null) { save(close: boolean = false, savedCallback: () => void = null) {
if (this.zoneOcrRunning()) return
this.networkActive.set(true) this.networkActive.set(true)
;(document.activeElement as HTMLElement)?.dispatchEvent(new Event('change')) ;(document.activeElement as HTMLElement)?.dispatchEvent(new Event('change'))
this.documentsService this.documentsService
@@ -1622,6 +1630,108 @@ export class DocumentDetailComponent
}) })
} }
/** Runs every template that matches, confirming only over unsaved changes. */
runZoneOcr() {
if (this.networkActive()) return
const documentId = this.documentId()
const versionId = this.getSelectedNonLatestVersionId()
if (!this.hasLocalEdits(this.document())) {
this.executeZoneOcr(documentId, null, versionId)
return
}
const modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Run Zone OCR`
modal.componentInstance.messageBold = $localize`You have unsaved changes.`
modal.componentInstance.message = $localize`Running Zone OCR will discard them.`
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Run Zone OCR`
modal.componentInstance.confirmClicked
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.close()
if (this.documentId() === documentId && !this.networkActive()) {
this.executeZoneOcr(documentId, null, versionId)
}
})
}
/** Opens the dialog to pick which template to run. */
runZoneOcrWithOptions() {
if (this.networkActive()) return
const documentId = this.documentId()
const versionId = this.getSelectedNonLatestVersionId()
const modal = this.modalService.open(ZoneOcrConfirmDialogComponent, {
backdrop: 'static',
})
const dialog = modal.componentInstance as ZoneOcrConfirmDialogComponent
dialog.title = $localize`Run Zone OCR`
dialog.btnCaption = $localize`Run Zone OCR`
if (this.hasLocalEdits(this.document())) {
dialog.messageBold = $localize`You have unsaved changes.`
dialog.message = $localize`Running Zone OCR will discard them.`
dialog.btnClass = 'btn-warning'
} else {
dialog.message = $localize`Fields filled by the template will be overwritten.`
}
dialog.confirmClicked
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.close()
if (this.documentId() === documentId && !this.networkActive()) {
this.executeZoneOcr(documentId, dialog.template, versionId)
}
})
}
private executeZoneOcr(
documentId: number,
templateId: number | null,
versionId: number | null
) {
this.zoneOcrRunning.set(true)
this.networkActive.set(true)
this.documentsService
.runZoneOcr(documentId, templateId, versionId)
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe({
next: (response) => {
if (response.result === 'skipped') {
this.finishZoneOcr()
this.toastService.showInfo(
$localize`No OCR template applies to this document.`
)
return
}
this.taskNotifications.showQueued(
response.task_id,
$localize`Zone OCR queued.`,
{ content: $localize`Zone OCR completed.` }
)
if (
this.documentId() === documentId &&
this.hasLocalEdits(this.document())
) {
this.docChangeNotifier.next(documentId)
this.loadDocument(documentId, true, () => this.finishZoneOcr())
} else {
this.finishZoneOcr()
}
},
error: (error) => {
this.finishZoneOcr()
this.toastService.showError($localize`Error running zone OCR.`, error)
},
})
}
private finishZoneOcr() {
this.zoneOcrRunning.set(false)
this.networkActive.set(false)
this.flushPendingIncomingUpdate()
}
private getSelectedNonLatestVersionId(): number | null { private getSelectedNonLatestVersionId(): number | null {
const versions = this.document()?.versions ?? [] const versions = this.document()?.versions ?? []
if (!versions.length || !this.selectedVersionId()) { if (!versions.length || !this.selectedVersionId()) {
@@ -103,6 +103,11 @@
<pngx-bs-icon name="play-circle" class="me-2"></pngx-bs-icon><ng-container i18n>Run workflow</ng-container> <pngx-bs-icon name="play-circle" class="me-2"></pngx-bs-icon><ng-container i18n>Run workflow</ng-container>
</button> </button>
} }
@if (permissionService.currentUserCan(PermissionAction.View, PermissionType.OcrTemplate)) {
<button ngbDropdownItem (click)="runZoneOcrSelected()" [disabled]="!userCanEditAll">
<pngx-bs-icon name="bounding-box-circles" class="me-2"></pngx-bs-icon><ng-container i18n>Run Zone OCR</ng-container>
</button>
}
</div> </div>
</div> </div>
</div> </div>
@@ -1386,6 +1386,70 @@ describe('BulkEditorComponent', () => {
) // listAllFilteredIds ) // listAllFilteredIds
}) })
it('should support run zone ocr', () => {
const queued = jest
.spyOn(TestBed.inject(TaskNotificationService), 'showQueued')
.mockImplementation(() => {})
const reload = jest.spyOn(documentListViewService, 'reload')
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
.spyOn(documentListViewService, 'documents', 'get')
.mockReturnValue([{ id: 3 }, { id: 4 }])
jest
.spyOn(documentListViewService, 'selected', 'get')
.mockReturnValue(new Set([3, 4]))
jest
.spyOn(permissionsService, 'currentUserHasObjectPermissions')
.mockReturnValue(true)
fixture.detectChanges()
component.runZoneOcrSelected()
expect(modal).not.toBeUndefined()
modal.componentInstance.confirm()
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/run_zone_ocr/`
)
req.flush({ result: 'OK', task_id: 'zone-ocr-1' })
expect(req.request.body).toEqual({
documents: [3, 4],
from_webui: true,
})
expect(queued).toHaveBeenLastCalledWith(
'zone-ocr-1',
"Bulk operation 'Run zone OCR' queued.",
{ content: "Bulk operation 'Run zone OCR' completed." }
)
expect(reload).not.toHaveBeenCalled()
component.runZoneOcrSelected()
modal.componentInstance.template = 3
modal.componentInstance.confirm()
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/run_zone_ocr/`
)
req.flush({ result: 'OK', task_id: 'zone-ocr-2' })
expect(req.request.body).toEqual({
documents: [3, 4],
template: 3,
from_webui: true,
})
expect(queued).toHaveBeenLastCalledWith(
'zone-ocr-2',
"Bulk operation 'Run zone OCR' queued.",
{ content: "Bulk operation 'Run zone OCR' completed." }
)
httpTestingController.match(
`${environment.apiBaseUrl}ocr_templates/?page=1&page_size=100000`
) // template list
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
})
it('should support rotate', () => { it('should support rotate', () => {
const toast = jest.spyOn(toastService, 'show') const toast = jest.spyOn(toastService, 'show')
let modal: NgbModalRef let modal: NgbModalRef
@@ -64,6 +64,7 @@ import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-c
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component' import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component' import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { RunWorkflowConfirmDialogComponent } from '../../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component' import { RunWorkflowConfirmDialogComponent } from '../../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component'
import { ZoneOcrConfirmDialogComponent } from '../../common/confirm-dialog/zone-ocr-confirm-dialog/zone-ocr-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component' import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component' import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component' import { DocumentTypeEditDialogComponent } from '../../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -1025,6 +1026,31 @@ export class BulkEditorComponent
}) })
} }
runZoneOcrSelected() {
const modal = this.modalService.open(ZoneOcrConfirmDialogComponent, {
backdrop: 'static',
})
const dialog = modal.componentInstance as ZoneOcrConfirmDialogComponent
dialog.title = $localize`Run zone OCR confirm`
dialog.messageBold = $localize`This operation will run zone OCR on ${this.getSelectionSize()} selected document(s).`
dialog.message = $localize`Fields filled by the template will be overwritten.`
dialog.btnClass = 'btn-warning'
dialog.btnCaption = $localize`Proceed`
dialog.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
dialog.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
'run_zone_ocr',
this.documentService.runZoneOcrDocuments(
this.getSelectionQuery(),
dialog.template
)
)
})
}
setPermissions() { setPermissions() {
let modal = this.modalService.open(PermissionsDialogComponent, { let modal = this.modalService.open(PermissionsDialogComponent, {
backdrop: 'static', backdrop: 'static',
@@ -0,0 +1,36 @@
@if (zones.length === 0) {
<p class="text-muted" i18n>No zones yet. Drag anywhere on the sample document to create one.</p>
} @else {
<div class="list-group">
@for (zone of zones; track $index; let i = $index) {
<div class="list-group-item list-group-item-action d-flex align-items-center gap-3 position-relative">
<span class="p-2 rounded flex-shrink-0" [style.background-color]="zoneColor(i)" aria-hidden="true"></span>
<div class="flex-grow-1 text-truncate">
<button type="button" class="btn btn-link p-0 fw-semibold text-reset text-decoration-none text-truncate mw-100 stretched-link" (click)="zoneSelected.emit(i)">{{ zone.name }}</button>
@if (!hasField(zone)) {
<div class="small text-warning text-truncate">
<pngx-bs-icon name="exclamation-triangle" class="me-1" aria-hidden="true"></pngx-bs-icon><span i18n>No field selected</span>
</div>
} @else {
<div class="small text-muted text-truncate">
{{ getZoneTargetName(zone) }}
@if (isCombined(zone)) {
<pngx-bs-icon name="braces" class="ms-1" title="Combined with other zones" i18n-title></pngx-bs-icon>
}
</div>
}
</div>
<small class="text-muted text-nowrap">
@if (zone.page === -1) {
<ng-container i18n>Last page</ng-container>
} @else {
<ng-container i18n>Page {{ zone.page ?? 1 }}</ng-container>
}
</small>
<button type="button" class="btn btn-sm btn-link text-danger position-relative z-2" (click)="zoneRemoved.emit(i)" title="Delete zone" i18n-title aria-label="Delete zone" i18n-aria-label>
<pngx-bs-icon name="trash"></pngx-bs-icon>
</button>
</div>
}
</div>
}
@@ -0,0 +1,115 @@
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { CustomField } from 'src/app/data/custom-field'
import {
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTransform,
} from 'src/app/data/ocr-template'
import { OcrTemplateEditorZoneListComponent } from './ocr-template-editor-zone-list.component'
function zone(overrides: Partial<OcrTemplateZone> = {}): OcrTemplateZone {
return {
name: 'Zone 1',
target: OcrZoneTarget.CustomField,
custom_field: 7,
x: 10,
y: 20,
width: 30,
height: 40,
page: 1,
ocr_language: 'eng',
transform: OcrZoneTransform.Strip,
validation_regex: '',
order: 0,
...overrides,
}
}
describe('OcrTemplateEditorZoneListComponent', () => {
let fixture: ComponentFixture<OcrTemplateEditorZoneListComponent>
let component: OcrTemplateEditorZoneListComponent
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [OcrTemplateEditorZoneListComponent, BootstrapIconComponent],
}).compileComponents()
fixture = TestBed.createComponent(OcrTemplateEditorZoneListComponent)
component = fixture.componentInstance
})
it('shows empty state when no zones are defined', () => {
fixture.detectChanges()
expect(fixture.nativeElement.textContent).toContain('No zones yet')
})
it('renders zone name, target, and page', () => {
component.zones = [zone(), zone({ name: 'Zone 2', page: -1 })]
component.customFields = [{ id: 7, name: 'Invoice Number' } as CustomField]
fixture.detectChanges()
const text = fixture.nativeElement.textContent
expect(text).toContain('Zone 1')
expect(text).toContain('Invoice Number')
expect(text).toContain('Page 1')
expect(text).toContain('Last page')
})
it('marks zones that share a field', () => {
component.zones = [
zone(),
zone({ name: 'Zone 2' }),
zone({ name: 'Zone 3', custom_field: 8 }),
]
fixture.detectChanges()
expect(component.isCombined(component.zones[0])).toBe(true)
expect(component.isCombined(component.zones[2])).toBe(false)
expect(
fixture.nativeElement.querySelectorAll(
'[title="Combined with other zones"]'
)
).toHaveLength(2)
})
it('warns only for zones without a field and clears the warning when assigned', () => {
component.zones = [
zone({ custom_field: null }),
zone({ target: OcrZoneTarget.Title, custom_field: null }),
zone(),
]
fixture.detectChanges()
const warnings = () =>
fixture.nativeElement.querySelectorAll('.text-warning')
expect(warnings()).toHaveLength(1)
expect(warnings()[0].textContent).toContain('No field selected')
component.zones[0].custom_field = 7
fixture.componentRef.setInput('zones', [...component.zones])
fixture.detectChanges()
expect(warnings()).toHaveLength(0)
})
it('emits select and remove events', () => {
component.zones = [zone()]
const selectSpy = jest.spyOn(component.zoneSelected, 'emit')
const removeSpy = jest.spyOn(component.zoneRemoved, 'emit')
fixture.detectChanges()
const buttons = fixture.nativeElement.querySelectorAll('button')
buttons[0].click()
buttons[1].click()
expect(selectSpy).toHaveBeenCalledWith(0)
expect(removeSpy).toHaveBeenCalledWith(0)
})
it('does not label an unknown built-in target', () => {
expect(
component.getZoneTargetName(zone({ target: 'unknown' as OcrZoneTarget }))
).toBe('')
})
})
@@ -0,0 +1,50 @@
import { Component, EventEmitter, Input, Output } from '@angular/core'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { CustomField } from 'src/app/data/custom-field'
import {
getZoneFieldKey,
OCR_BUILTIN_TARGETS,
OcrTemplateZone,
OcrZoneTarget,
} from 'src/app/data/ocr-template'
import { getZoneColor } from '../zone-geometry'
@Component({
selector: 'pngx-ocr-template-zone-list',
imports: [BootstrapIconComponent],
templateUrl: './ocr-template-editor-zone-list.component.html',
})
export class OcrTemplateEditorZoneListComponent {
@Input() zones: OcrTemplateZone[] = []
@Input() customFields: CustomField[] = []
@Output() public readonly zoneSelected = new EventEmitter<number>()
@Output() public readonly zoneRemoved = new EventEmitter<number>()
public readonly zoneColor = getZoneColor
hasField(zone: OcrTemplateZone): boolean {
return getZoneFieldKey(zone) !== null
}
isCombined(zone: OcrTemplateZone): boolean {
const key = getZoneFieldKey(zone)
return (
key !== null &&
this.zones.filter((z) => getZoneFieldKey(z) === key).length > 1
)
}
getZoneTargetName(zone: OcrTemplateZone): string {
if (zone.target === OcrZoneTarget.CustomField) {
return (
this.customFields.find((field) => field.id === zone.custom_field)
?.name ?? ''
)
}
return (
OCR_BUILTIN_TARGETS.find((target) => target.id === zone.target)?.name ??
''
)
}
}
@@ -0,0 +1,315 @@
<div class="modal-header">
<h4 class="modal-title" id="ocr-template-editor-title">{{ dialogTitle }}</h4>
@if (template.id) {
<span class="badge bg-primary text-primary-text-contrast ms-2">ID: {{ template.id }}</span>
}
<button type="button" class="btn-close" (click)="activeModal.dismiss()" [disabled]="saving()" aria-label="Close" i18n-aria-label></button>
</div>
<div class="modal-body d-flex flex-column">
<div class="row gx-3 editor-layout flex-grow-1 flex-lg-nowrap" [attr.inert]="saving() ? '' : null" [attr.aria-busy]="saving()">
<div class="col-lg-4 mb-3 mb-lg-0 overflow-auto">
<ul ngbNav #nav="ngbNav" [(activeId)]="activeTab" class="nav-underline flex-nowrap flex-md-wrap overflow-auto">
<li ngbNavItem="template">
<a ngbNavLink i18n>Template</a>
<ng-template ngbNavContent>
<div class="row mb-3">
<div class="col-9">
<pngx-input-text [(ngModel)]="template.name" title="Template name" i18n-title></pngx-input-text>
</div>
<div class="col-3">
<pngx-input-switch [(ngModel)]="template.enabled" title="Enabled" i18n-title></pngx-input-switch>
</div>
</div>
<pngx-input-select [(ngModel)]="template.matching_algorithm" [items]="matchingAlgorithms" title="Matching algorithm" i18n-title></pngx-input-select>
@if (patternRequired) {
<pngx-input-text [(ngModel)]="template.match" title="Matching pattern" i18n-title></pngx-input-text>
<pngx-input-check [(ngModel)]="template.is_insensitive" title="Case insensitive" i18n-title></pngx-input-check>
}
<div *pngxIfOwner="object">
<hr />
<pngx-permissions-form [users]="users()" [accordion]="true" [(ngModel)]="permissionsForm"></pngx-permissions-form>
</div>
</ng-template>
</li>
<li ngbNavItem="zones">
<a ngbNavLink><ng-container i18n>Zones</ng-container> <span class="badge bg-primary ms-2">{{ template.zones.length }}</span></a>
<ng-template ngbNavContent>
@if (selectedZone; as zone) {
<div class="d-flex align-items-center gap-2 mb-3">
<button type="button" class="btn btn-sm btn-link px-0 me-auto" (click)="deselectZone()">
<pngx-bs-icon name="arrow-left" class="me-1"></pngx-bs-icon><ng-container i18n>All zones</ng-container>
</button>
<button type="button" class="btn btn-sm btn-outline-primary" (click)="testZone()" [disabled]="!previewDocId || zoneTesting">
@if (zoneTesting) {
<span class="spinner-border spinner-border-sm me-1" aria-hidden="true"></span>
} @else {
<pngx-bs-icon name="play-fill" class="me-1"></pngx-bs-icon>
}
<ng-container i18n>Test</ng-container>
</button>
<button type="button" class="btn btn-sm btn-outline-danger" (click)="removeZone(selectedZoneIndex)" title="Delete zone" i18n-title aria-label="Delete zone" i18n-aria-label>
<pngx-bs-icon name="trash"></pngx-bs-icon>
</button>
</div>
@if (zoneTestResult; as result) {
@if (result.error) {
<div class="alert alert-warning py-2 mb-3 small" role="alert">{{ result.error }}</div>
} @else {
<dl class="row small border rounded mx-0 py-2 mb-3">
<dt class="col-4" i18n>OCR text</dt>
<dd class="col-8">
@if (result.raw_text) {
<code>{{ result.raw_text }}</code>
} @else {
<span class="text-muted" i18n>Nothing detected</span>
}
</dd>
<dt class="col-4" i18n>Value</dt>
<dd class="col-8" [class.mb-0]="!result.regex">
@if (result.value) {
<code>{{ result.value }}</code>
} @else {
<span class="text-muted" i18n>Empty</span>
}
</dd>
@if (result.regex) {
<dt class="col-4" i18n>Validation</dt>
<dd class="col-8 mb-0">
@if (result.regex_match) {
<span class="badge bg-success" i18n>Regex matches</span>
} @else {
<span class="badge bg-danger" i18n>Regex does not match</span>
}
</dd>
}
</dl>
}
}
<pngx-input-text [ngModel]="zone.name" (ngModelChange)="renameZone(zone, $event)" title="Name" i18n-title></pngx-input-text>
<pngx-input-select
[items]="fieldItems()"
[ngModel]="zoneFieldValue(zone)"
(ngModelChange)="setZoneField(zone, $event)"
(createNew)="createField($event)"
title="Field"
i18n-title
placeholder="Choose a field"
i18n-placeholder
hint="Where the extracted value is written."
i18n-hint
></pngx-input-select>
@if (isFieldShared(zone)) {
<pngx-input-text
[ngModel]="getCombineFormat(zone)"
(ngModelChange)="setCombineFormat(zone, $event)"
[placeholder]="defaultCombineFormat(zone)"
title="Combined value"
i18n-title
hint="Several zones write to this field. Set the format or leave empty to join them with spaces."
i18n-hint
></pngx-input-text>
<div class="d-flex flex-wrap gap-1 mt-n2 mb-3">
@for (z of zonesForField(zone); track z) {
<button type="button" class="btn btn-sm btn-outline-secondary font-monospace" (click)="insertCombineToken(zone, z)">{{ '{' + z.name + '}' }}</button>
}
</div>
}
<div class="row align-items-end">
<div class="col">
@if (!isLastPageZone(zone)) {
<pngx-input-number [(ngModel)]="zone.page" [showAdd]="false" title="Page" i18n-title></pngx-input-number>
}
</div>
<div class="col-auto">
<pngx-input-switch [ngModel]="isLastPageZone(zone)" (ngModelChange)="setLastPageZone(zone, $event)" [horizontal]="true" title="Last page" i18n-title></pngx-input-switch>
</div>
</div>
<h6 class="border-top pt-3 mt-1 mb-3" i18n>Processing</h6>
<pngx-input-select
[items]="languageOptions"
[multiple]="true"
[ngModel]="ocrLanguageArray(zone)"
(ngModelChange)="setOcrLanguages(zone, $event)"
title="OCR language"
i18n-title
placeholder="Select languages"
i18n-placeholder
></pngx-input-select>
<pngx-input-select [items]="transformOptions" [(ngModel)]="zone.transform" title="Transform" i18n-title></pngx-input-select>
@if (zone.transform === OcrZoneTransform.Date) {
<pngx-input-text
[(ngModel)]="zone.date_format"
title="Date format"
i18n-title
placeholder="Auto-detect"
i18n-placeholder
hint="Python date codes, e.g. %d.%m.%Y. Leave empty to detect the format automatically."
i18n-hint
></pngx-input-text>
}
<pngx-input-text [(ngModel)]="zone.validation_regex" title="Validation regex" i18n-title placeholder="\d{2}\.\d{2}\.\d{4}" hint="The value must match this pattern to be applied." i18n-hint></pngx-input-text>
} @else {
<pngx-ocr-template-zone-list
[zones]="template.zones"
[customFields]="customFields()"
(zoneSelected)="selectZone($event)"
(zoneRemoved)="removeZone($event)"
></pngx-ocr-template-zone-list>
}
</ng-template>
</li>
</ul>
<div [ngbNavOutlet]="nav" class="mt-3"></div>
</div>
<div class="col-lg-8 d-flex flex-column">
<div class="d-flex flex-wrap gap-2 mb-2 flex-shrink-0">
<pngx-input-document-link
class="flex-grow-1 paperless-input-select"
[ngModel]="sampleDocumentIds"
(ngModelChange)="setSampleDocument($event)"
[minimal]="true"
[multiple]="false"
placeholder="Search for a sample document"
i18n-placeholder
></pngx-input-document-link>
@if (previewDocId) {
<div class="input-group w-auto flex-nowrap">
<input class="form-control flex-grow-0 preview-page-input" type="number" min="1" [max]="previewPageCount()" [(ngModel)]="previewPageDisplay" [disabled]="!pageLoaded()" aria-label="Page" i18n-aria-label />
@if (previewPageCount(); as pageCount) {
<span class="input-group-text" i18n>of {{ pageCount }}</span>
}
</div>
<fieldset class="input-group w-auto flex-nowrap" aria-label="Zoom" i18n-aria-label>
<button type="button" class="btn btn-outline-secondary" aria-label="Zoom out" i18n-aria-label [disabled]="!pageLoaded() || zoom <= 0.5" (click)="zoomOut()"><pngx-bs-icon name="dash"></pngx-bs-icon></button>
<button type="button" class="btn btn-outline-secondary preview-zoom" title="Reset zoom" i18n-title [disabled]="!pageLoaded()" (click)="resetZoom()">{{ zoom | percent: '1.0-0' }}</button>
<button type="button" class="btn btn-outline-secondary" aria-label="Zoom in" i18n-aria-label [disabled]="!pageLoaded() || zoom >= 4" (click)="zoomIn()"><pngx-bs-icon name="plus"></pngx-bs-icon></button>
</fieldset>
}
</div>
<div class="flex-grow-1 overflow-hidden">
<div class="h-100 overflow-auto border" [attr.aria-busy]="previewDocId && !pageLoaded()">
@if (pageFailed()) {
<div class="py-5 px-3 text-center" role="alert">
<pngx-bs-icon name="exclamation-triangle" width="32" height="32" class="text-warning"></pngx-bs-icon>
<p class="my-3" i18n>Could not load this page.</p>
<button type="button" class="btn btn-sm btn-outline-secondary" (click)="setSampleDocument(sampleDocumentIds)"><pngx-bs-icon name="arrow-clockwise" class="me-1"></pngx-bs-icon><span i18n>Retry</span></button>
</div>
} @else if (previewDocId) {
<div class="position-relative" [class.d-none]="!pageLoaded()" [style.width.%]="zoom * 100">
<img
#pageImage
[src]="pageImageUrl"
(load)="onImageLoad()"
(error)="onImageError()"
alt="Sample document page"
i18n-alt
class="d-block w-100"
crossorigin="use-credentials"
/>
@if (pageLoaded()) {
<svg
#zoneOverlay
class="zone-overlay position-absolute top-0 start-0 w-100 h-100"
[attr.viewBox]="overlayViewBox()"
preserveAspectRatio="none"
[style.cursor]="overlayCursor"
(pointerdown)="onOverlayPointerDown($event)"
(pointermove)="onOverlayPointerMove($event)"
(pointerup)="onOverlayPointerUp($event)"
(pointercancel)="onOverlayPointerCancel($event)"
(lostpointercapture)="onOverlayPointerCancel($event)"
aria-label="Zone overlay" i18n-aria-label
>
@for (zone of template.zones; track $index; let i = $index) {
@if (zoneDisplayRect(i); as rect) {
<g>
<rect
class="zone-rect"
[class.zone-rect-selected]="selectedZoneIndex === i"
[attr.x]="rect.x"
[attr.y]="rect.y"
[attr.width]="rect.w"
[attr.height]="rect.h"
[attr.stroke]="zoneColor(i)"
[attr.fill]="zoneColor(i)"
></rect>
<text
class="zone-label fw-semibold pe-none"
[attr.x]="rect.x + overlayUnitSize(6)"
[attr.y]="zoneLabelY(rect)"
[attr.font-size]="overlayFontSize()"
[attr.fill]="zoneColor(i)"
>{{ zoneLabel(zone, i) }}</text>
@if (selectedZoneIndex === i) {
@for (handle of resizeHandles(rect); track handle.handle) {
<rect
class="zone-resize-handle"
[attr.x]="handle.x - overlayHandleSize() / 2"
[attr.y]="handle.y - overlayHandleSize() / 2"
[attr.width]="overlayHandleSize()"
[attr.height]="overlayHandleSize()"
[attr.fill]="zoneColor(i)"
></rect>
}
}
</g>
}
}
@if (drawingRect(); as rect) {
<rect
class="zone-drawing-rect"
[attr.x]="rect.x"
[attr.y]="rect.y"
[attr.width]="rect.w"
[attr.height]="rect.h"
></rect>
}
</svg>
}
</div>
@if (!pageLoaded()) {
<div class="p-5 d-flex justify-content-center align-items-center">
<output class="spinner-border">
<span class="visually-hidden" i18n>Loading page...</span>
</output>
</div>
}
} @else {
<div class="py-5 px-3 text-center text-muted">
<pngx-bs-icon name="file-text" width="32" height="32"></pngx-bs-icon>
<p class="mt-3 mb-0" i18n>No sample document selected.</p>
</div>
}
</div>
</div>
</div>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-outline-secondary" (click)="activeModal.dismiss()" [disabled]="saving()" i18n>Cancel</button>
<button type="button" class="btn btn-primary" (click)="save()" [disabled]="saving() || !isDirty">
@if (saving()) {
<span class="spinner-border spinner-border-sm me-1" aria-hidden="true"></span>
}
<span i18n>Save</span>
</button>
</div>
@@ -0,0 +1,64 @@
:host {
display: flex;
flex-direction: column;
min-height: 0;
}
@media (min-width: 992px) {
.modal-body {
overflow: hidden;
}
.editor-layout {
min-height: 0;
}
}
.preview-page-input {
width: 3.5rem;
}
.preview-zoom {
min-width: 4rem;
}
.zone-overlay {
touch-action: none;
}
.zone-rect,
.zone-drawing-rect {
vector-effect: non-scaling-stroke;
}
.zone-rect {
fill-opacity: 0.2;
stroke-width: 2;
}
.zone-rect-selected {
stroke-width: 3;
}
.zone-label {
font-family: var(--bs-font-sans-serif);
paint-order: stroke;
stroke: #fff;
stroke-linejoin: round;
stroke-width: 4px;
vector-effect: non-scaling-stroke;
}
.zone-resize-handle {
stroke: #fff;
stroke-width: 1;
vector-effect: non-scaling-stroke;
}
.zone-drawing-rect {
fill: var(--bs-primary);
fill-opacity: 0.2;
stroke: var(--bs-primary);
stroke-dasharray: 5 5;
stroke-width: 2;
}
@@ -0,0 +1,906 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ElementRef } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { By } from '@angular/platform-browser'
import { NgbActiveModal, NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { of, Subject, throwError } from 'rxjs'
import { PermissionsFormComponent } from 'src/app/components/common/input/permissions/permissions-form/permissions-form.component'
import { CustomField, CustomFieldDataType } from 'src/app/data/custom-field'
import { Document } from 'src/app/data/document'
import {
MATCH_ANY,
MATCH_AUTO,
MATCH_LITERAL,
MATCH_NONE,
} from 'src/app/data/matching-model'
import {
OcrTemplate,
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTestResult,
OcrZoneTransform,
} from 'src/app/data/ocr-template'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { PermissionsService } from 'src/app/services/permissions.service'
import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service'
import { DocumentService } from 'src/app/services/rest/document.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { OcrTemplateEditorComponent } from './ocr-template-editor.component'
const customFields: CustomField[] = [
{ id: 1, name: 'Invoice number', data_type: CustomFieldDataType.String },
{ id: 2, name: 'Status', data_type: CustomFieldDataType.Select },
]
function zone(overrides: Partial<OcrTemplateZone> = {}): OcrTemplateZone {
return {
name: 'Invoice number',
target: OcrZoneTarget.CustomField,
custom_field: 1,
x: 100,
y: 100,
width: 200,
height: 100,
page: 1,
ocr_language: 'eng',
transform: OcrZoneTransform.Strip,
validation_regex: '',
order: 0,
...overrides,
}
}
function pointer(x: number, y: number, pointerId = 1): PointerEvent {
return Object.assign(
new MouseEvent('pointermove', { clientX: x, clientY: y, button: 0 }),
{ pointerId }
) as PointerEvent
}
describe('OcrTemplateEditorComponent', () => {
let component: OcrTemplateEditorComponent
let fixture: ComponentFixture<OcrTemplateEditorComponent>
let documentService: DocumentService
let templateService: OcrTemplateService
let modalService: NgbModal
const activeModal = { close: jest.fn(), dismiss: jest.fn() }
beforeEach(() => {
TestBed.configureTestingModule({
imports: [BootstrapIconComponent, OcrTemplateEditorComponent],
providers: [
{ provide: NgbActiveModal, useValue: activeModal },
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
})
jest
.spyOn(TestBed.inject(CustomFieldsService), 'listAll')
.mockReturnValue(of({ count: 2, all: [1, 2], results: customFields }))
jest
.spyOn(TestBed.inject(UserService), 'listAll')
.mockReturnValue(of({ count: 0, all: [], results: [] }))
TestBed.inject(PermissionsService).initialize([], { id: 1 } as User)
documentService = TestBed.inject(DocumentService)
jest
.spyOn(documentService, 'get')
.mockReturnValue(of({ id: 5, page_count: 3 }))
templateService = TestBed.inject(OcrTemplateService)
modalService = TestBed.inject(NgbModal)
activeModal.close.mockReset()
fixture = TestBed.createComponent(OcrTemplateEditorComponent)
component = fixture.componentInstance
})
function prepareOverlay() {
const image = document.createElement('img')
Object.defineProperties(image, {
naturalWidth: { value: 1000 },
naturalHeight: { value: 1000 },
clientWidth: { value: 1000 },
})
const overlay = document.createElementNS(
'http://www.w3.org/2000/svg',
'svg'
)
Object.assign(overlay, {
getScreenCTM: () => ({ inverse: () => ({}) }),
createSVGPoint: () => ({
x: 0,
y: 0,
matrixTransform() {
return { x: this.x, y: this.y }
},
}),
setPointerCapture: jest.fn(),
hasPointerCapture: () => true,
releasePointerCapture: jest.fn(),
})
component.imageRef = new ElementRef(image)
component.overlayRef = new ElementRef(overlay)
component.pageLoaded.set(true)
}
function fieldDialog() {
const dialog = {
dialogMode: { set: jest.fn() },
object: null,
succeeded: new Subject<CustomField>(),
failed: new Subject(),
}
jest
.spyOn(modalService, 'open')
.mockReturnValue({ componentInstance: dialog } as any)
return dialog
}
it('updates screen-sized overlay units when the preview image resizes', () => {
let resize: ResizeObserverCallback
const observer = {
observe: jest.fn(),
unobserve: jest.fn(),
disconnect: jest.fn(),
}
const resizeObserver = jest
.spyOn(globalThis, 'ResizeObserver')
.mockImplementation((callback) => {
resize = callback
return observer
})
const image = document.createElement('img')
const ref = new ElementRef(image)
component.pageImage = ref
expect(observer.observe).toHaveBeenCalledWith(image)
resize([], observer)
expect(component.overlayUnitSize(8)).toBe(8)
Object.defineProperties(image, {
naturalWidth: { value: 1000 },
clientWidth: { value: 500 },
})
resize([], observer)
expect(component.overlayUnitSize(8)).toBe(16)
expect(component.overlayFontSize()).toBe(24)
component.pageImage = undefined
expect(observer.disconnect).toHaveBeenCalledTimes(1)
fixture.destroy()
expect(observer.disconnect).toHaveBeenCalledTimes(2)
resizeObserver.mockRestore()
})
it('renders loaded image dimensions, zone labels, and selection handles', () => {
component.object = {
...component.template,
id: 3,
sample_document: 5,
zones: [zone()],
}
component.selectedZoneIndex = 0
fixture.detectChanges()
const image = fixture.nativeElement.querySelector('img')
Object.defineProperties(image, {
naturalWidth: { value: 1000 },
naturalHeight: { value: 1500 },
})
image.dispatchEvent(new Event('load'))
fixture.detectChanges()
expect(component.template).toMatchObject({
source_width: 1000,
source_height: 1500,
})
expect(
fixture.nativeElement
.querySelector('.zone-overlay')
.getAttribute('viewBox')
).toBe('0 0 1000 1500')
expect(
fixture.nativeElement.querySelector('.zone-label').textContent
).toContain('Invoice number')
expect(
fixture.nativeElement.querySelectorAll('.zone-resize-handle')
).toHaveLength(8)
expect(component.zoneLabel(zone({ name: '' }), 2)).toBe('Zone 3')
expect(component.zoneLabelY({ x: 0, y: 0, w: 100, h: 100 })).toBe(14)
})
it('offers the supported language and transform choices', () => {
expect(component.languageOptions.map((option) => option.id)).toContain(
'eng'
)
expect(component.transformOptions.map((option) => option.id)).toEqual(
Object.values(OcrZoneTransform)
)
})
it('clamps page navigation and supports one-based page input', () => {
component.setSampleDocument([5])
component.previewPageDisplay = 2
expect(component.previewPage).toBe(1)
expect(component.previewPageDisplay).toBe(2)
expect(component.pageImageUrl).toContain('/5/2/')
component.pageLoaded.set(true)
component.goToPage(1)
expect(component.pageLoaded()).toBe(true)
component.goToPage(Infinity)
component.goToPage(NaN)
expect(component.previewPage).toBe(1)
component.goToPage(99)
expect(component.previewPage).toBe(2)
expect(component.pageLoaded()).toBe(false)
component.goToPage(-5)
expect(component.previewPage).toBe(0)
component.previewPageCount.set(null)
component.goToPage(4.7)
expect(component.previewPage).toBe(4)
})
it('uses the standard picker styling and shows a known page total', () => {
component.setSampleDocument([5])
fixture.detectChanges()
const picker = fixture.nativeElement.querySelector(
'pngx-input-document-link'
)
expect(picker.classList.contains('paperless-input-select')).toBe(true)
const pageInput = fixture.nativeElement.querySelector(
'input[aria-label="Page"]'
)
expect(pageInput.max).toBe('3')
expect(pageInput.parentElement.textContent.trim()).toBe('of 3')
})
it('does not display a question mark when the page total is unknown', () => {
jest
.spyOn(documentService, 'get')
.mockReturnValueOnce(of({ id: 5, page_count: null }))
component.setSampleDocument([5])
fixture.detectChanges()
const pageInput = fixture.nativeElement.querySelector(
'input[aria-label="Page"]'
)
expect(
pageInput.parentElement.querySelector('.input-group-text')
).toBeNull()
component.pageLoaded.set(true)
component.previewPageDisplay = 2
expect(component.previewPage).toBe(1)
})
it('limits zoom and resets it to the original size', () => {
for (let i = 0; i < 20; i++) component.zoomIn()
expect(component.zoom).toBe(4)
for (let i = 0; i < 20; i++) component.zoomOut()
expect(component.zoom).toBe(0.5)
component.resetZoom()
expect(component.zoom).toBe(1)
})
it('moves last-page zones to the last page and can pin them to a page', () => {
component.setSampleDocument([5])
const selected = zone()
component.template.zones = [selected]
expect(component.isLastPageZone(selected)).toBe(false)
component.setLastPageZone(selected, true)
expect(selected.page).toBe(-1)
expect(component.isLastPageZone(selected)).toBe(true)
component.selectZone(0)
expect(component.previewPage).toBe(2)
component.setLastPageZone(selected, false)
expect(selected.page).toBe(3)
})
it('ignores page-count errors from a previous sample document', () => {
const pending = new Subject<Document>()
jest
.spyOn(documentService, 'get')
.mockReturnValueOnce(pending)
.mockReturnValueOnce(of({ id: 6 }))
component.setSampleDocument([5])
component.setSampleDocument([6])
pending.error(new Error('Old document failed'))
expect(component.pageFailed()).toBe(false)
expect(component.previewPageCount()).toBeNull()
})
it('ignores non-primary, unloaded, and saving pointer gestures', () => {
prepareOverlay()
component.onOverlayPointerDown(
new MouseEvent('pointerdown', { button: 2 }) as PointerEvent
)
expect(component.drawingRect()).toBeNull()
component.pageLoaded.set(false)
component.onOverlayPointerDown(pointer(100, 100))
expect(component.drawingRect()).toBeNull()
component.pageLoaded.set(true)
component.saving.set(true)
component.onOverlayPointerDown(pointer(100, 100))
expect(component.drawingRect()).toBeNull()
})
it('ignores pointer events when the SVG coordinate transform is unavailable', () => {
component.onOverlayPointerDown(pointer(100, 100))
expect(component.drawingRect()).toBeNull()
prepareOverlay()
jest
.spyOn(component.overlayRef.nativeElement, 'getScreenCTM')
.mockReturnValue(null)
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerMove(pointer(200, 200))
expect(component.drawingRect()).toBeNull()
})
it('updates a drawing only for its captured pointer', () => {
prepareOverlay()
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerDown(pointer(500, 500, 2))
component.onOverlayPointerMove(pointer(900, 900, 2))
expect(component.drawingRect()).toEqual({ x: 100, y: 100, w: 0, h: 0 })
component.onOverlayPointerMove(pointer(250, 200))
expect(component.drawingRect()).toEqual({ x: 100, y: 100, w: 150, h: 100 })
component.onOverlayPointerUp(pointer(250, 200))
expect(component.template.zones[0]).toMatchObject({
x: 100,
y: 100,
width: 150,
height: 100,
})
})
it('resizes a selected zone using its corner handle', () => {
prepareOverlay()
const selected = zone()
component.template.zones = [selected]
component.selectZone(0)
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerMove(pointer(50, 75))
component.onOverlayPointerUp(pointer(60, 80))
expect(selected).toMatchObject({ x: 60, y: 80, width: 240, height: 120 })
expect(component.template.zones).toHaveLength(1)
})
it('shows the appropriate cursor over handles, zones, and empty space', () => {
prepareOverlay()
component.template.zones = [zone()]
component.onOverlayPointerMove(pointer(150, 150))
expect(component.overlayCursor).toBe('move')
component.selectZone(0)
component.onOverlayPointerMove(pointer(100, 100))
expect(component.overlayCursor).toBe('nw-resize')
component.onOverlayPointerMove(pointer(500, 500))
expect(component.overlayCursor).toBe('crosshair')
component.template.zones[0].page = 2
component.onOverlayPointerMove(pointer(100, 100))
expect(component.zoneDisplayRect(0)).toBeNull()
expect(component.overlayCursor).toBe('crosshair')
})
it('cancels only the captured pointer and tolerates an already-lost capture', () => {
prepareOverlay()
component.onOverlayPointerCancel(pointer(100, 100))
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerCancel(pointer(150, 150, 2))
expect(component.drawingRect()).not.toBeNull()
jest
.spyOn(component.overlayRef.nativeElement, 'hasPointerCapture')
.mockReturnValue(false)
component.onOverlayPointerCancel(pointer(150, 150))
expect(component.drawingRect()).toBeNull()
expect(
component.overlayRef.nativeElement.releasePointerCapture
).not.toHaveBeenCalled()
expect(component.template.zones).toEqual([])
})
it('preserves the selected zone when an earlier or later zone is removed', () => {
component.template.zones = [zone(), zone(), zone(), zone()]
component.selectZone(1)
const selected = component.selectedZone
component.removeZone(3)
expect(component.selectedZone).toBe(selected)
component.removeZone(0)
expect(component.selectedZoneIndex).toBe(0)
expect(component.selectedZone).toBe(selected)
component.selectZone(99)
expect(component.selectedZone).toBeNull()
})
it('does not test zones without a selection or a sample document', () => {
const test = jest.spyOn(templateService, 'testZone')
component.testZone()
component.template.zones = [zone()]
component.selectZone(0)
component.testZone()
expect(test).not.toHaveBeenCalled()
})
it.each([
{
response: { error: { error: 'OCR unavailable' } },
message: 'OCR unavailable',
},
{ response: new Error('Network failure'), message: 'Test failed' },
])('shows zone-test errors: $message', ({ response, message }) => {
component.template.zones = [zone({ page: undefined })]
component.selectZone(0)
component.setSampleDocument([5])
jest
.spyOn(templateService, 'testZone')
.mockReturnValue(throwError(() => response))
component.testZone()
expect(component.zoneTestResult).toEqual({ error: message })
expect(component.zoneTesting).toBe(false)
expect(templateService.testZone).toHaveBeenCalledWith(
5,
expect.objectContaining({ page: 1 })
)
})
it('switches between built-in targets, custom fields, and an unassigned field', () => {
const selected = zone()
expect(component.zoneFieldValue(selected)).toBe(1)
component.setZoneField(selected, OcrZoneTarget.Title)
expect(component.zoneFieldValue(selected)).toBe(OcrZoneTarget.Title)
expect(selected.custom_field).toBeNull()
component.setZoneField(selected, 7)
expect(selected).toMatchObject({
target: OcrZoneTarget.CustomField,
custom_field: 7,
})
component.setZoneField(selected, null)
expect(component.zoneFieldValue(selected)).toBeNull()
expect(component.zonesForField(selected)).toEqual([])
component.setCombineFormat(selected, 'Ignored')
expect(component.template.combine_formats).toEqual({})
})
it('creates combine formats when absent and leaves implicit formats alone on rename', () => {
const selected = zone({ target: OcrZoneTarget.Title, custom_field: null })
component.template.combine_formats = undefined
component.renameZone(selected, 'Renamed')
expect(selected.name).toBe('Renamed')
expect(component.getCombineFormat(selected)).toBe('')
component.setCombineFormat(selected, '{Renamed} ')
component.insertCombineToken(selected, selected)
expect(component.template.combine_formats).toEqual({
title: '{Renamed} {Renamed}',
})
})
it('saves templates without combine formats', () => {
component.template.combine_formats = undefined
fixture.detectChanges()
expect(component.isDirty).toBe(false)
jest
.spyOn(templateService, 'create')
.mockReturnValue(of(component.template))
component.save()
expect(component.template.combine_formats).toBeUndefined()
})
it('does not assign unsupported custom field types and surfaces creation failures', () => {
component.template.zones = [zone({ custom_field: null })]
component.selectZone(0)
const dialog = fieldDialog()
const error = jest.spyOn(TestBed.inject(ToastService), 'showError')
component.createField('Status')
expect(dialog.object).toEqual({ name: 'Status' })
dialog.succeeded.next(customFields[1])
expect(component.selectedZone.custom_field).toBeNull()
const failure = new Error('Could not save')
dialog.failed.next(failure)
expect(error).toHaveBeenCalledWith('Error saving field.', failure)
})
it('refreshes field choices when a field is created without a selected zone', () => {
const dialog = fieldDialog()
component.createField('Total')
dialog.succeeded.next(customFields[0])
expect(component.customFields()).toEqual(customFields)
expect(component.selectedZone).toBeNull()
})
it('should edit a copy of the template and load its sample document', () => {
const original: OcrTemplate = {
...component.template,
id: 3,
name: 'Invoices',
match: 'Northwind',
matching_algorithm: MATCH_LITERAL,
sample_document: 5,
zones: [zone()],
combine_formats: { '1': '{Invoice number}' },
}
component.object = original
fixture.detectChanges()
expect(component.dialogTitle).toEqual('Edit OCR Template')
expect(component.previewDocId).toEqual(5)
expect(component.previewPageCount()).toEqual(3)
expect(component.pageImageUrl).toContain('document-page-image/5/1/')
expect(component.template.match).toEqual('Northwind')
expect(component.isDirty).toBeFalsy()
component.template.zones[0].x = 500
component.template.combine_formats['1'] = 'Edited'
expect(original.zones[0].x).toEqual(100)
expect(original.combine_formats['1']).toEqual('{Invoice number}')
})
it('should offer every matching algorithm except automatic', () => {
expect(component.matchingAlgorithms.map((a) => a.id)).not.toContain(
MATCH_AUTO
)
expect(component.template.matching_algorithm).toEqual(MATCH_ANY)
expect(component.patternRequired).toBeTruthy()
component.template.matching_algorithm = MATCH_NONE
expect(component.patternRequired).toBeFalsy()
})
it('should initialize new template permissions from saved defaults', () => {
const defaults = {
[SETTINGS_KEYS.DEFAULT_PERMS_OWNER]: 1,
[SETTINGS_KEYS.DEFAULT_PERMS_VIEW_USERS]: [2],
[SETTINGS_KEYS.DEFAULT_PERMS_VIEW_GROUPS]: [3],
[SETTINGS_KEYS.DEFAULT_PERMS_EDIT_USERS]: [],
[SETTINGS_KEYS.DEFAULT_PERMS_EDIT_GROUPS]: [4],
}
jest
.spyOn(TestBed.inject(SettingsService), 'get')
.mockImplementation((key) => defaults[key])
fixture.detectChanges()
expect(component.permissionsForm).toEqual({
owner: 1,
set_permissions: {
view: { users: [2], groups: [3] },
change: { users: [], groups: [4] },
},
})
expect(
fixture.debugElement.query(By.directive(PermissionsFormComponent))
).toBeTruthy()
expect(component.isDirty).toBe(false)
const create = jest
.spyOn(templateService, 'create')
.mockReturnValue(of({ ...component.template, id: 3 }))
component.save()
expect(create).toHaveBeenCalledWith(
expect.objectContaining(component.permissionsForm)
)
})
it('should copy existing permissions and track changes without mutating the list item', () => {
component.object = {
...component.template,
id: 3,
owner: 1,
permissions: {
view: { users: [2], groups: [] },
change: { users: [], groups: [3] },
},
}
fixture.detectChanges()
expect(component.permissionsForm.owner).toBe(1)
expect(component.isDirty).toBe(false)
component.permissionsForm.set_permissions.view.users.push(4)
expect(component.isDirty).toBe(true)
expect(component.object.permissions.view.users).toEqual([2])
const update = jest
.spyOn(templateService, 'update')
.mockReturnValue(of(component.template))
component.save()
expect(update).toHaveBeenCalledWith(
expect.objectContaining({
owner: 1,
set_permissions: {
view: { users: [2, 4], groups: [] },
change: { users: [], groups: [3] },
},
})
)
})
it('should hide sharing controls and omit permission changes for a shared editor', () => {
component.object = {
...component.template,
id: 3,
owner: 2,
user_can_change: true,
}
fixture.detectChanges()
expect(
fixture.debugElement.query(By.directive(PermissionsFormComponent))
).toBeNull()
component.permissionsForm = {
owner: 1,
set_permissions: { view: { users: [1] } },
}
const update = jest
.spyOn(templateService, 'update')
.mockReturnValue(of(component.template))
component.save()
expect(update.mock.calls[0][0].owner).toBe(2)
expect(update.mock.calls[0][0]).not.toHaveProperty('set_permissions')
})
it('should keep only the last picked sample document', () => {
component.setSampleDocument([5, 6])
expect(component.sampleDocumentIds).toEqual([6])
component.setSampleDocument([])
expect(component.previewDocId).toBeNull()
expect(component.pageImageUrl).toBeNull()
})
it('should ignore a stale sample document response', () => {
const pending = new Subject<Document>()
jest.spyOn(documentService, 'get').mockReturnValueOnce(pending)
component.setSampleDocument([5])
component.setSampleDocument([6])
pending.next({ id: 5, page_count: 99 })
expect(component.previewPageCount()).toEqual(3)
})
it('should show a failed page and retry it', () => {
jest
.spyOn(documentService, 'get')
.mockReturnValueOnce(throwError(() => new Error('offline')))
component.setSampleDocument([5])
expect(component.pageFailed()).toBeTruthy()
component.setSampleDocument(component.sampleDocumentIds)
expect(component.pageFailed()).toBeFalsy()
component.onImageError()
expect(component.pageFailed()).toBeTruthy()
})
it('should support create and update', () => {
const createSpy = jest
.spyOn(templateService, 'create')
.mockReturnValue(of({ ...component.template, id: 3 }))
const updateSpy = jest
.spyOn(templateService, 'update')
.mockReturnValue(of(component.template))
const succeededSpy = jest.spyOn(component.succeeded, 'emit')
component.save()
expect(createSpy).toHaveBeenCalled()
expect(succeededSpy).toHaveBeenCalledWith({ ...component.template, id: 3 })
expect(activeModal.close).toHaveBeenCalled()
component.saving.set(false)
component.template.id = 3
component.save()
expect(updateSpy).toHaveBeenCalledWith(component.template)
})
it('should emit failed and allow editing again after a failed save', () => {
prepareOverlay()
component.template.zones = [zone()]
const pending = new Subject<OcrTemplate>()
jest.spyOn(templateService, 'create').mockReturnValue(pending)
const failedSpy = jest.spyOn(component.failed, 'emit')
component.onOverlayPointerDown(pointer(150, 150))
component.save()
component.save()
expect(templateService.create).toHaveBeenCalledTimes(1)
component.onOverlayPointerMove(pointer(250, 250))
component.onOverlayPointerUp(pointer(250, 250))
expect(component.template.zones[0].x).toEqual(100)
pending.error(new Error('Save failed'))
expect(failedSpy).toHaveBeenCalled()
expect(component.saving()).toBeFalsy()
component.onOverlayPointerDown(pointer(150, 150))
component.onOverlayPointerUp(pointer(250, 250))
expect(component.template.zones[0].x).toEqual(200)
})
it('should track unsaved changes', () => {
fixture.detectChanges()
expect(component.isDirty).toBeFalsy()
component.template.combine_formats = { '1': '' }
expect(component.isDirty).toBeFalsy()
component.template.name = 'Invoices'
expect(component.isDirty).toBeTruthy()
component.template.name = ''
expect(component.isDirty).toBeFalsy()
component.setSampleDocument([5])
expect(component.isDirty).toBeTruthy()
})
it('should draw a zone with no field in empty space', () => {
prepareOverlay()
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerUp(pointer(1200, 250))
expect(component.template.zones[0]).toMatchObject({
x: 100,
y: 100,
width: 900,
height: 150,
custom_field: null,
})
expect(component.selectedZoneIndex).toEqual(0)
})
it('should clear the selection on a click in empty space', () => {
prepareOverlay()
component.template.zones = [zone()]
component.selectZone(0)
component.onOverlayPointerDown(pointer(600, 600))
component.onOverlayPointerUp(pointer(602, 601))
expect(component.template.zones).toHaveLength(1)
expect(component.selectedZoneIndex).toBeNull()
})
it('should move a zone when dragging on it', () => {
prepareOverlay()
component.template.zones = [zone()]
component.onOverlayPointerDown(pointer(150, 150))
component.onOverlayPointerUp(pointer(250, 200))
expect(component.template.zones).toHaveLength(1)
expect(component.template.zones[0]).toMatchObject({ x: 200, y: 150 })
})
it('should cancel a drawing on escape and ignore other pointers', () => {
prepareOverlay()
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerUp(pointer(300, 200, 2))
expect(component.drawingRect()).not.toBeNull()
component.cancelDrawing()
component.onOverlayPointerUp(pointer(300, 200))
expect(component.template.zones).toEqual([])
})
it('should open a zone in the zones tab and go back to the list', () => {
component.template.zones = [zone(), zone()]
component.selectZone(1)
expect(component.activeTab).toEqual('zones')
expect(component.selectedZone).toBe(component.template.zones[1])
component.deselectZone()
expect(component.selectedZone).toBeNull()
component.selectZone(0)
component.removeZone(0)
expect(component.template.zones).toHaveLength(1)
expect(component.selectedZone).toBeNull()
})
it('should only offer custom fields OCR can write to', () => {
fixture.detectChanges()
expect(component.fieldItems().map((item) => item.id)).toEqual([
OcrZoneTarget.Title,
OcrZoneTarget.Asn,
OcrZoneTarget.Created,
1,
])
})
it('should create a custom field and assign it to the zone', () => {
const succeeded = new Subject<CustomField>()
const dialog = {
dialogMode: { set: jest.fn() },
object: null,
succeeded,
failed: new Subject(),
}
jest
.spyOn(modalService, 'open')
.mockReturnValue({ componentInstance: dialog } as any)
component.template.zones = [zone({ custom_field: null })]
component.selectZone(0)
component.createField()
expect(dialog.object).toEqual({ name: 'Invoice number' })
succeeded.next(customFields[0])
expect(component.template.zones[0].custom_field).toEqual(1)
})
it('should build and prune combine formats', () => {
const first = zone({ name: 'First' })
const second = zone({ name: 'Second' })
const other = zone({ name: 'Other', custom_field: 2 })
component.template.zones = [first, second, other]
component.template.combine_formats = { '2': '{Other}' }
expect(component.isFieldShared(first)).toBeTruthy()
expect(component.isFieldShared(other)).toBeFalsy()
expect(component.defaultCombineFormat(first)).toEqual('{First} {Second}')
component.insertCombineToken(first, second)
component.insertCombineToken(first, first)
expect(component.getCombineFormat(second)).toEqual('{Second} {First}')
jest.spyOn(templateService, 'create').mockReturnValue(of(null))
component.save()
expect(component.template.combine_formats).toEqual({
'1': '{Second} {First}',
})
})
it('should update exact tokens when a zone is renamed', () => {
const first = zone({ name: 'First [value]' })
component.template.zones = [first, zone({ name: 'Second' })]
component.template.combine_formats = {
'1': 'First [value]: {First [value]} / { First [value] } / {Second} / {First [value] extra}',
'2': '{First [value]}',
}
component.renameZone(first, '')
component.renameZone(first, 'Invoice number')
expect(first.name).toEqual('Invoice number')
expect(component.template.combine_formats).toEqual({
'1': 'First [value]: {Invoice number} / {Invoice number} / {Second} / {First [value] extra}',
'2': '{First [value]}',
})
})
it('should keep the language array stable until the value changes', () => {
const selected = zone({ ocr_language: 'deu+eng' })
const languages = component.ocrLanguageArray(selected)
expect(languages).toEqual(['deu', 'eng'])
expect(component.ocrLanguageArray(selected)).toBe(languages)
component.setOcrLanguages(selected, ['fra'])
expect(selected.ocr_language).toEqual('fra')
expect(component.ocrLanguageArray(selected)).toEqual(['fra'])
component.setOcrLanguages(selected, null)
expect(component.ocrLanguageArray(selected)).toEqual([])
})
it('should hide a test result once the zone changes', () => {
const selected = zone()
component.template.zones = [selected]
component.selectZone(0)
component.setSampleDocument([5])
jest
.spyOn(templateService, 'testZone')
.mockReturnValue(of({ value: '123' }))
component.testZone()
expect(component.zoneTestResult.value).toEqual('123')
selected.transform = OcrZoneTransform.Numeric
expect(component.zoneTestResult).toBeNull()
})
it('should drop a pending test when the zone or sample document changes', () => {
const pending = new Subject<OcrZoneTestResult>()
component.template.zones = [zone(), zone()]
component.selectZone(0)
component.setSampleDocument([5])
jest.spyOn(templateService, 'testZone').mockReturnValue(pending)
component.testZone()
expect(component.zoneTesting).toBeTruthy()
component.selectZone(1)
pending.next({ value: 'Old zone' })
expect(component.zoneTestResult).toBeNull()
component.testZone()
component.setSampleDocument([])
pending.next({ value: 'Old document' })
expect(component.zoneTestResult).toBeNull()
})
})
@@ -0,0 +1,963 @@
import { CommonModule } from '@angular/common'
import {
Component,
computed,
ElementRef,
EventEmitter,
HostListener,
inject,
Input,
OnDestroy,
OnInit,
Output,
signal,
ViewChild,
} from '@angular/core'
import { FormsModule } from '@angular/forms'
import {
NgbActiveModal,
NgbModal,
NgbNavModule,
} from '@ng-bootstrap/ng-bootstrap'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { Subject, takeUntil } from 'rxjs'
import { CustomFieldEditDialogComponent } from 'src/app/components/common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
import { EditDialogMode } from 'src/app/components/common/edit-dialog/edit-dialog.component'
import { DocumentLinkComponent } from 'src/app/components/common/input/document-link/document-link.component'
import { CheckComponent } from 'src/app/components/common/input/check/check.component'
import { NumberComponent } from 'src/app/components/common/input/number/number.component'
import {
PermissionsFormComponent,
PermissionsFormObject,
} from 'src/app/components/common/input/permissions/permissions-form/permissions-form.component'
import { SelectComponent } from 'src/app/components/common/input/select/select.component'
import { SwitchComponent } from 'src/app/components/common/input/switch/switch.component'
import { TextComponent } from 'src/app/components/common/input/text/text.component'
import { CustomField } from 'src/app/data/custom-field'
import {
MATCH_ANY,
MATCH_AUTO,
MATCH_NONE,
MATCHING_ALGORITHMS,
} from 'src/app/data/matching-model'
import {
DEFAULT_OCR_ZONE_LANGUAGE,
getZoneFieldKey,
OCR_BUILTIN_TARGETS,
OCR_SUPPORTED_FIELD_TYPES,
OcrTemplate,
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTestResult,
OcrZoneTransform,
ZoneTestRequest,
} from 'src/app/data/ocr-template'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
import { PermissionsService } from 'src/app/services/permissions.service'
import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service'
import { DocumentService } from 'src/app/services/rest/document.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { OcrTemplateEditorZoneListComponent } from './ocr-template-editor-zone-list/ocr-template-editor-zone-list.component'
import {
DisplayRect,
DrawingRect,
findHandleAt,
findZoneAt,
getResizeHandles,
getZoneColor,
getZoneDisplayRect,
getZonePage,
HANDLE_SIZE,
isZoneOnPage,
MIN_ZONE_SIZE,
MoveStart,
moveZone,
Point,
ResizeHandle,
resizeZone,
} from './zone-geometry'
type ActiveTab = 'template' | 'zones'
type ZoneFieldSelection = OcrZoneTarget | number | null
type OverlayInteraction =
| { kind: 'idle' }
| { kind: 'drawing'; pointerId: number; rect: DrawingRect }
| { kind: 'moving'; pointerId: number; zoneIndex: number; start: MoveStart }
| {
kind: 'resizing'
pointerId: number
zoneIndex: number
handle: ResizeHandle
}
interface ZoneTest {
zone: OcrTemplateZone
/** The sample document and zone settings the result was produced from. */
key: string
result: OcrZoneTestResult | null
}
export const ZONE_TRANSFORM_OPTIONS = [
{ id: OcrZoneTransform.None, name: $localize`None` },
{ id: OcrZoneTransform.Strip, name: $localize`Strip whitespace` },
{ id: OcrZoneTransform.Uppercase, name: $localize`Uppercase` },
{ id: OcrZoneTransform.Lowercase, name: $localize`Lowercase` },
{ id: OcrZoneTransform.Numeric, name: $localize`Numeric only` },
{
id: OcrZoneTransform.StripPunctuation,
name: $localize`Remove leading/trailing punctuation`,
},
{ id: OcrZoneTransform.Date, name: $localize`Parse date` },
{ id: OcrZoneTransform.QrCode, name: $localize`Read QR/barcode` },
]
export const ZONE_LANGUAGE_OPTIONS = [
{ id: 'eng', name: $localize`English` },
{ id: 'deu', name: $localize`German` },
{ id: 'fra', name: $localize`French` },
{ id: 'ita', name: $localize`Italian` },
{ id: 'spa', name: $localize`Spanish` },
{ id: 'por', name: $localize`Portuguese` },
{ id: 'nld', name: $localize`Dutch` },
]
const TEMPLATE_MATCHING_ALGORITHMS = MATCHING_ALGORITHMS.filter(
(algorithm) => algorithm.id !== MATCH_AUTO
)
const NO_OVERLAY_INTERACTION: OverlayInteraction = { kind: 'idle' }
const RESIZE_CURSOR: Record<ResizeHandle, string> = {
nw: 'nw-resize',
ne: 'ne-resize',
sw: 'sw-resize',
se: 'se-resize',
n: 'n-resize',
s: 's-resize',
w: 'w-resize',
e: 'e-resize',
}
@Component({
selector: 'pngx-ocr-template-editor',
imports: [
DocumentLinkComponent,
NumberComponent,
TextComponent,
CheckComponent,
SelectComponent,
SwitchComponent,
CommonModule,
FormsModule,
NgbNavModule,
BootstrapIconComponent,
OcrTemplateEditorZoneListComponent,
PermissionsFormComponent,
IfOwnerDirective,
],
templateUrl: './ocr-template-editor.component.html',
styleUrls: ['./ocr-template-editor.component.scss'],
})
export class OcrTemplateEditorComponent implements OnInit, OnDestroy {
public readonly activeModal = inject(NgbActiveModal)
private readonly templateService = inject(OcrTemplateService)
private readonly customFieldsService = inject(CustomFieldsService)
private readonly documentService = inject(DocumentService)
private readonly toastService = inject(ToastService)
private readonly modalService = inject(NgbModal)
private readonly userService = inject(UserService)
private readonly settingsService = inject(SettingsService)
private readonly permissionsService = inject(PermissionsService)
private readonly destroy$ = new Subject<void>()
private readonly cancelZoneTest$ = new Subject<void>()
@Input() object: OcrTemplate
@Output() public readonly succeeded = new EventEmitter<OcrTemplate>()
@Output() public readonly failed = new EventEmitter()
@ViewChild('zoneOverlay') overlayRef: ElementRef<SVGSVGElement>
imageRef: ElementRef<HTMLImageElement>
private previewResizeObserver: ResizeObserver | null = null
private readonly overlayScale = signal(1)
@ViewChild('pageImage')
set pageImage(ref: ElementRef<HTMLImageElement> | undefined) {
this.previewResizeObserver?.disconnect()
this.imageRef = ref
if (!ref) return
this.previewResizeObserver = new ResizeObserver(() => {
const image = ref.nativeElement
if (image.naturalWidth && image.clientWidth) {
this.overlayScale.set(image.naturalWidth / image.clientWidth)
}
})
this.previewResizeObserver.observe(ref.nativeElement)
}
template: OcrTemplate = {
id: null,
name: '',
match: '',
matching_algorithm: MATCH_ANY,
is_insensitive: true,
sample_document: null,
source_width: 0,
source_height: 0,
enabled: true,
combine_formats: {},
zones: [],
}
public readonly customFields = signal<CustomField[]>([])
public readonly fieldItems = computed<
{ id: ZoneFieldSelection; name: string }[]
>(() => [
...OCR_BUILTIN_TARGETS,
...this.customFields()
.filter((field) => OCR_SUPPORTED_FIELD_TYPES.includes(field.data_type))
.map((field) => ({ id: field.id, name: field.name })),
])
public readonly users = signal<User[]>([])
permissionsForm: PermissionsFormObject = {}
public readonly OcrZoneTransform = OcrZoneTransform
public readonly saving = signal(false)
/** Bound to the document picker, which works with id arrays. */
sampleDocumentIds: number[] = []
previewPage = 0
public readonly previewPageCount = signal<number | null>(null)
public readonly pageLoaded = signal(false)
public readonly pageFailed = signal(false)
zoom = 1
get previewDocId(): number | null {
return this.sampleDocumentIds[0] ?? null
}
get pageImageUrl(): string | null {
return this.previewDocId
? this.templateService.getPageImageUrl(
this.previewDocId,
this.previewPage + 1
)
: null
}
get transformOptions() {
return ZONE_TRANSFORM_OPTIONS
}
get languageOptions() {
return ZONE_LANGUAGE_OPTIONS
}
get matchingAlgorithms() {
return TEMPLATE_MATCHING_ALGORITHMS
}
get patternRequired(): boolean {
return (
this.template.matching_algorithm !== MATCH_AUTO &&
this.template.matching_algorithm !== MATCH_NONE
)
}
public get previewPageDisplay(): number {
return this.previewPage + 1
}
public set previewPageDisplay(value: number) {
this.goToPage(value - 1)
}
activeTab: ActiveTab = 'template'
selectedZoneIndex: number | null = null
private overlayInteraction: OverlayInteraction = NO_OVERLAY_INTERACTION
overlayCursor = 'crosshair'
private readonly zoneTest = signal<ZoneTest | null>(null)
get zoneTestResult(): OcrZoneTestResult | null {
return this.isCurrentZoneTest() ? this.zoneTest().result : null
}
get zoneTesting(): boolean {
return this.isCurrentZoneTest() && this.zoneTest().result === null
}
get selectedZone(): OcrTemplateZone | null {
return this.selectedZoneIndex !== null
? (this.template.zones[this.selectedZoneIndex] ?? null)
: null
}
private initialState = ''
get isDirty(): boolean {
return this.editorState() !== this.initialState
}
private editorState(): string {
const formats = Object.entries(this.template.combine_formats ?? {}).filter(
([, format]) => format
)
return JSON.stringify({
...this.template,
combine_formats: Object.fromEntries(formats),
sample_document: this.previewDocId ?? null,
permissions_form: this.permissionsForm,
})
}
get dialogTitle(): string {
return this.template.id
? $localize`Edit OCR Template`
: $localize`New OCR Template`
}
ngOnInit() {
this.userService
.listAll()
.pipe(takeUntil(this.destroy$))
.subscribe((r) => this.users.set(r.results))
this.customFieldsService
.listAll()
.pipe(takeUntil(this.destroy$))
.subscribe((r) => {
this.customFields.set(r.results)
})
if (this.object) {
this.template = {
...this.object,
zones: this.object.zones.map((zone) => ({ ...zone })),
combine_formats: { ...this.object.combine_formats },
}
if (this.template.sample_document) {
this.setSampleDocument([this.template.sample_document])
}
}
const permissions = this.object?.permissions
this.permissionsForm = this.object
? {
owner: this.object.owner,
set_permissions: {
view: {
users: [...(permissions?.view.users ?? [])],
groups: [...(permissions?.view.groups ?? [])],
},
change: {
users: [...(permissions?.change.users ?? [])],
groups: [...(permissions?.change.groups ?? [])],
},
},
}
: {
owner: this.settingsService.get(SETTINGS_KEYS.DEFAULT_PERMS_OWNER),
set_permissions: {
view: {
users: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_VIEW_USERS
),
groups: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_VIEW_GROUPS
),
},
change: {
users: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_EDIT_USERS
),
groups: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_EDIT_GROUPS
),
},
},
}
this.initialState = this.editorState()
}
setSampleDocument(ids: number[]) {
const id = ids.at(-1)
this.sampleDocumentIds = id ? [id] : []
this.previewPage = 0
this.previewPageCount.set(null)
this.resetPage()
if (!id) return
this.documentService
.get(id)
.pipe(takeUntil(this.destroy$))
.subscribe({
next: (doc) => {
if (this.previewDocId !== id) return
this.previewPageCount.set(doc.page_count ?? null)
},
error: () => {
if (this.previewDocId === id) this.pageFailed.set(true)
},
})
}
private resetPage() {
this.stopOverlayInteraction()
this.clearZoneTest()
this.pageLoaded.set(false)
this.pageFailed.set(false)
}
goToPage(page: number) {
if (!Number.isFinite(page)) return
const pageCount = this.previewPageCount()
const max = pageCount ? pageCount - 1 : page
const clamped = Math.max(0, Math.min(Math.trunc(page), max))
if (clamped === this.previewPage) return
this.previewPage = clamped
this.resetPage()
}
zoomIn() {
this.zoom = Math.min(4, Math.round((this.zoom + 0.25) * 100) / 100)
}
zoomOut() {
this.zoom = Math.max(0.5, Math.round((this.zoom - 0.25) * 100) / 100)
}
resetZoom() {
this.zoom = 1
}
zonePage(zone: OcrTemplateZone): number {
return getZonePage(zone, this.previewPage, this.previewPageCount())
}
isLastPageZone(zone: OcrTemplateZone): boolean {
return zone.page === -1
}
setLastPageZone(zone: OcrTemplateZone, lastPage: boolean) {
zone.page = lastPage ? -1 : this.previewPage + 1
}
private isOnCurrentPage(zone: OcrTemplateZone): boolean {
return isZoneOnPage(zone, this.previewPage, this.previewPageCount())
}
onImageLoad() {
this.pageLoaded.set(true)
const img = this.imageRef.nativeElement
this.template.source_width = img.naturalWidth
this.template.source_height = img.naturalHeight
}
onImageError() {
this.stopOverlayInteraction()
this.pageFailed.set(true)
}
@HostListener('document:keydown.escape')
cancelDrawing() {
this.stopOverlayInteraction()
}
onOverlayPointerDown(event: PointerEvent) {
if (this.saving()) return
if (event.button !== 0 || this.overlayInteraction.kind !== 'idle') return
const point = this.svgPointFromEvent(event)
if (!this.pageLoaded() || !point) return
event.preventDefault()
if (this.selectedZoneIndex !== null) {
const handle = this.handleAt(point, this.selectedZoneIndex)
if (handle) {
this.overlayInteraction = {
kind: 'resizing',
pointerId: event.pointerId,
zoneIndex: this.selectedZoneIndex,
handle,
}
this.overlayRef.nativeElement.setPointerCapture(event.pointerId)
return
}
}
const clickedIdx = this.zoneIndexAt(point)
if (clickedIdx !== null) {
this.selectZone(clickedIdx)
const zone = this.template.zones[clickedIdx]
this.overlayInteraction = {
kind: 'moving',
pointerId: event.pointerId,
zoneIndex: clickedIdx,
start: {
mouseX: point.x,
mouseY: point.y,
zoneX: zone.x,
zoneY: zone.y,
},
}
this.overlayRef.nativeElement.setPointerCapture(event.pointerId)
return
}
// Empty space: start drawing a new zone. A click without a drag just clears
// the selection (see onOverlayPointerUp).
this.clearZoneTest()
this.selectedZoneIndex = null
this.overlayInteraction = {
kind: 'drawing',
pointerId: event.pointerId,
rect: {
startX: point.x,
startY: point.y,
endX: point.x,
endY: point.y,
},
}
this.overlayRef.nativeElement.setPointerCapture(event.pointerId)
}
onOverlayPointerMove(event: PointerEvent) {
if (
this.overlayInteraction.kind !== 'idle' &&
this.overlayInteraction.pointerId !== event.pointerId
)
return
const point = this.svgPointFromEvent(event)
if (!point) return
if (this.overlayInteraction.kind === 'resizing') {
resizeZone(
this.template.zones[this.overlayInteraction.zoneIndex],
this.overlayInteraction.handle,
point,
this.imageNaturalSize()
)
return
}
if (this.overlayInteraction.kind === 'moving') {
moveZone(
this.template.zones[this.overlayInteraction.zoneIndex],
point,
this.overlayInteraction.start,
this.imageNaturalSize()
)
return
}
if (this.overlayInteraction.kind === 'drawing') {
this.overlayInteraction.rect.endX = point.x
this.overlayInteraction.rect.endY = point.y
return
}
this.updateOverlayCursor(point)
}
private updateOverlayCursor(point: Point) {
if (this.selectedZoneIndex !== null) {
const handle = this.handleAt(point, this.selectedZoneIndex)
if (handle) {
this.overlayCursor = RESIZE_CURSOR[handle]
return
}
}
this.overlayCursor = this.zoneIndexAt(point) !== null ? 'move' : 'crosshair'
}
onOverlayPointerUp(event: PointerEvent) {
if (this.overlayInteraction.kind === 'idle') return
if (this.overlayInteraction.pointerId !== event.pointerId) return
this.onOverlayPointerMove(event)
const interaction = this.overlayInteraction
this.stopOverlayInteraction()
if (interaction.kind !== 'drawing') return
const rect = this.sourceRectFromDrawing(interaction.rect)
// Ignore tiny accidental clicks.
if (rect.w < MIN_ZONE_SIZE || rect.h < MIN_ZONE_SIZE) {
return
}
this.template.zones.push(this.createZoneFromRect(rect))
this.selectZone(this.template.zones.length - 1)
}
private createZoneFromRect(rect: DisplayRect): OcrTemplateZone {
const imageSize = this.imageNaturalSize()
return {
name: `Zone ${this.template.zones.length + 1}`,
target: OcrZoneTarget.CustomField,
custom_field: null,
x: rect.x,
y: rect.y,
width: rect.w,
height: rect.h,
page: this.previewPageDisplay,
ocr_language: DEFAULT_OCR_ZONE_LANGUAGE,
transform: OcrZoneTransform.Strip,
date_format: '',
validation_regex: '',
order: this.template.zones.length,
zone_source_width: imageSize.width,
zone_source_height: imageSize.height,
}
}
onOverlayPointerCancel(event: PointerEvent) {
if (this.overlayInteraction.kind === 'idle') return
if (this.overlayInteraction.pointerId !== event.pointerId) return
this.stopOverlayInteraction()
}
private stopOverlayInteraction() {
const interaction = this.overlayInteraction
this.overlayInteraction = NO_OVERLAY_INTERACTION
this.overlayCursor = 'crosshair'
const overlay = this.overlayRef?.nativeElement
if (
interaction.kind !== 'idle' &&
overlay?.hasPointerCapture(interaction.pointerId)
) {
overlay.releasePointerCapture(interaction.pointerId)
}
}
drawingRect(): DisplayRect | null {
return this.overlayInteraction.kind === 'drawing'
? this.displayRectFromDrawing(this.overlayInteraction.rect)
: null
}
/** The zone's rectangle on the overlay, or null if it's on another page. */
zoneDisplayRect(zoneIndex: number): DisplayRect | null {
const zone = this.template.zones[zoneIndex]
if (!this.isOnCurrentPage(zone)) return null
return getZoneDisplayRect(zone, this.imageNaturalSize())
}
private handleAt(point: Point, zoneIndex: number): ResizeHandle | null {
const rect = this.zoneDisplayRect(zoneIndex)
return rect ? findHandleAt(point, rect, this.overlayHandleSize()) : null
}
private zoneIndexAt(point: Point): number | null {
return findZoneAt(
point,
this.template.zones,
this.previewPage,
this.previewPageCount(),
this.imageNaturalSize()
)
}
overlayViewBox(): string {
const imageSize = this.imageNaturalSize()
return `0 0 ${imageSize.width} ${imageSize.height}`
}
public readonly zoneColor = getZoneColor
zoneLabel(zone: OcrTemplateZone, index: number): string {
return zone.name || `Zone ${index + 1}`
}
zoneLabelY(rect: DisplayRect): number {
return Math.max(this.overlayUnitSize(14), rect.y - this.overlayUnitSize(4))
}
public readonly resizeHandles = getResizeHandles
overlayHandleSize(): number {
return this.overlayUnitSize(HANDLE_SIZE)
}
overlayFontSize(): number {
return this.overlayUnitSize(12)
}
overlayUnitSize(screenPixels: number): number {
return screenPixels * this.overlayScale()
}
private svgPointFromEvent(event: MouseEvent): Point | null {
const svg = this.overlayRef?.nativeElement
const matrix = svg?.getScreenCTM()
if (!svg || !matrix) return null
const point = svg.createSVGPoint()
point.x = event.clientX
point.y = event.clientY
const svgPoint = point.matrixTransform(matrix.inverse())
const size = this.imageNaturalSize()
return {
x: Math.max(0, Math.min(svgPoint.x, size.width)),
y: Math.max(0, Math.min(svgPoint.y, size.height)),
}
}
private displayRectFromDrawing(rect: DrawingRect): DisplayRect {
return {
x: Math.min(rect.startX, rect.endX),
y: Math.min(rect.startY, rect.endY),
w: Math.abs(rect.endX - rect.startX),
h: Math.abs(rect.endY - rect.startY),
}
}
private sourceRectFromDrawing(rect: DrawingRect): DisplayRect {
const displayRect = this.displayRectFromDrawing(rect)
return {
x: Math.round(displayRect.x),
y: Math.round(displayRect.y),
w: Math.round(displayRect.w),
h: Math.round(displayRect.h),
}
}
private imageNaturalSize() {
const img = this.imageRef.nativeElement
return { width: img.naturalWidth, height: img.naturalHeight }
}
removeZone(index: number) {
this.stopOverlayInteraction()
this.clearZoneTest()
this.template.zones.splice(index, 1)
if (this.selectedZoneIndex === index) {
this.selectedZoneIndex = null
} else if (this.selectedZoneIndex > index) {
this.selectedZoneIndex--
}
}
selectZone(index: number) {
this.selectedZoneIndex = index
this.activeTab = 'zones'
this.clearZoneTest()
const zone = this.template.zones[index]
if (zone) {
this.goToPage(this.zonePage(zone) - 1)
}
}
deselectZone() {
this.selectedZoneIndex = null
this.clearZoneTest()
}
testZone() {
const zone = this.selectedZone
if (!zone || !this.previewDocId) return
this.clearZoneTest()
const test: ZoneTest = { zone, key: this.zoneTestKey(zone), result: null }
this.zoneTest.set(test)
this.templateService
.testZone(this.previewDocId, this.zoneTestRequest(zone))
.pipe(takeUntil(this.destroy$), takeUntil(this.cancelZoneTest$))
.subscribe({
next: (res) => this.zoneTest.set({ ...test, result: res }),
error: (err) =>
this.zoneTest.set({
...test,
result: { error: err.error?.error || $localize`Test failed` },
}),
})
}
private clearZoneTest() {
this.cancelZoneTest$.next()
this.zoneTest.set(null)
}
/** A result is stale once its zone, settings or sample document change. */
private isCurrentZoneTest(): boolean {
const test = this.zoneTest()
return (
test?.zone === this.selectedZone &&
test.key === this.zoneTestKey(test.zone)
)
}
private zoneTestKey(zone: OcrTemplateZone): string {
return JSON.stringify([this.previewDocId, this.zoneTestRequest(zone)])
}
private zoneTestRequest(zone: OcrTemplateZone): ZoneTestRequest {
return {
name: zone.name,
x: zone.x,
y: zone.y,
width: zone.width,
height: zone.height,
page: zone.page ?? 1,
ocr_language: zone.ocr_language,
transform: zone.transform,
date_format: zone.date_format,
validation_regex: zone.validation_regex,
zone_source_width: zone.zone_source_width,
zone_source_height: zone.zone_source_height,
}
}
save() {
if (this.saving()) return
this.stopOverlayInteraction()
this.saving.set(true)
this.pruneCombineFormats()
this.template.sample_document = this.previewDocId
const payload = this.permissionsService.currentUserOwnsObject(this.object)
? { ...this.template, ...this.permissionsForm }
: this.template
const obs = this.template.id
? this.templateService.update(payload)
: this.templateService.create(payload)
obs.pipe(takeUntil(this.destroy$)).subscribe({
next: (saved) => {
this.activeModal.close()
this.succeeded.emit(saved)
},
error: (e) => {
this.saving.set(false)
this.failed.emit(e)
},
})
}
// ng-select needs a stable array between change detection runs, so the
// split language list is cached per zone until ocr_language changes.
private readonly ocrLanguageCache = new WeakMap<
OcrTemplateZone,
{ language: string; languages: string[] }
>()
ocrLanguageArray(zone: OcrTemplateZone): string[] {
const language = zone.ocr_language || ''
const cached = this.ocrLanguageCache.get(zone)
if (cached?.language === language) return cached.languages
const languages = language.split('+').filter(Boolean)
this.ocrLanguageCache.set(zone, { language, languages })
return languages
}
setOcrLanguages(zone: OcrTemplateZone, langs: string[]) {
zone.ocr_language = (langs ?? []).join('+')
}
/** The field select's value: a built-in target, or a custom field id. */
zoneFieldValue(zone: OcrTemplateZone): ZoneFieldSelection {
return zone.target === OcrZoneTarget.CustomField
? zone.custom_field
: zone.target
}
setZoneField(zone: OcrTemplateZone, value: ZoneFieldSelection) {
if (typeof value === 'string') {
zone.target = value
zone.custom_field = null
} else {
zone.target = OcrZoneTarget.CustomField
zone.custom_field = value
}
}
zonesForField(zone: OcrTemplateZone): OcrTemplateZone[] {
const key = getZoneFieldKey(zone)
if (!key) return []
return this.template.zones.filter((z) => getZoneFieldKey(z) === key)
}
isFieldShared(zone: OcrTemplateZone): boolean {
return this.zonesForField(zone).length > 1
}
renameZone(zone: OcrTemplateZone, name: string) {
const oldName = zone.name.trim()
const format = this.getCombineFormat(zone)
if (format) {
this.setCombineFormat(
zone,
format.replace(/\{([^{}]*)\}/g, (token, tokenName: string) =>
tokenName.trim() === oldName ? `{${name}}` : token
)
)
}
zone.name = name
}
getCombineFormat(zone: OcrTemplateZone): string {
return this.template.combine_formats?.[getZoneFieldKey(zone)] ?? ''
}
setCombineFormat(zone: OcrTemplateZone, value: string) {
const key = getZoneFieldKey(zone)
if (!key) return
this.template.combine_formats ??= {}
this.template.combine_formats[key] = value
}
defaultCombineFormat(zone: OcrTemplateZone): string {
return this.zonesForField(zone)
.map((z) => `{${z.name}}`)
.join(' ')
}
insertCombineToken(zone: OcrTemplateZone, tokenZone: OcrTemplateZone) {
const current = this.getCombineFormat(zone)
const sep = current && !current.endsWith(' ') ? ' ' : ''
this.setCombineFormat(zone, `${current}${sep}{${tokenZone.name}}`)
}
/** Drop formats that are empty or whose field no longer has several zones. */
private pruneCombineFormats() {
const formats = this.template.combine_formats
if (!formats) return
const keys = this.template.zones.map(getZoneFieldKey)
for (const [key, format] of Object.entries(formats)) {
if (!format || keys.filter((k) => k === key).length < 2) {
delete formats[key]
}
}
}
createField(name?: string) {
const zone = this.selectedZone
const modal = this.modalService.open(CustomFieldEditDialogComponent)
modal.componentInstance.dialogMode.set(EditDialogMode.CREATE)
modal.componentInstance.object = { name: name || zone?.name }
modal.componentInstance.succeeded
.pipe(takeUntil(this.destroy$))
.subscribe((field: CustomField) => {
this.toastService.showSuccess($localize`Saved field "${field.name}".`)
this.customFieldsService.clearCache()
this.customFieldsService
.listAll()
.pipe(takeUntil(this.destroy$))
.subscribe((r) => {
if (zone && OCR_SUPPORTED_FIELD_TYPES.includes(field.data_type)) {
this.setZoneField(zone, field.id)
}
this.customFields.set(r.results)
})
})
modal.componentInstance.failed
.pipe(takeUntil(this.destroy$))
.subscribe((e) => {
this.toastService.showError($localize`Error saving field.`, e)
})
}
ngOnDestroy() {
this.previewResizeObserver?.disconnect()
this.cancelZoneTest$.complete()
this.destroy$.next()
this.destroy$.complete()
}
}
@@ -0,0 +1,197 @@
import {
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTransform,
} from 'src/app/data/ocr-template'
import {
findHandleAt,
findZoneAt,
getResizeHandles,
getZoneDisplayRect,
getZonePage,
isZoneOnPage,
moveZone,
resizeZone,
} from './zone-geometry'
function zone(overrides: Partial<OcrTemplateZone> = {}): OcrTemplateZone {
return {
name: 'Zone',
target: OcrZoneTarget.CustomField,
custom_field: 1,
x: 100,
y: 200,
width: 300,
height: 400,
page: 1,
ocr_language: 'eng',
transform: OcrZoneTransform.Strip,
validation_regex: '',
order: 0,
...overrides,
}
}
describe('OCR template editor geometry', () => {
it('normalizes zone pages', () => {
expect(getZonePage(zone({ page: 2 }), 0, 5)).toBe(2)
expect(getZonePage(zone({ page: -1 }), 0, 5)).toBe(5)
expect(getZonePage(zone({ page: -1 }), 2, null)).toBe(3)
expect(getZonePage(zone({ page: 0 }), 0, 5)).toBe(1)
expect(getZonePage(zone({ page: undefined }), 0, 5)).toBe(1)
})
it('checks whether a zone is on the current preview page', () => {
expect(isZoneOnPage(zone({ page: 2 }), 1, 5)).toBe(true)
expect(isZoneOnPage(zone({ page: 2 }), 0, 5)).toBe(false)
expect(isZoneOnPage(zone({ page: -1 }), 4, 5)).toBe(true)
})
it('uses natural image coordinates when source dimensions are unset', () => {
expect(
getZoneDisplayRect(zone({ x: 100, y: 200, width: 300, height: 400 }), {
width: 1000,
height: 2000,
})
).toEqual({ x: 100, y: 200, w: 300, h: 400 })
})
it('uses per-zone source dimensions when present', () => {
expect(
getZoneDisplayRect(
zone({
x: 100,
y: 100,
width: 100,
height: 100,
zone_source_width: 1000,
zone_source_height: 1000,
}),
{ width: 500, height: 750 }
)
).toEqual({ x: 50, y: 75, w: 50, h: 75 })
})
it('finds zones from topmost to bottommost on the current page', () => {
const zones = [
zone({ name: 'first', x: 0, y: 0, width: 100, height: 100, page: 1 }),
zone({ name: 'second', x: 0, y: 0, width: 50, height: 50, page: 1 }),
zone({ name: 'third', x: 0, y: 0, width: 50, height: 50, page: 2 }),
]
expect(
findZoneAt({ x: 25, y: 25 }, zones, 0, 2, { width: 100, height: 100 })
).toBe(1)
})
it('finds resize handles around a display rect', () => {
const rect = { x: 10, y: 20, w: 100, h: 200 }
expect(findHandleAt({ x: 10, y: 20 }, rect)).toBe('nw')
expect(findHandleAt({ x: 110, y: 220 }, rect)).toBe('se')
expect(findHandleAt({ x: 60, y: 20 }, rect)).toBe('n')
expect(findHandleAt({ x: 90, y: 160 }, rect)).toBeNull()
expect(findHandleAt({ x: 19, y: 20 }, rect, 8)).toBeNull()
expect(findHandleAt({ x: 19, y: 20 }, rect, 10)).toBe('nw')
})
it('returns no zone when the point is outside every rectangle', () => {
expect(
findZoneAt({ x: 900, y: 900 }, [zone()], 0, 1, {
width: 1000,
height: 1000,
})
).toBeNull()
})
it('resizes north-west edges while keeping the opposite corner fixed', () => {
const z = zone({ x: 100, y: 200, width: 300, height: 400 })
resizeZone(z, 'nw', { x: 50, y: 100 }, { width: 1000, height: 1000 })
expect(z).toMatchObject({ x: 50, y: 100, width: 350, height: 500 })
resizeZone(z, 'nw', { x: 900, y: 900 }, { width: 1000, height: 1000 })
expect(z).toMatchObject({ x: 390, y: 590, width: 10, height: 10 })
})
it('clamps resized north-west edges to the source image bounds', () => {
const z = zone({
x: 50,
y: 50,
width: 100,
height: 100,
zone_source_width: 200,
zone_source_height: 300,
})
resizeZone(z, 'nw', { x: -50, y: -50 }, { width: 400, height: 600 })
expect(z).toMatchObject({ x: 0, y: 0, width: 150, height: 150 })
})
it('uses the rendered handle positions for hit testing', () => {
const rect = { x: 10, y: 20, w: 100, h: 200 }
const handles = getResizeHandles(rect)
expect(handles).toEqual([
{ handle: 'nw', x: 10, y: 20 },
{ handle: 'n', x: 60, y: 20 },
{ handle: 'ne', x: 110, y: 20 },
{ handle: 'w', x: 10, y: 120 },
{ handle: 'e', x: 110, y: 120 },
{ handle: 'sw', x: 10, y: 220 },
{ handle: 's', x: 60, y: 220 },
{ handle: 'se', x: 110, y: 220 },
])
for (const marker of handles) {
expect(findHandleAt(marker, rect)).toBe(marker.handle)
}
})
it('moves zones without leaving source image bounds', () => {
const z = zone({ x: 50, y: 50, width: 100, height: 100 })
moveZone(
z,
{ x: 500, y: 500 },
{ mouseX: 50, mouseY: 50, zoneX: 50, zoneY: 50 },
{ width: 500, height: 500 }
)
expect(z.x).toBe(400)
expect(z.y).toBe(400)
})
it('resizes zones without leaving source image bounds', () => {
const z = zone({
x: 50,
y: 50,
width: 100,
height: 100,
zone_source_width: 200,
zone_source_height: 200,
})
resizeZone(z, 'se', { x: 500, y: 500 }, { width: 500, height: 500 })
expect(z.width).toBe(150)
expect(z.height).toBe(150)
})
it('moves zones in their original source coordinates on a different-size preview', () => {
const z = zone({
x: 50,
y: 50,
width: 100,
height: 100,
zone_source_width: 1000,
zone_source_height: 2000,
})
moveZone(
z,
{ x: 75, y: 75 },
{ mouseX: 25, mouseY: 25, zoneX: 50, zoneY: 50 },
{ width: 500, height: 500 }
)
expect(z.x).toBe(150)
expect(z.y).toBe(250)
})
})
@@ -0,0 +1,204 @@
import { OcrTemplateZone } from 'src/app/data/ocr-template'
export interface DrawingRect {
startX: number
startY: number
endX: number
endY: number
}
export interface Dimensions {
width: number
height: number
}
export interface Point {
x: number
y: number
}
export interface DisplayRect {
x: number
y: number
w: number
h: number
}
export interface MoveStart {
mouseX: number
mouseY: number
zoneX: number
zoneY: number
}
export type ResizeHandle = 'n' | 's' | 'e' | 'w' | 'ne' | 'nw' | 'se' | 'sw'
export interface ResizeHandleMarker extends Point {
handle: ResizeHandle
}
export const HANDLE_SIZE = 8
export const MIN_ZONE_SIZE = 10
const ZONE_COLORS = [
'#4f8ff7',
'#ff6b6b',
'#51cf66',
'#ffd43b',
'#cc5de8',
'#ff922b',
'#20c997',
'#e599f7',
]
/** Outline color for a zone, shared by the preview overlay and the zone list. */
export function getZoneColor(index: number): string {
return ZONE_COLORS[index % ZONE_COLORS.length]
}
export function getZonePage(
zone: OcrTemplateZone,
previewPage: number,
previewPageCount: number | null
): number {
const page = zone.page ?? 1
if (page === -1) return previewPageCount ?? previewPage + 1
return Math.max(page, 1)
}
export function isZoneOnPage(
zone: OcrTemplateZone,
previewPage: number,
previewPageCount: number | null
): boolean {
return getZonePage(zone, previewPage, previewPageCount) === previewPage + 1
}
function getZoneSourceSize(
zone: OcrTemplateZone,
imageSize: Dimensions
): Dimensions {
return {
width: zone.zone_source_width || imageSize.width,
height: zone.zone_source_height || imageSize.height,
}
}
export function getZoneDisplayRect(
zone: OcrTemplateZone,
imageSize: Dimensions
): DisplayRect {
const sourceSize = getZoneSourceSize(zone, imageSize)
// The SVG viewBox uses the preview image's natural dimensions.
const scaleX = imageSize.width / sourceSize.width
const scaleY = imageSize.height / sourceSize.height
return {
x: zone.x * scaleX,
y: zone.y * scaleY,
w: zone.width * scaleX,
h: zone.height * scaleY,
}
}
export function getResizeHandles(rect: DisplayRect): ResizeHandleMarker[] {
return [
{ handle: 'nw', x: rect.x, y: rect.y },
{ handle: 'n', x: rect.x + rect.w / 2, y: rect.y },
{ handle: 'ne', x: rect.x + rect.w, y: rect.y },
{ handle: 'w', x: rect.x, y: rect.y + rect.h / 2 },
{ handle: 'e', x: rect.x + rect.w, y: rect.y + rect.h / 2 },
{ handle: 'sw', x: rect.x, y: rect.y + rect.h },
{ handle: 's', x: rect.x + rect.w / 2, y: rect.y + rect.h },
{ handle: 'se', x: rect.x + rect.w, y: rect.y + rect.h },
]
}
export function findHandleAt(
point: Point,
rect: DisplayRect,
handleSize = HANDLE_SIZE
): ResizeHandle | null {
return (
getResizeHandles(rect).find(
({ x, y }) =>
Math.abs(point.x - x) <= handleSize &&
Math.abs(point.y - y) <= handleSize
)?.handle ?? null
)
}
export function findZoneAt(
point: Point,
zones: OcrTemplateZone[],
previewPage: number,
previewPageCount: number | null,
imageSize: Dimensions
): number | null {
for (let i = zones.length - 1; i >= 0; i--) {
const zone = zones[i]
if (!isZoneOnPage(zone, previewPage, previewPageCount)) continue
const rect = getZoneDisplayRect(zone, imageSize)
if (
point.x >= rect.x &&
point.x <= rect.x + rect.w &&
point.y >= rect.y &&
point.y <= rect.y + rect.h
) {
return i
}
}
return null
}
export function moveZone(
zone: OcrTemplateZone,
point: Point,
moveStart: MoveStart,
imageSize: Dimensions
) {
const sourceSize = getZoneSourceSize(zone, imageSize)
const scaleX = sourceSize.width / imageSize.width
const scaleY = sourceSize.height / imageSize.height
const dx = Math.round((point.x - moveStart.mouseX) * scaleX)
const dy = Math.round((point.y - moveStart.mouseY) * scaleY)
zone.x = clamp(moveStart.zoneX + dx, 0, sourceSize.width - zone.width)
zone.y = clamp(moveStart.zoneY + dy, 0, sourceSize.height - zone.height)
}
export function resizeZone(
zone: OcrTemplateZone,
handle: ResizeHandle,
point: Point,
imageSize: Dimensions
) {
const sourceSize = getZoneSourceSize(zone, imageSize)
const scaleX = sourceSize.width / imageSize.width
const scaleY = sourceSize.height / imageSize.height
const imageX = clamp(Math.round(point.x * scaleX), 0, sourceSize.width)
const imageY = clamp(Math.round(point.y * scaleY), 0, sourceSize.height)
if (handle.includes('w')) {
const right = Math.min(zone.x + zone.width, sourceSize.width)
zone.x = clamp(imageX, 0, right - MIN_ZONE_SIZE)
zone.width = right - zone.x
}
if (handle.includes('e')) {
zone.width = Math.max(MIN_ZONE_SIZE, imageX - zone.x)
}
if (handle.includes('n')) {
const bottom = Math.min(zone.y + zone.height, sourceSize.height)
zone.y = clamp(imageY, 0, bottom - MIN_ZONE_SIZE)
zone.height = bottom - zone.y
}
if (handle.includes('s')) {
zone.height = Math.max(MIN_ZONE_SIZE, imageY - zone.y)
}
}
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(value, max))
}
@@ -0,0 +1,75 @@
<pngx-page-header
title="OCR Templates"
i18n-title
info="Define extraction zones on document types to automatically populate custom fields via OCR."
i18n-info
>
<button type="button" class="btn btn-sm btn-outline-primary" (click)="editTemplate()" *pngxIfPermissions="{ action: PermissionAction.Add, type: PermissionType.OcrTemplate }">
<pngx-bs-icon name="plus-circle" class="me-1"></pngx-bs-icon><ng-container i18n>Add Template</ng-container>
</button>
</pngx-page-header>
<ul class="list-group">
<li class="list-group-item">
<div class="row">
<div class="col" i18n>Name</div>
<div class="col d-none d-sm-flex" i18n>Matching</div>
<div class="col d-none d-sm-flex" i18n>Zones</div>
<div class="col" i18n>Status</div>
<div class="col" i18n>Actions</div>
</div>
</li>
@if (loading() && templates().length === 0) {
<li class="list-group-item">
<output class="spinner-border spinner-border-sm me-2"></output>
<ng-container i18n>Loading...</ng-container>
</li>
}
@for (template of templates(); track template.id) {
<li class="list-group-item">
<div class="row fade" [class.show]="show()">
<div class="col d-flex align-items-center"><button class="btn btn-link p-0 text-start" type="button" (click)="editTemplate(template)" [disabled]="!userCanEdit(template)">{{template.name}}</button></div>
<div class="col d-flex align-items-center d-none d-sm-flex">{{getMatching(template)}}</div>
<div class="col d-flex align-items-center d-none d-sm-flex"><code>{{template.zones.length}}</code></div>
<div class="col d-flex align-items-center">
<div class="form-check form-switch mb-0">
<input type="checkbox" class="form-check-input cursor-pointer" [id]="template.id+'_enable'" [(ngModel)]="template.enabled" (change)="toggleTemplateEnabled(template)" [disabled]="!userCanEdit(template)" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.OcrTemplate }">
<label class="form-check-label cursor-pointer" [for]="template.id+'_enable'">
<code> @if(template.enabled) { <ng-container i18n>Enabled</ng-container> } @else { <span i18n class="text-muted">Disabled</span> }</code>
</label>
</div>
</div>
<div class="col">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile{{template.id}}" (click)="$event.stopPropagation()" ngbDropdownToggle>
<pngx-bs-icon name="three-dots-vertical"></pngx-bs-icon>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile{{template.id}}">
<button (click)="editTemplate(template)" [disabled]="!userCanEdit(template)" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.OcrTemplate }" ngbDropdownItem i18n>Edit</button>
<button (click)="deleteTemplate(template)" [disabled]="!permissionsService.currentUserOwnsObject(template)" *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.OcrTemplate }" ngbDropdownItem i18n>Delete</button>
</div>
</div>
</div>
<div class="btn-toolbar d-none d-sm-flex gap-2" role="toolbar">
<div class="btn-group">
<button *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.OcrTemplate }" [disabled]="!userCanEdit(template)" class="btn btn-sm btn-outline-secondary" type="button" (click)="editTemplate(template)">
<pngx-bs-icon width="1em" height="1em" name="pencil" class="me-1"></pngx-bs-icon><ng-container i18n>Edit</ng-container>
</button>
<button *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.OcrTemplate }" [disabled]="!permissionsService.currentUserOwnsObject(template)" class="btn btn-sm btn-outline-danger" type="button" (click)="deleteTemplate(template)">
<pngx-bs-icon width="1em" height="1em" name="trash" class="me-1"></pngx-bs-icon><ng-container i18n>Delete</ng-container>
</button>
</div>
</div>
</div>
</div>
</li>
}
@if (!loading() && templates().length === 0) {
<li class="list-group-item" [class.show]="show()" i18n>No OCR templates defined.</li>
}
</ul>
@@ -0,0 +1,300 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { By } from '@angular/platform-browser'
import { NgbModal, NgbModalRef } from '@ng-bootstrap/ng-bootstrap'
import { of, throwError } from 'rxjs'
import { MATCH_LITERAL, MATCH_NONE } from 'src/app/data/matching-model'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { User } from 'src/app/data/user'
import { DirtyFormGuard } from 'src/app/guards/dirty-form.guard'
import { PermissionsService } from 'src/app/services/permissions.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { UserService } from 'src/app/services/rest/user.service'
import { ToastService } from 'src/app/services/toast.service'
import { ConfirmDialogComponent } from '../../common/confirm-dialog/confirm-dialog.component'
import { OcrTemplateEditorComponent } from './ocr-template-editor/ocr-template-editor.component'
import { OcrTemplatesComponent } from './ocr-templates.component'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
const templates: OcrTemplate[] = [
{
id: 1,
name: 'Invoices',
match: 'Northwind',
matching_algorithm: MATCH_LITERAL,
sample_document: null,
source_width: 0,
source_height: 0,
enabled: true,
zones: [],
},
]
describe('OcrTemplatesComponent', () => {
let component: OcrTemplatesComponent
let fixture: ComponentFixture<OcrTemplatesComponent>
let ocrTemplateService: OcrTemplateService
let modalService: NgbModal
let toastService: ToastService
let modal: NgbModalRef
beforeEach(() => {
TestBed.configureTestingModule({
imports: [OcrTemplatesComponent, BootstrapIconComponent],
providers: [
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
})
TestBed.inject(PermissionsService).initialize(
[
'view_ocrtemplate',
'add_ocrtemplate',
'change_ocrtemplate',
'delete_ocrtemplate',
],
{ id: 1, groups: [], is_superuser: false } as User
)
jest
.spyOn(TestBed.inject(UserService), 'listAll')
.mockReturnValue(of({ count: 0, all: [], results: [] }))
ocrTemplateService = TestBed.inject(OcrTemplateService)
jest
.spyOn(ocrTemplateService, 'listAll')
.mockReturnValue(of({ count: 1, all: [1], results: templates }))
modalService = TestBed.inject(NgbModal)
modalService.activeInstances.subscribe((m) => (modal = m[m.length - 1]))
toastService = TestBed.inject(ToastService)
fixture = TestBed.createComponent(OcrTemplatesComponent)
component = fixture.componentInstance
fixture.detectChanges()
})
afterEach(() => modalService.dismissAll())
it('should show templates with their matching', () => {
const text = fixture.nativeElement.textContent
expect(text).toContain('Invoices')
expect(text).toContain('Exact match: Northwind')
expect(
component.getMatching({ ...templates[0], matching_algorithm: MATCH_NONE })
).toEqual('None')
expect(component.getMatching({ ...templates[0], match: '' })).toEqual('-')
})
it.each([
{
name: 'owned',
owner: 1,
user_can_change: true,
editable: true,
deletable: true,
},
{
name: 'unowned',
owner: null,
user_can_change: true,
editable: true,
deletable: true,
},
{
name: 'view-only shared',
owner: 2,
user_can_change: false,
editable: false,
deletable: false,
},
{
name: 'editable shared',
owner: 2,
user_can_change: true,
editable: true,
deletable: false,
},
])(
'should respect permissions for $name templates',
async ({ owner, user_can_change, editable, deletable }) => {
const template = { ...templates[0], owner, user_can_change }
component.templates.set([template])
fixture.detectChanges()
await fixture.whenStable()
expect(fixture.nativeElement.querySelector('.text-start').disabled).toBe(
!editable
)
expect(
fixture.nativeElement.querySelector('.form-switch input').disabled
).toBe(!editable)
const buttons = Array.from<HTMLButtonElement>(
fixture.nativeElement.querySelectorAll('button')
)
for (const button of buttons.filter(
(button) => button.textContent.trim() === 'Edit'
)) {
expect(button.disabled).toBe(!editable)
}
for (const button of buttons.filter(
(button) => button.textContent.trim() === 'Delete'
)) {
expect(button.disabled).toBe(!deletable)
}
}
)
it('should allow a superuser to manage another users template', () => {
TestBed.inject(PermissionsService).initialize([], {
id: 1,
is_superuser: true,
} as User)
component.templates.set([
{ ...templates[0], owner: 2, user_can_change: false },
])
fixture.detectChanges()
expect(fixture.nativeElement.querySelector('.text-start').disabled).toBe(
false
)
expect(
fixture.nativeElement.querySelector('.btn-outline-danger').disabled
).toBe(false)
})
it('should not open mutation dialogs or patch a view-only template', () => {
const template = { ...templates[0], owner: 2, user_can_change: false }
const open = jest.spyOn(modalService, 'open')
const patch = jest.spyOn(ocrTemplateService, 'patch')
component.editTemplate(template)
component.toggleTemplateEnabled(template)
component.deleteTemplate(template)
expect(open).not.toHaveBeenCalled()
expect(patch).not.toHaveBeenCalled()
})
it('should require global edit permission even for an owned template', () => {
TestBed.inject(PermissionsService).initialize(['view_ocrtemplate'], {
id: 1,
} as User)
fixture.destroy()
fixture = TestBed.createComponent(OcrTemplatesComponent)
component = fixture.componentInstance
fixture.detectChanges()
expect(fixture.nativeElement.querySelector('.text-start').disabled).toBe(
true
)
expect(fixture.nativeElement.querySelector('.form-switch input')).toBeNull()
})
it('should support create, show notification on error / success', () => {
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSuccessSpy = jest.spyOn(toastService, 'showSuccess')
const reloadSpy = jest.spyOn(component, 'reload')
fixture.debugElement
.queryAll(By.css('pngx-page-header button'))
.find((button) => button.nativeElement.textContent.includes('Add'))
.triggerEventHandler('click')
const editor = modal.componentInstance as OcrTemplateEditorComponent
expect(editor).toBeInstanceOf(OcrTemplateEditorComponent)
expect(editor.object).toBeUndefined()
editor.failed.emit({ error: 'error creating template' })
expect(toastErrorSpy).toHaveBeenCalled()
expect(reloadSpy).not.toHaveBeenCalled()
editor.succeeded.emit(templates[0])
expect(toastSuccessSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalled()
})
it('should pass the template to the editor', () => {
component.editTemplate(templates[0])
expect(modal.componentInstance.object).toBe(templates[0])
})
it('should confirm closing the editor with unsaved changes', async () => {
const confirmSpy = jest
.spyOn(TestBed.inject(DirtyFormGuard), 'confirmChanges')
.mockReturnValue(of(false))
component.editTemplate()
const editor = modal.componentInstance as OcrTemplateEditorComponent
editor.ngOnInit()
modal.dismiss()
await fixture.whenStable()
expect(confirmSpy).not.toHaveBeenCalled()
component.editTemplate()
const dirtyEditor = modal.componentInstance as OcrTemplateEditorComponent
dirtyEditor.ngOnInit()
dirtyEditor.template.name = 'Changed'
modal.dismiss()
await fixture.whenStable()
expect(confirmSpy).toHaveBeenCalled()
expect(modalService.hasOpenModals()).toBe(true)
dirtyEditor.saving.set(true)
confirmSpy.mockClear()
modal.dismiss()
await fixture.whenStable()
expect(confirmSpy).not.toHaveBeenCalled()
expect(modalService.hasOpenModals()).toBe(true)
})
it('should only send the enabled flag when toggling', () => {
const patchSpy = jest
.spyOn(ocrTemplateService, 'patch')
.mockReturnValueOnce(throwError(() => new Error('error toggling')))
.mockReturnValueOnce(of(templates[0]))
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSuccessSpy = jest.spyOn(toastService, 'showSuccess')
const reloadSpy = jest.spyOn(component, 'reload')
const toggle = fixture.debugElement.query(By.css('.form-switch input'))
toggle.nativeElement.click()
expect(patchSpy).toHaveBeenCalledWith({ id: 1, enabled: false })
expect(toastErrorSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalledTimes(1)
component.toggleTemplateEnabled(templates[0])
expect(toastSuccessSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalledTimes(2)
})
it('should support delete, show notification on error / success', () => {
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSuccessSpy = jest.spyOn(toastService, 'showSuccess')
const deleteSpy = jest.spyOn(ocrTemplateService, 'delete')
const reloadSpy = jest.spyOn(component, 'reload')
component.deleteTemplate(templates[0])
const dialog = modal.componentInstance as ConfirmDialogComponent
deleteSpy.mockReturnValueOnce(throwError(() => new Error('error deleting')))
dialog.confirmClicked.emit()
expect(toastErrorSpy).toHaveBeenCalled()
expect(dialog.buttonsEnabled()).toBe(true)
expect(reloadSpy).not.toHaveBeenCalled()
deleteSpy.mockReturnValueOnce(of(true))
dialog.confirmClicked.emit()
expect(toastSuccessSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalled()
})
it.each([true, false])(
'reports the enabled state %s after the update succeeds',
(enabled) => {
const template = { ...templates[0], enabled }
jest.spyOn(ocrTemplateService, 'patch').mockReturnValue(of(template))
const toast = jest.spyOn(toastService, 'showSuccess')
component.toggleTemplateEnabled(template)
expect(toast).toHaveBeenCalledWith(
`${enabled ? 'Enabled' : 'Disabled'} OCR template "${template.name}"`
)
}
)
})
@@ -0,0 +1,185 @@
import { Component, OnInit, inject, signal } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { NgbDropdownModule, NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { firstValueFrom, takeUntil, tap } from 'rxjs'
import { MATCH_NONE, MATCHING_ALGORITHMS } from 'src/app/data/matching-model'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { DirtyFormGuard } from 'src/app/guards/dirty-form.guard'
import {
PermissionAction,
PermissionType,
PermissionsService,
} from 'src/app/services/permissions.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { ToastService } from 'src/app/services/toast.service'
import { ConfirmDialogComponent } from '../../common/confirm-dialog/confirm-dialog.component'
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
import { OcrTemplateEditorComponent } from './ocr-template-editor/ocr-template-editor.component'
@Component({
selector: 'pngx-ocr-templates',
templateUrl: './ocr-templates.component.html',
imports: [
PageHeaderComponent,
IfPermissionsDirective,
FormsModule,
NgbDropdownModule,
BootstrapIconComponent,
],
})
export class OcrTemplatesComponent
extends LoadingComponentWithPermissions
implements OnInit
{
private readonly ocrTemplateService = inject(OcrTemplateService)
public readonly permissionsService = inject(PermissionsService)
private readonly modalService = inject(NgbModal)
private readonly dirtyFormGuard = inject(DirtyFormGuard)
private readonly toastService = inject(ToastService)
public readonly templates = signal<OcrTemplate[]>([])
ngOnInit() {
this.reload()
}
reload() {
this.loading.set(true)
this.ocrTemplateService
.listAll()
.pipe(
takeUntil(this.unsubscribeNotifier),
tap((r) => this.templates.set(r.results))
)
.subscribe(() => {
this.show.set(true)
this.loading.set(false)
})
}
getMatching(template: OcrTemplate): string {
if (template.matching_algorithm === MATCH_NONE) {
return $localize`None`
}
if (!template.match) {
return '-'
}
const algorithm = MATCHING_ALGORITHMS.find(
(a) => a.id === template.matching_algorithm
)
return `${algorithm?.shortName}: ${template.match}`
}
editTemplate(template?: OcrTemplate) {
if (template && !this.userCanEdit(template)) return
const modal = this.modalService.open(OcrTemplateEditorComponent, {
backdrop: 'static',
keyboard: false,
size: 'xxl',
fullscreen: 'lg',
scrollable: true,
ariaLabelledBy: 'ocr-template-editor-title',
beforeDismiss: () => {
const editor: OcrTemplateEditorComponent = modal.componentInstance
if (editor.saving()) return false
if (!editor.isDirty) return true
return firstValueFrom(
this.dirtyFormGuard.confirmChanges(undefined, $localize`Close`)
)
},
})
modal.componentInstance.object = template
modal.componentInstance.succeeded
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((saved: OcrTemplate) => {
this.toastService.showSuccess(
$localize`Saved OCR template "${saved.name}".`
)
this.ocrTemplateService.clearCache()
this.reload()
})
modal.componentInstance.failed
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((e) => {
this.toastService.showError($localize`Error saving OCR template.`, e)
})
}
toggleTemplateEnabled(template: OcrTemplate) {
if (!this.userCanEdit(template)) return
this.ocrTemplateService
.patch({ id: template.id, enabled: template.enabled } as OcrTemplate)
.subscribe({
next: () => {
this.toastService.showSuccess(
template.enabled
? $localize`Enabled OCR template "${template.name}"`
: $localize`Disabled OCR template "${template.name}"`
)
this.reload()
},
error: (e) => {
this.toastService.showError(
$localize`Error toggling OCR template "${template.name}".`,
e
)
this.reload()
},
})
}
deleteTemplate(template: OcrTemplate) {
if (
!this.permissionsService.currentUserCan(
PermissionAction.Delete,
PermissionType.OcrTemplate
) ||
!this.permissionsService.currentUserOwnsObject(template)
)
return
const modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Confirm delete OCR template`
modal.componentInstance.messageBold = $localize`This operation will permanently delete this OCR template.`
modal.componentInstance.message = $localize`This operation cannot be undone.`
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false)
this.ocrTemplateService.delete(template).subscribe({
next: () => {
modal.close()
this.toastService.showSuccess(
$localize`Deleted OCR template "${template.name}".`
)
this.ocrTemplateService.clearCache()
this.reload()
},
error: (e) => {
modal.componentInstance.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Error deleting OCR template "${template.name}".`,
e
)
},
})
})
}
userCanEdit(template: OcrTemplate): boolean {
return (
this.permissionsService.currentUserCan(
PermissionAction.Change,
PermissionType.OcrTemplate
) &&
this.permissionsService.currentUserHasObjectPermissions(
PermissionAction.Change,
template
)
)
}
}
+105
View File
@@ -0,0 +1,105 @@
import { CustomFieldDataType } from './custom-field'
import { MatchingModel } from './matching-model'
import { ObjectWithPermissions } from './object-with-permissions'
export enum OcrZoneTarget {
CustomField = 'custom_field',
Title = 'title',
Asn = 'asn',
Created = 'created',
}
export enum OcrZoneTransform {
None = 'none',
Strip = 'strip',
Uppercase = 'uppercase',
Lowercase = 'lowercase',
Numeric = 'numeric',
StripPunctuation = 'strip_punctuation',
Date = 'date',
QrCode = 'qr_code',
}
export const DEFAULT_OCR_ZONE_LANGUAGE = 'eng'
/** The field a zone writes to, keyed by OcrTemplate.combine_formats. */
export function getZoneFieldKey(zone: OcrTemplateZone): string | null {
if (zone.target !== OcrZoneTarget.CustomField) return zone.target
return zone.custom_field === null ? null : String(zone.custom_field)
}
/** Custom field types zone OCR can write to. See models.py OCR_SUPPORTED_FIELD_TYPES. */
export const OCR_SUPPORTED_FIELD_TYPES: CustomFieldDataType[] = [
CustomFieldDataType.String,
CustomFieldDataType.Url,
CustomFieldDataType.Date,
CustomFieldDataType.Integer,
CustomFieldDataType.Float,
CustomFieldDataType.Monetary,
CustomFieldDataType.LongText,
CustomFieldDataType.Boolean,
]
export const OCR_BUILTIN_TARGETS = [
{ id: OcrZoneTarget.Title, name: $localize`Title` },
{ id: OcrZoneTarget.Asn, name: $localize`Archive serial number` },
{ id: OcrZoneTarget.Created, name: $localize`Date created` },
]
export interface OcrTemplateZone {
id?: number
name: string
target: OcrZoneTarget
custom_field: number | null
page?: number
x: number
y: number
width: number
height: number
ocr_language: string
transform: OcrZoneTransform
date_format?: string
validation_regex: string
order: number
zone_source_width?: number
zone_source_height?: number
}
export interface OcrTemplate extends MatchingModel, ObjectWithPermissions {
name: string
sample_document: number | null
source_width: number
source_height: number
enabled: boolean
combine_formats?: Record<string, string>
created?: string
updated?: string
zones: OcrTemplateZone[]
}
export interface ZoneTestRequest {
name: string
x: number
y: number
width: number
height: number
page: number
ocr_language: string
transform: OcrZoneTransform
date_format?: string
validation_regex: string
zone_source_width?: number
zone_source_height?: number
}
export interface OcrZoneTestResult {
raw_text?: string | null
value?: string | null
regex?: string
regex_match?: boolean | null
error?: string
}
export type OcrZoneRunResponse =
| { result: 'OK'; task_id: string }
| { result: 'skipped'; reason: 'no_template' }
+1
View File
@@ -35,6 +35,7 @@ export enum HideableSidebarItemID {
SavedViews = 'saved_views', SavedViews = 'saved_views',
ShareLinks = 'share_links', ShareLinks = 'share_links',
Workflows = 'workflows', Workflows = 'workflows',
OcrTemplates = 'ocr_templates',
Mail = 'mail', Mail = 'mail',
Documentation = 'documentation', Documentation = 'documentation',
} }
+6 -2
View File
@@ -3,12 +3,16 @@ import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { DirtyCheckGuard } from '@ngneat/dirty-check-forms' import { DirtyCheckGuard } from '@ngneat/dirty-check-forms'
import { Observable, Subject } from 'rxjs' import { Observable, Subject } from 'rxjs'
import { ConfirmDialogComponent } from 'src/app/components/common/confirm-dialog/confirm-dialog.component' import { ConfirmDialogComponent } from 'src/app/components/common/confirm-dialog/confirm-dialog.component'
import { ActivatedRouteSnapshot } from '@angular/router'
@Injectable({ providedIn: 'root' }) @Injectable({ providedIn: 'root' })
export class DirtyFormGuard extends DirtyCheckGuard { export class DirtyFormGuard extends DirtyCheckGuard {
private modalService = inject(NgbModal) private modalService = inject(NgbModal)
confirmChanges(): Observable<boolean> { confirmChanges(
_currentRoute?: ActivatedRouteSnapshot,
leaveLabel: string = $localize`Leave page`
): Observable<boolean> {
let modal = this.modalService.open(ConfirmDialogComponent, { let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static', backdrop: 'static',
}) })
@@ -16,7 +20,7 @@ export class DirtyFormGuard extends DirtyCheckGuard {
modal.componentInstance.messageBold = $localize`You have unsaved changes.` modal.componentInstance.messageBold = $localize`You have unsaved changes.`
modal.componentInstance.message = $localize`Are you sure you want to leave?` modal.componentInstance.message = $localize`Are you sure you want to leave?`
modal.componentInstance.btnClass = 'btn-warning' modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Leave page` modal.componentInstance.btnCaption = leaveLabel
modal.componentInstance.confirmClicked.subscribe(() => { modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false) modal.componentInstance.buttonsEnabled.set(false)
modal.close() modal.close()
@@ -283,6 +283,10 @@ describe('PermissionsService', () => {
'view_workflow', 'view_workflow',
'change_workflow', 'change_workflow',
'delete_workflow', 'delete_workflow',
'add_ocrtemplate',
'view_ocrtemplate',
'change_ocrtemplate',
'delete_ocrtemplate',
'add_customfield', 'add_customfield',
'view_customfield', 'view_customfield',
'change_customfield', 'change_customfield',
@@ -29,6 +29,7 @@ export enum PermissionType {
ShareLinkBundle = '%s_sharelinkbundle', ShareLinkBundle = '%s_sharelinkbundle',
CustomField = '%s_customfield', CustomField = '%s_customfield',
Workflow = '%s_workflow', Workflow = '%s_workflow',
OcrTemplate = '%s_ocrtemplate',
ProcessedMail = '%s_processedmail', ProcessedMail = '%s_processedmail',
GlobalStatistics = '%s_global_statistics', GlobalStatistics = '%s_global_statistics',
SystemMonitoring = '%s_system_monitoring', SystemMonitoring = '%s_system_monitoring',
@@ -323,6 +323,27 @@ describe(`DocumentService`, () => {
}) })
}) })
it.each([null, 3])('should queue zone OCR with template %s', (template) => {
const queued = jest.fn()
subscription = service
.runZoneOcrDocuments({ documents: [1, 2] }, template)
.subscribe(queued)
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/run_zone_ocr/`
)
expect(req.request.method).toBe('POST')
expect(req.request.body).toEqual({
documents: [1, 2],
...(template ? { template } : {}),
from_webui: true,
})
req.flush({ result: 'OK', task_id: 'zone-ocr-task' })
expect(queued).toHaveBeenCalledWith({
result: 'OK',
task_id: 'zone-ocr-task',
})
})
it('should call appropriate api endpoint for rotate documents', () => { it('should call appropriate api endpoint for rotate documents', () => {
const ids = [1, 2, 3] const ids = [1, 2, 3]
subscription = service.rotateDocuments({ documents: ids }, 90).subscribe() subscription = service.rotateDocuments({ documents: ids }, 90).subscribe()
@@ -574,6 +595,20 @@ describe(`DocumentService`, () => {
req.flush('task-id') req.flush('task-id')
}) })
it('should call appropriate api endpoint for running zone OCR', () => {
const url = `${environment.apiBaseUrl}${endpoint}/${documents[0].id}/run-zone-ocr/`
subscription = service.runZoneOcr(documents[0].id).subscribe()
let req = httpTestingController.expectOne(url)
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({ from_webui: true })
req.flush({ result: 'OK', task_id: 'zone-ocr-1' })
subscription = service.runZoneOcr(documents[0].id, 3, 10).subscribe()
req = httpTestingController.expectOne(`${url}?version=10`)
expect(req.request.body).toEqual({ template: 3, from_webui: true })
req.flush({ result: 'OK', task_id: 'zone-ocr-2' })
})
}) })
it('should construct sort fields respecting permissions', () => { it('should construct sort fields respecting permissions', () => {
@@ -12,6 +12,7 @@ import {
import { DocumentMetadata } from 'src/app/data/document-metadata' import { DocumentMetadata } from 'src/app/data/document-metadata'
import { DocumentSuggestions } from 'src/app/data/document-suggestions' import { DocumentSuggestions } from 'src/app/data/document-suggestions'
import { FilterRule } from 'src/app/data/filter-rule' import { FilterRule } from 'src/app/data/filter-rule'
import { OcrZoneRunResponse } from 'src/app/data/ocr-template'
import { Results, SelectionData } from 'src/app/data/results' import { Results, SelectionData } from 'src/app/data/results'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings' import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { BulkOperationResponse } from '../../data/bulk-operation' import { BulkOperationResponse } from '../../data/bulk-operation'
@@ -371,6 +372,32 @@ export class DocumentService extends AbstractPaperlessService<Document> {
) )
} }
runZoneOcrDocuments(
selection: DocumentSelectionQuery,
templateId: number | null = null
) {
return this.http.post<BulkOperationResponse>(
this.getResourceUrl(null, 'run_zone_ocr'),
{
...selection,
...(templateId ? { template: templateId } : {}),
from_webui: true,
}
)
}
runZoneOcr(
id: number,
templateId: number | null = null,
versionId: number | null = null
): Observable<OcrZoneRunResponse> {
return this.http.post<OcrZoneRunResponse>(
this.getResourceUrl(id, 'run-zone-ocr'),
{ ...(templateId ? { template: templateId } : {}), from_webui: true },
{ params: versionId ? { version: versionId.toString() } : {} }
)
}
rotateDocuments( rotateDocuments(
selection: DocumentSelectionQuery, selection: DocumentSelectionQuery,
degrees: number, degrees: number,
@@ -0,0 +1,78 @@
import { provideHttpClient } from '@angular/common/http'
import {
HttpTestingController,
provideHttpClientTesting,
} from '@angular/common/http/testing'
import { TestBed } from '@angular/core/testing'
import { OcrZoneTransform, ZoneTestRequest } from 'src/app/data/ocr-template'
import { environment } from 'src/environments/environment'
import { OcrTemplateService } from './ocr-template.service'
describe('OcrTemplateService', () => {
let service: OcrTemplateService
let http: HttpTestingController
beforeEach(() => {
TestBed.configureTestingModule({
providers: [provideHttpClient(), provideHttpClientTesting()],
})
service = TestBed.inject(OcrTemplateService)
http = TestBed.inject(HttpTestingController)
})
afterEach(() => http.verify())
it('builds page image URLs with zero-based page numbers', () => {
expect(service.getPageImageUrl(12, 0)).toBe(
`${environment.apiBaseUrl}ocr_templates/document-page-image/12/0/`
)
expect(service.getPageImageUrl(12, 2)).toBe(
`${environment.apiBaseUrl}ocr_templates/document-page-image/12/2/`
)
})
it('posts a zone test and returns the extraction result', () => {
const zone: ZoneTestRequest = {
name: 'Total',
x: 10,
y: 20,
width: 100,
height: 50,
page: -1,
ocr_language: 'eng',
transform: OcrZoneTransform.Numeric,
validation_regex: '\\d+',
zone_source_width: 1000,
zone_source_height: 1500,
}
const result = {
raw_text: 'Total: 123',
value: '123',
regex: '\\d+',
regex_match: true,
}
const next = jest.fn()
service.testZone(12, zone).subscribe(next)
const request = http.expectOne(
`${environment.apiBaseUrl}ocr_templates/test-zone/`
)
expect(request.request.method).toBe('POST')
expect(request.request.body).toEqual({ document: 12, zone })
request.flush(result)
expect(next).toHaveBeenCalledWith(result)
})
it('passes zone-test failures to the caller', () => {
const error = jest.fn()
service.testZone(12, {} as ZoneTestRequest).subscribe({ error })
http
.expectOne(`${environment.apiBaseUrl}ocr_templates/test-zone/`)
.flush(
{ error: 'Invalid zone' },
{ status: 400, statusText: 'Bad Request' }
)
expect(error).toHaveBeenCalledWith(
expect.objectContaining({ status: 400, error: { error: 'Invalid zone' } })
)
})
})
@@ -0,0 +1,30 @@
import { Injectable } from '@angular/core'
import { Observable } from 'rxjs'
import {
OcrTemplate,
OcrZoneTestResult,
ZoneTestRequest,
} from '../../data/ocr-template'
import { AbstractPaperlessService } from './abstract-paperless-service'
@Injectable({ providedIn: 'root' })
export class OcrTemplateService extends AbstractPaperlessService<OcrTemplate> {
constructor() {
super()
this.resourceName = 'ocr_templates'
}
getPageImageUrl(docId: number, page: number): string {
return `${this.baseUrl}${this.resourceName}/document-page-image/${docId}/${page}/`
}
testZone(
docId: number,
zone: ZoneTestRequest
): Observable<OcrZoneTestResult> {
return this.http.post<OcrZoneTestResult>(
`${this.baseUrl}${this.resourceName}/test-zone/`,
{ document: docId, zone }
)
}
}
@@ -1,6 +1,13 @@
import { getDocumentOperationLabel } from './document-operation' import {
documentOperationHasFollowUpTasks,
getDocumentOperationLabel,
} from './document-operation'
describe('Document Operation Utils', () => { describe('Document Operation Utils', () => {
it('names zone OCR and treats task completion as finished extraction', () => {
expect(getDocumentOperationLabel('run_zone_ocr')).toBe('Run zone OCR')
expect(documentOperationHasFollowUpTasks('run_zone_ocr')).toBe(false)
})
it('should retrieve the document operation label correctly for an existing operation, null otherwise', () => { it('should retrieve the document operation label correctly for an existing operation, null otherwise', () => {
const result = getDocumentOperationLabel('modify_custom_fields') const result = getDocumentOperationLabel('modify_custom_fields')
expect(result).toBe('Update custom fields') expect(result).toBe('Update custom fields')
@@ -10,6 +10,7 @@ const OPERATION_LABELS: Record<string, string> = {
delete: $localize`Move to trash`, delete: $localize`Move to trash`,
reprocess: $localize`Document reprocessing`, reprocess: $localize`Document reprocessing`,
run_workflow: $localize`Run workflow`, run_workflow: $localize`Run workflow`,
run_zone_ocr: $localize`Run zone OCR`,
rotate: $localize`Rotate documents`, rotate: $localize`Rotate documents`,
merge: $localize`Merge documents`, merge: $localize`Merge documents`,
merge_as_versions: $localize`Merge as versions`, merge_as_versions: $localize`Merge as versions`,
+6
View File
@@ -691,6 +691,12 @@ table.table {
} }
} }
@media (min-width: 992px) {
.modal-xxl {
--bs-modal-width: min(1440px, calc(100vw - 2 * var(--bs-modal-margin)));
}
}
.toast { .toast {
--bs-toast-max-width: var(--pngx-toast-max-width); --bs-toast-max-width: var(--pngx-toast-max-width);
} }
+77 -2
View File
@@ -33,6 +33,7 @@ from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplate
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
@@ -46,7 +47,11 @@ from documents.tasks import run_workflows_for_documents
from documents.tasks import update_document_content_maybe_archive_file from documents.tasks import update_document_content_maybe_archive_file
from documents.versioning import get_latest_version_for_root from documents.versioning import get_latest_version_for_root
from documents.versioning import get_root_document from documents.versioning import get_root_document
from documents.versioning import latest_version
from documents.versioning import versions_newest_first from documents.versioning import versions_newest_first
from documents.zone_ocr import get_templates_for_document
from documents.zone_ocr import run_zone_extraction
from documents.zone_ocr import zone_results_written
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable from collections.abc import Callable
@@ -442,7 +447,7 @@ def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
A root document with versions reprocesses its latest version, which is the A root document with versions reprocesses its latest version, which is the
file whose content, archive and thumbnail are shown for it. file whose content, archive and thumbnail are shown for it.
""" """
latest_version = versions_newest_first( newest_version_id = versions_newest_first(
Document.objects.filter(root_document=OuterRef("pk")), Document.objects.filter(root_document=OuterRef("pk")),
).values("id")[:1] ).values("id")[:1]
source_ids = ( source_ids = (
@@ -450,7 +455,7 @@ def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
.annotate( .annotate(
source_id=Case( source_id=Case(
When(root_document__isnull=False, then=F("id")), When(root_document__isnull=False, then=F("id")),
default=Coalesce(Subquery(latest_version), F("id")), default=Coalesce(Subquery(newest_version_id), F("id")),
), ),
) )
.order_by() .order_by()
@@ -1255,3 +1260,73 @@ def remove_doclink(
target_doc_field_instance.value.remove(document.id) target_doc_field_instance.value.remove(document.id)
target_doc_field_instance.save() target_doc_field_instance.save()
Document.objects.filter(id=target_doc_id).update(modified=timezone.now()) Document.objects.filter(id=target_doc_id).update(modified=timezone.now())
def run_zone_ocr(
doc_ids: list[int],
*,
template: int | None = None,
user: User | None = None,
resolve_latest: bool = True,
) -> Literal["OK"]:
"""
Run zone OCR within the tracked bulk operation task, then finish once for
the root documents that had a value written. Root documents are run on
their newest version unless resolve_latest is False. Without a template,
every template matching the document that the user can view is run.
A failing document is logged and the rest still run; the task then fails.
"""
selected_template: OcrTemplate | None = None
if template is not None:
selected_template = (
OcrTemplate.objects.filter(pk=template)
.prefetch_related("zones", "zones__custom_field")
.first()
)
if selected_template is None:
logger.warning(
"OCR template %s no longer exists, not running zone OCR",
template,
)
return "OK"
documents = {d.pk: d for d in Document.objects.filter(pk__in=doc_ids)}
missing_ids = [pk for pk in dict.fromkeys(doc_ids) if pk not in documents]
if missing_ids:
logger.warning("Documents %s not found, skipping zone OCR", missing_ids)
targets: dict[int, Document] = {}
for document_id in doc_ids:
document = documents.get(document_id)
if document is None:
continue
if resolve_latest:
document = latest_version(document)
targets.setdefault(document.pk, document)
written_ids: list[int] = []
failed_ids: list[int] = []
try:
for document in targets.values():
try:
templates = (
[selected_template]
if selected_template is not None
else get_templates_for_document(document, requester=user)
)
results = run_zone_extraction(document, templates=templates)
if zone_results_written(results):
written_ids.append(get_root_document(document).pk)
except Exception:
logger.exception("Zone OCR failed for document %s", document.pk)
failed_ids.append(document.pk)
finally:
if written_ids:
bulk_update_documents(list(dict.fromkeys(written_ids)))
if failed_ids:
raise RuntimeError(
f"Zone OCR failed for {len(failed_ids)} of {len(targets)} documents",
)
return "OK"
+15 -2
View File
@@ -13,8 +13,10 @@ from rest_framework.exceptions import PermissionDenied
from documents import bulk_edit from documents import bulk_edit
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import OcrTemplate
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.permissions import has_document_operation_permissions from documents.permissions import has_document_operation_permissions
from documents.permissions import has_perms_owner_aware
from documents.tasks import bulk_update_documents from documents.tasks import bulk_update_documents
MODIFIED_FIELDS = { MODIFIED_FIELDS = {
@@ -34,6 +36,7 @@ QUEUED_OPERATIONS = frozenset(
"delete", "delete",
"reprocess", "reprocess",
"run_workflow", "run_workflow",
"run_zone_ocr",
"rotate", "rotate",
"merge", "merge",
"merge_as_versions", "merge_as_versions",
@@ -52,10 +55,11 @@ USER_OPERATIONS = frozenset(
"edit_pdf", "edit_pdf",
"remove_password", "remove_password",
"merge_as_versions", "merge_as_versions",
"run_zone_ocr",
}, },
) )
# merge_as_versions doesn't queue any consume tasks # These don't queue any consume tasks
TRIGGER_SOURCE_OPERATIONS = USER_OPERATIONS - {"merge_as_versions"} TRIGGER_SOURCE_OPERATIONS = USER_OPERATIONS - {"merge_as_versions", "run_zone_ocr"}
def snapshot_field(doc_ids: list[int], field: str) -> dict[int, Any]: def snapshot_field(doc_ids: list[int], field: str) -> dict[int, Any]:
@@ -113,6 +117,15 @@ def execute_bulk_operation(
if operation == "run_workflow" and not user.has_perm("documents.view_workflow"): if operation == "run_workflow" and not user.has_perm("documents.view_workflow"):
raise PermissionDenied("Insufficient permissions to run this workflow") raise PermissionDenied("Insufficient permissions to run this workflow")
if operation == "run_zone_ocr" and parameters.get("template") is not None:
template = OcrTemplate.objects.filter(pk=parameters["template"]).first()
if template is not None and not has_perms_owner_aware(
user,
"view_ocrtemplate",
template,
):
raise PermissionDenied("Insufficient permissions to run this OCR template")
actor_context = set_actor(user) if settings.AUDIT_LOG_ENABLED else nullcontext() actor_context = set_actor(user) if settings.AUDIT_LOG_ENABLED else nullcontext()
if operation not in MODIFIED_FIELDS: if operation not in MODIFIED_FIELDS:
+127
View File
@@ -34,6 +34,7 @@ from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentBarcode from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplateZone
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
from documents.models import WorkflowTrigger from documents.models import WorkflowTrigger
@@ -56,6 +57,13 @@ from documents.utils import compute_checksum
from documents.utils import copy_basic_file_stats from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess from documents.utils import run_subprocess
from documents.versioning import get_root_document
from documents.zone_ocr import FieldKey
from documents.zone_ocr import TemplatePlan
from documents.zone_ocr import apply_zone_plans
from documents.zone_ocr import field_key
from documents.zone_ocr import get_templates_for_content
from documents.zone_ocr import plan_zone_extraction
from paperless.config import BarcodeConfig from paperless.config import BarcodeConfig
from paperless.config import OcrConfig from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig from paperless.config import RemoteOCRConfig
@@ -588,6 +596,16 @@ class ConsumerPlugin(
exception=e, exception=e,
) )
# Zone OCR renders and reads pages, so it runs here, before the
# transaction opens. Its values are written once the document
# is stored.
zone_plans = self._plan_zone_ocr(
text=text,
archive_path=archive_path,
mime_type=mime_type,
page_count=page_count,
)
# Prepare the document classifier. # Prepare the document classifier.
# TODO: I don't really like to do this here, but this way we avoid # TODO: I don't really like to do this here, but this way we avoid
@@ -672,6 +690,11 @@ class ConsumerPlugin(
mime_type=mime_type, mime_type=mime_type,
) )
# Before the consumption finished signal, so DOCUMENT_ADDED
# workflows and the other handlers of that signal see the
# zone values.
self._apply_zone_ocr(document, zone_plans)
# If we get here, it was successful. Proceed with post-consume # If we get here, it was successful. Proceed with post-consume
# hooks. If they fail, nothing will get changed. # hooks. If they fail, nothing will get changed.
@@ -986,6 +1009,110 @@ class ConsumerPlugin(
self._store_barcodes(document) self._store_barcodes(document)
def _plan_zone_ocr(
self,
*,
text: str | None,
archive_path: Path | None,
mime_type: str,
page_count: int | None,
) -> list[TemplatePlan]:
"""
Select the OCR templates matching the parsed text and compute their zone
values, without writing anything. Zones are read from the archive, the
file the template editor draws on, or from the working copy of a PDF or
image without one. Any error is logged and yields no zone values.
"""
try:
if self.input_doc.root_document_id is not None:
root_doc = Document.objects.select_related("owner").get(
pk=self.input_doc.root_document_id,
)
owner = get_root_document(root_doc).owner
elif self.metadata.owner_id is not None:
owner = User.objects.get(pk=self.metadata.owner_id)
else:
owner = None
templates = get_templates_for_content(
text or "",
owner,
title=self.filename,
)
if not templates:
return []
is_image = False
if archive_path is not None and Path(archive_path).is_file():
doc_path: Path | None = Path(archive_path)
elif mime_type == "application/pdf" or mime_type.startswith("image/"):
doc_path = self.working_copy
is_image = mime_type.startswith("image/")
else:
doc_path = None
return plan_zone_extraction(
doc_path,
page_count,
templates,
label=self.filename,
is_image=is_image,
)
except Exception:
self.log.exception(
f"Zone OCR failed for {self.filename}, continuing without zone values",
)
return []
def _apply_zone_ocr(self, document: Document, plans: list[TemplatePlan]) -> None:
"""
Write the planned zone values to the stored document, or to the root of a
new version. For a new document, a value supplied explicitly in the
metadata overrides is kept rather than replaced by a zone.
"""
if not plans:
return
try:
protected = (
frozenset()
if self.input_doc.root_document_id is not None
else self._explicit_zone_targets()
)
root_doc = get_root_document(document)
results = apply_zone_plans(
root_doc,
plans,
label=f"document {root_doc.pk}",
protected=protected,
)
except Exception:
self.log.exception(
f"Zone OCR could not write values for {self.filename}",
)
return
for result in results:
self.log.debug(
f"Zone OCR: template '{result.template}' zone '{result.zone}' "
f"for {result.custom_field}: "
f"{result.reason if result.reason is not None else 'written'}",
)
def _explicit_zone_targets(self) -> frozenset[FieldKey]:
"""The zone targets given a value by the metadata overrides.
A custom field listed without a value is not a supplied value.
"""
targets = set()
if self.metadata.title is not None:
targets.add(field_key(OcrTemplateZone.TargetType.TITLE, None))
if self.metadata.asn is not None:
targets.add(field_key(OcrTemplateZone.TargetType.ASN, None))
if self.metadata.created is not None:
targets.add(field_key(OcrTemplateZone.TargetType.CREATED, None))
for field_id, value in (self.metadata.custom_fields or {}).items():
if value is not None:
targets.add(
field_key(OcrTemplateZone.TargetType.CUSTOM_FIELD, int(field_id)),
)
return frozenset(targets)
def _read_version_barcodes(self, mime_type: str, work_dir: Path) -> None: def _read_version_barcodes(self, mime_type: str, work_dir: Path) -> None:
barcode_settings = BarcodeConfig() barcode_settings = BarcodeConfig()
if not barcode_settings.barcode_store_values: if not barcode_settings.barcode_store_values:
@@ -49,6 +49,8 @@ from documents.models import Document
from documents.models import DocumentBarcode from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import Note from documents.models import Note
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import SavedView from documents.models import SavedView
from documents.models import SavedViewFilterRule from documents.models import SavedViewFilterRule
from documents.models import ShareLink from documents.models import ShareLink
@@ -364,6 +366,8 @@ class Command(CryptMixin, PaperlessCommand):
"workflows": Workflow.objects.all(), "workflows": Workflow.objects.all(),
"custom_fields": CustomField.objects.all(), "custom_fields": CustomField.objects.all(),
"custom_field_instances": CustomFieldInstance.global_objects.all(), "custom_field_instances": CustomFieldInstance.global_objects.all(),
"ocr_templates": OcrTemplate.objects.all(),
"ocr_template_zones": OcrTemplateZone.objects.all(),
"document_barcodes": DocumentBarcode.objects.all(), "document_barcodes": DocumentBarcode.objects.all(),
"app_configs": ApplicationConfiguration.objects.all(), "app_configs": ApplicationConfiguration.objects.all(),
"notes": Note.global_objects.all(), "notes": Note.global_objects.all(),
+6 -2
View File
@@ -14,6 +14,7 @@ from documents.models import Correspondent
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import MatchingModel from documents.models import MatchingModel
from documents.models import OcrTemplate
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
from documents.models import Workflow from documents.models import Workflow
@@ -30,7 +31,7 @@ logger = logging.getLogger("paperless.matching")
def log_reason( def log_reason(
matching_model: MatchingModel | WorkflowTrigger, matching_model: MatchingModel | WorkflowTrigger | OcrTemplate,
document: Document, document: Document,
reason: str, reason: str,
) -> None: ) -> None:
@@ -161,7 +162,10 @@ def match_storage_paths(document: Document, classifier: DocumentClassifier, user
) )
def matches(matching_model: MatchingModel, document: Document): def matches(
matching_model: MatchingModel | WorkflowTrigger | OcrTemplate,
document: Document,
) -> bool:
search_flags = 0 search_flags = 0
document_content = document.get_effective_content() or "" document_content = document.get_effective_content() or ""
@@ -0,0 +1,310 @@
# Generated by Django 5.2.18 on 2026-11-19 17:29
import django.core.validators
import django.db.models.deletion
import django.utils.timezone
from django.conf import settings
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0030_note_format"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name="OcrTemplate",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("name", models.CharField(max_length=128, verbose_name="name")),
(
"match",
models.CharField(blank=True, max_length=256, verbose_name="match"),
),
(
"matching_algorithm",
models.PositiveSmallIntegerField(
choices=[
(0, "None"),
(1, "Any word"),
(2, "All words"),
(3, "Exact match"),
(4, "Regular expression"),
(5, "Fuzzy word"),
],
default=1,
verbose_name="matching algorithm",
),
),
(
"is_insensitive",
models.BooleanField(default=True, verbose_name="is insensitive"),
),
(
"source_width",
models.PositiveIntegerField(
help_text="Width of the image the zones were drawn on (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="source width",
),
),
(
"source_height",
models.PositiveIntegerField(
help_text="Height of the image the zones were drawn on (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="source height",
),
),
("enabled", models.BooleanField(default=True, verbose_name="enabled")),
(
"combine_formats",
models.JSONField(
blank=True,
default=dict,
help_text="Per-target format strings for combining several zones into one field, keyed by target (custom field id, or 'title'/'asn'/'created'). Tokens like {Zone Name} are replaced with that zone's value.",
verbose_name="combine formats",
),
),
(
"created",
models.DateTimeField(
db_index=True,
default=django.utils.timezone.now,
editable=False,
verbose_name="created",
),
),
(
"updated",
models.DateTimeField(auto_now=True, verbose_name="updated"),
),
(
"owner",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
to=settings.AUTH_USER_MODEL,
verbose_name="owner",
),
),
(
"sample_document",
models.ForeignKey(
blank=True,
help_text="Document used for previewing zones in the editor",
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="+",
to="documents.document",
verbose_name="sample document",
),
),
],
options={
"verbose_name": "OCR template",
"verbose_name_plural": "OCR templates",
"ordering": ("name",),
"abstract": False,
},
),
migrations.CreateModel(
name="OcrTemplateZone",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"name",
models.CharField(
help_text="Descriptive name for this zone (e.g. 'Invoice Number')",
max_length=128,
verbose_name="zone name",
),
),
(
"target",
models.CharField(
choices=[
("custom_field", "Custom field"),
("title", "Title"),
("asn", "Archive serial number"),
("created", "Date created"),
],
default="custom_field",
help_text="Where the extracted value is written: a custom field, or a built-in document field (title, ASN, created date)",
max_length=20,
verbose_name="target",
),
),
(
"page",
models.IntegerField(
blank=True,
help_text="Page (1 = first, -1 = last; blank = first page)",
null=True,
verbose_name="page",
),
),
(
"x",
models.PositiveIntegerField(
help_text="Left edge (px)",
verbose_name="x",
),
),
(
"y",
models.PositiveIntegerField(
help_text="Top edge (px)",
verbose_name="y",
),
),
(
"width",
models.PositiveIntegerField(
help_text="Zone width (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="width",
),
),
(
"height",
models.PositiveIntegerField(
help_text="Zone height (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="height",
),
),
(
"zone_source_width",
models.PositiveIntegerField(
blank=True,
help_text="Width of the page image this zone was drawn on (px). Falls back to template source_width if unset.",
null=True,
verbose_name="zone source width",
),
),
(
"zone_source_height",
models.PositiveIntegerField(
blank=True,
help_text="Height of the page image this zone was drawn on (px). Falls back to template source_height if unset.",
null=True,
verbose_name="zone source height",
),
),
(
"ocr_language",
models.CharField(
default="eng",
help_text="Tesseract language code(s), e.g. 'deu+eng'",
max_length=32,
verbose_name="OCR language",
),
),
(
"transform",
models.CharField(
choices=[
("none", "None"),
("strip", "Strip whitespace"),
("uppercase", "Uppercase"),
("lowercase", "Lowercase"),
("numeric", "Numeric only"),
(
"strip_punctuation",
"Remove leading/trailing punctuation",
),
("date", "Parse date"),
("qr_code", "Read QR/barcode"),
],
default="strip",
max_length=20,
verbose_name="transform",
),
),
(
"date_format",
models.CharField(
blank=True,
default="",
help_text="Python strptime format for the 'Parse date' transform (e.g. %d.%m.%Y). Blank = auto-detect.",
max_length=64,
verbose_name="date format",
),
),
(
"validation_regex",
models.CharField(
blank=True,
default="",
help_text="Optional regex pattern. Extracted text is only accepted if it fully matches.",
max_length=256,
verbose_name="validation regex",
),
),
("order", models.PositiveIntegerField(default=0, verbose_name="order")),
(
"custom_field",
models.ForeignKey(
blank=True,
help_text="Target custom field (only used when target is 'custom_field')",
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="ocr_zones",
to="documents.customfield",
verbose_name="custom field",
),
),
(
"template",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="zones",
to="documents.ocrtemplate",
verbose_name="template",
),
),
],
options={
"verbose_name": "OCR template zone",
"verbose_name_plural": "OCR template zones",
"ordering": ("template", "order"),
},
),
migrations.AddConstraint(
model_name="ocrtemplate",
constraint=models.UniqueConstraint(
fields=("name", "owner"),
name="documents_ocrtemplate_unique_name_owner",
),
),
migrations.AddConstraint(
model_name="ocrtemplate",
constraint=models.UniqueConstraint(
condition=models.Q(("owner__isnull", True)),
fields=("name",),
name="documents_ocrtemplate_name_uniq",
),
),
]
+226
View File
@@ -2216,3 +2216,229 @@ class WorkflowRun(SoftDeleteModel):
def __str__(self) -> str: def __str__(self) -> str:
return f"WorkflowRun of {self.workflow} at {self.run_at} on {self.document}" return f"WorkflowRun of {self.workflow} at {self.run_at} on {self.document}"
class OcrTemplate(ModelWithOwner):
"""Matching rules and extraction zones for documents with a common layout."""
name = models.CharField(_("name"), max_length=128)
match = models.CharField(_("match"), max_length=256, blank=True)
# Templates are never assigned by the classifier, so there is no auto matching.
matching_algorithm = models.PositiveSmallIntegerField(
_("matching algorithm"),
choices=WorkflowTrigger.WorkflowTriggerMatching.choices,
default=MatchingModel.MATCH_ANY,
)
is_insensitive = models.BooleanField(_("is insensitive"), default=True)
source_width = models.PositiveIntegerField(
_("source width"),
validators=[MinValueValidator(1)],
help_text=_("Width of the image the zones were drawn on (px)"),
)
source_height = models.PositiveIntegerField(
_("source height"),
validators=[MinValueValidator(1)],
help_text=_("Height of the image the zones were drawn on (px)"),
)
sample_document = models.ForeignKey(
"documents.Document",
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="+",
verbose_name=_("sample document"),
help_text=_("Document used for previewing zones in the editor"),
)
enabled = models.BooleanField(_("enabled"), default=True)
combine_formats = models.JSONField(
_("combine formats"),
default=dict,
blank=True,
help_text=_(
"Per-target format strings for combining several zones into one "
"field, keyed by target (custom field id, or 'title'/'asn'/'created'). "
"Tokens like {Zone Name} are replaced with that zone's value.",
),
)
created = models.DateTimeField(
_("created"),
default=timezone.now,
db_index=True,
editable=False,
)
updated = models.DateTimeField(
_("updated"),
auto_now=True,
)
class Meta(ModelWithOwner.Meta):
abstract = False
verbose_name = _("OCR template")
verbose_name_plural = _("OCR templates")
ordering = ("name",)
constraints = [
models.UniqueConstraint(
fields=["name", "owner"],
name="%(app_label)s_%(class)s_unique_name_owner",
),
models.UniqueConstraint(
name="%(app_label)s_%(class)s_name_uniq",
fields=["name"],
condition=models.Q(owner__isnull=True),
),
]
def __str__(self) -> str: # pragma: no cover
return self.name
class OcrTemplateZone(models.Model):
"""A page region and the field its extracted value is written to."""
template = models.ForeignKey(
OcrTemplate,
on_delete=models.CASCADE,
related_name="zones",
verbose_name=_("template"),
)
name = models.CharField(
_("zone name"),
max_length=128,
help_text=_("Descriptive name for this zone (e.g. 'Invoice Number')"),
)
class TargetType(models.TextChoices):
CUSTOM_FIELD = ("custom_field", _("Custom field"))
TITLE = ("title", _("Title"))
ASN = ("asn", _("Archive serial number"))
CREATED = ("created", _("Date created"))
target = models.CharField(
_("target"),
max_length=20,
choices=TargetType.choices,
default=TargetType.CUSTOM_FIELD,
help_text=_(
"Where the extracted value is written: a custom field, or a "
"built-in document field (title, ASN, created date)",
),
)
custom_field = models.ForeignKey(
"documents.CustomField",
on_delete=models.SET_NULL,
related_name="ocr_zones",
verbose_name=_("custom field"),
null=True,
blank=True,
help_text=_("Target custom field (only used when target is 'custom_field')"),
)
page = models.IntegerField(
_("page"),
null=True,
blank=True,
help_text=_("Page (1 = first, -1 = last; blank = first page)"),
)
x = models.PositiveIntegerField(_("x"), help_text=_("Left edge (px)"))
y = models.PositiveIntegerField(_("y"), help_text=_("Top edge (px)"))
width = models.PositiveIntegerField(
_("width"),
validators=[MinValueValidator(1)],
help_text=_("Zone width (px)"),
)
height = models.PositiveIntegerField(
_("height"),
validators=[MinValueValidator(1)],
help_text=_("Zone height (px)"),
)
# Mixed-size PDFs need each zone's original page dimensions for scaling.
zone_source_width = models.PositiveIntegerField(
_("zone source width"),
null=True,
blank=True,
help_text=_(
"Width of the page image this zone was drawn on (px). "
"Falls back to template source_width if unset.",
),
)
zone_source_height = models.PositiveIntegerField(
_("zone source height"),
null=True,
blank=True,
help_text=_(
"Height of the page image this zone was drawn on (px). "
"Falls back to template source_height if unset.",
),
)
ocr_language = models.CharField(
_("OCR language"),
max_length=32,
default="eng",
help_text=_("Tesseract language code(s), e.g. 'deu+eng'"),
)
class TransformType(models.TextChoices):
NONE = ("none", _("None"))
STRIP = ("strip", _("Strip whitespace"))
UPPERCASE = ("uppercase", _("Uppercase"))
LOWERCASE = ("lowercase", _("Lowercase"))
NUMERIC = ("numeric", _("Numeric only"))
STRIP_PUNCTUATION = (
"strip_punctuation",
_("Remove leading/trailing punctuation"),
)
DATE = ("date", _("Parse date"))
QR_CODE = ("qr_code", _("Read QR/barcode"))
transform = models.CharField(
_("transform"),
max_length=20,
choices=TransformType.choices,
default=TransformType.STRIP,
)
date_format = models.CharField(
_("date format"),
max_length=64,
blank=True,
default="",
help_text=_(
"Python strptime format for the 'Parse date' transform "
"(e.g. %d.%m.%Y). Blank = auto-detect.",
),
)
validation_regex = models.CharField(
_("validation regex"),
max_length=256,
blank=True,
default="",
help_text=_(
"Optional regex pattern. Extracted text is only accepted if it fully matches.",
),
)
order = models.PositiveIntegerField(_("order"), default=0)
class Meta:
ordering = ("template", "order")
verbose_name = _("OCR template zone")
verbose_name_plural = _("OCR template zones")
def __str__(self) -> str: # pragma: no cover
return f"{self.template.name} -> {self.name}"
+20 -5
View File
@@ -90,18 +90,31 @@ def rasterize_pdf_page_to_png(
out_path: Path, out_path: Path,
*, *,
dpi: float, dpi: float,
page: int = 1,
timeout: float | None = None,
logging_group=None, logging_group=None,
) -> None: ) -> None:
""" """
Rasterizes the first page of a PDF to a PNG with pdftoppm. Rasterizes one page of a PDF to a PNG with pdftoppm.
Args:
in_path: The PDF to render.
out_path: Where to write the PNG. Exactly this one file is written.
dpi: Resolution, which may be fractional.
page: The 1-based page number to render, the first page by default.
timeout: Seconds to allow pdftoppm to run, no limit by default.
logging_group: Group attached to the debug log record.
Raises:
ParseError: pdftoppm failed, timed out or could not render the page.
""" """
args = [ args = [
"pdftoppm", "pdftoppm",
# Only the first page is needed, first and last page are both 1 # Only the requested page is needed, first and last page are the same
"-f", "-f",
"1", str(page),
"-l", "-l",
"1", str(page),
# Resolution in DPI, which may be fractional # Resolution in DPI, which may be fractional
"-r", "-r",
str(dpi), str(dpi),
@@ -121,9 +134,11 @@ def rasterize_pdf_page_to_png(
logger.debug("Execute: " + " ".join(args), extra={"group": logging_group}) logger.debug("Execute: " + " ".join(args), extra={"group": logging_group})
try: try:
run_subprocess(args, logger=logger) run_subprocess(args, logger=logger, timeout=timeout)
except subprocess.CalledProcessError as e: except subprocess.CalledProcessError as e:
raise ParseError(f"pdftoppm failed at {args}") from e raise ParseError(f"pdftoppm failed at {args}") from e
except subprocess.TimeoutExpired as e:
raise ParseError(f"pdftoppm timed out after {timeout}s at {args}") from e
except Exception as e: # pragma: no cover except Exception as e: # pragma: no cover
raise ParseError("Unknown error running pdftoppm") from e raise ParseError("Unknown error running pdftoppm") from e
+15
View File
@@ -699,6 +699,21 @@ class ViewDocumentsPermissions(GlobalPermissionsByMethod):
} }
class ViewOcrTemplatesPermissions(GlobalPermissionsByMethod):
"""
Requires the global permission to view OCR templates for OPTIONS, GET and
POST, for the template editor's read-only preview endpoints.
"""
perms_map = {
"OPTIONS": ["documents.view_ocrtemplate"],
"GET": ["documents.view_ocrtemplate"],
# The zone preview is a POST only to carry the zone in the body; it
# creates and changes nothing, so viewing templates is enough.
"POST": ["documents.view_ocrtemplate"],
}
class TrashPermissions(GlobalPermissionsByMethod): class TrashPermissions(GlobalPermissionsByMethod):
"""Check the global document permission for each trash operation.""" """Check the global document permission for each trash operation."""
+72 -56
View File
@@ -2,12 +2,19 @@ from __future__ import annotations
import logging import logging
import textwrap import textwrap
from typing import TYPE_CHECKING
from typing import TypeVar
import regex import regex
from django.conf import settings from django.conf import settings
if TYPE_CHECKING:
from collections.abc import Callable
logger = logging.getLogger("paperless.regex") logger = logging.getLogger("paperless.regex")
T = TypeVar("T")
REGEX_TIMEOUT_SECONDS: float = getattr(settings, "MATCH_REGEX_TIMEOUT_SECONDS", 0.1) REGEX_TIMEOUT_SECONDS: float = getattr(settings, "MATCH_REGEX_TIMEOUT_SECONDS", 0.1)
@@ -23,31 +30,56 @@ def validate_regex_pattern(pattern: str) -> None:
raise ValueError(exc.msg) from exc raise ValueError(exc.msg) from exc
def _shorten(pattern: str) -> str:
return textwrap.shorten(pattern, width=80, placeholder="…")
def _compile_or_none(
pattern: str,
flags: int,
*,
log: Callable[..., None],
) -> regex.Pattern[str] | None:
"""Validate and compile a pattern, logging and returning None on failure."""
try:
validate_regex_pattern(pattern)
return regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
log(
"Error while processing regular expression %s: %s",
_shorten(pattern),
exc,
)
return None
def _run_with_timeout(
pattern: str,
operation: str,
func: Callable[..., T],
*args: object,
) -> T | None:
"""Call a compiled pattern method with the timeout, logging a timeout."""
try:
return func(*args, timeout=REGEX_TIMEOUT_SECONDS)
except TimeoutError:
logger.warning(
"Regular expression %s timed out for pattern %s",
operation,
_shorten(pattern),
)
return None
def safe_regex_search(pattern: str, text: str, *, flags: int = 0): def safe_regex_search(pattern: str, text: str, *, flags: int = 0):
""" """
Run a regex search with a timeout. Returns a match object or None. Run a regex search with a timeout. Returns a match object or None.
Validation errors and timeouts are logged and treated as no match. Validation errors and timeouts are logged and treated as no match.
""" """
compiled = _compile_or_none(pattern, flags, log=logger.error)
try: if compiled is None:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
logger.error(
"Error while processing regular expression %s: %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
exc,
)
return None
try:
return compiled.search(text, timeout=REGEX_TIMEOUT_SECONDS)
except TimeoutError:
logger.warning(
"Regular expression matching timed out for pattern %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
)
return None return None
return _run_with_timeout(pattern, "matching", compiled.search, text)
def safe_regex_match(pattern: str, text: str, *, flags: int = 0): def safe_regex_match(pattern: str, text: str, *, flags: int = 0):
@@ -55,26 +87,26 @@ def safe_regex_match(pattern: str, text: str, *, flags: int = 0):
Run a regex match with a timeout. Returns a match object or None. Run a regex match with a timeout. Returns a match object or None.
Validation errors and timeouts are logged and treated as no match. Validation errors and timeouts are logged and treated as no match.
""" """
compiled = _compile_or_none(pattern, flags, log=logger.exception)
try: if compiled is None:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
logger.exception(
"Error while processing regular expression %s: %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
exc,
)
return None return None
return _run_with_timeout(pattern, "matching", compiled.match, text)
try:
return compiled.match(text, timeout=REGEX_TIMEOUT_SECONDS) def safe_regex_fullmatch(
except TimeoutError: pattern: str,
logger.warning( text: str,
"Regular expression matching timed out for pattern %s", *,
textwrap.shorten(pattern, width=80, placeholder="…"), flags: int = 0,
) ) -> regex.Match[str] | None:
"""
Run a regex full match with a timeout. Returns a match object or None.
Validation errors and timeouts are logged and treated as no match.
"""
compiled = _compile_or_none(pattern, flags, log=logger.exception)
if compiled is None:
return None return None
return _run_with_timeout(pattern, "matching", compiled.fullmatch, text)
def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str | None: def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str | None:
@@ -82,26 +114,10 @@ def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str
Run a regex substitution with a timeout. Returns the substituted string, Run a regex substitution with a timeout. Returns the substituted string,
or None on error/timeout. or None on error/timeout.
""" """
compiled = _compile_or_none(pattern, flags, log=logger.exception)
try: if compiled is None:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
logger.exception(
"Error while processing regular expression %s: %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
exc,
)
return None
try:
return compiled.sub(repl, text, timeout=REGEX_TIMEOUT_SECONDS)
except TimeoutError:
logger.warning(
"Regular expression substitution timed out for pattern %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
)
return None return None
return _run_with_timeout(pattern, "substitution", compiled.sub, repl, text)
def safe_regex_finditer(compiled_pattern: regex.Pattern, text: str): def safe_regex_finditer(compiled_pattern: regex.Pattern, text: str):
@@ -115,6 +131,6 @@ def safe_regex_finditer(compiled_pattern: regex.Pattern, text: str):
except TimeoutError: except TimeoutError:
logger.warning( logger.warning(
"Regular expression finditer timed out for pattern %s", "Regular expression finditer timed out for pattern %s",
textwrap.shorten(compiled_pattern.pattern, width=80, placeholder="…"), _shorten(compiled_pattern.pattern),
) )
return return
+453 -32
View File
@@ -5,9 +5,9 @@ import math
import re import re
from datetime import datetime from datetime import datetime
from datetime import timedelta from datetime import timedelta
from decimal import Decimal
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from typing import Any from typing import Any
from typing import NotRequired
from typing import TypedDict from typing import TypedDict
import magic import magic
@@ -16,13 +16,12 @@ from django.contrib.auth.models import Group
from django.contrib.auth.models import User from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import ValidationError from django.core.exceptions import ValidationError
from django.core.validators import DecimalValidator
from django.core.validators import EmailValidator from django.core.validators import EmailValidator
from django.core.validators import MaxLengthValidator from django.core.validators import MaxLengthValidator
from django.core.validators import MaxValueValidator from django.core.validators import MaxValueValidator
from django.core.validators import MinValueValidator from django.core.validators import MinValueValidator
from django.core.validators import RegexValidator
from django.core.validators import integer_validator from django.core.validators import integer_validator
from django.db import transaction
from django.db.models import Count from django.db.models import Count
from django.db.models import Q from django.db.models import Q
from django.db.models.functions import Lower from django.db.models.functions import Lower
@@ -62,6 +61,8 @@ from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import MatchingModel from documents.models import MatchingModel
from documents.models import Note from documents.models import Note
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import SavedView from documents.models import SavedView
from documents.models import SavedViewFilterRule from documents.models import SavedViewFilterRule
@@ -85,16 +86,21 @@ from documents.permissions import get_document_count_filter_for_user
from documents.permissions import get_permission_grantees from documents.permissions import get_permission_grantees
from documents.permissions import has_perms_owner_aware from documents.permissions import has_perms_owner_aware
from documents.permissions import permitted_document_ids from documents.permissions import permitted_document_ids
from documents.permissions import permitted_object_ids
from documents.permissions import restrict_queryset_to_visible from documents.permissions import restrict_queryset_to_visible
from documents.permissions import set_permissions_for_objects from documents.permissions import set_permissions_for_objects
from documents.regex import validate_regex_pattern from documents.regex import validate_regex_pattern
from documents.templating.filepath import validate_filepath_template_and_render from documents.templating.filepath import validate_filepath_template_and_render
from documents.templating.utils import convert_format_str_to_template_format from documents.templating.utils import convert_format_str_to_template_format
from documents.templating.workflows import validate_workflow_template from documents.templating.workflows import validate_workflow_template
from documents.validators import monetary_validator
from documents.validators import uri_validator from documents.validators import uri_validator
from documents.validators import url_validator from documents.validators import url_validator
from documents.versioning import has_prefetched_effective_content from documents.versioning import has_prefetched_effective_content
from documents.versioning import sort_versions_newest_first from documents.versioning import sort_versions_newest_first
from documents.zone_ocr import OCR_SUPPORTED_FIELD_TYPES
from documents.zone_ocr import ZoneOcrSkipReason
from documents.zone_ocr import field_key
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Iterable from collections.abc import Iterable
@@ -168,36 +174,29 @@ class DocumentUpdateFieldsModelSerializer(DynamicFieldsModelSerializer):
return instance return instance
class MatchingModelSerializer(serializers.ModelSerializer[Any]): # Name and owner uniqueness and match regex validation for models with a name,
document_count = serializers.IntegerField(read_only=True) # an owner and match rules. A comment, not a docstring, so it stays out of the
# API schema descriptions of every serializer built on it.
def get_slug(self, obj) -> str: class MatchRuleSerializer(serializers.ModelSerializer[Any]):
return slugify(obj.name) def validate(self, attrs: dict[str, Any]) -> dict[str, Any]:
slug = SerializerMethodField()
def validate(self, data):
# TODO: remove pending https://github.com/encode/django-rest-framework/issues/7173 # TODO: remove pending https://github.com/encode/django-rest-framework/issues/7173
name = data.get( name = attrs.get("name", getattr(self.instance, "name", None))
"name",
self.instance.name if hasattr(self.instance, "name") else None,
)
owner = ( owner = (
data["owner"] attrs["owner"]
if "owner" in data if "owner" in attrs
else self.user else self.user
if hasattr(self, "user") if hasattr(self, "user")
else None else None
) )
pk = self.instance.pk if hasattr(self.instance, "pk") else None pk = getattr(self.instance, "pk", None)
if ("name" in data or "owner" in data) and self.Meta.model.objects.filter( if ("name" in attrs or "owner" in attrs) and self.Meta.model.objects.filter(
name=name, name=name,
owner=owner, owner=owner,
).exclude(pk=pk).exists(): ).exclude(pk=pk).exists():
raise serializers.ValidationError( raise serializers.ValidationError(
{"error": "Object violates owner / name unique constraint"}, {"error": "Object violates owner / name unique constraint"},
) )
return data return attrs
def validate_match(self, match): def validate_match(self, match):
if ( if (
@@ -214,6 +213,15 @@ class MatchingModelSerializer(serializers.ModelSerializer[Any]):
return match return match
class MatchingModelSerializer(MatchRuleSerializer):
document_count = serializers.IntegerField(read_only=True)
def get_slug(self, obj) -> str:
return slugify(obj.name)
slug = SerializerMethodField()
class SetPermissionsMixin: class SetPermissionsMixin:
def _validate_user_ids(self, user_ids): def _validate_user_ids(self, user_ids):
users = User.objects.none() users = User.objects.none()
@@ -904,17 +912,7 @@ class CustomFieldInstanceSerializer(serializers.ModelSerializer[CustomFieldInsta
field.data_type == CustomField.FieldDataType.MONETARY field.data_type == CustomField.FieldDataType.MONETARY
and data["value"] != "" and data["value"] != ""
): ):
try: monetary_validator(data["value"])
# First try to validate as a number from legacy format
DecimalValidator(max_digits=12, decimal_places=2)(
Decimal(str(data["value"])),
)
except Exception:
# If that fails, try to validate as a monetary string
RegexValidator(
regex=r"^[A-Z]{3}-?\d+(\.\d{1,2})$",
message="Must be a two-decimal number with optional currency code e.g. GBP123.45",
)(data["value"])
elif field.data_type == CustomField.FieldDataType.STRING: elif field.data_type == CustomField.FieldDataType.STRING:
MaxLengthValidator(limit_value=128)(data["value"]) MaxLengthValidator(limit_value=128)(data["value"])
elif field.data_type == CustomField.FieldDataType.SELECT: elif field.data_type == CustomField.FieldDataType.SELECT:
@@ -1808,6 +1806,65 @@ class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
from_webui = serializers.BooleanField(required=False, default=False) from_webui = serializers.BooleanField(required=False, default=False)
class OcrTemplateField(serializers.PrimaryKeyRelatedField[OcrTemplate]):
pass
class RunZoneOcrSerializer(serializers.Serializer[Any]):
template = OcrTemplateField(
queryset=OcrTemplate.objects.none(),
required=False,
allow_null=True,
)
from_webui = serializers.BooleanField(required=False, default=False)
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, **kwargs)
request = self.context.get("request")
user = getattr(request, "user", None) if request else None
if user is not None and user.is_authenticated:
template_field = self.fields.get("template")
if not isinstance(
template_field,
serializers.PrimaryKeyRelatedField,
): # pragma: no cover
return
template_field.queryset = OcrTemplate.objects.filter(
id__in=permitted_object_ids(user, OcrTemplate, "view_ocrtemplate"),
).prefetch_related("zones", "zones__custom_field")
class RunZoneOcrResponseSerializer(serializers.Serializer[dict[str, Any]]):
"""The response of queueing zone OCR on one document."""
result = serializers.ChoiceField(
choices=["OK", "skipped"],
read_only=True,
help_text="OK when the run was queued, skipped when it was not.",
)
task_id = serializers.CharField(
read_only=True,
required=False,
help_text="The queued task. Absent when the run was skipped.",
)
reason = serializers.ChoiceField(
choices=[reason.value for reason in ZoneOcrSkipReason],
read_only=True,
required=False,
help_text="Present only when the run was skipped.",
)
class OcrErrorSerializer(serializers.Serializer[dict[str, Any]]):
"""The body of an error response from the OCR template actions."""
error = serializers.CharField(read_only=True)
class RunZoneOcrDocumentsSerializer(DocumentSelectionSerializer, RunZoneOcrSerializer):
pass
class BulkEditSerializer( class BulkEditSerializer(
SerializerWithPerms, SerializerWithPerms,
DocumentSelectionSerializer, DocumentSelectionSerializer,
@@ -3719,3 +3776,367 @@ class StoragePathTestSerializer(SerializerWithPerms):
document_field.queryset = Document.objects.filter( document_field.queryset = Document.objects.filter(
id__in=permitted_document_ids(user), id__in=permitted_document_ids(user),
) )
class ZoneData(TypedDict):
"""Validated data of one OcrTemplateZoneSerializer."""
name: str
x: int
y: int
width: int
height: int
target: NotRequired[str]
custom_field: NotRequired[CustomField | None]
page: NotRequired[int | None]
ocr_language: NotRequired[str]
transform: NotRequired[str]
date_format: NotRequired[str]
order: NotRequired[int]
zone_source_width: NotRequired[int | None]
zone_source_height: NotRequired[int | None]
validation_regex: NotRequired[str]
class OcrTemplateZoneSerializer(serializers.ModelSerializer[OcrTemplateZone]):
class Meta:
model = OcrTemplateZone
fields = [
"id",
"name",
"target",
"custom_field",
"page",
"x",
"y",
"width",
"height",
"ocr_language",
"transform",
"date_format",
"order",
"zone_source_width",
"zone_source_height",
"validation_regex",
]
def validate_custom_field(
self,
value: CustomField | None,
) -> CustomField | None:
if value is None:
return value
if value.data_type not in OCR_SUPPORTED_FIELD_TYPES:
supported = ", ".join(
str(data_type.label).lower()
for data_type in CustomField.FieldDataType
if data_type in OCR_SUPPORTED_FIELD_TYPES
)
raise serializers.ValidationError(
f"Custom field type '{value.data_type}' is not supported for OCR extraction. "
f"Use one of: {supported}.",
)
return value
def validate_ocr_language(self, value: str) -> str:
# Tesseract language codes joined by "+", e.g. "deu+eng", "chi_sim" or
# "script/Latin".
code = r"[A-Za-z0-9_]+(/[A-Za-z0-9_]+)?"
if not re.fullmatch(rf"{code}(\+{code})*", value):
raise serializers.ValidationError(
"Use Tesseract language codes joined by '+', e.g. 'deu+eng'.",
)
return value
def validate_date_format(self, value: str) -> str:
# The format must read back a full date: without a day, month or year
# strptime fills in 1 or 1900 instead of failing. A missing year is
# rejected before parsing, since Python deprecates parsing a day of month
# without one.
if value:
if not re.search(r"%[Yy]", value.replace("%%", "")):
raise serializers.ValidationError(
"The date format must include the day, month and year.",
)
sample = datetime(2024, 3, 15)
try:
round_trip = datetime.strptime(sample.strftime(value), value)
except ValueError as e:
logger.debug(f"Invalid zone date format {value!r}: {e!s}")
raise serializers.ValidationError(
"Invalid date format, see log for details.",
) from e
if round_trip.date() != sample.date():
raise serializers.ValidationError(
"The date format must read back a full date (day, month and year).",
)
return value
def validate_validation_regex(self, value: str) -> str:
if value:
try:
validate_regex_pattern(value)
except ValueError as e:
logger.debug(f"Invalid regular expression: {e!s}")
raise serializers.ValidationError(
"Invalid regular expression, see log for details.",
) from e
return value
def validate_page(self, value: int | None) -> int | None:
if value is not None and value != -1 and value < 1:
raise serializers.ValidationError(
"Page must be positive or -1 for the last page.",
)
return value
def validate(self, attrs: ZoneData) -> ZoneData:
# A template PATCH still replaces each zone in full.
if self.root.partial:
for name, field in self.fields.items():
if field.required and name not in attrs:
raise serializers.ValidationError(
{name: field.error_messages["required"]},
)
# A custom field target may be left unassigned, as the editor creates new
# zones, and a zone is left unassigned when its custom field is deleted;
# built-in targets never use a custom field.
if (
attrs.get("target", OcrTemplateZone.TargetType.CUSTOM_FIELD)
!= OcrTemplateZone.TargetType.CUSTOM_FIELD
and attrs.get("custom_field") is not None
):
raise serializers.ValidationError(
{"custom_field": "Only zones targeting a custom field can set one."},
)
return attrs
class OcrTemplateZoneTestSerializer(OcrTemplateZoneSerializer):
name = serializers.CharField(default="test")
x = serializers.IntegerField(min_value=0, default=0)
y = serializers.IntegerField(min_value=0, default=0)
width = serializers.IntegerField(min_value=2)
height = serializers.IntegerField(min_value=2)
class Meta(OcrTemplateZoneSerializer.Meta):
fields = [
"name",
"x",
"y",
"width",
"height",
"page",
"ocr_language",
"transform",
"date_format",
"validation_regex",
"zone_source_width",
"zone_source_height",
]
extra_kwargs = {
"zone_source_width": {"min_value": 1},
"zone_source_height": {"min_value": 1},
}
class DocumentField(serializers.PrimaryKeyRelatedField[Document]):
pass
class OcrTemplateTestSerializer(serializers.Serializer[dict[str, Any]]):
document = DocumentField(queryset=Document.objects.none())
zone = OcrTemplateZoneTestSerializer()
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, **kwargs)
request = self.context.get("request")
user = getattr(request, "user", None) if request else None
if user is not None and user.is_authenticated:
document_field = self.fields.get("document")
if not isinstance(
document_field,
serializers.PrimaryKeyRelatedField,
): # pragma: no cover
return
document_field.queryset = Document.objects.filter(
id__in=permitted_document_ids(user),
)
class OcrTemplateTestResultSerializer(serializers.Serializer[dict[str, Any]]):
"""The extraction preview returned for an ad-hoc zone."""
raw_text = serializers.CharField(read_only=True, allow_null=True)
value = serializers.CharField(read_only=True, allow_null=True)
regex = serializers.CharField(read_only=True, allow_null=True)
regex_match = serializers.BooleanField(
read_only=True,
allow_null=True,
help_text="Whether the value fully matches the regex, or null when not checked.",
)
class OcrTemplateSerializer(MatchRuleSerializer, OwnedObjectSerializer):
zones = OcrTemplateZoneSerializer(many=True, required=False)
combine_formats = serializers.DictField(
child=serializers.CharField(allow_blank=True, trim_whitespace=False),
required=False,
)
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, **kwargs)
# Writes accept documents the user can view, plus the sample document
# the template already stores, so saving a template unchanged keeps it.
# Reads serialize the stored pk regardless of the queryset.
sample_field = self.fields.get("sample_document")
if not isinstance(
sample_field,
serializers.PrimaryKeyRelatedField,
): # pragma: no cover
return
if self.user is None:
sample_field.queryset = Document.objects.none()
return
allowed = Q(id__in=permitted_document_ids(self.user))
if isinstance(self.instance, OcrTemplate) and self.instance.sample_document_id:
allowed |= Q(id=self.instance.sample_document_id)
sample_field.queryset = Document.objects.filter(allowed)
class Meta:
model = OcrTemplate
fields = [
"id",
"name",
"match",
"matching_algorithm",
"is_insensitive",
"source_width",
"source_height",
"sample_document",
"enabled",
"combine_formats",
"created",
"updated",
"zones",
"owner",
"permissions",
"user_can_change",
"set_permissions",
]
read_only_fields = ["created", "updated"]
def validate(self, attrs: dict[str, Any]) -> dict[str, Any]:
attrs = super().validate(attrs)
zones: list[ZoneData] | None = attrs.get("zones")
if zones is not None:
names = [zone["name"] for zone in zones]
duplicates = sorted({name for name in names if names.count(name) > 1})
if duplicates:
# Combine format tokens refer to zones by name.
raise serializers.ValidationError(
{"zones": f"Zone names must be unique: {', '.join(duplicates)}"},
)
if "combine_formats" in attrs:
if zones is not None:
keys = {
field_key(
OcrTemplateZone.TargetType(
zone.get("target", OcrTemplateZone.TargetType.CUSTOM_FIELD),
),
custom_field.pk
if (custom_field := zone.get("custom_field"))
else None,
)
for zone in zones
}
elif self.instance:
keys = {
field_key(OcrTemplateZone.TargetType(target), custom_field_id)
for target, custom_field_id in self.instance.zones.values_list(
"target",
"custom_field",
)
}
else:
keys = set()
unknown = sorted(set(attrs["combine_formats"]) - keys)
if unknown:
raise serializers.ValidationError(
{
"combine_formats": (
"Keys must match the field of a zone: " + ", ".join(unknown)
),
},
)
# Each zone's name, x, y, width, height and source size.
boxes: list[tuple[str, int, int, int, int, int | None, int | None]] = []
if zones is not None:
boxes = [
(
zone["name"],
zone["x"],
zone["y"],
zone["width"],
zone["height"],
zone.get("zone_source_width"),
zone.get("zone_source_height"),
)
for zone in zones
]
elif self.instance and ("source_width" in attrs or "source_height" in attrs):
boxes = list(
self.instance.zones.values_list(
"name",
"x",
"y",
"width",
"height",
"zone_source_width",
"zone_source_height",
),
)
source_width = attrs.get(
"source_width",
getattr(self.instance, "source_width", None),
)
source_height = attrs.get(
"source_height",
getattr(self.instance, "source_height", None),
)
if not boxes:
return attrs
assert source_width is not None
assert source_height is not None
for name, x, y, width, height, zone_width, zone_height in boxes:
if x + width > (zone_width or source_width) or y + height > (
zone_height or source_height
):
raise serializers.ValidationError(
{"zones": f"Zone '{name}' extends beyond its source dimensions."},
)
return attrs
@transaction.atomic
def create(self, validated_data: dict[str, Any]) -> OcrTemplate:
zones_data: list[ZoneData] = validated_data.pop("zones", [])
template = super().create(validated_data)
for zone_data in zones_data:
OcrTemplateZone.objects.create(template=template, **zone_data)
return template
@transaction.atomic
def update(
self,
instance: OcrTemplate,
validated_data: dict[str, Any],
) -> OcrTemplate:
zones_data: list[ZoneData] | None = validated_data.pop("zones", None)
instance = super().update(instance, validated_data)
if zones_data is not None:
instance.zones.all().delete()
for zone_data in zones_data:
OcrTemplateZone.objects.create(template=instance, **zone_data)
return instance
+13 -1
View File
@@ -45,6 +45,8 @@ from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import SavedView from documents.models import SavedView
from documents.models import StoragePath from documents.models import StoragePath
@@ -717,7 +719,8 @@ def check_paths_and_prune_custom_fields(
@receiver(models.signals.post_delete, sender=CustomField) @receiver(models.signals.post_delete, sender=CustomField)
def cleanup_custom_field_deletion(sender, instance: CustomField, **kwargs) -> None: def cleanup_custom_field_deletion(sender, instance: CustomField, **kwargs) -> None:
""" """
When a custom field is deleted, ensure no saved views reference it. When a custom field is deleted, ensure no saved views reference it and no
OCR template keeps a combine format for it.
""" """
field_identifier = SavedView.DisplayFields.CUSTOM_FIELD % instance.pk field_identifier = SavedView.DisplayFields.CUSTOM_FIELD % instance.pk
# remove field from display_fields of all saved views # remove field from display_fields of all saved views
@@ -740,6 +743,15 @@ def cleanup_custom_field_deletion(sender, instance: CustomField, **kwargs) -> No
f"Removing custom field {instance} from sort field of {views_with_sort_updated} views", f"Removing custom field {instance} from sort field of {views_with_sort_updated} views",
) )
# zone_ocr imports this module, so it cannot be imported at module level
from documents.zone_ocr import field_key
combine_key = field_key(OcrTemplateZone.TargetType.CUSTOM_FIELD, instance.pk)
for template in OcrTemplate.objects.only("pk", "combine_formats"):
if combine_key in template.combine_formats:
del template.combine_formats[combine_key]
template.save(update_fields=["combine_formats"])
@receiver(models.signals.post_save, sender=Document) @receiver(models.signals.post_save, sender=Document)
def update_llm_suggestions_cache(sender, instance, **kwargs): def update_llm_suggestions_cache(sender, instance, **kwargs):
+10
View File
@@ -8,6 +8,10 @@ import pytest
from paperless_testing.factories import DocumentFactory from paperless_testing.factories import DocumentFactory
if TYPE_CHECKING: if TYPE_CHECKING:
from unittest.mock import MagicMock
from pytest_mock import MockerFixture
from documents.models import Document from documents.models import Document
from paperless_testing.dirs import PaperlessDirs from paperless_testing.dirs import PaperlessDirs
@@ -65,3 +69,9 @@ def searchable_document(_search_index: None) -> "Document":
doc = DocumentFactory.create(title="quarterly invoice", content="acme corp") doc = DocumentFactory.create(title="quarterly invoice", content="acme corp")
get_backend().add_or_update(doc) get_backend().add_or_update(doc)
return doc return doc
@pytest.fixture
def mock_zone_ocr_run(mocker: "MockerFixture") -> "MagicMock":
"""Patch the subprocess runner used by zone OCR (tesseract and pdftoppm)."""
return mocker.patch("documents.zone_ocr.subprocess.run")
+164
View File
@@ -1,10 +1,13 @@
import json import json
from unittest import mock from unittest import mock
import pytest
from auditlog.models import LogEntry from auditlog.models import LogEntry
from django.contrib.auth.models import User from django.contrib.auth.models import User
from django.test import override_settings from django.test import override_settings
from pytest_mock import MockerFixture
from rest_framework import status from rest_framework import status
from rest_framework.test import APIClient
from rest_framework.test import APITestCase from rest_framework.test import APITestCase
from documents.bulk_operations import execute_bulk_operation from documents.bulk_operations import execute_bulk_operation
@@ -20,11 +23,14 @@ from documents.models import WorkflowAction
from documents.models import WorkflowTrigger from documents.models import WorkflowTrigger
from paperless_testing.dirs import DirectoriesMixin from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import OcrTemplateFactory
from paperless_testing.factories import UserFactory from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_all_global from paperless_testing.permissions import grant_all_global
from paperless_testing.permissions import grant_global from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object from paperless_testing.permissions import grant_object
RUN_ZONE_OCR_BULK_URL = "/api/documents/run_zone_ocr/"
class TestBulkEditAPI(DirectoriesMixin, APITestCase): class TestBulkEditAPI(DirectoriesMixin, APITestCase):
def setUp(self) -> None: def setUp(self) -> None:
@@ -1236,6 +1242,31 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
) )
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
def test_api_bulk_edit_rejects_update_callback_parameter(self) -> None:
with mock.patch(
"documents.views.execute_bulk_operation.apply_async",
return_value=mock.Mock(id="test-bulk-operation"),
) as enqueue:
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc2.id],
"method": "modify_tags",
"parameters": {
"add_tags": [self.t2.id],
"remove_tags": [],
"update_callback": "x",
},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
enqueue.assert_not_called()
self.async_task.assert_not_called()
self.assertEqual(list(self.doc2.tags.all()), [self.t1])
def test_api_modify_tags_no_tags(self) -> None: def test_api_modify_tags_no_tags(self) -> None:
response = self.client.post( response = self.client.post(
"/api/documents/bulk_edit/", "/api/documents/bulk_edit/",
@@ -2867,3 +2898,136 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn(b"notadate", response.content) self.assertIn(b"notadate", response.content)
@pytest.mark.django_db
@pytest.mark.usefixtures("paperless_dirs")
class TestRunZoneOcrBulkEditAPI:
"""Bulk run_zone_ocr endpoint for selected documents and chosen templates."""
@pytest.fixture
def mock_run_zone_ocr(self, mocker: MockerFixture) -> mock.MagicMock:
"""
Mocks the bulk method and runs accepted requests through the worker
synchronously, as the TestBulkEditAPI setup does.
"""
mocker.patch("documents.bulk_operations.bulk_update_documents")
method = mocker.patch("documents.views.bulk_edit.run_zone_ocr")
method.return_value = "OK"
method.__name__ = "run_zone_ocr"
def run_operation(*, kwargs: dict, headers: dict) -> mock.Mock:
execute_bulk_operation(**kwargs)
return mocker.Mock(id="test-bulk-operation")
mocker.patch(
"documents.views.execute_bulk_operation.apply_async",
side_effect=run_operation,
)
return method
@pytest.fixture
def document(self) -> Document:
return DocumentFactory()
def test_rejects_private_template(
self,
rest_api_client: APIClient,
document: Document,
mock_run_zone_ocr: mock.MagicMock,
) -> None:
"""
GIVEN:
- A user who can change the documents but not view the template
WHEN:
- API is called with that template
THEN:
- The whole request is rejected with 400 on the template field
"""
user = UserFactory(username="editor")
grant_global(user, "view_document", "change_document")
template = OcrTemplateFactory(
name="Private",
owner=UserFactory(username="other"),
)
rest_api_client.force_authenticate(user=user)
response = rest_api_client.post(
RUN_ZONE_OCR_BULK_URL,
{"documents": [document.id], "template": template.pk},
format="json",
)
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert "template" in response.json()
mock_run_zone_ocr.assert_not_called()
def test_rejects_owned_document_without_global_change_permission(
self,
rest_api_client: APIClient,
mock_run_zone_ocr: mock.MagicMock,
) -> None:
"""
GIVEN:
- A user without the global change_document permission
- A document owned by that user
WHEN:
- API is called with that document
THEN:
- The request is rejected with 403 and nothing is run
"""
user = UserFactory(username="viewer")
grant_global(user, "view_document")
document = DocumentFactory(owner=user)
rest_api_client.force_authenticate(user=user)
response = rest_api_client.post(
RUN_ZONE_OCR_BULK_URL,
{"documents": [document.id]},
format="json",
)
assert response.status_code == status.HTTP_403_FORBIDDEN
mock_run_zone_ocr.assert_not_called()
def test_object_change_permission_required_for_foreign_document(
self,
rest_api_client: APIClient,
mock_run_zone_ocr: mock.MagicMock,
) -> None:
"""
GIVEN:
- A user with global change_document permission
- A document owned by someone else, with no object permission for the user
WHEN:
- API is called with that document
- API is called again after the user is granted change on the document
THEN:
- The first request is rejected with 403 and nothing is run
- The second request is accepted and run_zone_ocr is called
"""
user = UserFactory(username="editor")
grant_global(user, "view_document", "change_document")
document = DocumentFactory(owner=UserFactory(username="other"))
rest_api_client.force_authenticate(user=user)
denied = rest_api_client.post(
RUN_ZONE_OCR_BULK_URL,
{"documents": [document.id]},
format="json",
)
assert denied.status_code == status.HTTP_403_FORBIDDEN
mock_run_zone_ocr.assert_not_called()
grant_object(user, document, "change_document")
allowed = rest_api_client.post(
RUN_ZONE_OCR_BULK_URL,
{"documents": [document.id]},
format="json",
)
assert allowed.status_code == status.HTTP_200_OK
mock_run_zone_ocr.assert_called_once()
assert mock_run_zone_ocr.call_args.args[0] == [document.id]
File diff suppressed because it is too large. Load diff
+319
View File
@@ -26,8 +26,12 @@ from documents.models import Tag
from documents.pdf_ops import PageSpec from documents.pdf_ops import PageSpec
from documents.permissions import PermissionsSpec from documents.permissions import PermissionsSpec
from documents.permissions import set_permissions_for_objects from documents.permissions import set_permissions_for_objects
from documents.zone_ocr import ZoneOcrReason
from documents.zone_ocr import ZoneOcrResult
from paperless_testing.dirs import DirectoriesMixin from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import OcrTemplateFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_object from paperless_testing.permissions import grant_object
from paperless_testing.samples import MULTI_PAGE_DIGITAL_PDF from paperless_testing.samples import MULTI_PAGE_DIGITAL_PDF
from paperless_testing.samples import SIMPLE_DIGITAL_PDF from paperless_testing.samples import SIMPLE_DIGITAL_PDF
@@ -1934,6 +1938,321 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIn("Error removing password from document", cm.output[0]) self.assertIn("Error removing password from document", cm.output[0])
def _written_result() -> ZoneOcrResult:
return ZoneOcrResult(template="t", zone="z", custom_field="", reason=None)
def _unwritten_result() -> ZoneOcrResult:
return ZoneOcrResult(
template="t",
zone="z",
custom_field="",
reason=ZoneOcrReason.NO_VALUE,
)
@pytest.mark.django_db
class TestBulkEditRunZoneOcr:
"""Bulk run_zone_ocr runs zone extraction for each selected document."""
@pytest.fixture
def mock_extract(self, mocker: MockerFixture) -> mock.MagicMock:
return mocker.patch("documents.bulk_edit.run_zone_extraction", return_value=[])
@pytest.fixture
def mock_finish(self, mocker: MockerFixture) -> mock.MagicMock:
return mocker.patch("documents.bulk_edit.bulk_update_documents")
def test_selects_templates_for_the_requester(
self,
mock_extract: mock.MagicMock,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- Two selected documents and a user
WHEN:
- run_zone_ocr is called for the documents without a template
THEN:
- Templates are selected for each document with the requester
- Extraction runs inline for each document
"""
select = mocker.patch("documents.bulk_edit.get_templates_for_document")
docs = DocumentFactory.create_batch(2)
user = UserFactory()
result = bulk_edit.run_zone_ocr([doc.id for doc in docs], user=user)
assert result == "OK"
assert select.call_args_list == [
mocker.call(doc, requester=user) for doc in docs
]
assert mock_extract.call_args_list == [
mocker.call(doc, templates=select.return_value) for doc in docs
]
def test_runs_the_chosen_template(
self,
mock_extract: mock.MagicMock,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- Two documents and an OCR template
WHEN:
- run_zone_ocr is called for the documents with the template
THEN:
- Extraction runs for each document with only that template
- Template matching is not used
"""
select = mocker.patch("documents.bulk_edit.get_templates_for_document")
docs = DocumentFactory.create_batch(2)
template = OcrTemplateFactory()
bulk_edit.run_zone_ocr([doc.id for doc in docs], template=template.pk)
select.assert_not_called()
assert mock_extract.call_args_list == [
mocker.call(doc, templates=[template]) for doc in docs
]
def test_deleted_template_runs_nothing(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- A chosen template that no longer exists
WHEN:
- run_zone_ocr is called
THEN:
- OK is returned without running extraction or finishing
"""
doc = DocumentFactory()
result = bulk_edit.run_zone_ocr([doc.pk], template=999)
assert result == "OK"
mock_extract.assert_not_called()
mock_finish.assert_not_called()
def test_missing_document_does_not_abort_the_others(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- A selection holding a nonexistent id and a trashed document
alongside a live document
WHEN:
- run_zone_ocr is called
THEN:
- Only the live document is extracted
- The live document is finished
"""
live = DocumentFactory()
trashed = DocumentFactory()
trashed.delete()
template = OcrTemplateFactory()
mock_extract.return_value = [_written_result()]
bulk_edit.run_zone_ocr([999999, trashed.pk, live.pk], template=template.pk)
mock_extract.assert_called_once_with(live, templates=[template])
mock_finish.assert_called_once_with([live.pk])
def test_failing_document_does_not_abort_the_others(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- Three selected documents, the second of which raises during extraction
WHEN:
- run_zone_ocr is called
THEN:
- The failure is logged and the remaining document still runs
- bulk_update_documents runs once with the written documents
- The task then fails, naming the number of failed documents
"""
docs = DocumentFactory.create_batch(3)
template = OcrTemplateFactory()
mock_extract.side_effect = [
[_written_result()],
RuntimeError("boom"),
[_written_result()],
]
with pytest.raises(RuntimeError, match="1 of 3 documents"):
bulk_edit.run_zone_ocr([d.pk for d in docs], template=template.pk)
assert mock_extract.call_count == 3
mock_finish.assert_called_once_with([docs[0].pk, docs[2].pk])
def test_finishes_each_root_once(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- Two documents whose extraction writes, one being a version of the other
when not resolving the latest version
WHEN:
- run_zone_ocr is called without resolving the latest version
THEN:
- bulk_update_documents runs once with the root id listed once
"""
root = DocumentFactory()
version = DocumentFactory(root_document=root, version_index=1)
template = OcrTemplateFactory()
mock_extract.return_value = [_written_result()]
bulk_edit.run_zone_ocr(
[root.pk, version.pk],
template=template.pk,
resolve_latest=False,
)
assert mock_extract.call_count == 2
mock_finish.assert_called_once_with([root.pk])
def test_does_not_finish_when_nothing_was_written(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- Selected documents with no successful write
WHEN:
- run_zone_ocr is called
THEN:
- bulk_update_documents is not run
"""
docs = DocumentFactory.create_batch(2)
template = OcrTemplateFactory()
mock_extract.return_value = [_unwritten_result()]
bulk_edit.run_zone_ocr([d.pk for d in docs], template=template.pk)
mock_finish.assert_not_called()
def test_finishes_written_documents_when_the_run_is_interrupted(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- Two selected documents, the second of which raises an exception
that is not an Exception subclass
WHEN:
- run_zone_ocr is called
THEN:
- The interruption propagates
- bulk_update_documents still runs with the first document's id
"""
docs = DocumentFactory.create_batch(2)
template = OcrTemplateFactory()
mock_extract.side_effect = [[_written_result()], KeyboardInterrupt()]
with pytest.raises(KeyboardInterrupt):
bulk_edit.run_zone_ocr([d.pk for d in docs], template=template.pk)
mock_finish.assert_called_once_with([docs[0].pk])
@pytest.mark.parametrize(
("resolve_latest", "processes_root"),
[
pytest.param(True, False, id="resolves-latest"),
pytest.param(False, True, id="exact-document"),
],
)
def test_resolves_latest_only_when_asked(
self,
mock_extract: mock.MagicMock,
resolve_latest: bool, # noqa: FBT001
processes_root: bool, # noqa: FBT001
) -> None:
"""
GIVEN:
- A root document with a newer version and a template
WHEN:
- run_zone_ocr runs for the root id with resolve_latest on or off
THEN:
- The newest version is processed when resolve_latest is on
- The root document itself is processed when it is off
"""
root = DocumentFactory()
newest = DocumentFactory(root_document=root, version_index=1)
template = OcrTemplateFactory()
bulk_edit.run_zone_ocr(
[root.pk],
template=template.pk,
resolve_latest=resolve_latest,
)
mock_extract.assert_called_once_with(
root if processes_root else newest,
templates=[template],
)
def test_root_and_its_version_are_processed_once(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- A root document with a version, both listed in the selection
WHEN:
- run_zone_ocr runs with resolve_latest on
THEN:
- The newest version is extracted once
- The root is finished once
"""
root = DocumentFactory()
version = DocumentFactory(root_document=root, version_index=1)
template = OcrTemplateFactory()
mock_extract.return_value = [_written_result()]
bulk_edit.run_zone_ocr([root.pk, version.pk], template=template.pk)
mock_extract.assert_called_once_with(version, templates=[template])
mock_finish.assert_called_once_with([root.pk])
def test_a_version_reports_its_root(
self,
mock_extract: mock.MagicMock,
mock_finish: mock.MagicMock,
) -> None:
"""
GIVEN:
- A root with two versions and the older version selected
WHEN:
- run_zone_ocr runs without resolving the latest version
THEN:
- Extraction runs on that version, not the newest
- The root is finished
"""
root = DocumentFactory()
older = DocumentFactory(root_document=root, version_index=1)
DocumentFactory(root_document=root, version_index=2)
template = OcrTemplateFactory()
mock_extract.return_value = [_written_result()]
bulk_edit.run_zone_ocr([older.pk], template=template.pk, resolve_latest=False)
mock_extract.assert_called_once_with(older, templates=[template])
mock_finish.assert_called_once_with([root.pk])
@pytest.mark.django_db @pytest.mark.django_db
class TestBulkEditReprocess: class TestBulkEditReprocess:
@pytest.fixture @pytest.fixture
+173
View File
@@ -1,11 +1,13 @@
"""Request acceptance, worker execution, and task tracking.""" """Request acceptance, worker execution, and task tracking."""
import json
import uuid import uuid
from unittest import mock from unittest import mock
import pytest import pytest
from auditlog.models import LogEntry from auditlog.models import LogEntry
from django.contrib.auth.models import User from django.contrib.auth.models import User
from pytest_mock import MockerFixture
from rest_framework.exceptions import PermissionDenied from rest_framework.exceptions import PermissionDenied
from rest_framework.test import APIClient from rest_framework.test import APIClient
@@ -17,7 +19,12 @@ from documents.signals.handlers import before_task_publish_handler
from documents.signals.handlers import task_failure_handler from documents.signals.handlers import task_failure_handler
from documents.signals.handlers import task_postrun_handler from documents.signals.handlers import task_postrun_handler
from documents.signals.handlers import task_revoked_handler from documents.signals.handlers import task_revoked_handler
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import OcrTemplateFactory
from paperless_testing.factories import UserFactory from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
RUN_ZONE_OCR_BULK_URL = "/api/documents/run_zone_ocr/"
@pytest.fixture @pytest.fixture
@@ -35,6 +42,7 @@ def bulk_setup(paperless_dirs):
"endpoint,operation,parameters", "endpoint,operation,parameters",
[ [
("bulk_edit", "set_correspondent", {"correspondent": None}), ("bulk_edit", "set_correspondent", {"correspondent": None}),
("run_zone_ocr", "run_zone_ocr", {}),
], ],
) )
def test_request_only_queues(bulk_setup, endpoint, operation, parameters): def test_request_only_queues(bulk_setup, endpoint, operation, parameters):
@@ -43,6 +51,8 @@ def test_request_only_queues(bulk_setup, endpoint, operation, parameters):
data = {"documents": [doc.pk], "from_webui": True} data = {"documents": [doc.pk], "from_webui": True}
if endpoint == "bulk_edit": if endpoint == "bulk_edit":
data.update(method=operation, parameters=parameters) data.update(method=operation, parameters=parameters)
else:
data.update(parameters)
with mock.patch("documents.views.execute_bulk_operation.apply_async") as enqueue: with mock.patch("documents.views.execute_bulk_operation.apply_async") as enqueue:
enqueue.return_value.id = task_id enqueue.return_value.id = task_id
response = client.post(f"/api/documents/{endpoint}/", data, format="json") response = client.post(f"/api/documents/{endpoint}/", data, format="json")
@@ -59,6 +69,169 @@ def test_request_only_queues(bulk_setup, endpoint, operation, parameters):
assert enqueue.call_args.kwargs["headers"]["trigger_source"] == "web_ui" assert enqueue.call_args.kwargs["headers"]["trigger_source"] == "web_ui"
@pytest.mark.django_db
@pytest.mark.usefixtures("paperless_dirs")
class TestRunZoneOcrBulkOperation:
"""Zone OCR runs as a tracked bulk operation."""
@pytest.fixture
def document(self) -> Document:
return DocumentFactory()
def test_queues_only_template_id_and_returns_tracked_task(
self,
mocker: MockerFixture,
admin_client: APIClient,
admin_user: User,
document: Document,
) -> None:
"""
GIVEN:
- A zone OCR run request naming a template
WHEN:
- The request is posted to the API
THEN:
- Only the template id is queued, with a tracked task id returned
- Extraction does not run in the request
"""
template = OcrTemplateFactory()
enqueue = mocker.patch("documents.views.execute_bulk_operation.apply_async")
extract = mocker.patch(
"documents.bulk_edit.run_zone_extraction",
return_value=[],
)
enqueue.return_value.id = "zone-ocr-task"
response = admin_client.post(
RUN_ZONE_OCR_BULK_URL,
{
"documents": [document.pk],
"template": template.pk,
"from_webui": True,
},
format="json",
)
assert response.status_code == 200
assert response.data == {"result": "OK", "task_id": "zone-ocr-task"}
kwargs = enqueue.call_args.kwargs["kwargs"]
assert json.loads(json.dumps(kwargs)) == {
"operation": "run_zone_ocr",
"document_ids": [document.pk],
"parameters": {"template": template.pk},
"user_id": admin_user.pk,
"trigger_source": "web_ui",
}
extract.assert_not_called()
def test_finishes_extraction_within_the_tracked_task(
self,
mocker: MockerFixture,
admin_user: User,
document: Document,
) -> None:
"""
GIVEN:
- A queued zone OCR run naming a template
WHEN:
- The tracked task executes it
THEN:
- Extraction runs synchronously with only that template
- The result reports the operation and document count
"""
template = OcrTemplateFactory()
extract = mocker.patch(
"documents.bulk_edit.run_zone_extraction",
return_value=[],
)
result = execute_bulk_operation(
"run_zone_ocr",
[document.pk],
{"template": template.pk},
admin_user.pk,
)
extract.assert_called_once_with(document, templates=[template])
assert result == {"operation": "run_zone_ocr", "document_count": 1}
def test_without_template_selects_for_the_requester(
self,
mocker: MockerFixture,
admin_user: User,
document: Document,
) -> None:
"""
GIVEN:
- A queued zone OCR run without a chosen template
WHEN:
- The tracked task executes it
THEN:
- Templates are selected for the requesting user so they are limited
to those they can view
"""
select = mocker.patch("documents.bulk_edit.get_templates_for_document")
extract = mocker.patch(
"documents.bulk_edit.run_zone_extraction",
return_value=[],
)
execute_bulk_operation("run_zone_ocr", [document.pk], {}, admin_user.pk)
select.assert_called_once_with(document, requester=admin_user)
extract.assert_called_once_with(document, templates=select.return_value)
@pytest.mark.parametrize(
"revoked",
[
pytest.param("document", id="document"),
pytest.param("template", id="template"),
pytest.param("user", id="user"),
],
)
def test_rechecks_access_before_extraction(
self,
mocker: MockerFixture,
user_client: APIClient,
regular_user: User,
document: Document,
revoked: str,
) -> None:
"""
GIVEN:
- A zone OCR run accepted for a user with access to the document
and template
WHEN:
- Access to the document, the template, or the user is revoked before
the tracked task executes
THEN:
- Execution raises PermissionDenied
- Extraction never runs
"""
grant_global(regular_user, "change_document")
template = OcrTemplateFactory(owner=regular_user)
enqueue = mocker.patch("documents.views.execute_bulk_operation.apply_async")
enqueue.return_value.id = "zone-ocr-task"
response = user_client.post(
RUN_ZONE_OCR_BULK_URL,
{
"documents": [document.pk],
"template": template.pk,
},
format="json",
)
assert response.status_code == 200
if revoked == "document":
document.owner = UserFactory()
document.save(update_fields=["owner"])
elif revoked == "template":
template.owner = UserFactory()
template.save(update_fields=["owner"])
else:
User.objects.filter(pk=regular_user.pk).update(is_active=False)
extract = mocker.patch(
"documents.bulk_edit.run_zone_extraction",
return_value=[],
)
with pytest.raises(PermissionDenied):
execute_bulk_operation(**enqueue.call_args.kwargs["kwargs"])
extract.assert_not_called()
@pytest.mark.django_db @pytest.mark.django_db
def test_metadata_task_includes_audit_and_followup(bulk_setup, settings): def test_metadata_task_includes_audit_and_followup(bulk_setup, settings):
user, _, doc, correspondent = bulk_setup user, _, doc, correspondent = bulk_setup
@@ -0,0 +1,653 @@
"""Tests for zone OCR during consumption."""
from __future__ import annotations
import json
import logging
import shutil
import tempfile
from datetime import date
from pathlib import Path
from typing import TYPE_CHECKING
from typing import Any
from typing import Self
import pytest
from django.conf import settings
from django.db import transaction
from PIL import Image
from documents import consumer as consumer_module
from documents.consumer import ConsumerPlugin
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from paperless_testing.factories import CustomFieldFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import OcrTemplateFactory
from paperless_testing.factories import OcrTemplateZoneFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory
from paperless_testing.fakes.progress import FakeProgressManager
from paperless_testing.samples import SIMPLE_DIGITAL_PDF
if TYPE_CHECKING:
from collections.abc import Callable
from unittest.mock import MagicMock
from pytest_django.fixtures import SettingsWrapper
from pytest_mock import MockerFixture
from documents.zone_ocr import TemplatePlan
from paperless_testing.dirs import PaperlessDirs
CONTENT = "Northwind Power & Light statement"
# The text each zone's fake OCR returns, by zone name.
ZONE_TEXT = {
"Title": "Zone title",
"ASN": "ASN 777",
"Created": "2024-03-05",
"Code": "INV42",
}
MIME_TYPES = {
".pdf": "application/pdf",
".png": "image/png",
".eml": "message/rfc822",
}
class ZoneTestParser:
"""A parser returning CONTENT, with a PDF archive when one is requested."""
name = "zone-test-parser"
version = "0.1"
author = "test"
url = "test"
@classmethod
def supported_mime_types(cls) -> dict[str, str]:
return {mime: suffix for suffix, mime in MIME_TYPES.items()}
@classmethod
def score(cls, mime_type: str, filename: str, path: Path | None = None) -> int:
return 0
@property
def can_produce_archive(self) -> bool:
return True
@property
def requires_pdf_rendition(self) -> bool:
return False
def __enter__(self) -> Self:
self._tmpdir = Path(tempfile.mkdtemp(dir=settings.SCRATCH_DIR))
self._archive: Path | None = None
self._thumb = self._tmpdir / "thumb.webp"
self._thumb.write_bytes(b"thumbnail")
return self
def __exit__(self, *args: object) -> None:
shutil.rmtree(self._tmpdir, ignore_errors=True)
def configure(self, context: object) -> None:
"""The test parser ignores its context."""
def parse(
self,
document_path: Path,
mime_type: str,
*,
produce_archive: bool = True,
) -> None:
if produce_archive:
self._archive = self._tmpdir / "archive.pdf"
shutil.copy(SIMPLE_DIGITAL_PDF, self._archive)
def get_text(self) -> str:
return CONTENT
def get_date(self) -> None:
return None
def get_archive_path(self) -> Path | None:
return self._archive
def get_thumbnail(self, document_path: Path, mime_type: str) -> Path:
return self._thumb
def get_page_count(self, document_path: Path, mime_type: str) -> int:
return 1
def extract_metadata(self, document_path: Path, mime_type: str) -> list[Any]:
return []
@pytest.fixture
def page_image(tmp_path: Path) -> Path:
page = tmp_path / "page.png"
with Image.new("RGB", (200, 200), "white") as image:
image.save(page)
return page
@pytest.fixture
def mock_render(mocker: MockerFixture, page_image: Path) -> MagicMock:
"""Fake page rendering, recording whether a transaction was open."""
render = mocker.patch("documents.zone_ocr._render_pages")
render.in_transaction = []
def record(*args: Any, **kwargs: Any) -> dict[int, Path]:
render.in_transaction.append(transaction.get_connection().in_atomic_block)
return {0: page_image}
render.side_effect = record
return render
@pytest.fixture
def mock_ocr(mocker: MockerFixture) -> MagicMock:
"""Fake zone OCR returning ZONE_TEXT, recording whether a transaction was open."""
ocr = mocker.patch("documents.zone_ocr._ocr_text")
ocr.in_transaction = []
def read(cropped: Image.Image, zone: OcrTemplateZone, tmp_dir: Path) -> str:
ocr.in_transaction.append(transaction.get_connection().in_atomic_block)
return ZONE_TEXT[zone.name]
ocr.side_effect = read
return ocr
@pytest.fixture
def consume(
paperless_dirs: PaperlessDirs,
mocker: MockerFixture,
) -> Callable[..., Document]:
"""Consume a file of the given suffix through the ConsumerPlugin."""
registry = mocker.patch("documents.consumer.get_parser_registry")
registry.return_value.get_parser_for_file.return_value = ZoneTestParser
mocker.patch(
"documents.consumer.magic.from_file",
side_effect=lambda path, *, mime=False: MIME_TYPES[Path(path).suffix],
)
def run(
suffix: str = ".pdf",
overrides: DocumentMetadataOverrides | None = None,
root_document_id: int | None = None,
) -> Document:
source = paperless_dirs.scratch_dir / f"upload{suffix}"
if suffix == ".pdf":
shutil.copy(SIMPLE_DIGITAL_PDF, source)
else:
source.write_bytes(f"{suffix} {source.name}".encode())
plugin = ConsumerPlugin(
ConsumableDocument(
DocumentSource.ApiUpload,
original_file=source,
root_document_id=root_document_id,
),
overrides or DocumentMetadataOverrides(),
FakeProgressManager(source.name, None),
paperless_dirs.scratch_dir,
"task-id",
)
plugin.setup()
try:
result = plugin.run()
finally:
plugin.cleanup()
return Document.objects.get(pk=result["document_id"])
return run
def make_template(
name: str = "Invoice",
targets: tuple[str, ...] = ("Title",),
custom_field: CustomField | None = None,
**kwargs: Any,
) -> OcrTemplate:
"""A template matching CONTENT with one zone per named target."""
template = OcrTemplateFactory(
name=name,
match="Northwind",
source_width=100,
source_height=100,
**kwargs,
)
zone_targets = {
"Title": OcrTemplateZone.TargetType.TITLE,
"ASN": OcrTemplateZone.TargetType.ASN,
"Created": OcrTemplateZone.TargetType.CREATED,
"Code": OcrTemplateZone.TargetType.CUSTOM_FIELD,
}
for order, zone_name in enumerate(targets):
OcrTemplateZoneFactory(
template=template,
name=zone_name,
target=zone_targets[zone_name],
custom_field=custom_field if zone_name == "Code" else None,
y=order * 20,
width=50,
height=20,
order=order,
)
return template
def custom_field_value(document: Document, field: CustomField) -> Any:
instance = CustomFieldInstance.objects.filter(
document=document,
field=field,
).first()
return instance.value if instance is not None else None
@pytest.mark.django_db(transaction=True)
@pytest.mark.usefixtures("mock_ocr")
class TestConsumerZoneOcrTransaction:
"""Zone OCR runs outside the consumer's database transaction."""
def test_ocr_runs_before_the_transaction_opens(
self,
consume: Callable[..., Document],
mock_render: MagicMock,
mock_ocr: MagicMock,
) -> None:
"""
GIVEN:
- A template matching the consumed document's text
WHEN:
- The document is consumed
THEN:
- Pages are rendered and zones are read with no transaction open
- The zone value is written to the document
"""
make_template()
document = consume()
assert mock_render.in_transaction == [False]
assert mock_ocr.in_transaction == [False]
assert document.title == "Zone title"
@pytest.mark.django_db
@pytest.mark.usefixtures("mock_ocr")
class TestConsumerZoneOcr:
"""Zone values are computed before and written during consumption."""
@pytest.fixture
def code_field(self) -> CustomField:
return CustomFieldFactory(
name="Invoice code",
data_type=CustomField.FieldDataType.STRING,
)
@pytest.mark.parametrize(
("suffix", "generation", "expected_file"),
[
pytest.param(".pdf", "always", "archive.pdf", id="pdf-archive"),
pytest.param(".eml", "always", "archive.pdf", id="mail-archive"),
pytest.param(".pdf", "never", "upload.pdf", id="pdf-working-copy"),
pytest.param(".png", "never", "upload.png", id="image-working-copy"),
],
)
def test_zones_are_read_from_the_archive_or_working_copy(
self,
consume: Callable[..., Document],
mock_render: MagicMock,
settings: SettingsWrapper,
suffix: str,
generation: str,
expected_file: str,
) -> None:
"""
GIVEN:
- A template matching the consumed document's text
WHEN:
- A document is consumed with or without an archive
THEN:
- Zones are read from the archive when there is one, otherwise
from the working copy of the PDF or image
"""
settings.ARCHIVE_FILE_GENERATION = generation
make_template()
document = consume(suffix)
mock_render.assert_called_once()
assert mock_render.call_args.args[0].name == expected_file
assert document.title == "Zone title"
def test_mail_without_archive_is_not_rendered(
self,
consume: Callable[..., Document],
mock_render: MagicMock,
settings: SettingsWrapper,
) -> None:
"""
GIVEN:
- A template matching the consumed document's text
WHEN:
- An email is consumed without an archive
THEN:
- No page is rendered from the email file
- The document is consumed without zone values
"""
settings.ARCHIVE_FILE_GENERATION = "never"
make_template()
document = consume(".eml")
mock_render.assert_not_called()
assert document.title == "upload"
@pytest.mark.usefixtures("mock_render")
def test_values_are_written_before_document_added_workflows(
self,
consume: Callable[..., Document],
code_field: CustomField,
) -> None:
"""
GIVEN:
- A template writing a custom field
- A DOCUMENT_ADDED workflow filtering on that custom field's value
and assigning a tag
WHEN:
- A matching document is consumed
THEN:
- The workflow sees the zone value and assigns its tag
"""
make_template(targets=("Code",), custom_field=code_field)
tag = TagFactory(name="Zone matched")
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
filter_custom_field_query=json.dumps(
["AND", [[code_field.pk, "exact", "INV42"]]],
),
)
action = WorkflowAction.objects.create()
action.assign_tags.add(tag)
workflow = Workflow.objects.create(name="On zone value", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(action)
document = consume()
assert custom_field_value(document, code_field) == "INV42"
assert list(document.tags.all()) == [tag]
@pytest.mark.parametrize(
"overridden",
[
pytest.param("title", id="title"),
pytest.param("asn", id="asn"),
pytest.param("created", id="created"),
pytest.param("custom_field", id="custom-field"),
],
)
@pytest.mark.usefixtures("mock_render")
def test_explicit_values_are_not_overwritten(
self,
consume: Callable[..., Document],
code_field: CustomField,
overridden: str,
) -> None:
"""
GIVEN:
- A template with title, ASN, created and custom field zones
- Metadata overrides giving one of those fields a value
WHEN:
- A matching document is consumed
THEN:
- The supplied value is kept
- Every other zone value is written
"""
make_template(
targets=("Title", "ASN", "Created", "Code"),
custom_field=code_field,
)
explicit = {
"title": "Upload title",
"asn": 123,
"created": date(2020, 1, 2),
"custom_field": "Given",
}
overrides = {
"title": DocumentMetadataOverrides(title="Upload title"),
"asn": DocumentMetadataOverrides(asn=123),
"created": DocumentMetadataOverrides(created=date(2020, 1, 2)),
"custom_field": DocumentMetadataOverrides(
custom_fields={code_field.pk: "Given"},
),
}[overridden]
expected = {
"title": "Zone title",
"asn": 777,
"created": date(2024, 3, 5),
"custom_field": "INV42",
} | {overridden: explicit[overridden]}
document = consume(overrides=overrides)
assert {
"title": document.title,
"asn": document.archive_serial_number,
"created": document.created,
"custom_field": custom_field_value(document, code_field),
} == expected
@pytest.mark.usefixtures("mock_render")
def test_custom_field_listed_without_a_value_is_filled(
self,
consume: Callable[..., Document],
code_field: CustomField,
) -> None:
"""
GIVEN:
- A template with a custom field zone
- Metadata overrides listing that custom field without a value
WHEN:
- A matching document is consumed
THEN:
- The zone value is written to the custom field
"""
make_template(targets=("Code",), custom_field=code_field)
document = consume(
overrides=DocumentMetadataOverrides(custom_fields={code_field.pk: None}),
)
assert custom_field_value(document, code_field) == "INV42"
@pytest.mark.usefixtures("mock_render")
def test_custom_field_deleted_before_writing_is_skipped(
self,
consume: Callable[..., Document],
code_field: CustomField,
mocker: MockerFixture,
caplog: pytest.LogCaptureFixture,
) -> None:
"""
GIVEN:
- A template with title and custom field zones
- The custom field deleted after the zone values are planned
WHEN:
- A matching document is consumed
THEN:
- The document is stored with the title
- The custom field zone is reported as unassigned
"""
make_template(targets=("Title", "Code"), custom_field=code_field)
plan = consumer_module.plan_zone_extraction
def plan_then_delete(*args: Any, **kwargs: Any) -> list[TemplatePlan]:
plans = plan(*args, **kwargs)
CustomField.objects.filter(pk=code_field.pk).delete()
return plans
mocker.patch(
"documents.consumer.plan_zone_extraction",
side_effect=plan_then_delete,
)
caplog.set_level(logging.DEBUG, logger="paperless.consumer")
document = consume()
assert document.title == "Zone title"
assert not CustomFieldInstance.objects.filter(document=document).exists()
assert "zone 'Code' for Invoice code: unassigned" in caplog.text
@pytest.mark.parametrize(
("with_owner", "expected_title", "expected_code"),
[
pytest.param(True, "Zone title", None, id="owner-sees-own-templates"),
pytest.param(False, "Zone title", "INV42", id="no-owner-sees-all"),
],
)
@pytest.mark.usefixtures("mock_render")
def test_templates_are_selected_by_the_metadata_owner(
self,
consume: Callable[..., Document],
code_field: CustomField,
with_owner: bool, # noqa: FBT001
expected_title: str,
expected_code: str | None,
) -> None:
"""
GIVEN:
- A template owned by the upload's owner writing the title
- A private template of another user writing a custom field
WHEN:
- A matching document is consumed with or without an owner
THEN:
- With an owner, only the templates that owner can view apply
- Without an owner, every template applies
"""
owner = UserFactory()
make_template("Mine", owner=owner)
make_template(
"Private",
targets=("Code",),
custom_field=code_field,
owner=UserFactory(),
)
document = consume(
overrides=DocumentMetadataOverrides(
owner_id=owner.pk if with_owner else None,
),
)
assert document.title == expected_title
assert custom_field_value(document, code_field) == expected_code
@pytest.mark.parametrize(
"failing",
[
pytest.param(
"documents.consumer.get_templates_for_content",
id="template-selection",
),
pytest.param("documents.consumer.plan_zone_extraction", id="extraction"),
],
)
@pytest.mark.usefixtures("mock_render")
def test_read_phase_failure_does_not_fail_consumption(
self,
consume: Callable[..., Document],
mocker: MockerFixture,
caplog: pytest.LogCaptureFixture,
failing: str,
) -> None:
"""
GIVEN:
- A template matching the consumed document's text
- Template selection or zone extraction raising an error
WHEN:
- The document is consumed
THEN:
- The document is stored without zone values
- The failure is logged
"""
make_template()
mocker.patch(failing, side_effect=RuntimeError("zone OCR broke"))
document = consume()
assert document.title == "upload"
assert "Zone OCR failed for upload.pdf" in caplog.text
@pytest.mark.usefixtures("mock_render")
def test_write_failure_does_not_fail_consumption(
self,
consume: Callable[..., Document],
mocker: MockerFixture,
caplog: pytest.LogCaptureFixture,
) -> None:
"""
GIVEN:
- A template matching the consumed document's text
- Applying the planned zone values raising an error
WHEN:
- The document is consumed
THEN:
- The document is stored without the template's values
- The failure is logged
"""
make_template()
mocker.patch(
"documents.consumer.apply_zone_plans",
side_effect=RuntimeError("write failed"),
)
document = consume()
assert document.title == "upload"
assert "Zone OCR could not write values for upload.pdf" in caplog.text
@pytest.mark.usefixtures("mock_render")
def test_version_values_are_written_to_the_root(
self,
consume: Callable[..., Document],
code_field: CustomField,
) -> None:
"""
GIVEN:
- A root document whose owner can view a template writing the title
and a custom field
- A private template of another user
WHEN:
- A new version of the root document is consumed
THEN:
- The zone values are written to the root document
- The version keeps the root's previous title and gets no custom field
- The other user's template does not apply
"""
owner = UserFactory()
root = DocumentFactory(title="Root title", owner=owner, content=CONTENT)
make_template(targets=("Title", "Code"), custom_field=code_field, owner=owner)
make_template("Private", targets=("ASN",), owner=UserFactory())
version = consume(root_document_id=root.pk)
root.refresh_from_db()
assert version.root_document_id == root.pk
assert root.title == "Zone title"
assert custom_field_value(root, code_field) == "INV42"
assert root.archive_serial_number is None
assert version.title == "Root title"
assert custom_field_value(version, code_field) is None
@@ -36,6 +36,8 @@ from documents.models import Document
from documents.models import DocumentBarcode from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import Note from documents.models import Note
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import ShareLink from documents.models import ShareLink
from documents.models import ShareLinkBundle from documents.models import ShareLinkBundle
from documents.models import StoragePath from documents.models import StoragePath
@@ -58,6 +60,8 @@ from paperless_testing.dirs import PaperlessDirs
from paperless_testing.dirs import paperless_environment from paperless_testing.dirs import paperless_environment
from paperless_testing.factories import DocumentBarcodeFactory from paperless_testing.factories import DocumentBarcodeFactory
from paperless_testing.factories import DocumentFactory from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import OcrTemplateFactory
from paperless_testing.factories import OcrTemplateZoneFactory
from paperless_testing.permissions import grant_object from paperless_testing.permissions import grant_object
from paperless_testing.samples import install_document_samples from paperless_testing.samples import install_document_samples
@@ -415,6 +419,56 @@ class TestExportImport(
b"share-bundle-contents", b"share-bundle-contents",
) )
def test_exporter_includes_ocr_templates(self) -> None:
"""
GIVEN:
- An OCR template with a sample document and zones targeting a
custom field and a built-in field
WHEN:
- The data is exported, the templates are deleted, and the export is
imported again
THEN:
- The template and its zones are restored
"""
shutil.rmtree(Path(self.dirs.media_dir) / "documents")
install_document_samples(Path(self.dirs.media_dir) / "documents")
template = OcrTemplateFactory(
name="Invoice",
owner=self.user,
sample_document=self.d1,
combine_formats={str(self.cf1.pk): "{Number}"},
)
cf_zone = OcrTemplateZoneFactory(
template=template,
name="Number",
target=OcrTemplateZone.TargetType.CUSTOM_FIELD,
custom_field=self.cf1,
)
title_zone = OcrTemplateZoneFactory(template=template, name="Title")
manifest = self._do_export()
exported_models = [record["model"] for record in manifest]
self.assertEqual(exported_models.count("documents.ocrtemplate"), 1)
self.assertEqual(exported_models.count("documents.ocrtemplatezone"), 2)
with paperless_environment():
OcrTemplate.objects.all().delete()
call_command(
"document_importer",
"--no-progress-bar",
self.target,
skip_checks=True,
)
imported = OcrTemplate.objects.get(pk=template.pk)
self.assertEqual(
list(imported.zones.order_by("pk").values_list("pk", flat=True)),
[cf_zone.pk, title_zone.pk],
)
def test_update_export_changed_time(self) -> None: def test_update_export_changed_time(self) -> None:
shutil.rmtree(Path(self.dirs.media_dir) / "documents") shutil.rmtree(Path(self.dirs.media_dir) / "documents")
install_document_samples(Path(self.dirs.media_dir) / "documents") install_document_samples(Path(self.dirs.media_dir) / "documents")
+91
View File
@@ -421,6 +421,97 @@ class TestRasterizePdfPageToPng:
with pytest.raises(ParseError): with pytest.raises(ParseError):
rasterize_pdf_page_to_png(bad, tmp_path / "page1.png", dpi=72) rasterize_pdf_page_to_png(bad, tmp_path / "page1.png", dpi=72)
def test_renders_the_requested_page(self, tmp_path: Path) -> None:
"""
GIVEN:
- A two page PDF with a 144x72 first page and a 300x300 second page
WHEN:
- The second page is rasterized at 72 DPI
THEN:
- Only out_path is written, sized like the second page
"""
pdf_path = self._write_pdf(tmp_path / "in.pdf")
out_dir = tmp_path / "out"
out_dir.mkdir()
out_path = out_dir / "page2.png"
rasterize_pdf_page_to_png(pdf_path, out_path, dpi=72, page=2, timeout=30)
assert list(out_dir.iterdir()) == [out_path]
with Image.open(out_path) as im:
assert im.size == (300, 300)
def test_page_out_of_range_raises_parse_error(self, tmp_path: Path) -> None:
"""
GIVEN:
- A two page PDF
WHEN:
- A page past the end is rasterized
THEN:
- A ParseError is raised and nothing is written
"""
pdf_path = self._write_pdf(tmp_path / "in.pdf")
out_path = tmp_path / "page3.png"
with pytest.raises(ParseError):
rasterize_pdf_page_to_png(pdf_path, out_path, dpi=72, page=3)
assert not out_path.exists()
def test_timeout_and_resolution_are_passed_to_pdftoppm(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- A subprocess runner that succeeds
WHEN:
- Page 4 is rasterized at a non-72 dpi with a timeout
THEN:
- The timeout reaches the runner
- The resolution is passed to pdftoppm
"""
run = mocker.patch("documents.parsers.run_subprocess")
rasterize_pdf_page_to_png(
tmp_path / "in.pdf",
tmp_path / "out.png",
dpi=150,
page=4,
timeout=12,
)
args = run.call_args.args[0]
assert args[args.index("-r") + 1] == "150"
assert run.call_args.kwargs["timeout"] == 12
def test_timeout_raises_parse_error(
self,
tmp_path: Path,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- A subprocess runner that times out
WHEN:
- A page is rasterized
THEN:
- A ParseError is raised
"""
mocker.patch(
"documents.parsers.run_subprocess",
side_effect=subprocess.TimeoutExpired(["pdftoppm"], 5),
)
with pytest.raises(ParseError, match="timed out"):
rasterize_pdf_page_to_png(
tmp_path / "in.pdf",
tmp_path / "out.png",
dpi=72,
timeout=5,
)
class TestEncodeThumbnailWebp: class TestEncodeThumbnailWebp:
@pytest.mark.parametrize( @pytest.mark.parametrize(
+16 -1
View File
@@ -4,6 +4,7 @@ from django.conf import settings
from pytest_mock import MockerFixture from pytest_mock import MockerFixture
from documents.regex import safe_regex_finditer from documents.regex import safe_regex_finditer
from documents.regex import safe_regex_fullmatch
from documents.regex import safe_regex_match from documents.regex import safe_regex_match
from documents.regex import safe_regex_search from documents.regex import safe_regex_search
from documents.regex import safe_regex_sub from documents.regex import safe_regex_sub
@@ -26,7 +27,7 @@ class TestValidateRegexPattern:
class TestSafeRegexSearchAndMatch: class TestSafeRegexSearchAndMatch:
"""Tests for safe_regex_search and safe_regex_match (same contract).""" """Tests for safe_regex_search, _match and _fullmatch (same contract)."""
@pytest.mark.parametrize( @pytest.mark.parametrize(
("func", "pattern", "text", "expected_group"), ("func", "pattern", "text", "expected_group"),
@@ -45,6 +46,13 @@ class TestSafeRegexSearchAndMatch:
"123", "123",
id="match-match-found", id="match-match-found",
), ),
pytest.param(
safe_regex_fullmatch,
r"\d+",
"123",
"123",
id="fullmatch-match-found",
),
], ],
) )
def test_match_found(self, func, pattern, text, expected_group) -> None: def test_match_found(self, func, pattern, text, expected_group) -> None:
@@ -57,6 +65,12 @@ class TestSafeRegexSearchAndMatch:
[ [
pytest.param(safe_regex_search, r"\d+", "abcdef", id="search-no-match"), pytest.param(safe_regex_search, r"\d+", "abcdef", id="search-no-match"),
pytest.param(safe_regex_match, r"\d+", "abc123", id="match-no-match"), pytest.param(safe_regex_match, r"\d+", "abc123", id="match-no-match"),
pytest.param(
safe_regex_fullmatch,
r"\d+",
"123abc",
id="fullmatch-no-match",
),
], ],
) )
def test_no_match(self, func, pattern, text) -> None: def test_no_match(self, func, pattern, text) -> None:
@@ -67,6 +81,7 @@ class TestSafeRegexSearchAndMatch:
[ [
pytest.param(safe_regex_search, id="search"), pytest.param(safe_regex_search, id="search"),
pytest.param(safe_regex_match, id="match"), pytest.param(safe_regex_match, id="match"),
pytest.param(safe_regex_fullmatch, id="fullmatch"),
], ],
) )
def test_invalid_pattern_returns_none(self, func) -> None: def test_invalid_pattern_returns_none(self, func) -> None:
+55
View File
@@ -0,0 +1,55 @@
import pytest
from django.core.exceptions import ValidationError
from documents.validators import monetary_validator
class TestMonetaryValidator:
"""Tests for the monetary_validator function."""
@pytest.mark.parametrize(
"value",
[
pytest.param("12", id="whole-number"),
pytest.param("-12.50", id="negative"),
pytest.param("9999999999.99", id="twelve-digits"),
pytest.param("GBP123.45", id="code-with-amount"),
pytest.param("USD-12.50", id="code-with-negative"),
pytest.param("EUR12.5", id="code-with-one-decimal"),
],
)
def test_valid_values(self, value: str) -> None:
"""
GIVEN:
- A plain decimal or an amount prefixed with a currency code
WHEN:
- The value is validated
THEN:
- No error is raised
"""
monetary_validator(value)
@pytest.mark.parametrize(
"value",
[
pytest.param("12.345", id="too-many-decimals"),
pytest.param("99999999999.99", id="too-many-digits"),
pytest.param("GBP12.345", id="code-with-too-many-decimals"),
pytest.param("GBP12", id="code-without-decimals"),
pytest.param("gbp12.50", id="lowercase-code"),
pytest.param("GB12.50", id="short-code"),
pytest.param("abc", id="not-a-number"),
pytest.param("NaN", id="nan"),
],
)
def test_invalid_values(self, value: str) -> None:
"""
GIVEN:
- A value that is neither a valid decimal nor a coded amount
WHEN:
- The value is validated
THEN:
- A ValidationError is raised
"""
with pytest.raises(ValidationError):
monetary_validator(value)
File diff suppressed because it is too large. Load diff
+11 -2
View File
@@ -164,14 +164,23 @@ def run_subprocess(
check_exit_code: bool = True, check_exit_code: bool = True,
log_stdout: bool = True, log_stdout: bool = True,
log_stderr: bool = True, log_stderr: bool = True,
timeout: float | None = None,
) -> CompletedProcess: ) -> CompletedProcess:
""" """
Runs a subprocess and logs its output, checking return code if requested Runs a subprocess and logs its output, checking return code if requested.
A timeout, in seconds, raises subprocess.TimeoutExpired once exceeded.
""" """
proc_name = arguments[0] proc_name = arguments[0]
completed_proc = run(args=arguments, env=env, capture_output=True, check=False) completed_proc = run(
args=arguments,
env=env,
capture_output=True,
check=False,
timeout=timeout,
)
if logger: if logger:
logger.info(f"{proc_name} exited {completed_proc.returncode}") logger.info(f"{proc_name} exited {completed_proc.returncode}")
+28
View File
@@ -1,6 +1,9 @@
from decimal import Decimal
from urllib.parse import urlparse from urllib.parse import urlparse
from django.core.exceptions import ValidationError from django.core.exceptions import ValidationError
from django.core.validators import DecimalValidator
from django.core.validators import RegexValidator
from django.utils.translation import gettext_lazy as _ from django.utils.translation import gettext_lazy as _
@@ -58,3 +61,28 @@ def url_validator(value) -> None:
ValidationError: If the URL is invalid or not using http/https scheme ValidationError: If the URL is invalid or not using http/https scheme
""" """
uri_validator(value, allowed_schemes={"http", "https"}) uri_validator(value, allowed_schemes={"http", "https"})
def monetary_validator(value: str) -> None:
"""
Validates the stored value of a monetary custom field.
Accepts either a plain decimal with at most 12 digits and 2 decimal places
(the legacy format), or a two-decimal number prefixed by a three letter
uppercase currency code, e.g. GBP123.45 or USD-12.50.
Args:
value: The monetary value to validate
Raises:
ValidationError: If the value matches neither format
"""
try:
# First try to validate as a number from legacy format
DecimalValidator(max_digits=12, decimal_places=2)(Decimal(str(value)))
except Exception:
# If that fails, try to validate as a monetary string
RegexValidator(
regex=r"^[A-Z]{3}-?\d+(\.\d{1,2})$",
message="Must be a two-decimal number with optional currency code e.g. GBP123.45",
)(value)
+344 -25
View File
@@ -10,6 +10,7 @@ from collections import deque
from collections.abc import AsyncIterator from collections.abc import AsyncIterator
from collections.abc import Callable from collections.abc import Callable
from collections.abc import Generator from collections.abc import Generator
from collections.abc import Sequence
from datetime import datetime from datetime import datetime
from datetime import timedelta from datetime import timedelta
from http import HTTPStatus from http import HTTPStatus
@@ -22,6 +23,7 @@ from typing import TYPE_CHECKING
from typing import Any from typing import Any
from typing import Final from typing import Final
from typing import NamedTuple from typing import NamedTuple
from typing import cast
from unicodedata import normalize from unicodedata import normalize
from urllib.parse import quote from urllib.parse import quote
from urllib.parse import urlparse from urllib.parse import urlparse
@@ -162,6 +164,8 @@ from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import Note from documents.models import Note
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import SavedView from documents.models import SavedView
from documents.models import ShareLink from documents.models import ShareLink
@@ -172,12 +176,15 @@ from documents.models import UiSettings
from documents.models import Workflow from documents.models import Workflow
from documents.models import WorkflowAction from documents.models import WorkflowAction
from documents.models import WorkflowTrigger from documents.models import WorkflowTrigger
from documents.parsers import ParseError
from documents.parsers import rasterize_pdf_page_to_png
from documents.permissions import AcknowledgeTasksPermissions from documents.permissions import AcknowledgeTasksPermissions
from documents.permissions import PaperlessAdminPermissions from documents.permissions import PaperlessAdminPermissions
from documents.permissions import PaperlessNotePermissions from documents.permissions import PaperlessNotePermissions
from documents.permissions import PaperlessObjectPermissions from documents.permissions import PaperlessObjectPermissions
from documents.permissions import TrashPermissions from documents.permissions import TrashPermissions
from documents.permissions import ViewDocumentsPermissions from documents.permissions import ViewDocumentsPermissions
from documents.permissions import ViewOcrTemplatesPermissions
from documents.permissions import annotate_document_count_for_related_queryset from documents.permissions import annotate_document_count_for_related_queryset
from documents.permissions import get_document_count_filter_for_user from documents.permissions import get_document_count_filter_for_user
from documents.permissions import get_permission_grantees from documents.permissions import get_permission_grantees
@@ -191,6 +198,7 @@ from documents.permissions import restrict_queryset_to_visible
from documents.permissions import set_permissions_for_objects from documents.permissions import set_permissions_for_objects
from documents.permissions import user_is_unrestricted from documents.permissions import user_is_unrestricted
from documents.plugins.date_parsing import get_date_parser from documents.plugins.date_parsing import get_date_parser
from documents.regex import safe_regex_fullmatch
from documents.schema import generate_object_with_permissions_schema from documents.schema import generate_object_with_permissions_schema
from documents.search import SearchHit from documents.search import SearchHit
from documents.serialisers import AcknowledgeTasksViewSerializer from documents.serialisers import AcknowledgeTasksViewSerializer
@@ -212,11 +220,18 @@ from documents.serialisers import MergeDocumentsAsVersionsSerializer
from documents.serialisers import MergeDocumentsSerializer from documents.serialisers import MergeDocumentsSerializer
from documents.serialisers import NoteCreateSerializer from documents.serialisers import NoteCreateSerializer
from documents.serialisers import NotesSerializer from documents.serialisers import NotesSerializer
from documents.serialisers import OcrErrorSerializer
from documents.serialisers import OcrTemplateSerializer
from documents.serialisers import OcrTemplateTestResultSerializer
from documents.serialisers import OcrTemplateTestSerializer
from documents.serialisers import PostDocumentSerializer from documents.serialisers import PostDocumentSerializer
from documents.serialisers import RemovePasswordDocumentsSerializer from documents.serialisers import RemovePasswordDocumentsSerializer
from documents.serialisers import ReprocessDocumentsSerializer from documents.serialisers import ReprocessDocumentsSerializer
from documents.serialisers import RotateDocumentsSerializer from documents.serialisers import RotateDocumentsSerializer
from documents.serialisers import RunTaskSerializer from documents.serialisers import RunTaskSerializer
from documents.serialisers import RunZoneOcrDocumentsSerializer
from documents.serialisers import RunZoneOcrResponseSerializer
from documents.serialisers import RunZoneOcrSerializer
from documents.serialisers import SavedViewSerializer from documents.serialisers import SavedViewSerializer
from documents.serialisers import SearchResultSerializer from documents.serialisers import SearchResultSerializer
from documents.serialisers import SerializerWithPerms from documents.serialisers import SerializerWithPerms
@@ -248,6 +263,7 @@ from documents.versioning import get_latest_version_for_root
from documents.versioning import get_request_version_param from documents.versioning import get_request_version_param
from documents.versioning import get_root_document from documents.versioning import get_root_document
from documents.versioning import latest_version_content_prefetch from documents.versioning import latest_version_content_prefetch
from documents.versioning import resolve_effective_document
from documents.versioning import resolve_requested_version_for_root from documents.versioning import resolve_requested_version_for_root
from documents.versioning import versions_newest_first from documents.versioning import versions_newest_first
from paperless import version from paperless import version
@@ -2259,6 +2275,98 @@ class DocumentViewSet(
raise Http404 raise Http404
return version_doc return version_doc
@extend_schema(
operation_id="documents_run_zone_ocr",
parameters=[
OpenApiParameter(
name="version",
type=OpenApiTypes.INT,
location=OpenApiParameter.QUERY,
description="File version ID. Defaults to the latest version.",
),
],
request=RunZoneOcrSerializer,
responses={
(200, "application/json"): RunZoneOcrResponseSerializer,
(404, "application/json"): OcrErrorSerializer,
},
)
@action(
methods=["post"],
detail=True,
url_path="run-zone-ocr",
permission_classes=[IsAuthenticated],
)
def run_zone_ocr(self, request: Request, pk: str | None = None) -> HttpResponse:
"""
Queue zone-based OCR extraction on this document, with the given
template or with every template that matches it. The file version
given by the version parameter is run when it is not the root
document. Responds with the task id, or skipped when no template
applies.
"""
serializer = RunZoneOcrSerializer(
data=request.data,
context={"request": request},
)
serializer.is_valid(raise_exception=True)
template = serializer.validated_data.get("template")
document: Document = get_object_or_404(
Document.objects.select_related("root_document__owner"),
pk=pk,
)
user = cast("User", request.user)
if not user.has_perm(
"documents.change_document",
) or not has_perms_owner_aware(user, "change_document", document):
return HttpResponseForbidden("Insufficient permissions")
from documents.zone_ocr import ZoneOcrSkipReason
from documents.zone_ocr import get_templates_for_document
from documents.zone_ocr import resolve_zone_source
resolution = resolve_effective_document(document, request)
if resolution.error == VersionResolutionError.INVALID:
raise NotFound("Invalid version parameter")
file_document = resolution.document
if file_document is None:
raise Http404
if template is None and not get_templates_for_document(
file_document,
requester=user,
):
return Response(
{
"result": "skipped",
"reason": ZoneOcrSkipReason.NO_TEMPLATE,
},
)
if resolve_zone_source(file_document) is None:
return Response(
{"error": "Document file not found"},
status=status.HTTP_404_NOT_FOUND,
)
parameters: dict[str, Any] = {"resolve_latest": False}
if template is not None:
parameters["template"] = template.pk
try:
return queue_bulk_operation(
user,
bulk_edit.run_zone_ocr,
[file_document.pk],
parameters,
from_webui=serializer.validated_data["from_webui"],
)
except Exception as e:
logger.warning(f"An error occurred performing document zone OCR: {e!s}")
return HttpResponseBadRequest(
"Error performing document zone OCR, check logs for more detail.",
)
@extend_schema( @extend_schema(
operation_id="documents_delete_version", operation_id="documents_delete_version",
parameters=[ parameters=[
@@ -2993,11 +3101,18 @@ class DocumentSelectionMixin:
return list(filtered_documents.values_list("pk", flat=True)) return list(filtered_documents.values_list("pk", flat=True))
class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin): def queue_bulk_operation(
permission_classes = (IsAuthenticated,) user: User,
parser_classes = (parsers.JSONParser,) method: Callable[..., Any],
documents: list[int],
def _queue_bulk_operation(self, method, documents, parameters, from_webui): parameters: dict[str, Any],
*,
from_webui: bool,
) -> Response:
"""
Validate the parameters against the bulk edit method, queue it as a tracked
bulk operation task and return the task id.
"""
if "update_callback" in parameters: if "update_callback" in parameters:
raise ValueError("Unsupported bulk operation parameter: update_callback") raise ValueError("Unsupported bulk operation parameter: update_callback")
trigger_source = ( trigger_source = (
@@ -3007,7 +3122,7 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
) )
final_parameters = parameters.copy() final_parameters = parameters.copy()
if method.__name__ in USER_OPERATIONS: if method.__name__ in USER_OPERATIONS:
final_parameters["user"] = self.request.user final_parameters["user"] = user
if method.__name__ in TRIGGER_SOURCE_OPERATIONS: if method.__name__ in TRIGGER_SOURCE_OPERATIONS:
final_parameters["trigger_source"] = trigger_source final_parameters["trigger_source"] = trigger_source
signature(method).bind(documents, **final_parameters) signature(method).bind(documents, **final_parameters)
@@ -3016,13 +3131,27 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
"operation": method.__name__, "operation": method.__name__,
"document_ids": documents, "document_ids": documents,
"parameters": parameters, "parameters": parameters,
"user_id": self.request.user.pk, "user_id": user.pk,
"trigger_source": trigger_source, "trigger_source": trigger_source,
}, },
headers={"trigger_source": trigger_source}, headers={"trigger_source": trigger_source},
) )
return Response({"result": "OK", "task_id": task.id}) return Response({"result": "OK", "task_id": task.id})
class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
permission_classes = (IsAuthenticated,)
parser_classes = (parsers.JSONParser,)
def _queue_bulk_operation(self, method, documents, parameters, from_webui):
return queue_bulk_operation(
cast("User", self.request.user),
method,
documents,
parameters,
from_webui=from_webui,
)
def _has_document_permissions( def _has_document_permissions(
self, self,
*, *,
@@ -3376,6 +3505,37 @@ class ReprocessDocumentsView(DocumentOperationPermissionMixin):
) )
@extend_schema_view(
post=extend_schema(
operation_id="documents_run_zone_ocr_bulk",
description="Run OCR template zone extraction on selected documents",
responses={
200: inline_serializer(
name="RunZoneOcrDocumentsResult",
fields={
"result": serializers.CharField(),
"task_id": serializers.CharField(),
},
),
},
),
)
class RunZoneOcrDocumentsView(DocumentOperationPermissionMixin):
serializer_class = RunZoneOcrDocumentsSerializer
def post(self, request: Request, *args: Any, **kwargs: Any) -> HttpResponse:
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
template = serializer.validated_data.get("template")
if template is not None:
serializer.validated_data["template"] = template.pk
return self._execute_document_action(
method=bulk_edit.run_zone_ocr,
validated_data=serializer.validated_data,
operation_label="document zone OCR",
)
@extend_schema_view( @extend_schema_view(
post=extend_schema( post=extend_schema(
operation_id="documents_edit_pdf", operation_id="documents_edit_pdf",
@@ -5318,8 +5478,6 @@ class CustomFieldViewSet(PermissionsAwareDocumentCountMixin, ModelViewSet[Custom
class SystemStatusView(PassUserMixin): class SystemStatusView(PassUserMixin):
permission_classes = (IsAuthenticated,) permission_classes = (IsAuthenticated,)
TASK_SUMMARY_DAYS = 30 TASK_SUMMARY_DAYS = 30
# Seconds to wait on the broker before reporting it as unreachable
BROKER_SOCKET_TIMEOUT = 5
def get(self, request, format=None): def get(self, request, format=None):
if not has_system_status_permission(request.user): if not has_system_status_permission(request.user):
@@ -5365,11 +5523,7 @@ class SystemStatusView(PassUserMixin):
if redis_url_parsed.hostname is not None: if redis_url_parsed.hostname is not None:
redis_constructed_url += f":{redis_url_parsed.port}" redis_constructed_url += f":{redis_url_parsed.port}"
redis_error = None redis_error = None
with Redis.from_url( with Redis.from_url(url=redis_url) as client:
url=redis_url,
socket_timeout=self.BROKER_SOCKET_TIMEOUT,
socket_connect_timeout=self.BROKER_SOCKET_TIMEOUT,
) as client:
try: try:
client.ping() client.ping()
redis_status = "OK" redis_status = "OK"
@@ -5384,18 +5538,8 @@ class SystemStatusView(PassUserMixin):
celery_url = None celery_url = None
try: try:
celery_ping = None celery_ping = None
# Use a dedicated connection with socket timeouts. Without them a
# stalled broker blocks this request (and its DB connection) forever
with celery_app.connection_for_read(
transport_options={
"socket_timeout": self.BROKER_SOCKET_TIMEOUT,
"socket_connect_timeout": self.BROKER_SOCKET_TIMEOUT,
},
) as celery_connection:
for ping_attempt in range(3): for ping_attempt in range(3):
celery_ping = celery_app.control.inspect( celery_ping = celery_app.control.inspect().ping()
connection=celery_connection,
).ping()
if celery_ping: if celery_ping:
break break
if ping_attempt < 2: if ping_attempt < 2:
@@ -5693,3 +5837,178 @@ def serve_logo(request: HttpRequest, filename: str | None = None) -> FileRespons
filename=logo_name, filename=logo_name,
as_attachment=True, as_attachment=True,
) )
class OcrTemplateViewSet(BulkPermissionMixin, PassUserMixin, ModelViewSet[OcrTemplate]):
"""CRUD for OCR templates with zone definitions."""
model = OcrTemplate
queryset = (
OcrTemplate.objects.select_related("owner")
.prefetch_related(
"zones",
"zones__custom_field",
)
.order_by("name")
)
serializer_class = OcrTemplateSerializer
pagination_class = StandardPagination
permission_classes = (IsAuthenticated, PaperlessObjectPermissions)
filter_backends = (
OrderingFilter,
PermittedObjectsFilter,
)
ordering_fields = ("name",)
@staticmethod
def _resolve_file_document(document: Document, request: Request) -> Document:
"""The version whose file zone OCR reads for this request."""
resolution = resolve_effective_document(document, request)
if resolution.error == VersionResolutionError.INVALID:
raise NotFound("Invalid version parameter")
if resolution.document is None:
raise Http404
resolved = resolution.document
if resolved.get_deferred_fields():
return Document.objects.get(pk=resolved.pk)
return resolved
def get_permissions(self) -> Sequence[Any]:
if self.action in {"document_page_image", "test_zone"}:
# Read-only previews: viewing templates is enough, even for the POST.
return [
IsAuthenticated(),
ViewOcrTemplatesPermissions(),
ViewDocumentsPermissions(),
]
return super().get_permissions()
@action(
detail=False,
methods=["get"],
url_path=r"document-page-image/(?P<doc_id>[0-9]+)/(?P<page>[0-9]+)",
)
def document_page_image(
self,
request: Request,
doc_id: str,
page: str,
) -> HttpResponse:
"""
Render a page for the template editor's zone overlay. The page in the URL is
1-based, like a zone's page.
"""
from documents.zone_ocr import resolve_zone_source
document: Document = get_object_or_404(
Document.objects.select_related("root_document__owner"),
pk=doc_id,
)
if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions")
file_doc = self._resolve_file_document(document, request)
page_number = int(page)
if page_number < 1:
raise Http404("Page numbers start at 1")
if file_doc.page_count and page_number > file_doc.page_count:
raise Http404(
f"Page {page_number} out of range (document has {file_doc.page_count} pages)",
)
source = resolve_zone_source(file_doc)
if source is None:
raise Http404("Document file not found")
doc_path = source.path
if source.is_image:
return HttpResponse(
doc_path.read_bytes(),
content_type=file_doc.mime_type,
)
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir:
rendered = Path(tmp_dir) / "page.png"
try:
rasterize_pdf_page_to_png(
doc_path,
rendered,
dpi=150,
page=page_number,
timeout=30,
)
except ParseError as error:
logger.warning(
"Failed to render page %s of document %s: %s",
page_number,
file_doc.pk,
error,
)
raise Http404("Failed to render page") from error
if not rendered.is_file():
raise Http404("No rendered page found")
content = rendered.read_bytes()
return HttpResponse(content, content_type="image/png")
@extend_schema(
request=OcrTemplateTestSerializer,
responses={
(200, "application/json"): OcrTemplateTestResultSerializer,
(404, "application/json"): OcrErrorSerializer,
},
)
@action(detail=False, methods=["post"], url_path="test-zone")
def test_zone(self, request: Request) -> HttpResponse:
"""Extract an ad-hoc zone without changing the document."""
from documents.zone_ocr import extract_zone_preview
from documents.zone_ocr import resolve_zone_source
serializer = OcrTemplateTestSerializer(
data=request.data,
context={"request": request},
)
serializer.is_valid(raise_exception=True)
document: Document = serializer.validated_data["document"]
file_doc = self._resolve_file_document(document, request)
source = resolve_zone_source(file_doc)
if source is None:
return Response(
{"error": "Document file not found"},
status=status.HTTP_404_NOT_FOUND,
)
zone = OcrTemplateZone(**serializer.validated_data["zone"])
result = extract_zone_preview(
source.path,
zone,
zone.zone_source_width or 0,
zone.zone_source_height or 0,
file_doc.page_count,
is_image=source.is_image,
)
regex_match = None
if zone.validation_regex and result.value is not None:
regex_match = (
safe_regex_fullmatch(zone.validation_regex, result.value) is not None
)
return Response(
{
"raw_text": result.raw_text,
"value": result.value,
"regex": zone.validation_regex,
"regex_match": regex_match,
},
)
File diff suppressed because it is too large. Load diff
+1
View File
@@ -227,6 +227,7 @@ SPECTACULAR_SETTINGS = {
}, },
"ENUM_NAME_OVERRIDES": { "ENUM_NAME_OVERRIDES": {
"MatchingAlgorithm": "documents.models.MatchingModel.MATCHING_ALGORITHMS", "MatchingAlgorithm": "documents.models.MatchingModel.MATCHING_ALGORITHMS",
"WorkflowTriggerMatchingAlgorithmEnum": "documents.models.WorkflowTrigger.WorkflowTriggerMatching",
"BarcodeFormatEnum": "documents.models.DocumentBarcode.Format", "BarcodeFormatEnum": "documents.models.DocumentBarcode.Format",
"NoteFormatEnum": "documents.models.Note.Format", "NoteFormatEnum": "documents.models.Note.Format",
}, },
+8
View File
@@ -30,11 +30,13 @@ from documents.views import IndexView
from documents.views import LogViewSet from documents.views import LogViewSet
from documents.views import MergeDocumentsAsVersionsView from documents.views import MergeDocumentsAsVersionsView
from documents.views import MergeDocumentsView from documents.views import MergeDocumentsView
from documents.views import OcrTemplateViewSet
from documents.views import PostDocumentView from documents.views import PostDocumentView
from documents.views import RemoteVersionView from documents.views import RemoteVersionView
from documents.views import RemovePasswordDocumentsView from documents.views import RemovePasswordDocumentsView
from documents.views import ReprocessDocumentsView from documents.views import ReprocessDocumentsView
from documents.views import RotateDocumentsView from documents.views import RotateDocumentsView
from documents.views import RunZoneOcrDocumentsView
from documents.views import SavedViewViewSet from documents.views import SavedViewViewSet
from documents.views import SearchAutoCompleteView from documents.views import SearchAutoCompleteView
from documents.views import SelectionDataView from documents.views import SelectionDataView
@@ -90,6 +92,7 @@ api_router.register(r"workflow_triggers", WorkflowTriggerViewSet)
api_router.register(r"workflow_actions", WorkflowActionViewSet) api_router.register(r"workflow_actions", WorkflowActionViewSet)
api_router.register(r"workflows", WorkflowViewSet) api_router.register(r"workflows", WorkflowViewSet)
api_router.register(r"custom_fields", CustomFieldViewSet) api_router.register(r"custom_fields", CustomFieldViewSet)
api_router.register(r"ocr_templates", OcrTemplateViewSet)
api_router.register(r"config", ApplicationConfigurationViewSet) api_router.register(r"config", ApplicationConfigurationViewSet)
api_router.register(r"processed_mail", ProcessedMailViewSet) api_router.register(r"processed_mail", ProcessedMailViewSet)
@@ -166,6 +169,11 @@ urlpatterns = [
ReprocessDocumentsView.as_view(), ReprocessDocumentsView.as_view(),
name="reprocess_documents", name="reprocess_documents",
), ),
re_path(
"^run_zone_ocr/",
RunZoneOcrDocumentsView.as_view(),
name="run_zone_ocr_documents",
),
re_path( re_path(
"^rotate/", "^rotate/",
RotateDocumentsView.as_view(), RotateDocumentsView.as_view(),
+35
View File
@@ -8,10 +8,13 @@ import factory
from django.contrib.auth.models import User from django.contrib.auth.models import User
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import CustomField
from documents.models import Document from documents.models import Document
from documents.models import DocumentBarcode from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import MatchingModel from documents.models import MatchingModel
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
@@ -46,6 +49,14 @@ class TagFactory(TypedModelFactory[Tag]):
is_inbox_tag = False is_inbox_tag = False
class CustomFieldFactory(TypedModelFactory[CustomField]):
class Meta:
model = CustomField
name = factory.Sequence(lambda n: f"Custom field {n}")
data_type = CustomField.FieldDataType.STRING
class StoragePathFactory(TypedModelFactory[StoragePath]): class StoragePathFactory(TypedModelFactory[StoragePath]):
class Meta: class Meta:
model = StoragePath model = StoragePath
@@ -108,3 +119,27 @@ class PaperlessTaskFactory(TypedModelFactory[PaperlessTask]):
input_data = factory.LazyFunction(dict) input_data = factory.LazyFunction(dict)
result_data = None result_data = None
acknowledged = False acknowledged = False
class OcrTemplateFactory(TypedModelFactory[OcrTemplate]):
class Meta:
model = OcrTemplate
name = factory.Sequence(lambda n: f"OCR template {n}")
match = ""
matching_algorithm = MatchingModel.MATCH_ANY
source_width = 1000
source_height = 1000
class OcrTemplateZoneFactory(TypedModelFactory[OcrTemplateZone]):
class Meta:
model = OcrTemplateZone
template = factory.SubFactory(OcrTemplateFactory)
name = factory.Sequence(lambda n: f"Zone {n}")
target = OcrTemplateZone.TargetType.TITLE
x = 0
y = 0
width = 100
height = 50
Generated
+19 -54
View File
@@ -3,14 +3,14 @@ revision = 5
requires-python = ">=3.11" requires-python = ">=3.11"
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'", "python_full_version >= '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'",
"python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version < '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'", "python_full_version < '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')", "(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')", "(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
] ]
supported-markers = [ supported-markers = [
@@ -2547,9 +2547,7 @@ dependencies = [
{ name = "scikit-learn" }, { name = "scikit-learn" },
{ name = "sentence-transformers" }, { name = "sentence-transformers" },
{ name = "setproctitle" }, { name = "setproctitle" },
{ name = "sqlite-vec", version = "0.1.10+paperless.3", source = { url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-macosx_11_0_arm64.whl" }, marker = "sys_platform != 'linux'" }, { name = "sqlite-vec" },
{ name = "sqlite-vec", version = "0.1.10+paperless.3", source = { url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_aarch64.whl" }, marker = "platform_machine != 'x86_64' and sys_platform == 'linux'" },
{ name = "sqlite-vec", version = "0.1.10+paperless.3", source = { url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_x86_64.whl" }, marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
{ name = "tantivy" }, { name = "tantivy" },
{ name = "tika-client", extra = ["httpx"] }, { name = "tika-client", extra = ["httpx"] },
{ name = "torch", version = "2.14.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform != 'linux'" }, { name = "torch", version = "2.14.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform != 'linux'" },
@@ -2708,10 +2706,7 @@ requires-dist = [
{ name = "scikit-learn", specifier = "~=1.9.0" }, { name = "scikit-learn", specifier = "~=1.9.0" },
{ name = "sentence-transformers", specifier = ">=5.6.1" }, { name = "sentence-transformers", specifier = ">=5.6.1" },
{ name = "setproctitle", specifier = "~=1.3.4" }, { name = "setproctitle", specifier = "~=1.3.4" },
{ name = "sqlite-vec", marker = "(platform_machine != 'aarch64' and platform_machine != 'arm64' and platform_machine != 'x86_64') or (platform_machine == 'aarch64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'arm64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux')", specifier = "==0.1.10+paperless.3" }, { name = "sqlite-vec", specifier = "==0.1.9" },
{ name = "sqlite-vec", marker = "platform_machine == 'arm64' and sys_platform == 'darwin'", url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-macosx_11_0_arm64.whl" },
{ name = "sqlite-vec", marker = "platform_machine == 'aarch64' and sys_platform == 'linux'", url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_aarch64.whl" },
{ name = "sqlite-vec", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'", url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_x86_64.whl" },
{ name = "tantivy", specifier = "~=0.26.0" }, { name = "tantivy", specifier = "~=0.26.0" },
{ name = "tika-client", extras = ["httpx"], specifier = "~=1.0" }, { name = "tika-client", extras = ["httpx"], specifier = "~=1.0" },
{ name = "torch", specifier = ">=2.13,<2.15", index = "https://download.pytorch.org/whl/cpu" }, { name = "torch", specifier = ">=2.13,<2.15", index = "https://download.pytorch.org/whl/cpu" },
@@ -3112,10 +3107,10 @@ version = "3.3.4"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'", "python_full_version >= '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'",
"python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version < '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'", "python_full_version < '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'",
"python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')", "(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')", "(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
] ]
sdist = { url = "https://files.pythonhosted.org/packages/21/7c/c08364f2eab2913e4068b3b955d963e7a3491986a85429990969525def30/psycopg_c-3.3.4.tar.gz", hash = "sha256:ed8106128b2d04359c185fc9641b4409abfce4d0b6fb1d1ff6800646e27f1a22", size = 647111, upload-time = "2026-05-01T23:31:58.032Z" } sdist = { url = "https://files.pythonhosted.org/packages/21/7c/c08364f2eab2913e4068b3b955d963e7a3491986a85429990969525def30/psycopg_c-3.3.4.tar.gz", hash = "sha256:ed8106128b2d04359c185fc9641b4409abfce4d0b6fb1d1ff6800646e27f1a22", size = 647111, upload-time = "2026-05-01T23:31:58.032Z" }
@@ -4098,42 +4093,12 @@ asyncio = [
[[package]] [[package]]
name = "sqlite-vec" name = "sqlite-vec"
version = "0.1.10+paperless.3" version = "0.1.9"
source = { url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-macosx_11_0_arm64.whl" } source = { registry = "https://pypi.org/simple" }
resolution-markers = [
"python_full_version >= '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'",
"python_full_version < '3.15' and platform_machine == 'arm64' and platform_python_implementation == 'CPython' and sys_platform == 'darwin'",
]
wheels = [ wheels = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-macosx_11_0_arm64.whl", hash = "sha256:8cd483975df4e833cfa10d3026bb6a7e71be2cbe3fcb7546952178849d45fa03" }, { url = "https://files.pythonhosted.org/packages/a4/3d/3677e0cd2f92e5ebc43cd29fbf565b75582bff1ccfa0b8327c7508e1084f/sqlite_vec-0.1.9-py3-none-macosx_11_0_arm64.whl", hash = "sha256:1d52e30513bae4cc9778ddbf6145610434081be4c3afe57cd877893bad9f6b6c", size = 165434, upload-time = "2026-03-31T08:02:32.712Z" },
] { url = "https://files.pythonhosted.org/packages/00/d4/f2b936d3bdc38eadcbd2a87875815db36430fab0363182ba5d12cd8e0b51/sqlite_vec-0.1.9-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e921e592f24a5f9a18f590b6ddd530eb637e2d474e3b1972f9bbeb773aa3cb9", size = 160076, upload-time = "2026-03-31T08:02:33.796Z" },
{ url = "https://files.pythonhosted.org/packages/6f/ad/6afd073b0f817b3e03f9e37ad626ae341805891f23c74b5292818f49ac63/sqlite_vec-0.1.9-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux1_x86_64.whl", hash = "sha256:1515727990b49e79bcaf75fdee2ffc7d461f8b66905013231251f1c8938e7786", size = 163388, upload-time = "2026-03-31T08:02:34.888Z" },
[[package]]
name = "sqlite-vec"
version = "0.1.10+paperless.3"
source = { url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_aarch64.whl" }
resolution-markers = [
"python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
]
wheels = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_aarch64.whl", hash = "sha256:71508033874aef98ef1a4006b4e1675be9bd9f866b3444491db480ce870a62c2" },
]
[[package]]
name = "sqlite-vec"
version = "0.1.10+paperless.3"
source = { url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_x86_64.whl" }
resolution-markers = [
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
]
wheels = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/sqlite-vec-0.1.10%2Bpaperless.3/sqlite_vec-0.1.10%2Bpaperless.3-py3-none-manylinux_2_29_x86_64.whl", hash = "sha256:9c8972c37295498460223ed592e6d8ab8e81d093604e645821fd36ec4ca30090" },
] ]
[[package]] [[package]]
@@ -4397,13 +4362,13 @@ name = "torch"
version = "2.14.0+cpu" version = "2.14.0+cpu"
source = { registry = "https://download.pytorch.org/whl/cpu" } source = { registry = "https://download.pytorch.org/whl/cpu" }
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'", "python_full_version == '3.14.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'", "python_full_version == '3.12.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'", "python_full_version >= '3.15' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')", "(python_full_version < '3.12' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
"python_full_version == '3.14.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"python_full_version >= '3.15' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'",
"(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')", "(python_full_version < '3.12' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux') or (python_full_version == '3.13.*' and platform_machine == 'x86_64' and platform_python_implementation == 'CPython' and sys_platform == 'linux')",
] ]
dependencies = [ dependencies = [