Compare commits

..
Author SHA1 Message Date
shamoon aede2e6a5a Merge branch 'dev' into feature/ocr-templates 2026-11-06 08:46:46 -08:00
shamoon fa792d33c2 Fix icon imports 2026-11-06 08:46:42 -08:00
shamoonandCodex 6fd91beae2 Chore: CI apt caching for flakiness (#14647)
---------

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-11-06 16:36:46 +00:00
Trenton H a4543c03dc Fix: release the LLM index read lock before streaming the chat answer (#14674)
Document chat held the shared index read lock for the whole request,
including the LLM stream, which can take minutes. A compaction or
migration swap waiting for exclusive access could time out behind it
and be skipped until the next run.

Now that the answer is synthesized from the nodes already retrieved,
nothing after the vector search touches the store. Hold the lock only
while loading the index and searching, and release it before synthesis.
2026-11-06 07:03:44 -08:00
Trenton H 45a2ac4879 Fix: retrieve once per document chat question (#14673)
Document chat searched the vector store twice per question: once
explicitly to build the reference list, and again inside
RetrieverQueryEngine.query(), which retrieves before synthesizing. That
meant two query-embedding calls and two full vector searches each time.

The engine added nothing beyond the retriever call (no node
postprocessors are configured), so drop it and synthesize directly from
the nodes already retrieved. The streaming response is unchanged.
2026-11-06 07:03:43 -08:00
GitHub Actions 8bc9db816a Auto translate strings 2026-11-06 03:07:21 +00:00
shamoon c5239f9f74 Fixhancement: rework saved view widget queue processing to expected (#14699)
* activateSavedViewWithDocuments, move most of loadSavedView to a helper

* Consolidate the page size thing

* Pass widget's documents to list service fpr next / previous

* Tests for the page size + widget click

* Fix close from a dashboard widget list
2026-11-05 19:06:10 -08:00
shamoon e88303cc16 Fix: apply PDF/A conversion once for mail archives (#14698) 2026-11-05 18:03:36 -08:00
shamoon cb3513c6f3 Fix default MFA trusted AMR setting (#14693) 2026-11-04 23:46:15 +00:00
dependabot[bot] 4468a510a6 Chore(deps): Bump the actions group across 1 directory with 17 updates (#14691)
Bumps the actions group with 17 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [actions/setup-python](https://github.com/actions/setup-python) | `7.0.0` | `7.1.0` |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `10.3.2` | `10.3.3` |
| [docker/login-action](https://github.com/docker/login-action) | `4.6.0` | `4.7.0` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.1` | `7.0.2` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `8.0.1` | `8.0.2` |
| [actions/setup-node](https://github.com/actions/setup-node) | `7.0.0` | `7.1.0` |
| [j178/prek-action](https://github.com/j178/prek-action) | `3.0.0` | `3.1.0` |
| [release-drafter/release-drafter](https://github.com/release-drafter/release-drafter) | `7.7.0` | `7.10.0` |
| [shogo82148/actions-upload-release-asset](https://github.com/shogo82148/actions-upload-release-asset) | `1.10.4` | `1.10.7` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.6.4` | `0.8.0` |
| [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) | `4.38.2` | `4.39.1` |
| [stumpylog/image-cleaner-action/ephemeral](https://github.com/stumpylog/image-cleaner-action) | `0.13.0` | `0.14.0` |
| [stumpylog/image-cleaner-action/untagged](https://github.com/stumpylog/image-cleaner-action) | `0.13.0` | `0.14.0` |
| [github/codeql-action/init](https://github.com/github/codeql-action) | `4.38.2` | `4.39.1` |
| [github/codeql-action/analyze](https://github.com/github/codeql-action) | `4.38.2` | `4.39.1` |
| [crowdin/github-action](https://github.com/crowdin/github-action) | `3.3.0` | `3.5.0` |
| [stefanzweifel/git-auto-commit-action](https://github.com/stefanzweifel/git-auto-commit-action) | `7.2.0` | `7.2.1` |



Updates `actions/setup-python` from 7.0.0 to 7.1.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/5fda3b95a4ea91299a34e894583c3862153e4b97...6d4fd22b20465ccc0f7354476588e8df7c9ca0e0)

Updates `astral-sh/setup-uv` from 10.3.2 to 10.3.3
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](https://github.com/astral-sh/setup-uv/compare/07a047e608db84b3d8c06cd9a7c7dcc9c355c18e...68559e99cc807c1eb7e9238fd43a15bba460eec2)

Updates `docker/login-action` from 4.6.0 to 4.7.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/dbcb813823bdd20940b903addbd779551569679f...81c6f93d4bc991e510b342a00b419b3bb4efc5c6)

Updates `actions/upload-artifact` from 7.0.1 to 7.0.2
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/043fb46d1a93c77aae656e7c1c64a875d1fc6a0a...cf430e030ddbb5b0abf93d22962f4752f3646cd9)

Updates `actions/download-artifact` from 8.0.1 to 8.0.2
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c...9000827ccba6bdab643e8b6fd33ac0654aef8333)

Updates `actions/setup-node` from 7.0.0 to 7.1.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/820762786026740c76f36085b0efc47a31fe5020...949feb2413d6458794dcd2491c4babbbce0c15c1)

Updates `j178/prek-action` from 3.0.0 to 3.1.0
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](https://github.com/j178/prek-action/compare/4e14d07f9231acabce116ccfca13b13dd9755ece...44924fe9ef4a6f88253c4b6b1bf11c689d3ce53d)

Updates `release-drafter/release-drafter` from 7.7.0 to 7.10.0
- [Release notes](https://github.com/release-drafter/release-drafter/releases)
- [Commits](https://github.com/release-drafter/release-drafter/compare/34d80673e067bdc0c24568d3af899c216adcfaa9...978bb0b30c5ee8cdd332ced253758157150d43f0)

Updates `shogo82148/actions-upload-release-asset` from 1.10.4 to 1.10.7
- [Release notes](https://github.com/shogo82148/actions-upload-release-asset/releases)
- [Commits](https://github.com/shogo82148/actions-upload-release-asset/compare/aaba0f56bdbc1071f4af234d5cb16055e8a400de...73f83b114970a3485e1423ecc1fb93d887c9b096)

Updates `zizmorcore/zizmor-action` from 0.6.4 to 0.8.0
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](https://github.com/zizmorcore/zizmor-action/compare/cc914d7f3750a2d13d75c7f184a1060aa0e9d482...53290badfa369cb0bf7ad16342cb62c2fa47ac7b)

Updates `github/codeql-action/upload-sarif` from 4.38.2 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

Updates `stumpylog/image-cleaner-action/ephemeral` from 0.13.0 to 0.14.0
- [Release notes](https://github.com/stumpylog/image-cleaner-action/releases)
- [Changelog](https://github.com/stumpylog/image-cleaner-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/stumpylog/image-cleaner-action/compare/21f875bab2376314e0525c614e923c669434f0e4...9796867b2a7994f0a3b6b1f2e38a6579f502c977)

Updates `stumpylog/image-cleaner-action/untagged` from 0.13.0 to 0.14.0
- [Release notes](https://github.com/stumpylog/image-cleaner-action/releases)
- [Changelog](https://github.com/stumpylog/image-cleaner-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/stumpylog/image-cleaner-action/compare/21f875bab2376314e0525c614e923c669434f0e4...9796867b2a7994f0a3b6b1f2e38a6579f502c977)

Updates `github/codeql-action/init` from 4.38.2 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

Updates `github/codeql-action/analyze` from 4.38.2 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

Updates `crowdin/github-action` from 3.3.0 to 3.5.0
- [Release notes](https://github.com/crowdin/github-action/releases)
- [Commits](https://github.com/crowdin/github-action/compare/9c23991700c0ec5256fd41089b9d9d7d540e424e...959dc74ac104447731cba99a016f85d0ae719788)

Updates `stefanzweifel/git-auto-commit-action` from 7.2.0 to 7.2.1
- [Release notes](https://github.com/stefanzweifel/git-auto-commit-action/releases)
- [Changelog](https://github.com/stefanzweifel/git-auto-commit-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/stefanzweifel/git-auto-commit-action/compare/4a55954c782fc1ea30b9056cd3e7a2b40ca8887d...d0b01107b5db882071f96de5dd4b54961ea93f4c)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: astral-sh/setup-uv
  dependency-version: 10.3.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/setup-node
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 3.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: release-drafter/release-drafter
  dependency-version: 7.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: shogo82148/actions-upload-release-asset
  dependency-version: 1.10.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: stumpylog/image-cleaner-action/ephemeral
  dependency-version: 0.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: stumpylog/image-cleaner-action/untagged
  dependency-version: 0.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: github/codeql-action/init
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: crowdin/github-action
  dependency-version: 3.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: stefanzweifel/git-auto-commit-action
  dependency-version: 7.2.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-04 13:40:41 -08:00
dependabot[bot] 09d70ed7da docker(deps): bump astral-sh/uv (#14635)
Bumps [astral-sh/uv](https://github.com/astral-sh/uv) from 0.13.5-python3.14-trixie-slim to 0.13.7-python3.14-trixie-slim.
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/uv/compare/0.13.5...0.13.7)

---
updated-dependencies:
- dependency-name: astral-sh/uv
  dependency-version: 0.13.6-python3.14-trixie-slim
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-04 12:30:59 -08:00
GitHub Actions 057aee8bbd Auto translate strings 2026-11-04 19:07:00 +00:00
shamoon fa83a4c5dc Fix: check descendant tag permissions on delete (#14687) 2026-11-04 11:05:48 -08:00
shamoon 22746aa202 Fix: lock rows for first-user signup (#14688) 2026-11-04 16:33:28 +00:00
shamoon 137342c1f9 Documentation: clarify metadata exposure reporting scope 2026-11-04 07:29:41 -08:00
GitHub Actions 0ae5e2d3d4 Auto translate strings 2026-11-04 14:47:43 +00:00
shamoon b7b0de100f Fix: prevent ID badge wrapping under title on mobile (#14682) 2026-11-04 06:46:22 -08:00
Trenton H 8531d3e0c9 chore(deps): bump django-allauth to 65.21.1 (#14672)
django-allauth 65.20.1 -> 65.21.1. Security fixes:

- MFA: authentication records from an inactive user's passwordless WebAuthn login could carry over to a later login in the same session, skipping MFA
- Rate limits with multiple windows for the same scope shared a cache entry, allowing longer windows to be bypassed
- Passwordless WebAuthn registration and login now always require user verification (MFA_WEBAUTHN_USER_VERIFICATION)
- Headless: reaching the phone verification attempt limit now aborts pending flows

Released 2026-10-30, inside the usual 7 day dependency cooldown, taken deliberately for the security fixes.
2026-11-04 01:25:10 +00:00
Trenton H 05f7545a4a chore(deps): bump dev tooling (#14671)
- prek 0.5.3 -> 0.5.7
- ruff 0.16.9 -> 0.16.10
- faker 40.39.0 -> 40.43.0
- pytest-env 1.7.1 -> 1.8.1
- pytest-mock 3.15.1 -> 3.17.0
- time-machine 3.5.1 -> 3.6.0
- zensical 0.0.65 -> 0.0.73
- python-dotenv 1.2.3 -> 1.2.4 (pulled in by the above)
2026-11-04 00:53:22 +00:00
shamoon ade5901a4d Update for new icons 2026-11-02 21:46:45 -08:00
shamoon 2b620a2823 Merge branch 'dev' into feature/ocr-templates 2026-11-02 21:44:11 -08:00
shamoon 45b8977623 Merge branch 'dev' into feature/ocr-templates 2026-10-29 15:46:31 -07:00
shamoon 800748d5ab Merge branch 'dev' into feature/ocr-templates 2026-10-25 10:29:32 -07:00
shamoon 182b0b4712 Merge branch 'dev' into feature/ocr-templates 2026-10-22 15:39:03 -07:00
shamoon c718636f79 Update usage.md 2026-10-20 17:07:09 -07:00
shamoonandOpenAI Codex 54f7f43f35 Reuse OCR page renders and simplify value conversion
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 17:07:00 -07:00
shamoonandOpenAI Codex 94da3d4c57 Report actual OCR extraction outcomes instead of existing field values
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 17:02:23 -07:00
shamoonandOpenAI Codex 453e89d6dd Prepare and atomically apply OCR template field updates
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 17:00:03 -07:00
shamoonandOpenAI Codex 6562415894 Unify OCR zone extraction for preview and saved templates
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-20 16:58:10 -07:00
shamoon 257c33453d Update test_api_ocr_templates.py 2026-10-20 16:49:39 -07:00
shamoon 87913411ef Update serialisers.py 2026-10-20 16:49:35 -07:00
shamoon cec393dcae Update views.py 2026-10-20 16:49:32 -07:00
shamoon f5526d5700 Improve backend tests 2026-10-20 16:47:48 -07:00
shamoon 2534d3c16b Absurd comments 2026-10-20 16:46:21 -07:00
shamoon 95c8bf2058 Cleanup the serializers and views 2026-10-20 16:45:04 -07:00
shamoon 930d23956b Up frontend coverage 2026-10-20 16:34:22 -07:00
shamoon d2e6834602 Make sort order stable 2026-10-20 16:28:31 -07:00
shamoon a7bf04ddee Update spacing 2026-10-20 16:17:56 -07:00
shamoon 8ab2bf63aa Merge branch 'dev' into feature/ocr-templates 2026-10-20 16:16:31 -07:00
shamoon c63a7e25e7 Update docs 2026-10-20 15:37:08 -07:00
shamoon 9171a1aa6a Changeup the icon 2026-10-20 15:33:01 -07:00
shamoon 55c5d9121f Yes, cute little split button that opens the dialog 2026-10-20 15:31:48 -07:00
shamoon e4efb62667 And bulk edit 2026-10-20 14:02:04 -07:00
shamoon f0597a7c54 doc detail use it this way 2026-10-20 14:00:25 -07:00
shamoon bbdc016a32 Backend for manual tempalte choice 2026-10-20 13:51:54 -07:00
shamoon eae392469f Update the frontend for the matching stuff 2026-10-20 13:41:55 -07:00
shamoon bedefd85f2 Remove the handlers, make the matching work 2026-10-20 13:39:09 -07:00
shamoon bd73fe1e80 Update the serializer 2026-10-20 13:31:08 -07:00
shamoon 8b77a924dc Make it a matchingmodel 2026-10-20 13:29:42 -07:00
shamoon 1bdeb7ebcd Mas sonar 2026-10-20 13:15:17 -07:00
shamoon fa3acfe13f Sure sonar 2026-10-20 13:12:26 -07:00
shamoon 2bc22b14bd Make perms accordion 2026-10-20 13:03:04 -07:00
shamoon dbce8c1409 User perms on frontend 2026-10-20 11:25:48 -07:00
shamoon e976b2ec7a Tests 2026-10-20 11:23:44 -07:00
shamoon 05da0ba5a3 Apply only visible templates 2026-10-20 11:15:31 -07:00
shamoon 4ff2cb9c72 Make them owned, duh
Update 0028_ocrtemplate_ocrtemplatezone.py
2026-10-20 11:15:30 -07:00
shamoon 77a7fb40ec safe regex 2026-10-20 11:05:49 -07:00
shamoon 8100a8e1eb Lol and run zone ocr 2026-10-20 11:04:17 -07:00
shamoon 1e334903f3 And test zone 2026-10-20 11:01:11 -07:00
shamoon a958d89bf4 Add perms to document_page_image 2026-10-20 10:58:54 -07:00
shamoon 8b3295fce0 Docs 2026-10-20 10:42:16 -07:00
shamoon 9783c4995f Fix the tests for now 2026-10-20 10:42:12 -07:00
shamoon e89495dc02 Frontend use the bulk edit 2026-10-20 10:32:48 -07:00
shamoon 8bb2f88015 Backend work for proper bulk zone ocr 2026-10-20 10:20:28 -07:00
shamoon 7d9b8c770d Cleanup doc detail run zone ocr 2026-10-20 10:16:58 -07:00
shamoon 3296fd7cb5 Formatting, remove hard-coded alpha stuff 2026-10-20 10:14:03 -07:00
shamoon bff41907c2 Move back controls, drop (lol) dnd 2026-10-20 10:08:43 -07:00
shamoon 592f882394 Get rid of the x for the picker 2026-10-20 10:06:40 -07:00
shamoon 29ea9cec66 Gosh so much cleanup 2026-10-20 10:03:55 -07:00
shamoon 123df2e826 Pngx conventions 2026-10-20 09:50:29 -07:00
shamoon 639827ec65 Lol, no not deu+eng default 2026-10-20 09:45:30 -07:00
shamoon b3106e408c Lots of enum cleanup 2026-10-20 09:44:37 -07:00
shamoon ea31b6a043 Make it close not leave page 2026-10-20 09:38:29 -07:00
shamoon 0572b54579 Proper zoneless stuff 2026-10-20 09:35:39 -07:00
shamoon 2afec3e0b7 Lots of dead code 2026-10-20 09:31:59 -07:00
shamoon 57019bccbe Cleanup save now that we're modal 2026-10-20 09:30:12 -07:00
shamoon 26b3964a20 Cleanup the geometry stuff 2026-10-20 09:26:42 -07:00
shamoon a2eeadca7f Fix the delete handler 2026-10-20 09:24:26 -07:00
shamoon ba5474ba81 Lol, just one icon 2026-10-20 09:20:06 -07:00
shamoon e3592804bd Embiggen 2026-10-20 09:18:36 -07:00
shamoon 2d54927be8 Fuck it, make it a modal like literally everything else, drop the routes 2026-10-20 09:14:52 -07:00
shamoon 1d2832ac30 Update ocr-template-editor.component.ts 2026-10-20 09:05:24 -07:00
shamoon 3321ba46d6 Automatically update the combine string when changing zone name 2026-10-20 09:00:17 -07:00
shamoon bf316cba8a Dont fail if one bulk fails, collect results separately 2026-10-20 08:57:24 -07:00
shamoon 36c31b3a73 Also prevent editing template while saving 2026-10-20 08:47:26 -07:00
shamoon d10d303980 Warn about losing changes, disable while zone ocr running 2026-10-20 08:36:22 -07:00
shamoon a51e6a71f4 Fix reload after running zone ocr 2026-10-20 08:20:44 -07:00
shamoon ebd8dba3bb Fix the no zones note 2026-10-20 08:20:25 -07:00
shamoon 3ed9586347 Remove the create link from doc details 2026-10-20 08:11:40 -07:00
shamoon 7055f7a745 Fix perms test 2026-10-20 08:07:46 -07:00
shamoon 8f7500ba64 Fix the signal thing 2026-10-20 08:03:33 -07:00
shamoon 816774815c Add a little warning for empty zones 2026-10-20 07:39:47 -07:00
shamoon c5e6748771 Use braces icon instead 2026-10-20 07:18:01 -07:00
shamoon 9b42170554 Make the floating controls movable 2026-10-20 07:15:18 -07:00
shamoon d2ed0c55a4 dirtycheck 2026-10-19 23:12:09 -07:00
shamoon 4d31063082 Default to empty field 2026-10-19 23:07:12 -07:00
shamoon a4fb9bf2b0 Drop the silly mode toggle 2026-10-19 23:04:35 -07:00
shamoon e028257da5 Cleanup the zone combine stuff 2026-10-19 22:55:23 -07:00
shamoon 68eb4fe96d Make the zone list more useful, show colors 2026-10-19 22:43:12 -07:00
shamoon d2d434535b Remove the slop quick create CF stuff 2026-10-19 22:36:17 -07:00
shamoon 1d75c0c99b Use pngx inputs, move test button, generally make less terrible 2026-10-19 22:31:33 -07:00
shamoon 1de1261728 Ok thats a better start, two tabs 2026-10-19 22:23:50 -07:00
shamoon f5f33e0744 make the ui usable 2026-10-19 22:06:11 -07:00
shamoon c9d62afa50 Make it hideable too 2026-10-19 21:43:10 -07:00
shamoon b5a9353780 Move migration 2026-10-19 21:39:26 -07:00
shamoon cf6d16edcd Merge branch 'dev' into feature/ocr-templates 2026-10-19 21:37:23 -07:00
shamoon 551445773a Use SVG, get rid of all the manual redraw stuff 2026-09-09 12:43:11 -07:00
shamoon c54cb4a84b Use a real canvas interaction model
[skip ci]
2026-09-09 12:43:11 -07:00
shamoon af61044b63 Fix some permissions stuff, show error on toggle 2026-09-09 12:43:11 -07:00
shamoon 1ef5a6d3cf Get rid of absurd alert 2026-09-09 12:43:11 -07:00
shamoon 4da311f91f Proper data types
[skip ci]
2026-09-09 12:43:10 -07:00
shamoon cbb84649c7 Pull out the zone list into a sub-component 2026-09-09 12:43:10 -07:00
shamoon 8da734a495 Extract the geometry stuff a bit 2026-09-09 12:43:10 -07:00
shamoon 351671011d Remove afterviewinit, fix page change, custom date format per zone 2026-09-09 12:43:09 -07:00
15c95c251d Feature: OCR Templates (#13043)
[skip ci]

Signed-off-by: dependabot[bot] <support@github.com>
Co-Authored-By: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-Authored-By: stumpylog <797416+stumpylog@users.noreply.github.com>
Co-Authored-By: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-Authored-By: shamoon <4887959+shamoon@users.noreply.github.com>
2026-09-09 12:43:09 -07:00
111 changed files with 10046 additions and 2621 deletions

No files matched your search

+34 -34
View File
@@ -1,53 +1,53 @@
name: Install apt packages name: Install apt packages
description: > description: >
Installs apt packages with a per-attempt timeout and retries. The hosted runner's Azure apt mirror intermittently serves archives at a crawl, which apt's own timeout does not catch. A stuck attempt is killed and retried, which usually lands on a healthy mirror node. Installs apt packages with bounded download retries and Ubuntu mirror fallback. Caches downloaded packages as an optimization; apt resolves dependencies on every run.
inputs: inputs:
packages: packages:
description: Space separated list of packages to install description: Space separated list of package names to install
required: true required: true
attempts: attempts:
description: Maximum number of attempts description: Maximum number of download attempts
default: "3" default: "3"
attempt-timeout: attempt-timeout:
description: Seconds before a single attempt is killed description: Seconds before a single download attempt is killed
default: "180" default: "180"
runs: runs:
using: composite using: composite
steps: steps:
- name: Identify apt cache
id: cache-key
shell: bash
env:
PACKAGES: ${{ inputs.packages }}
run: |
. /etc/os-release
packages_hash=$(printf '%s' "$PACKAGES" | sha256sum | cut -d ' ' -f 1)
# Cached downloads are optional: apt fetches any missing/newer packages.
# Refresh weekly, including inside job containers without ImageVersion.
echo "key=apt-v2-${ID}-${VERSION_ID}-$(dpkg --print-architecture)-$(date -u +%G-%V)-${packages_hash}" >> "$GITHUB_OUTPUT"
- name: Restore apt packages
id: cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
${{ runner.temp }}/apt-packages/*.deb
${{ runner.temp }}/apt-packages/.keep
key: ${{ steps.cache-key.outputs.key }}
- name: Install ${{ inputs.packages }} - name: Install ${{ inputs.packages }}
shell: bash shell: bash
env: env:
PACKAGES: ${{ inputs.packages }} PACKAGES: ${{ inputs.packages }}
ATTEMPTS: ${{ inputs.attempts }} ATTEMPTS: ${{ inputs.attempts }}
ATTEMPT_TIMEOUT: ${{ inputs.attempt-timeout }} ATTEMPT_TIMEOUT: ${{ inputs.attempt-timeout }}
run: | CACHE_DIR: ${{ runner.temp }}/apt-packages
SUDO="" run: bash "$GITHUB_ACTION_PATH/install.sh"
if [ "$(id -u)" -ne 0 ]; then # Save immediately: a later test failure must not discard downloaded packages.
SUDO="sudo" - name: Save apt packages
fi if: steps.cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
attempt_install() { with:
$SUDO apt-get update -q path: |
# Intentionally unquoted, this is a list of package names ${{ runner.temp }}/apt-packages/*.deb
$SUDO apt-get install -q --yes --no-install-recommends $PACKAGES ${{ runner.temp }}/apt-packages/.keep
} key: ${{ steps.cache-key.outputs.key }}
export -f attempt_install
export SUDO PACKAGES
for attempt in $(seq 1 "${ATTEMPTS}"); do
echo "::group::apt install attempt ${attempt}/${ATTEMPTS}"
start=$(date +%s)
if timeout --kill-after=10 "${ATTEMPT_TIMEOUT}" bash -c attempt_install; then
echo "::endgroup::"
echo "Installed in $(( $(date +%s) - start ))s"
exit 0
fi
echo "::endgroup::"
echo "::warning::apt install attempt ${attempt}/${ATTEMPTS} failed or exceeded ${ATTEMPT_TIMEOUT}s"
# A killed attempt can leave dpkg half configured
$SUDO dpkg --configure -a || true
done
echo "::error::apt install failed after ${ATTEMPTS} attempts"
exit 1
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
set -euo pipefail
# Run as root in the timed child process. Only downloads may be interrupted.
if [ "${1:-}" = download ]; then
download_archive_dir=$2
shift 2
apt-get update -q
apt-get install -q --yes --no-install-recommends --download-only \
-o "Dir::Cache::archives=$download_archive_dir" "$@"
exit 0
fi
SUDO=()
if [ "$(id -u)" -ne 0 ]; then
SUDO=(sudo)
fi
read -r -a requested_packages <<< "$PACKAGES"
mkdir -p "$CACHE_DIR"
touch "$CACHE_DIR/.keep"
use_ubuntu_mirrors() {
# Hosted runners use a mirror list; job containers can point directly at Azure.
# Handle both layouts, leaving other repositories and signature checks intact.
if [ -f /etc/apt/apt-mirrors.txt ]; then
"${SUDO[@]}" sed -i '\|azure\.archive\.ubuntu\.com|d' /etc/apt/apt-mirrors.txt
fi
for source in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
if [ -f "$source" ]; then
"${SUDO[@]}" sed -i 's|azure\.archive\.ubuntu\.com|archive.ubuntu.com|g' "$source"
fi
done
}
for attempt in $(seq 1 "$ATTEMPTS"); do
echo "::group::apt download attempt ${attempt}/${ATTEMPTS}"
if timeout --kill-after=10 "$ATTEMPT_TIMEOUT" "${SUDO[@]}" bash "$0" download "$CACHE_DIR" "${requested_packages[@]}"; then
echo "::endgroup::"
# Apt handles aliases (e.g. libmagic1 -> libmagic1t64) and dependencies.
"${SUDO[@]}" apt-get install -q --yes --no-install-recommends --no-download \
-o "Dir::Cache::archives=$CACHE_DIR" "${requested_packages[@]}"
# Allow the unprivileged cache action to read archives downloaded as root.
find "$CACHE_DIR" -maxdepth 1 -name '*.deb' -exec "${SUDO[@]}" chmod a+r {} +
exit 0
fi
echo "::endgroup::"
echo "::warning::apt download attempt ${attempt}/${ATTEMPTS} failed or exceeded ${ATTEMPT_TIMEOUT}s"
if [ "$attempt" -eq 1 ] && [ "$ATTEMPTS" -gt 1 ]; then
echo "Switching from Azure to Ubuntu archive mirrors for remaining attempts"
use_ubuntu_mirrors
fi
done
echo "::error::apt download failed after ${ATTEMPTS} attempts"
exit 1
+5 -4
View File
@@ -77,6 +77,7 @@ jobs:
- 'uv.lock' - 'uv.lock'
- 'docker/compose/docker-compose.ci-test.yml' - 'docker/compose/docker-compose.ci-test.yml'
- '.github/workflows/ci-backend.yml' - '.github/workflows/ci-backend.yml'
- '.github/actions/apt-install/**'
test: test:
needs: changes needs: changes
if: needs.changes.outputs.backend_changed == 'true' if: needs.changes.outputs.backend_changed == 'true'
@@ -99,11 +100,11 @@ jobs:
docker compose --file docker/compose/docker-compose.ci-test.yml up --detach docker compose --file docker/compose/docker-compose.ci-test.yml up --detach
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: "${{ matrix.python-version }}" python-version: "${{ matrix.python-version }}"
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: ${{ env.DEFAULT_UV_VERSION }} version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true enable-cache: true
@@ -169,11 +170,11 @@ jobs:
persist-credentials: false persist-credentials: false
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: "${{ env.DEFAULT_PYTHON }}" python-version: "${{ env.DEFAULT_PYTHON }}"
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: ${{ env.DEFAULT_UV_VERSION }} version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true enable-cache: true
+6 -6
View File
@@ -109,7 +109,7 @@ jobs:
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GitHub Container Registry - name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}
@@ -155,7 +155,7 @@ jobs:
echo "${digest}" > "/tmp/digests/digest-${{ matrix.arch }}.txt" echo "${digest}" > "/tmp/digests/digest-${{ matrix.arch }}.txt"
- name: Upload digest - name: Upload digest
if: steps.check-push.outputs.should-push == 'true' if: steps.check-push.outputs.should-push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with: with:
name: digests-${{ matrix.arch }} name: digests-${{ matrix.arch }}
path: /tmp/digests/digest-${{ matrix.arch }}.txt path: /tmp/digests/digest-${{ matrix.arch }}.txt
@@ -173,7 +173,7 @@ jobs:
packages: write packages: write
steps: steps:
- name: Download digests - name: Download digests
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with: with:
path: /tmp/digests path: /tmp/digests
pattern: digest-*.txt pattern: digest-*.txt
@@ -185,20 +185,20 @@ jobs:
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GitHub Container Registry - name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to Docker Hub - name: Login to Docker Hub
if: needs.build-arch.outputs.push-external == 'true' if: needs.build-arch.outputs.push-external == 'true'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with: with:
username: ${{ secrets.DOCKERHUB_USERNAME }} username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }} password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to Quay.io - name: Login to Quay.io
if: needs.build-arch.outputs.push-external == 'true' if: needs.build-arch.outputs.push-external == 'true'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with: with:
registry: quay.io registry: quay.io
username: ${{ secrets.QUAY_USERNAME }} username: ${{ secrets.QUAY_USERNAME }}
+2 -2
View File
@@ -75,11 +75,11 @@ jobs:
persist-credentials: false persist-credentials: false
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }} python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: ${{ env.DEFAULT_UV_VERSION }} version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true enable-cache: true
+8 -7
View File
@@ -69,6 +69,7 @@ jobs:
frontend: frontend:
- 'src-ui/**' - 'src-ui/**'
- '.github/workflows/ci-frontend.yml' - '.github/workflows/ci-frontend.yml'
- '.github/actions/apt-install/**'
install-dependencies: install-dependencies:
needs: changes needs: changes
if: needs.changes.outputs.frontend_changed == 'true' if: needs.changes.outputs.frontend_changed == 'true'
@@ -86,7 +87,7 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
cache: 'pnpm' cache: 'pnpm'
@@ -118,7 +119,7 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
cache: 'pnpm' cache: 'pnpm'
@@ -157,7 +158,7 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
cache: 'pnpm' cache: 'pnpm'
@@ -206,18 +207,18 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
cache: 'pnpm' cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml' cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: '3.12' python-version: '3.12'
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: '0.13.x' version: '0.13.x'
enable-cache: false enable-cache: false
@@ -261,7 +262,7 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
cache: 'pnpm' cache: 'pnpm'
+2 -2
View File
@@ -22,8 +22,8 @@ jobs:
with: with:
persist-credentials: false persist-credentials: false
- name: Install Python - name: Install Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: "3.14" python-version: "3.14"
- name: Run prek - name: Run prek
uses: j178/prek-action@4e14d07f9231acabce116ccfca13b13dd9755ece # v3.0.0 uses: j178/prek-action@44924fe9ef4a6f88253c4b6b1bf11c689d3ce53d # v3.1.0
+9 -9
View File
@@ -44,7 +44,7 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
package-manager-cache: false package-manager-cache: false
@@ -55,11 +55,11 @@ jobs:
# ---- Backend Setup ---- # ---- Backend Setup ----
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }} python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: ${{ env.DEFAULT_UV_VERSION }} version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: false enable-cache: false
@@ -137,7 +137,7 @@ jobs:
sudo chown -R 1000:1000 paperless-ngx/ sudo chown -R 1000:1000 paperless-ngx/
tar -cJf paperless-ngx.tar.xz paperless-ngx/ tar -cJf paperless-ngx.tar.xz paperless-ngx/
- name: Upload release artifact - name: Upload release artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with: with:
name: release name: release
path: dist/paperless-ngx.tar.xz path: dist/paperless-ngx.tar.xz
@@ -155,7 +155,7 @@ jobs:
version: ${{ steps.get-version.outputs.version }} version: ${{ steps.get-version.outputs.version }}
steps: steps:
- name: Download release artifact - name: Download release artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with: with:
name: release name: release
path: ./ path: ./
@@ -172,7 +172,7 @@ jobs:
fi fi
- name: Create release and changelog - name: Create release and changelog
id: create-release id: create-release
uses: release-drafter/release-drafter@34d80673e067bdc0c24568d3af899c216adcfaa9 # v7.7.0 uses: release-drafter/release-drafter@978bb0b30c5ee8cdd332ced253758157150d43f0 # v7.10.0
with: with:
name: Paperless-ngx ${{ steps.get-version.outputs.version }} name: Paperless-ngx ${{ steps.get-version.outputs.version }}
tag: ${{ steps.get-version.outputs.version }} tag: ${{ steps.get-version.outputs.version }}
@@ -183,7 +183,7 @@ jobs:
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release archive - name: Upload release archive
uses: shogo82148/actions-upload-release-asset@aaba0f56bdbc1071f4af234d5cb16055e8a400de # v1.10.4 uses: shogo82148/actions-upload-release-asset@73f83b114970a3485e1423ecc1fb93d887c9b096 # v1.10.7
with: with:
github_token: ${{ secrets.GITHUB_TOKEN }} github_token: ${{ secrets.GITHUB_TOKEN }}
upload_url: ${{ steps.create-release.outputs.upload_url }} upload_url: ${{ steps.create-release.outputs.upload_url }}
@@ -209,11 +209,11 @@ jobs:
persist-credentials: true # for pushing changelog branch persist-credentials: true # for pushing changelog branch
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with: with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }} python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: ${{ env.DEFAULT_UV_VERSION }} version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: false enable-cache: false
+2 -2
View File
@@ -27,7 +27,7 @@ jobs:
with: with:
persist-credentials: false persist-credentials: false
- name: Run zizmor - name: Run zizmor
uses: zizmorcore/zizmor-action@cc914d7f3750a2d13d75c7f184a1060aa0e9d482 # v0.6.4 uses: zizmorcore/zizmor-action@53290badfa369cb0bf7ad16342cb62c2fa47ac7b # v0.8.0
semgrep: semgrep:
name: Semgrep CE name: Semgrep CE
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
@@ -45,7 +45,7 @@ jobs:
- name: Run Semgrep - name: Run Semgrep
run: semgrep scan --config auto --sarif-output results.sarif run: semgrep scan --config auto --sarif-output results.sarif
- name: Upload results to GitHub code scanning - name: Upload results to GitHub code scanning
uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4.38.2 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
if: always() if: always()
with: with:
sarif_file: results.sarif sarif_file: results.sarif
+2 -2
View File
@@ -29,7 +29,7 @@ jobs:
steps: steps:
- name: Clean temporary images - name: Clean temporary images
if: "${{ env.TOKEN != '' }}" if: "${{ env.TOKEN != '' }}"
uses: stumpylog/image-cleaner-action/ephemeral@21f875bab2376314e0525c614e923c669434f0e4 # v0.13.0 uses: stumpylog/image-cleaner-action/ephemeral@9796867b2a7994f0a3b6b1f2e38a6579f502c977 # v0.14.0
with: with:
token: "${{ env.TOKEN }}" token: "${{ env.TOKEN }}"
owner: "${{ github.repository_owner }}" owner: "${{ github.repository_owner }}"
@@ -56,7 +56,7 @@ jobs:
steps: steps:
- name: Clean untagged images - name: Clean untagged images
if: "${{ env.TOKEN != '' }}" if: "${{ env.TOKEN != '' }}"
uses: stumpylog/image-cleaner-action/untagged@21f875bab2376314e0525c614e923c669434f0e4 # v0.13.0 uses: stumpylog/image-cleaner-action/untagged@9796867b2a7994f0a3b6b1f2e38a6579f502c977 # v0.14.0
with: with:
token: "${{ env.TOKEN }}" token: "${{ env.TOKEN }}"
owner: "${{ github.repository_owner }}" owner: "${{ github.repository_owner }}"
+2 -2
View File
@@ -39,7 +39,7 @@ jobs:
persist-credentials: false persist-credentials: false
# Initializes the CodeQL tools for scanning. # Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4.38.2 uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
languages: ${{ matrix.language }} languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file. # If you wish to specify custom queries, you can do so here or in a config file.
@@ -47,4 +47,4 @@ jobs:
# Prefix the list here with "+" to use these queries and those in the config file. # Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main # queries: ./path/to/local/query, your-org/your-repo/queries@main
- name: Perform CodeQL Analysis - name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4.38.2 uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
token: ${{ secrets.PNGX_BOT_PAT }} token: ${{ secrets.PNGX_BOT_PAT }}
persist-credentials: false persist-credentials: false
- name: crowdin action - name: crowdin action
uses: crowdin/github-action@9c23991700c0ec5256fd41089b9d9d7d540e424e # v3.3.0 uses: crowdin/github-action@959dc74ac104447731cba99a016f85d0ae719788 # v3.5.0
with: with:
upload_translations: false upload_translations: false
download_translations: true download_translations: true
+1 -1
View File
@@ -19,6 +19,6 @@ jobs:
if: github.event_name == 'pull_request_target' && (github.event.action == 'opened' || github.event.action == 'reopened') && github.event.pull_request.user.login != 'dependabot' if: github.event_name == 'pull_request_target' && (github.event.action == 'opened' || github.event.action == 'reopened') && github.event.pull_request.user.login != 'dependabot'
steps: steps:
- name: Label PR with release-drafter - name: Label PR with release-drafter
uses: release-drafter/release-drafter@34d80673e067bdc0c24568d3af899c216adcfaa9 # v7.7.0 uses: release-drafter/release-drafter@978bb0b30c5ee8cdd332ced253758157150d43f0 # v7.10.0
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+4 -4
View File
@@ -23,14 +23,14 @@ jobs:
persist-credentials: true # for pushing translation branch persist-credentials: true # for pushing translation branch
- name: Set up Python - name: Set up Python
id: setup-python id: setup-python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
- name: Install system dependencies - name: Install system dependencies
timeout-minutes: 12 timeout-minutes: 12
uses: $/.github/actions/apt-install uses: $/.github/actions/apt-install
with: with:
packages: gettext packages: gettext
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2 uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with: with:
version: ${{ env.DEFAULT_UV_VERSION }} version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true enable-cache: true
@@ -48,7 +48,7 @@ jobs:
with: with:
package_json_file: src-ui/package.json package_json_file: src-ui/package.json
- name: Use Node.js 24 - name: Use Node.js 24
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with: with:
node-version: 24.x node-version: 24.x
cache: 'pnpm' cache: 'pnpm'
@@ -68,7 +68,7 @@ jobs:
cd src-ui cd src-ui
pnpm run ng extract-i18n pnpm run ng extract-i18n
- name: Commit changes - name: Commit changes
uses: stefanzweifel/git-auto-commit-action@4a55954c782fc1ea30b9056cd3e7a2b40ca8887d # v7.2.0 uses: stefanzweifel/git-auto-commit-action@d0b01107b5db882071f96de5dd4b54961ea93f4c # v7.2.1
with: with:
file_pattern: 'src-ui/messages.xlf src/locale/en_US/LC_MESSAGES/django.po' file_pattern: 'src-ui/messages.xlf src/locale/en_US/LC_MESSAGES/django.po'
commit_message: "Auto translate strings" commit_message: "Auto translate strings"
+1 -1
View File
@@ -30,7 +30,7 @@ RUN set -eux \
# Purpose: Installs s6-overlay and rootfs # Purpose: Installs s6-overlay and rootfs
# Comments: # Comments:
# - Don't leave anything extra in here either # - Don't leave anything extra in here either
FROM ghcr.io/astral-sh/uv:0.13.5-python3.14-trixie-slim@sha256:d8e732839dfc4225a1ff698803b478b2223c05636c055d3add41ab9dbec0f315 AS s6-overlay-base FROM ghcr.io/astral-sh/uv:0.13.7-python3.14-trixie-slim@sha256:e80c05a310400cfc90af00b5b7271ef9ad24816e3265e1162af8a678e1aad099 AS s6-overlay-base
WORKDIR /usr/src/s6 WORKDIR /usr/src/s6
+1 -1
View File
@@ -66,7 +66,7 @@ The following are not generally considered vulnerabilities unless accompanied by
- optional webhook, mail, AI, OCR, or integration behavior described without a product-level vulnerability - optional webhook, mail, AI, OCR, or integration behavior described without a product-level vulnerability
- missing limits or hardening settings presented without concrete impact - missing limits or hardening settings presented without concrete impact
- generic AI or static-analysis output that is not confirmed against the current codebase and a real deployment scenario - generic AI or static-analysis output that is not confirmed against the current codebase and a real deployment scenario
- metadata names visible in a document's custom storage path, even when the user cannot access the underlying metadata object; this is expected behavior - metadata names or normalized words visible in a document's custom storage path or through search/autocomplete for documents the user can access, even when the user cannot access the underlying metadata object; this is expected behavior
- the ability to attach objects that a user cannot access to a document by ID is an intentional design choice, and not considered a vulnerability - the ability to attach objects that a user cannot access to a document by ID is an intentional design choice, and not considered a vulnerability
## Transparency ## Transparency
+79 -1
View File
@@ -109,7 +109,7 @@ Think of versions as **file history** for a document.
Paperless-ngx includes management lists for tags, correspondents, document types Paperless-ngx includes management lists for tags, correspondents, document types
and more. These areas allow you to view, add, edit, delete and manage permissions and more. These areas allow you to view, add, edit, delete and manage permissions
for these objects. You can also manage saved views, mail accounts, mail rules, for these objects. You can also manage saved views, mail accounts, mail rules,
workflows and more from the management sections. workflows, [OCR templates](#ocr-templates) and more from the management sections.
### Nested Tags ### Nested Tags
@@ -795,6 +795,84 @@ The following custom field types are supported:
- `Document Link`: reference(s) to other document(s) displayed as links, automatically creates a symmetrical link in reverse - `Document Link`: reference(s) to other document(s) displayed as links, automatically creates a symmetrical link in reverse
- `Select`: a pre-defined list of strings from which the user can choose - `Select`: a pre-defined list of strings from which the user can choose
## OCR Templates {#ocr-templates}
OCR templates read text from fixed areas ("zones") of a document and write it to a
[custom field](#custom-fields), or to the document's title, archive serial number or created date. They are
useful for documents that always share the same layout, such as bills from a particular utility. They allow you
to draw a zone around the account number once, and every new bill gets its account number filled in.
Like tags or document types, each template has a [matching algorithm](advanced_usage.md#matching) that
decides which documents it automatically applies to (or not), based on their content.
### Creating a template
1. Open **OCR Templates** under "Manage" and choose **Add Template**.
2. Give the template a name and set up its matching, for example **Any word** with the utility's name. Choose
**None** for a template that should only ever be run manually.
3. Search for a sample document with the layout the template is for. Its pages are shown next to the template
settings.
4. Draw zones on the document. Click a zone to select and edit it.
5. Choose the field each zone fills.
6. Use **Test** to run OCR on the selected zone of the sample document and see the text it finds and the value
that would be written.
Zones are positioned relative to the sample document's page, so templates work best when the documents they
match have an identical layout. Keep the matching specific enough that it only catches documents with that
layout.
### Zone settings
- **Page**: the page the zone is on. Turn on **Last page** for zones that are always on the final page,
regardless of how many pages a document has.
- **OCR language**: the Tesseract languages used to read the zone. Languages other than the defaults must be
installed, see [`PAPERLESS_OCR_LANGUAGES`](configuration.md#PAPERLESS_OCR_LANGUAGES).
- **Transform**: post-processing applied to the text, such as stripping whitespace, keeping only numbers,
parsing a date or reading a QR code or barcode instead of text.
- **Date format**: for the "Parse date" transform, the format of the date using
[Python date codes](https://docs.python.org/3/library/datetime.html#format-codes), e.g. `%d.%m.%Y`.
Leave it empty to detect the format automatically.
- **Validation regex**: a regular expression the whole value must match. Values that do not match are not
written.
Zones can fill custom fields of type text, long text, URL, date, integer, number, monetary or boolean.
Document link and select fields are not supported.
### Combining zones
When several zones fill the same field, a **Combined value** setting appears for that field. Use it to arrange
the zones' values with `{Zone name}` tokens and any text in between, for example `{Start} - {End}`. If it
is left empty, the values are joined with spaces in zone order.
### When templates run
When a document is added, every enabled template that matches its content runs automatically. A template is
only used for a document if the document's owner can view the template, so users' templates apply to their own
documents, and unowned templates apply to everyone's. Templates run in name order, so if two templates fill the
same field, the later one wins.
Turn off **Enabled** to pause a template without deleting it. Disabled templates never run automatically, but
can still be run manually.
### Running templates manually
To run templates on documents that are already in Paperless-ngx, open the **Actions** menu on the document
details page:
- **Run Zone OCR** runs every enabled template that matches the document.
- The button next to it opens a dialog where you can choose one template to run
instead. A template you choose runs even if it does not match the document or is disabled.
The bulk editor's **Run Zone OCR** action opens the same dialog for the selected documents. Bulk runs are
processed in the background.
!!! warning
Zone OCR overwrites the fields it fills. A zone that finds no value leaves its field unchanged.
Managing OCR templates requires the OCR template permissions. Running zone OCR on documents requires
permission to change them, and choosing a template requires permission to view it.
## PDF Actions ## PDF Actions
Paperless-ngx supports basic editing operations for PDFs (these operations currently cannot be performed on non-PDF files). When viewing an individual document you can Paperless-ngx supports basic editing operations for PDFs (these operations currently cannot be performed on non-PDF files). When viewing an individual document you can
+4 -4
View File
@@ -27,7 +27,7 @@ dependencies = [
# WARNING: django does not use semver. # WARNING: django does not use semver.
# Only patch versions are guaranteed to not introduce breaking changes. # Only patch versions are guaranteed to not introduce breaking changes.
"django~=5.2.18", "django~=5.2.18",
"django-allauth[mfa,socialaccount]~=65.20.0", "django-allauth[mfa,socialaccount]~=65.21.1",
"django-auditlog~=3.4.1", "django-auditlog~=3.4.1",
"django-cachalot~=2.9.0", "django-cachalot~=2.9.0",
"django-compression-middleware~=0.5.0", "django-compression-middleware~=0.5.0",
@@ -112,14 +112,14 @@ lint = [
testing = [ testing = [
"daphne", "daphne",
"factory-boy~=3.3.1", "factory-boy~=3.3.1",
"faker>=40.36,<40.40", "faker>=40.36,<40.44",
"imagehash", "imagehash",
"pytest~=9.1.1", "pytest~=9.1.1",
"pytest-cov~=7.1.0", "pytest-cov~=7.1.0",
"pytest-django>=4.12,<4.15", "pytest-django>=4.12,<4.15",
"pytest-env~=1.7.0", "pytest-env~=1.8.1",
"pytest-httpx", "pytest-httpx",
"pytest-mock~=3.15.1", "pytest-mock~=3.17.0",
"pytest-randomly~=5.0.0", "pytest-randomly~=5.0.0",
"pytest-rerunfailures~=16.4", "pytest-rerunfailures~=16.4",
"pytest-sugar", "pytest-sugar",
+34 -34
View File
@@ -839,7 +839,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context> <context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context>
<context context-type="linenumber">30</context> <context context-type="linenumber">32</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/permissions-dialog/permissions-dialog.component.html</context> <context context-type="sourcefile">src/app/components/common/permissions-dialog/permissions-dialog.component.html</context>
@@ -1141,7 +1141,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context> <context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context>
<context context-type="linenumber">18</context> <context context-type="linenumber">20</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/permissions-select/permissions-select.component.html</context> <context context-type="sourcefile">src/app/components/common/permissions-select/permissions-select.component.html</context>
@@ -1366,7 +1366,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1913</context> <context context-type="linenumber">1918</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="1577733187050997705" datatype="html"> <trans-unit id="1577733187050997705" datatype="html">
@@ -3401,11 +3401,11 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1527</context> <context context-type="linenumber">1532</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1914</context> <context context-type="linenumber">1919</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context> <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -4152,7 +4152,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1476</context> <context context-type="linenumber">1481</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context> <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -4296,7 +4296,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1967</context> <context context-type="linenumber">1972</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6661109599266152398" datatype="html"> <trans-unit id="6661109599266152398" datatype="html">
@@ -4307,7 +4307,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1968</context> <context context-type="linenumber">1973</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="5162686434580248853" datatype="html"> <trans-unit id="5162686434580248853" datatype="html">
@@ -4318,7 +4318,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1969</context> <context context-type="linenumber">1974</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6665634854532231106" datatype="html"> <trans-unit id="6665634854532231106" datatype="html">
@@ -6575,7 +6575,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1480</context> <context context-type="linenumber">1485</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context> <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -7156,14 +7156,14 @@
<source>Copy ID</source> <source>Copy ID</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context> <context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context>
<context context-type="linenumber">6</context> <context context-type="linenumber">7</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="5392341774767336507" datatype="html"> <trans-unit id="5392341774767336507" datatype="html">
<source>Copied!</source> <source>Copied!</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context> <context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context>
<context context-type="linenumber">8</context> <context context-type="linenumber">9</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/profile-edit-dialog/profile-edit-dialog.component.html</context> <context context-type="sourcefile">src/app/components/common/profile-edit-dialog/profile-edit-dialog.component.html</context>
@@ -7198,7 +7198,7 @@
<source>Read more</source> <source>Read more</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context> <context context-type="sourcefile">src/app/components/common/page-header/page-header.component.html</context>
<context context-type="linenumber">24,25</context> <context context-type="linenumber">26,27</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/permissions-select/permissions-select.component.html</context> <context context-type="sourcefile">src/app/components/common/permissions-select/permissions-select.component.html</context>
@@ -8473,7 +8473,7 @@
<source>No documents</source> <source>No documents</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.html</context> <context context-type="sourcefile">src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.html</context>
<context context-type="linenumber">189,192</context> <context context-type="linenumber">191,194</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="1069523139277190436" datatype="html"> <trans-unit id="1069523139277190436" datatype="html">
@@ -9142,14 +9142,14 @@
<source>Do you really want to move the document &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash?</source> <source>Do you really want to move the document &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash?</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1477</context> <context context-type="linenumber">1482</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="282586936710748252" datatype="html"> <trans-unit id="282586936710748252" datatype="html">
<source>Documents can be restored prior to permanent deletion.</source> <source>Documents can be restored prior to permanent deletion.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1478</context> <context context-type="linenumber">1483</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context> <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -9160,21 +9160,21 @@
<source>Moved &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash</source> <source>Moved &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1496</context> <context context-type="linenumber">1501</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="7295637485862454066" datatype="html"> <trans-unit id="7295637485862454066" datatype="html">
<source>Error deleting document</source> <source>Error deleting document</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1503</context> <context context-type="linenumber">1508</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="619486176823357521" datatype="html"> <trans-unit id="619486176823357521" datatype="html">
<source>Reprocess confirm</source> <source>Reprocess confirm</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1523</context> <context context-type="linenumber">1528</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context> <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -9185,102 +9185,102 @@
<source>This operation will permanently recreate the archive file for this document.</source> <source>This operation will permanently recreate the archive file for this document.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1524</context> <context context-type="linenumber">1529</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="302054111564709516" datatype="html"> <trans-unit id="302054111564709516" datatype="html">
<source>The archive file will be re-generated with the current settings.</source> <source>The archive file will be re-generated with the current settings.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1525</context> <context context-type="linenumber">1530</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="4700389117298802932" datatype="html"> <trans-unit id="4700389117298802932" datatype="html">
<source>Reprocess operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source> <source>Reprocess operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1538</context> <context context-type="linenumber">1543</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="4409560272830824468" datatype="html"> <trans-unit id="4409560272830824468" datatype="html">
<source>Error executing operation</source> <source>Error executing operation</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1549</context> <context context-type="linenumber">1554</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6030453331794586802" datatype="html"> <trans-unit id="6030453331794586802" datatype="html">
<source>Error downloading document</source> <source>Error downloading document</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1613</context> <context context-type="linenumber">1618</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="4458954481601077369" datatype="html"> <trans-unit id="4458954481601077369" datatype="html">
<source>Page Fit</source> <source>Page Fit</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1691</context> <context context-type="linenumber">1696</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="4663705961777238777" datatype="html"> <trans-unit id="4663705961777238777" datatype="html">
<source>PDF edit operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source> <source>PDF edit operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1934</context> <context context-type="linenumber">1939</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="9043972994040261999" datatype="html"> <trans-unit id="9043972994040261999" datatype="html">
<source>Error executing PDF edit operation</source> <source>Error executing PDF edit operation</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1946</context> <context context-type="linenumber">1951</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6172690334763056188" datatype="html"> <trans-unit id="6172690334763056188" datatype="html">
<source>Please enter the current password before attempting to remove it.</source> <source>Please enter the current password before attempting to remove it.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1957</context> <context context-type="linenumber">1962</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="968660764814228922" datatype="html"> <trans-unit id="968660764814228922" datatype="html">
<source>Password removal operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source> <source>Password removal operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1991</context> <context context-type="linenumber">1996</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="2282118435712883014" datatype="html"> <trans-unit id="2282118435712883014" datatype="html">
<source>Error executing password removal operation</source> <source>Error executing password removal operation</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2005</context> <context context-type="linenumber">2010</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="3740891324955700797" datatype="html"> <trans-unit id="3740891324955700797" datatype="html">
<source>Print failed.</source> <source>Print failed.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2055</context> <context context-type="linenumber">2060</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6457245677384603573" datatype="html"> <trans-unit id="6457245677384603573" datatype="html">
<source>Error loading document for printing.</source> <source>Error loading document for printing.</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2064</context> <context context-type="linenumber">2069</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6085793215710522488" datatype="html"> <trans-unit id="6085793215710522488" datatype="html">
<source>An error occurred loading tiff: <x id="PH" equiv-text="err.toString()"/></source> <source>An error occurred loading tiff: <x id="PH" equiv-text="err.toString()"/></source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2147</context> <context context-type="linenumber">2152</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context> <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2153</context> <context context-type="linenumber">2158</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="4958946940233632319" datatype="html"> <trans-unit id="4958946940233632319" datatype="html">
+14 -1
View File
@@ -13,6 +13,7 @@ import { DocumentDetailComponent } from './components/document-detail/document-d
import { DocumentListComponent } from './components/document-list/document-list.component' import { DocumentListComponent } from './components/document-list/document-list.component'
import { DocumentAttributesComponent } from './components/manage/document-attributes/document-attributes.component' import { DocumentAttributesComponent } from './components/manage/document-attributes/document-attributes.component'
import { MailComponent } from './components/manage/mail/mail.component' import { MailComponent } from './components/manage/mail/mail.component'
import { OcrTemplatesComponent } from './components/manage/ocr-templates/ocr-templates.component'
import { SavedViewsComponent } from './components/manage/saved-views/saved-views.component' import { SavedViewsComponent } from './components/manage/saved-views/saved-views.component'
import { ShareLinksComponent } from './components/manage/share-links/share-links.component' import { ShareLinksComponent } from './components/manage/share-links/share-links.component'
import { WorkflowsComponent } from './components/manage/workflows/workflows.component' import { WorkflowsComponent } from './components/manage/workflows/workflows.component'
@@ -37,7 +38,7 @@ export const routes: Routes = [
path: 'dashboard', path: 'dashboard',
component: DashboardComponent, component: DashboardComponent,
data: { data: {
componentName: 'AppFrameComponent', componentName: 'DashboardComponent',
}, },
}, },
{ {
@@ -275,6 +276,18 @@ export const routes: Routes = [
componentName: 'WorkflowsComponent', componentName: 'WorkflowsComponent',
}, },
}, },
{
path: 'ocr-templates',
component: OcrTemplatesComponent,
canActivate: [PermissionsGuard],
data: {
requiredPermission: {
action: PermissionAction.View,
type: PermissionType.OcrTemplate,
},
componentName: 'OcrTemplatesComponent',
},
},
{ {
path: 'mail', path: 'mail',
component: MailComponent, component: MailComponent,
@@ -113,6 +113,7 @@ const sidebarItemLabels: Record<HideableSidebarItemID, string> = {
[HideableSidebarItemID.SavedViews]: $localize`Saved Views`, [HideableSidebarItemID.SavedViews]: $localize`Saved Views`,
[HideableSidebarItemID.ShareLinks]: $localize`Share Links`, [HideableSidebarItemID.ShareLinks]: $localize`Share Links`,
[HideableSidebarItemID.Workflows]: $localize`Workflows`, [HideableSidebarItemID.Workflows]: $localize`Workflows`,
[HideableSidebarItemID.OcrTemplates]: $localize`OCR Templates`,
[HideableSidebarItemID.Mail]: $localize`Mail`, [HideableSidebarItemID.Mail]: $localize`Mail`,
[HideableSidebarItemID.Documentation]: $localize`Documentation`, [HideableSidebarItemID.Documentation]: $localize`Documentation`,
} }
@@ -289,6 +289,17 @@
<pngx-input-switch class="position-absolute top-50 end-0 translate-middle-y me-1" [class.d-none]="slimSidebarEnabled || slimSidebarAnimating()" [compact]="true" title="Workflows" i18n-title [ngModel]="!settingsService.sidebarItemIsHidden(HideableSidebarItemID.Workflows)" (ngModelChange)="toggleSidebarItem(HideableSidebarItemID.Workflows, $event)"></pngx-input-switch> <pngx-input-switch class="position-absolute top-50 end-0 translate-middle-y me-1" [class.d-none]="slimSidebarEnabled || slimSidebarAnimating()" [compact]="true" title="Workflows" i18n-title [ngModel]="!settingsService.sidebarItemIsHidden(HideableSidebarItemID.Workflows)" (ngModelChange)="toggleSidebarItem(HideableSidebarItemID.Workflows, $event)"></pngx-input-switch>
} }
</li> </li>
<li class="nav-item app-link position-relative" [class.d-none]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.OcrTemplates) && !settingsService.organizingSidebarItems()"
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.OcrTemplate }">
<a class="nav-link" [class.opacity-50]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.OcrTemplates)" [class.pe-5]="settingsService.organizingSidebarItems() && !slimSidebarEnabled && !slimSidebarAnimating()" routerLink="ocr-templates" routerLinkActive="active" (click)="closeMenu()"
ngbPopover="OCR Templates" i18n-ngbPopover [disablePopover]="!slimSidebarPopoversEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<pngx-bs-icon class="me-2" name="bounding-box-circles"></pngx-bs-icon><span class="nav-link-label"><ng-container i18n>OCR Templates</ng-container></span>
</a>
@if (settingsService.organizingSidebarItems()) {
<pngx-input-switch class="position-absolute top-50 end-0 translate-middle-y me-1" [class.d-none]="slimSidebarEnabled || slimSidebarAnimating()" [compact]="true" title="OCR Templates" i18n-title [ngModel]="!settingsService.sidebarItemIsHidden(HideableSidebarItemID.OcrTemplates)" (ngModelChange)="toggleSidebarItem(HideableSidebarItemID.OcrTemplates, $event)"></pngx-input-switch>
}
</li>
<li class="nav-item app-link position-relative" [class.d-none]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail) && !settingsService.organizingSidebarItems()" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.MailAccount }" <li class="nav-item app-link position-relative" [class.d-none]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail) && !settingsService.organizingSidebarItems()" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.MailAccount }"
tourAnchor="tour.mail"> tourAnchor="tour.mail">
<a class="nav-link" [class.opacity-50]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail)" [class.pe-5]="settingsService.organizingSidebarItems() && !slimSidebarEnabled && !slimSidebarAnimating()" routerLink="mail" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Mail" <a class="nav-link" [class.opacity-50]="settingsService.sidebarItemIsHidden(HideableSidebarItemID.Mail)" [class.pe-5]="settingsService.organizingSidebarItems() && !slimSidebarEnabled && !slimSidebarAnimating()" routerLink="mail" routerLinkActive="active" (click)="closeMenu()" ngbPopover="Mail"
@@ -293,6 +293,7 @@ describe('AppFrameComponent', () => {
HideableSidebarItemID.Dashboard, HideableSidebarItemID.Dashboard,
HideableSidebarItemID.Workflows, HideableSidebarItemID.Workflows,
HideableSidebarItemID.ShareLinks, HideableSidebarItemID.ShareLinks,
HideableSidebarItemID.OcrTemplates,
]) ])
fixture.detectChanges() fixture.detectChanges()
@@ -308,6 +309,10 @@ describe('AppFrameComponent', () => {
fixture.nativeElement.querySelector('[routerLink="share-links"]') fixture.nativeElement.querySelector('[routerLink="share-links"]')
.parentElement.classList .parentElement.classList
).toContain('d-none') ).toContain('d-none')
expect(
fixture.nativeElement.querySelector('[routerLink="ocr-templates"]')
.parentElement.classList
).toContain('d-none')
expect( expect(
fixture.nativeElement.querySelector('[routerLink="mail"]').parentElement fixture.nativeElement.querySelector('[routerLink="mail"]').parentElement
.classList .classList
@@ -320,12 +325,21 @@ describe('AppFrameComponent', () => {
]) ])
settingsService.sidebarHiddenItemsEditing.set([ settingsService.sidebarHiddenItemsEditing.set([
HideableSidebarItemID.Dashboard, HideableSidebarItemID.Dashboard,
HideableSidebarItemID.OcrTemplates,
]) ])
fixture.detectChanges() fixture.detectChanges()
expect( expect(
fixture.nativeElement.querySelectorAll('pngx-input-switch').length fixture.nativeElement.querySelectorAll('pngx-input-switch').length
).toBe(6) ).toBe(7)
const ocrTemplatesLink = fixture.nativeElement.querySelector(
'[routerLink="ocr-templates"]'
)
expect(ocrTemplatesLink.parentElement.classList).not.toContain('d-none')
expect(ocrTemplatesLink.classList).toContain('opacity-50')
expect(
ocrTemplatesLink.parentElement.querySelector('pngx-input-switch')
).not.toBeNull()
expect( expect(
fixture.nativeElement.querySelector('[routerLink="dashboard"]') fixture.nativeElement.querySelector('[routerLink="dashboard"]')
.parentElement.classList .parentElement.classList
@@ -16,6 +16,7 @@ import basket from 'bootstrap-icons/icons/basket.svg'
import bell from 'bootstrap-icons/icons/bell.svg' import bell from 'bootstrap-icons/icons/bell.svg'
import bodyText from 'bootstrap-icons/icons/body-text.svg' import bodyText from 'bootstrap-icons/icons/body-text.svg'
import bookmark from 'bootstrap-icons/icons/bookmark.svg' import bookmark from 'bootstrap-icons/icons/bookmark.svg'
import boundingBoxCircles from 'bootstrap-icons/icons/bounding-box-circles.svg'
import boxArrowInRight from 'bootstrap-icons/icons/box-arrow-in-right.svg' import boxArrowInRight from 'bootstrap-icons/icons/box-arrow-in-right.svg'
import boxArrowUpRight from 'bootstrap-icons/icons/box-arrow-up-right.svg' import boxArrowUpRight from 'bootstrap-icons/icons/box-arrow-up-right.svg'
import boxArrowUp from 'bootstrap-icons/icons/box-arrow-up.svg' import boxArrowUp from 'bootstrap-icons/icons/box-arrow-up.svg'
@@ -77,6 +78,7 @@ import fileEarmarkMinus from 'bootstrap-icons/icons/file-earmark-minus.svg'
import fileEarmarkPerson from 'bootstrap-icons/icons/file-earmark-person.svg' import fileEarmarkPerson from 'bootstrap-icons/icons/file-earmark-person.svg'
import fileEarmarkPlus from 'bootstrap-icons/icons/file-earmark-plus.svg' import fileEarmarkPlus from 'bootstrap-icons/icons/file-earmark-plus.svg'
import fileEarmarkRichtext from 'bootstrap-icons/icons/file-earmark-richtext.svg' import fileEarmarkRichtext from 'bootstrap-icons/icons/file-earmark-richtext.svg'
import fileEarmarkRuled from 'bootstrap-icons/icons/file-earmark-ruled.svg'
import fileEarmarkSpreadsheet from 'bootstrap-icons/icons/file-earmark-spreadsheet.svg' import fileEarmarkSpreadsheet from 'bootstrap-icons/icons/file-earmark-spreadsheet.svg'
import fileEarmark from 'bootstrap-icons/icons/file-earmark.svg' import fileEarmark from 'bootstrap-icons/icons/file-earmark.svg'
import fileText from 'bootstrap-icons/icons/file-text.svg' import fileText from 'bootstrap-icons/icons/file-text.svg'
@@ -130,6 +132,7 @@ import search from 'bootstrap-icons/icons/search.svg'
import send from 'bootstrap-icons/icons/send.svg' import send from 'bootstrap-icons/icons/send.svg'
import shop from 'bootstrap-icons/icons/shop.svg' import shop from 'bootstrap-icons/icons/shop.svg'
import slashCircle from 'bootstrap-icons/icons/slash-circle.svg' import slashCircle from 'bootstrap-icons/icons/slash-circle.svg'
import sliders2 from 'bootstrap-icons/icons/sliders2.svg'
import sliders2Vertical from 'bootstrap-icons/icons/sliders2-vertical.svg' import sliders2Vertical from 'bootstrap-icons/icons/sliders2-vertical.svg'
import sortAlphaDown from 'bootstrap-icons/icons/sort-alpha-down.svg' import sortAlphaDown from 'bootstrap-icons/icons/sort-alpha-down.svg'
import sortAlphaUpAlt from 'bootstrap-icons/icons/sort-alpha-up-alt.svg' import sortAlphaUpAlt from 'bootstrap-icons/icons/sort-alpha-up-alt.svg'
@@ -178,6 +181,7 @@ export const icons: Record<string, string> = {
braces, braces,
bodyText, bodyText,
bookmark, bookmark,
boundingBoxCircles,
boxArrowInRight, boxArrowInRight,
boxArrowUp, boxArrowUp,
boxArrowUpRight, boxArrowUpRight,
@@ -239,6 +243,7 @@ export const icons: Record<string, string> = {
fileEarmarkPerson, fileEarmarkPerson,
fileEarmarkPlus, fileEarmarkPlus,
fileEarmarkRichtext, fileEarmarkRichtext,
fileEarmarkRuled,
fileEarmarkSpreadsheet, fileEarmarkSpreadsheet,
files, files,
fileText, fileText,
@@ -290,6 +295,7 @@ export const icons: Record<string, string> = {
search, search,
send, send,
slashCircle, slashCircle,
sliders2,
shop, shop,
sliders2Vertical, sliders2Vertical,
sortAlphaDown, sortAlphaDown,
@@ -0,0 +1,30 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (messageBold) {
<p class="text-break"><b>{{messageBold}}</b></p>
}
@if (message) {
<p class="text-break" [innerHTML]="message"></p>
}
<pngx-input-select
[items]="templates()"
[(ngModel)]="template"
[allowNull]="true"
title="OCR template"
i18n-title
placeholder="All matching templates"
i18n-placeholder
></pngx-input-select>
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled()">
{{btnCaption}}
</button>
</div>
@@ -0,0 +1,52 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { of } from 'rxjs'
import { MATCH_NONE } from 'src/app/data/matching-model'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { ZoneOcrConfirmDialogComponent } from './zone-ocr-confirm-dialog.component'
const templates: OcrTemplate[] = [
{
id: 1,
name: 'Invoices',
match: '',
matching_algorithm: MATCH_NONE,
sample_document: null,
source_width: 0,
source_height: 0,
enabled: true,
zones: [],
},
]
describe('ZoneOcrConfirmDialogComponent', () => {
let component: ZoneOcrConfirmDialogComponent
let fixture: ComponentFixture<ZoneOcrConfirmDialogComponent>
beforeEach(() => {
TestBed.configureTestingModule({
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
imports: [ZoneOcrConfirmDialogComponent],
})
jest
.spyOn(TestBed.inject(OcrTemplateService), 'listAll')
.mockReturnValue(of({ count: 1, all: [1], results: templates }))
fixture = TestBed.createComponent(ZoneOcrConfirmDialogComponent)
component = fixture.componentInstance
fixture.detectChanges()
})
it('should offer templates and default to all matching templates', () => {
expect(component.templates()).toEqual(templates)
expect(component.template).toBeNull()
expect(fixture.nativeElement.textContent).toContain('OCR template')
})
})
@@ -0,0 +1,31 @@
import { Component, OnInit, inject, signal } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { takeUntil } from 'rxjs'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { SelectComponent } from '../../input/select/select.component'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-zone-ocr-confirm-dialog',
templateUrl: './zone-ocr-confirm-dialog.component.html',
imports: [FormsModule, SelectComponent],
})
export class ZoneOcrConfirmDialogComponent
extends ConfirmDialogComponent
implements OnInit
{
private readonly ocrTemplateService = inject(OcrTemplateService)
public readonly templates = signal<OcrTemplate[]>([])
/** The template to run, or null for every template that matches. */
template: number | null = null
ngOnInit() {
this.ocrTemplateService
.listAll()
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((r) => this.templates.set(r.results))
}
}
@@ -40,7 +40,7 @@
(change)="onChange(selectedDocumentIDs)"> (change)="onChange(selectedDocumentIDs)">
<ng-template ng-label-tmp let-document="item"> <ng-template ng-label-tmp let-document="item">
<div class="d-flex align-items-center"> <div class="d-flex align-items-center">
@if (!disabled) { @if (!disabled && multiple) {
<button class="btn p-0 lh-1" (click)="unselect(document)" (mousedown)="$event.stopImmediatePropagation()" type="button" title="Remove link" i18n-title><pngx-bs-icon name="x"></pngx-bs-icon></button> <button class="btn p-0 lh-1" (click)="unselect(document)" (mousedown)="$event.stopImmediatePropagation()" type="button" title="Remove link" i18n-title><pngx-bs-icon name="x"></pngx-bs-icon></button>
} }
@if (document.title) { @if (document.title) {
@@ -81,6 +81,9 @@ export class DocumentLinkComponent
@Input() @Input()
placeholder: string = $localize`Search for documents` placeholder: string = $localize`Search for documents`
@Input()
multiple: boolean = true
/** /**
* Parent for ng-select dropdown, needed to prevent close on click. * Parent for ng-select dropdown, needed to prevent close on click.
*/ */
@@ -1,16 +1,18 @@
<div class="row pt-3 pb-2 align-items-center"> <div class="row pt-3 pb-2 align-items-center">
<div class="col-md text-truncate"> <div class="col-md text-truncate">
<h3 class="d-flex align-items-center mb-2 mb-md-1" style="line-height: 1.4"> <h3 class="d-flex align-items-center mb-2 mb-md-1" style="line-height: 1.4">
<span class="text-truncate">{{title()}}</span> <span class="d-flex align-items-center mw-100">
@if (id()) { <span class="text-truncate">{{title()}}</span>
<button type="button" class="badge border-0 bg-primary text-primary-text-contrast ms-3 small fs-normal" (click)="copyID()" title="Copy ID" i18n-title> @if (id()) {
@if (copied()) { <button type="button" class="badge border-0 bg-primary text-primary-text-contrast ms-3 small fs-normal flex-shrink-0 text-nowrap" (click)="copyID()" title="Copy ID" i18n-title>
<pngx-bs-icon width="1em" height="1em" name="clipboard-check" class="me-1"></pngx-bs-icon><ng-container i18n>Copied!</ng-container> @if (copied()) {
} @else { <pngx-bs-icon width="1em" height="1em" name="clipboard-check" class="me-1"></pngx-bs-icon><ng-container i18n>Copied!</ng-container>
ID: {{id()}} } @else {
} ID: {{id()}}
</button> }
} </button>
}
</span>
@if (subTitle()) { @if (subTitle()) {
<span class="page-subtitle h6 mb-0 mt-1 fw-normal ms-md-3 text-truncate" style="line-height: 1.4">{{subTitle()}}</span> <span class="page-subtitle h6 mb-0 mt-1 fw-normal ms-md-3 text-truncate" style="line-height: 1.4">{{subTitle()}}</span>
} }
@@ -62,13 +62,13 @@
} @else { } @else {
@switch (field) { @switch (field) {
@case (DisplayField.ADDED) { @case (DisplayField.ADDED) {
<a routerLink="/documents/{{doc.id}}" class="btn-link text-dark text-decoration-none py-2 py-md-3" title="Open document" i18n-title>{{doc.added | customDate}}</a> <a routerLink="/documents/{{doc.id}}" (click)="activateViewForNavigation()" class="btn-link text-dark text-decoration-none py-2 py-md-3" title="Open document" i18n-title>{{doc.added | customDate}}</a>
} }
@case (DisplayField.CREATED) { @case (DisplayField.CREATED) {
<a routerLink="/documents/{{doc.id}}" class="btn-link text-dark text-decoration-none py-2 py-md-3" title="Open document" i18n-title>{{doc.created | customDate}}</a> <a routerLink="/documents/{{doc.id}}" (click)="activateViewForNavigation()" class="btn-link text-dark text-decoration-none py-2 py-md-3" title="Open document" i18n-title>{{doc.created | customDate}}</a>
} }
@case (DisplayField.TITLE) { @case (DisplayField.TITLE) {
<a routerLink="/documents/{{doc.id}}" title="Open document" i18n-title class="btn-link text-dark text-decoration-none py-2 py-md-3">{{doc.title | documentTitle}}</a> <a routerLink="/documents/{{doc.id}}" (click)="activateViewForNavigation()" title="Open document" i18n-title class="btn-link text-dark text-decoration-none py-2 py-md-3">{{doc.title | documentTitle}}</a>
} }
@case (DisplayField.CORRESPONDENT) { @case (DisplayField.CORRESPONDENT) {
@if (doc.correspondent) { @if (doc.correspondent) {
@@ -106,7 +106,7 @@
} }
@case (DisplayField.NOTES) { @case (DisplayField.NOTES) {
@if (doc.notes.length) { @if (doc.notes.length) {
<a routerLink="/documents/{{doc.id}}/notes" class="btn btn-sm p-0"> <a routerLink="/documents/{{doc.id}}/notes" (click)="activateViewForNavigation()" class="btn btn-sm p-0">
<span class="badge rounded-pill bg-light border text-primary"> <span class="badge rounded-pill bg-light border text-primary">
<pngx-bs-icon width="1.2em" height="1.2em" class="ms-1 me-1" name="chat-left-text"></pngx-bs-icon> <pngx-bs-icon width="1.2em" height="1.2em" class="ms-1 me-1" name="chat-left-text"></pngx-bs-icon>
{{doc.notes.length}}</span> {{doc.notes.length}}</span>
@@ -150,6 +150,7 @@
<pngx-document-card-small <pngx-document-card-small
class="p-0" class="p-0"
[ngStyle]="{ opacity: show() ? 1 : 1 - (i * 1/documents().length) }" [ngStyle]="{ opacity: show() ? 1 : 1 - (i * 1/documents().length) }"
(click)="activateViewForNavigation()"
(dblClickDocument)="openDocumentDetail(d)" (dblClickDocument)="openDocumentDetail(d)"
[document]="d" [document]="d"
[displayFields]="displayFields()" [displayFields]="displayFields()"
@@ -173,6 +174,7 @@
} }
@for (d of documents(); track d.id; let i = $index) { @for (d of documents(); track d.id; let i = $index) {
<pngx-document-card-large <pngx-document-card-large
(click)="activateViewForNavigation()"
(dblClickDocument)="openDocumentDetail(d)" (dblClickDocument)="openDocumentDetail(d)"
[document]="d" [document]="d"
[ngStyle]="{ opacity: show() ? 1 : 1 - (i * 1/documents().length) }" [ngStyle]="{ opacity: show() ? 1 : 1 - (i * 1/documents().length) }"
@@ -16,7 +16,11 @@ import { routes } from 'src/app/app-routing.module'
import { CustomFieldDisplayComponent } from 'src/app/components/common/custom-field-display/custom-field-display.component' import { CustomFieldDisplayComponent } from 'src/app/components/common/custom-field-display/custom-field-display.component'
import { PreviewPopupComponent } from 'src/app/components/common/preview-popup/preview-popup.component' import { PreviewPopupComponent } from 'src/app/components/common/preview-popup/preview-popup.component'
import { CustomFieldDataType } from 'src/app/data/custom-field' import { CustomFieldDataType } from 'src/app/data/custom-field'
import { DisplayField, DisplayMode } from 'src/app/data/document' import {
DEFAULT_DASHBOARD_VIEW_PAGE_SIZE,
DisplayField,
DisplayMode,
} from 'src/app/data/document'
import { import {
FILTER_CORRESPONDENT, FILTER_CORRESPONDENT,
FILTER_DOCUMENT_TYPE, FILTER_DOCUMENT_TYPE,
@@ -277,10 +281,40 @@ describe('SavedViewWidgetComponent', () => {
it('should navigate to document', () => { it('should navigate to document', () => {
const routerSpy = jest.spyOn(router, 'navigate') const routerSpy = jest.spyOn(router, 'navigate')
const activateSpy = jest.spyOn(
documentListViewService,
'activateSavedViewWithDocuments'
)
component.openDocumentDetail(documentResults[0]) component.openDocumentDetail(documentResults[0])
expect(activateSpy).toHaveBeenCalled()
expect(routerSpy).toHaveBeenCalledWith(['documents', documentResults[0].id]) expect(routerSpy).toHaveBeenCalledWith(['documents', documentResults[0].id])
}) })
it('should hand its documents to the list service when opening a document', () => {
jest.spyOn(documentService, 'listFiltered').mockReturnValue(
of({
all: [2, 3],
count: 25,
results: documentResults,
})
)
jest.spyOn(router, 'navigateByUrl').mockResolvedValue(true)
const activateSpy = jest
.spyOn(documentListViewService, 'activateSavedViewWithDocuments')
.mockImplementation(() => {})
component.ngOnInit()
fixture.detectChanges()
fixture.debugElement
.query(By.css('a[title="Open document"]'))
.nativeElement.click()
expect(activateSpy).toHaveBeenCalledWith(savedView, documentResults, 25, 20)
})
it('should fall back to the default dashboard page size', () => {
component.savedView = { ...savedView, page_size: undefined }
expect(component.pageSize).toEqual(DEFAULT_DASHBOARD_VIEW_PAGE_SIZE)
})
it('should navigate via quickfilter on click tag', () => { it('should navigate via quickfilter on click tag', () => {
const qfSpy = jest.spyOn(documentListViewService, 'quickFilter') const qfSpy = jest.spyOn(documentListViewService, 'quickFilter')
component.clickTag(11, new MouseEvent('click')) component.clickTag(11, new MouseEvent('click'))
@@ -132,7 +132,7 @@ export class SavedViewWidgetComponent
ngOnInit(): void { ngOnInit(): void {
this.placeholderRows = Array.from( this.placeholderRows = Array.from(
{ {
length: this.savedView?.page_size ?? DEFAULT_DASHBOARD_VIEW_PAGE_SIZE, length: this.pageSize,
}, },
(_, index) => index (_, index) => index
) )
@@ -187,7 +187,7 @@ export class SavedViewWidgetComponent
this.documentService this.documentService
.listFiltered( .listFiltered(
1, 1,
this.savedView?.page_size ?? DEFAULT_DASHBOARD_VIEW_PAGE_SIZE, this.pageSize,
this.savedView.sort_field, this.savedView.sort_field,
this.savedView.sort_reverse, this.savedView.sort_reverse,
this.savedView.filter_rules, this.savedView.filter_rules,
@@ -292,7 +292,24 @@ export class SavedViewWidgetComponent
]) ])
} }
get pageSize(): number {
return this.savedView?.page_size ?? DEFAULT_DASHBOARD_VIEW_PAGE_SIZE
}
/**
* Pass the widget's documents to the list service to enable next / previous.
*/
activateViewForNavigation() {
this.documentListViewService.activateSavedViewWithDocuments(
this.savedView,
this.documents(),
this.count(),
this.pageSize
)
}
openDocumentDetail(document: Document) { openDocumentDetail(document: Document) {
this.activateViewForNavigation()
this.router.navigate(['documents', document.id]) this.router.navigate(['documents', document.id])
} }
@@ -61,6 +61,22 @@
<pngx-bs-icon name="pencil" class="me-2"></pngx-bs-icon><ng-container i18n>PDF Editor</ng-container> <pngx-bs-icon name="pencil" class="me-2"></pngx-bs-icon><ng-container i18n>PDF Editor</ng-container>
</button> </button>
<div class="d-flex" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.OcrTemplate }">
<button ngbDropdownItem (click)="runZoneOcr()" [disabled]="!userCanEdit || networkActive()">
<pngx-bs-icon width="1em" height="1em" name="bounding-box-circles" class="me-2"></pngx-bs-icon><span i18n>Run Zone OCR</span>
</button>
<button
ngbDropdownItem
class="w-auto flex-shrink-0 px-2"
(click)="runZoneOcrWithOptions()"
[disabled]="!userCanEdit || networkActive()"
title="Choose OCR template"
i18n-title
>
<pngx-bs-icon width="1em" height="1em" name="sliders2"></pngx-bs-icon>
</button>
</div>
@if (userIsOwner && (requiresPassword || password)) { @if (userIsOwner && (requiresPassword || password)) {
<button ngbDropdownItem (click)="removePassword()" [disabled]="!password"> <button ngbDropdownItem (click)="removePassword()" [disabled]="!password">
<pngx-bs-icon name="unlock" class="me-2"></pngx-bs-icon><ng-container i18n>Remove Password</ng-container> <pngx-bs-icon name="unlock" class="me-2"></pngx-bs-icon><ng-container i18n>Remove Password</ng-container>
@@ -90,7 +106,7 @@
<div class="row"> <div class="row">
<div class="col-md-6 col-xl-5 mb-4"> <div class="col-md-6 col-xl-5 mb-4">
<form [formGroup]='documentForm' (ngSubmit)="save()"> <form [formGroup]='documentForm' (ngSubmit)="save()" [attr.inert]="zoneOcrRunning() ? '' : null" [attr.aria-busy]="zoneOcrRunning()">
<div class="btn-toolbar mb-1 pb-3 gap-2 row-gap-2 border-bottom"> <div class="btn-toolbar mb-1 pb-3 gap-2 row-gap-2 border-bottom">
<div class="btn-group"> <div class="btn-group">
@@ -940,6 +940,20 @@ describe('DocumentDetailComponent', () => {
expect(navigateSpy).toHaveBeenCalledWith(['view', 77]) expect(navigateSpy).toHaveBeenCalledWith(['view', 77])
}) })
it('should close back to the dashboard even if a saved view is active', () => {
initNormally()
jest
.spyOn(componentRouterService, 'getComponentURLBefore')
.mockReturnValue('dashboard')
jest
.spyOn(documentListViewService, 'activeSavedViewId', 'get')
.mockReturnValue(77)
const navigateSpy = jest.spyOn(router, 'navigate')
component.close()
expect(navigateSpy).toHaveBeenCalledWith(['dashboard'])
expect(navigateSpy).not.toHaveBeenCalledWith(['view', 77])
})
it('should not close if e.g. user-cancelled', () => { it('should not close if e.g. user-cancelled', () => {
initNormally() initNormally()
jest.spyOn(openDocumentsService, 'closeDocument').mockReturnValue(of(false)) jest.spyOn(openDocumentsService, 'closeDocument').mockReturnValue(of(false))
@@ -1895,6 +1909,196 @@ describe('DocumentDetailComponent', () => {
expect(toastSpy).toHaveBeenCalledWith('Document reloaded.') expect(toastSpy).toHaveBeenCalledWith('Document reloaded.')
}) })
it('should run zone OCR, report results and reload the document', () => {
component.documentId.set(doc.id)
component.document.set(doc)
jest.spyOn(documentService, 'runZoneOcr').mockReturnValue(
of({
results: [
{ zone: 'Total', value: '102.10' },
{ zone: 'Due', value: '' },
],
} as any)
)
const loadSpy = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
const notifySpy = jest.spyOn(component.docChangeNotifier, 'next')
const toastSpy = jest.spyOn(toastService, 'showInfo')
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcrWithOptions()
const dialog = modalSpy.mock.results[0].value.componentInstance
expect(dialog.message).toBe(
'Fields filled by the template will be overwritten.'
)
dialog.template = 3
dialog.confirmClicked.emit()
expect(documentService.runZoneOcr).toHaveBeenCalledWith(doc.id, 3)
expect(toastSpy).toHaveBeenCalledWith(
'Filled 1 of 2 fields. No value found for: Due'
)
expect(notifySpy).toHaveBeenCalledWith(doc.id)
expect(loadSpy).toHaveBeenCalledWith(doc.id, true, expect.any(Function))
expect(component.zoneOcrRunning()).toBe(true)
loadSpy.mock.calls[0][2]()
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
})
it('should summarize zone OCR results', () => {
component.documentId.set(doc.id)
component.document.set(doc)
jest.spyOn(component as any, 'loadDocument').mockImplementation(() => {})
const runSpy = jest.spyOn(documentService, 'runZoneOcr')
const toastSpy = jest.spyOn(toastService, 'showInfo')
const modalSpy = jest.spyOn(modalService, 'open')
runSpy.mockReturnValueOnce(of({ status: 'completed', results: [] }))
component.runZoneOcr()
expect(modalSpy).not.toHaveBeenCalled()
expect(runSpy).toHaveBeenCalledWith(doc.id, null)
expect(toastSpy).toHaveBeenLastCalledWith(
'Zone OCR found no fields to fill.'
)
component.networkActive.set(false)
runSpy.mockReturnValueOnce(
of({
status: 'completed',
results: [
{ template: 'T', zone: 'Total', custom_field: 'Amount', value: 0 },
],
})
)
component.runZoneOcr()
expect(toastSpy).toHaveBeenLastCalledWith('Filled 1 of 1 fields.')
})
it('should confirm before running zone OCR over unsaved changes', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
component.documentForm.patchValue({ title: 'Unsaved title' })
const runSpy = jest.spyOn(documentService, 'runZoneOcr')
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcr()
const modal = modalSpy.mock.results[0].value
expect(modal.componentInstance.message).toBe(
'Running Zone OCR will discard them.'
)
expect(runSpy).not.toHaveBeenCalled()
modal.componentInstance.cancel()
expect(runSpy).not.toHaveBeenCalled()
expect(component.documentForm.get('title').value).toBe('Unsaved title')
expect(component.zoneOcrRunning()).toBe(false)
})
it('should warn about unsaved changes in the zone OCR options dialog', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
const runSpy = jest.spyOn(documentService, 'runZoneOcr')
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcrWithOptions()
const dialog = modalSpy.mock.results[0].value.componentInstance
expect(dialog.message).toBe('Running Zone OCR will discard them.')
expect(dialog.btnClass).toBe('btn-warning')
dialog.cancel()
expect(runSpy).not.toHaveBeenCalled()
component.networkActive.set(true)
component.runZoneOcrWithOptions()
expect(modalSpy).toHaveBeenCalledTimes(1)
})
it('should not allow saving or a second zone OCR run while one is running', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
const pending = new Subject<any>()
const runSpy = jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(pending)
const modalSpy = jest.spyOn(modalService, 'open')
component.runZoneOcr()
modalSpy.mock.results[0].value.componentInstance.confirmClicked.emit()
expect(runSpy).toHaveBeenCalledWith(doc.id, null)
expect(component.zoneOcrRunning()).toBe(true)
expect(component.networkActive()).toBe(true)
const saveSpy = jest.spyOn(documentService, 'patch')
component.save()
expect(saveSpy).not.toHaveBeenCalled()
component.runZoneOcr()
expect(modalSpy).toHaveBeenCalledTimes(1)
expect(runSpy).toHaveBeenCalledTimes(1)
pending.error(new Error('failed'))
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
expect(component.document().__changedFields).toEqual(['title'])
})
it('should not reload after zone OCR if another document is open', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: [] })
const pending = new Subject<any>()
jest.spyOn(documentService, 'runZoneOcr').mockReturnValue(pending)
const loadSpy = jest.spyOn(component as any, 'loadDocument')
component.runZoneOcr()
component.documentId.set(99)
pending.next({ results: [] })
expect(loadSpy).not.toHaveBeenCalled()
expect(component.zoneOcrRunning()).toBe(false)
pending.complete()
})
it('should keep unsaved changes when zone OCR is skipped', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, __changedFields: ['title'] })
component.documentForm.patchValue({ title: 'Unsaved title' })
const modalSpy = jest.spyOn(modalService, 'open')
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(of({ status: 'skipped', results: [] }))
const loadSpy = jest.spyOn(component as any, 'loadDocument')
const infoSpy = jest.spyOn(toastService, 'showInfo')
component.runZoneOcr()
modalSpy.mock.results[0].value.componentInstance.confirm()
expect(loadSpy).not.toHaveBeenCalled()
expect(component.documentForm.get('title').value).toBe('Unsaved title')
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
expect(infoSpy).toHaveBeenCalledWith(
'No OCR template applies to this document.'
)
})
it('should show an error when zone OCR fails', () => {
component.document.set(doc)
jest
.spyOn(documentService, 'runZoneOcr')
.mockReturnValue(throwError(() => new Error('nope')))
const loadSpy = jest.spyOn(component as any, 'loadDocument')
const errorSpy = jest.spyOn(toastService, 'showError')
component.runZoneOcr()
expect(errorSpy).toHaveBeenCalledWith(
'Error running zone OCR.',
expect.any(Error)
)
expect(loadSpy).not.toHaveBeenCalled()
expect(component.zoneOcrRunning()).toBe(false)
expect(component.networkActive()).toBe(false)
})
it('should change preview element by render type', () => { it('should change preview element by render type', () => {
initNormally() initNormally()
component.document.update((document) => ({ component.document.update((document) => ({
@@ -64,6 +64,7 @@ import {
FILTER_STORAGE_PATH, FILTER_STORAGE_PATH,
} from 'src/app/data/filter-rule-type' } from 'src/app/data/filter-rule-type'
import { ObjectWithId } from 'src/app/data/object-with-id' import { ObjectWithId } from 'src/app/data/object-with-id'
import { OcrZoneRunResult } from 'src/app/data/ocr-template'
import { StoragePath } from 'src/app/data/storage-path' import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag' import { Tag } from 'src/app/data/tag'
import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings' import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings'
@@ -104,6 +105,7 @@ import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component' import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component' import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component' import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { ZoneOcrConfirmDialogComponent } from '../common/confirm-dialog/zone-ocr-confirm-dialog/zone-ocr-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component' import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component' import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component' import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -266,6 +268,7 @@ export class DocumentDetailComponent
readonly error = signal<any>(undefined) readonly error = signal<any>(undefined)
readonly networkActive = signal(false) readonly networkActive = signal(false)
readonly zoneOcrRunning = signal(false)
readonly documentId = signal<number>(undefined) readonly documentId = signal<number>(undefined)
readonly document = signal<Document>(undefined) readonly document = signal<Document>(undefined)
@@ -543,7 +546,11 @@ export class DocumentDetailComponent
) )
} }
private loadDocument(documentId: number, forceRemote: boolean = false): void { private loadDocument(
documentId: number,
forceRemote: boolean = false,
onComplete?: () => void
): void {
let redirectedToRoot = false let redirectedToRoot = false
this.closeIncomingUpdateModal() this.closeIncomingUpdateModal()
this.pendingIncomingUpdate = null this.pendingIncomingUpdate = null
@@ -600,7 +607,8 @@ export class DocumentDetailComponent
}), }),
first(), first(),
takeUntil(this.unsubscribeNotifier), takeUntil(this.unsubscribeNotifier),
takeUntil(this.docChangeNotifier) takeUntil(this.docChangeNotifier),
finalize(() => onComplete?.())
) )
.subscribe({ .subscribe({
next: (doc) => { next: (doc) => {
@@ -1334,6 +1342,7 @@ export class DocumentDetailComponent
} }
save(close: boolean = false, savedCallback: () => void = null) { save(close: boolean = false, savedCallback: () => void = null) {
if (this.zoneOcrRunning()) return
this.networkActive.set(true) this.networkActive.set(true)
;(document.activeElement as HTMLElement)?.dispatchEvent(new Event('change')) ;(document.activeElement as HTMLElement)?.dispatchEvent(new Event('change'))
this.documentsService this.documentsService
@@ -1454,7 +1463,12 @@ export class DocumentDetailComponent
.subscribe((closed) => { .subscribe((closed) => {
if (!closed) return if (!closed) return
if (closedCallback) closedCallback() if (closedCallback) closedCallback()
if (this.documentListViewService.activeSavedViewId) { if (
this.componentRouterService.getComponentURLBefore() === 'dashboard'
) {
// e.g. opened from a dashboard widget
this.router.navigate(['dashboard'])
} else if (this.documentListViewService.activeSavedViewId) {
this.router.navigate([ this.router.navigate([
'view', 'view',
this.documentListViewService.activeSavedViewId, this.documentListViewService.activeSavedViewId,
@@ -1554,6 +1568,107 @@ export class DocumentDetailComponent
}) })
} }
/** Runs every template that matches, confirming only over unsaved changes. */
runZoneOcr() {
if (this.networkActive()) return
const documentId = this.documentId()
if (!this.hasLocalEdits(this.document())) {
this.executeZoneOcr(documentId, null)
return
}
const modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Run Zone OCR`
modal.componentInstance.messageBold = $localize`You have unsaved changes.`
modal.componentInstance.message = $localize`Running Zone OCR will discard them.`
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Run Zone OCR`
modal.componentInstance.confirmClicked
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.close()
if (this.documentId() === documentId && !this.networkActive()) {
this.executeZoneOcr(documentId, null)
}
})
}
/** Opens the dialog to pick which template to run. */
runZoneOcrWithOptions() {
if (this.networkActive()) return
const documentId = this.documentId()
const modal = this.modalService.open(ZoneOcrConfirmDialogComponent, {
backdrop: 'static',
})
const dialog = modal.componentInstance as ZoneOcrConfirmDialogComponent
dialog.title = $localize`Run Zone OCR`
dialog.btnCaption = $localize`Run Zone OCR`
if (this.hasLocalEdits(this.document())) {
dialog.messageBold = $localize`You have unsaved changes.`
dialog.message = $localize`Running Zone OCR will discard them.`
dialog.btnClass = 'btn-warning'
} else {
dialog.message = $localize`Fields filled by the template will be overwritten.`
}
dialog.confirmClicked
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.close()
if (this.documentId() === documentId && !this.networkActive()) {
this.executeZoneOcr(documentId, dialog.template)
}
})
}
private executeZoneOcr(documentId: number, templateId: number | null) {
this.zoneOcrRunning.set(true)
this.networkActive.set(true)
this.documentsService
.runZoneOcr(documentId, templateId)
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe({
next: (response) => {
if (
this.documentId() !== documentId ||
response.status === 'skipped'
) {
this.finishZoneOcr()
if (response.status === 'skipped') {
this.toastService.showInfo(
$localize`No OCR template applies to this document.`
)
}
return
}
this.toastService.showInfo(this.zoneOcrSummary(response.results))
this.docChangeNotifier.next(documentId)
this.loadDocument(documentId, true, () => this.finishZoneOcr())
},
error: (error) => {
this.finishZoneOcr()
this.toastService.showError($localize`Error running zone OCR.`, error)
},
})
}
private zoneOcrSummary(results: OcrZoneRunResult[]): string {
if (!results.length) return $localize`Zone OCR found no fields to fill.`
const missed = results
.filter((result) => !`${result.value ?? ''}`.trim())
.map((result) => result.zone)
const filled = results.length - missed.length
return missed.length
? $localize`Filled ${filled} of ${results.length} fields. No value found for: ${missed.join(', ')}`
: $localize`Filled ${filled} of ${results.length} fields.`
}
private finishZoneOcr() {
this.zoneOcrRunning.set(false)
this.networkActive.set(false)
this.flushPendingIncomingUpdate()
}
private getSelectedNonLatestVersionId(): number | null { private getSelectedNonLatestVersionId(): number | null {
const versions = this.document()?.versions ?? [] const versions = this.document()?.versions ?? []
if (!versions.length || !this.selectedVersionId()) { if (!versions.length || !this.selectedVersionId()) {
@@ -98,6 +98,11 @@
<button ngbDropdownItem (click)="mergeSelectedAsVersions()" [disabled]="!userOwnsAll || !userCanEditAll || !userCanDelete || list.allSelected || list.selectedCount < 2"> <button ngbDropdownItem (click)="mergeSelectedAsVersions()" [disabled]="!userOwnsAll || !userCanEditAll || !userCanDelete || list.allSelected || list.selectedCount < 2">
<pngx-bs-icon name="journal-bookmark-fill" class="me-2"></pngx-bs-icon><ng-container i18n>Merge as versions</ng-container> <pngx-bs-icon name="journal-bookmark-fill" class="me-2"></pngx-bs-icon><ng-container i18n>Merge as versions</ng-container>
</button> </button>
@if (permissionService.currentUserCan(PermissionAction.View, PermissionType.OcrTemplate)) {
<button ngbDropdownItem (click)="runZoneOcrSelected()" [disabled]="!userCanEditAll">
<pngx-bs-icon name="bounding-box-circles" class="me-2"></pngx-bs-icon><ng-container i18n>Run Zone OCR</ng-container>
</button>
}
</div> </div>
</div> </div>
</div> </div>
@@ -1285,6 +1285,53 @@ describe('BulkEditorComponent', () => {
) // listAllFilteredIds ) // listAllFilteredIds
}) })
it('should support run zone ocr', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
.spyOn(documentListViewService, 'documents', 'get')
.mockReturnValue([{ id: 3 }, { id: 4 }])
jest
.spyOn(documentListViewService, 'selected', 'get')
.mockReturnValue(new Set([3, 4]))
jest
.spyOn(permissionsService, 'currentUserHasObjectPermissions')
.mockReturnValue(true)
fixture.detectChanges()
component.runZoneOcrSelected()
expect(modal).not.toBeUndefined()
modal.componentInstance.confirm()
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/run_zone_ocr/`
)
req.flush(true)
expect(req.request.body).toEqual({
documents: [3, 4],
})
component.runZoneOcrSelected()
modal.componentInstance.template = 3
modal.componentInstance.confirm()
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/run_zone_ocr/`
)
req.flush(true)
expect(req.request.body).toEqual({
documents: [3, 4],
template: 3,
})
httpTestingController.match(
`${environment.apiBaseUrl}ocr_templates/?page=1&page_size=100000`
) // template list
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
})
it('should support rotate', () => { it('should support rotate', () => {
let modal: NgbModalRef let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0])) modalService.activeInstances.subscribe((m) => (modal = m[0]))
@@ -64,6 +64,7 @@ import { MergeAsVersionsConfirmDialogComponent } from '../../common/confirm-dial
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component' import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component' import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component' import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { ZoneOcrConfirmDialogComponent } from '../../common/confirm-dialog/zone-ocr-confirm-dialog/zone-ocr-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component' import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component' import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component' import { DocumentTypeEditDialogComponent } from '../../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -966,6 +967,30 @@ export class BulkEditorComponent
}) })
} }
runZoneOcrSelected() {
const modal = this.modalService.open(ZoneOcrConfirmDialogComponent, {
backdrop: 'static',
})
const dialog = modal.componentInstance as ZoneOcrConfirmDialogComponent
dialog.title = $localize`Run zone OCR confirm`
dialog.messageBold = $localize`This operation will run zone OCR on ${this.getSelectionSize()} selected document(s).`
dialog.message = $localize`Fields filled by the template will be overwritten.`
dialog.btnClass = 'btn-warning'
dialog.btnCaption = $localize`Proceed`
dialog.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
dialog.buttonsEnabled.set(false)
this.executeDocumentAction(
modal,
this.documentService.runZoneOcrDocuments(
this.getSelectionQuery(),
dialog.template
)
)
})
}
setPermissions() { setPermissions() {
let modal = this.modalService.open(PermissionsDialogComponent, { let modal = this.modalService.open(PermissionsDialogComponent, {
backdrop: 'static', backdrop: 'static',
@@ -0,0 +1,36 @@
@if (zones.length === 0) {
<p class="text-muted" i18n>No zones yet. Drag anywhere on the sample document to create one.</p>
} @else {
<div class="list-group">
@for (zone of zones; track $index; let i = $index) {
<div class="list-group-item list-group-item-action d-flex align-items-center gap-3 position-relative">
<span class="p-2 rounded flex-shrink-0" [style.background-color]="zoneColor(i)" aria-hidden="true"></span>
<div class="flex-grow-1 text-truncate">
<button type="button" class="btn btn-link p-0 fw-semibold text-reset text-decoration-none text-truncate mw-100 stretched-link" (click)="zoneSelected.emit(i)">{{ zone.name }}</button>
@if (!hasField(zone)) {
<div class="small text-warning text-truncate">
<pngx-bs-icon name="exclamation-triangle" class="me-1" aria-hidden="true"></pngx-bs-icon><span i18n>No field selected</span>
</div>
} @else {
<div class="small text-muted text-truncate">
{{ getZoneTargetName(zone) }}
@if (isCombined(zone)) {
<pngx-bs-icon name="braces" class="ms-1" title="Combined with other zones" i18n-title></pngx-bs-icon>
}
</div>
}
</div>
<small class="text-muted text-nowrap">
@if (zone.page === -1) {
<ng-container i18n>Last page</ng-container>
} @else {
<ng-container i18n>Page {{ zone.page ?? 1 }}</ng-container>
}
</small>
<button type="button" class="btn btn-sm btn-link text-danger position-relative z-2" (click)="zoneRemoved.emit(i)" title="Delete zone" i18n-title aria-label="Delete zone" i18n-aria-label>
<pngx-bs-icon name="trash"></pngx-bs-icon>
</button>
</div>
}
</div>
}
@@ -0,0 +1,115 @@
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { CustomField } from 'src/app/data/custom-field'
import {
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTransform,
} from 'src/app/data/ocr-template'
import { OcrTemplateEditorZoneListComponent } from './ocr-template-editor-zone-list.component'
function zone(overrides: Partial<OcrTemplateZone> = {}): OcrTemplateZone {
return {
name: 'Zone 1',
target: OcrZoneTarget.CustomField,
custom_field: 7,
x: 10,
y: 20,
width: 30,
height: 40,
page: 1,
ocr_language: 'eng',
transform: OcrZoneTransform.Strip,
validation_regex: '',
order: 0,
...overrides,
}
}
describe('OcrTemplateEditorZoneListComponent', () => {
let fixture: ComponentFixture<OcrTemplateEditorZoneListComponent>
let component: OcrTemplateEditorZoneListComponent
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [OcrTemplateEditorZoneListComponent, BootstrapIconComponent],
}).compileComponents()
fixture = TestBed.createComponent(OcrTemplateEditorZoneListComponent)
component = fixture.componentInstance
})
it('shows empty state when no zones are defined', () => {
fixture.detectChanges()
expect(fixture.nativeElement.textContent).toContain('No zones yet')
})
it('renders zone name, target, and page', () => {
component.zones = [zone(), zone({ name: 'Zone 2', page: -1 })]
component.customFields = [{ id: 7, name: 'Invoice Number' } as CustomField]
fixture.detectChanges()
const text = fixture.nativeElement.textContent
expect(text).toContain('Zone 1')
expect(text).toContain('Invoice Number')
expect(text).toContain('Page 1')
expect(text).toContain('Last page')
})
it('marks zones that share a field', () => {
component.zones = [
zone(),
zone({ name: 'Zone 2' }),
zone({ name: 'Zone 3', custom_field: 8 }),
]
fixture.detectChanges()
expect(component.isCombined(component.zones[0])).toBe(true)
expect(component.isCombined(component.zones[2])).toBe(false)
expect(
fixture.nativeElement.querySelectorAll(
'[title="Combined with other zones"]'
)
).toHaveLength(2)
})
it('warns only for zones without a field and clears the warning when assigned', () => {
component.zones = [
zone({ custom_field: null }),
zone({ target: OcrZoneTarget.Title, custom_field: null }),
zone(),
]
fixture.detectChanges()
const warnings = () =>
fixture.nativeElement.querySelectorAll('.text-warning')
expect(warnings()).toHaveLength(1)
expect(warnings()[0].textContent).toContain('No field selected')
component.zones[0].custom_field = 7
fixture.componentRef.setInput('zones', [...component.zones])
fixture.detectChanges()
expect(warnings()).toHaveLength(0)
})
it('emits select and remove events', () => {
component.zones = [zone()]
const selectSpy = jest.spyOn(component.zoneSelected, 'emit')
const removeSpy = jest.spyOn(component.zoneRemoved, 'emit')
fixture.detectChanges()
const buttons = fixture.nativeElement.querySelectorAll('button')
buttons[0].click()
buttons[1].click()
expect(selectSpy).toHaveBeenCalledWith(0)
expect(removeSpy).toHaveBeenCalledWith(0)
})
it('does not label an unknown built-in target', () => {
expect(
component.getZoneTargetName(zone({ target: 'unknown' as OcrZoneTarget }))
).toBe('')
})
})
@@ -0,0 +1,50 @@
import { Component, EventEmitter, Input, Output } from '@angular/core'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { CustomField } from 'src/app/data/custom-field'
import {
getZoneFieldKey,
OCR_BUILTIN_TARGETS,
OcrTemplateZone,
OcrZoneTarget,
} from 'src/app/data/ocr-template'
import { getZoneColor } from '../zone-geometry'
@Component({
selector: 'pngx-ocr-template-zone-list',
imports: [BootstrapIconComponent],
templateUrl: './ocr-template-editor-zone-list.component.html',
})
export class OcrTemplateEditorZoneListComponent {
@Input() zones: OcrTemplateZone[] = []
@Input() customFields: CustomField[] = []
@Output() public readonly zoneSelected = new EventEmitter<number>()
@Output() public readonly zoneRemoved = new EventEmitter<number>()
public readonly zoneColor = getZoneColor
hasField(zone: OcrTemplateZone): boolean {
return getZoneFieldKey(zone) !== null
}
isCombined(zone: OcrTemplateZone): boolean {
const key = getZoneFieldKey(zone)
return (
key !== null &&
this.zones.filter((z) => getZoneFieldKey(z) === key).length > 1
)
}
getZoneTargetName(zone: OcrTemplateZone): string {
if (zone.target === OcrZoneTarget.CustomField) {
return (
this.customFields.find((field) => field.id === zone.custom_field)
?.name ?? ''
)
}
return (
OCR_BUILTIN_TARGETS.find((target) => target.id === zone.target)?.name ??
''
)
}
}
@@ -0,0 +1,313 @@
<div class="modal-header">
<h4 class="modal-title" id="ocr-template-editor-title">{{ dialogTitle }}</h4>
@if (template.id) {
<span class="badge bg-primary text-primary-text-contrast ms-2">ID: {{ template.id }}</span>
}
<button type="button" class="btn-close" (click)="activeModal.dismiss()" [disabled]="saving()" aria-label="Close" i18n-aria-label></button>
</div>
<div class="modal-body d-flex flex-column">
<div class="row gx-3 editor-layout flex-grow-1 flex-lg-nowrap" [attr.inert]="saving() ? '' : null" [attr.aria-busy]="saving()">
<div class="col-lg-4 mb-3 mb-lg-0 overflow-auto">
<ul ngbNav #nav="ngbNav" [(activeId)]="activeTab" class="nav-underline flex-nowrap flex-md-wrap overflow-auto">
<li ngbNavItem="template">
<a ngbNavLink i18n>Template</a>
<ng-template ngbNavContent>
<div class="row mb-3">
<div class="col-9">
<pngx-input-text [(ngModel)]="template.name" title="Template name" i18n-title></pngx-input-text>
</div>
<div class="col-3">
<pngx-input-switch [(ngModel)]="template.enabled" title="Enabled" i18n-title></pngx-input-switch>
</div>
</div>
<pngx-input-select [(ngModel)]="template.matching_algorithm" [items]="matchingAlgorithms" title="Matching algorithm" i18n-title></pngx-input-select>
@if (patternRequired) {
<pngx-input-text [(ngModel)]="template.match" title="Matching pattern" i18n-title></pngx-input-text>
<pngx-input-check [(ngModel)]="template.is_insensitive" title="Case insensitive" i18n-title></pngx-input-check>
}
<div *pngxIfOwner="object">
<hr />
<pngx-permissions-form [users]="users()" [accordion]="true" [(ngModel)]="permissionsForm"></pngx-permissions-form>
</div>
</ng-template>
</li>
<li ngbNavItem="zones">
<a ngbNavLink><ng-container i18n>Zones</ng-container> <span class="badge bg-primary ms-2">{{ template.zones.length }}</span></a>
<ng-template ngbNavContent>
@if (selectedZone; as zone) {
<div class="d-flex align-items-center gap-2 mb-3">
<button type="button" class="btn btn-sm btn-link px-0 me-auto" (click)="deselectZone()">
<pngx-bs-icon name="arrow-left" class="me-1"></pngx-bs-icon><ng-container i18n>All zones</ng-container>
</button>
<button type="button" class="btn btn-sm btn-outline-primary" (click)="testZone()" [disabled]="!previewDocId || zoneTesting">
@if (zoneTesting) {
<span class="spinner-border spinner-border-sm me-1" aria-hidden="true"></span>
} @else {
<pngx-bs-icon name="play-fill" class="me-1"></pngx-bs-icon>
}
<ng-container i18n>Test</ng-container>
</button>
<button type="button" class="btn btn-sm btn-outline-danger" (click)="removeZone(selectedZoneIndex)" title="Delete zone" i18n-title aria-label="Delete zone" i18n-aria-label>
<pngx-bs-icon name="trash"></pngx-bs-icon>
</button>
</div>
@if (zoneTestResult; as result) {
@if (result.error) {
<div class="alert alert-warning py-2 mb-3 small" role="alert">{{ result.error }}</div>
} @else {
<dl class="row small border rounded mx-0 py-2 mb-3">
<dt class="col-4" i18n>OCR text</dt>
<dd class="col-8">
@if (result.raw_text) {
<code>{{ result.raw_text }}</code>
} @else {
<span class="text-muted" i18n>Nothing detected</span>
}
</dd>
<dt class="col-4" i18n>Value</dt>
<dd class="col-8" [class.mb-0]="!result.regex">
@if (result.value) {
<code>{{ result.value }}</code>
} @else {
<span class="text-muted" i18n>Empty</span>
}
</dd>
@if (result.regex) {
<dt class="col-4" i18n>Validation</dt>
<dd class="col-8 mb-0">
@if (result.regex_match) {
<span class="badge bg-success" i18n>Regex matches</span>
} @else {
<span class="badge bg-danger" i18n>Regex does not match</span>
}
</dd>
}
</dl>
}
}
<pngx-input-text [ngModel]="zone.name" (ngModelChange)="renameZone(zone, $event)" title="Name" i18n-title></pngx-input-text>
<pngx-input-select
[items]="fieldItems()"
[ngModel]="zoneFieldValue(zone)"
(ngModelChange)="setZoneField(zone, $event)"
(createNew)="createField($event)"
title="Field"
i18n-title
placeholder="Choose a field"
i18n-placeholder
hint="Where the extracted value is written."
i18n-hint
></pngx-input-select>
@if (isFieldShared(zone)) {
<pngx-input-text
[ngModel]="getCombineFormat(zone)"
(ngModelChange)="setCombineFormat(zone, $event)"
[placeholder]="defaultCombineFormat(zone)"
title="Combined value"
i18n-title
hint="Several zones write to this field. Set the format or leave empty to join them with spaces."
i18n-hint
></pngx-input-text>
<div class="d-flex flex-wrap gap-1 mt-n2 mb-3">
@for (z of zonesForField(zone); track z) {
<button type="button" class="btn btn-sm btn-outline-secondary font-monospace" (click)="insertCombineToken(zone, z)">{{ '{' + z.name + '}' }}</button>
}
</div>
}
<div class="row align-items-end">
<div class="col">
@if (!isLastPageZone(zone)) {
<pngx-input-number [(ngModel)]="zone.page" [showAdd]="false" title="Page" i18n-title></pngx-input-number>
}
</div>
<div class="col-auto">
<pngx-input-switch [ngModel]="isLastPageZone(zone)" (ngModelChange)="setLastPageZone(zone, $event)" [horizontal]="true" title="Last page" i18n-title></pngx-input-switch>
</div>
</div>
<h6 class="border-top pt-3 mt-1 mb-3" i18n>Processing</h6>
<pngx-input-select
[items]="languageOptions"
[multiple]="true"
[ngModel]="ocrLanguageArray(zone)"
(ngModelChange)="setOcrLanguages(zone, $event)"
title="OCR language"
i18n-title
placeholder="Select languages"
i18n-placeholder
></pngx-input-select>
<pngx-input-select [items]="transformOptions" [(ngModel)]="zone.transform" title="Transform" i18n-title></pngx-input-select>
@if (zone.transform === OcrZoneTransform.Date) {
<pngx-input-text
[(ngModel)]="zone.date_format"
title="Date format"
i18n-title
placeholder="Auto-detect"
i18n-placeholder
hint="Python date codes, e.g. %d.%m.%Y. Leave empty to detect the format automatically."
i18n-hint
></pngx-input-text>
}
<pngx-input-text [(ngModel)]="zone.validation_regex" title="Validation regex" i18n-title placeholder="\d{2}\.\d{2}\.\d{4}" hint="The value must match this pattern to be applied." i18n-hint></pngx-input-text>
} @else {
<pngx-ocr-template-zone-list
[zones]="template.zones"
[customFields]="customFields()"
(zoneSelected)="selectZone($event)"
(zoneRemoved)="removeZone($event)"
></pngx-ocr-template-zone-list>
}
</ng-template>
</li>
</ul>
<div [ngbNavOutlet]="nav" class="mt-3"></div>
</div>
<div class="col-lg-8 d-flex flex-column">
<div class="d-flex flex-wrap gap-2 mb-2 flex-shrink-0">
<pngx-input-document-link
class="flex-grow-1"
[ngModel]="sampleDocumentIds"
(ngModelChange)="setSampleDocument($event)"
[minimal]="true"
[multiple]="false"
placeholder="Search for a sample document"
i18n-placeholder
></pngx-input-document-link>
@if (previewDocId) {
<div class="input-group w-auto flex-nowrap">
<input class="form-control flex-grow-0 preview-page-input" type="number" min="1" [max]="previewPageCount()" [(ngModel)]="previewPageDisplay" [disabled]="!pageLoaded()" aria-label="Page" i18n-aria-label />
<span class="input-group-text" i18n>of {{ previewPageCount() ?? '?' }}</span>
</div>
<fieldset class="input-group w-auto flex-nowrap" aria-label="Zoom" i18n-aria-label>
<button type="button" class="btn btn-outline-secondary" aria-label="Zoom out" i18n-aria-label [disabled]="!pageLoaded() || zoom <= 0.5" (click)="zoomOut()"><pngx-bs-icon name="dash"></pngx-bs-icon></button>
<button type="button" class="btn btn-outline-secondary preview-zoom" title="Reset zoom" i18n-title [disabled]="!pageLoaded()" (click)="resetZoom()">{{ zoom | percent: '1.0-0' }}</button>
<button type="button" class="btn btn-outline-secondary" aria-label="Zoom in" i18n-aria-label [disabled]="!pageLoaded() || zoom >= 4" (click)="zoomIn()"><pngx-bs-icon name="plus"></pngx-bs-icon></button>
</fieldset>
}
</div>
<div class="flex-grow-1 overflow-hidden">
<div class="h-100 overflow-auto border" [attr.aria-busy]="previewDocId && !pageLoaded()">
@if (pageFailed()) {
<div class="py-5 px-3 text-center" role="alert">
<pngx-bs-icon name="exclamation-triangle" width="32" height="32" class="text-warning"></pngx-bs-icon>
<p class="my-3" i18n>Could not load this page.</p>
<button type="button" class="btn btn-sm btn-outline-secondary" (click)="setSampleDocument(sampleDocumentIds)"><pngx-bs-icon name="arrow-clockwise" class="me-1"></pngx-bs-icon><span i18n>Retry</span></button>
</div>
} @else if (previewDocId) {
<div class="position-relative" [class.d-none]="!pageLoaded()" [style.width.%]="zoom * 100">
<img
#pageImage
[src]="pageImageUrl"
(load)="onImageLoad()"
(error)="onImageError()"
alt="Sample document page"
i18n-alt
class="d-block w-100"
crossorigin="use-credentials"
/>
@if (pageLoaded()) {
<svg
#zoneOverlay
class="zone-overlay position-absolute top-0 start-0 w-100 h-100"
[attr.viewBox]="overlayViewBox()"
preserveAspectRatio="none"
[style.cursor]="overlayCursor"
(pointerdown)="onOverlayPointerDown($event)"
(pointermove)="onOverlayPointerMove($event)"
(pointerup)="onOverlayPointerUp($event)"
(pointercancel)="onOverlayPointerCancel($event)"
(lostpointercapture)="onOverlayPointerCancel($event)"
aria-label="Zone overlay" i18n-aria-label
>
@for (zone of template.zones; track $index; let i = $index) {
@if (zoneDisplayRect(i); as rect) {
<g>
<rect
class="zone-rect"
[class.zone-rect-selected]="selectedZoneIndex === i"
[attr.x]="rect.x"
[attr.y]="rect.y"
[attr.width]="rect.w"
[attr.height]="rect.h"
[attr.stroke]="zoneColor(i)"
[attr.fill]="zoneColor(i)"
></rect>
<text
class="zone-label fw-semibold pe-none"
[attr.x]="rect.x + overlayUnitSize(6)"
[attr.y]="zoneLabelY(rect)"
[attr.font-size]="overlayFontSize()"
[attr.fill]="zoneColor(i)"
>{{ zoneLabel(zone, i) }}</text>
@if (selectedZoneIndex === i) {
@for (handle of resizeHandles(rect); track handle.handle) {
<rect
class="zone-resize-handle"
[attr.x]="handle.x - overlayHandleSize() / 2"
[attr.y]="handle.y - overlayHandleSize() / 2"
[attr.width]="overlayHandleSize()"
[attr.height]="overlayHandleSize()"
[attr.fill]="zoneColor(i)"
></rect>
}
}
</g>
}
}
@if (drawingRect(); as rect) {
<rect
class="zone-drawing-rect"
[attr.x]="rect.x"
[attr.y]="rect.y"
[attr.width]="rect.w"
[attr.height]="rect.h"
></rect>
}
</svg>
}
</div>
@if (!pageLoaded()) {
<div class="p-5 d-flex justify-content-center align-items-center">
<output class="spinner-border">
<span class="visually-hidden" i18n>Loading page...</span>
</output>
</div>
}
} @else {
<div class="py-5 px-3 text-center text-muted">
<pngx-bs-icon name="file-text" width="32" height="32"></pngx-bs-icon>
<p class="mt-3 mb-0" i18n>No sample document selected.</p>
</div>
}
</div>
</div>
</div>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-outline-secondary" (click)="activeModal.dismiss()" [disabled]="saving()" i18n>Cancel</button>
<button type="button" class="btn btn-primary" (click)="save()" [disabled]="saving() || !isDirty">
@if (saving()) {
<span class="spinner-border spinner-border-sm me-1" aria-hidden="true"></span>
}
<span i18n>Save</span>
</button>
</div>
@@ -0,0 +1,64 @@
:host {
display: flex;
flex-direction: column;
min-height: 0;
}
@media (min-width: 992px) {
.modal-body {
overflow: hidden;
}
.editor-layout {
min-height: 0;
}
}
.preview-page-input {
width: 3.5rem;
}
.preview-zoom {
min-width: 4rem;
}
.zone-overlay {
touch-action: none;
}
.zone-rect,
.zone-drawing-rect {
vector-effect: non-scaling-stroke;
}
.zone-rect {
fill-opacity: 0.2;
stroke-width: 2;
}
.zone-rect-selected {
stroke-width: 3;
}
.zone-label {
font-family: var(--bs-font-sans-serif);
paint-order: stroke;
stroke: #fff;
stroke-linejoin: round;
stroke-width: 4px;
vector-effect: non-scaling-stroke;
}
.zone-resize-handle {
stroke: #fff;
stroke-width: 1;
vector-effect: non-scaling-stroke;
}
.zone-drawing-rect {
fill: var(--bs-primary);
fill-opacity: 0.2;
stroke: var(--bs-primary);
stroke-dasharray: 5 5;
stroke-width: 2;
}
@@ -0,0 +1,875 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ElementRef } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { By } from '@angular/platform-browser'
import { NgbActiveModal, NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { of, Subject, throwError } from 'rxjs'
import { PermissionsFormComponent } from 'src/app/components/common/input/permissions/permissions-form/permissions-form.component'
import { CustomField, CustomFieldDataType } from 'src/app/data/custom-field'
import { Document } from 'src/app/data/document'
import {
MATCH_ANY,
MATCH_AUTO,
MATCH_LITERAL,
MATCH_NONE,
} from 'src/app/data/matching-model'
import {
OcrTemplate,
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTestResult,
OcrZoneTransform,
} from 'src/app/data/ocr-template'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { PermissionsService } from 'src/app/services/permissions.service'
import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service'
import { DocumentService } from 'src/app/services/rest/document.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { OcrTemplateEditorComponent } from './ocr-template-editor.component'
const customFields: CustomField[] = [
{ id: 1, name: 'Invoice number', data_type: CustomFieldDataType.String },
{ id: 2, name: 'Status', data_type: CustomFieldDataType.Select },
]
function zone(overrides: Partial<OcrTemplateZone> = {}): OcrTemplateZone {
return {
name: 'Invoice number',
target: OcrZoneTarget.CustomField,
custom_field: 1,
x: 100,
y: 100,
width: 200,
height: 100,
page: 1,
ocr_language: 'eng',
transform: OcrZoneTransform.Strip,
validation_regex: '',
order: 0,
...overrides,
}
}
function pointer(x: number, y: number, pointerId = 1): PointerEvent {
return Object.assign(
new MouseEvent('pointermove', { clientX: x, clientY: y, button: 0 }),
{ pointerId }
) as PointerEvent
}
describe('OcrTemplateEditorComponent', () => {
let component: OcrTemplateEditorComponent
let fixture: ComponentFixture<OcrTemplateEditorComponent>
let documentService: DocumentService
let templateService: OcrTemplateService
let modalService: NgbModal
const activeModal = { close: jest.fn(), dismiss: jest.fn() }
beforeEach(() => {
TestBed.configureTestingModule({
imports: [BootstrapIconComponent, OcrTemplateEditorComponent],
providers: [
{ provide: NgbActiveModal, useValue: activeModal },
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
})
jest
.spyOn(TestBed.inject(CustomFieldsService), 'listAll')
.mockReturnValue(of({ count: 2, all: [1, 2], results: customFields }))
jest
.spyOn(TestBed.inject(UserService), 'listAll')
.mockReturnValue(of({ count: 0, all: [], results: [] }))
TestBed.inject(PermissionsService).initialize([], { id: 1 } as User)
documentService = TestBed.inject(DocumentService)
jest
.spyOn(documentService, 'get')
.mockReturnValue(of({ id: 5, page_count: 3 }))
templateService = TestBed.inject(OcrTemplateService)
modalService = TestBed.inject(NgbModal)
activeModal.close.mockReset()
fixture = TestBed.createComponent(OcrTemplateEditorComponent)
component = fixture.componentInstance
})
function prepareOverlay() {
const image = document.createElement('img')
Object.defineProperties(image, {
naturalWidth: { value: 1000 },
naturalHeight: { value: 1000 },
clientWidth: { value: 1000 },
})
const overlay = document.createElementNS(
'http://www.w3.org/2000/svg',
'svg'
)
Object.assign(overlay, {
getScreenCTM: () => ({ inverse: () => ({}) }),
createSVGPoint: () => ({
x: 0,
y: 0,
matrixTransform() {
return { x: this.x, y: this.y }
},
}),
setPointerCapture: jest.fn(),
hasPointerCapture: () => true,
releasePointerCapture: jest.fn(),
})
component.imageRef = new ElementRef(image)
component.overlayRef = new ElementRef(overlay)
component.pageLoaded.set(true)
}
function fieldDialog() {
const dialog = {
dialogMode: { set: jest.fn() },
object: null,
succeeded: new Subject<CustomField>(),
failed: new Subject(),
}
jest
.spyOn(modalService, 'open')
.mockReturnValue({ componentInstance: dialog } as any)
return dialog
}
it('updates screen-sized overlay units when the preview image resizes', () => {
let resize: ResizeObserverCallback
const observer = {
observe: jest.fn(),
unobserve: jest.fn(),
disconnect: jest.fn(),
}
const resizeObserver = jest
.spyOn(globalThis, 'ResizeObserver')
.mockImplementation((callback) => {
resize = callback
return observer
})
const image = document.createElement('img')
const ref = new ElementRef(image)
component.pageImage = ref
expect(observer.observe).toHaveBeenCalledWith(image)
resize([], observer)
expect(component.overlayUnitSize(8)).toBe(8)
Object.defineProperties(image, {
naturalWidth: { value: 1000 },
clientWidth: { value: 500 },
})
resize([], observer)
expect(component.overlayUnitSize(8)).toBe(16)
expect(component.overlayFontSize()).toBe(24)
component.pageImage = undefined
expect(observer.disconnect).toHaveBeenCalledTimes(1)
fixture.destroy()
expect(observer.disconnect).toHaveBeenCalledTimes(2)
resizeObserver.mockRestore()
})
it('renders loaded image dimensions, zone labels, and selection handles', () => {
component.object = {
...component.template,
id: 3,
sample_document: 5,
zones: [zone()],
}
component.selectedZoneIndex = 0
fixture.detectChanges()
const image = fixture.nativeElement.querySelector('img')
Object.defineProperties(image, {
naturalWidth: { value: 1000 },
naturalHeight: { value: 1500 },
})
image.dispatchEvent(new Event('load'))
fixture.detectChanges()
expect(component.template).toMatchObject({
source_width: 1000,
source_height: 1500,
})
expect(
fixture.nativeElement
.querySelector('.zone-overlay')
.getAttribute('viewBox')
).toBe('0 0 1000 1500')
expect(
fixture.nativeElement.querySelector('.zone-label').textContent
).toContain('Invoice number')
expect(
fixture.nativeElement.querySelectorAll('.zone-resize-handle')
).toHaveLength(8)
expect(component.zoneLabel(zone({ name: '' }), 2)).toBe('Zone 3')
expect(component.zoneLabelY({ x: 0, y: 0, w: 100, h: 100 })).toBe(14)
})
it('offers the supported language and transform choices', () => {
expect(component.languageOptions.map((option) => option.id)).toContain(
'eng'
)
expect(component.transformOptions.map((option) => option.id)).toEqual(
Object.values(OcrZoneTransform)
)
})
it('clamps page navigation and supports one-based page input', () => {
component.setSampleDocument([5])
component.previewPageDisplay = 2
expect(component.previewPage).toBe(1)
expect(component.previewPageDisplay).toBe(2)
expect(component.pageImageUrl).toContain('/5/1/')
component.pageLoaded.set(true)
component.goToPage(1)
expect(component.pageLoaded()).toBe(true)
component.goToPage(Infinity)
component.goToPage(NaN)
expect(component.previewPage).toBe(1)
component.goToPage(99)
expect(component.previewPage).toBe(2)
expect(component.pageLoaded()).toBe(false)
component.goToPage(-5)
expect(component.previewPage).toBe(0)
component.previewPageCount.set(null)
component.goToPage(4.7)
expect(component.previewPage).toBe(4)
})
it('limits zoom and resets it to the original size', () => {
for (let i = 0; i < 20; i++) component.zoomIn()
expect(component.zoom).toBe(4)
for (let i = 0; i < 20; i++) component.zoomOut()
expect(component.zoom).toBe(0.5)
component.resetZoom()
expect(component.zoom).toBe(1)
})
it('moves last-page zones to the last page and can pin them to a page', () => {
component.setSampleDocument([5])
const selected = zone()
component.template.zones = [selected]
expect(component.isLastPageZone(selected)).toBe(false)
component.setLastPageZone(selected, true)
expect(selected.page).toBe(-1)
expect(component.isLastPageZone(selected)).toBe(true)
component.selectZone(0)
expect(component.previewPage).toBe(2)
component.setLastPageZone(selected, false)
expect(selected.page).toBe(3)
})
it('ignores page-count errors from a previous sample document', () => {
const pending = new Subject<Document>()
jest
.spyOn(documentService, 'get')
.mockReturnValueOnce(pending)
.mockReturnValueOnce(of({ id: 6 }))
component.setSampleDocument([5])
component.setSampleDocument([6])
pending.error(new Error('Old document failed'))
expect(component.pageFailed()).toBe(false)
expect(component.previewPageCount()).toBeNull()
})
it('ignores non-primary, unloaded, and saving pointer gestures', () => {
prepareOverlay()
component.onOverlayPointerDown(
new MouseEvent('pointerdown', { button: 2 }) as PointerEvent
)
expect(component.drawingRect()).toBeNull()
component.pageLoaded.set(false)
component.onOverlayPointerDown(pointer(100, 100))
expect(component.drawingRect()).toBeNull()
component.pageLoaded.set(true)
component.saving.set(true)
component.onOverlayPointerDown(pointer(100, 100))
expect(component.drawingRect()).toBeNull()
})
it('ignores pointer events when the SVG coordinate transform is unavailable', () => {
component.onOverlayPointerDown(pointer(100, 100))
expect(component.drawingRect()).toBeNull()
prepareOverlay()
jest
.spyOn(component.overlayRef.nativeElement, 'getScreenCTM')
.mockReturnValue(null)
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerMove(pointer(200, 200))
expect(component.drawingRect()).toBeNull()
})
it('updates a drawing only for its captured pointer', () => {
prepareOverlay()
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerDown(pointer(500, 500, 2))
component.onOverlayPointerMove(pointer(900, 900, 2))
expect(component.drawingRect()).toEqual({ x: 100, y: 100, w: 0, h: 0 })
component.onOverlayPointerMove(pointer(250, 200))
expect(component.drawingRect()).toEqual({ x: 100, y: 100, w: 150, h: 100 })
component.onOverlayPointerUp(pointer(250, 200))
expect(component.template.zones[0]).toMatchObject({
x: 100,
y: 100,
width: 150,
height: 100,
})
})
it('resizes a selected zone using its corner handle', () => {
prepareOverlay()
const selected = zone()
component.template.zones = [selected]
component.selectZone(0)
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerMove(pointer(50, 75))
component.onOverlayPointerUp(pointer(60, 80))
expect(selected).toMatchObject({ x: 60, y: 80, width: 240, height: 120 })
expect(component.template.zones).toHaveLength(1)
})
it('shows the appropriate cursor over handles, zones, and empty space', () => {
prepareOverlay()
component.template.zones = [zone()]
component.onOverlayPointerMove(pointer(150, 150))
expect(component.overlayCursor).toBe('move')
component.selectZone(0)
component.onOverlayPointerMove(pointer(100, 100))
expect(component.overlayCursor).toBe('nw-resize')
component.onOverlayPointerMove(pointer(500, 500))
expect(component.overlayCursor).toBe('crosshair')
component.template.zones[0].page = 2
component.onOverlayPointerMove(pointer(100, 100))
expect(component.zoneDisplayRect(0)).toBeNull()
expect(component.overlayCursor).toBe('crosshair')
})
it('cancels only the captured pointer and tolerates an already-lost capture', () => {
prepareOverlay()
component.onOverlayPointerCancel(pointer(100, 100))
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerCancel(pointer(150, 150, 2))
expect(component.drawingRect()).not.toBeNull()
jest
.spyOn(component.overlayRef.nativeElement, 'hasPointerCapture')
.mockReturnValue(false)
component.onOverlayPointerCancel(pointer(150, 150))
expect(component.drawingRect()).toBeNull()
expect(
component.overlayRef.nativeElement.releasePointerCapture
).not.toHaveBeenCalled()
expect(component.template.zones).toEqual([])
})
it('preserves the selected zone when an earlier or later zone is removed', () => {
component.template.zones = [zone(), zone(), zone(), zone()]
component.selectZone(1)
const selected = component.selectedZone
component.removeZone(3)
expect(component.selectedZone).toBe(selected)
component.removeZone(0)
expect(component.selectedZoneIndex).toBe(0)
expect(component.selectedZone).toBe(selected)
component.selectZone(99)
expect(component.selectedZone).toBeNull()
})
it('does not test zones without a selection or a sample document', () => {
const test = jest.spyOn(templateService, 'testZone')
component.testZone()
component.template.zones = [zone()]
component.selectZone(0)
component.testZone()
expect(test).not.toHaveBeenCalled()
})
it.each([
{
response: { error: { error: 'OCR unavailable' } },
message: 'OCR unavailable',
},
{ response: new Error('Network failure'), message: 'Test failed' },
])('shows zone-test errors: $message', ({ response, message }) => {
component.template.zones = [zone({ page: undefined })]
component.selectZone(0)
component.setSampleDocument([5])
jest
.spyOn(templateService, 'testZone')
.mockReturnValue(throwError(() => response))
component.testZone()
expect(component.zoneTestResult).toEqual({ error: message })
expect(component.zoneTesting).toBe(false)
expect(templateService.testZone).toHaveBeenCalledWith(
5,
expect.objectContaining({ page: 1 })
)
})
it('switches between built-in targets, custom fields, and an unassigned field', () => {
const selected = zone()
expect(component.zoneFieldValue(selected)).toBe(1)
component.setZoneField(selected, OcrZoneTarget.Title)
expect(component.zoneFieldValue(selected)).toBe(OcrZoneTarget.Title)
expect(selected.custom_field).toBeNull()
component.setZoneField(selected, 7)
expect(selected).toMatchObject({
target: OcrZoneTarget.CustomField,
custom_field: 7,
})
component.setZoneField(selected, null)
expect(component.zoneFieldValue(selected)).toBeNull()
expect(component.zonesForField(selected)).toEqual([])
component.setCombineFormat(selected, 'Ignored')
expect(component.template.combine_formats).toEqual({})
})
it('creates combine formats when absent and leaves implicit formats alone on rename', () => {
const selected = zone({ target: OcrZoneTarget.Title, custom_field: null })
component.template.combine_formats = undefined
component.renameZone(selected, 'Renamed')
expect(selected.name).toBe('Renamed')
expect(component.getCombineFormat(selected)).toBe('')
component.setCombineFormat(selected, '{Renamed} ')
component.insertCombineToken(selected, selected)
expect(component.template.combine_formats).toEqual({
title: '{Renamed} {Renamed}',
})
})
it('saves templates without combine formats', () => {
component.template.combine_formats = undefined
fixture.detectChanges()
expect(component.isDirty).toBe(false)
jest
.spyOn(templateService, 'create')
.mockReturnValue(of(component.template))
component.save()
expect(component.template.combine_formats).toBeUndefined()
})
it('does not assign unsupported custom field types and surfaces creation failures', () => {
component.template.zones = [zone({ custom_field: null })]
component.selectZone(0)
const dialog = fieldDialog()
const error = jest.spyOn(TestBed.inject(ToastService), 'showError')
component.createField('Status')
expect(dialog.object).toEqual({ name: 'Status' })
dialog.succeeded.next(customFields[1])
expect(component.selectedZone.custom_field).toBeNull()
const failure = new Error('Could not save')
dialog.failed.next(failure)
expect(error).toHaveBeenCalledWith('Error saving field.', failure)
})
it('refreshes field choices when a field is created without a selected zone', () => {
const dialog = fieldDialog()
component.createField('Total')
dialog.succeeded.next(customFields[0])
expect(component.customFields()).toEqual(customFields)
expect(component.selectedZone).toBeNull()
})
it('should edit a copy of the template and load its sample document', () => {
const original: OcrTemplate = {
...component.template,
id: 3,
name: 'Invoices',
match: 'Northwind',
matching_algorithm: MATCH_LITERAL,
sample_document: 5,
zones: [zone()],
combine_formats: { '1': '{Invoice number}' },
}
component.object = original
fixture.detectChanges()
expect(component.dialogTitle).toEqual('Edit OCR Template')
expect(component.previewDocId).toEqual(5)
expect(component.previewPageCount()).toEqual(3)
expect(component.pageImageUrl).toContain('document-page-image/5/0/')
expect(component.template.match).toEqual('Northwind')
expect(component.isDirty).toBeFalsy()
component.template.zones[0].x = 500
component.template.combine_formats['1'] = 'Edited'
expect(original.zones[0].x).toEqual(100)
expect(original.combine_formats['1']).toEqual('{Invoice number}')
})
it('should offer every matching algorithm except automatic', () => {
expect(component.matchingAlgorithms.map((a) => a.id)).not.toContain(
MATCH_AUTO
)
expect(component.template.matching_algorithm).toEqual(MATCH_ANY)
expect(component.patternRequired).toBeTruthy()
component.template.matching_algorithm = MATCH_NONE
expect(component.patternRequired).toBeFalsy()
})
it('should initialize new template permissions from saved defaults', () => {
const defaults = {
[SETTINGS_KEYS.DEFAULT_PERMS_OWNER]: 1,
[SETTINGS_KEYS.DEFAULT_PERMS_VIEW_USERS]: [2],
[SETTINGS_KEYS.DEFAULT_PERMS_VIEW_GROUPS]: [3],
[SETTINGS_KEYS.DEFAULT_PERMS_EDIT_USERS]: [],
[SETTINGS_KEYS.DEFAULT_PERMS_EDIT_GROUPS]: [4],
}
jest
.spyOn(TestBed.inject(SettingsService), 'get')
.mockImplementation((key) => defaults[key])
fixture.detectChanges()
expect(component.permissionsForm).toEqual({
owner: 1,
set_permissions: {
view: { users: [2], groups: [3] },
change: { users: [], groups: [4] },
},
})
expect(
fixture.debugElement.query(By.directive(PermissionsFormComponent))
).toBeTruthy()
expect(component.isDirty).toBe(false)
const create = jest
.spyOn(templateService, 'create')
.mockReturnValue(of({ ...component.template, id: 3 }))
component.save()
expect(create).toHaveBeenCalledWith(
expect.objectContaining(component.permissionsForm)
)
})
it('should copy existing permissions and track changes without mutating the list item', () => {
component.object = {
...component.template,
id: 3,
owner: 1,
permissions: {
view: { users: [2], groups: [] },
change: { users: [], groups: [3] },
},
}
fixture.detectChanges()
expect(component.permissionsForm.owner).toBe(1)
expect(component.isDirty).toBe(false)
component.permissionsForm.set_permissions.view.users.push(4)
expect(component.isDirty).toBe(true)
expect(component.object.permissions.view.users).toEqual([2])
const update = jest
.spyOn(templateService, 'update')
.mockReturnValue(of(component.template))
component.save()
expect(update).toHaveBeenCalledWith(
expect.objectContaining({
owner: 1,
set_permissions: {
view: { users: [2, 4], groups: [] },
change: { users: [], groups: [3] },
},
})
)
})
it('should hide sharing controls and omit permission changes for a shared editor', () => {
component.object = {
...component.template,
id: 3,
owner: 2,
user_can_change: true,
}
fixture.detectChanges()
expect(
fixture.debugElement.query(By.directive(PermissionsFormComponent))
).toBeNull()
component.permissionsForm = {
owner: 1,
set_permissions: { view: { users: [1] } },
}
const update = jest
.spyOn(templateService, 'update')
.mockReturnValue(of(component.template))
component.save()
expect(update.mock.calls[0][0].owner).toBe(2)
expect(update.mock.calls[0][0]).not.toHaveProperty('set_permissions')
})
it('should keep only the last picked sample document', () => {
component.setSampleDocument([5, 6])
expect(component.sampleDocumentIds).toEqual([6])
component.setSampleDocument([])
expect(component.previewDocId).toBeNull()
expect(component.pageImageUrl).toBeNull()
})
it('should ignore a stale sample document response', () => {
const pending = new Subject<Document>()
jest.spyOn(documentService, 'get').mockReturnValueOnce(pending)
component.setSampleDocument([5])
component.setSampleDocument([6])
pending.next({ id: 5, page_count: 99 })
expect(component.previewPageCount()).toEqual(3)
})
it('should show a failed page and retry it', () => {
jest
.spyOn(documentService, 'get')
.mockReturnValueOnce(throwError(() => new Error('offline')))
component.setSampleDocument([5])
expect(component.pageFailed()).toBeTruthy()
component.setSampleDocument(component.sampleDocumentIds)
expect(component.pageFailed()).toBeFalsy()
component.onImageError()
expect(component.pageFailed()).toBeTruthy()
})
it('should support create and update', () => {
const createSpy = jest
.spyOn(templateService, 'create')
.mockReturnValue(of({ ...component.template, id: 3 }))
const updateSpy = jest
.spyOn(templateService, 'update')
.mockReturnValue(of(component.template))
const succeededSpy = jest.spyOn(component.succeeded, 'emit')
component.save()
expect(createSpy).toHaveBeenCalled()
expect(succeededSpy).toHaveBeenCalledWith({ ...component.template, id: 3 })
expect(activeModal.close).toHaveBeenCalled()
component.saving.set(false)
component.template.id = 3
component.save()
expect(updateSpy).toHaveBeenCalledWith(component.template)
})
it('should emit failed and allow editing again after a failed save', () => {
prepareOverlay()
component.template.zones = [zone()]
const pending = new Subject<OcrTemplate>()
jest.spyOn(templateService, 'create').mockReturnValue(pending)
const failedSpy = jest.spyOn(component.failed, 'emit')
component.onOverlayPointerDown(pointer(150, 150))
component.save()
component.save()
expect(templateService.create).toHaveBeenCalledTimes(1)
component.onOverlayPointerMove(pointer(250, 250))
component.onOverlayPointerUp(pointer(250, 250))
expect(component.template.zones[0].x).toEqual(100)
pending.error(new Error('Save failed'))
expect(failedSpy).toHaveBeenCalled()
expect(component.saving()).toBeFalsy()
component.onOverlayPointerDown(pointer(150, 150))
component.onOverlayPointerUp(pointer(250, 250))
expect(component.template.zones[0].x).toEqual(200)
})
it('should track unsaved changes', () => {
fixture.detectChanges()
expect(component.isDirty).toBeFalsy()
component.template.combine_formats = { '1': '' }
expect(component.isDirty).toBeFalsy()
component.template.name = 'Invoices'
expect(component.isDirty).toBeTruthy()
component.template.name = ''
expect(component.isDirty).toBeFalsy()
component.setSampleDocument([5])
expect(component.isDirty).toBeTruthy()
})
it('should draw a zone with no field in empty space', () => {
prepareOverlay()
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerUp(pointer(1200, 250))
expect(component.template.zones[0]).toMatchObject({
x: 100,
y: 100,
width: 900,
height: 150,
custom_field: null,
})
expect(component.selectedZoneIndex).toEqual(0)
})
it('should clear the selection on a click in empty space', () => {
prepareOverlay()
component.template.zones = [zone()]
component.selectZone(0)
component.onOverlayPointerDown(pointer(600, 600))
component.onOverlayPointerUp(pointer(602, 601))
expect(component.template.zones).toHaveLength(1)
expect(component.selectedZoneIndex).toBeNull()
})
it('should move a zone when dragging on it', () => {
prepareOverlay()
component.template.zones = [zone()]
component.onOverlayPointerDown(pointer(150, 150))
component.onOverlayPointerUp(pointer(250, 200))
expect(component.template.zones).toHaveLength(1)
expect(component.template.zones[0]).toMatchObject({ x: 200, y: 150 })
})
it('should cancel a drawing on escape and ignore other pointers', () => {
prepareOverlay()
component.onOverlayPointerDown(pointer(100, 100))
component.onOverlayPointerUp(pointer(300, 200, 2))
expect(component.drawingRect()).not.toBeNull()
component.cancelDrawing()
component.onOverlayPointerUp(pointer(300, 200))
expect(component.template.zones).toEqual([])
})
it('should open a zone in the zones tab and go back to the list', () => {
component.template.zones = [zone(), zone()]
component.selectZone(1)
expect(component.activeTab).toEqual('zones')
expect(component.selectedZone).toBe(component.template.zones[1])
component.deselectZone()
expect(component.selectedZone).toBeNull()
component.selectZone(0)
component.removeZone(0)
expect(component.template.zones).toHaveLength(1)
expect(component.selectedZone).toBeNull()
})
it('should only offer custom fields OCR can write to', () => {
fixture.detectChanges()
expect(component.fieldItems().map((item) => item.id)).toEqual([
OcrZoneTarget.Title,
OcrZoneTarget.Asn,
OcrZoneTarget.Created,
1,
])
})
it('should create a custom field and assign it to the zone', () => {
const succeeded = new Subject<CustomField>()
const dialog = {
dialogMode: { set: jest.fn() },
object: null,
succeeded,
failed: new Subject(),
}
jest
.spyOn(modalService, 'open')
.mockReturnValue({ componentInstance: dialog } as any)
component.template.zones = [zone({ custom_field: null })]
component.selectZone(0)
component.createField()
expect(dialog.object).toEqual({ name: 'Invoice number' })
succeeded.next(customFields[0])
expect(component.template.zones[0].custom_field).toEqual(1)
})
it('should build and prune combine formats', () => {
const first = zone({ name: 'First' })
const second = zone({ name: 'Second' })
const other = zone({ name: 'Other', custom_field: 2 })
component.template.zones = [first, second, other]
component.template.combine_formats = { '2': '{Other}' }
expect(component.isFieldShared(first)).toBeTruthy()
expect(component.isFieldShared(other)).toBeFalsy()
expect(component.defaultCombineFormat(first)).toEqual('{First} {Second}')
component.insertCombineToken(first, second)
component.insertCombineToken(first, first)
expect(component.getCombineFormat(second)).toEqual('{Second} {First}')
jest.spyOn(templateService, 'create').mockReturnValue(of(null))
component.save()
expect(component.template.combine_formats).toEqual({
'1': '{Second} {First}',
})
})
it('should update exact tokens when a zone is renamed', () => {
const first = zone({ name: 'First [value]' })
component.template.zones = [first, zone({ name: 'Second' })]
component.template.combine_formats = {
'1': 'First [value]: {First [value]} / { First [value] } / {Second} / {First [value] extra}',
'2': '{First [value]}',
}
component.renameZone(first, '')
component.renameZone(first, 'Invoice number')
expect(first.name).toEqual('Invoice number')
expect(component.template.combine_formats).toEqual({
'1': 'First [value]: {Invoice number} / {Invoice number} / {Second} / {First [value] extra}',
'2': '{First [value]}',
})
})
it('should keep the language array stable until the value changes', () => {
const selected = zone({ ocr_language: 'deu+eng' })
const languages = component.ocrLanguageArray(selected)
expect(languages).toEqual(['deu', 'eng'])
expect(component.ocrLanguageArray(selected)).toBe(languages)
component.setOcrLanguages(selected, ['fra'])
expect(selected.ocr_language).toEqual('fra')
expect(component.ocrLanguageArray(selected)).toEqual(['fra'])
component.setOcrLanguages(selected, null)
expect(component.ocrLanguageArray(selected)).toEqual([])
})
it('should hide a test result once the zone changes', () => {
const selected = zone()
component.template.zones = [selected]
component.selectZone(0)
component.setSampleDocument([5])
jest
.spyOn(templateService, 'testZone')
.mockReturnValue(of({ value: '123' }))
component.testZone()
expect(component.zoneTestResult.value).toEqual('123')
selected.transform = OcrZoneTransform.Numeric
expect(component.zoneTestResult).toBeNull()
})
it('should drop a pending test when the zone or sample document changes', () => {
const pending = new Subject<OcrZoneTestResult>()
component.template.zones = [zone(), zone()]
component.selectZone(0)
component.setSampleDocument([5])
jest.spyOn(templateService, 'testZone').mockReturnValue(pending)
component.testZone()
expect(component.zoneTesting).toBeTruthy()
component.selectZone(1)
pending.next({ value: 'Old zone' })
expect(component.zoneTestResult).toBeNull()
component.testZone()
component.setSampleDocument([])
pending.next({ value: 'Old document' })
expect(component.zoneTestResult).toBeNull()
})
})
@@ -0,0 +1,963 @@
import { CommonModule } from '@angular/common'
import {
Component,
computed,
ElementRef,
EventEmitter,
HostListener,
inject,
Input,
OnDestroy,
OnInit,
Output,
signal,
ViewChild,
} from '@angular/core'
import { FormsModule } from '@angular/forms'
import {
NgbActiveModal,
NgbModal,
NgbNavModule,
} from '@ng-bootstrap/ng-bootstrap'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { Subject, takeUntil } from 'rxjs'
import { CustomFieldEditDialogComponent } from 'src/app/components/common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
import { EditDialogMode } from 'src/app/components/common/edit-dialog/edit-dialog.component'
import { DocumentLinkComponent } from 'src/app/components/common/input/document-link/document-link.component'
import { CheckComponent } from 'src/app/components/common/input/check/check.component'
import { NumberComponent } from 'src/app/components/common/input/number/number.component'
import {
PermissionsFormComponent,
PermissionsFormObject,
} from 'src/app/components/common/input/permissions/permissions-form/permissions-form.component'
import { SelectComponent } from 'src/app/components/common/input/select/select.component'
import { SwitchComponent } from 'src/app/components/common/input/switch/switch.component'
import { TextComponent } from 'src/app/components/common/input/text/text.component'
import { CustomField } from 'src/app/data/custom-field'
import {
MATCH_ANY,
MATCH_AUTO,
MATCH_NONE,
MATCHING_ALGORITHMS,
} from 'src/app/data/matching-model'
import {
DEFAULT_OCR_ZONE_LANGUAGE,
getZoneFieldKey,
OCR_BUILTIN_TARGETS,
OCR_SUPPORTED_FIELD_TYPES,
OcrTemplate,
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTestResult,
OcrZoneTransform,
ZoneTestRequest,
} from 'src/app/data/ocr-template'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { IfOwnerDirective } from 'src/app/directives/if-owner.directive'
import { PermissionsService } from 'src/app/services/permissions.service'
import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service'
import { DocumentService } from 'src/app/services/rest/document.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { OcrTemplateEditorZoneListComponent } from './ocr-template-editor-zone-list/ocr-template-editor-zone-list.component'
import {
DisplayRect,
DrawingRect,
findHandleAt,
findZoneAt,
getResizeHandles,
getZoneColor,
getZoneDisplayRect,
getZonePage,
HANDLE_SIZE,
isZoneOnPage,
MIN_ZONE_SIZE,
MoveStart,
moveZone,
Point,
ResizeHandle,
resizeZone,
} from './zone-geometry'
type ActiveTab = 'template' | 'zones'
type ZoneFieldSelection = OcrZoneTarget | number | null
type OverlayInteraction =
| { kind: 'idle' }
| { kind: 'drawing'; pointerId: number; rect: DrawingRect }
| { kind: 'moving'; pointerId: number; zoneIndex: number; start: MoveStart }
| {
kind: 'resizing'
pointerId: number
zoneIndex: number
handle: ResizeHandle
}
interface ZoneTest {
zone: OcrTemplateZone
/** The sample document and zone settings the result was produced from. */
key: string
result: OcrZoneTestResult | null
}
export const ZONE_TRANSFORM_OPTIONS = [
{ id: OcrZoneTransform.None, name: $localize`None` },
{ id: OcrZoneTransform.Strip, name: $localize`Strip whitespace` },
{ id: OcrZoneTransform.Uppercase, name: $localize`Uppercase` },
{ id: OcrZoneTransform.Lowercase, name: $localize`Lowercase` },
{ id: OcrZoneTransform.Numeric, name: $localize`Numeric only` },
{
id: OcrZoneTransform.StripPunctuation,
name: $localize`Remove leading/trailing punctuation`,
},
{ id: OcrZoneTransform.Date, name: $localize`Parse date` },
{ id: OcrZoneTransform.QrCode, name: $localize`Read QR/barcode` },
]
export const ZONE_LANGUAGE_OPTIONS = [
{ id: 'eng', name: $localize`English` },
{ id: 'deu', name: $localize`German` },
{ id: 'fra', name: $localize`French` },
{ id: 'ita', name: $localize`Italian` },
{ id: 'spa', name: $localize`Spanish` },
{ id: 'por', name: $localize`Portuguese` },
{ id: 'nld', name: $localize`Dutch` },
]
const TEMPLATE_MATCHING_ALGORITHMS = MATCHING_ALGORITHMS.filter(
(algorithm) => algorithm.id !== MATCH_AUTO
)
const NO_OVERLAY_INTERACTION: OverlayInteraction = { kind: 'idle' }
const RESIZE_CURSOR: Record<ResizeHandle, string> = {
nw: 'nw-resize',
ne: 'ne-resize',
sw: 'sw-resize',
se: 'se-resize',
n: 'n-resize',
s: 's-resize',
w: 'w-resize',
e: 'e-resize',
}
@Component({
selector: 'pngx-ocr-template-editor',
imports: [
DocumentLinkComponent,
NumberComponent,
TextComponent,
CheckComponent,
SelectComponent,
SwitchComponent,
CommonModule,
FormsModule,
NgbNavModule,
BootstrapIconComponent,
OcrTemplateEditorZoneListComponent,
PermissionsFormComponent,
IfOwnerDirective,
],
templateUrl: './ocr-template-editor.component.html',
styleUrls: ['./ocr-template-editor.component.scss'],
})
export class OcrTemplateEditorComponent implements OnInit, OnDestroy {
public readonly activeModal = inject(NgbActiveModal)
private readonly templateService = inject(OcrTemplateService)
private readonly customFieldsService = inject(CustomFieldsService)
private readonly documentService = inject(DocumentService)
private readonly toastService = inject(ToastService)
private readonly modalService = inject(NgbModal)
private readonly userService = inject(UserService)
private readonly settingsService = inject(SettingsService)
private readonly permissionsService = inject(PermissionsService)
private readonly destroy$ = new Subject<void>()
private readonly cancelZoneTest$ = new Subject<void>()
@Input() object: OcrTemplate
@Output() public readonly succeeded = new EventEmitter<OcrTemplate>()
@Output() public readonly failed = new EventEmitter()
@ViewChild('zoneOverlay') overlayRef: ElementRef<SVGSVGElement>
imageRef: ElementRef<HTMLImageElement>
private previewResizeObserver: ResizeObserver | null = null
private readonly overlayScale = signal(1)
@ViewChild('pageImage')
set pageImage(ref: ElementRef<HTMLImageElement> | undefined) {
this.previewResizeObserver?.disconnect()
this.imageRef = ref
if (!ref) return
this.previewResizeObserver = new ResizeObserver(() => {
const image = ref.nativeElement
if (image.naturalWidth && image.clientWidth) {
this.overlayScale.set(image.naturalWidth / image.clientWidth)
}
})
this.previewResizeObserver.observe(ref.nativeElement)
}
template: OcrTemplate = {
id: null,
name: '',
match: '',
matching_algorithm: MATCH_ANY,
is_insensitive: true,
sample_document: null,
source_width: 0,
source_height: 0,
enabled: true,
combine_formats: {},
zones: [],
}
public readonly customFields = signal<CustomField[]>([])
public readonly fieldItems = computed<
{ id: ZoneFieldSelection; name: string }[]
>(() => [
...OCR_BUILTIN_TARGETS,
...this.customFields()
.filter((field) => OCR_SUPPORTED_FIELD_TYPES.includes(field.data_type))
.map((field) => ({ id: field.id, name: field.name })),
])
public readonly users = signal<User[]>([])
permissionsForm: PermissionsFormObject = {}
public readonly OcrZoneTransform = OcrZoneTransform
public readonly saving = signal(false)
/** Bound to the document picker, which works with id arrays. */
sampleDocumentIds: number[] = []
previewPage = 0
public readonly previewPageCount = signal<number | null>(null)
public readonly pageLoaded = signal(false)
public readonly pageFailed = signal(false)
zoom = 1
get previewDocId(): number | null {
return this.sampleDocumentIds[0] ?? null
}
get pageImageUrl(): string | null {
return this.previewDocId
? this.templateService.getPageImageUrl(
this.previewDocId,
this.previewPage
)
: null
}
get transformOptions() {
return ZONE_TRANSFORM_OPTIONS
}
get languageOptions() {
return ZONE_LANGUAGE_OPTIONS
}
get matchingAlgorithms() {
return TEMPLATE_MATCHING_ALGORITHMS
}
get patternRequired(): boolean {
return (
this.template.matching_algorithm !== MATCH_AUTO &&
this.template.matching_algorithm !== MATCH_NONE
)
}
public get previewPageDisplay(): number {
return this.previewPage + 1
}
public set previewPageDisplay(value: number) {
this.goToPage(value - 1)
}
activeTab: ActiveTab = 'template'
selectedZoneIndex: number | null = null
private overlayInteraction: OverlayInteraction = NO_OVERLAY_INTERACTION
overlayCursor = 'crosshair'
private readonly zoneTest = signal<ZoneTest | null>(null)
get zoneTestResult(): OcrZoneTestResult | null {
return this.isCurrentZoneTest() ? this.zoneTest().result : null
}
get zoneTesting(): boolean {
return this.isCurrentZoneTest() && this.zoneTest().result === null
}
get selectedZone(): OcrTemplateZone | null {
return this.selectedZoneIndex !== null
? (this.template.zones[this.selectedZoneIndex] ?? null)
: null
}
private initialState = ''
get isDirty(): boolean {
return this.editorState() !== this.initialState
}
private editorState(): string {
const formats = Object.entries(this.template.combine_formats ?? {}).filter(
([, format]) => format
)
return JSON.stringify({
...this.template,
combine_formats: Object.fromEntries(formats),
sample_document: this.previewDocId ?? null,
permissions_form: this.permissionsForm,
})
}
get dialogTitle(): string {
return this.template.id
? $localize`Edit OCR Template`
: $localize`New OCR Template`
}
ngOnInit() {
this.userService
.listAll()
.pipe(takeUntil(this.destroy$))
.subscribe((r) => this.users.set(r.results))
this.customFieldsService
.listAll()
.pipe(takeUntil(this.destroy$))
.subscribe((r) => {
this.customFields.set(r.results)
})
if (this.object) {
this.template = {
...this.object,
zones: this.object.zones.map((zone) => ({ ...zone })),
combine_formats: { ...this.object.combine_formats },
}
if (this.template.sample_document) {
this.setSampleDocument([this.template.sample_document])
}
}
const permissions = this.object?.permissions
this.permissionsForm = this.object
? {
owner: this.object.owner,
set_permissions: {
view: {
users: [...(permissions?.view.users ?? [])],
groups: [...(permissions?.view.groups ?? [])],
},
change: {
users: [...(permissions?.change.users ?? [])],
groups: [...(permissions?.change.groups ?? [])],
},
},
}
: {
owner: this.settingsService.get(SETTINGS_KEYS.DEFAULT_PERMS_OWNER),
set_permissions: {
view: {
users: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_VIEW_USERS
),
groups: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_VIEW_GROUPS
),
},
change: {
users: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_EDIT_USERS
),
groups: this.settingsService.get(
SETTINGS_KEYS.DEFAULT_PERMS_EDIT_GROUPS
),
},
},
}
this.initialState = this.editorState()
}
setSampleDocument(ids: number[]) {
const id = ids.at(-1)
this.sampleDocumentIds = id ? [id] : []
this.previewPage = 0
this.previewPageCount.set(null)
this.resetPage()
if (!id) return
this.documentService
.get(id)
.pipe(takeUntil(this.destroy$))
.subscribe({
next: (doc) => {
if (this.previewDocId !== id) return
this.previewPageCount.set(doc.page_count ?? null)
},
error: () => {
if (this.previewDocId === id) this.pageFailed.set(true)
},
})
}
private resetPage() {
this.stopOverlayInteraction()
this.clearZoneTest()
this.pageLoaded.set(false)
this.pageFailed.set(false)
}
goToPage(page: number) {
if (!Number.isFinite(page)) return
const pageCount = this.previewPageCount()
const max = pageCount ? pageCount - 1 : page
const clamped = Math.max(0, Math.min(Math.trunc(page), max))
if (clamped === this.previewPage) return
this.previewPage = clamped
this.resetPage()
}
zoomIn() {
this.zoom = Math.min(4, Math.round((this.zoom + 0.25) * 100) / 100)
}
zoomOut() {
this.zoom = Math.max(0.5, Math.round((this.zoom - 0.25) * 100) / 100)
}
resetZoom() {
this.zoom = 1
}
zonePage(zone: OcrTemplateZone): number {
return getZonePage(zone, this.previewPage, this.previewPageCount())
}
isLastPageZone(zone: OcrTemplateZone): boolean {
return zone.page === -1
}
setLastPageZone(zone: OcrTemplateZone, lastPage: boolean) {
zone.page = lastPage ? -1 : this.previewPage + 1
}
private isOnCurrentPage(zone: OcrTemplateZone): boolean {
return isZoneOnPage(zone, this.previewPage, this.previewPageCount())
}
onImageLoad() {
this.pageLoaded.set(true)
const img = this.imageRef.nativeElement
this.template.source_width = img.naturalWidth
this.template.source_height = img.naturalHeight
}
onImageError() {
this.stopOverlayInteraction()
this.pageFailed.set(true)
}
@HostListener('document:keydown.escape')
cancelDrawing() {
this.stopOverlayInteraction()
}
onOverlayPointerDown(event: PointerEvent) {
if (this.saving()) return
if (event.button !== 0 || this.overlayInteraction.kind !== 'idle') return
const point = this.svgPointFromEvent(event)
if (!this.pageLoaded() || !point) return
event.preventDefault()
if (this.selectedZoneIndex !== null) {
const handle = this.handleAt(point, this.selectedZoneIndex)
if (handle) {
this.overlayInteraction = {
kind: 'resizing',
pointerId: event.pointerId,
zoneIndex: this.selectedZoneIndex,
handle,
}
this.overlayRef.nativeElement.setPointerCapture(event.pointerId)
return
}
}
const clickedIdx = this.zoneIndexAt(point)
if (clickedIdx !== null) {
this.selectZone(clickedIdx)
const zone = this.template.zones[clickedIdx]
this.overlayInteraction = {
kind: 'moving',
pointerId: event.pointerId,
zoneIndex: clickedIdx,
start: {
mouseX: point.x,
mouseY: point.y,
zoneX: zone.x,
zoneY: zone.y,
},
}
this.overlayRef.nativeElement.setPointerCapture(event.pointerId)
return
}
// Empty space: start drawing a new zone. A click without a drag just clears
// the selection (see onOverlayPointerUp).
this.clearZoneTest()
this.selectedZoneIndex = null
this.overlayInteraction = {
kind: 'drawing',
pointerId: event.pointerId,
rect: {
startX: point.x,
startY: point.y,
endX: point.x,
endY: point.y,
},
}
this.overlayRef.nativeElement.setPointerCapture(event.pointerId)
}
onOverlayPointerMove(event: PointerEvent) {
if (
this.overlayInteraction.kind !== 'idle' &&
this.overlayInteraction.pointerId !== event.pointerId
)
return
const point = this.svgPointFromEvent(event)
if (!point) return
if (this.overlayInteraction.kind === 'resizing') {
resizeZone(
this.template.zones[this.overlayInteraction.zoneIndex],
this.overlayInteraction.handle,
point,
this.imageNaturalSize()
)
return
}
if (this.overlayInteraction.kind === 'moving') {
moveZone(
this.template.zones[this.overlayInteraction.zoneIndex],
point,
this.overlayInteraction.start,
this.imageNaturalSize()
)
return
}
if (this.overlayInteraction.kind === 'drawing') {
this.overlayInteraction.rect.endX = point.x
this.overlayInteraction.rect.endY = point.y
return
}
this.updateOverlayCursor(point)
}
private updateOverlayCursor(point: Point) {
if (this.selectedZoneIndex !== null) {
const handle = this.handleAt(point, this.selectedZoneIndex)
if (handle) {
this.overlayCursor = RESIZE_CURSOR[handle]
return
}
}
this.overlayCursor = this.zoneIndexAt(point) !== null ? 'move' : 'crosshair'
}
onOverlayPointerUp(event: PointerEvent) {
if (this.overlayInteraction.kind === 'idle') return
if (this.overlayInteraction.pointerId !== event.pointerId) return
this.onOverlayPointerMove(event)
const interaction = this.overlayInteraction
this.stopOverlayInteraction()
if (interaction.kind !== 'drawing') return
const rect = this.sourceRectFromDrawing(interaction.rect)
// Ignore tiny accidental clicks.
if (rect.w < MIN_ZONE_SIZE || rect.h < MIN_ZONE_SIZE) {
return
}
this.template.zones.push(this.createZoneFromRect(rect))
this.selectZone(this.template.zones.length - 1)
}
private createZoneFromRect(rect: DisplayRect): OcrTemplateZone {
const imageSize = this.imageNaturalSize()
return {
name: `Zone ${this.template.zones.length + 1}`,
target: OcrZoneTarget.CustomField,
custom_field: null,
x: rect.x,
y: rect.y,
width: rect.w,
height: rect.h,
page: this.previewPageDisplay,
ocr_language: DEFAULT_OCR_ZONE_LANGUAGE,
transform: OcrZoneTransform.Strip,
date_format: '',
validation_regex: '',
order: this.template.zones.length,
zone_source_width: imageSize.width,
zone_source_height: imageSize.height,
}
}
onOverlayPointerCancel(event: PointerEvent) {
if (this.overlayInteraction.kind === 'idle') return
if (this.overlayInteraction.pointerId !== event.pointerId) return
this.stopOverlayInteraction()
}
private stopOverlayInteraction() {
const interaction = this.overlayInteraction
this.overlayInteraction = NO_OVERLAY_INTERACTION
this.overlayCursor = 'crosshair'
const overlay = this.overlayRef?.nativeElement
if (
interaction.kind !== 'idle' &&
overlay?.hasPointerCapture(interaction.pointerId)
) {
overlay.releasePointerCapture(interaction.pointerId)
}
}
drawingRect(): DisplayRect | null {
return this.overlayInteraction.kind === 'drawing'
? this.displayRectFromDrawing(this.overlayInteraction.rect)
: null
}
/** The zone's rectangle on the overlay, or null if it's on another page. */
zoneDisplayRect(zoneIndex: number): DisplayRect | null {
const zone = this.template.zones[zoneIndex]
if (!this.isOnCurrentPage(zone)) return null
return getZoneDisplayRect(zone, this.imageNaturalSize())
}
private handleAt(point: Point, zoneIndex: number): ResizeHandle | null {
const rect = this.zoneDisplayRect(zoneIndex)
return rect ? findHandleAt(point, rect, this.overlayHandleSize()) : null
}
private zoneIndexAt(point: Point): number | null {
return findZoneAt(
point,
this.template.zones,
this.previewPage,
this.previewPageCount(),
this.imageNaturalSize()
)
}
overlayViewBox(): string {
const imageSize = this.imageNaturalSize()
return `0 0 ${imageSize.width} ${imageSize.height}`
}
public readonly zoneColor = getZoneColor
zoneLabel(zone: OcrTemplateZone, index: number): string {
return zone.name || `Zone ${index + 1}`
}
zoneLabelY(rect: DisplayRect): number {
return Math.max(this.overlayUnitSize(14), rect.y - this.overlayUnitSize(4))
}
public readonly resizeHandles = getResizeHandles
overlayHandleSize(): number {
return this.overlayUnitSize(HANDLE_SIZE)
}
overlayFontSize(): number {
return this.overlayUnitSize(12)
}
overlayUnitSize(screenPixels: number): number {
return screenPixels * this.overlayScale()
}
private svgPointFromEvent(event: MouseEvent): Point | null {
const svg = this.overlayRef?.nativeElement
const matrix = svg?.getScreenCTM()
if (!svg || !matrix) return null
const point = svg.createSVGPoint()
point.x = event.clientX
point.y = event.clientY
const svgPoint = point.matrixTransform(matrix.inverse())
const size = this.imageNaturalSize()
return {
x: Math.max(0, Math.min(svgPoint.x, size.width)),
y: Math.max(0, Math.min(svgPoint.y, size.height)),
}
}
private displayRectFromDrawing(rect: DrawingRect): DisplayRect {
return {
x: Math.min(rect.startX, rect.endX),
y: Math.min(rect.startY, rect.endY),
w: Math.abs(rect.endX - rect.startX),
h: Math.abs(rect.endY - rect.startY),
}
}
private sourceRectFromDrawing(rect: DrawingRect): DisplayRect {
const displayRect = this.displayRectFromDrawing(rect)
return {
x: Math.round(displayRect.x),
y: Math.round(displayRect.y),
w: Math.round(displayRect.w),
h: Math.round(displayRect.h),
}
}
private imageNaturalSize() {
const img = this.imageRef.nativeElement
return { width: img.naturalWidth, height: img.naturalHeight }
}
removeZone(index: number) {
this.stopOverlayInteraction()
this.clearZoneTest()
this.template.zones.splice(index, 1)
if (this.selectedZoneIndex === index) {
this.selectedZoneIndex = null
} else if (this.selectedZoneIndex > index) {
this.selectedZoneIndex--
}
}
selectZone(index: number) {
this.selectedZoneIndex = index
this.activeTab = 'zones'
this.clearZoneTest()
const zone = this.template.zones[index]
if (zone) {
this.goToPage(this.zonePage(zone) - 1)
}
}
deselectZone() {
this.selectedZoneIndex = null
this.clearZoneTest()
}
testZone() {
const zone = this.selectedZone
if (!zone || !this.previewDocId) return
this.clearZoneTest()
const test: ZoneTest = { zone, key: this.zoneTestKey(zone), result: null }
this.zoneTest.set(test)
this.templateService
.testZone(this.previewDocId, this.zoneTestRequest(zone))
.pipe(takeUntil(this.destroy$), takeUntil(this.cancelZoneTest$))
.subscribe({
next: (res) => this.zoneTest.set({ ...test, result: res }),
error: (err) =>
this.zoneTest.set({
...test,
result: { error: err.error?.error || $localize`Test failed` },
}),
})
}
private clearZoneTest() {
this.cancelZoneTest$.next()
this.zoneTest.set(null)
}
/** A result is stale once its zone, settings or sample document change. */
private isCurrentZoneTest(): boolean {
const test = this.zoneTest()
return (
test?.zone === this.selectedZone &&
test.key === this.zoneTestKey(test.zone)
)
}
private zoneTestKey(zone: OcrTemplateZone): string {
return JSON.stringify([this.previewDocId, this.zoneTestRequest(zone)])
}
private zoneTestRequest(zone: OcrTemplateZone): ZoneTestRequest {
return {
name: zone.name,
x: zone.x,
y: zone.y,
width: zone.width,
height: zone.height,
page: zone.page ?? 1,
ocr_language: zone.ocr_language,
transform: zone.transform,
date_format: zone.date_format,
validation_regex: zone.validation_regex,
zone_source_width: zone.zone_source_width,
zone_source_height: zone.zone_source_height,
}
}
save() {
if (this.saving()) return
this.stopOverlayInteraction()
this.saving.set(true)
this.pruneCombineFormats()
this.template.sample_document = this.previewDocId
const payload = this.permissionsService.currentUserOwnsObject(this.object)
? { ...this.template, ...this.permissionsForm }
: this.template
const obs = this.template.id
? this.templateService.update(payload)
: this.templateService.create(payload)
obs.pipe(takeUntil(this.destroy$)).subscribe({
next: (saved) => {
this.activeModal.close()
this.succeeded.emit(saved)
},
error: (e) => {
this.saving.set(false)
this.failed.emit(e)
},
})
}
// ng-select needs a stable array between change detection runs, so the
// split language list is cached per zone until ocr_language changes.
private readonly ocrLanguageCache = new WeakMap<
OcrTemplateZone,
{ language: string; languages: string[] }
>()
ocrLanguageArray(zone: OcrTemplateZone): string[] {
const language = zone.ocr_language || ''
const cached = this.ocrLanguageCache.get(zone)
if (cached?.language === language) return cached.languages
const languages = language.split('+').filter(Boolean)
this.ocrLanguageCache.set(zone, { language, languages })
return languages
}
setOcrLanguages(zone: OcrTemplateZone, langs: string[]) {
zone.ocr_language = (langs ?? []).join('+')
}
/** The field select's value: a built-in target, or a custom field id. */
zoneFieldValue(zone: OcrTemplateZone): ZoneFieldSelection {
return zone.target === OcrZoneTarget.CustomField
? zone.custom_field
: zone.target
}
setZoneField(zone: OcrTemplateZone, value: ZoneFieldSelection) {
if (typeof value === 'string') {
zone.target = value
zone.custom_field = null
} else {
zone.target = OcrZoneTarget.CustomField
zone.custom_field = value
}
}
zonesForField(zone: OcrTemplateZone): OcrTemplateZone[] {
const key = getZoneFieldKey(zone)
if (!key) return []
return this.template.zones.filter((z) => getZoneFieldKey(z) === key)
}
isFieldShared(zone: OcrTemplateZone): boolean {
return this.zonesForField(zone).length > 1
}
renameZone(zone: OcrTemplateZone, name: string) {
const oldName = zone.name.trim()
const format = this.getCombineFormat(zone)
if (format) {
this.setCombineFormat(
zone,
format.replace(/\{([^{}]*)\}/g, (token, tokenName: string) =>
tokenName.trim() === oldName ? `{${name}}` : token
)
)
}
zone.name = name
}
getCombineFormat(zone: OcrTemplateZone): string {
return this.template.combine_formats?.[getZoneFieldKey(zone)] ?? ''
}
setCombineFormat(zone: OcrTemplateZone, value: string) {
const key = getZoneFieldKey(zone)
if (!key) return
this.template.combine_formats ??= {}
this.template.combine_formats[key] = value
}
defaultCombineFormat(zone: OcrTemplateZone): string {
return this.zonesForField(zone)
.map((z) => `{${z.name}}`)
.join(' ')
}
insertCombineToken(zone: OcrTemplateZone, tokenZone: OcrTemplateZone) {
const current = this.getCombineFormat(zone)
const sep = current && !current.endsWith(' ') ? ' ' : ''
this.setCombineFormat(zone, `${current}${sep}{${tokenZone.name}}`)
}
/** Drop formats that are empty or whose field no longer has several zones. */
private pruneCombineFormats() {
const formats = this.template.combine_formats
if (!formats) return
const keys = this.template.zones.map(getZoneFieldKey)
for (const [key, format] of Object.entries(formats)) {
if (!format || keys.filter((k) => k === key).length < 2) {
delete formats[key]
}
}
}
createField(name?: string) {
const zone = this.selectedZone
const modal = this.modalService.open(CustomFieldEditDialogComponent)
modal.componentInstance.dialogMode.set(EditDialogMode.CREATE)
modal.componentInstance.object = { name: name || zone?.name }
modal.componentInstance.succeeded
.pipe(takeUntil(this.destroy$))
.subscribe((field: CustomField) => {
this.toastService.showSuccess($localize`Saved field "${field.name}".`)
this.customFieldsService.clearCache()
this.customFieldsService
.listAll()
.pipe(takeUntil(this.destroy$))
.subscribe((r) => {
if (zone && OCR_SUPPORTED_FIELD_TYPES.includes(field.data_type)) {
this.setZoneField(zone, field.id)
}
this.customFields.set(r.results)
})
})
modal.componentInstance.failed
.pipe(takeUntil(this.destroy$))
.subscribe((e) => {
this.toastService.showError($localize`Error saving field.`, e)
})
}
ngOnDestroy() {
this.previewResizeObserver?.disconnect()
this.cancelZoneTest$.complete()
this.destroy$.next()
this.destroy$.complete()
}
}
@@ -0,0 +1,197 @@
import {
OcrTemplateZone,
OcrZoneTarget,
OcrZoneTransform,
} from 'src/app/data/ocr-template'
import {
findHandleAt,
findZoneAt,
getResizeHandles,
getZoneDisplayRect,
getZonePage,
isZoneOnPage,
moveZone,
resizeZone,
} from './zone-geometry'
function zone(overrides: Partial<OcrTemplateZone> = {}): OcrTemplateZone {
return {
name: 'Zone',
target: OcrZoneTarget.CustomField,
custom_field: 1,
x: 100,
y: 200,
width: 300,
height: 400,
page: 1,
ocr_language: 'eng',
transform: OcrZoneTransform.Strip,
validation_regex: '',
order: 0,
...overrides,
}
}
describe('OCR template editor geometry', () => {
it('normalizes zone pages', () => {
expect(getZonePage(zone({ page: 2 }), 0, 5)).toBe(2)
expect(getZonePage(zone({ page: -1 }), 0, 5)).toBe(5)
expect(getZonePage(zone({ page: -1 }), 2, null)).toBe(3)
expect(getZonePage(zone({ page: 0 }), 0, 5)).toBe(1)
expect(getZonePage(zone({ page: undefined }), 0, 5)).toBe(1)
})
it('checks whether a zone is on the current preview page', () => {
expect(isZoneOnPage(zone({ page: 2 }), 1, 5)).toBe(true)
expect(isZoneOnPage(zone({ page: 2 }), 0, 5)).toBe(false)
expect(isZoneOnPage(zone({ page: -1 }), 4, 5)).toBe(true)
})
it('uses natural image coordinates when source dimensions are unset', () => {
expect(
getZoneDisplayRect(zone({ x: 100, y: 200, width: 300, height: 400 }), {
width: 1000,
height: 2000,
})
).toEqual({ x: 100, y: 200, w: 300, h: 400 })
})
it('uses per-zone source dimensions when present', () => {
expect(
getZoneDisplayRect(
zone({
x: 100,
y: 100,
width: 100,
height: 100,
zone_source_width: 1000,
zone_source_height: 1000,
}),
{ width: 500, height: 750 }
)
).toEqual({ x: 50, y: 75, w: 50, h: 75 })
})
it('finds zones from topmost to bottommost on the current page', () => {
const zones = [
zone({ name: 'first', x: 0, y: 0, width: 100, height: 100, page: 1 }),
zone({ name: 'second', x: 0, y: 0, width: 50, height: 50, page: 1 }),
zone({ name: 'third', x: 0, y: 0, width: 50, height: 50, page: 2 }),
]
expect(
findZoneAt({ x: 25, y: 25 }, zones, 0, 2, { width: 100, height: 100 })
).toBe(1)
})
it('finds resize handles around a display rect', () => {
const rect = { x: 10, y: 20, w: 100, h: 200 }
expect(findHandleAt({ x: 10, y: 20 }, rect)).toBe('nw')
expect(findHandleAt({ x: 110, y: 220 }, rect)).toBe('se')
expect(findHandleAt({ x: 60, y: 20 }, rect)).toBe('n')
expect(findHandleAt({ x: 90, y: 160 }, rect)).toBeNull()
expect(findHandleAt({ x: 19, y: 20 }, rect, 8)).toBeNull()
expect(findHandleAt({ x: 19, y: 20 }, rect, 10)).toBe('nw')
})
it('returns no zone when the point is outside every rectangle', () => {
expect(
findZoneAt({ x: 900, y: 900 }, [zone()], 0, 1, {
width: 1000,
height: 1000,
})
).toBeNull()
})
it('resizes north-west edges while keeping the opposite corner fixed', () => {
const z = zone({ x: 100, y: 200, width: 300, height: 400 })
resizeZone(z, 'nw', { x: 50, y: 100 }, { width: 1000, height: 1000 })
expect(z).toMatchObject({ x: 50, y: 100, width: 350, height: 500 })
resizeZone(z, 'nw', { x: 900, y: 900 }, { width: 1000, height: 1000 })
expect(z).toMatchObject({ x: 390, y: 590, width: 10, height: 10 })
})
it('clamps resized north-west edges to the source image bounds', () => {
const z = zone({
x: 50,
y: 50,
width: 100,
height: 100,
zone_source_width: 200,
zone_source_height: 300,
})
resizeZone(z, 'nw', { x: -50, y: -50 }, { width: 400, height: 600 })
expect(z).toMatchObject({ x: 0, y: 0, width: 150, height: 150 })
})
it('uses the rendered handle positions for hit testing', () => {
const rect = { x: 10, y: 20, w: 100, h: 200 }
const handles = getResizeHandles(rect)
expect(handles).toEqual([
{ handle: 'nw', x: 10, y: 20 },
{ handle: 'n', x: 60, y: 20 },
{ handle: 'ne', x: 110, y: 20 },
{ handle: 'w', x: 10, y: 120 },
{ handle: 'e', x: 110, y: 120 },
{ handle: 'sw', x: 10, y: 220 },
{ handle: 's', x: 60, y: 220 },
{ handle: 'se', x: 110, y: 220 },
])
for (const marker of handles) {
expect(findHandleAt(marker, rect)).toBe(marker.handle)
}
})
it('moves zones without leaving source image bounds', () => {
const z = zone({ x: 50, y: 50, width: 100, height: 100 })
moveZone(
z,
{ x: 500, y: 500 },
{ mouseX: 50, mouseY: 50, zoneX: 50, zoneY: 50 },
{ width: 500, height: 500 }
)
expect(z.x).toBe(400)
expect(z.y).toBe(400)
})
it('resizes zones without leaving source image bounds', () => {
const z = zone({
x: 50,
y: 50,
width: 100,
height: 100,
zone_source_width: 200,
zone_source_height: 200,
})
resizeZone(z, 'se', { x: 500, y: 500 }, { width: 500, height: 500 })
expect(z.width).toBe(150)
expect(z.height).toBe(150)
})
it('moves zones in their original source coordinates on a different-size preview', () => {
const z = zone({
x: 50,
y: 50,
width: 100,
height: 100,
zone_source_width: 1000,
zone_source_height: 2000,
})
moveZone(
z,
{ x: 75, y: 75 },
{ mouseX: 25, mouseY: 25, zoneX: 50, zoneY: 50 },
{ width: 500, height: 500 }
)
expect(z.x).toBe(150)
expect(z.y).toBe(250)
})
})
@@ -0,0 +1,204 @@
import { OcrTemplateZone } from 'src/app/data/ocr-template'
export interface DrawingRect {
startX: number
startY: number
endX: number
endY: number
}
export interface Dimensions {
width: number
height: number
}
export interface Point {
x: number
y: number
}
export interface DisplayRect {
x: number
y: number
w: number
h: number
}
export interface MoveStart {
mouseX: number
mouseY: number
zoneX: number
zoneY: number
}
export type ResizeHandle = 'n' | 's' | 'e' | 'w' | 'ne' | 'nw' | 'se' | 'sw'
export interface ResizeHandleMarker extends Point {
handle: ResizeHandle
}
export const HANDLE_SIZE = 8
export const MIN_ZONE_SIZE = 10
const ZONE_COLORS = [
'#4f8ff7',
'#ff6b6b',
'#51cf66',
'#ffd43b',
'#cc5de8',
'#ff922b',
'#20c997',
'#e599f7',
]
/** Outline color for a zone, shared by the preview overlay and the zone list. */
export function getZoneColor(index: number): string {
return ZONE_COLORS[index % ZONE_COLORS.length]
}
export function getZonePage(
zone: OcrTemplateZone,
previewPage: number,
previewPageCount: number | null
): number {
const page = zone.page ?? 1
if (page === -1) return previewPageCount ?? previewPage + 1
return Math.max(page, 1)
}
export function isZoneOnPage(
zone: OcrTemplateZone,
previewPage: number,
previewPageCount: number | null
): boolean {
return getZonePage(zone, previewPage, previewPageCount) === previewPage + 1
}
function getZoneSourceSize(
zone: OcrTemplateZone,
imageSize: Dimensions
): Dimensions {
return {
width: zone.zone_source_width || imageSize.width,
height: zone.zone_source_height || imageSize.height,
}
}
export function getZoneDisplayRect(
zone: OcrTemplateZone,
imageSize: Dimensions
): DisplayRect {
const sourceSize = getZoneSourceSize(zone, imageSize)
// The SVG viewBox uses the preview image's natural dimensions.
const scaleX = imageSize.width / sourceSize.width
const scaleY = imageSize.height / sourceSize.height
return {
x: zone.x * scaleX,
y: zone.y * scaleY,
w: zone.width * scaleX,
h: zone.height * scaleY,
}
}
export function getResizeHandles(rect: DisplayRect): ResizeHandleMarker[] {
return [
{ handle: 'nw', x: rect.x, y: rect.y },
{ handle: 'n', x: rect.x + rect.w / 2, y: rect.y },
{ handle: 'ne', x: rect.x + rect.w, y: rect.y },
{ handle: 'w', x: rect.x, y: rect.y + rect.h / 2 },
{ handle: 'e', x: rect.x + rect.w, y: rect.y + rect.h / 2 },
{ handle: 'sw', x: rect.x, y: rect.y + rect.h },
{ handle: 's', x: rect.x + rect.w / 2, y: rect.y + rect.h },
{ handle: 'se', x: rect.x + rect.w, y: rect.y + rect.h },
]
}
export function findHandleAt(
point: Point,
rect: DisplayRect,
handleSize = HANDLE_SIZE
): ResizeHandle | null {
return (
getResizeHandles(rect).find(
({ x, y }) =>
Math.abs(point.x - x) <= handleSize &&
Math.abs(point.y - y) <= handleSize
)?.handle ?? null
)
}
export function findZoneAt(
point: Point,
zones: OcrTemplateZone[],
previewPage: number,
previewPageCount: number | null,
imageSize: Dimensions
): number | null {
for (let i = zones.length - 1; i >= 0; i--) {
const zone = zones[i]
if (!isZoneOnPage(zone, previewPage, previewPageCount)) continue
const rect = getZoneDisplayRect(zone, imageSize)
if (
point.x >= rect.x &&
point.x <= rect.x + rect.w &&
point.y >= rect.y &&
point.y <= rect.y + rect.h
) {
return i
}
}
return null
}
export function moveZone(
zone: OcrTemplateZone,
point: Point,
moveStart: MoveStart,
imageSize: Dimensions
) {
const sourceSize = getZoneSourceSize(zone, imageSize)
const scaleX = sourceSize.width / imageSize.width
const scaleY = sourceSize.height / imageSize.height
const dx = Math.round((point.x - moveStart.mouseX) * scaleX)
const dy = Math.round((point.y - moveStart.mouseY) * scaleY)
zone.x = clamp(moveStart.zoneX + dx, 0, sourceSize.width - zone.width)
zone.y = clamp(moveStart.zoneY + dy, 0, sourceSize.height - zone.height)
}
export function resizeZone(
zone: OcrTemplateZone,
handle: ResizeHandle,
point: Point,
imageSize: Dimensions
) {
const sourceSize = getZoneSourceSize(zone, imageSize)
const scaleX = sourceSize.width / imageSize.width
const scaleY = sourceSize.height / imageSize.height
const imageX = clamp(Math.round(point.x * scaleX), 0, sourceSize.width)
const imageY = clamp(Math.round(point.y * scaleY), 0, sourceSize.height)
if (handle.includes('w')) {
const right = Math.min(zone.x + zone.width, sourceSize.width)
zone.x = clamp(imageX, 0, right - MIN_ZONE_SIZE)
zone.width = right - zone.x
}
if (handle.includes('e')) {
zone.width = Math.max(MIN_ZONE_SIZE, imageX - zone.x)
}
if (handle.includes('n')) {
const bottom = Math.min(zone.y + zone.height, sourceSize.height)
zone.y = clamp(imageY, 0, bottom - MIN_ZONE_SIZE)
zone.height = bottom - zone.y
}
if (handle.includes('s')) {
zone.height = Math.max(MIN_ZONE_SIZE, imageY - zone.y)
}
}
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(value, max))
}
@@ -0,0 +1,75 @@
<pngx-page-header
title="OCR Templates"
i18n-title
info="Define extraction zones on document types to automatically populate custom fields via OCR."
i18n-info
>
<button type="button" class="btn btn-sm btn-outline-primary" (click)="editTemplate()" *pngxIfPermissions="{ action: PermissionAction.Add, type: PermissionType.OcrTemplate }">
<pngx-bs-icon name="plus-circle" class="me-1"></pngx-bs-icon><ng-container i18n>Add Template</ng-container>
</button>
</pngx-page-header>
<ul class="list-group">
<li class="list-group-item">
<div class="row">
<div class="col" i18n>Name</div>
<div class="col d-none d-sm-flex" i18n>Matching</div>
<div class="col d-none d-sm-flex" i18n>Zones</div>
<div class="col" i18n>Status</div>
<div class="col" i18n>Actions</div>
</div>
</li>
@if (loading() && templates().length === 0) {
<li class="list-group-item">
<output class="spinner-border spinner-border-sm me-2"></output>
<ng-container i18n>Loading...</ng-container>
</li>
}
@for (template of templates(); track template.id) {
<li class="list-group-item">
<div class="row fade" [class.show]="show()">
<div class="col d-flex align-items-center"><button class="btn btn-link p-0 text-start" type="button" (click)="editTemplate(template)" [disabled]="!userCanEdit(template)">{{template.name}}</button></div>
<div class="col d-flex align-items-center d-none d-sm-flex">{{getMatching(template)}}</div>
<div class="col d-flex align-items-center d-none d-sm-flex"><code>{{template.zones.length}}</code></div>
<div class="col d-flex align-items-center">
<div class="form-check form-switch mb-0">
<input type="checkbox" class="form-check-input cursor-pointer" [id]="template.id+'_enable'" [(ngModel)]="template.enabled" (change)="toggleTemplateEnabled(template)" [disabled]="!userCanEdit(template)" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.OcrTemplate }">
<label class="form-check-label cursor-pointer" [for]="template.id+'_enable'">
<code> @if(template.enabled) { <ng-container i18n>Enabled</ng-container> } @else { <span i18n class="text-muted">Disabled</span> }</code>
</label>
</div>
</div>
<div class="col">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile{{template.id}}" (click)="$event.stopPropagation()" ngbDropdownToggle>
<pngx-bs-icon name="three-dots-vertical"></pngx-bs-icon>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile{{template.id}}">
<button (click)="editTemplate(template)" [disabled]="!userCanEdit(template)" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.OcrTemplate }" ngbDropdownItem i18n>Edit</button>
<button (click)="deleteTemplate(template)" [disabled]="!permissionsService.currentUserOwnsObject(template)" *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.OcrTemplate }" ngbDropdownItem i18n>Delete</button>
</div>
</div>
</div>
<div class="btn-toolbar d-none d-sm-flex gap-2" role="toolbar">
<div class="btn-group">
<button *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.OcrTemplate }" [disabled]="!userCanEdit(template)" class="btn btn-sm btn-outline-secondary" type="button" (click)="editTemplate(template)">
<pngx-bs-icon width="1em" height="1em" name="pencil" class="me-1"></pngx-bs-icon><ng-container i18n>Edit</ng-container>
</button>
<button *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.OcrTemplate }" [disabled]="!permissionsService.currentUserOwnsObject(template)" class="btn btn-sm btn-outline-danger" type="button" (click)="deleteTemplate(template)">
<pngx-bs-icon width="1em" height="1em" name="trash" class="me-1"></pngx-bs-icon><ng-container i18n>Delete</ng-container>
</button>
</div>
</div>
</div>
</div>
</li>
}
@if (!loading() && templates().length === 0) {
<li class="list-group-item" [class.show]="show()" i18n>No OCR templates defined.</li>
}
</ul>
@@ -0,0 +1,300 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { By } from '@angular/platform-browser'
import { NgbModal, NgbModalRef } from '@ng-bootstrap/ng-bootstrap'
import { of, throwError } from 'rxjs'
import { MATCH_LITERAL, MATCH_NONE } from 'src/app/data/matching-model'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { User } from 'src/app/data/user'
import { DirtyFormGuard } from 'src/app/guards/dirty-form.guard'
import { PermissionsService } from 'src/app/services/permissions.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { UserService } from 'src/app/services/rest/user.service'
import { ToastService } from 'src/app/services/toast.service'
import { ConfirmDialogComponent } from '../../common/confirm-dialog/confirm-dialog.component'
import { OcrTemplateEditorComponent } from './ocr-template-editor/ocr-template-editor.component'
import { OcrTemplatesComponent } from './ocr-templates.component'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
const templates: OcrTemplate[] = [
{
id: 1,
name: 'Invoices',
match: 'Northwind',
matching_algorithm: MATCH_LITERAL,
sample_document: null,
source_width: 0,
source_height: 0,
enabled: true,
zones: [],
},
]
describe('OcrTemplatesComponent', () => {
let component: OcrTemplatesComponent
let fixture: ComponentFixture<OcrTemplatesComponent>
let ocrTemplateService: OcrTemplateService
let modalService: NgbModal
let toastService: ToastService
let modal: NgbModalRef
beforeEach(() => {
TestBed.configureTestingModule({
imports: [OcrTemplatesComponent, BootstrapIconComponent],
providers: [
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
})
TestBed.inject(PermissionsService).initialize(
[
'view_ocrtemplate',
'add_ocrtemplate',
'change_ocrtemplate',
'delete_ocrtemplate',
],
{ id: 1, groups: [], is_superuser: false } as User
)
jest
.spyOn(TestBed.inject(UserService), 'listAll')
.mockReturnValue(of({ count: 0, all: [], results: [] }))
ocrTemplateService = TestBed.inject(OcrTemplateService)
jest
.spyOn(ocrTemplateService, 'listAll')
.mockReturnValue(of({ count: 1, all: [1], results: templates }))
modalService = TestBed.inject(NgbModal)
modalService.activeInstances.subscribe((m) => (modal = m[m.length - 1]))
toastService = TestBed.inject(ToastService)
fixture = TestBed.createComponent(OcrTemplatesComponent)
component = fixture.componentInstance
fixture.detectChanges()
})
afterEach(() => modalService.dismissAll())
it('should show templates with their matching', () => {
const text = fixture.nativeElement.textContent
expect(text).toContain('Invoices')
expect(text).toContain('Exact match: Northwind')
expect(
component.getMatching({ ...templates[0], matching_algorithm: MATCH_NONE })
).toEqual('None')
expect(component.getMatching({ ...templates[0], match: '' })).toEqual('-')
})
it.each([
{
name: 'owned',
owner: 1,
user_can_change: true,
editable: true,
deletable: true,
},
{
name: 'unowned',
owner: null,
user_can_change: true,
editable: true,
deletable: true,
},
{
name: 'view-only shared',
owner: 2,
user_can_change: false,
editable: false,
deletable: false,
},
{
name: 'editable shared',
owner: 2,
user_can_change: true,
editable: true,
deletable: false,
},
])(
'should respect permissions for $name templates',
async ({ owner, user_can_change, editable, deletable }) => {
const template = { ...templates[0], owner, user_can_change }
component.templates.set([template])
fixture.detectChanges()
await fixture.whenStable()
expect(fixture.nativeElement.querySelector('.text-start').disabled).toBe(
!editable
)
expect(
fixture.nativeElement.querySelector('.form-switch input').disabled
).toBe(!editable)
const buttons = Array.from<HTMLButtonElement>(
fixture.nativeElement.querySelectorAll('button')
)
for (const button of buttons.filter(
(button) => button.textContent.trim() === 'Edit'
)) {
expect(button.disabled).toBe(!editable)
}
for (const button of buttons.filter(
(button) => button.textContent.trim() === 'Delete'
)) {
expect(button.disabled).toBe(!deletable)
}
}
)
it('should allow a superuser to manage another users template', () => {
TestBed.inject(PermissionsService).initialize([], {
id: 1,
is_superuser: true,
} as User)
component.templates.set([
{ ...templates[0], owner: 2, user_can_change: false },
])
fixture.detectChanges()
expect(fixture.nativeElement.querySelector('.text-start').disabled).toBe(
false
)
expect(
fixture.nativeElement.querySelector('.btn-outline-danger').disabled
).toBe(false)
})
it('should not open mutation dialogs or patch a view-only template', () => {
const template = { ...templates[0], owner: 2, user_can_change: false }
const open = jest.spyOn(modalService, 'open')
const patch = jest.spyOn(ocrTemplateService, 'patch')
component.editTemplate(template)
component.toggleTemplateEnabled(template)
component.deleteTemplate(template)
expect(open).not.toHaveBeenCalled()
expect(patch).not.toHaveBeenCalled()
})
it('should require global edit permission even for an owned template', () => {
TestBed.inject(PermissionsService).initialize(['view_ocrtemplate'], {
id: 1,
} as User)
fixture.destroy()
fixture = TestBed.createComponent(OcrTemplatesComponent)
component = fixture.componentInstance
fixture.detectChanges()
expect(fixture.nativeElement.querySelector('.text-start').disabled).toBe(
true
)
expect(fixture.nativeElement.querySelector('.form-switch input')).toBeNull()
})
it('should support create, show notification on error / success', () => {
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSuccessSpy = jest.spyOn(toastService, 'showSuccess')
const reloadSpy = jest.spyOn(component, 'reload')
fixture.debugElement
.queryAll(By.css('pngx-page-header button'))
.find((button) => button.nativeElement.textContent.includes('Add'))
.triggerEventHandler('click')
const editor = modal.componentInstance as OcrTemplateEditorComponent
expect(editor).toBeInstanceOf(OcrTemplateEditorComponent)
expect(editor.object).toBeUndefined()
editor.failed.emit({ error: 'error creating template' })
expect(toastErrorSpy).toHaveBeenCalled()
expect(reloadSpy).not.toHaveBeenCalled()
editor.succeeded.emit(templates[0])
expect(toastSuccessSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalled()
})
it('should pass the template to the editor', () => {
component.editTemplate(templates[0])
expect(modal.componentInstance.object).toBe(templates[0])
})
it('should confirm closing the editor with unsaved changes', async () => {
const confirmSpy = jest
.spyOn(TestBed.inject(DirtyFormGuard), 'confirmChanges')
.mockReturnValue(of(false))
component.editTemplate()
const editor = modal.componentInstance as OcrTemplateEditorComponent
editor.ngOnInit()
modal.dismiss()
await fixture.whenStable()
expect(confirmSpy).not.toHaveBeenCalled()
component.editTemplate()
const dirtyEditor = modal.componentInstance as OcrTemplateEditorComponent
dirtyEditor.ngOnInit()
dirtyEditor.template.name = 'Changed'
modal.dismiss()
await fixture.whenStable()
expect(confirmSpy).toHaveBeenCalled()
expect(modalService.hasOpenModals()).toBe(true)
dirtyEditor.saving.set(true)
confirmSpy.mockClear()
modal.dismiss()
await fixture.whenStable()
expect(confirmSpy).not.toHaveBeenCalled()
expect(modalService.hasOpenModals()).toBe(true)
})
it('should only send the enabled flag when toggling', () => {
const patchSpy = jest
.spyOn(ocrTemplateService, 'patch')
.mockReturnValueOnce(throwError(() => new Error('error toggling')))
.mockReturnValueOnce(of(templates[0]))
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSuccessSpy = jest.spyOn(toastService, 'showSuccess')
const reloadSpy = jest.spyOn(component, 'reload')
const toggle = fixture.debugElement.query(By.css('.form-switch input'))
toggle.nativeElement.click()
expect(patchSpy).toHaveBeenCalledWith({ id: 1, enabled: false })
expect(toastErrorSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalledTimes(1)
component.toggleTemplateEnabled(templates[0])
expect(toastSuccessSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalledTimes(2)
})
it('should support delete, show notification on error / success', () => {
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSuccessSpy = jest.spyOn(toastService, 'showSuccess')
const deleteSpy = jest.spyOn(ocrTemplateService, 'delete')
const reloadSpy = jest.spyOn(component, 'reload')
component.deleteTemplate(templates[0])
const dialog = modal.componentInstance as ConfirmDialogComponent
deleteSpy.mockReturnValueOnce(throwError(() => new Error('error deleting')))
dialog.confirmClicked.emit()
expect(toastErrorSpy).toHaveBeenCalled()
expect(dialog.buttonsEnabled()).toBe(true)
expect(reloadSpy).not.toHaveBeenCalled()
deleteSpy.mockReturnValueOnce(of(true))
dialog.confirmClicked.emit()
expect(toastSuccessSpy).toHaveBeenCalled()
expect(reloadSpy).toHaveBeenCalled()
})
it.each([true, false])(
'reports the enabled state %s after the update succeeds',
(enabled) => {
const template = { ...templates[0], enabled }
jest.spyOn(ocrTemplateService, 'patch').mockReturnValue(of(template))
const toast = jest.spyOn(toastService, 'showSuccess')
component.toggleTemplateEnabled(template)
expect(toast).toHaveBeenCalledWith(
`${enabled ? 'Enabled' : 'Disabled'} OCR template "${template.name}"`
)
}
)
})
@@ -0,0 +1,185 @@
import { Component, OnInit, inject, signal } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { NgbDropdownModule, NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
import { firstValueFrom, takeUntil, tap } from 'rxjs'
import { MATCH_NONE, MATCHING_ALGORITHMS } from 'src/app/data/matching-model'
import { OcrTemplate } from 'src/app/data/ocr-template'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { DirtyFormGuard } from 'src/app/guards/dirty-form.guard'
import {
PermissionAction,
PermissionType,
PermissionsService,
} from 'src/app/services/permissions.service'
import { OcrTemplateService } from 'src/app/services/rest/ocr-template.service'
import { ToastService } from 'src/app/services/toast.service'
import { ConfirmDialogComponent } from '../../common/confirm-dialog/confirm-dialog.component'
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
import { OcrTemplateEditorComponent } from './ocr-template-editor/ocr-template-editor.component'
@Component({
selector: 'pngx-ocr-templates',
templateUrl: './ocr-templates.component.html',
imports: [
PageHeaderComponent,
IfPermissionsDirective,
FormsModule,
NgbDropdownModule,
BootstrapIconComponent,
],
})
export class OcrTemplatesComponent
extends LoadingComponentWithPermissions
implements OnInit
{
private readonly ocrTemplateService = inject(OcrTemplateService)
public readonly permissionsService = inject(PermissionsService)
private readonly modalService = inject(NgbModal)
private readonly dirtyFormGuard = inject(DirtyFormGuard)
private readonly toastService = inject(ToastService)
public readonly templates = signal<OcrTemplate[]>([])
ngOnInit() {
this.reload()
}
reload() {
this.loading.set(true)
this.ocrTemplateService
.listAll()
.pipe(
takeUntil(this.unsubscribeNotifier),
tap((r) => this.templates.set(r.results))
)
.subscribe(() => {
this.show.set(true)
this.loading.set(false)
})
}
getMatching(template: OcrTemplate): string {
if (template.matching_algorithm === MATCH_NONE) {
return $localize`None`
}
if (!template.match) {
return '-'
}
const algorithm = MATCHING_ALGORITHMS.find(
(a) => a.id === template.matching_algorithm
)
return `${algorithm?.shortName}: ${template.match}`
}
editTemplate(template?: OcrTemplate) {
if (template && !this.userCanEdit(template)) return
const modal = this.modalService.open(OcrTemplateEditorComponent, {
backdrop: 'static',
keyboard: false,
size: 'xxl',
fullscreen: 'lg',
scrollable: true,
ariaLabelledBy: 'ocr-template-editor-title',
beforeDismiss: () => {
const editor: OcrTemplateEditorComponent = modal.componentInstance
if (editor.saving()) return false
if (!editor.isDirty) return true
return firstValueFrom(
this.dirtyFormGuard.confirmChanges(undefined, $localize`Close`)
)
},
})
modal.componentInstance.object = template
modal.componentInstance.succeeded
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((saved: OcrTemplate) => {
this.toastService.showSuccess(
$localize`Saved OCR template "${saved.name}".`
)
this.ocrTemplateService.clearCache()
this.reload()
})
modal.componentInstance.failed
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((e) => {
this.toastService.showError($localize`Error saving OCR template.`, e)
})
}
toggleTemplateEnabled(template: OcrTemplate) {
if (!this.userCanEdit(template)) return
this.ocrTemplateService
.patch({ id: template.id, enabled: template.enabled } as OcrTemplate)
.subscribe({
next: () => {
this.toastService.showSuccess(
template.enabled
? $localize`Enabled OCR template "${template.name}"`
: $localize`Disabled OCR template "${template.name}"`
)
this.reload()
},
error: (e) => {
this.toastService.showError(
$localize`Error toggling OCR template "${template.name}".`,
e
)
this.reload()
},
})
}
deleteTemplate(template: OcrTemplate) {
if (
!this.permissionsService.currentUserCan(
PermissionAction.Delete,
PermissionType.OcrTemplate
) ||
!this.permissionsService.currentUserOwnsObject(template)
)
return
const modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Confirm delete OCR template`
modal.componentInstance.messageBold = $localize`This operation will permanently delete this OCR template.`
modal.componentInstance.message = $localize`This operation cannot be undone.`
modal.componentInstance.btnClass = 'btn-danger'
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false)
this.ocrTemplateService.delete(template).subscribe({
next: () => {
modal.close()
this.toastService.showSuccess(
$localize`Deleted OCR template "${template.name}".`
)
this.ocrTemplateService.clearCache()
this.reload()
},
error: (e) => {
modal.componentInstance.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Error deleting OCR template "${template.name}".`,
e
)
},
})
})
}
userCanEdit(template: OcrTemplate): boolean {
return (
this.permissionsService.currentUserCan(
PermissionAction.Change,
PermissionType.OcrTemplate
) &&
this.permissionsService.currentUserHasObjectPermissions(
PermissionAction.Change,
template
)
)
}
}
+115
View File
@@ -0,0 +1,115 @@
import { CustomFieldDataType } from './custom-field'
import { MatchingModel } from './matching-model'
import { ObjectWithPermissions } from './object-with-permissions'
export enum OcrZoneTarget {
CustomField = 'custom_field',
Title = 'title',
Asn = 'asn',
Created = 'created',
}
export enum OcrZoneTransform {
None = 'none',
Strip = 'strip',
Uppercase = 'uppercase',
Lowercase = 'lowercase',
Numeric = 'numeric',
StripPunctuation = 'strip_punctuation',
Date = 'date',
QrCode = 'qr_code',
}
export const DEFAULT_OCR_ZONE_LANGUAGE = 'eng'
/** The field a zone writes to, keyed by OcrTemplate.combine_formats. */
export function getZoneFieldKey(zone: OcrTemplateZone): string | null {
if (zone.target !== OcrZoneTarget.CustomField) return zone.target
return zone.custom_field === null ? null : String(zone.custom_field)
}
/** Custom field types zone OCR can write to. See models.py OCR_SUPPORTED_FIELD_TYPES. */
export const OCR_SUPPORTED_FIELD_TYPES: CustomFieldDataType[] = [
CustomFieldDataType.String,
CustomFieldDataType.Url,
CustomFieldDataType.Date,
CustomFieldDataType.Integer,
CustomFieldDataType.Float,
CustomFieldDataType.Monetary,
CustomFieldDataType.LongText,
CustomFieldDataType.Boolean,
]
export const OCR_BUILTIN_TARGETS = [
{ id: OcrZoneTarget.Title, name: $localize`Title` },
{ id: OcrZoneTarget.Asn, name: $localize`Archive serial number` },
{ id: OcrZoneTarget.Created, name: $localize`Date created` },
]
export interface OcrTemplateZone {
id?: number
name: string
target: OcrZoneTarget
custom_field: number | null
page?: number
x: number
y: number
width: number
height: number
ocr_language: string
transform: OcrZoneTransform
date_format?: string
validation_regex: string
order: number
zone_source_width?: number
zone_source_height?: number
}
export interface OcrTemplate extends MatchingModel, ObjectWithPermissions {
name: string
sample_document: number | null
source_width: number
source_height: number
enabled: boolean
combine_formats?: Record<string, string>
created?: string
updated?: string
zones: OcrTemplateZone[]
}
export interface ZoneTestRequest {
name: string
x: number
y: number
width: number
height: number
page: number
ocr_language: string
transform: OcrZoneTransform
date_format?: string
validation_regex: string
zone_source_width?: number
zone_source_height?: number
}
export interface OcrZoneTestResult {
raw_text?: string | null
value?: string | null
regex?: string
regex_match?: boolean | null
error?: string
}
export interface OcrZoneRunResult {
template: string
zone: string
custom_field: string
value: string | number | boolean | null
reason?: string | null
}
export interface OcrZoneRunResponse {
status: 'completed' | 'skipped'
reason?: 'no_template'
results: OcrZoneRunResult[]
}
+1
View File
@@ -35,6 +35,7 @@ export enum HideableSidebarItemID {
SavedViews = 'saved_views', SavedViews = 'saved_views',
ShareLinks = 'share_links', ShareLinks = 'share_links',
Workflows = 'workflows', Workflows = 'workflows',
OcrTemplates = 'ocr_templates',
Mail = 'mail', Mail = 'mail',
Documentation = 'documentation', Documentation = 'documentation',
} }
+6 -2
View File
@@ -3,12 +3,16 @@ import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { DirtyCheckGuard } from '@ngneat/dirty-check-forms' import { DirtyCheckGuard } from '@ngneat/dirty-check-forms'
import { Observable, Subject } from 'rxjs' import { Observable, Subject } from 'rxjs'
import { ConfirmDialogComponent } from 'src/app/components/common/confirm-dialog/confirm-dialog.component' import { ConfirmDialogComponent } from 'src/app/components/common/confirm-dialog/confirm-dialog.component'
import { ActivatedRouteSnapshot } from '@angular/router'
@Injectable({ providedIn: 'root' }) @Injectable({ providedIn: 'root' })
export class DirtyFormGuard extends DirtyCheckGuard { export class DirtyFormGuard extends DirtyCheckGuard {
private modalService = inject(NgbModal) private modalService = inject(NgbModal)
confirmChanges(): Observable<boolean> { confirmChanges(
_currentRoute?: ActivatedRouteSnapshot,
leaveLabel: string = $localize`Leave page`
): Observable<boolean> {
let modal = this.modalService.open(ConfirmDialogComponent, { let modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static', backdrop: 'static',
}) })
@@ -16,7 +20,7 @@ export class DirtyFormGuard extends DirtyCheckGuard {
modal.componentInstance.messageBold = $localize`You have unsaved changes.` modal.componentInstance.messageBold = $localize`You have unsaved changes.`
modal.componentInstance.message = $localize`Are you sure you want to leave?` modal.componentInstance.message = $localize`Are you sure you want to leave?`
modal.componentInstance.btnClass = 'btn-warning' modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Leave page` modal.componentInstance.btnCaption = leaveLabel
modal.componentInstance.confirmClicked.subscribe(() => { modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false) modal.componentInstance.buttonsEnabled.set(false)
modal.close() modal.close()
@@ -386,6 +386,46 @@ describe('DocumentListViewService', () => {
documentListViewService.activateSavedView(null) documentListViewService.activateSavedView(null)
}) })
it('should support activating saved view with pre-fetched documents', () => {
const routerSpy = jest.spyOn(router, 'navigate')
routerSpy.mockClear()
documentListViewService.activateSavedViewWithDocuments(
view,
documents.slice(0, 2),
documents.length,
2
)
// no navigation, no reload
expect(routerSpy).not.toHaveBeenCalled()
httpTestingController.expectNone(
(req) => req.url === `${environment.apiBaseUrl}documents/`
)
expect(documentListViewService.activeSavedViewId).toEqual(view.id)
expect(documentListViewService.activeSavedViewTitle).toEqual(view.name)
expect(documentListViewService.documents).toEqual(documents.slice(0, 2))
expect(documentListViewService.collectionSize).toEqual(documents.length)
expect(documentListViewService.currentPage).toEqual(1)
expect(documentListViewService.pageSize).toEqual(2)
expect(documentListViewService.hasNext(documents[0].id)).toBeTruthy()
let nextDocId: number
documentListViewService
.getNext(documents[1].id)
.subscribe((docId) => (nextDocId = docId))
// crossing the page uses the view's sort + filters and the given page size
// (the currentPage setter reloads too, so the first request is cancelled)
const reqs = httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=2&page_size=2&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
)
expect(reqs.length).toBeGreaterThan(0)
reqs[reqs.length - 1].flush({
count: documents.length,
results: documents.slice(2, 4),
})
expect(nextDocId).toEqual(documents[2].id)
documentListViewService.activateSavedView(null)
})
it('should support navigating next / previous', () => { it('should support navigating next / previous', () => {
documentListViewService.setFilterRules([]) documentListViewService.setFilterRules([])
let req = httpTestingController.expectOne( let req = httpTestingController.expectOne(
@@ -270,11 +270,28 @@ export class DocumentListViewService {
this.currentPage = viewState.currentPage this.currentPage = viewState.currentPage
} }
loadSavedView(view: SavedView, closeCurrentView: boolean = false) { /**
if (closeCurrentView) { * Activate a saved view using documents that were already fetched, e.g. by a
this._activeSavedViewId = null * dashboard widget. Does not navigate or reload.
} */
activateSavedViewWithDocuments(
view: SavedView,
documents: Document[],
collectionSize: number,
pageSize: number
) {
this.cancelPending()
this.rangeSelectionAnchorIndex = this.lastRangeSelectionToIndex = null
this._activeSavedViewId = view.id
this.applySavedViewConfig(view)
this.activeListViewState.pageSize = pageSize // match the page size the documents were fetched with
this.activeListViewState.currentPage = 1
this.activeListViewState.documents = documents
this.activeListViewState.collectionSize = collectionSize
this.markChanged()
}
private applySavedViewConfig(view: SavedView) {
this.activeListViewState.filterRules = cloneFilterRules(view.filter_rules) this.activeListViewState.filterRules = cloneFilterRules(view.filter_rules)
this.activeListViewState.sortField = view.sort_field this.activeListViewState.sortField = view.sort_field
this.activeListViewState.sortReverse = view.sort_reverse this.activeListViewState.sortReverse = view.sort_reverse
@@ -284,6 +301,14 @@ export class DocumentListViewService {
this.activeListViewState.displayMode = view.display_mode this.activeListViewState.displayMode = view.display_mode
this.activeListViewState.pageSize = view.page_size this.activeListViewState.pageSize = view.page_size
this.activeListViewState.displayFields = view.display_fields this.activeListViewState.displayFields = view.display_fields
}
loadSavedView(view: SavedView, closeCurrentView: boolean = false) {
if (closeCurrentView) {
this._activeSavedViewId = null
}
this.applySavedViewConfig(view)
this.reduceSelectionToFilter() this.reduceSelectionToFilter()
this.markChanged() this.markChanged()
@@ -283,6 +283,10 @@ describe('PermissionsService', () => {
'view_workflow', 'view_workflow',
'change_workflow', 'change_workflow',
'delete_workflow', 'delete_workflow',
'add_ocrtemplate',
'view_ocrtemplate',
'change_ocrtemplate',
'delete_ocrtemplate',
'add_customfield', 'add_customfield',
'view_customfield', 'view_customfield',
'change_customfield', 'change_customfield',
@@ -29,6 +29,7 @@ export enum PermissionType {
ShareLinkBundle = '%s_sharelinkbundle', ShareLinkBundle = '%s_sharelinkbundle',
CustomField = '%s_customfield', CustomField = '%s_customfield',
Workflow = '%s_workflow', Workflow = '%s_workflow',
OcrTemplate = '%s_ocrtemplate',
ProcessedMail = '%s_processedmail', ProcessedMail = '%s_processedmail',
GlobalStatistics = '%s_global_statistics', GlobalStatistics = '%s_global_statistics',
SystemMonitoring = '%s_system_monitoring', SystemMonitoring = '%s_system_monitoring',
@@ -563,6 +563,20 @@ describe(`DocumentService`, () => {
req.flush('task-id') req.flush('task-id')
}) })
it('should call appropriate api endpoint for running zone OCR', () => {
const url = `${environment.apiBaseUrl}${endpoint}/${documents[0].id}/run-zone-ocr/`
subscription = service.runZoneOcr(documents[0].id).subscribe()
let req = httpTestingController.expectOne(url)
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({})
req.flush({ status: 'completed', results: [] })
subscription = service.runZoneOcr(documents[0].id, 3).subscribe()
req = httpTestingController.expectOne(url)
expect(req.request.body).toEqual({ template: 3 })
req.flush({ status: 'completed', results: [] })
})
}) })
it('should construct sort fields respecting permissions', () => { it('should construct sort fields respecting permissions', () => {
@@ -12,6 +12,7 @@ import {
import { DocumentMetadata } from 'src/app/data/document-metadata' import { DocumentMetadata } from 'src/app/data/document-metadata'
import { DocumentSuggestions } from 'src/app/data/document-suggestions' import { DocumentSuggestions } from 'src/app/data/document-suggestions'
import { FilterRule } from 'src/app/data/filter-rule' import { FilterRule } from 'src/app/data/filter-rule'
import { OcrZoneRunResponse } from 'src/app/data/ocr-template'
import { Results, SelectionData } from 'src/app/data/results' import { Results, SelectionData } from 'src/app/data/results'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings' import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { queryParamsFromFilterRules } from '../../utils/query-params' import { queryParamsFromFilterRules } from '../../utils/query-params'
@@ -360,6 +361,26 @@ export class DocumentService extends AbstractPaperlessService<Document> {
}) })
} }
runZoneOcrDocuments(
selection: DocumentSelectionQuery,
templateId: number | null = null
) {
return this.http.post(this.getResourceUrl(null, 'run_zone_ocr'), {
...selection,
...(templateId ? { template: templateId } : {}),
})
}
runZoneOcr(
id: number,
templateId: number | null = null
): Observable<OcrZoneRunResponse> {
return this.http.post<OcrZoneRunResponse>(
this.getResourceUrl(id, 'run-zone-ocr'),
templateId ? { template: templateId } : {}
)
}
rotateDocuments( rotateDocuments(
selection: DocumentSelectionQuery, selection: DocumentSelectionQuery,
degrees: number, degrees: number,
@@ -0,0 +1,78 @@
import { provideHttpClient } from '@angular/common/http'
import {
HttpTestingController,
provideHttpClientTesting,
} from '@angular/common/http/testing'
import { TestBed } from '@angular/core/testing'
import { OcrZoneTransform, ZoneTestRequest } from 'src/app/data/ocr-template'
import { environment } from 'src/environments/environment'
import { OcrTemplateService } from './ocr-template.service'
describe('OcrTemplateService', () => {
let service: OcrTemplateService
let http: HttpTestingController
beforeEach(() => {
TestBed.configureTestingModule({
providers: [provideHttpClient(), provideHttpClientTesting()],
})
service = TestBed.inject(OcrTemplateService)
http = TestBed.inject(HttpTestingController)
})
afterEach(() => http.verify())
it('builds page image URLs with zero-based page numbers', () => {
expect(service.getPageImageUrl(12, 0)).toBe(
`${environment.apiBaseUrl}ocr_templates/document-page-image/12/0/`
)
expect(service.getPageImageUrl(12, 2)).toBe(
`${environment.apiBaseUrl}ocr_templates/document-page-image/12/2/`
)
})
it('posts a zone test and returns the extraction result', () => {
const zone: ZoneTestRequest = {
name: 'Total',
x: 10,
y: 20,
width: 100,
height: 50,
page: -1,
ocr_language: 'eng',
transform: OcrZoneTransform.Numeric,
validation_regex: '\\d+',
zone_source_width: 1000,
zone_source_height: 1500,
}
const result = {
raw_text: 'Total: 123',
value: '123',
regex: '\\d+',
regex_match: true,
}
const next = jest.fn()
service.testZone(12, zone).subscribe(next)
const request = http.expectOne(
`${environment.apiBaseUrl}ocr_templates/test-zone/`
)
expect(request.request.method).toBe('POST')
expect(request.request.body).toEqual({ document: 12, zone })
request.flush(result)
expect(next).toHaveBeenCalledWith(result)
})
it('passes zone-test failures to the caller', () => {
const error = jest.fn()
service.testZone(12, {} as ZoneTestRequest).subscribe({ error })
http
.expectOne(`${environment.apiBaseUrl}ocr_templates/test-zone/`)
.flush(
{ error: 'Invalid zone' },
{ status: 400, statusText: 'Bad Request' }
)
expect(error).toHaveBeenCalledWith(
expect.objectContaining({ status: 400, error: { error: 'Invalid zone' } })
)
})
})
@@ -0,0 +1,30 @@
import { Injectable } from '@angular/core'
import { Observable } from 'rxjs'
import {
OcrTemplate,
OcrZoneTestResult,
ZoneTestRequest,
} from '../../data/ocr-template'
import { AbstractPaperlessService } from './abstract-paperless-service'
@Injectable({ providedIn: 'root' })
export class OcrTemplateService extends AbstractPaperlessService<OcrTemplate> {
constructor() {
super()
this.resourceName = 'ocr_templates'
}
getPageImageUrl(docId: number, page: number): string {
return `${this.baseUrl}${this.resourceName}/document-page-image/${docId}/${page}/`
}
testZone(
docId: number,
zone: ZoneTestRequest
): Observable<OcrZoneTestResult> {
return this.http.post<OcrZoneTestResult>(
`${this.baseUrl}${this.resourceName}/test-zone/`,
{ document: docId, zone }
)
}
}
+6
View File
@@ -691,6 +691,12 @@ table.table {
} }
} }
@media (min-width: 992px) {
.modal-xxl {
--bs-modal-width: min(1440px, calc(100vw - 2 * var(--bs-modal-margin)));
}
}
.toast { .toast {
--bs-toast-max-width: var(--pngx-toast-max-width); --bs-toast-max-width: var(--pngx-toast-max-width);
} }
+3
View File
@@ -15,6 +15,7 @@ class DocumentsConfig(AppConfig):
from documents.signals.handlers import add_to_index from documents.signals.handlers import add_to_index
from documents.signals.handlers import run_workflows_added from documents.signals.handlers import run_workflows_added
from documents.signals.handlers import run_workflows_updated from documents.signals.handlers import run_workflows_updated
from documents.signals.handlers import run_zone_ocr_extraction
from documents.signals.handlers import send_websocket_document_updated from documents.signals.handlers import send_websocket_document_updated
from documents.signals.handlers import set_correspondent from documents.signals.handlers import set_correspondent
from documents.signals.handlers import set_document_type from documents.signals.handlers import set_document_type
@@ -29,6 +30,8 @@ class DocumentsConfig(AppConfig):
document_consumption_finished.connect(run_workflows_added) document_consumption_finished.connect(run_workflows_added)
document_consumption_finished.connect(add_to_index) document_consumption_finished.connect(add_to_index)
document_consumption_finished.connect(add_or_update_document_in_llm_index) document_consumption_finished.connect(add_or_update_document_in_llm_index)
document_consumption_finished.connect(run_zone_ocr_extraction)
document_updated.connect(run_workflows_updated) document_updated.connect(run_workflows_updated)
document_updated.connect(send_websocket_document_updated) document_updated.connect(send_websocket_document_updated)
document_updated.connect(add_or_update_document_in_llm_index) document_updated.connect(add_or_update_document_in_llm_index)
+21
View File
@@ -33,6 +33,7 @@ from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplate
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
@@ -41,6 +42,7 @@ from documents.permissions import set_permissions_for_objects
from documents.plugins.helpers import DocumentsStatusManager from documents.plugins.helpers import DocumentsStatusManager
from documents.tasks import bulk_update_documents from documents.tasks import bulk_update_documents
from documents.tasks import consume_file from documents.tasks import consume_file
from documents.tasks import extract_zones
from documents.tasks import remove_document_from_index from documents.tasks import remove_document_from_index
from documents.tasks import update_document_content_maybe_archive_file from documents.tasks import update_document_content_maybe_archive_file
from documents.versioning import get_latest_version_for_root from documents.versioning import get_latest_version_for_root
@@ -1197,3 +1199,22 @@ def remove_doclink(
target_doc_field_instance.value.remove(document.id) target_doc_field_instance.value.remove(document.id)
target_doc_field_instance.save() target_doc_field_instance.save()
Document.objects.filter(id=target_doc_id).update(modified=timezone.now()) Document.objects.filter(id=target_doc_id).update(modified=timezone.now())
def run_zone_ocr(
doc_ids: list[int],
*,
template: OcrTemplate | None = None,
) -> Literal["OK"]:
"""
Queue zone OCR for the given documents, with the given template or with the
templates that match each document's content when its task runs.
"""
template_id = template.pk if template is not None else None
for document_id in doc_ids:
extract_zones.apply_async(
kwargs={"document_id": document_id, "template_id": template_id},
headers={"trigger_source": PaperlessTask.TriggerSource.MANUAL},
)
return "OK"
+10 -19
View File
@@ -15,6 +15,7 @@ from django.db import transaction
from django.db.models import Max from django.db.models import Max
from django.db.models import Q from django.db.models import Q
from django.utils import timezone from django.utils import timezone
from filelock import FileLock
from rest_framework.reverse import reverse from rest_framework.reverse import reverse
from documents.barcodes import read_barcode_values from documents.barcodes import read_barcode_values
@@ -26,10 +27,7 @@ from documents.file_handling import create_source_path_directory
from documents.file_handling import generate_filename from documents.file_handling import generate_filename
from documents.file_handling import generate_unique_filename from documents.file_handling import generate_unique_filename
from documents.file_handling import validate_path_in_root from documents.file_handling import validate_path_in_root
from documents.file_moves import defer_file_moves
from documents.file_moves import flush_pending_file_moves
from documents.loggers import LoggingMixin from documents.loggers import LoggingMixin
from documents.media_lock import media_lock
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import CustomField from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
@@ -607,13 +605,7 @@ class ConsumerPlugin(
# now that everything is done, we can start to store the document # now that everything is done, we can start to store the document
# in the system. This will be a transaction and reasonably fast. # in the system. This will be a transaction and reasonably fast.
try: try:
# File moves requested in here run after commit and after with transaction.atomic():
# media.lock is released. media.lock is taken before the
# transaction: the version path locks the root row with
# select_for_update, and workflow email actions and file
# placement below re-enter media.lock instead of waiting
# on it while rows are locked.
with defer_file_moves(), media_lock(), transaction.atomic():
# store the document. # store the document.
if self.input_doc.root_document_id: if self.input_doc.root_document_id:
# If this is a new version of an existing document, we need # If this is a new version of an existing document, we need
@@ -703,7 +695,7 @@ class ConsumerPlugin(
# After everything is in the database, copy the files into # After everything is in the database, copy the files into
# place. If this fails, we'll also rollback the transaction. # place. If this fails, we'll also rollback the transaction.
with media_lock(): with FileLock(settings.MEDIA_LOCK):
generated_filename = generate_unique_filename(document) generated_filename = generate_unique_filename(document)
if ( if (
len(str(generated_filename)) len(str(generated_filename))
@@ -767,9 +759,10 @@ class ConsumerPlugin(
document.archive_path, document.archive_path,
) )
# Any filename follow-up from this save is deferred by # Don't save with the lock active. Saving will cause the file
# the scope above until after commit. # renaming logic to acquire the lock as well.
document.save(write_file_paths=True) # This triggers things like file renaming
document.save()
if document.root_document_id: if document.root_document_id:
document_updated.send( document_updated.send(
@@ -810,11 +803,6 @@ class ConsumerPlugin(
exception=e, exception=e,
) )
# The storage scope above nests under the consume task's scope, so its
# moves are still pending; place the file before the script reads its path.
flush_pending_file_moves()
document.refresh_from_db()
self.run_post_consume_script(document) self.run_post_consume_script(document)
self.log.info(f"Document {document} consumption finished") self.log.info(f"Document {document} consumption finished")
@@ -827,6 +815,9 @@ class ConsumerPlugin(
document.id, document.id,
) )
# Return the most up to date fields
document.refresh_from_db()
return ConsumeFileSuccessResult(document_id=document.pk) return ConsumeFileSuccessResult(document_id=document.pk)
def _parse_title_placeholders(self, title: str) -> str: def _parse_title_placeholders(self, title: str) -> str:
-464
View File
@@ -1,464 +0,0 @@
"""
Document file moves, coalesced to once per unit of work.
Every save that can change a document's generated filename asks for a file
move through request_file_move(). Inside a scope opened by defer_file_moves()
(a request, a Celery task, the consumer, or a per-item command step) the
request is only recorded, and the outermost scope moves each recorded document
once, after its own writes have committed. Outside any scope the move happens
immediately.
The mover always takes media.lock first and then reads the document fresh, so
it starts from the latest committed state and never waits on media.lock while
its own transaction holds document rows.
"""
from __future__ import annotations
import logging
import shutil
import threading
from contextlib import asynccontextmanager
from contextlib import contextmanager
from contextvars import ContextVar
from pathlib import Path
from typing import TYPE_CHECKING
from typing import Final
from asgiref.sync import sync_to_async
from django.conf import settings
from django.db import DatabaseError
from django.db import transaction
from django.utils import timezone
from filelock import Timeout
from documents.caching import clear_document_caches
from documents.file_handling import create_source_path_directory
from documents.file_handling import delete_empty_directories
from documents.file_handling import generate_filename
from documents.file_handling import generate_unique_filename
from documents.media_lock import media_lock
from documents.models import Document
from documents.utils import compute_checksum
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from collections.abc import Iterator
logger = logging.getLogger("paperless.handlers")
MEDIA_LOCK_TIMEOUT_SECONDS: Final[int] = 300
class CannotMoveFilesException(Exception):
pass
def in_user_transaction() -> bool:
"""
True when the default connection is inside an atomic block that was not
created by Django's test case machinery.
Django marks the atomic blocks TestCase (and pytest-django) wrap around
each test with _from_testcase. Ignoring them keeps tests on the same path
as production.
"""
connection = transaction.get_connection()
return any(not block._from_testcase for block in connection.atomic_blocks)
class FileMoveCollector:
"""Pending document pks for one file move scope."""
def __init__(self) -> None:
self._pending: set[int] = set()
self._lock = threading.Lock()
self.closed = False
def add(self, pk: int) -> bool:
"""Record a pending move. Returns False if the collector is closed."""
with self._lock:
if self.closed:
return False
self._pending.add(pk)
return True
def take_pending(self) -> list[int]:
"""Remove and return the pending pks, sorted, leaving the collector open."""
with self._lock:
pending = sorted(self._pending)
self._pending.clear()
return pending
def close(self) -> list[int]:
"""Close the collector and return what was still pending, sorted."""
with self._lock:
self.closed = True
pending = sorted(self._pending)
self._pending.clear()
return pending
# Only the scope context managers call set() / reset(). Everything else reads
# it and mutates the collector, so adds made in a copied context (thread or
# task hops) are visible to the scope that flushes.
_active_collector: ContextVar[FileMoveCollector | None] = ContextVar(
"paperless_file_move_collector",
default=None,
)
def _path_matches_checksum(path: Path, checksum: str | None) -> bool:
if checksum is None or not path.is_file():
return False
return compute_checksum(path) == checksum
def move_files_for_document(pk: int) -> Document | None:
"""
Move the document's original and archive to the paths its current
committed state generates, then do the same for its versions.
Returns the fresh instance with its final filename columns, or None when
the document does not exist, has no filename yet, or the lock timed out.
"""
# Cheap check without the lock: most saves during consumption happen
# before the file is placed. filename is never cleared once set, so this
# stays true under the lock. values_list() keeps the check off
# Document.global_objects.filter(), which tests patch to inject database
# errors into and observe the mover's own writes.
filename = (
Document.global_objects.values_list("filename", flat=True).filter(pk=pk).first()
)
if not filename:
return None
try:
with media_lock(timeout=MEDIA_LOCK_TIMEOUT_SECONDS):
try:
document = Document.global_objects.get(pk=pk)
except Document.DoesNotExist:
logger.debug(f"Document {pk} no longer exists, skipping file move")
return None
_move_document_files(document)
except Timeout:
logger.error(
f"Timed out after {MEDIA_LOCK_TIMEOUT_SECONDS}s waiting for "
f"{settings.MEDIA_LOCK} to move files of document {pk}; the file "
f"keeps its current name and will be renamed on its next save",
)
return None
# Keep version files in sync with root
if document.root_document_id is None:
version_pks = list(
Document.global_objects.values_list("pk", flat=True).filter(
root_document_id=pk,
),
)
for version_pk in version_pks:
move_files_for_document(version_pk)
return document
def _flush(pks: list[int]) -> None:
"""Move each document; one failure never stops the rest or raises."""
for pk in pks:
try:
move_files_for_document(pk)
except Exception:
logger.exception(f"Unexpected error moving files of document {pk}")
def _flush_after_commit(pks: list[int]) -> None:
if not pks:
return
if in_user_transaction():
transaction.on_commit(lambda: _flush(pks))
else:
_flush(pks)
def _close_and_flush(collector: FileMoveCollector) -> None:
_flush_after_commit(collector.close())
@contextmanager
def defer_file_moves() -> Iterator[None]:
"""
Collect file move requests and perform each once when the outermost scope
exits, after its own writes have committed.
"""
current = _active_collector.get()
if current is not None and not current.closed:
yield
return
collector = FileMoveCollector()
token = _active_collector.set(collector)
try:
yield
finally:
_active_collector.reset(token)
_close_and_flush(collector)
def flush_pending_file_moves() -> None:
"""
Perform the current scope's pending moves now, keeping the scope open.
From an eager task inside a request this flushes the request's collector
early, which is harmless: moves still run after commit, on fresh state.
"""
collector = _active_collector.get()
if collector is None:
return
_flush_after_commit(collector.take_pending())
def _validate_move(
document: Document,
old_path: Path,
new_path: Path,
root: Path,
) -> None:
if not new_path.is_relative_to(root):
msg = (
f"Document {document!s}: Refusing to move file outside root {root}: "
f"{new_path}."
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
if not old_path.is_file():
# Can't do anything if the old file does not exist anymore.
msg = f"Document {document!s}: File {old_path} doesn't exist."
logger.fatal(msg)
raise CannotMoveFilesException(msg)
if new_path.is_file():
# Can't do anything if the new file already exists. Skip updating file.
msg = f"Document {document!s}: Cannot rename file since target path {new_path} already exists."
logger.warning(msg)
raise CannotMoveFilesException(msg)
def _move_document_files(document: Document) -> None:
try:
old_filename = document.filename
old_source_path = document.source_path
move_original = False
original_already_moved = False
old_archive_filename = document.archive_filename
old_archive_path = document.archive_path
move_archive = False
archive_already_moved = False
candidate_filename = generate_filename(document)
if len(str(candidate_filename)) > Document.MAX_STORED_FILENAME_LENGTH:
msg = (
f"Document {document!s}: Generated filename exceeds db path "
f"limit ({len(str(candidate_filename))} > "
f"{Document.MAX_STORED_FILENAME_LENGTH}): {candidate_filename!s}"
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
candidate_source_path = (settings.ORIGINALS_DIR / candidate_filename).resolve()
if candidate_filename == Path(old_filename):
new_filename = Path(old_filename)
elif (
candidate_source_path.exists() and candidate_source_path != old_source_path
):
if not old_source_path.is_file() and _path_matches_checksum(
candidate_source_path,
document.checksum,
):
new_filename = candidate_filename
original_already_moved = True
else:
# Only fall back to unique search when there is an actual conflict
new_filename = generate_unique_filename(document)
else:
new_filename = candidate_filename
# Need to convert to string to be able to save it to the db
document.filename = str(new_filename)
move_original = old_filename != document.filename and not original_already_moved
if document.has_archive_version:
archive_candidate = generate_filename(document, archive_filename=True)
if len(str(archive_candidate)) > Document.MAX_STORED_FILENAME_LENGTH:
msg = (
f"Document {document!s}: Generated archive filename exceeds "
f"db path limit ({len(str(archive_candidate))} > "
f"{Document.MAX_STORED_FILENAME_LENGTH}): {archive_candidate!s}"
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
archive_candidate_path = (
settings.ARCHIVE_DIR / archive_candidate
).resolve()
if archive_candidate == Path(old_archive_filename):
new_archive_filename = Path(old_archive_filename)
elif (
archive_candidate_path.exists()
and archive_candidate_path != old_archive_path
):
if not old_archive_path.is_file() and _path_matches_checksum(
archive_candidate_path,
document.archive_checksum,
):
new_archive_filename = archive_candidate
archive_already_moved = True
else:
new_archive_filename = generate_unique_filename(
document,
archive_filename=True,
)
else:
new_archive_filename = archive_candidate
document.archive_filename = str(new_archive_filename)
move_archive = (
old_archive_filename != document.archive_filename
and not archive_already_moved
)
else:
move_archive = False
if not move_original and not move_archive:
updates: dict[str, object] = {}
if old_filename != document.filename:
updates["filename"] = document.filename
if old_archive_filename != document.archive_filename:
updates["archive_filename"] = document.archive_filename
if updates:
# global_objects so a trashed document's path is fixed too.
updates["modified"] = timezone.now()
# Don't save() here to prevent infinite recursion.
Document.global_objects.filter(pk=document.pk).update(**updates)
return
if move_original:
_validate_move(
document,
old_source_path,
document.source_path,
settings.ORIGINALS_DIR,
)
create_source_path_directory(document.source_path)
shutil.move(old_source_path, document.source_path)
if move_archive:
_validate_move(
document,
old_archive_path,
document.archive_path,
settings.ARCHIVE_DIR,
)
create_source_path_directory(document.archive_path)
shutil.move(old_archive_path, document.archive_path)
# Don't save() here to prevent infinite recursion.
Document.global_objects.filter(pk=document.pk).update(
filename=document.filename,
archive_filename=document.archive_filename,
modified=timezone.now(),
)
# Clear any caching for this document. Slightly overkill, but not terrible
clear_document_caches(document.pk)
except (OSError, DatabaseError, CannotMoveFilesException) as e:
logger.warning(f"Exception during file handling: {e}")
# This happens when either:
# - moving the files failed due to file system errors
# - saving to the database failed due to database errors
# In both cases, we need to revert to the original state.
# Try to move files to their original location.
try:
if move_original and document.source_path.is_file():
logger.info("Restoring previous original path")
shutil.move(document.source_path, old_source_path)
if move_archive and document.archive_path.is_file():
logger.info("Restoring previous archive path")
shutil.move(document.archive_path, old_archive_path)
except Exception:
# This is fine, since:
# A: if we managed to move source from A to B, we will also
# manage to move it from B to A. If not, we have a serious
# issue that's going to get caught by the santiy checker.
# All files remain in place and will never be overwritten,
# so this is not the end of the world.
# B: if moving the original file failed, nothing has changed
# anyway.
pass
# restore old values on the document
document.filename = old_filename
document.archive_filename = old_archive_filename
# finally, remove any empty sub folders. This will do nothing if
# something has failed above.
if not old_source_path.is_file():
delete_empty_directories(
Path(old_source_path).parent,
root=settings.ORIGINALS_DIR,
)
if document.has_archive_version and not old_archive_path.is_file():
delete_empty_directories(
Path(old_archive_path).parent,
root=settings.ARCHIVE_DIR,
)
def request_file_move(document: Document) -> None:
"""
Ask for the document's files to follow its committed state.
Inside an open scope the request is recorded and performed when the scope
ends. Otherwise the move happens now and only the final filename columns
are copied onto the caller's instance, so source_path is current after
save() while other in-memory fields are left alone.
"""
collector = _active_collector.get()
if collector is not None and collector.add(document.pk):
return
moved = move_files_for_document(document.pk)
if moved is not None:
document.filename = moved.filename
document.archive_filename = moved.archive_filename
@asynccontextmanager
async def adefer_file_moves() -> AsyncIterator[None]:
"""
Async twin of defer_file_moves().
When something was collected, the flush runs through sync_to_async: its
transaction check needs the thread owning the DB connection, and the
blocking media.lock must never run on the event loop.
"""
current = _active_collector.get()
if current is not None and not current.closed:
yield
return
collector = FileMoveCollector()
token = _active_collector.set(collector)
try:
yield
finally:
_active_collector.reset(token)
pks = collector.close()
if pks:
await sync_to_async(_flush_after_commit, thread_sensitive=True)(pks)
+3 -3
View File
@@ -4,9 +4,9 @@ from dataclasses import dataclass
from email import message_from_bytes from email import message_from_bytes
from pathlib import Path from pathlib import Path
from django.conf import settings
from django.core.mail import EmailMessage from django.core.mail import EmailMessage
from filelock import FileLock
from documents.media_lock import media_lock
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -48,7 +48,7 @@ def send_email(
used_filenames: set[str] = set() used_filenames: set[str] = set()
# Something could be renaming the file concurrently so it can't be attached # Something could be renaming the file concurrently so it can't be attached
with media_lock(): with FileLock(settings.MEDIA_LOCK):
for attachment in attachments: for attachment in attachments:
filename = _get_unique_filename( filename = _get_unique_filename(
attachment.friendly_name, attachment.friendly_name,
@@ -18,11 +18,10 @@ from django.core import serializers
from django.core.management.base import CommandError from django.core.management.base import CommandError
from django.db import transaction from django.db import transaction
from django.utils import timezone from django.utils import timezone
from filelock import FileLock
from guardian.models import GroupObjectPermission from guardian.models import GroupObjectPermission
from guardian.models import UserObjectPermission from guardian.models import UserObjectPermission
from documents.media_lock import media_lock
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Generator from collections.abc import Generator
@@ -329,7 +328,7 @@ class Command(CryptMixin, PaperlessCommand):
) )
# Prevent any ongoing changes in the documents while exporting # Prevent any ongoing changes in the documents while exporting
with media_lock(), sink: with FileLock(settings.MEDIA_LOCK), sink:
self.dump(sink) self.dump(sink)
def dump(self, sink: ExportSink) -> None: def dump(self, sink: ExportSink) -> None:
@@ -29,6 +29,7 @@ from django.db.models import GeneratedField
from django.db.models import Model from django.db.models import Model
from django.db.models.signals import m2m_changed from django.db.models.signals import m2m_changed
from django.db.models.signals import post_save from django.db.models.signals import post_save
from filelock import FileLock
from guardian.shortcuts import clear_ct_cache from guardian.shortcuts import clear_ct_cache
from documents.export.compression import compress_type_readable from documents.export.compression import compress_type_readable
@@ -36,7 +37,6 @@ from documents.export.compression import unreadable_method_names
from documents.file_handling import create_source_path_directory from documents.file_handling import create_source_path_directory
from documents.management.commands.base import PaperlessCommand from documents.management.commands.base import PaperlessCommand
from documents.management.commands.mixins import CryptMixin from documents.management.commands.mixins import CryptMixin
from documents.media_lock import media_lock
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import CustomField from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
@@ -644,7 +644,7 @@ class Command(CryptMixin, PaperlessCommand):
else: else:
archive_path = None archive_path = None
with media_lock(): with FileLock(settings.MEDIA_LOCK):
if Path(document.source_path).is_file(): if Path(document.source_path).is_file():
raise FileExistsError(document.source_path) raise FileExistsError(document.source_path)
@@ -678,7 +678,7 @@ class Command(CryptMixin, PaperlessCommand):
f"Share link bundle {bundle.pk} does not have a valid file path.", f"Share link bundle {bundle.pk} does not have a valid file path.",
) )
with media_lock(): with FileLock(settings.MEDIA_LOCK):
bundle_target_path.parent.mkdir(parents=True, exist_ok=True) bundle_target_path.parent.mkdir(parents=True, exist_ok=True)
copy_file_with_basic_stats( copy_file_with_basic_stats(
bundle_source_path, bundle_source_path,
@@ -1,4 +1,5 @@
from documents.file_moves import move_files_for_document from django.db.models.signals import post_save
from documents.management.commands.base import PaperlessCommand from documents.management.commands.base import PaperlessCommand
from documents.models import Document from documents.models import Document
@@ -10,9 +11,5 @@ class Command(PaperlessCommand):
supports_multiprocessing = False supports_multiprocessing = False
def handle(self, *args, **options): def handle(self, *args, **options):
for document in self.track( for document in self.track(Document.objects.all(), description="Renaming..."):
# move_files_for_document handles a root's versions itself post_save.send(Document, instance=document, created=False)
Document.objects.filter(root_document__isnull=True).only("pk"),
description="Renaming...",
):
move_files_for_document(document.pk)
@@ -9,7 +9,6 @@ from rich.table import Table
from rich.text import Text from rich.text import Text
from documents.classifier import load_classifier from documents.classifier import load_classifier
from documents.file_moves import defer_file_moves
from documents.management.commands.base import PaperlessCommand from documents.management.commands.base import PaperlessCommand
from documents.models import Document from documents.models import Document
from documents.signals.handlers import set_correspondent from documents.signals.handlers import set_correspondent
@@ -274,63 +273,61 @@ class Command(PaperlessCommand):
description="Retagging...", description="Retagging...",
stats_renderer=render_stats, stats_renderer=render_stats,
): ):
# One file move per document, after all of its assignments. suggestion = DocumentSuggestion(document=document)
with defer_file_moves():
suggestion = DocumentSuggestion(document=document)
if do_correspondent: if do_correspondent:
correspondent = set_correspondent( correspondent = set_correspondent(
None, None,
document, document,
classifier=classifier, classifier=classifier,
replace=overwrite, replace=overwrite,
use_first=use_first, use_first=use_first,
dry_run=suggest, dry_run=suggest,
) )
if correspondent is not None: if correspondent is not None:
stats.correspondents += 1 stats.correspondents += 1
suggestion.correspondent = correspondent suggestion.correspondent = correspondent
if do_document_type: if do_document_type:
document_type = set_document_type( document_type = set_document_type(
None, None,
document, document,
classifier=classifier, classifier=classifier,
replace=overwrite, replace=overwrite,
use_first=use_first, use_first=use_first,
dry_run=suggest, dry_run=suggest,
) )
if document_type is not None: if document_type is not None:
stats.document_types += 1 stats.document_types += 1
suggestion.document_type = document_type suggestion.document_type = document_type
if do_tags: if do_tags:
tags_to_add, tags_to_remove = set_tags( tags_to_add, tags_to_remove = set_tags(
None, None,
document, document,
classifier=classifier, classifier=classifier,
replace=overwrite, replace=overwrite,
dry_run=suggest, dry_run=suggest,
) )
stats.tags_added += len(tags_to_add) stats.tags_added += len(tags_to_add)
stats.tags_removed += len(tags_to_remove) stats.tags_removed += len(tags_to_remove)
suggestion.tags_to_add = frozenset(tags_to_add) suggestion.tags_to_add = frozenset(tags_to_add)
suggestion.tags_to_remove = frozenset(tags_to_remove) suggestion.tags_to_remove = frozenset(tags_to_remove)
if do_storage_path: if do_storage_path:
storage_path = set_storage_path( storage_path = set_storage_path(
None, None,
document, document,
classifier=classifier, classifier=classifier,
replace=overwrite, replace=overwrite,
use_first=use_first, use_first=use_first,
dry_run=suggest, dry_run=suggest,
) )
if storage_path is not None: if storage_path is not None:
stats.storage_paths += 1 stats.storage_paths += 1
suggestion.storage_path = storage_path suggestion.storage_path = storage_path
stats.documents_processed += 1 stats.documents_processed += 1
if suggest: if suggest:
suggestions.append(suggestion) suggestions.append(suggestion)
-36
View File
@@ -1,36 +0,0 @@
"""
The global media lock, reentrant within a thread.
Code that both needs media.lock and opens a transaction that locks document
rows must take media.lock first, so no transaction ever waits on it while
holding rows. Reentrancy lets code already holding the lock (for example the
consumer) call into code that takes it again (for example a workflow email
action attaching files) without blocking itself.
"""
from __future__ import annotations
from contextlib import contextmanager
from typing import TYPE_CHECKING
from django.conf import settings
from filelock import FileLock
if TYPE_CHECKING:
from collections.abc import Iterator
@contextmanager
def media_lock(timeout: float = -1) -> Iterator[None]:
"""
Hold settings.MEDIA_LOCK for the block.
is_singleton returns one FileLock instance per path in this process, and
the default thread_local=True keeps the hold count per thread, so the
same thread re-enters while other threads and processes are excluded.
The timeout goes to acquire() because a singleton rejects construction
with different parameters. A negative timeout waits forever.
"""
lock = FileLock(settings.MEDIA_LOCK, is_singleton=True)
with lock.acquire(timeout=timeout):
yield
@@ -0,0 +1,311 @@
# Generated by Django 5.2.17 on 2026-10-20 20:29
import django.core.validators
import django.db.models.deletion
import django.utils.timezone
from django.conf import settings
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0027_documentbarcode"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name="OcrTemplate",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("name", models.CharField(max_length=128, verbose_name="name")),
(
"match",
models.CharField(blank=True, max_length=256, verbose_name="match"),
),
(
"matching_algorithm",
models.PositiveSmallIntegerField(
choices=[
(0, "None"),
(1, "Any word"),
(2, "All words"),
(3, "Exact match"),
(4, "Regular expression"),
(5, "Fuzzy word"),
(6, "Automatic"),
],
default=1,
verbose_name="matching algorithm",
),
),
(
"is_insensitive",
models.BooleanField(default=True, verbose_name="is insensitive"),
),
(
"source_width",
models.PositiveIntegerField(
help_text="Width of the image the zones were drawn on (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="source width",
),
),
(
"source_height",
models.PositiveIntegerField(
help_text="Height of the image the zones were drawn on (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="source height",
),
),
("enabled", models.BooleanField(default=True, verbose_name="enabled")),
(
"combine_formats",
models.JSONField(
blank=True,
default=dict,
help_text="Per-target format strings for combining several zones into one field, keyed by target (custom field id, or 'title'/'asn'/'created'). Tokens like {Zone Name} are replaced with that zone's value.",
verbose_name="combine formats",
),
),
(
"created",
models.DateTimeField(
db_index=True,
default=django.utils.timezone.now,
editable=False,
verbose_name="created",
),
),
(
"updated",
models.DateTimeField(auto_now=True, verbose_name="updated"),
),
(
"owner",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
to=settings.AUTH_USER_MODEL,
verbose_name="owner",
),
),
(
"sample_document",
models.ForeignKey(
blank=True,
help_text="Document used for previewing zones in the editor",
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="+",
to="documents.document",
verbose_name="sample document",
),
),
],
options={
"verbose_name": "OCR template",
"verbose_name_plural": "OCR templates",
"ordering": ("name",),
"abstract": False,
},
),
migrations.CreateModel(
name="OcrTemplateZone",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"name",
models.CharField(
help_text="Descriptive name for this zone (e.g. 'Invoice Number')",
max_length=128,
verbose_name="zone name",
),
),
(
"target",
models.CharField(
choices=[
("custom_field", "Custom field"),
("title", "Title"),
("asn", "Archive serial number"),
("created", "Date created"),
],
default="custom_field",
help_text="Where the extracted value is written: a custom field, or a built-in document field (title, ASN, created date)",
max_length=20,
verbose_name="target",
),
),
(
"page",
models.IntegerField(
blank=True,
help_text="Page (1 = first, -1 = last; blank uses the template default)",
null=True,
verbose_name="page",
),
),
(
"x",
models.PositiveIntegerField(
help_text="Left edge (px)",
verbose_name="x",
),
),
(
"y",
models.PositiveIntegerField(
help_text="Top edge (px)",
verbose_name="y",
),
),
(
"width",
models.PositiveIntegerField(
help_text="Zone width (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="width",
),
),
(
"height",
models.PositiveIntegerField(
help_text="Zone height (px)",
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="height",
),
),
(
"zone_source_width",
models.PositiveIntegerField(
blank=True,
help_text="Width of the page image this zone was drawn on (px). Falls back to template source_width if unset.",
null=True,
verbose_name="zone source width",
),
),
(
"zone_source_height",
models.PositiveIntegerField(
blank=True,
help_text="Height of the page image this zone was drawn on (px). Falls back to template source_height if unset.",
null=True,
verbose_name="zone source height",
),
),
(
"ocr_language",
models.CharField(
default="eng",
help_text="Tesseract language code(s), e.g. 'deu+eng'",
max_length=20,
verbose_name="OCR language",
),
),
(
"transform",
models.CharField(
choices=[
("none", "None"),
("strip", "Strip whitespace"),
("uppercase", "Uppercase"),
("lowercase", "Lowercase"),
("numeric", "Numeric only"),
(
"strip_punctuation",
"Remove leading/trailing punctuation",
),
("date", "Parse date"),
("qr_code", "Read QR/barcode"),
],
default="strip",
max_length=20,
verbose_name="transform",
),
),
(
"date_format",
models.CharField(
blank=True,
default="",
help_text="Python strptime format for the 'Parse date' transform (e.g. %d.%m.%Y). Blank = auto-detect.",
max_length=64,
verbose_name="date format",
),
),
(
"validation_regex",
models.CharField(
blank=True,
default="",
help_text="Optional regex pattern — extracted text is only accepted if it matches",
max_length=256,
verbose_name="validation regex",
),
),
("order", models.PositiveIntegerField(default=0, verbose_name="order")),
(
"custom_field",
models.ForeignKey(
blank=True,
help_text="Target custom field (only used when target is 'custom_field')",
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="ocr_zones",
to="documents.customfield",
verbose_name="custom field",
),
),
(
"template",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="zones",
to="documents.ocrtemplate",
verbose_name="template",
),
),
],
options={
"verbose_name": "OCR template zone",
"verbose_name_plural": "OCR template zones",
"ordering": ("template", "order"),
},
),
migrations.AddConstraint(
model_name="ocrtemplate",
constraint=models.UniqueConstraint(
fields=("name", "owner"),
name="documents_ocrtemplate_unique_name_owner",
),
),
migrations.AddConstraint(
model_name="ocrtemplate",
constraint=models.UniqueConstraint(
condition=models.Q(("owner__isnull", True)),
fields=("name",),
name="documents_ocrtemplate_name_uniq",
),
),
]
+212 -32
View File
@@ -1,7 +1,6 @@
import datetime import datetime
import uuid import uuid
from pathlib import Path from pathlib import Path
from typing import Any
from typing import Final from typing import Final
import pathvalidate import pathvalidate
@@ -158,12 +157,6 @@ class StoragePath(MatchingModel):
class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-missing] class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-missing]
MAX_STORED_FILENAME_LENGTH: Final[int] = 1024 MAX_STORED_FILENAME_LENGTH: Final[int] = 1024
# Written only by the file mover, the consumer's file placement and the
# archiver. A full save() of a stale instance must never write them, or
# the DB can point at a path the file has already been moved away from.
FILE_PATH_FIELDS: Final[frozenset[str]] = frozenset(
{"filename", "archive_filename"},
)
# Per-component limit of common filesystems (ext4, XFS, btrfs, ZFS, APFS) # Per-component limit of common filesystems (ext4, XFS, btrfs, ZFS, APFS)
MAX_PUBLIC_FILENAME_BYTES: Final[int] = 255 MAX_PUBLIC_FILENAME_BYTES: Final[int] = 255
@@ -375,31 +368,6 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
res += f" {self.title}" res += f" {self.title}"
return res return res
def save(self, *, write_file_paths: bool = False, **kwargs: Any) -> None:
"""
Save the document without writing filename / archive_filename on an
existing row, unless write_file_paths=True or update_fields names them.
This mirrors how Django narrows a save on a deferred instance: only the
loaded, concrete, non-generated, non-primary-key fields are written.
"""
if (
not write_file_paths
and not self._state.adding
and kwargs.get("update_fields") is None
and not kwargs.get("force_insert", False)
):
deferred = self.get_deferred_fields()
kwargs["update_fields"] = [
field.attname
for field in self._meta.concrete_fields
if not field.primary_key
and not field.generated
and field.name not in self.FILE_PATH_FIELDS
and field.attname not in deferred
]
super().save(**kwargs)
def get_effective_barcodes(self) -> list["DocumentBarcode"]: def get_effective_barcodes(self) -> list["DocumentBarcode"]:
""" """
Returns the stored barcodes for the document, like Returns the stored barcodes for the document, like
@@ -2181,3 +2149,215 @@ class WorkflowRun(SoftDeleteModel):
def __str__(self) -> str: def __str__(self) -> str:
return f"WorkflowRun of {self.workflow} at {self.run_at} on {self.document}" return f"WorkflowRun of {self.workflow} at {self.run_at} on {self.document}"
class OcrTemplate(MatchingModel):
"""Matching rules and extraction zones for documents with a common layout."""
source_width = models.PositiveIntegerField(
_("source width"),
validators=[MinValueValidator(1)],
help_text=_("Width of the image the zones were drawn on (px)"),
)
source_height = models.PositiveIntegerField(
_("source height"),
validators=[MinValueValidator(1)],
help_text=_("Height of the image the zones were drawn on (px)"),
)
sample_document = models.ForeignKey(
"documents.Document",
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="+",
verbose_name=_("sample document"),
help_text=_("Document used for previewing zones in the editor"),
)
enabled = models.BooleanField(_("enabled"), default=True)
combine_formats = models.JSONField(
_("combine formats"),
default=dict,
blank=True,
help_text=_(
"Per-target format strings for combining several zones into one "
"field, keyed by target (custom field id, or 'title'/'asn'/'created'). "
"Tokens like {Zone Name} are replaced with that zone's value.",
),
)
created = models.DateTimeField(
_("created"),
default=timezone.now,
db_index=True,
editable=False,
)
updated = models.DateTimeField(
_("updated"),
auto_now=True,
)
class Meta(MatchingModel.Meta):
verbose_name = _("OCR template")
verbose_name_plural = _("OCR templates")
class OcrTemplateZone(models.Model):
"""A page region and the field its extracted value is written to."""
template = models.ForeignKey(
OcrTemplate,
on_delete=models.CASCADE,
related_name="zones",
verbose_name=_("template"),
)
name = models.CharField(
_("zone name"),
max_length=128,
help_text=_("Descriptive name for this zone (e.g. 'Invoice Number')"),
)
class TargetType(models.TextChoices):
CUSTOM_FIELD = ("custom_field", _("Custom field"))
TITLE = ("title", _("Title"))
ASN = ("asn", _("Archive serial number"))
CREATED = ("created", _("Date created"))
target = models.CharField(
_("target"),
max_length=20,
choices=TargetType.choices,
default=TargetType.CUSTOM_FIELD,
help_text=_(
"Where the extracted value is written: a custom field, or a "
"built-in document field (title, ASN, created date)",
),
)
custom_field = models.ForeignKey(
"documents.CustomField",
on_delete=models.CASCADE,
related_name="ocr_zones",
verbose_name=_("custom field"),
null=True,
blank=True,
help_text=_("Target custom field (only used when target is 'custom_field')"),
)
page = models.IntegerField(
_("page"),
null=True,
blank=True,
help_text=_("Page (1 = first, -1 = last; blank uses the template default)"),
)
x = models.PositiveIntegerField(_("x"), help_text=_("Left edge (px)"))
y = models.PositiveIntegerField(_("y"), help_text=_("Top edge (px)"))
width = models.PositiveIntegerField(
_("width"),
validators=[MinValueValidator(1)],
help_text=_("Zone width (px)"),
)
height = models.PositiveIntegerField(
_("height"),
validators=[MinValueValidator(1)],
help_text=_("Zone height (px)"),
)
# Mixed-size PDFs need each zone's original page dimensions for scaling.
zone_source_width = models.PositiveIntegerField(
_("zone source width"),
null=True,
blank=True,
help_text=_(
"Width of the page image this zone was drawn on (px). "
"Falls back to template source_width if unset.",
),
)
zone_source_height = models.PositiveIntegerField(
_("zone source height"),
null=True,
blank=True,
help_text=_(
"Height of the page image this zone was drawn on (px). "
"Falls back to template source_height if unset.",
),
)
ocr_language = models.CharField(
_("OCR language"),
max_length=20,
default="eng",
help_text=_("Tesseract language code(s), e.g. 'deu+eng'"),
)
class TransformType(models.TextChoices):
NONE = ("none", _("None"))
STRIP = ("strip", _("Strip whitespace"))
UPPERCASE = ("uppercase", _("Uppercase"))
LOWERCASE = ("lowercase", _("Lowercase"))
NUMERIC = ("numeric", _("Numeric only"))
STRIP_PUNCTUATION = (
"strip_punctuation",
_("Remove leading/trailing punctuation"),
)
DATE = ("date", _("Parse date"))
QR_CODE = ("qr_code", _("Read QR/barcode"))
transform = models.CharField(
_("transform"),
max_length=20,
choices=TransformType.choices,
default=TransformType.STRIP,
)
date_format = models.CharField(
_("date format"),
max_length=64,
blank=True,
default="",
help_text=_(
"Python strptime format for the 'Parse date' transform "
"(e.g. %d.%m.%Y). Blank = auto-detect.",
),
)
validation_regex = models.CharField(
_("validation regex"),
max_length=256,
blank=True,
default="",
help_text=_(
"Optional regex pattern — extracted text is only accepted if it matches",
),
)
order = models.PositiveIntegerField(_("order"), default=0)
class Meta:
ordering = ("template", "order")
verbose_name = _("OCR template zone")
verbose_name_plural = _("OCR template zones")
def __str__(self) -> str:
return f"{self.template.name} -> {self.name}"
# Document links and select fields require object references, not OCR text.
OCR_SUPPORTED_FIELD_TYPES = frozenset(
{
CustomField.FieldDataType.STRING,
CustomField.FieldDataType.URL,
CustomField.FieldDataType.DATE,
CustomField.FieldDataType.INT,
CustomField.FieldDataType.FLOAT,
CustomField.FieldDataType.MONETARY,
CustomField.FieldDataType.LONG_TEXT,
CustomField.FieldDataType.BOOL,
},
)
+27
View File
@@ -77,6 +77,33 @@ def safe_regex_match(pattern: str, text: str, *, flags: int = 0):
return None return None
def safe_regex_fullmatch(pattern: str, text: str, *, flags: int = 0):
"""
Run a regex full match with a timeout. Returns a match object or None.
Validation errors and timeouts are logged and treated as no match.
"""
try:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
logger.exception(
"Error while processing regular expression %s: %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
exc,
)
return None
try:
return compiled.fullmatch(text, timeout=REGEX_TIMEOUT_SECONDS)
except TimeoutError:
logger.warning(
"Regular expression matching timed out for pattern %s",
textwrap.shorten(pattern, width=80, placeholder="…"),
)
return None
def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str | None: def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str | None:
""" """
Run a regex substitution with a timeout. Returns the substituted string, Run a regex substitution with a timeout. Returns the substituted string,
+194
View File
@@ -23,6 +23,7 @@ from django.core.validators import MaxValueValidator
from django.core.validators import MinValueValidator from django.core.validators import MinValueValidator
from django.core.validators import RegexValidator from django.core.validators import RegexValidator
from django.core.validators import integer_validator from django.core.validators import integer_validator
from django.db import transaction
from django.db.models import Count from django.db.models import Count
from django.db.models import Q from django.db.models import Q
from django.db.models.functions import Lower from django.db.models.functions import Lower
@@ -54,6 +55,7 @@ if settings.AUDIT_LOG_ENABLED:
from documents import bulk_edit from documents import bulk_edit
from documents.data_models import DocumentSource from documents.data_models import DocumentSource
from documents.filters import CustomFieldQueryParser from documents.filters import CustomFieldQueryParser
from documents.models import OCR_SUPPORTED_FIELD_TYPES
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import CustomField from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
@@ -62,6 +64,8 @@ from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import MatchingModel from documents.models import MatchingModel
from documents.models import Note from documents.models import Note
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import SavedView from documents.models import SavedView
from documents.models import SavedViewFilterRule from documents.models import SavedViewFilterRule
@@ -1821,6 +1825,18 @@ class ReprocessDocumentsSerializer(DocumentSelectionSerializer):
remote_ocr = serializers.BooleanField(required=False, default=False) remote_ocr = serializers.BooleanField(required=False, default=False)
class RunZoneOcrSerializer(serializers.Serializer):
template = serializers.PrimaryKeyRelatedField(
queryset=OcrTemplate.objects.prefetch_related("zones", "zones__custom_field"),
required=False,
allow_null=True,
)
class RunZoneOcrDocumentsSerializer(DocumentSelectionSerializer, RunZoneOcrSerializer):
pass
class BulkEditSerializer( class BulkEditSerializer(
SerializerWithPerms, SerializerWithPerms,
DocumentSelectionSerializer, DocumentSelectionSerializer,
@@ -3698,3 +3714,181 @@ class StoragePathTestSerializer(SerializerWithPerms):
document_field.queryset = Document.objects.filter( document_field.queryset = Document.objects.filter(
id__in=permitted_document_ids(user), id__in=permitted_document_ids(user),
) )
class OcrTemplateZoneSerializer(serializers.ModelSerializer):
class Meta:
model = OcrTemplateZone
fields = [
"id",
"name",
"target",
"custom_field",
"page",
"x",
"y",
"width",
"height",
"ocr_language",
"transform",
"date_format",
"order",
"zone_source_width",
"zone_source_height",
"validation_regex",
]
def validate_custom_field(self, value):
if value is None:
return value
if value.data_type not in OCR_SUPPORTED_FIELD_TYPES:
raise serializers.ValidationError(
f"Custom field type '{value.data_type}' is not supported for OCR extraction. "
f"Use string, integer, float, date, monetary, boolean, URL, or long text.",
)
return value
def validate_page(self, value):
if value is not None and value != -1 and value < 1:
raise serializers.ValidationError(
"Page must be positive or -1 for the last page.",
)
return value
def validate(self, data):
# A template PATCH still replaces each zone in full.
if self.root.partial:
for name, field in self.fields.items():
if field.required and name not in data:
raise serializers.ValidationError(
{name: field.error_messages["required"]},
)
return data
class OcrTemplateZoneTestSerializer(OcrTemplateZoneSerializer):
name = serializers.CharField(default="test")
x = serializers.IntegerField(min_value=0, default=0)
y = serializers.IntegerField(min_value=0, default=0)
width = serializers.IntegerField(min_value=2)
height = serializers.IntegerField(min_value=2)
class Meta(OcrTemplateZoneSerializer.Meta):
fields = [
"name",
"x",
"y",
"width",
"height",
"page",
"ocr_language",
"transform",
"date_format",
"validation_regex",
"zone_source_width",
"zone_source_height",
]
extra_kwargs = {
"zone_source_width": {"min_value": 1},
"zone_source_height": {"min_value": 1},
}
class OcrTemplateTestSerializer(serializers.Serializer):
document = serializers.PrimaryKeyRelatedField(
queryset=Document.objects.select_related("root_document__owner"),
)
zone = OcrTemplateZoneTestSerializer()
class OcrTemplateSerializer(MatchingModelSerializer, OwnedObjectSerializer):
zones = OcrTemplateZoneSerializer(many=True, required=False)
combine_formats = serializers.DictField(
child=serializers.CharField(allow_blank=True, trim_whitespace=False),
required=False,
)
# Templates are never assigned to documents
slug = None
document_count = None
class Meta:
model = OcrTemplate
fields = [
"id",
"name",
"match",
"matching_algorithm",
"is_insensitive",
"source_width",
"source_height",
"sample_document",
"enabled",
"combine_formats",
"created",
"updated",
"zones",
"owner",
"permissions",
"user_can_change",
"set_permissions",
]
read_only_fields = ["created", "updated"]
def validate_matching_algorithm(self, value):
if value == MatchingModel.MATCH_AUTO:
raise serializers.ValidationError(
"Automatic matching is not supported for OCR templates.",
)
return value
def validate(self, data):
data = super().validate(data)
zones = data.get("zones", [])
if (
self.instance
and "zones" not in data
and ("source_width" in data or "source_height" in data)
):
zones = self.instance.zones.values()
source_width = data.get(
"source_width",
getattr(self.instance, "source_width", None),
)
source_height = data.get(
"source_height",
getattr(self.instance, "source_height", None),
)
for zone in zones:
width = zone.get("zone_source_width") or source_width
height = zone.get("zone_source_height") or source_height
if (
zone.get("x", 0) + zone.get("width", 0) > width
or zone.get("y", 0) + zone.get("height", 0) > height
):
raise serializers.ValidationError(
{
"zones": f"Zone '{zone.get('name', '?')}' extends beyond its source dimensions.",
},
)
return data
@transaction.atomic
def create(self, validated_data):
zones_data = validated_data.pop("zones", [])
template = super().create(validated_data)
for zone_data in zones_data:
OcrTemplateZone.objects.create(template=template, **zone_data)
return template
@transaction.atomic
def update(self, instance, validated_data):
zones_data = validated_data.pop("zones", None)
instance = super().update(instance, validated_data)
if zones_data is not None:
instance.zones.all().delete()
for zone_data in zones_data:
OcrTemplateZone.objects.create(template=instance, **zone_data)
return instance
+257 -19
View File
@@ -19,22 +19,25 @@ from celery.signals import worker_process_shutdown
from django.conf import settings from django.conf import settings
from django.contrib.auth.models import Group from django.contrib.auth.models import Group
from django.contrib.auth.models import User from django.contrib.auth.models import User
from django.db import DatabaseError
from django.db import close_old_connections from django.db import close_old_connections
from django.db import connections from django.db import connections
from django.db import models from django.db import models
from django.db.models import Q from django.db.models import Q
from django.dispatch import receiver from django.dispatch import receiver
from django.utils import timezone from django.utils import timezone
from filelock import FileLock
from rest_framework import serializers from rest_framework import serializers
from documents import matching from documents import matching
from documents.caching import clear_document_caches
from documents.caching import invalidate_llm_suggestions_cache from documents.caching import invalidate_llm_suggestions_cache
from documents.caching import invalidate_suggestions_cache from documents.caching import invalidate_suggestions_cache
from documents.data_models import ConsumableDocument from documents.data_models import ConsumableDocument
from documents.file_handling import create_source_path_directory
from documents.file_handling import delete_empty_directories from documents.file_handling import delete_empty_directories
from documents.file_moves import move_files_for_document from documents.file_handling import generate_filename
from documents.file_moves import request_file_move from documents.file_handling import generate_unique_filename
from documents.media_lock import media_lock
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import CustomField from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
@@ -52,6 +55,7 @@ from documents.models import WorkflowTrigger
from documents.permissions import restrict_queryset_to_visible from documents.permissions import restrict_queryset_to_visible
from documents.plugins.helpers import DocumentsStatusManager from documents.plugins.helpers import DocumentsStatusManager
from documents.templating.utils import convert_format_str_to_template_format from documents.templating.utils import convert_format_str_to_template_format
from documents.utils import compute_checksum
from documents.utils import copy_file_with_basic_stats from documents.utils import copy_file_with_basic_stats
from documents.workflows.actions import build_workflow_action_context from documents.workflows.actions import build_workflow_action_context
from documents.workflows.actions import execute_email_action from documents.workflows.actions import execute_email_action
@@ -335,7 +339,7 @@ def set_storage_path(
# see empty_trash in documents/tasks.py for signal handling # see empty_trash in documents/tasks.py for signal handling
def cleanup_document_deletion(sender, instance, **kwargs) -> None: def cleanup_document_deletion(sender, instance, **kwargs) -> None:
with media_lock(): with FileLock(settings.MEDIA_LOCK):
if settings.EMPTY_TRASH_DIR: if settings.EMPTY_TRASH_DIR:
# Find a non-conflicting filename in case a document with the same # Find a non-conflicting filename in case a document with the same
# name was moved to trash earlier # name was moved to trash earlier
@@ -401,6 +405,17 @@ def cleanup_document_deletion(sender, instance, **kwargs) -> None:
) )
class CannotMoveFilesException(Exception):
pass
def _path_matches_checksum(path: Path, checksum: str | None) -> bool:
if checksum is None or not path.is_file():
return False
return compute_checksum(path) == checksum
def _filename_template_uses_custom_fields(doc: Document) -> bool: def _filename_template_uses_custom_fields(doc: Document) -> bool:
template = None template = None
if doc.storage_path is not None: if doc.storage_path is not None:
@@ -423,24 +438,235 @@ def update_filename_and_move_files(
instance: Document | CustomFieldInstance, instance: Document | CustomFieldInstance,
**kwargs, **kwargs,
) -> None: ) -> None:
action = kwargs.get("action")
if action is not None and action not in {"post_add", "post_remove", "post_clear"}:
# m2m_changed fires pre_* and post_* actions; only post_* describe the
# relation state the filename should follow.
return
if kwargs.get("reverse"):
# tag.documents.add(doc): instance is the Tag, pk_set holds Document pks.
for document in Document.objects.filter(pk__in=kwargs.get("pk_set") or ()):
request_file_move(document)
return
if isinstance(instance, CustomFieldInstance): if isinstance(instance, CustomFieldInstance):
if not _filename_template_uses_custom_fields(instance.document): if not _filename_template_uses_custom_fields(instance.document):
return return
instance = instance.document instance = instance.document
request_file_move(instance) def validate_move(instance, old_path: Path, new_path: Path, root: Path) -> None:
if not new_path.is_relative_to(root):
msg = (
f"Document {instance!s}: Refusing to move file outside root {root}: "
f"{new_path}."
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
if not old_path.is_file():
# Can't do anything if the old file does not exist anymore.
msg = f"Document {instance!s}: File {old_path} doesn't exist."
logger.fatal(msg)
raise CannotMoveFilesException(msg)
if new_path.is_file():
# Can't do anything if the new file already exists. Skip updating file.
msg = f"Document {instance!s}: Cannot rename file since target path {new_path} already exists."
logger.warning(msg)
raise CannotMoveFilesException(msg)
if not instance.filename:
# Can't update the filename if there is no filename to begin with
# This happens when the consumer creates a new document.
# The document is modified and saved multiple times, and only after
# everything is done (i.e., the generated filename is final),
# filename will be set to the location where the consumer has put
# the file.
#
# This will in turn cause this logic to move the file where it belongs.
return
with FileLock(settings.MEDIA_LOCK):
# If this was waiting for the lock, the filename or archive_filename
# of this document may have been updated. This happens if multiple updates
# get queued from the UI for the same document
# So freshen up the data before doing anything
instance.refresh_from_db()
try:
old_filename = instance.filename
old_source_path = instance.source_path
move_original = False
original_already_moved = False
old_archive_filename = instance.archive_filename
old_archive_path = instance.archive_path
move_archive = False
archive_already_moved = False
candidate_filename = generate_filename(instance)
if len(str(candidate_filename)) > Document.MAX_STORED_FILENAME_LENGTH:
msg = (
f"Document {instance!s}: Generated filename exceeds db path "
f"limit ({len(str(candidate_filename))} > "
f"{Document.MAX_STORED_FILENAME_LENGTH}): {candidate_filename!s}"
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
candidate_source_path = (
settings.ORIGINALS_DIR / candidate_filename
).resolve()
if candidate_filename == Path(old_filename):
new_filename = Path(old_filename)
elif (
candidate_source_path.exists()
and candidate_source_path != old_source_path
):
if not old_source_path.is_file() and _path_matches_checksum(
candidate_source_path,
instance.checksum,
):
new_filename = candidate_filename
original_already_moved = True
else:
# Only fall back to unique search when there is an actual conflict
new_filename = generate_unique_filename(instance)
else:
new_filename = candidate_filename
# Need to convert to string to be able to save it to the db
instance.filename = str(new_filename)
move_original = (
old_filename != instance.filename and not original_already_moved
)
if instance.has_archive_version:
archive_candidate = generate_filename(instance, archive_filename=True)
if len(str(archive_candidate)) > Document.MAX_STORED_FILENAME_LENGTH:
msg = (
f"Document {instance!s}: Generated archive filename exceeds "
f"db path limit ({len(str(archive_candidate))} > "
f"{Document.MAX_STORED_FILENAME_LENGTH}): {archive_candidate!s}"
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
archive_candidate_path = (
settings.ARCHIVE_DIR / archive_candidate
).resolve()
if archive_candidate == Path(old_archive_filename):
new_archive_filename = Path(old_archive_filename)
elif (
archive_candidate_path.exists()
and archive_candidate_path != old_archive_path
):
if not old_archive_path.is_file() and _path_matches_checksum(
archive_candidate_path,
instance.archive_checksum,
):
new_archive_filename = archive_candidate
archive_already_moved = True
else:
new_archive_filename = generate_unique_filename(
instance,
archive_filename=True,
)
else:
new_archive_filename = archive_candidate
instance.archive_filename = str(new_archive_filename)
move_archive = (
old_archive_filename != instance.archive_filename
and not archive_already_moved
)
else:
move_archive = False
if not move_original and not move_archive:
updates = {"modified": timezone.now()}
if old_filename != instance.filename:
updates["filename"] = instance.filename
if old_archive_filename != instance.archive_filename:
updates["archive_filename"] = instance.archive_filename
# Don't save() here to prevent infinite recursion.
Document.objects.filter(pk=instance.pk).update(**updates)
return
if move_original:
validate_move(
instance,
old_source_path,
instance.source_path,
settings.ORIGINALS_DIR,
)
create_source_path_directory(instance.source_path)
shutil.move(old_source_path, instance.source_path)
if move_archive:
validate_move(
instance,
old_archive_path,
instance.archive_path,
settings.ARCHIVE_DIR,
)
create_source_path_directory(instance.archive_path)
shutil.move(old_archive_path, instance.archive_path)
# Don't save() here to prevent infinite recursion.
Document.global_objects.filter(pk=instance.pk).update(
filename=instance.filename,
archive_filename=instance.archive_filename,
modified=timezone.now(),
)
# Clear any caching for this document. Slightly overkill, but not terrible
clear_document_caches(instance.pk)
except (OSError, DatabaseError, CannotMoveFilesException) as e:
logger.warning(f"Exception during file handling: {e}")
# This happens when either:
# - moving the files failed due to file system errors
# - saving to the database failed due to database errors
# In both cases, we need to revert to the original state.
# Try to move files to their original location.
try:
if move_original and instance.source_path.is_file():
logger.info("Restoring previous original path")
shutil.move(instance.source_path, old_source_path)
if move_archive and instance.archive_path.is_file():
logger.info("Restoring previous archive path")
shutil.move(instance.archive_path, old_archive_path)
except Exception:
# This is fine, since:
# A: if we managed to move source from A to B, we will also
# manage to move it from B to A. If not, we have a serious
# issue that's going to get caught by the santiy checker.
# All files remain in place and will never be overwritten,
# so this is not the end of the world.
# B: if moving the original file failed, nothing has changed
# anyway.
pass
# restore old values on the instance
instance.filename = old_filename
instance.archive_filename = old_archive_filename
# finally, remove any empty sub folders. This will do nothing if
# something has failed above.
if not old_source_path.is_file():
delete_empty_directories(
Path(old_source_path).parent,
root=settings.ORIGINALS_DIR,
)
if instance.has_archive_version and not old_archive_path.is_file():
delete_empty_directories(
Path(old_archive_path).parent,
root=settings.ARCHIVE_DIR,
)
# Keep version files in sync with root
if instance.root_document_id is None:
for version_doc in Document.objects.filter(root_document_id=instance.pk).only(
"pk",
):
update_filename_and_move_files(
Document,
version_doc,
)
@shared_task @shared_task
@@ -465,8 +691,7 @@ def process_cf_select_update(custom_field: CustomField) -> None:
for cf_instance in custom_field.fields.select_related("document").iterator(): for cf_instance in custom_field.fields.select_related("document").iterator():
# Update the filename and move files if necessary # Update the filename and move files if necessary
if _filename_template_uses_custom_fields(cf_instance.document): update_filename_and_move_files(CustomFieldInstance, cf_instance)
move_files_for_document(cf_instance.document_id)
# should be disabled in /src/documents/management/commands/document_importer.py handle # should be disabled in /src/documents/management/commands/document_importer.py handle
@@ -1187,6 +1412,19 @@ def close_connection_pool_on_worker_init(**kwargs) -> None:
conn.close_pool() conn.close_pool()
def run_zone_ocr_extraction(sender, document, original_file=None, **kwargs):
"""Apply matching templates without failing consumption if extraction fails."""
try:
from documents.zone_ocr import run_zone_extraction
run_zone_extraction(document, Path(original_file) if original_file else None)
except Exception:
logger.exception(
"Zone OCR extraction failed for document %s",
document.pk,
)
@worker_process_shutdown.connect @worker_process_shutdown.connect
def close_connection_pool_on_worker_shutdown(**kwargs) -> None: # pragma: no cover def close_connection_pool_on_worker_shutdown(**kwargs) -> None: # pragma: no cover
""" """
+35 -26
View File
@@ -16,6 +16,7 @@ from django.db import models
from django.db import transaction from django.db import transaction
from django.db.models.signals import post_save from django.db.models.signals import post_save
from django.utils import timezone from django.utils import timezone
from filelock import FileLock
from documents import sanity_checker from documents import sanity_checker
from documents.barcodes import BarcodePlugin from documents.barcodes import BarcodePlugin
@@ -41,12 +42,12 @@ from documents.double_sided import CollatePlugin
from documents.file_handling import create_source_path_directory from documents.file_handling import create_source_path_directory
from documents.file_handling import generate_unique_filename from documents.file_handling import generate_unique_filename
from documents.matching import prefilter_documents_by_workflowtrigger from documents.matching import prefilter_documents_by_workflowtrigger
from documents.media_lock import media_lock
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentBarcode from documents.models import DocumentBarcode
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplate
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import ShareLink from documents.models import ShareLink
from documents.models import ShareLinkBundle from documents.models import ShareLinkBundle
@@ -69,6 +70,7 @@ from documents.utils import compute_checksum
from documents.utils import identity from documents.utils import identity
from documents.versioning import annotate_effective_content from documents.versioning import annotate_effective_content
from documents.workflows.utils import get_workflows_for_trigger from documents.workflows.utils import get_workflows_for_trigger
from documents.zone_ocr import run_zone_extraction
from paperless.config import AIConfig from paperless.config import AIConfig
from paperless.config import BarcodeConfig from paperless.config import BarcodeConfig
from paperless.config import RemoteOCRConfig from paperless.config import RemoteOCRConfig
@@ -310,7 +312,6 @@ def sanity_check(*, raise_on_error: bool = True) -> str:
@shared_task @shared_task
def bulk_update_documents(document_ids) -> None: def bulk_update_documents(document_ids) -> None:
from documents.file_moves import flush_pending_file_moves
from documents.search import get_backend from documents.search import get_backend
document_ids = list(document_ids) document_ids = list(document_ids)
@@ -331,9 +332,6 @@ def bulk_update_documents(document_ids) -> None:
skip_ai_index=True, # bulk path calls update_llm_index once below skip_ai_index=True, # bulk path calls update_llm_index once below
) )
post_save.send(Document, instance=doc, created=False) post_save.send(Document, instance=doc, created=False)
# Move this document's files now rather than at the end of the task,
# so a long bulk edit makes steady progress.
flush_pending_file_moves()
with get_backend().batch_update() as batch: with get_backend().batch_update() as batch:
batch.add_or_update_ids(document_ids) batch.add_or_update_ids(document_ids)
@@ -371,6 +369,31 @@ def _read_barcodes_for_reprocess(document: Document) -> list[StoredBarcode] | No
return None return None
@shared_task
def extract_zones(document_id: int, template_id: int | None = None) -> None:
"""
Run an OCR template against a stored document, or every template that
matches it when no template is given.
"""
document = Document.objects.get(id=document_id)
if template_id is None:
run_zone_extraction(document, None)
return
try:
template = OcrTemplate.objects.prefetch_related(
"zones",
"zones__custom_field",
).get(id=template_id)
except OcrTemplate.DoesNotExist:
logger.warning(
"OCR template %s no longer exists, not running zone OCR",
template_id,
)
return
run_zone_extraction(document, None, templates=[template])
@shared_task @shared_task
def update_document_content_maybe_archive_file( def update_document_content_maybe_archive_file(
document_id, document_id,
@@ -421,13 +444,7 @@ def update_document_content_maybe_archive_file(
thumbnail = parser.get_thumbnail(document.source_path, mime_type) thumbnail = parser.get_thumbnail(document.source_path, mime_type)
# media.lock first, then the transaction: never wait on media.lock with transaction.atomic():
# while this transaction holds the document row. Also keeps the
# unique archive filename generation under the lock.
with media_lock(), transaction.atomic():
# Loaded before the (long) parse; read current metadata now
# that the lock is held, for the archive filename.
document.refresh_from_db()
oldDocument = Document.objects.get(pk=document.pk) oldDocument = Document.objects.get(pk=document.pk)
if parser.get_archive_path(): if parser.get_archive_path():
checksum = compute_checksum(parser.get_archive_path()) checksum = compute_checksum(parser.get_archive_path())
@@ -492,10 +509,11 @@ def update_document_content_maybe_archive_file(
modified=timezone.now(), modified=timezone.now(),
) )
if parser.get_archive_path(): with FileLock(settings.MEDIA_LOCK):
create_source_path_directory(document.archive_path) if parser.get_archive_path():
shutil.move(parser.get_archive_path(), document.archive_path) create_source_path_directory(document.archive_path)
shutil.move(thumbnail, document.thumbnail_path) shutil.move(parser.get_archive_path(), document.archive_path)
shutil.move(thumbnail, document.thumbnail_path)
document.refresh_from_db() document.refresh_from_db()
logger.info( logger.info(
@@ -540,11 +558,7 @@ def empty_trash(doc_ids=None) -> None:
deleted_document_ids = list(documents.values_list("id", flat=True)) deleted_document_ids = list(documents.values_list("id", flat=True))
# Temporarily connect the cleanup handler # Temporarily connect the cleanup handler
models.signals.post_delete.connect(cleanup_document_deletion, sender=Document) models.signals.post_delete.connect(cleanup_document_deletion, sender=Document)
# media.lock before Django's delete transaction: post_delete runs documents.delete() # this is effectively a hard delete
# cleanup_document_deletion inside it while the rows are locked, and
# that handler re-enters the lock instead of waiting.
with media_lock():
documents.delete() # this is effectively a hard delete
logger.info(f"Deleted {len(deleted_document_ids)} documents from trash") logger.info(f"Deleted {len(deleted_document_ids)} documents from trash")
if settings.AUDIT_LOG_ENABLED: if settings.AUDIT_LOG_ENABLED:
@@ -574,8 +588,6 @@ def check_scheduled_workflows() -> None:
Once a document satisfies this condition, and recurring/non-recurring constraints are met, the workflow is run. Once a document satisfies this condition, and recurring/non-recurring constraints are met, the workflow is run.
""" """
from documents.file_moves import flush_pending_file_moves
scheduled_workflows = get_workflows_for_trigger( scheduled_workflows = get_workflows_for_trigger(
WorkflowTrigger.WorkflowTriggerType.SCHEDULED, WorkflowTrigger.WorkflowTriggerType.SCHEDULED,
) )
@@ -703,9 +715,6 @@ def check_scheduled_workflows() -> None:
sender=None, sender=None,
document=document, document=document,
) )
# Move this document's files now rather than at the end
# of the task, so a long run makes steady progress.
flush_pending_file_moves()
def update_document_parent_tags(tag: Tag, new_parent: Tag) -> None: def update_document_parent_tags(tag: Tag, new_parent: Tag) -> None:
+82
View File
@@ -12,6 +12,7 @@ from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplate
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
from paperless_testing.dirs import DirectoriesMixin from paperless_testing.dirs import DirectoriesMixin
@@ -583,6 +584,87 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(args[0], [self.doc1.id]) self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True}) self.assertEqual(kwargs, {"remote_ocr": True})
@mock.patch("documents.views.bulk_edit.run_zone_ocr")
def test_run_zone_ocr_documents_endpoint(self, m) -> None:
"""
GIVEN:
- API data to run zone OCR on a document
WHEN:
- API is called
THEN:
- run_zone_ocr is called with the document
"""
self.setup_mock(m, "run_zone_ocr")
response = self.client.post(
"/api/documents/run_zone_ocr/",
json.dumps({"documents": [self.doc1.id]}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {})
@mock.patch("documents.views.bulk_edit.run_zone_ocr")
def test_run_zone_ocr_documents_endpoint_with_template(self, m) -> None:
"""
GIVEN:
- API data to run a chosen OCR template on a document
WHEN:
- API is called
THEN:
- run_zone_ocr is called with the template
"""
self.setup_mock(m, "run_zone_ocr")
template = OcrTemplate.objects.create(
name="Bills",
source_width=100,
source_height=100,
)
response = self.client.post(
"/api/documents/run_zone_ocr/",
json.dumps({"documents": [self.doc1.id], "template": template.pk}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
m.assert_called_once()
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"template": template})
@mock.patch("documents.views.bulk_edit.run_zone_ocr")
def test_run_zone_ocr_documents_endpoint_rejects_private_template(
self,
m,
) -> None:
"""
GIVEN:
- A user who can change the documents but not view the template
WHEN:
- API is called with that template
THEN:
- The whole request is refused
"""
user = UserFactory(username="editor")
grant_global(user, "view_document", "change_document")
template = OcrTemplate.objects.create(
name="Private",
owner=UserFactory(username="other"),
source_width=100,
source_height=100,
)
self.client.force_authenticate(user=user)
response = self.client.post(
"/api/documents/run_zone_ocr/",
json.dumps({"documents": [self.doc1.id], "template": template.pk}),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
m.assert_not_called()
@mock.patch("documents.serialisers.bulk_edit.set_storage_path") @mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None: def test_api_set_storage_path(self, m) -> None:
""" """
File diff suppressed because it is too large. Load diff
+110 -3
View File
@@ -21,11 +21,13 @@ from documents.models import CustomField
from documents.models import CustomFieldInstance from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import OcrTemplate
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
from documents.pdf_ops import PageSpec from documents.pdf_ops import PageSpec
from documents.permissions import PermissionsSpec from documents.permissions import PermissionsSpec
from documents.permissions import set_permissions_for_objects from documents.permissions import set_permissions_for_objects
from documents.tasks import extract_zones
from paperless_testing.dirs import DirectoriesMixin from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory from paperless_testing.factories import DocumentFactory
from paperless_testing.permissions import grant_object from paperless_testing.permissions import grant_object
@@ -739,7 +741,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
mime_type="application/pdf", mime_type="application/pdf",
) )
self.doc1.archive_filename = sample1_archive self.doc1.archive_filename = sample1_archive
self.doc1.save(write_file_paths=True) self.doc1.save()
self.doc2 = Document.objects.create( self.doc2 = Document.objects.create(
checksum="B", checksum="B",
title="B", title="B",
@@ -749,7 +751,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
created=date(2023, 1, 2), created=date(2023, 1, 2),
) )
self.doc2.archive_filename = sample2_archive self.doc2.archive_filename = sample2_archive
self.doc2.save(write_file_paths=True) self.doc2.save()
self.doc3 = Document.objects.create( self.doc3 = Document.objects.create(
checksum="C", checksum="C",
title="C", title="C",
@@ -774,7 +776,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
created=date(2023, 1, 3), created=date(2023, 1, 3),
) )
self.img_doc.archive_filename = img_doc_archive self.img_doc.archive_filename = img_doc_archive
self.img_doc.save(write_file_paths=True) self.img_doc.save()
@staticmethod @staticmethod
def fake_decrypt( def fake_decrypt(
@@ -1874,6 +1876,111 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIn("Error removing password from document", cm.output[0]) self.assertIn("Error removing password from document", cm.output[0])
@pytest.mark.django_db
class TestBulkEditRunZoneOcr:
@pytest.fixture
def mock_task(self, mocker: MockerFixture) -> mock.MagicMock:
return mocker.patch("documents.bulk_edit.extract_zones")
def test_queues_every_selected_document(
self,
mock_task: mock.MagicMock,
) -> None:
"""
GIVEN:
- Selected documents
WHEN:
- run_zone_ocr is called with them
THEN:
- Each document is queued once, templates are matched when its
task runs
"""
docs = DocumentFactory.create_batch(2)
result = bulk_edit.run_zone_ocr([doc.id for doc in docs])
assert result == "OK"
assert [
call.kwargs["kwargs"] for call in mock_task.apply_async.call_args_list
] == [{"document_id": doc.id, "template_id": None} for doc in docs]
def test_queues_the_chosen_template(self, mock_task: mock.MagicMock) -> None:
"""
GIVEN:
- A chosen OCR template
WHEN:
- run_zone_ocr is called with it
THEN:
- The template is queued with the document
"""
doc = DocumentFactory()
template = OcrTemplate.objects.create(
name="Bills",
source_width=100,
source_height=100,
)
bulk_edit.run_zone_ocr([doc.id], template=template)
assert mock_task.apply_async.call_args.kwargs["kwargs"] == {
"document_id": doc.id,
"template_id": template.pk,
}
def test_task_runs_extraction(self, mocker: MockerFixture) -> None:
"""
GIVEN:
- A queued zone OCR task
WHEN:
- The task runs
THEN:
- Zone extraction runs on the stored document
"""
extract = mocker.patch("documents.tasks.run_zone_extraction")
doc = DocumentFactory()
extract_zones(doc.id)
extract.assert_called_once_with(doc, None)
def test_task_runs_the_chosen_template(self, mocker: MockerFixture) -> None:
"""
GIVEN:
- A queued zone OCR task with a chosen template
WHEN:
- The task runs
THEN:
- Zone extraction runs with only that template
"""
extract = mocker.patch("documents.tasks.run_zone_extraction")
doc = DocumentFactory()
template = OcrTemplate.objects.create(
name="Bills",
source_width=100,
source_height=100,
)
extract_zones(doc.id, template.pk)
extract.assert_called_once_with(doc, None, templates=[template])
def test_task_skips_a_deleted_template(self, mocker: MockerFixture) -> None:
"""
GIVEN:
- A template deleted between queueing and running
WHEN:
- The task runs
THEN:
- Nothing is extracted
"""
extract = mocker.patch("documents.tasks.run_zone_extraction")
doc = DocumentFactory()
extract_zones(doc.id, 999)
extract.assert_not_called()
@pytest.mark.django_db @pytest.mark.django_db
class TestBulkEditReprocess: class TestBulkEditReprocess:
@pytest.fixture @pytest.fixture
+1 -160
View File
@@ -27,10 +27,6 @@ from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowActionEmail
from documents.models import WorkflowTrigger
from documents.parsers import ParseError from documents.parsers import ParseError
from documents.plugins.helpers import ProgressStatusOptions from documents.plugins.helpers import ProgressStatusOptions
from documents.tasks import sanity_check from documents.tasks import sanity_check
@@ -42,7 +38,6 @@ from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import TagFactory from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory from paperless_testing.factories import UserFactory
from paperless_testing.fakes.progress import FakeProgressManager from paperless_testing.fakes.progress import FakeProgressManager
from paperless_testing.locks import record_media_lock_acquisitions
from paperless_testing.permissions import grant_object from paperless_testing.permissions import grant_object
from paperless_testing.samples import MULTI_PAGE_DIGITAL_PDF from paperless_testing.samples import MULTI_PAGE_DIGITAL_PDF
from paperless_testing.samples import MULTI_PAGE_IMAGES_PDF from paperless_testing.samples import MULTI_PAGE_IMAGES_PDF
@@ -226,160 +221,6 @@ class TestConsumer(
shutil.copy(src, dst) shutil.copy(src, dst)
return dst return dst
@mock.patch("documents.consumer.load_classifier")
def test_consume_takes_media_lock_outside_transaction(self, m) -> None:
"""
GIVEN:
- A new document to consume
WHEN:
- It is consumed
THEN:
- media.lock is acquired
- media.lock is never first acquired inside a transaction
- The document's file is in place
"""
m.return_value = MagicMock()
with record_media_lock_acquisitions() as acquisitions:
with self.get_consumer(self.get_test_file()) as consumer:
consumer.run()
self.assertTrue(acquisitions)
self.assertFalse(any(a.in_user_transaction for a in acquisitions))
document = Document.objects.first()
assert document is not None
self.assertIsFile(document.source_path)
@mock.patch("documents.consumer.load_classifier")
def test_consume_places_file_before_post_consume_script(self, m) -> None:
"""
GIVEN:
- A new document to consume
WHEN:
- It is consumed
THEN:
- Pending file moves are flushed before the post-consume script runs
"""
m.return_value = MagicMock()
order = MagicMock()
with (
mock.patch(
"documents.consumer.flush_pending_file_moves",
order.flush,
),
mock.patch.object(
ConsumerPlugin,
"run_post_consume_script",
order.script,
),
):
with self.get_consumer(self.get_test_file()) as consumer:
consumer.run()
self.assertEqual(
[c[0] for c in order.mock_calls],
["flush", "script"],
)
@mock.patch("documents.consumer.load_classifier")
def test_consume_version_takes_media_lock_outside_transaction(self, m) -> None:
"""
GIVEN:
- An existing root document
WHEN:
- A new version of it is consumed (the root row is locked with
select_for_update inside the consumer's transaction)
THEN:
- media.lock is never first acquired inside a transaction
- The root's modified is bumped
"""
m.return_value = MagicMock()
with self.get_consumer(self.get_test_file()) as consumer:
consumer.run()
root_doc = Document.objects.first()
assert root_doc is not None
original_modified = timezone.now() - datetime.timedelta(days=1)
Document.objects.filter(pk=root_doc.pk).update(modified=original_modified)
version_file = self.get_test_file2()
status = FakeProgressManager(version_file.name, None)
overrides = DocumentMetadataOverrides(version_label="v2")
doc = ConsumableDocument(
DocumentSource.ApiUpload,
original_file=version_file,
root_document_id=root_doc.pk,
)
preflight = ConsumerPreflightPlugin(
doc,
overrides,
status, # type: ignore[arg-type]
self.dirs.scratch_dir,
"task-id",
)
preflight.setup()
preflight.run()
consumer = ConsumerPlugin(
doc,
overrides,
status, # type: ignore[arg-type]
self.dirs.scratch_dir,
"task-id",
)
consumer.setup()
try:
with record_media_lock_acquisitions() as acquisitions:
consumer.run()
finally:
consumer.cleanup()
self.assertTrue(acquisitions)
self.assertFalse(any(a.in_user_transaction for a in acquisitions))
root_doc.refresh_from_db()
self.assertGreater(root_doc.modified, original_modified)
@override_settings(
PAPERLESS_EMAIL_HOST="localhost",
EMAIL_ENABLED=True,
PAPERLESS_URL="http://localhost:8000",
)
@mock.patch("documents.consumer.load_classifier")
@mock.patch("django.core.mail.message.EmailMessage.send", return_value=1)
def test_consumption_workflow_email_runs_under_held_lock(self, send, m) -> None:
"""
GIVEN:
- A document added workflow with an email action
WHEN:
- A document is consumed (the consumer holds media.lock while
workflows run)
THEN:
- The email action re-enters media.lock and sends without blocking
"""
m.return_value = MagicMock()
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
)
email = WorkflowActionEmail.objects.create(
subject="added",
body="body",
to="someone@example.com",
include_document=True,
)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.EMAIL,
email=email,
)
workflow = Workflow.objects.create(name="email on add")
workflow.triggers.add(trigger)
workflow.actions.add(action)
with self.get_consumer(self.get_test_file()) as consumer:
consumer.run()
send.assert_called_once()
@override_settings( @override_settings(
FILENAME_FORMAT=None, FILENAME_FORMAT=None,
TIME_ZONE="America/Chicago", TIME_ZONE="America/Chicago",
@@ -909,7 +750,7 @@ class TestConsumer(
FILENAME_FORMAT="{correspondent}/{title}", FILENAME_FORMAT="{correspondent}/{title}",
ARCHIVE_FILE_GENERATION="always", ARCHIVE_FILE_GENERATION="always",
) )
@mock.patch("documents.file_moves.generate_unique_filename") @mock.patch("documents.signals.handlers.generate_unique_filename")
def testFilenameHandlingUnstableFormat(self, m) -> None: def testFilenameHandlingUnstableFormat(self, m) -> None:
filenames = ["this", "that", "now this", "i cannot decide"] filenames = ["this", "that", "now this", "i cannot decide"]
+25 -23
View File
@@ -92,7 +92,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
# Ensure that filename is properly generated # Ensure that filename is properly generated
self.assertEqual(document.filename, Path("none/none.pdf")) self.assertEqual(document.filename, Path("none/none.pdf"))
document.save(write_file_paths=True) document.save()
# test that creating dirs for the source_path creates the correct directory # test that creating dirs for the source_path creates the correct directory
create_source_path_directory(document.source_path) create_source_path_directory(document.source_path)
@@ -190,7 +190,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
# Set a correspondent and save the document # Set a correspondent and save the document
document.correspondent = Correspondent.objects.get_or_create(name="test")[0] document.correspondent = Correspondent.objects.get_or_create(name="test")[0]
document.save(write_file_paths=True) document.save()
# Check proper handling of files # Check proper handling of files
self.assertIsFile( self.assertIsFile(
@@ -232,7 +232,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
disable_auditlog(), disable_auditlog(),
): ):
m.side_effect = DatabaseError() m.side_effect = DatabaseError()
document.save(write_file_paths=True) document.save()
# Check proper handling of files # Check proper handling of files
self.assertIsFile(document.source_path) self.assertIsFile(document.source_path)
@@ -336,7 +336,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
# Ensure that filename is properly generated # Ensure that filename is properly generated
document.filename = generate_filename(document) document.filename = generate_filename(document)
document.save(write_file_paths=True) document.save()
self.assertEqual(document.filename, "none/none.pdf") self.assertEqual(document.filename, "none/none.pdf")
create_source_path_directory(document.source_path) create_source_path_directory(document.source_path)
@@ -362,7 +362,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
# Ensure that filename is properly generated # Ensure that filename is properly generated
document.filename = generate_filename(document) document.filename = generate_filename(document)
document.save(write_file_paths=True) document.save()
self.assertEqual(document.filename, "none/none.pdf") self.assertEqual(document.filename, "none/none.pdf")
create_source_path_directory(document.source_path) create_source_path_directory(document.source_path)
@@ -385,7 +385,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
document.save() document.save()
document.filename = generate_filename(document) document.filename = generate_filename(document)
document.save(write_file_paths=True) document.save()
create_source_path_directory(document.source_path) create_source_path_directory(document.source_path)
Path(document.source_path).touch() Path(document.source_path).touch()
document.delete() document.delete()
@@ -421,7 +421,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
# Set a correspondent and save the document # Set a correspondent and save the document
document.correspondent = Correspondent.objects.get_or_create(name="test")[0] document.correspondent = Correspondent.objects.get_or_create(name="test")[0]
document.save(write_file_paths=True) document.save()
# Check proper handling of files # Check proper handling of files
self.assertIsDir(settings.ORIGINALS_DIR / "test") self.assertIsDir(settings.ORIGINALS_DIR / "test")
@@ -476,7 +476,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
def test_filename_relative(self) -> None: def test_filename_relative(self) -> None:
doc = Document.objects.create(title="doc1", mime_type="application/pdf") doc = Document.objects.create(title="doc1", mime_type="application/pdf")
doc.filename = generate_filename(doc) doc.filename = generate_filename(doc)
doc.save(write_file_paths=True) doc.save()
self.assertEqual( self.assertEqual(
doc.source_path, doc.source_path,
@@ -532,7 +532,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
# Ensure that filename is properly generated # Ensure that filename is properly generated
document.filename = generate_filename(document) document.filename = generate_filename(document)
document.save(write_file_paths=True) document.save()
self.assertEqual(document.filename, "none/none/none.pdf") self.assertEqual(document.filename, "none/none/none.pdf")
create_source_path_directory(document.source_path) create_source_path_directory(document.source_path)
Path(document.source_path).touch() Path(document.source_path).touch()
@@ -620,13 +620,13 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
Path(document.source_path).touch() Path(document.source_path).touch()
Path(document2.source_path).touch() Path(document2.source_path).touch()
document.filename = "0000001.pdf" document.filename = "0000001.pdf"
document.save(write_file_paths=True) document.save()
self.assertIsFile(document.source_path) self.assertIsFile(document.source_path)
self.assertEqual(document.filename, "qwe.pdf") self.assertEqual(document.filename, "qwe.pdf")
document2.filename = "0000002.pdf" document2.filename = "0000002.pdf"
document2.save(write_file_paths=True) document2.save()
self.assertIsFile(document.source_path) self.assertIsFile(document.source_path)
self.assertEqual(document2.filename, "qwe_01.pdf") self.assertEqual(document2.filename, "qwe_01.pdf")
@@ -656,17 +656,19 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
self.assertEqual(document2.filename, "qwe.pdf") self.assertEqual(document2.filename, "qwe.pdf")
@override_settings(FILENAME_FORMAT="{title}") @override_settings(FILENAME_FORMAT="{title}")
@mock.patch("documents.file_moves.Document.global_objects.filter") @mock.patch("documents.signals.handlers.Document.objects.filter")
@mock.patch("documents.file_moves.shutil.move") @mock.patch("documents.signals.handlers.shutil.move")
def test_no_move_only_save(self, mock_move, mock_filter) -> None: def test_no_move_only_save(self, mock_move, mock_filter) -> None:
""" """
GIVEN: GIVEN:
- A document with a filename - A document with a filename
- The document is saved
- The filename is not changed
WHEN: WHEN:
- The document is saved without a change that affects its path - The document is saved
THEN: THEN:
- The document modified date is updated by the save itself - The document modified date is updated
- The file is not moved and the mover writes nothing - The document is not moved
""" """
with disable_auditlog(): with disable_auditlog():
doc = Document.objects.create( doc = Document.objects.create(
@@ -684,7 +686,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
doc.save() doc.save()
doc.refresh_from_db() doc.refresh_from_db()
mock_filter.assert_not_called() mock_filter.assert_called()
self.assertNotEqual(original_modified, doc.modified) self.assertNotEqual(original_modified, doc.modified)
mock_move.assert_not_called() mock_move.assert_not_called()
@@ -692,7 +694,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
FILENAME_FORMAT="{{title}}_{{custom_fields|get_cf_value('test')}}", FILENAME_FORMAT="{{title}}_{{custom_fields|get_cf_value('test')}}",
CELERY_TASK_ALWAYS_EAGER=True, CELERY_TASK_ALWAYS_EAGER=True,
) )
@mock.patch("documents.signals.handlers.move_files_for_document") @mock.patch("documents.signals.handlers.update_filename_and_move_files")
def test_select_cf_updated(self, m) -> None: def test_select_cf_updated(self, m) -> None:
""" """
GIVEN: GIVEN:
@@ -2043,7 +2045,7 @@ class TestCustomFieldFilenameUpdates(
Path(self.doc.source_path).parent.mkdir(parents=True, exist_ok=True) Path(self.doc.source_path).parent.mkdir(parents=True, exist_ok=True)
Path(self.doc.source_path).touch() Path(self.doc.source_path).touch()
with mock.patch("documents.file_moves.generate_unique_filename") as m: with mock.patch("documents.signals.handlers.generate_unique_filename") as m:
m.side_effect = generate_unique_filename m.side_effect = generate_unique_filename
self.cfi.value_text = "updated" self.cfi.value_text = "updated"
self.cfi.save() self.cfi.save()
@@ -2065,7 +2067,7 @@ class TestCustomFieldFilenameUpdates(
Path(self.doc.source_path).parent.mkdir(parents=True, exist_ok=True) Path(self.doc.source_path).parent.mkdir(parents=True, exist_ok=True)
Path(self.doc.source_path).touch() Path(self.doc.source_path).touch()
with mock.patch("documents.file_moves.generate_unique_filename") as m: with mock.patch("documents.signals.handlers.generate_unique_filename") as m:
m.side_effect = generate_unique_filename m.side_effect = generate_unique_filename
self.cfi.value_text = "updated" self.cfi.value_text = "updated"
self.cfi.save() self.cfi.save()
@@ -2143,7 +2145,7 @@ def test_refresh_failure_surfaces_database_error(
GIVEN: GIVEN:
- A document with a stored filename - A document with a stored filename
WHEN: WHEN:
- Saving it triggers a filename update and loading it from the database fails - Saving it triggers a filename update and refreshing it from the database fails
THEN: THEN:
- The database error is raised, not an unbound variable error from the rollback - The database error is raised, not an unbound variable error from the rollback
""" """
@@ -2151,8 +2153,8 @@ def test_refresh_failure_surfaces_database_error(
with ( with (
mock.patch.object( mock.patch.object(
Document.global_objects, Document,
"get", "refresh_from_db",
side_effect=DatabaseError("database is locked"), side_effect=DatabaseError("database is locked"),
), ),
pytest.raises(DatabaseError, match="database is locked"), pytest.raises(DatabaseError, match="database is locked"),
-945
View File
@@ -1,945 +0,0 @@
from __future__ import annotations
import datetime
import hashlib
import logging
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from asgiref.sync import async_to_sync
from asgiref.sync import sync_to_async
from celery.contrib.django.task import DjangoTask
from django.db import transaction
from django.http import HttpRequest
from django.http import HttpResponse
from django.test import RequestFactory
from django.utils import timezone
from django.utils.dateparse import parse_datetime
from filelock import Timeout
from rest_framework import status
from documents.file_moves import _active_collector
from documents.file_moves import defer_file_moves
from documents.file_moves import flush_pending_file_moves
from documents.file_moves import move_files_for_document
from documents.file_moves import request_file_move
from documents.models import CustomField
from documents.models import Document
from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from documents.tasks import bulk_update_documents
from documents.tasks import check_scheduled_workflows
from documents.tasks import empty_trash
from paperless.celery import app as celery_app
from paperless.middleware import FileMoveMiddleware
from paperless_testing.factories import CorrespondentFactory
from paperless_testing.factories import CustomFieldFactory
from paperless_testing.factories import CustomFieldInstanceFactory
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import StoragePathFactory
from paperless_testing.factories import TagFactory
from paperless_testing.locks import record_media_lock_acquisitions
if TYPE_CHECKING:
from collections.abc import Callable
from contextlib import AbstractContextManager
from pytest_django.fixtures import Settings
from pytest_mock import MockerFixture
from rest_framework.test import APIClient
from paperless_testing.dirs import PaperlessDirs
OnCommitCapture = Callable[..., AbstractContextManager[list[Callable[[], object]]]]
CONTENT = b"original content"
@pytest.fixture
def title_format(settings: Settings) -> None:
settings.FILENAME_FORMAT = "{title}"
@pytest.fixture
def document(paperless_dirs: PaperlessDirs, title_format: None) -> Document:
"""A document named after its title, with its original on disk."""
(paperless_dirs.originals_dir / "original.pdf").write_bytes(CONTENT)
return DocumentFactory(
title="original",
filename="original.pdf",
mime_type="application/pdf",
checksum=hashlib.sha256(CONTENT).hexdigest(),
)
@celery_app.task(name="documents.tests.test_file_moves.rename_then_fail")
def rename_then_fail(pk: int, originals_dir: str) -> None:
"""Test-only task: rename the document, check nothing moved yet, then fail."""
doc = Document.objects.get(pk=pk)
doc.title = "renamed"
doc.save()
assert (Path(originals_dir) / "original.pdf").is_file()
raise RuntimeError("task failed after saving")
@pytest.mark.django_db
class TestMoveFilesForDocument:
def test_moves_using_fresh_db_state(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A document whose title changed in the DB without any signal
WHEN:
- move_files_for_document is called with its pk
THEN:
- The file and the DB filename follow the DB title
- The returned instance carries the final filename
"""
Document.objects.filter(pk=document.pk).update(title="renamed")
moved = move_files_for_document(document.pk)
assert moved is not None
assert moved.filename == "renamed.pdf"
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
assert not (paperless_dirs.originals_dir / "original.pdf").exists()
document.refresh_from_db()
assert document.filename == "renamed.pdf"
def test_missing_document_returns_none(self, paperless_dirs: PaperlessDirs) -> None:
"""
GIVEN:
- A pk with no document row
WHEN:
- move_files_for_document is called
THEN:
- It returns None without raising
"""
assert move_files_for_document(999_999) is None
def test_document_without_filename_skips_lock(
self,
paperless_dirs: PaperlessDirs,
title_format: None,
) -> None:
"""
GIVEN:
- A document still being consumed (no filename yet)
WHEN:
- move_files_for_document is called
THEN:
- It returns None, writes nothing, and never takes media.lock
"""
doc = DocumentFactory(filename=None, mime_type="application/pdf")
with record_media_lock_acquisitions() as acquisitions:
assert move_files_for_document(doc.pk) is None
assert acquisitions == []
doc.refresh_from_db()
assert doc.filename is None
def test_no_change_does_not_write_modified(
self,
document: Document,
) -> None:
"""
GIVEN:
- A document already at its generated path, modified in the past
WHEN:
- move_files_for_document is called
THEN:
- modified is not written
"""
past = timezone.now() - datetime.timedelta(days=1)
Document.objects.filter(pk=document.pk).update(modified=past)
move_files_for_document(document.pk)
document.refresh_from_db()
assert document.modified == past
def test_trashed_already_moved_document_gets_db_path_fixed(
self,
paperless_dirs: PaperlessDirs,
title_format: None,
) -> None:
"""
GIVEN:
- A trashed document whose DB filename is stale
- Its file already sits at the generated path (checksum matches)
WHEN:
- move_files_for_document is called
THEN:
- The DB filename is updated to where the file is
"""
doc = DocumentFactory(
title="current",
filename="stale.pdf",
mime_type="application/pdf",
checksum=hashlib.sha256(CONTENT).hexdigest(),
)
assert Document.objects.get(pk=doc.pk).filename == "stale.pdf"
doc.delete()
(paperless_dirs.originals_dir / "current.pdf").write_bytes(CONTENT)
move_files_for_document(doc.pk)
assert Document.global_objects.get(pk=doc.pk).filename == "current.pdf"
def test_lock_timeout_leaves_consistent_state(
self,
document: Document,
paperless_dirs: PaperlessDirs,
mocker: MockerFixture,
caplog: pytest.LogCaptureFixture,
) -> None:
"""
GIVEN:
- A document whose title changed in the DB
- media.lock cannot be acquired
WHEN:
- move_files_for_document is called
THEN:
- An error is logged and None is returned
- The lock was requested with the 300 second timeout
- The DB filename and the file on disk are unchanged
"""
Document.objects.filter(pk=document.pk).update(title="renamed")
lock = mocker.patch("documents.file_moves.media_lock")
lock.return_value.__enter__.side_effect = Timeout("media.lock")
with caplog.at_level(logging.ERROR, logger="paperless.handlers"):
assert move_files_for_document(document.pk) is None
assert "Timed out" in caplog.text
lock.assert_called_once_with(timeout=300)
document.refresh_from_db()
assert document.filename == "original.pdf"
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
@pytest.mark.django_db
class TestRequestFileMoveWithoutScope:
def test_moves_immediately_and_syncs_only_file_columns(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- No file move scope is open
- A caller instance with an unsaved in-memory change
WHEN:
- request_file_move is called after the title changed in the DB
THEN:
- The file moves immediately
- The caller's filename is updated
- The caller's unsaved change is kept
"""
Document.objects.filter(pk=document.pk).update(title="renamed")
unsaved = CorrespondentFactory()
document.correspondent = unsaved
request_file_move(document)
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
assert document.filename == "renamed.pdf"
assert document.correspondent == unsaved
assert document.title == "original"
@pytest.mark.django_db
class TestReceiver:
def test_reverse_m2m_requests_a_move_per_document(
self,
document: Document,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- A tag
WHEN:
- A document is added through the tag's reverse relation
THEN:
- A move is requested for the added document, never the tag
"""
tag = TagFactory()
request = mocker.patch("documents.signals.handlers.request_file_move")
tag.documents.add(document)
request.assert_called_once()
assert request.call_args.args[0].pk == document.pk
@pytest.mark.django_db
class TestDeferFileMoves:
def test_moves_once_on_exit(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- An open file move scope
WHEN:
- The document is saved with a new title inside the scope
THEN:
- Nothing moves while the scope is open
- On exit the file and DB filename follow the final title
"""
with defer_file_moves():
document.title = "renamed"
document.save()
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
assert Document.objects.get(pk=document.pk).filename == "original.pdf"
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
assert not (paperless_dirs.originals_dir / "original.pdf").exists()
assert Document.objects.get(pk=document.pk).filename == "renamed.pdf"
def test_nested_scope_defers_to_outer(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A scope nested inside another
WHEN:
- The document is saved inside the inner scope and the inner scope exits
THEN:
- Nothing moves until the outer scope exits
"""
with defer_file_moves():
with defer_file_moves():
document.title = "renamed"
document.save()
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_closed_collector_moves_immediately(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A collector that has already been closed (work outliving its scope)
WHEN:
- A save asks for a move while that collector is still current
THEN:
- The move happens immediately instead of being lost
"""
with defer_file_moves():
collector = _active_collector.get()
assert collector is not None
token = _active_collector.set(collector)
try:
document.title = "renamed"
document.save()
finally:
_active_collector.reset(token)
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_scope_closing_inside_atomic_flushes_on_commit(
self,
document: Document,
paperless_dirs: PaperlessDirs,
django_capture_on_commit_callbacks: OnCommitCapture,
) -> None:
"""
GIVEN:
- A scope opened and closed inside an atomic block
WHEN:
- The atomic block commits
THEN:
- The move runs only once the commit callbacks run
"""
with django_capture_on_commit_callbacks(execute=True):
with transaction.atomic(), defer_file_moves():
document.title = "renamed"
document.save()
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_scope_closing_inside_rolled_back_atomic_moves_nothing(
self,
document: Document,
paperless_dirs: PaperlessDirs,
django_capture_on_commit_callbacks: OnCommitCapture,
) -> None:
"""
GIVEN:
- A scope opened and closed inside an atomic block
WHEN:
- The atomic block rolls back
THEN:
- The title change is gone and the file is not moved
"""
class Rollback(Exception):
pass
with django_capture_on_commit_callbacks(execute=True):
with pytest.raises(Rollback):
with transaction.atomic(), defer_file_moves():
document.title = "renamed"
document.save()
raise Rollback
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
assert not (paperless_dirs.originals_dir / "renamed.pdf").exists()
assert Document.objects.get(pk=document.pk).title == "original"
def test_flush_pending_moves_early(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- An open scope with a pending move
WHEN:
- flush_pending_file_moves is called inside the scope
THEN:
- The move happens before the scope exits
"""
with defer_file_moves():
document.title = "renamed"
document.save()
flush_pending_file_moves()
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_document_without_filename_in_scope_is_noop(
self,
paperless_dirs: PaperlessDirs,
title_format: None,
) -> None:
"""
GIVEN:
- A document still being consumed (no filename yet)
WHEN:
- It is saved inside a scope and the scope exits
THEN:
- Nothing is written, media.lock is never taken, nothing raises
"""
doc = DocumentFactory(filename=None, mime_type="application/pdf")
with record_media_lock_acquisitions() as acquisitions:
with defer_file_moves():
doc.title = "renamed"
doc.save()
assert acquisitions == []
doc.refresh_from_db()
assert doc.filename is None
def test_mover_keeps_metadata_changed_after_save(
self,
document: Document,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
"""
GIVEN:
- A filename format using correspondent and title
- A scope in which the title is saved
WHEN:
- Another writer changes the correspondent before the scope exits
THEN:
- The correspondent change survives the flush
- The final path reflects both changes
"""
settings.FILENAME_FORMAT = "{correspondent}/{title}"
correspondent = CorrespondentFactory(name="acme")
with defer_file_moves():
document.title = "renamed"
document.save()
Document.objects.filter(pk=document.pk).update(correspondent=correspondent)
document.refresh_from_db()
assert document.correspondent == correspondent
assert document.filename == "acme/renamed.pdf"
assert (paperless_dirs.originals_dir / "acme" / "renamed.pdf").is_file()
def test_one_failure_does_not_stop_the_flush(
self,
paperless_dirs: PaperlessDirs,
title_format: None,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- Two documents saved in one scope
- Moving the first raises an unexpected error
WHEN:
- The scope exits
THEN:
- The scope does not raise
- The second document is still moved
"""
(paperless_dirs.originals_dir / "a.pdf").write_bytes(CONTENT)
first = DocumentFactory(
filename="a.pdf",
title="a",
mime_type="application/pdf",
)
(paperless_dirs.originals_dir / "b.pdf").write_bytes(CONTENT)
second = DocumentFactory(
filename="b.pdf",
title="b",
mime_type="application/pdf",
)
real_move = move_files_for_document
def failing_for_first(pk: int) -> Document | None:
if pk == first.pk:
raise RuntimeError("boom")
return real_move(pk)
mocker.patch(
"documents.file_moves.move_files_for_document",
side_effect=failing_for_first,
)
with defer_file_moves():
first.title = "a2"
first.save()
second.title = "b2"
second.save()
assert (paperless_dirs.originals_dir / "b2.pdf").is_file()
assert (paperless_dirs.originals_dir / "a.pdf").is_file()
@pytest.mark.django_db
class TestFileMoveMiddleware:
def test_sync_mode_flushes_before_returning(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- The middleware wrapping a sync view that renames the document
WHEN:
- A request passes through it
THEN:
- Nothing moves while the view runs
- The file is at its final path when the middleware returns
"""
def view(request: HttpRequest) -> HttpResponse:
document.title = "renamed"
document.save()
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
return HttpResponse()
middleware = FileMoveMiddleware(view)
assert middleware.async_mode is False
middleware(RequestFactory().get("/"))
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_async_mode_flushes_after_await(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- The middleware wrapping an async view that saves through
sync_to_async
WHEN:
- A request passes through it
THEN:
- Nothing moves while the view runs
- The file is at its final path after the await, so the collector
survived the context copy into the sync thread
"""
def rename() -> None:
document.title = "renamed"
document.save()
async def view(request: HttpRequest) -> HttpResponse:
await sync_to_async(rename)()
assert (paperless_dirs.originals_dir / "original.pdf").is_file()
return HttpResponse()
middleware = FileMoveMiddleware(view)
assert middleware.async_mode is True
async_to_sync(middleware)(RequestFactory().get("/"))
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_view_raising_after_save_still_moves(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A view that saves a rename and then raises
WHEN:
- A request passes through the middleware
THEN:
- The exception propagates
- The committed rename is still moved
"""
def view(request: HttpRequest) -> HttpResponse:
document.title = "renamed"
document.save()
raise RuntimeError("view failed")
with pytest.raises(RuntimeError, match="view failed"):
FileMoveMiddleware(view)(RequestFactory().get("/"))
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
@pytest.mark.django_db
class TestFileMovesThroughApi:
def test_media_lock_never_taken_inside_a_transaction(
self,
admin_client: APIClient,
document: Document,
) -> None:
"""
GIVEN:
- A document and a tag
WHEN:
- A tag + title PATCH and a note POST are made through the API
THEN:
- media.lock is acquired at least once
- media.lock is never first acquired while a transaction is open
"""
tag = TagFactory()
with record_media_lock_acquisitions() as acquisitions:
response = admin_client.patch(
f"/api/documents/{document.pk}/",
{"tags": [tag.pk], "title": "renamed"},
format="json",
)
assert response.status_code == status.HTTP_200_OK
response = admin_client.post(
f"/api/documents/{document.pk}/notes/",
{"note": "a note"},
format="json",
)
assert response.status_code == status.HTTP_200_OK
assert acquisitions
assert not any(a.in_user_transaction for a in acquisitions)
def test_patch_moves_to_final_path(
self,
admin_client: APIClient,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A document named after its title
WHEN:
- Its title and tags are changed through the API
THEN:
- The file is at the final path when the response returns
"""
response = admin_client.patch(
f"/api/documents/{document.pk}/",
{"tags": [TagFactory().pk], "title": "renamed"},
format="json",
)
assert response.status_code == status.HTTP_200_OK
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
assert Document.objects.get(pk=document.pk).filename == "renamed.pdf"
def test_patch_response_reports_post_move_modified(
self,
admin_client: APIClient,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A document named after its title
WHEN:
- Its title is changed through the API, causing a rename
THEN:
- The modified time in the response equals the stored value,
which the file move bumped after the edit was saved
"""
response = admin_client.patch(
f"/api/documents/{document.pk}/",
{"title": "renamed"},
format="json",
)
assert response.status_code == status.HTTP_200_OK
stored = Document.objects.get(pk=document.pk)
assert stored.filename == "renamed.pdf"
assert parse_datetime(response.json()["modified"]) == stored.modified
def test_trash_restore_keeps_db_and_disk_consistent(
self,
admin_client: APIClient,
document: Document,
) -> None:
"""
GIVEN:
- A document moved to the trash
WHEN:
- It is restored through the trash API
THEN:
- Its DB filename points at a file that exists
"""
admin_client.delete(f"/api/documents/{document.pk}/")
response = admin_client.post(
"/api/trash/",
{"action": "restore", "documents": [document.pk]},
format="json",
)
assert response.status_code == status.HTTP_200_OK
restored = Document.objects.get(pk=document.pk)
assert restored.source_path.is_file()
def test_root_storage_path_change_moves_versions(
self,
admin_client: APIClient,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A root document with one version, both with files on disk
WHEN:
- The root's storage path is changed through the API
THEN:
- The root and version files both follow the storage path
"""
(paperless_dirs.originals_dir / "version.pdf").write_bytes(CONTENT)
version = DocumentFactory(
root_document=document,
title="version",
filename="version.pdf",
mime_type="application/pdf",
checksum=hashlib.sha256(CONTENT).hexdigest(),
)
storage_path = StoragePathFactory(path="stored/{{ title }}")
response = admin_client.patch(
f"/api/documents/{document.pk}/",
{"storage_path": storage_path.pk},
format="json",
)
assert response.status_code == status.HTTP_200_OK
document.refresh_from_db()
version.refresh_from_db()
assert document.filename.startswith("stored/")
assert version.filename.startswith("stored/")
assert document.source_path.is_file()
assert version.source_path.is_file()
def test_custom_field_change_moves_to_final_path(
self,
admin_client: APIClient,
document: Document,
paperless_dirs: PaperlessDirs,
settings: Settings,
) -> None:
"""
GIVEN:
- A filename format that uses a text custom field
- A document with that field set
WHEN:
- The field value is changed through the API
THEN:
- The file is at the path for the new value
"""
settings.FILENAME_FORMAT = (
"{{ custom_fields|get_cf_value('flavor') }}/{{ title }}"
)
field = CustomFieldFactory(
name="flavor",
data_type=CustomField.FieldDataType.STRING,
)
CustomFieldInstanceFactory(field=field, document=document, value_text="apple")
response = admin_client.patch(
f"/api/documents/{document.pk}/",
{"custom_fields": [{"field": field.pk, "value": "cherry"}]},
format="json",
)
assert response.status_code == status.HTTP_200_OK
document.refresh_from_db()
assert document.filename == "cherry/original.pdf"
assert (paperless_dirs.originals_dir / "cherry" / "original.pdf").is_file()
@pytest.mark.django_db
class TestCeleryTaskScope:
def test_task_class_opens_scope(self) -> None:
"""
GIVEN:
- The paperless Celery app
WHEN:
- A registered task is inspected
THEN:
- It runs through FileMoveScopedTask
- It keeps Celery's DjangoTask API, such as delay_on_commit
"""
from paperless.celery import FileMoveScopedTask
assert isinstance(bulk_update_documents, FileMoveScopedTask)
assert isinstance(bulk_update_documents, DjangoTask)
def test_failing_task_still_moves_committed_change(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A task that renames a document, then raises
WHEN:
- The task is applied
THEN:
- The task fails with its own error
- Nothing moved while the task body ran
- The committed rename is moved when the task ends
"""
result = rename_then_fail.apply(
args=[document.pk, str(paperless_dirs.originals_dir)],
)
assert result.failed()
assert isinstance(result.result, RuntimeError)
assert (paperless_dirs.originals_dir / "renamed.pdf").is_file()
def test_bulk_update_moves_each_document(
self,
document: Document,
paperless_dirs: PaperlessDirs,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- Two documents whose titles changed in the DB without signals
WHEN:
- bulk_update_documents runs for both
THEN:
- Each document's file follows its new title
"""
mocker.patch("documents.search.get_backend")
(paperless_dirs.originals_dir / "second.pdf").write_bytes(CONTENT)
second = DocumentFactory(
title="second",
filename="second.pdf",
mime_type="application/pdf",
checksum=hashlib.sha256(CONTENT).hexdigest(),
)
Document.objects.filter(pk=document.pk).update(title="first-renamed")
Document.objects.filter(pk=second.pk).update(title="second-renamed")
bulk_update_documents.apply(args=[[document.pk, second.pk]])
assert (paperless_dirs.originals_dir / "first-renamed.pdf").is_file()
assert (paperless_dirs.originals_dir / "second-renamed.pdf").is_file()
def test_scheduled_workflows_move_each_document(
self,
document: Document,
paperless_dirs: PaperlessDirs,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- Two documents matched by a scheduled workflow, where running the
workflow renames the document
WHEN:
- check_scheduled_workflows runs
THEN:
- The first document's file is already moved when the workflow
runs for the second, not only when the task ends
"""
(paperless_dirs.originals_dir / "second.pdf").write_bytes(CONTENT)
second = DocumentFactory(
title="second",
filename="second.pdf",
mime_type="application/pdf",
checksum=hashlib.sha256(CONTENT).hexdigest(),
)
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.SCHEDULED,
schedule_offset_days=0,
schedule_date_field=WorkflowTrigger.ScheduleDateField.ADDED,
)
workflow = Workflow.objects.create(name="scheduled", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(WorkflowAction.objects.create(assign_title="unused"))
seen: dict[int, bool] = {}
def rename(*, document: Document, **kwargs: object) -> None:
seen[document.pk] = (paperless_dirs.originals_dir / "first.pdf").is_file()
document.title = "first" if document.pk == first_pk else "other"
document.save()
first_pk = min(document.pk, second.pk)
mocker.patch("documents.tasks.run_workflows", side_effect=rename)
mocker.patch("documents.tasks.send_websocket_document_updated")
check_scheduled_workflows.apply()
second_pk = max(document.pk, second.pk)
assert seen[first_pk] is False
assert seen[second_pk] is True
@pytest.mark.django_db
class TestEmptyTrashLockOrdering:
def test_hard_delete_takes_media_lock_outside_transaction(
self,
document: Document,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- A trashed document with its file on disk
WHEN:
- The trash is emptied for it
THEN:
- media.lock is never first acquired inside a transaction
- The document row and its file are gone
"""
document.delete()
with record_media_lock_acquisitions() as acquisitions:
empty_trash([document.pk])
assert acquisitions
assert not any(a.in_user_transaction for a in acquisitions)
assert not Document.global_objects.filter(pk=document.pk).exists()
assert not (paperless_dirs.originals_dir / "original.pdf").exists()
+1 -61
View File
@@ -18,13 +18,11 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture from pytest_mock import MockerFixture
from documents.file_handling import generate_filename from documents.file_handling import generate_filename
from documents.file_moves import move_files_for_document
from documents.models import Document from documents.models import Document
from documents.tasks import update_document_content_maybe_archive_file from documents.tasks import update_document_content_maybe_archive_file
from paperless_testing.assertions import FileSystemAssertsMixin from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory from paperless_testing.factories import DocumentFactory
from paperless_testing.locks import record_media_lock_acquisitions
from paperless_testing.samples import SIMPLE_DIGITAL_PDF from paperless_testing.samples import SIMPLE_DIGITAL_PDF
sample_file: Path = SIMPLE_DIGITAL_PDF sample_file: Path = SIMPLE_DIGITAL_PDF
@@ -65,28 +63,6 @@ class TestArchiver(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
self.assertTrue(filecmp.cmp(sample_file, doc.source_path)) self.assertTrue(filecmp.cmp(sample_file, doc.source_path))
self.assertEqual(doc.archive_filename, "none/A.pdf") self.assertEqual(doc.archive_filename, "none/A.pdf")
def test_handle_document_takes_media_lock_outside_transaction(self) -> None:
"""
GIVEN:
- A document whose archive version is being generated
WHEN:
- update_document_content_maybe_archive_file runs
THEN:
- media.lock is acquired
- media.lock is never first acquired while a transaction holds the
document row
"""
doc = self.make_models()
shutil.copy(sample_file, Path(self.dirs.originals_dir) / f"{doc.id:07}.pdf")
with record_media_lock_acquisitions() as acquisitions:
update_document_content_maybe_archive_file(doc.pk)
self.assertTrue(acquisitions)
self.assertFalse(any(a.in_user_transaction for a in acquisitions))
doc.refresh_from_db()
self.assertIsFile(doc.archive_path)
def test_unknown_mime_type(self) -> None: def test_unknown_mime_type(self) -> None:
doc = self.make_models() doc = self.make_models()
doc.mime_type = "sdgfh" doc.mime_type = "sdgfh"
@@ -260,7 +236,7 @@ class TestRenamer(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
doc = Document.objects.create(title="test", mime_type="image/jpeg") doc = Document.objects.create(title="test", mime_type="image/jpeg")
doc.filename = generate_filename(doc) doc.filename = generate_filename(doc)
doc.archive_filename = generate_filename(doc, archive_filename=True) doc.archive_filename = generate_filename(doc, archive_filename=True)
doc.save(write_file_paths=True) doc.save()
Path(doc.source_path).touch() Path(doc.source_path).touch()
Path(doc.archive_path).touch() Path(doc.archive_path).touch()
@@ -277,42 +253,6 @@ class TestRenamer(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
self.assertIsFile(doc2.source_path) self.assertIsFile(doc2.source_path)
self.assertIsFile(doc2.archive_path) self.assertIsFile(doc2.archive_path)
@override_settings(FILENAME_FORMAT="")
def test_rename_moves_each_document_by_direct_call(self) -> None:
"""
GIVEN:
- Two documents whose filename format changed
WHEN:
- document_renamer runs
THEN:
- move_files_for_document is called once per document
- Both files are at their new paths
"""
docs = []
for title in ("first", "second"):
doc = Document.objects.create(title=title, mime_type="image/jpeg")
doc.filename = generate_filename(doc)
doc.save(write_file_paths=True)
Path(doc.source_path).touch()
docs.append(doc)
with (
override_settings(FILENAME_FORMAT="{title}"),
mock.patch(
"documents.management.commands.document_renamer.move_files_for_document",
wraps=move_files_for_document,
) as mover,
):
call_command("document_renamer", skip_checks=True)
self.assertCountEqual(
[c.args[0] for c in mover.call_args_list],
[d.pk for d in docs],
)
for doc in docs:
doc.refresh_from_db()
self.assertIsFile(doc.source_path)
@pytest.mark.management @pytest.mark.management
class TestCreateClassifier: class TestCreateClassifier:
@@ -4,13 +4,10 @@ Tests for the document_retagger management command.
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING
import pytest import pytest
from django.core.management import call_command from django.core.management import call_command
from django.core.management.base import CommandError from django.core.management.base import CommandError
from documents.file_moves import _active_collector
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
@@ -24,12 +21,6 @@ from paperless_testing.factories import DocumentTypeFactory
from paperless_testing.factories import StoragePathFactory from paperless_testing.factories import StoragePathFactory
from paperless_testing.factories import TagFactory from paperless_testing.factories import TagFactory
if TYPE_CHECKING:
from pytest_django.fixtures import Settings
from pytest_mock import MockerFixture
from paperless_testing.dirs import PaperlessDirs
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Module-level type aliases # Module-level type aliases
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -493,69 +484,3 @@ class TestRetaggerEdgeCases(DirectoriesMixin):
assert d_first.tags.count() == 0 assert d_first.tags.count() == 0
assert d_unrelated.tags.count() == 2 assert d_unrelated.tags.count() == 2
@pytest.mark.management
@pytest.mark.django_db
class TestRetaggerFileMoves:
def test_each_document_is_processed_in_its_own_scope(
self,
paperless_dirs: PaperlessDirs,
settings: Settings,
mocker: MockerFixture,
) -> None:
"""
GIVEN:
- Two documents and a filename format using the correspondent
- A classifier suggestion assigning a correspondent to each
WHEN:
- document_retagger runs with --correspondent
THEN:
- Each document is processed inside an open file move scope
- When the second document is processed, the first one's file has
already moved
- Both files end at their final paths
"""
settings.FILENAME_FORMAT = "{correspondent}/{title}"
(paperless_dirs.originals_dir / "none").mkdir(exist_ok=True)
for title in ("first", "second"):
(paperless_dirs.originals_dir / "none" / f"{title}.pdf").write_bytes(b"x")
DocumentFactory(
title=title,
filename=f"none/{title}.pdf",
mime_type="application/pdf",
)
correspondent = CorrespondentFactory(name="acme")
processed: list[Document] = []
scope_open: list[bool] = []
seen_first_moved: list[bool] = []
def fake_set_correspondent(sender, document, **kwargs):
scope_open.append(_active_collector.get() is not None)
# Order-agnostic: on the second call, check the first one's file.
if processed:
first = processed[0]
seen_first_moved.append(
(
paperless_dirs.originals_dir / "acme" / f"{first.title}.pdf"
).is_file(),
)
processed.append(document)
document.correspondent = correspondent
document.save(update_fields=("correspondent",))
return correspondent
mocker.patch(
"documents.management.commands.document_retagger.load_classifier",
)
mocker.patch(
"documents.management.commands.document_retagger.set_correspondent",
side_effect=fake_set_correspondent,
)
call_command("document_retagger", "--correspondent", skip_checks=True)
assert scope_open == [True, True]
assert seen_first_moved == [True]
for title in ("first", "second"):
assert (paperless_dirs.originals_dir / "acme" / f"{title}.pdf").is_file()
-121
View File
@@ -1,121 +0,0 @@
from __future__ import annotations
import threading
from typing import TYPE_CHECKING
import pytest
from django.db import transaction
from filelock import Timeout
from documents.file_moves import in_user_transaction
from documents.media_lock import media_lock
from paperless_testing.locks import record_media_lock_acquisitions
if TYPE_CHECKING:
from paperless_testing.dirs import PaperlessDirs
@pytest.mark.usefixtures("paperless_dirs")
class TestMediaLock:
def test_same_thread_reenters_without_blocking(self) -> None:
"""
GIVEN:
- media.lock held by the current thread
WHEN:
- The same thread acquires it again with a zero timeout
THEN:
- The nested acquisition succeeds immediately
"""
with media_lock(), media_lock(timeout=0):
pass
def test_other_thread_is_excluded(self) -> None:
"""
GIVEN:
- media.lock held by the current thread
WHEN:
- Another thread tries to acquire it with a short timeout
THEN:
- The other thread times out
- After release, the other thread can acquire it
"""
outcomes: list[str] = []
def try_acquire() -> None:
try:
with media_lock(timeout=0.2):
outcomes.append("acquired")
except Timeout:
outcomes.append("timeout")
with media_lock():
worker = threading.Thread(target=try_acquire)
worker.start()
worker.join()
worker = threading.Thread(target=try_acquire)
worker.start()
worker.join()
assert outcomes == ["timeout", "acquired"]
def test_lock_file_is_the_configured_path(
self,
paperless_dirs: PaperlessDirs,
) -> None:
"""
GIVEN:
- The test's media directory layout
WHEN:
- media.lock is acquired
THEN:
- The lock file is created at settings.MEDIA_LOCK
"""
with media_lock():
assert paperless_dirs.media_lock.exists()
@pytest.mark.django_db
class TestInUserTransaction:
def test_false_inside_test_case_wrapper(self) -> None:
"""
GIVEN:
- The transaction pytest-django wraps around the test
WHEN:
- in_user_transaction is called
THEN:
- It returns False
"""
assert in_user_transaction() is False
def test_true_inside_atomic(self) -> None:
"""
GIVEN:
- An atomic block opened by code under test
WHEN:
- in_user_transaction is called inside it
THEN:
- It returns True
"""
with transaction.atomic():
assert in_user_transaction() is True
@pytest.mark.django_db
@pytest.mark.usefixtures("paperless_dirs")
class TestRecordMediaLockAcquisitions:
def test_records_only_first_acquisitions(self) -> None:
"""
GIVEN:
- Nested media.lock acquisitions, the outer one inside a transaction
WHEN:
- They are recorded
THEN:
- Only the outer, first acquisition is recorded, flagged as inside
a transaction
"""
with record_media_lock_acquisitions() as acquisitions:
with transaction.atomic(), media_lock(), media_lock():
pass
assert [a.in_user_transaction for a in acquisitions] == [True]
-101
View File
@@ -1,7 +1,4 @@
import pytest import pytest
from django.db import DatabaseError
from django.db import models
from django.db import transaction
from documents.models import Correspondent from documents.models import Correspondent
from documents.models import Document from documents.models import Document
@@ -25,101 +22,3 @@ class TestDocument:
assert Correspondent.objects.count() == 0 assert Correspondent.objects.count() == 0
assert Document.objects.count() == 1 assert Document.objects.count() == 1
assert Document.objects.first().correspondent is None assert Document.objects.first().correspondent is None
@pytest.mark.django_db
@pytest.mark.usefixtures("paperless_dirs")
class TestDocumentFilePathWrites:
def test_full_save_does_not_write_file_paths(self) -> None:
"""
GIVEN:
- A document whose filename columns were changed in the DB after
an instance was loaded
WHEN:
- The stale instance is saved without update_fields
THEN:
- The DB filename columns keep the newer values
- Other fields from the stale instance are written
"""
doc = DocumentFactory(filename="old.pdf", archive_filename="old.pdf")
stale = Document.objects.get(pk=doc.pk)
Document.objects.filter(pk=doc.pk).update(
filename="new.pdf",
archive_filename="new-archive.pdf",
)
stale.title = "changed"
stale.save()
doc.refresh_from_db()
assert doc.filename == "new.pdf"
assert doc.archive_filename == "new-archive.pdf"
assert doc.title == "changed"
@pytest.mark.parametrize(
"save_kwargs",
[
pytest.param({"write_file_paths": True}, id="write-file-paths"),
pytest.param(
{"update_fields": ["filename", "archive_filename"]},
id="explicit-update-fields",
),
],
)
def test_opt_in_writes_file_paths(self, save_kwargs: dict[str, object]) -> None:
"""
GIVEN:
- An existing document
WHEN:
- Its filename columns are changed in memory and saved with an
explicit opt in
THEN:
- The DB filename columns are updated
"""
doc = DocumentFactory(filename="old.pdf", archive_filename="old.pdf")
doc.filename = "new.pdf"
doc.archive_filename = "new.pdf"
doc.save(**save_kwargs)
doc.refresh_from_db()
assert doc.filename == "new.pdf"
assert doc.archive_filename == "new.pdf"
def test_creation_writes_file_paths(self) -> None:
"""
GIVEN:
- A new, unsaved document with filename columns set
WHEN:
- It is saved
THEN:
- The filename columns are written
"""
doc = DocumentFactory.build(filename="a.pdf", archive_filename="b.pdf")
doc.save()
doc.refresh_from_db()
assert doc.filename == "a.pdf"
assert doc.archive_filename == "b.pdf"
def test_full_save_of_deleted_row_raises(self) -> None:
"""
GIVEN:
- An instance whose row was hard-deleted elsewhere
WHEN:
- The instance is saved without update_fields
THEN:
- A DatabaseError is raised instead of re-inserting the row
"""
doc = DocumentFactory()
stale = Document.objects.get(pk=doc.pk)
# A plain QuerySet bypasses the soft-delete manager, so the row is
# really gone (as after emptying the trash).
models.QuerySet(model=Document).filter(pk=doc.pk).delete()
# The savepoint keeps the failed UPDATE from poisoning the test transaction.
with pytest.raises(DatabaseError), transaction.atomic():
stale.save()
assert not Document.global_objects.filter(pk=doc.pk).exists()
+18 -1
View File
@@ -4,6 +4,7 @@ from django.conf import settings
from pytest_mock import MockerFixture from pytest_mock import MockerFixture
from documents.regex import safe_regex_finditer from documents.regex import safe_regex_finditer
from documents.regex import safe_regex_fullmatch
from documents.regex import safe_regex_match from documents.regex import safe_regex_match
from documents.regex import safe_regex_search from documents.regex import safe_regex_search
from documents.regex import safe_regex_sub from documents.regex import safe_regex_sub
@@ -26,7 +27,7 @@ class TestValidateRegexPattern:
class TestSafeRegexSearchAndMatch: class TestSafeRegexSearchAndMatch:
"""Tests for safe_regex_search and safe_regex_match (same contract).""" """Tests for safe_regex_search, _match and _fullmatch (same contract)."""
@pytest.mark.parametrize( @pytest.mark.parametrize(
("func", "pattern", "text", "expected_group"), ("func", "pattern", "text", "expected_group"),
@@ -45,6 +46,13 @@ class TestSafeRegexSearchAndMatch:
"123", "123",
id="match-match-found", id="match-match-found",
), ),
pytest.param(
safe_regex_fullmatch,
r"\d+",
"123",
"123",
id="fullmatch-match-found",
),
], ],
) )
def test_match_found(self, func, pattern, text, expected_group) -> None: def test_match_found(self, func, pattern, text, expected_group) -> None:
@@ -57,6 +65,12 @@ class TestSafeRegexSearchAndMatch:
[ [
pytest.param(safe_regex_search, r"\d+", "abcdef", id="search-no-match"), pytest.param(safe_regex_search, r"\d+", "abcdef", id="search-no-match"),
pytest.param(safe_regex_match, r"\d+", "abc123", id="match-no-match"), pytest.param(safe_regex_match, r"\d+", "abc123", id="match-no-match"),
pytest.param(
safe_regex_fullmatch,
r"\d+",
"123abc",
id="fullmatch-no-match",
),
], ],
) )
def test_no_match(self, func, pattern, text) -> None: def test_no_match(self, func, pattern, text) -> None:
@@ -67,6 +81,7 @@ class TestSafeRegexSearchAndMatch:
[ [
pytest.param(safe_regex_search, id="search"), pytest.param(safe_regex_search, id="search"),
pytest.param(safe_regex_match, id="match"), pytest.param(safe_regex_match, id="match"),
pytest.param(safe_regex_fullmatch, id="fullmatch"),
], ],
) )
def test_invalid_pattern_returns_none(self, func) -> None: def test_invalid_pattern_returns_none(self, func) -> None:
@@ -77,6 +92,7 @@ class TestSafeRegexSearchAndMatch:
[ [
pytest.param(safe_regex_search, id="search"), pytest.param(safe_regex_search, id="search"),
pytest.param(safe_regex_match, id="match"), pytest.param(safe_regex_match, id="match"),
pytest.param(safe_regex_fullmatch, id="fullmatch"),
], ],
) )
def test_flags_respected(self, func) -> None: def test_flags_respected(self, func) -> None:
@@ -87,6 +103,7 @@ class TestSafeRegexSearchAndMatch:
[ [
pytest.param(safe_regex_search, "search", id="search"), pytest.param(safe_regex_search, "search", id="search"),
pytest.param(safe_regex_match, "match", id="match"), pytest.param(safe_regex_match, "match", id="match"),
pytest.param(safe_regex_fullmatch, "fullmatch", id="fullmatch"),
], ],
) )
def test_timeout_returns_none( def test_timeout_returns_none(
+2 -2
View File
@@ -132,7 +132,7 @@ class TestCheckSanityOriginal:
class TestCheckSanityArchive: class TestCheckSanityArchive:
def test_checksum_without_filename(self, sample_doc: Document) -> None: def test_checksum_without_filename(self, sample_doc: Document) -> None:
sample_doc.archive_filename = None sample_doc.archive_filename = None
sample_doc.save(write_file_paths=True) sample_doc.save()
messages = check_sanity() messages = check_sanity()
assert messages.has_error assert messages.has_error
assert any( assert any(
@@ -185,7 +185,7 @@ class TestCheckSanityArchive:
Path(sample_doc.archive_path).unlink() Path(sample_doc.archive_path).unlink()
sample_doc.archive_checksum = None sample_doc.archive_checksum = None
sample_doc.archive_filename = None sample_doc.archive_filename = None
sample_doc.save(write_file_paths=True) sample_doc.save()
messages = check_sanity() messages = check_sanity()
assert not messages.has_error assert not messages.has_error
@@ -366,7 +366,7 @@ class ShareLinkBundleBuildTaskTests(DirectoriesMixin, APITestCase):
def _write_document_file(self, *, archive: bool, content: bytes) -> Path: def _write_document_file(self, *, archive: bool, content: bytes) -> Path:
if archive: if archive:
self.document.archive_filename = f"{self.document.pk:07}.pdf" self.document.archive_filename = f"{self.document.pk:07}.pdf"
self.document.save(write_file_paths=True) self.document.save()
path = self.document.archive_path path = self.document.archive_path
assert path is not None assert path is not None
else: else:
+62
View File
@@ -12,11 +12,73 @@ from documents.models import WorkflowTrigger
from documents.serialisers import TagSerializer from documents.serialisers import TagSerializer
from documents.signals.handlers import run_workflows from documents.signals.handlers import run_workflows
from paperless_testing.dirs import DirectoriesMixin from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import TagFactory
from paperless_testing.factories import UserFactory from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
class TestTagHierarchyPermissions(APITestCase): class TestTagHierarchyPermissions(APITestCase):
def test_delete_checks_descendant_permissions(self) -> None:
for mode in ("detail", "bulk", "all"):
for ownership in ("owned", "unowned", "private", "shared", "superuser"):
with self.subTest(mode=mode, ownership=ownership):
requester = UserFactory(superuser=ownership == "superuser")
grant_global(requester, "view_tag", "delete_tag")
self.client.force_authenticate(user=requester)
if ownership == "owned":
descendant_owner = requester
elif ownership == "unowned":
descendant_owner = None
else:
descendant_owner = UserFactory()
parent = TagFactory(owner=requester)
child = TagFactory(tn_parent=parent)
grandchild = TagFactory(
owner=descendant_owner,
tn_parent=child,
)
if ownership == "shared":
grant_object(requester, grandchild, "delete_tag")
document = DocumentFactory()
document.tags.add(grandchild)
ids = [parent.pk, child.pk, grandchild.pk]
if mode == "detail":
response = self.client.delete(f"/api/tags/{parent.pk}/")
else:
selection = (
{"all": True, "filters": {"id__in": str(parent.pk)}}
if mode == "all"
else {"objects": [parent.pk]}
)
response = self.client.post(
"/api/bulk_edit_objects/",
{"object_type": "tags", "operation": "delete", **selection},
format="json",
)
# Detail deletion accepts explicit delete grants; bulk
# deletion requires ownership (or an unowned object).
allowed = ownership in ("owned", "unowned", "superuser") or (
ownership == "shared" and mode == "detail"
)
expected_status = status.HTTP_403_FORBIDDEN
if allowed:
expected_status = (
status.HTTP_204_NO_CONTENT
if mode == "detail"
else status.HTTP_200_OK
)
assert response.status_code == expected_status
assert Tag.objects.filter(pk__in=ids).count() == (
0 if allowed else 3
)
assert document.tags.filter(pk=grandchild.pk).exists() == (
not allowed
)
def test_children_only_include_visible_tags(self) -> None: def test_children_only_include_visible_tags(self) -> None:
owner = UserFactory(username="owner") owner = UserFactory(username="owner")
requester = UserFactory(username="requester") requester = UserFactory(username="requester")
+38
View File
@@ -13,14 +13,52 @@ from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource from documents.data_models import DocumentSource
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.signals.handlers import before_task_publish_handler from documents.signals.handlers import before_task_publish_handler
from documents.signals.handlers import run_zone_ocr_extraction
from documents.signals.handlers import task_failure_handler from documents.signals.handlers import task_failure_handler
from documents.signals.handlers import task_postrun_handler from documents.signals.handlers import task_postrun_handler
from documents.signals.handlers import task_prerun_handler from documents.signals.handlers import task_prerun_handler
from documents.signals.handlers import task_revoked_handler from documents.signals.handlers import task_revoked_handler
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import PaperlessTaskFactory from paperless_testing.factories import PaperlessTaskFactory
from paperless_testing.factories import UserFactory from paperless_testing.factories import UserFactory
@pytest.mark.django_db
@pytest.mark.parametrize(
"original_file",
[None, "/tmp/original.pdf", Path("/tmp/original.pdf")],
)
def test_consumption_runs_zone_extraction(
original_file,
mocker: pytest_mock.MockerFixture,
):
extract = mocker.patch("documents.zone_ocr.run_zone_extraction")
document = DocumentFactory()
run_zone_ocr_extraction(sender=None, document=document, original_file=original_file)
extract.assert_called_once_with(
document,
Path(original_file) if original_file else None,
)
@pytest.mark.django_db
def test_zone_extraction_failure_does_not_fail_consumption(
mocker: pytest_mock.MockerFixture,
caplog,
):
mocker.patch(
"documents.zone_ocr.run_zone_extraction",
side_effect=RuntimeError("OCR failed"),
)
document = DocumentFactory()
run_zone_ocr_extraction(sender=None, document=document)
assert f"Zone OCR extraction failed for document {document.pk}" in caplog.text
@pytest.fixture @pytest.fixture
def consume_input_doc(): def consume_input_doc():
doc = mock.MagicMock(spec=ConsumableDocument) doc = mock.MagicMock(spec=ConsumableDocument)
File diff suppressed because it is too large. Load diff
+271 -3
View File
@@ -3,10 +3,12 @@ import logging
import os import os
import platform import platform
import re import re
import subprocess
import tempfile import tempfile
import zipfile import zipfile
from collections import deque from collections import deque
from collections.abc import Callable from collections.abc import Callable
from dataclasses import asdict
from datetime import datetime from datetime import datetime
from datetime import timedelta from datetime import timedelta
from http import HTTPStatus from http import HTTPStatus
@@ -126,7 +128,6 @@ from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource from documents.data_models import DocumentSource
from documents.file_handling import format_filename from documents.file_handling import format_filename
from documents.file_moves import flush_pending_file_moves
from documents.filters import CorrespondentFilterSet from documents.filters import CorrespondentFilterSet
from documents.filters import CustomFieldFilterSet from documents.filters import CustomFieldFilterSet
from documents.filters import DocumentFilterSet from documents.filters import DocumentFilterSet
@@ -153,6 +154,8 @@ from documents.models import CustomFieldInstance
from documents.models import Document from documents.models import Document
from documents.models import DocumentType from documents.models import DocumentType
from documents.models import Note from documents.models import Note
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.models import SavedView from documents.models import SavedView
from documents.models import ShareLink from documents.models import ShareLink
@@ -182,6 +185,7 @@ from documents.permissions import restrict_queryset_to_visible
from documents.permissions import set_permissions_for_objects from documents.permissions import set_permissions_for_objects
from documents.permissions import user_is_unrestricted from documents.permissions import user_is_unrestricted
from documents.plugins.date_parsing import get_date_parser from documents.plugins.date_parsing import get_date_parser
from documents.regex import safe_regex_fullmatch
from documents.schema import generate_object_with_permissions_schema from documents.schema import generate_object_with_permissions_schema
from documents.search import SearchHit from documents.search import SearchHit
from documents.serialisers import AcknowledgeTasksViewSerializer from documents.serialisers import AcknowledgeTasksViewSerializer
@@ -202,11 +206,15 @@ from documents.serialisers import EmailSerializer
from documents.serialisers import MergeDocumentsAsVersionsSerializer from documents.serialisers import MergeDocumentsAsVersionsSerializer
from documents.serialisers import MergeDocumentsSerializer from documents.serialisers import MergeDocumentsSerializer
from documents.serialisers import NotesSerializer from documents.serialisers import NotesSerializer
from documents.serialisers import OcrTemplateSerializer
from documents.serialisers import OcrTemplateTestSerializer
from documents.serialisers import PostDocumentSerializer from documents.serialisers import PostDocumentSerializer
from documents.serialisers import RemovePasswordDocumentsSerializer from documents.serialisers import RemovePasswordDocumentsSerializer
from documents.serialisers import ReprocessDocumentsSerializer from documents.serialisers import ReprocessDocumentsSerializer
from documents.serialisers import RotateDocumentsSerializer from documents.serialisers import RotateDocumentsSerializer
from documents.serialisers import RunTaskSerializer from documents.serialisers import RunTaskSerializer
from documents.serialisers import RunZoneOcrDocumentsSerializer
from documents.serialisers import RunZoneOcrSerializer
from documents.serialisers import SavedViewSerializer from documents.serialisers import SavedViewSerializer
from documents.serialisers import SearchResultSerializer from documents.serialisers import SearchResultSerializer
from documents.serialisers import SerializerWithPerms from documents.serialisers import SerializerWithPerms
@@ -577,6 +585,15 @@ class CorrespondentViewSet(
return super().retrieve(request, *args, **kwargs) return super().retrieve(request, *args, **kwargs)
def _with_tag_descendants(tags: QuerySet[Tag]) -> QuerySet[Tag]:
"""Include every tag that deleting the selected tags will affect."""
tag_ids = set()
for tag in tags:
tag_ids.add(tag.pk)
tag_ids.update(tag.get_descendants_pks())
return Tag.objects.filter(pk__in=tag_ids)
@extend_schema_view(**generate_object_with_permissions_schema(TagSerializer)) @extend_schema_view(**generate_object_with_permissions_schema(TagSerializer))
class TagViewSet(PermissionsAwareDocumentCountMixin, ModelViewSet[Tag]): class TagViewSet(PermissionsAwareDocumentCountMixin, ModelViewSet[Tag]):
model = Tag model = Tag
@@ -598,6 +615,14 @@ class TagViewSet(PermissionsAwareDocumentCountMixin, ModelViewSet[Tag]):
filterset_class = TagFilterSet filterset_class = TagFilterSet
ordering_fields = ("color", "name", "matching_algorithm", "match", "document_count") ordering_fields = ("color", "name", "matching_algorithm", "match", "document_count")
def perform_destroy(self, instance):
tags = _with_tag_descendants(Tag.objects.filter(pk=instance.pk))
if tags.exclude(
pk__in=permitted_object_ids(self.request.user, Tag, "delete_tag"),
).exists():
raise PermissionDenied(_("Insufficient permissions."))
super().perform_destroy(instance)
def get_serializer_context(self): def get_serializer_context(self):
context = super().get_serializer_context() context = super().get_serializer_context()
context["document_count_filter"] = self.get_document_count_filter() context["document_count_filter"] = self.get_document_count_filter()
@@ -1265,8 +1290,6 @@ class DocumentViewSet(
) )
serializer.is_valid(raise_exception=True) serializer.is_valid(raise_exception=True)
self.perform_update(serializer) self.perform_update(serializer)
# The response must reflect the post-move filename and modified time
flush_pending_file_moves()
if content_updated and content_doc.id != root_doc.id: if content_updated and content_doc.id != root_doc.id:
content_doc.content = ( content_doc.content = (
@@ -2226,6 +2249,76 @@ class DocumentViewSet(
raise Http404 raise Http404
return version_doc return version_doc
@extend_schema(
operation_id="documents_run_zone_ocr",
request=RunZoneOcrSerializer,
responses=inline_serializer(
name="RunZoneOcrResult",
fields={
"status": serializers.ChoiceField(choices=["completed", "skipped"]),
"reason": serializers.ChoiceField(
choices=["no_template"],
required=False,
),
"results": serializers.ListField(child=serializers.DictField()),
},
),
)
@action(
methods=["post"],
detail=True,
url_path="run-zone-ocr",
permission_classes=[IsAuthenticated],
)
def run_zone_ocr(self, request, pk=None):
"""
Run zone-based OCR extraction on this document, with the given template
or with every template that matches it.
"""
serializer = RunZoneOcrSerializer(data=request.data)
serializer.is_valid(raise_exception=True)
template = serializer.validated_data.get("template")
document = get_object_or_404(
Document.objects.select_related("root_document__owner"),
pk=pk,
)
if not request.user.has_perm(
"documents.change_document",
) or not has_perms_owner_aware(request.user, "change_document", document):
return HttpResponseForbidden("Insufficient permissions")
if template is not None and not has_perms_owner_aware(
request.user,
"view_ocrtemplate",
template,
):
return HttpResponseForbidden("Insufficient permissions")
from documents.zone_ocr import get_templates_for_document
templates = (
[template] if template is not None else get_templates_for_document(document)
)
if not templates:
return Response(
{"status": "skipped", "reason": "no_template", "results": []},
)
doc_path = document.archive_path or document.source_path
if not doc_path or not Path(doc_path).is_file():
return Response(
{"error": "Document file not found"},
status=status.HTTP_404_NOT_FOUND,
)
from documents.zone_ocr import run_zone_extraction
results = run_zone_extraction(document, None, templates=templates)
return Response(
{"status": "completed", "results": [asdict(result) for result in results]},
)
@extend_schema( @extend_schema(
operation_id="documents_delete_version", operation_id="documents_delete_version",
parameters=[ parameters=[
@@ -3392,6 +3485,40 @@ class ReprocessDocumentsView(DocumentOperationPermissionMixin):
) )
@extend_schema_view(
post=extend_schema(
operation_id="documents_run_zone_ocr_bulk",
description="Run OCR template zone extraction on selected documents",
responses={
200: inline_serializer(
name="RunZoneOcrDocumentsResult",
fields={
"result": serializers.CharField(),
},
),
},
),
)
class RunZoneOcrDocumentsView(DocumentOperationPermissionMixin):
serializer_class = RunZoneOcrDocumentsSerializer
def post(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
template = serializer.validated_data.get("template")
if template is not None and not has_perms_owner_aware(
request.user,
"view_ocrtemplate",
template,
):
return HttpResponseForbidden("Insufficient permissions")
return self._execute_document_action(
method=bulk_edit.run_zone_ocr,
validated_data=serializer.validated_data,
operation_label="document zone OCR",
)
@extend_schema_view( @extend_schema_view(
post=extend_schema( post=extend_schema(
operation_id="documents_edit_pdf", operation_id="documents_edit_pdf",
@@ -5078,6 +5205,9 @@ class BulkEditObjectsView(PassUserMixin):
pk__in=object_ids, pk__in=object_ids,
) )
if object_type == "tags" and operation == "delete":
objs = _with_tag_descendants(objs)
if not user.is_superuser: if not user.is_superuser:
perm = f"documents.{perm_codename}" perm = f"documents.{perm_codename}"
# Limited to the owner (or unowned), same as documents, see BulkEditView # Limited to the owner (or unowned), same as documents, see BulkEditView
@@ -5691,3 +5821,141 @@ def serve_logo(request: HttpRequest, filename: str | None = None) -> FileRespons
filename=logo_name, filename=logo_name,
as_attachment=True, as_attachment=True,
) )
class OcrTemplateViewSet(BulkPermissionMixin, PassUserMixin, ModelViewSet[OcrTemplate]):
"""CRUD for OCR templates with zone definitions."""
model = OcrTemplate
queryset = (
OcrTemplate.objects.select_related("owner")
.prefetch_related(
"zones",
"zones__custom_field",
)
.order_by("name")
)
serializer_class = OcrTemplateSerializer
pagination_class = StandardPagination
permission_classes = (IsAuthenticated, PaperlessObjectPermissions)
filter_backends = (
OrderingFilter,
PermittedObjectsFilter,
)
ordering_fields = ("name",)
@action(
detail=False,
methods=["get"],
url_path=r"document-page-image/(?P<doc_id>[0-9]+)/(?P<page>[0-9]+)",
)
def document_page_image(self, request, doc_id=None, page=None):
"""Render a page for the template editor's zone overlay."""
document = get_object_or_404(
Document.objects.select_related("root_document__owner"),
pk=doc_id,
)
if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions")
page_num = int(page)
if document.page_count and page_num >= document.page_count:
raise Http404(
f"Page {page_num} out of range (document has {document.page_count} pages)",
)
doc_path = document.archive_path or document.source_path
if not doc_path or not Path(doc_path).is_file():
raise Http404("Document file not found")
if document.mime_type and document.mime_type.startswith("image/"):
content = Path(doc_path).read_bytes()
return HttpResponse(content, content_type=document.mime_type)
with tempfile.TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir:
output_prefix = Path(tmp_dir) / "page"
try:
subprocess.run(
[
"pdftoppm",
"-png",
"-r",
"150",
"-f",
str(page_num + 1),
"-l",
str(page_num + 1),
str(doc_path),
str(output_prefix),
],
check=True,
capture_output=True,
timeout=30,
)
except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as error:
logger.warning(
"Failed to render page %s of document %s: %s",
page_num,
document.pk,
error,
)
raise Http404("Failed to render page") from error
except FileNotFoundError as error:
logger.warning("pdftoppm is not available")
raise Http404("Failed to render page") from error
rendered = sorted(Path(tmp_dir).glob("page-*.png"))
if not rendered:
raise Http404("No rendered page found")
content = rendered[0].read_bytes()
return HttpResponse(content, content_type="image/png")
@extend_schema(request=OcrTemplateTestSerializer)
@action(detail=False, methods=["post"], url_path="test-zone")
def test_zone(self, request):
"""Extract an ad-hoc zone without changing the document."""
from documents.zone_ocr import extract_zone_preview
serializer = OcrTemplateTestSerializer(data=request.data)
serializer.is_valid(raise_exception=True)
document = serializer.validated_data["document"]
if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions")
doc_path = document.archive_path or document.source_path
if not doc_path or not Path(doc_path).is_file():
return Response(
{"error": "Document file not found"},
status=status.HTTP_404_NOT_FOUND,
)
zone = OcrTemplateZone(**serializer.validated_data["zone"])
result = extract_zone_preview(
Path(doc_path),
zone,
zone.zone_source_width or 0,
zone.zone_source_height or 0,
document.page_count,
)
regex_match = None
if zone.validation_regex and result.get("value") is not None:
regex_match = (
safe_regex_fullmatch(zone.validation_regex, result["value"]) is not None
)
return Response(
{
"raw_text": result.get("raw_text"),
"value": result.get("value"),
"regex": zone.validation_regex,
"regex_match": regex_match,
},
)
+672
View File
@@ -0,0 +1,672 @@
from __future__ import annotations
import logging
import math
import re
import string
import subprocess
import tempfile
from dataclasses import dataclass
from datetime import date
from datetime import datetime
from decimal import Decimal
from decimal import InvalidOperation
from pathlib import Path
import dateparser
import magic
import zxingcpp
from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import transaction
from PIL import Image
from documents.matching import matches
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import OcrTemplate
from documents.models import OcrTemplateZone
from documents.permissions import permitted_object_ids
from documents.regex import safe_regex_fullmatch
from documents.versioning import get_root_document
logger = logging.getLogger("paperless.zone_ocr")
@dataclass(frozen=True)
class ZoneExtraction:
raw_text: str | None = None
value: str | None = None
@dataclass(frozen=True)
class FieldUpdate:
field_name: str
value: object
custom_field: CustomField | None = None
@dataclass
class ZoneOcrResult:
template: str
zone: str
custom_field: str
value: object = None
reason: str | None = None
def _zone_result(
template: OcrTemplate,
zone: OcrTemplateZone,
*,
value: object = None,
reason: str | None = None,
) -> ZoneOcrResult:
if zone.target == OcrTemplateZone.TargetType.CUSTOM_FIELD:
label = zone.custom_field.name if zone.custom_field_id else "(no field)"
else:
label = {"title": "Title", "asn": "ASN", "created": "Created"}[zone.target]
return ZoneOcrResult(template.name, zone.name, label, value, reason)
def get_templates_for_document(document: Document) -> list[OcrTemplate]:
"""
The enabled templates that match the document's content and that its owner
can view, the same rule as matching. A version is owned through its root
document.
"""
owner = get_root_document(document).owner
templates = (
OcrTemplate.objects.filter(
enabled=True,
id__in=permitted_object_ids(owner, OcrTemplate, "view_ocrtemplate"),
)
.order_by("name", "pk")
.prefetch_related("zones", "zones__custom_field")
)
return [template for template in templates if matches(template, document)]
def run_zone_extraction(
document: Document,
original_file: Path | None,
templates: list[OcrTemplate] | None = None,
) -> list[ZoneOcrResult]:
"""
Run zone-based OCR extraction for the given templates, or for every
template that matches the document when none are given.
"""
if templates is None:
templates = get_templates_for_document(document)
if not templates:
return []
doc_path = _resolve_doc_path(document, original_file)
if doc_path is None:
logger.warning(
"Zone OCR: no accessible file for document %d",
document.pk,
)
return [
_zone_result(template, zone, reason="missing_file")
for template in templates
for zone in template.zones.all()
]
template_zones = [(template, list(template.zones.all())) for template in templates]
pages = {
_resolve_page_idx(zone.page, document.page_count)
for _, zones in template_zones
for zone in zones
}
if not pages:
return []
results = []
with tempfile.TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir:
tmp_path = Path(tmp_dir)
page_images = _render_pages(doc_path, pages, tmp_path, document.page_count)
for template, zones in template_zones:
try:
results.extend(
_process_template(document, page_images, template, zones, tmp_path),
)
except Exception:
# One failed template must not prevent the remaining templates from running.
logger.exception(
"Zone OCR: error processing template '%s' for document %d",
template.name,
document.pk,
)
results.extend(
_zone_result(template, zone, reason="template_failed")
for zone in zones
)
return results
def _resolve_doc_path(
document: Document,
original_file: Path | None,
) -> Path | None:
"""Prefer the archive, then the source, then the ingestion signal's file."""
candidates = []
if document.has_archive_version:
candidates.append(document.archive_path)
candidates.append(document.source_path)
if original_file is not None:
candidates.append(original_file)
for path in candidates:
if path is not None and Path(path).is_file():
return Path(path)
return None
def _resolve_page_idx(page_value: int | None, page_count: int | None) -> int:
"""Resolve a 1-based page or -1 (last page), defaulting to the first page."""
if page_value == -1:
return max((page_count or 1) - 1, 0)
return max((page_value or 1) - 1, 0)
def _process_template(
document: Document,
page_images: dict[int, Path],
template: OcrTemplate,
zones: list[OcrTemplateZone],
tmp_path: Path,
) -> list[ZoneOcrResult]:
"""Extract and combine zone values before applying a template's updates."""
grouped: dict[str, list[tuple[OcrTemplateZone, ZoneOcrResult]]] = {}
results = []
for zone in zones:
page_idx = _resolve_page_idx(zone.page, document.page_count)
extracted = None
reason = "missing_page"
if page_idx in page_images:
extraction = _extract_zone(
page_images[page_idx],
zone,
zone.zone_source_width or template.source_width,
zone.zone_source_height or template.source_height,
tmp_path,
)
extracted = extraction.value
reason = None if extracted else "no_value"
if (
extracted is not None
and zone.validation_regex
and not safe_regex_fullmatch(zone.validation_regex, extracted)
):
reason = "regex_mismatch"
result = _zone_result(
template,
zone,
value=extracted if reason is None else None,
reason=reason,
)
results.append(result)
grouped.setdefault(_field_key(zone), []).append((zone, result))
updates = []
combine_formats = template.combine_formats or {}
for key, field_zones in grouped.items():
accepted = [result for _, result in field_zones if result.reason is None]
if not accepted:
continue
value = _combine_field_value(
combine_formats.get(key, ""),
[(zone.name, result.value) for zone, result in field_zones],
)
target_zone = field_zones[0][0]
update = _prepare_field_update(target_zone, value) if value else None
if update is not None:
updates.append(update)
for result in accepted:
result.value = update.value if update else None
if update is None:
result.reason = (
"unassigned" if key == "custom_field" else "invalid_value"
)
_apply_field_updates(document, updates)
return results
def _field_key(zone: OcrTemplateZone) -> str:
"""Use the same target keys as the editor's combination formats."""
target = zone.target
if target == OcrTemplateZone.TargetType.CUSTOM_FIELD and zone.custom_field_id:
return str(zone.custom_field_id)
return target
def _combine_field_value(
fmt: str,
field_zones: list[tuple[str, str | None]],
) -> str:
"""Combine the OCR values of all zones targeting one field.
With a format string, `{Zone Name}` tokens are replaced by that zone's value
and literal text is kept; separators left dangling by an empty token are
cleaned up. Without a format, the zone values are joined in order by a space.
"""
values = {name: value or "" for name, value in field_zones}
if not fmt:
return " ".join(value for _, value in field_zones if value).strip()
def _replace(match: re.Match) -> str:
return values.get(match.group(1).strip(), "")
combined = re.sub(r"\{([^{}]+)\}", _replace, fmt)
# Tidy up separators an empty token may have left behind.
combined = re.sub(r"\s{2,}", " ", combined)
combined = re.sub(r"([^\w\s])\s*\1+", r"\1", combined)
return combined.strip().strip("-/.,;:| \t")
def _render_pages(
doc_path: Path,
pages: set[int],
tmp_dir: Path,
page_count: int | None,
) -> dict[int, Path]:
"""Render specific PDF pages as PNG images using pdftoppm (poppler-utils)."""
result: dict[int, Path] = {}
mime = _detect_mime(doc_path)
if mime and mime.startswith("image/"):
if 0 in pages:
result[0] = doc_path
return result
for actual_page in sorted(pages):
if actual_page < 0 or (page_count and actual_page >= page_count):
continue
output_prefix = tmp_dir / f"page_{actual_page}"
try:
subprocess.run(
[
"pdftoppm",
"-png",
"-singlefile",
"-r",
"300",
"-f",
str(actual_page + 1), # pdftoppm is 1-indexed
"-l",
str(actual_page + 1),
str(doc_path),
str(output_prefix),
],
check=True,
capture_output=True,
timeout=60,
)
except subprocess.TimeoutExpired:
logger.error("Zone OCR: pdftoppm timed out for page %d", actual_page)
continue
except subprocess.CalledProcessError as e:
logger.error(
"Zone OCR: pdftoppm failed for page %d: %s",
actual_page,
e.stderr.decode(errors="replace") if e.stderr else str(e),
)
continue
except FileNotFoundError:
logger.error("Zone OCR: pdftoppm not found")
return result
rendered = output_prefix.with_suffix(".png")
if rendered.is_file():
result[actual_page] = rendered
return result
def _crop_zone(
page_img: Path,
zone: OcrTemplateZone,
source_width: int,
source_height: int,
) -> Image.Image | None:
"""Crop a zone from the page image and return the PIL Image."""
try:
with Image.open(page_img) as img:
img_width, img_height = img.size
if not source_width or not source_height:
source_width, source_height = img.size
scale_x = img_width / source_width
scale_y = img_height / source_height
crop_left = int(zone.x * scale_x)
crop_top = int(zone.y * scale_y)
crop_right = int((zone.x + zone.width) * scale_x)
crop_bottom = int((zone.y + zone.height) * scale_y)
# Clamp to the image so an oversized zone can't crop out of bounds.
crop_left = max(0, min(crop_left, img_width))
crop_top = max(0, min(crop_top, img_height))
crop_right = min(crop_right, img_width)
crop_bottom = min(crop_bottom, img_height)
if crop_right - crop_left < 2 or crop_bottom - crop_top < 2:
logger.warning("Zone OCR: crop too small for zone '%s'", zone.name)
return None
return img.crop((crop_left, crop_top, crop_right, crop_bottom))
except (OSError, ValueError, Image.DecompressionBombError):
logger.exception("Zone OCR: crop failed for zone '%s'", zone.name)
return None
def _read_barcode(cropped: Image.Image, zone_name: str) -> str | None:
"""Read QR/barcode from a cropped image using zxingcpp."""
try:
results = zxingcpp.read_barcodes(cropped)
return results[0].text if results else None
except (RuntimeError, ValueError, TypeError):
logger.exception("Zone OCR: barcode read failed for zone '%s'", zone_name)
return None
def _ocr_text(cropped: Image.Image, zone: OcrTemplateZone, tmp_dir: Path) -> str | None:
"""OCR a cropped image with Tesseract."""
crop_path = tmp_dir / f"zone_{zone.pk}.png"
cropped.save(crop_path)
try:
proc = subprocess.run(
[
"tesseract",
str(crop_path),
"stdout",
"-l",
zone.ocr_language,
"--psm",
"6", # Assume uniform block of text
],
capture_output=True,
text=True,
timeout=30,
check=True,
)
return proc.stdout.strip() or None
except subprocess.TimeoutExpired:
logger.error("Zone OCR: Tesseract timed out for zone '%s'", zone.name)
return None
except subprocess.CalledProcessError as e:
logger.error(
"Zone OCR: Tesseract failed for zone '%s': %s",
zone.name,
e.stderr[:200] if e.stderr else str(e),
)
return None
except FileNotFoundError:
logger.error("Zone OCR: Tesseract not found")
return None
def _extract_zone(
page_img: Path,
zone: OcrTemplateZone,
source_width: int,
source_height: int,
tmp_dir: Path,
) -> ZoneExtraction:
"""Read and transform a zone, retaining the text for the editor's preview."""
cropped = _crop_zone(page_img, zone, source_width, source_height)
if cropped is None:
return ZoneExtraction()
try:
raw_text = (
_read_barcode(cropped, zone.name)
if zone.transform == OcrTemplateZone.TransformType.QR_CODE
else _ocr_text(cropped, zone, tmp_dir)
)
finally:
cropped.close()
return ZoneExtraction(
raw_text=raw_text,
value=_apply_transform(raw_text, zone.transform, zone.date_format)
if raw_text
else None,
)
def extract_zone_preview(
doc_path: Path,
zone: OcrTemplateZone,
source_width: int,
source_height: int,
page_count: int | None,
) -> dict:
"""Extract a zone without writing its value to the document."""
page_idx = _resolve_page_idx(zone.page, page_count)
with tempfile.TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir:
tmp_path = Path(tmp_dir)
page_images = _render_pages(doc_path, {page_idx}, tmp_path, page_count)
if page_idx not in page_images:
return {"raw_text": None, "value": None}
result = _extract_zone(
page_images[page_idx],
zone,
source_width,
source_height,
tmp_path,
)
return {"raw_text": result.raw_text, "value": result.value}
def _parse_iso_date(text: str) -> date | None:
match = re.search(r"\d{4}-\d{2}-\d{2}", text)
return date.fromisoformat(match[0]) if match else None
def _parse_date(text: str, fmt: str = "") -> date | None:
"""Try the zone's date format, ISO dates, then automatic date detection."""
if fmt:
try:
return datetime.strptime(text, fmt).date()
except ValueError:
pass
try:
iso_date = _parse_iso_date(text)
except ValueError:
iso_date = None
if iso_date is not None:
return iso_date
try:
parsed = dateparser.parse(
text,
settings={
"PREFER_DAY_OF_MONTH": "first",
"RETURN_AS_TIMEZONE_AWARE": False,
},
)
if parsed:
return parsed.date()
except (ValueError, OverflowError):
logger.debug("Zone OCR: date parsing failed")
return None
def _apply_transform(text: str, transform: str, date_format: str = "") -> str:
"""Apply post-processing transform to extracted text."""
text = text.strip()
if not text:
return text
if transform in ("strip", "none"):
return text
elif transform == "date":
parsed = _parse_date(text, date_format)
return parsed.isoformat() if parsed else text
elif transform == "uppercase":
return text.upper()
elif transform == "lowercase":
return text.lower()
elif transform == "numeric":
result = re.sub(r"[^\d.,\-]", "", text)
return result if result else text
elif transform == "strip_punctuation":
return text.strip(string.punctuation + " \t\r\n")
return text
def _prepare_field_update(
zone: OcrTemplateZone,
value: str,
) -> FieldUpdate | None:
"""Convert extracted text and enforce the destination field's constraints."""
custom_field = None
if zone.target == OcrTemplateZone.TargetType.CUSTOM_FIELD:
if not zone.custom_field_id:
return None
custom_field = zone.custom_field
field_name = CustomFieldInstance.get_value_field_name(custom_field.data_type)
typed_value = _convert_value(value, custom_field.data_type)
model = CustomFieldInstance
elif zone.target == OcrTemplateZone.TargetType.TITLE:
field_name, typed_value = "title", value[:128]
model = Document
elif zone.target == OcrTemplateZone.TargetType.ASN:
digits = re.sub(r"[^\d]", "", value)
field_name = "archive_serial_number"
typed_value = int(digits) if digits else None
model = Document
elif zone.target == OcrTemplateZone.TargetType.CREATED:
field_name, typed_value = "created", _parse_date(value)
model = Document
else:
return None
if typed_value is None:
return None
try:
typed_value = model._meta.get_field(field_name).clean(typed_value, None)
except ValidationError:
logger.info("Zone OCR: invalid value for zone '%s'", zone.name)
return None
return FieldUpdate(field_name, typed_value, custom_field)
def _apply_field_updates(document: Document, updates: list[FieldUpdate]) -> None:
builtin_values = {
update.field_name: update.value
for update in updates
if update.custom_field is None
}
original_values = {name: getattr(document, name) for name in builtin_values}
try:
with transaction.atomic():
for update in updates:
if update.custom_field is not None:
CustomFieldInstance.objects.update_or_create(
document=document,
field=update.custom_field,
defaults={update.field_name: update.value},
)
if builtin_values:
for name, value in builtin_values.items():
setattr(document, name, value)
document.save(update_fields=list(builtin_values))
except Exception:
# Database rollback does not restore attributes on the model instance.
for name, value in original_values.items():
setattr(document, name, value)
raise
def _normalize_number(value: str) -> str:
cleaned = re.sub(r"[^\d.,\-]", "", value)
# When both separators occur, the rightmost one is the decimal separator.
if "," in cleaned and "." in cleaned:
if cleaned.rindex(",") > cleaned.rindex("."):
cleaned = cleaned.replace(".", "").replace(",", ".")
else:
cleaned = cleaned.replace(",", "")
else:
cleaned = cleaned.replace(",", ".")
Decimal(cleaned)
return cleaned
def _convert_value(value: str, data_type: str) -> object | None:
"""Convert an extracted OCR string to the appropriate type for the custom field."""
if not value:
return None
try:
if data_type in (
CustomField.FieldDataType.STRING,
CustomField.FieldDataType.URL,
):
return value[:128]
elif data_type == CustomField.FieldDataType.LONG_TEXT:
return value
elif data_type == CustomField.FieldDataType.INT:
digits = re.sub(r"[^\d\-]", "", value)
digits = digits.lstrip("-")
if not digits:
return None
if value.strip().startswith("-"):
digits = "-" + digits
return int(digits)
elif data_type == CustomField.FieldDataType.FLOAT:
number = float(_normalize_number(value))
return number if math.isfinite(number) else None
elif data_type == CustomField.FieldDataType.DATE:
parsed = _parse_iso_date(value)
return parsed.isoformat() if parsed else None
elif data_type == CustomField.FieldDataType.MONETARY:
return _normalize_number(value)
elif data_type == CustomField.FieldDataType.BOOL:
lower = value.lower().strip()
if lower in ("true", "yes", "1", "ja", "oui", "si", "x"):
return True
elif lower in ("false", "no", "0", "nein", "non"):
return False
return None
else:
logger.debug(
"Zone OCR: unsupported custom field type %s for OCR extraction",
data_type,
)
return None
except (ValueError, InvalidOperation):
logger.info("Zone OCR: could not convert value to %s", data_type)
return None
def _detect_mime(path: Path) -> str | None:
"""Detect MIME type of a file."""
try:
return magic.from_file(str(path), mime=True)
except (OSError, magic.MagicException):
logger.debug("Zone OCR: magic failed for %s, falling back to extension", path)
suffix = path.suffix.lower()
return {
".pdf": "application/pdf",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".tiff": "image/tiff",
".tif": "image/tiff",
".webp": "image/webp",
".bmp": "image/bmp",
".gif": "image/gif",
}.get(suffix)
+12 -12
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr "" msgstr ""
"Project-Id-Version: paperless-ngx\n" "Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n" "Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-10-31 16:33+0000\n" "POT-Creation-Date: 2026-11-04 19:06+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n" "PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n" "Last-Translator: \n"
"Language-Team: English\n" "Language-Team: English\n"
@@ -1657,8 +1657,8 @@ msgid "workflow runs"
msgstr "" msgstr ""
#: documents/serialisers.py:483 documents/serialisers.py:840 #: documents/serialisers.py:483 documents/serialisers.py:840
#: documents/serialisers.py:2872 documents/views.py:348 documents/views.py:2731 #: documents/serialisers.py:2872 documents/views.py:348 documents/views.py:614
#: paperless_mail/serialisers.py:156 #: documents/views.py:2748 paperless_mail/serialisers.py:156
msgid "Insufficient permissions." msgid "Insufficient permissions."
msgstr "" msgstr ""
@@ -1698,7 +1698,7 @@ msgstr ""
msgid "Duplicate document identifiers are not allowed." msgid "Duplicate document identifiers are not allowed."
msgstr "" msgstr ""
#: documents/serialisers.py:2958 documents/views.py:4781 #: documents/serialisers.py:2958 documents/views.py:4798
#, python-format #, python-format
msgid "Documents not found: %(ids)s" msgid "Documents not found: %(ids)s"
msgstr "" msgstr ""
@@ -1953,36 +1953,36 @@ msgstr ""
msgid "Specify only one of text, title_search, query, or more_like_id." msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr "" msgstr ""
#: documents/views.py:341 documents/views.py:2728 #: documents/views.py:341 documents/views.py:2745
msgid "Invalid more_like_id" msgid "Invalid more_like_id"
msgstr "" msgstr ""
#: documents/views.py:1652 #: documents/views.py:1669
msgid "Invalid AI configuration." msgid "Invalid AI configuration."
msgstr "" msgstr ""
#: documents/views.py:1663 #: documents/views.py:1680
msgid "AI backend request timed out." msgid "AI backend request timed out."
msgstr "" msgstr ""
#: documents/views.py:1675 #: documents/views.py:1692
msgid "AI backend rejected the request. Check logs for details." msgid "AI backend rejected the request. Check logs for details."
msgstr "" msgstr ""
#: documents/views.py:4797 #: documents/views.py:4814
#, python-format #, python-format
msgid "Insufficient permissions to share document %(id)s." msgid "Insufficient permissions to share document %(id)s."
msgstr "" msgstr ""
#: documents/views.py:4844 #: documents/views.py:4861
msgid "Bundle is already being processed." msgid "Bundle is already being processed."
msgstr "" msgstr ""
#: documents/views.py:4910 #: documents/views.py:4927
msgid "The share link bundle is still being prepared. Please try again later." msgid "The share link bundle is still being prepared. Please try again later."
msgstr "" msgstr ""
#: documents/views.py:4924 #: documents/views.py:4941
msgid "The share link bundle is unavailable." msgid "The share link bundle is unavailable."
msgstr "" msgstr ""
+10 -1
View File
@@ -9,6 +9,8 @@ from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
from django.conf import settings from django.conf import settings
from django.contrib.auth.models import Group from django.contrib.auth.models import Group
from django.contrib.auth.models import User from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.db import transaction
from django.forms import ValidationError from django.forms import ValidationError
from django.http import HttpRequest from django.http import HttpRequest
from django.urls import reverse from django.urls import reverse
@@ -111,13 +113,20 @@ class CustomAccountAdapter(DefaultAccountAdapter):
path = path.replace("UID-KEY", quote(key)) path = path.replace("UID-KEY", quote(key))
return settings.PAPERLESS_URL + path return settings.PAPERLESS_URL + path
@transaction.atomic
def save_user(self, request, user, form, commit=True): # noqa: FBT002 def save_user(self, request, user, form, commit=True): # noqa: FBT002
""" """
Save the user instance. Default groups are assigned to the user, if Save the user instance. Default groups are assigned to the user, if
specified in the settings. specified in the settings.
""" """
if ( if commit:
ContentType.objects.select_for_update().get(
app_label=User._meta.app_label,
model=User._meta.model_name,
)
if commit and (
User.objects.exclude(username__in=["consumer", "AnonymousUser"]).count() User.objects.exclude(username__in=["consumer", "AnonymousUser"]).count()
== 0 == 0
and Document.global_objects.count() == 0 and Document.global_objects.count() == 0
+1 -23
View File
@@ -1,7 +1,6 @@
import os import os
from celery import Celery from celery import Celery
from celery.contrib.django.task import DjangoTask
from celery.signals import worker_process_init from celery.signals import worker_process_init
from kombu.serialization import register from kombu.serialization import register
@@ -27,28 +26,7 @@ register(
content_encoding="binary", content_encoding="binary",
) )
app = Celery("paperless")
class FileMoveScopedTask(DjangoTask):
"""
Run every task inside a document file move scope, so each document a task
changes is moved once when the task ends, after its writes have committed.
The scope also flushes when the task raises, for writes that committed
before the failure.
Subclasses DjangoTask because a custom task_cls replaces the Django fixup's
default, and delay_on_commit lives there.
"""
def __call__(self, *args, **kwargs):
# Function-local: this module is imported by paperless/__init__.py
# before Django apps are loaded.
from documents.file_moves import defer_file_moves
with defer_file_moves():
return super().__call__(*args, **kwargs)
app = Celery("paperless", task_cls=FileMoveScopedTask)
# Using a string here means the worker doesn't have to serialize # Using a string here means the worker doesn't have to serialize
# the configuration object to child processes. # the configuration object to child processes.
Loaded 100 of 111 files, more files were not shown because too many files have changed in this diff. Show more