mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-07-30 15:45:58 +00:00
Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 stumpylog
|
dfdc8aa3bb | ||
|
stumpylog
|
a9161e7d84 | ||
 stumpylogandClaude Sonnet 5
|
42208ec70c | ||
 GitHub Actions
|
09c9fa03cf | ||
  shamoonandGitHub
|
04779d72bb | ||
|
Trenton HandGitHub
|
1b32b9d678 | ||
|
Trenton HandGitHub
|
12d318deff | ||
  |
9c98f30d3f | ||
|
shamoonandGitHub
|
6a1d7b1bca | ||
|
shamoonandGitHub
|
de9f520143 | ||
|
shamoonandGitHub
|
a68c487d60 | ||
|
github-actions[bot]andGitHub
|
217421bceb |
@@ -1,5 +1,51 @@
|
||||
# Changelog
|
||||
|
||||
## paperless-ngx 3.0.4
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
|
||||
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
|
||||
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
|
||||
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
|
||||
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
|
||||
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
|
||||
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
|
||||
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
|
||||
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
|
||||
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
|
||||
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
|
||||
|
||||
### Documentation
|
||||
|
||||
- Documentation: fix PAPERLESS\_AI\_LLM\_OUTPUT\_LANGUAGE heading level [@SuperSandro2000](https://github.com/SuperSandro2000) ([#13341](https://github.com/paperless-ngx/paperless-ngx/pull/13341))
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
|
||||
|
||||
### All App Changes
|
||||
|
||||
<details>
|
||||
<summary>14 changes</summary>
|
||||
|
||||
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
|
||||
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
|
||||
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
|
||||
- Performance: prefetch notes and custom fields for LLM index text building [@stumpylog](https://github.com/stumpylog) ([#13350](https://github.com/paperless-ngx/paperless-ngx/pull/13350))
|
||||
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
|
||||
- Performance: Scope llm index updates to actually modified documents [@stumpylog](https://github.com/stumpylog) ([#13322](https://github.com/paperless-ngx/paperless-ngx/pull/13322))
|
||||
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
|
||||
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
|
||||
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
|
||||
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
|
||||
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
|
||||
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
|
||||
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
|
||||
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
|
||||
|
||||
</details>
|
||||
|
||||
## paperless-ngx 3.0.3
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
|
||||
If a "Move to Trash" action is executed in a consume pipeline, the consumption
|
||||
will be aborted and the file will be deleted.
|
||||
|
||||
##### Password Removal {#workflow-action-password-removal}
|
||||
|
||||
"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
|
||||
|
||||
- One or more passwords to try, separated by commas or new lines
|
||||
- Each password is tried in order until one successfully unlocks the document
|
||||
|
||||
Password removal never modifies a file in place. Instead, once a working password is found, the
|
||||
decrypted content is consumed as a new [document version](#document-file-versions), leaving the
|
||||
original (still encrypted) version in the document's version history.
|
||||
|
||||
**Consumption Started**: because this trigger fires before the document exists yet, the password
|
||||
removal itself is deferred until after the initial consumption of the encrypted file has completed.
|
||||
OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
|
||||
extracted text (unless the file already contained extractable text outside of OCR). Immediately
|
||||
afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
|
||||
new version of the same document, this time with normal OCR/text extraction applied. In other words,
|
||||
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
|
||||
the properly processed version is created.
|
||||
|
||||
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
|
||||
already exists, so password removal happens immediately: the decrypted content is queued for
|
||||
consumption as a new version right away. Note that if the document's initial consumption also
|
||||
happened while it was still encrypted, that original version will likewise be missing OCR text.
|
||||
|
||||
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
|
||||
multiple PDF types with different passwords, create separate workflows for each use case.
|
||||
|
||||
#### Workflow placeholders
|
||||
|
||||
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
|
||||
|
||||
+35
-35
@@ -659,7 +659,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">445</context>
|
||||
<context context-type="linenumber">449</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
|
||||
@@ -831,7 +831,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">464</context>
|
||||
<context context-type="linenumber">468</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
||||
@@ -1355,7 +1355,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">390</context>
|
||||
<context context-type="linenumber">394</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||
@@ -1596,7 +1596,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">165</context>
|
||||
<context context-type="linenumber">169</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="2691296884221415710" datatype="html">
|
||||
@@ -1607,7 +1607,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">170</context>
|
||||
<context context-type="linenumber">174</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||
@@ -1638,7 +1638,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">174</context>
|
||||
<context context-type="linenumber">178</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||
@@ -1669,7 +1669,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">178</context>
|
||||
<context context-type="linenumber">182</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||
@@ -4914,7 +4914,7 @@
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">356</context>
|
||||
<context context-type="linenumber">360</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="8057014866157903311" datatype="html">
|
||||
@@ -7760,14 +7760,14 @@
|
||||
<source>Details</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">160</context>
|
||||
<context context-type="linenumber">164</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="5701618810648052610" datatype="html">
|
||||
<source>Title</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">163</context>
|
||||
<context context-type="linenumber">167</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
||||
@@ -7790,14 +7790,14 @@
|
||||
<source>Date created</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">167</context>
|
||||
<context context-type="linenumber">171</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="5607669932062416162" datatype="html">
|
||||
<source>Default</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">179</context>
|
||||
<context context-type="linenumber">183</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
|
||||
@@ -7808,14 +7808,14 @@
|
||||
<source>Content</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">286</context>
|
||||
<context context-type="linenumber">290</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="218403386307979629" datatype="html">
|
||||
<source>Metadata</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">295</context>
|
||||
<context context-type="linenumber">299</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
|
||||
@@ -7826,147 +7826,147 @@
|
||||
<source>Date modified</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">302</context>
|
||||
<context context-type="linenumber">306</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6392918669949841614" datatype="html">
|
||||
<source>Date added</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">306</context>
|
||||
<context context-type="linenumber">310</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="146828917013192897" datatype="html">
|
||||
<source>Media filename</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">310</context>
|
||||
<context context-type="linenumber">314</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="4500855521601039868" datatype="html">
|
||||
<source>Original filename</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">314</context>
|
||||
<context context-type="linenumber">318</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="2659735245739197634" datatype="html">
|
||||
<source>Original SHA256 checksum</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">318</context>
|
||||
<context context-type="linenumber">322</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="5888243105821763422" datatype="html">
|
||||
<source>Original file size</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">322</context>
|
||||
<context context-type="linenumber">326</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="2696647325713149563" datatype="html">
|
||||
<source>Original mime type</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">326</context>
|
||||
<context context-type="linenumber">330</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6714358112223607756" datatype="html">
|
||||
<source>Archive SHA256 checksum</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">331</context>
|
||||
<context context-type="linenumber">335</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6033581412811562084" datatype="html">
|
||||
<source>Archive file size</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">337</context>
|
||||
<context context-type="linenumber">341</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6992781481378431874" datatype="html">
|
||||
<source>Original document metadata</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">346</context>
|
||||
<context context-type="linenumber">350</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="2846565152091361585" datatype="html">
|
||||
<source>Archived document metadata</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">349</context>
|
||||
<context context-type="linenumber">353</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="7206723502037428235" datatype="html">
|
||||
<source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="<span class="badge text-bg-secondary ms-1">"/><x id="INTERPOLATION" equiv-text="length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">368,371</context>
|
||||
<context context-type="linenumber">372,375</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="186236568870281953" datatype="html">
|
||||
<source>History</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">379</context>
|
||||
<context context-type="linenumber">383</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="8236092845697214347" datatype="html">
|
||||
<source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="<span class="badge text-bg-secondary ms-1">"/><x id="INTERPOLATION" equiv-text="cate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/></source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">401,405</context>
|
||||
<context context-type="linenumber">405,409</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6449374629822973702" datatype="html">
|
||||
<source>Duplicate documents detected:</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">407</context>
|
||||
<context context-type="linenumber">411</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="14058600336670816" datatype="html">
|
||||
<source>In trash</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">418</context>
|
||||
<context context-type="linenumber">422</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="5129524307369213584" datatype="html">
|
||||
<source>Save & next</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">447</context>
|
||||
<context context-type="linenumber">451</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="4910102545766233758" datatype="html">
|
||||
<source>Save & close</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">449</context>
|
||||
<context context-type="linenumber">453</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="3823219296477075982" datatype="html">
|
||||
<source>Discard</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">451</context>
|
||||
<context context-type="linenumber">455</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="1309556917227148591" datatype="html">
|
||||
<source>Document loading...</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">459</context>
|
||||
<context context-type="linenumber">463</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="8191371354890763172" datatype="html">
|
||||
<source>Enter Password</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
<context context-type="linenumber">513</context>
|
||||
<context context-type="linenumber">517</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="5758784066858623886" datatype="html">
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
</div>
|
||||
</pngx-page-header>
|
||||
|
||||
@if (!tasksService.completedFileTasks && tasksService.loading) {
|
||||
@if (loading() && pagedTasks().length === 0) {
|
||||
<div class="spinner-border spinner-border-sm fw-normal ms-2 me-auto" role="status"></div>
|
||||
<div class="visually-hidden" i18n>Loading...</div>
|
||||
}
|
||||
|
||||
@@ -113,8 +113,8 @@
|
||||
|
||||
<form [formGroup]='documentForm' (ngSubmit)="save()">
|
||||
|
||||
<div class="btn-toolbar mb-1 border-bottom">
|
||||
<div class="btn-group pb-3">
|
||||
<div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
|
||||
<div class="btn-group me-auto">
|
||||
<button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
|
||||
<i-bs width="1.2em" height="1.2em" name="x"></i-bs>
|
||||
</button>
|
||||
@@ -127,32 +127,36 @@
|
||||
</div>
|
||||
|
||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||
<div class="btn-group pb-3 ms-auto">
|
||||
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
||||
[disabled]="!userCanEdit || suggestionsLoading()"
|
||||
[loading]="suggestionsLoading()"
|
||||
[suggestions]="suggestions()"
|
||||
[aiEnabled]="aiEnabled"
|
||||
(getSuggestions)="getSuggestions()"
|
||||
(addTag)="createTag($event)"
|
||||
(addDocumentType)="createDocumentType($event)"
|
||||
(addCorrespondent)="createCorrespondent($event)">
|
||||
</pngx-suggestions-dropdown>
|
||||
<div class="d-flex gap-2">
|
||||
<div class="btn-group">
|
||||
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
||||
[disabled]="!userCanEdit || suggestionsLoading()"
|
||||
[loading]="suggestionsLoading()"
|
||||
[suggestions]="suggestions()"
|
||||
[aiEnabled]="aiEnabled"
|
||||
(getSuggestions)="getSuggestions()"
|
||||
(addTag)="createTag($event)"
|
||||
(addDocumentType)="createDocumentType($event)"
|
||||
(addCorrespondent)="createCorrespondent($event)">
|
||||
</pngx-suggestions-dropdown>
|
||||
</div>
|
||||
|
||||
<div class="btn-group">
|
||||
<pngx-custom-fields-dropdown
|
||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
||||
[documentId]="documentId()"
|
||||
[disabled]="!userCanEdit"
|
||||
[existingFields]="document()?.custom_fields"
|
||||
(created)="refreshCustomFields()"
|
||||
(added)="addField($event)">
|
||||
</pngx-custom-fields-dropdown>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="btn-group pb-3 ms-2">
|
||||
<pngx-custom-fields-dropdown
|
||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
||||
[documentId]="documentId()"
|
||||
[disabled]="!userCanEdit"
|
||||
[existingFields]="document()?.custom_fields"
|
||||
(created)="refreshCustomFields()"
|
||||
(added)="addField($event)">
|
||||
</pngx-custom-fields-dropdown>
|
||||
<div class="ps-3">
|
||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||
</div>
|
||||
</ng-container>
|
||||
|
||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||
</div>
|
||||
|
||||
<ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
|
||||
@@ -276,7 +280,7 @@
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="d-flex border-top pt-3">
|
||||
<div class="d-flex justify-content-end border-top pt-3">
|
||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||
</div>
|
||||
</ng-template>
|
||||
@@ -440,7 +444,7 @@
|
||||
</div>
|
||||
|
||||
<ng-template #saveButtons>
|
||||
<div class="btn-group pb-3 ms-4">
|
||||
<div class="btn-group">
|
||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
|
||||
@if (hasNext()) {
|
||||
|
||||
@@ -2269,4 +2269,34 @@ describe('FilterEditorComponent', () => {
|
||||
component.itemSelected({ item: 'world', preventDefault: () => true })
|
||||
expect(component.textFilter).toEqual('hello world ')
|
||||
})
|
||||
|
||||
it('should choose the active autocomplete item with Enter', () => {
|
||||
component.textFilterTarget = 'fulltext-query'
|
||||
jest
|
||||
.spyOn(searchService, 'autocomplete')
|
||||
.mockReturnValue(of(['hello', 'help']))
|
||||
|
||||
const input = component.textFilterInput.nativeElement as HTMLInputElement
|
||||
input.value = 'he'
|
||||
input.dispatchEvent(new Event('input'))
|
||||
tick(250)
|
||||
|
||||
input.dispatchEvent(
|
||||
new KeyboardEvent('keydown', {
|
||||
key: 'ArrowDown',
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
})
|
||||
)
|
||||
input.dispatchEvent(
|
||||
new KeyboardEvent('keydown', {
|
||||
key: 'Enter',
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
})
|
||||
)
|
||||
fixture.detectChanges()
|
||||
|
||||
expect(component.textFilter).toEqual('help ')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1314,6 +1314,10 @@ export class FilterEditorComponent
|
||||
|
||||
textFilterKeydown(event: KeyboardEvent) {
|
||||
if (event.key == 'Enter') {
|
||||
if (event.defaultPrevented) {
|
||||
// NgbTypeahead calls preventDefault, so use that to detect if the Enter key was for the dropdown
|
||||
return
|
||||
}
|
||||
const filterString = (
|
||||
this.textFilterInput.nativeElement as HTMLInputElement
|
||||
).value
|
||||
|
||||
@@ -50,13 +50,66 @@ describe('TasksService', () => {
|
||||
req.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('does not call tasks api endpoint on reload if already loading', () => {
|
||||
tasksService.loading = true
|
||||
it('cancels an in-progress reload when reloading again', () => {
|
||||
tasksService.reload()
|
||||
httpTestingController.expectNone(
|
||||
const staleReload = httpTestingController.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
tasksService.reload()
|
||||
|
||||
expect(staleReload.cancelled).toBe(true)
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('continues reloading after a reload request fails', () => {
|
||||
tasksService.reload()
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
.flush('error', { status: 500, statusText: 'error' })
|
||||
|
||||
expect(tasksService.loading).toBe(false)
|
||||
|
||||
tasksService.reload()
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/`
|
||||
)
|
||||
.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('reloads after dismissing a task while a reload is already in progress', () => {
|
||||
tasksService.reload()
|
||||
const staleReload = httpTestingController.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||
req.params.get('acknowledged') === 'false'
|
||||
)
|
||||
|
||||
tasksService.dismissTasks(new Set([1])).subscribe()
|
||||
httpTestingController
|
||||
.expectOne(`${environment.apiBaseUrl}tasks/acknowledge/`)
|
||||
.flush([])
|
||||
|
||||
expect(staleReload.cancelled).toBe(true)
|
||||
httpTestingController
|
||||
.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||
req.params.get('acknowledged') === 'false'
|
||||
)
|
||||
.flush({ count: 0, results: [] })
|
||||
|
||||
expect(tasksService.needsAttentionTasks).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('calls acknowledge_tasks api endpoint on dismiss and reloads', () => {
|
||||
@@ -101,79 +154,6 @@ describe('TasksService', () => {
|
||||
.flush({ count: 0, results: [] })
|
||||
})
|
||||
|
||||
it('groups mixed task types by status when reloading', () => {
|
||||
expect(tasksService.total).toEqual(0)
|
||||
const mockTasks = [
|
||||
{
|
||||
task_type: PaperlessTaskType.ConsumeFile,
|
||||
trigger_source: PaperlessTaskTriggerSource.FolderConsume,
|
||||
status: PaperlessTaskStatus.Success,
|
||||
acknowledged: false,
|
||||
task_id: '1234',
|
||||
input_data: { filename: 'file1.pdf' },
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.SanityCheck,
|
||||
trigger_source: PaperlessTaskTriggerSource.System,
|
||||
status: PaperlessTaskStatus.Failure,
|
||||
acknowledged: false,
|
||||
task_id: '1235',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.MailFetch,
|
||||
trigger_source: PaperlessTaskTriggerSource.Scheduled,
|
||||
status: PaperlessTaskStatus.Pending,
|
||||
acknowledged: false,
|
||||
task_id: '1236',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.LlmIndex,
|
||||
trigger_source: PaperlessTaskTriggerSource.WebUI,
|
||||
status: PaperlessTaskStatus.Started,
|
||||
acknowledged: false,
|
||||
task_id: '1237',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
{
|
||||
task_type: PaperlessTaskType.EmptyTrash,
|
||||
trigger_source: PaperlessTaskTriggerSource.Manual,
|
||||
status: PaperlessTaskStatus.Success,
|
||||
acknowledged: false,
|
||||
task_id: '1238',
|
||||
input_data: {},
|
||||
date_created: new Date(),
|
||||
related_document_ids: [],
|
||||
},
|
||||
]
|
||||
|
||||
tasksService.reload()
|
||||
|
||||
const req = httpTestingController.expectOne(
|
||||
(req: HttpRequest<unknown>) =>
|
||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||
req.params.get('acknowledged') === 'false' &&
|
||||
req.params.get('page_size') === '1000'
|
||||
)
|
||||
|
||||
req.flush({ count: mockTasks.length, results: mockTasks })
|
||||
|
||||
expect(tasksService.allFileTasks).toHaveLength(5)
|
||||
expect(tasksService.completedFileTasks).toHaveLength(2)
|
||||
expect(tasksService.failedFileTasks).toHaveLength(1)
|
||||
expect(tasksService.queuedFileTasks).toHaveLength(1)
|
||||
expect(tasksService.startedFileTasks).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('includes revoked tasks in needs attention', () => {
|
||||
const mockTasks = [
|
||||
{
|
||||
|
||||
@@ -1,7 +1,15 @@
|
||||
import { HttpClient } from '@angular/common/http'
|
||||
import { Injectable, inject, signal } from '@angular/core'
|
||||
import { Observable, Subject } from 'rxjs'
|
||||
import { first, map, takeUntil, tap } from 'rxjs/operators'
|
||||
import { EMPTY, Observable, Subject } from 'rxjs'
|
||||
import {
|
||||
catchError,
|
||||
finalize,
|
||||
first,
|
||||
map,
|
||||
switchMap,
|
||||
takeUntil,
|
||||
tap,
|
||||
} from 'rxjs/operators'
|
||||
import {
|
||||
PaperlessTask,
|
||||
PaperlessTaskStatus,
|
||||
@@ -23,44 +31,40 @@ export class TasksService {
|
||||
|
||||
public loading: boolean = false
|
||||
|
||||
private readonly fileTasks = signal<PaperlessTask[]>([])
|
||||
private readonly tasks = signal<PaperlessTask[]>([])
|
||||
private readonly reloadNotifier = new Subject<void>()
|
||||
|
||||
private unsubscribeNotifer: Subject<any> = new Subject()
|
||||
|
||||
public get total(): number {
|
||||
return this.fileTasks().length
|
||||
}
|
||||
|
||||
public get allFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().slice(0)
|
||||
}
|
||||
|
||||
public get queuedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Pending
|
||||
)
|
||||
}
|
||||
|
||||
public get startedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Started
|
||||
)
|
||||
}
|
||||
|
||||
public get completedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Success
|
||||
)
|
||||
}
|
||||
|
||||
public get failedFileTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter(
|
||||
(t) => t.status === PaperlessTaskStatus.Failure
|
||||
)
|
||||
constructor() {
|
||||
this.reloadNotifier
|
||||
.pipe(
|
||||
switchMap(() => {
|
||||
this.loading = true
|
||||
return this.http
|
||||
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
|
||||
params: {
|
||||
acknowledged: 'false',
|
||||
page_size: this.defaultReloadPageSize,
|
||||
},
|
||||
})
|
||||
.pipe(
|
||||
map((response) => response.results),
|
||||
takeUntil(this.unsubscribeNotifer),
|
||||
catchError(() => EMPTY),
|
||||
finalize(() => {
|
||||
this.loading = false
|
||||
})
|
||||
)
|
||||
})
|
||||
)
|
||||
.subscribe((tasks) => {
|
||||
this.tasks.set(tasks)
|
||||
})
|
||||
}
|
||||
|
||||
public get needsAttentionTasks(): PaperlessTask[] {
|
||||
return this.fileTasks().filter((t) =>
|
||||
return this.tasks().filter((t) =>
|
||||
[PaperlessTaskStatus.Failure, PaperlessTaskStatus.Revoked].includes(
|
||||
t.status
|
||||
)
|
||||
@@ -68,22 +72,7 @@ export class TasksService {
|
||||
}
|
||||
|
||||
public reload() {
|
||||
if (this.loading) return
|
||||
this.loading = true
|
||||
|
||||
this.http
|
||||
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
|
||||
params: {
|
||||
acknowledged: 'false',
|
||||
page_size: this.defaultReloadPageSize,
|
||||
},
|
||||
})
|
||||
.pipe(map((r) => r.results))
|
||||
.pipe(takeUntil(this.unsubscribeNotifer), first())
|
||||
.subscribe((r) => {
|
||||
this.fileTasks.set(r)
|
||||
this.loading = false
|
||||
})
|
||||
this.reloadNotifier.next()
|
||||
}
|
||||
|
||||
public list(
|
||||
|
||||
@@ -39,6 +39,7 @@ from documents.search._tokenizer import ascii_fold
|
||||
from documents.search._tokenizer import autocomplete_tokens
|
||||
from documents.search._tokenizer import register_tokenizers
|
||||
from documents.utils import IterWrapper
|
||||
from documents.utils import QuerySetStream
|
||||
from documents.utils import identity
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -1035,14 +1036,15 @@ _EMPTY_VIEWER_GRANT: Final[ViewerGrant] = ViewerGrant(
|
||||
)
|
||||
|
||||
|
||||
class _DocumentViewerStream:
|
||||
class _DocumentViewerStream(QuerySetStream["Document"]):
|
||||
"""Yield document permission data while batch-loading grants.
|
||||
|
||||
Viewer permissions are fetched in batches (see
|
||||
``_bulk_get_viewer_permissions``), but documents are yielded individually so a
|
||||
progress bar wrapped around this stream advances per document rather than
|
||||
jumping a whole chunk at a time. ``__len__`` lets the progress helper still
|
||||
discover the total (it inspects ``QuerySet``/``Sized``).
|
||||
jumping a whole chunk at a time. ``__len__`` (inherited from
|
||||
``QuerySetStream``) lets the progress helper still discover the total (it
|
||||
inspects ``QuerySet``/``Sized``).
|
||||
|
||||
The viewer and group ids travel with each document in the yielded pair
|
||||
rather than through a separate mutable attribute, so the pairing survives
|
||||
@@ -1051,18 +1053,11 @@ class _DocumentViewerStream:
|
||||
generator in lock-step.
|
||||
"""
|
||||
|
||||
def __init__(self, documents: QuerySet[Document], *, chunk_size: int) -> None:
|
||||
self._documents = documents
|
||||
self._chunk_size = chunk_size
|
||||
|
||||
def __len__(self) -> int:
|
||||
return self._documents.count()
|
||||
|
||||
def __iter__(self) -> Iterator[tuple[Document, ViewerGrant]]:
|
||||
# iterator(chunk_size=…) streams from a server-side cursor instead of
|
||||
# materialising the whole queryset in memory; since Django 4.1 it still
|
||||
# honours prefetch_related, running the prefetches one batch at a time.
|
||||
documents = self._documents.iterator(chunk_size=self._chunk_size)
|
||||
documents = self._queryset.iterator(chunk_size=self._chunk_size)
|
||||
for chunk in chunked(documents, self._chunk_size):
|
||||
grants_by_pk = _bulk_get_viewer_permissions([doc.pk for doc in chunk])
|
||||
for doc in chunk:
|
||||
|
||||
@@ -43,8 +43,16 @@ def _fmt(dt: datetime) -> str:
|
||||
|
||||
|
||||
def _iso_range(lo: datetime, hi: datetime) -> str:
|
||||
"""Format a [lo TO hi] range string in ISO 8601 for Tantivy query syntax."""
|
||||
return f"[{_fmt(lo)} TO {_fmt(hi)}]"
|
||||
"""
|
||||
Format a half-open ``[lo TO hi)`` range in ISO 8601 for Tantivy query syntax.
|
||||
|
||||
``hi`` is always the exclusive ceiling of a computed period (the start of
|
||||
the *next* day/week/month/quarter/year), so the closing bracket must be
|
||||
the Tantivy exclusive-range brace ``}`` rather than ``]`` — otherwise the
|
||||
first instant of the following period (e.g. the 1st of next month) is
|
||||
incorrectly included in the match.
|
||||
"""
|
||||
return f"[{_fmt(lo)} TO {_fmt(hi)}}}"
|
||||
|
||||
|
||||
def _quarter_start(d: date) -> date:
|
||||
|
||||
@@ -566,6 +566,17 @@ def translate_range(field: str, lo: str, hi: str, tz: tzinfo) -> str:
|
||||
lo_pair, hi_pair = hi_pair, lo_pair
|
||||
|
||||
lo_iso = _fmt(lo_pair[0]) if lo_pair is not None else OPEN_LO
|
||||
hi_iso = _fmt(hi_pair[1]) if hi_pair is not None else OPEN_HI
|
||||
|
||||
return f"{field}:[{lo_iso} TO {hi_iso}]"
|
||||
# A bound resolves to (floor, ceil) where floor == ceil for an exact instant
|
||||
# (a full ISO datetime, "now", or a "+/-N unit" offset) and floor != ceil for
|
||||
# a coarser period token (year/month/day precision). Only the latter needs a
|
||||
# half-open close: its ceil is the start of the *next* period and must be
|
||||
# excluded, or that instant (e.g. the 1st of next month) wrongly matches.
|
||||
if hi_pair is not None:
|
||||
hi_iso = _fmt(hi_pair[1])
|
||||
hi_close = "]" if hi_pair[0] == hi_pair[1] else "}"
|
||||
else:
|
||||
hi_iso = OPEN_HI
|
||||
hi_close = "]"
|
||||
|
||||
return f"{field}:[{lo_iso} TO {hi_iso}{hi_close}"
|
||||
|
||||
@@ -1134,6 +1134,11 @@ def before_task_publish_handler(
|
||||
return
|
||||
|
||||
try:
|
||||
# Close stale connections without disrupting a transaction publishing a task
|
||||
for connection in connections.all(initialized_only=True):
|
||||
if not connection.in_atomic_block:
|
||||
connection.close_if_unusable_or_obsolete()
|
||||
|
||||
_, task_kwargs, _ = body
|
||||
task_id = headers["id"]
|
||||
|
||||
|
||||
@@ -32,7 +32,9 @@ AUCKLAND = ZoneInfo("Pacific/Auckland") # UTC+13 in southern-hemisphere summer
|
||||
|
||||
|
||||
def _range(result: str, field: str) -> tuple[str, str]:
|
||||
m = re.search(rf"{field}:\[(.+?) TO (.+?)\]", result)
|
||||
# Half-open period ranges close with "}" (exclusive); exact-instant ranges
|
||||
# (full ISO datetimes, "now", relative offsets) close with "]" (inclusive).
|
||||
m = re.search(rf"{field}:\[(.+?) TO (.+?)[\]}}]", result)
|
||||
assert m, f"No range for {field!r} in: {result!r}"
|
||||
return m.group(1), m.group(2)
|
||||
|
||||
|
||||
@@ -214,27 +214,27 @@ class TestTranslateScalar:
|
||||
(
|
||||
"created",
|
||||
"2020",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||
),
|
||||
(
|
||||
"created",
|
||||
"202003",
|
||||
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
|
||||
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
|
||||
),
|
||||
(
|
||||
"created",
|
||||
"20200115",
|
||||
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
|
||||
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
|
||||
),
|
||||
(
|
||||
"created",
|
||||
"2020-01-15",
|
||||
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
|
||||
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
|
||||
),
|
||||
(
|
||||
"created",
|
||||
"2020-03",
|
||||
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
|
||||
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
|
||||
),
|
||||
],
|
||||
)
|
||||
@@ -248,9 +248,9 @@ class TestTranslateScalar:
|
||||
assert exc_info.value.value == "202023"
|
||||
|
||||
def test_keyword_delegates(self) -> None:
|
||||
# keyword path produces a range; just assert it is a created range
|
||||
# keyword path produces a half-open range; just assert it is a created range
|
||||
out = translate_scalar("created", "today", UTC)
|
||||
assert out.startswith("created:[") and out.endswith("]")
|
||||
assert out.startswith("created:[") and out.endswith("}")
|
||||
|
||||
def test_14digit_compact_datetime(self) -> None:
|
||||
out = translate_scalar("created", "20240115120000", UTC)
|
||||
@@ -279,21 +279,21 @@ class TestTranslateRange:
|
||||
@pytest.mark.parametrize(
|
||||
("lo", "hi", "expected"),
|
||||
[
|
||||
("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"),
|
||||
("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"),
|
||||
(
|
||||
"202001",
|
||||
"202006",
|
||||
"created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z]",
|
||||
"created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z}",
|
||||
),
|
||||
(
|
||||
"20200101",
|
||||
"20201231",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||
),
|
||||
(
|
||||
"2020-01-01",
|
||||
"2020-12-31",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||
),
|
||||
],
|
||||
)
|
||||
@@ -302,7 +302,7 @@ class TestTranslateRange:
|
||||
|
||||
def test_reversed_swaps(self):
|
||||
assert translate_range("created", "2009", "2005", UTC) == (
|
||||
"created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"
|
||||
"created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"
|
||||
)
|
||||
|
||||
def test_open_upper(self):
|
||||
@@ -311,7 +311,7 @@ class TestTranslateRange:
|
||||
|
||||
def test_open_lower(self):
|
||||
out = translate_range("created", "", "2020", UTC)
|
||||
assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z]"
|
||||
assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z}}"
|
||||
|
||||
def test_invalid_bound_raises(self):
|
||||
with pytest.raises(InvalidDateQuery) as exc_info:
|
||||
@@ -334,16 +334,16 @@ class TestTranslateQuery:
|
||||
[
|
||||
(
|
||||
"created:2020",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||
),
|
||||
("tag:foo,bar", "tag:foo AND tag:bar"),
|
||||
# 'type' is a user-facing alias rewritten to 'document_type' (the real schema field)
|
||||
("tag:foo,type:bar", "tag:foo AND document_type:bar"),
|
||||
(
|
||||
"created:[2020 TO 2021],added:[2022 TO 2023]",
|
||||
"created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
|
||||
"created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
|
||||
" AND "
|
||||
"added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z]",
|
||||
"added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z}",
|
||||
),
|
||||
# correspondent is not multi-value: comma stays literal inside the value
|
||||
("correspondent:foo,bar", "correspondent:foo,bar"),
|
||||
@@ -506,7 +506,7 @@ class TestOperatorNormalization:
|
||||
def test_date_range_preserved(self) -> None:
|
||||
out = translate_query("created:[2020 TO 2021]", UTC)
|
||||
# Must not corrupt the ISO range
|
||||
assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
|
||||
assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
|
||||
|
||||
def test_date_scalar_with_or(self) -> None:
|
||||
out = translate_query("created:2020 OR foo", UTC)
|
||||
@@ -581,42 +581,42 @@ class TestKeywordDateResolution:
|
||||
[
|
||||
pytest.param(
|
||||
"today",
|
||||
"created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z]",
|
||||
"created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z}",
|
||||
id="today",
|
||||
),
|
||||
pytest.param(
|
||||
"yesterday",
|
||||
"created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z]",
|
||||
"created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z}",
|
||||
id="yesterday",
|
||||
),
|
||||
pytest.param(
|
||||
"previous week",
|
||||
"created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z]",
|
||||
"created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z}",
|
||||
id="previous-week",
|
||||
),
|
||||
pytest.param(
|
||||
"this month",
|
||||
"created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z]",
|
||||
"created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z}",
|
||||
id="this-month",
|
||||
),
|
||||
pytest.param(
|
||||
"previous month",
|
||||
"created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z]",
|
||||
"created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z}",
|
||||
id="previous-month",
|
||||
),
|
||||
pytest.param(
|
||||
"this year",
|
||||
"created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z]",
|
||||
"created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z}",
|
||||
id="this-year",
|
||||
),
|
||||
pytest.param(
|
||||
"previous year",
|
||||
"created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
|
||||
"created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
|
||||
id="previous-year",
|
||||
),
|
||||
pytest.param(
|
||||
"previous quarter",
|
||||
"created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
|
||||
"created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
|
||||
id="previous-quarter",
|
||||
),
|
||||
],
|
||||
@@ -637,42 +637,42 @@ class TestKeywordDateResolution:
|
||||
[
|
||||
pytest.param(
|
||||
"today",
|
||||
"added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z]",
|
||||
"added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z}",
|
||||
id="today",
|
||||
),
|
||||
pytest.param(
|
||||
"yesterday",
|
||||
"added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z]",
|
||||
"added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z}",
|
||||
id="yesterday",
|
||||
),
|
||||
pytest.param(
|
||||
"previous week",
|
||||
"added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z]",
|
||||
"added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z}",
|
||||
id="previous-week",
|
||||
),
|
||||
pytest.param(
|
||||
"this month",
|
||||
"added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z]",
|
||||
"added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z}",
|
||||
id="this-month",
|
||||
),
|
||||
pytest.param(
|
||||
"previous month",
|
||||
"added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z]",
|
||||
"added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z}",
|
||||
id="previous-month",
|
||||
),
|
||||
pytest.param(
|
||||
"this year",
|
||||
"added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z]",
|
||||
"added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z}",
|
||||
id="this-year",
|
||||
),
|
||||
pytest.param(
|
||||
"previous year",
|
||||
"added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z]",
|
||||
"added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z}",
|
||||
id="previous-year",
|
||||
),
|
||||
pytest.param(
|
||||
"previous quarter",
|
||||
"added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z]",
|
||||
"added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z}",
|
||||
id="previous-quarter",
|
||||
),
|
||||
],
|
||||
@@ -719,7 +719,7 @@ class TestISODatetimeBounds:
|
||||
def test_translate_query_text_before_comma_separated_date_clause(self) -> None:
|
||||
result = translate_query("schäfersee,created:previous year", UTC)
|
||||
assert result == (
|
||||
"schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]"
|
||||
"schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}"
|
||||
)
|
||||
|
||||
def test_invalid_iso_datetime_raises(self) -> None:
|
||||
|
||||
@@ -720,6 +720,48 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
|
||||
self.assertEqual(results[0]["id"], 3)
|
||||
self.assertEqual(results[0]["title"], "bank statement 3")
|
||||
|
||||
def test_search_added_previous_month_excludes_next_period_start(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- One document added at the last instant of last month
|
||||
- One document added exactly at the first instant of this month
|
||||
WHEN:
|
||||
- Query for documents added in the previous month
|
||||
THEN:
|
||||
- Only the document from last month is returned; the document dated
|
||||
exactly at the start of this month (the exclusive upper bound of
|
||||
the range) is not
|
||||
"""
|
||||
d1 = DocumentFactory.create(
|
||||
title="end of last month",
|
||||
content="last instant of last month",
|
||||
checksum="A",
|
||||
pk=1,
|
||||
added=timezone.make_aware(datetime.datetime(2024, 1, 31, 23, 59, 59)),
|
||||
)
|
||||
d2 = DocumentFactory.create(
|
||||
title="start of this month",
|
||||
content="first instant of this month",
|
||||
checksum="B",
|
||||
pk=2,
|
||||
added=timezone.make_aware(datetime.datetime(2024, 2, 1, 0, 0, 0)),
|
||||
)
|
||||
|
||||
backend = get_backend()
|
||||
backend.add_or_update(d1)
|
||||
backend.add_or_update(d2)
|
||||
|
||||
with time_machine.travel(
|
||||
timezone.make_aware(datetime.datetime(2024, 2, 15, 12, 0, 0)),
|
||||
tick=False,
|
||||
):
|
||||
response = self.client.get("/api/documents/?query=added:previous month")
|
||||
results = response.data["results"]
|
||||
|
||||
self.assertEqual(len(results), 1)
|
||||
self.assertEqual(results[0]["id"], 1)
|
||||
self.assertEqual(results[0]["title"], "end of last month")
|
||||
|
||||
def test_search_added_invalid_date(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -56,6 +56,32 @@ def send_publish(
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestBeforeTaskPublishHandler:
|
||||
@mock.patch("documents.signals.handlers.connections.all")
|
||||
def test_closes_old_connections_outside_atomic_blocks(
|
||||
self,
|
||||
connections_all,
|
||||
) -> None:
|
||||
connection = mock.Mock(in_atomic_block=False)
|
||||
connections_all.return_value = [connection]
|
||||
|
||||
task_id = send_publish("documents.tasks.train_classifier", (), {})
|
||||
|
||||
connection.close_if_unusable_or_obsolete.assert_called_once_with()
|
||||
assert PaperlessTask.objects.filter(task_id=task_id).exists()
|
||||
|
||||
@mock.patch("documents.signals.handlers.connections.all")
|
||||
def test_keeps_connections_open_inside_atomic_blocks(
|
||||
self,
|
||||
connections_all,
|
||||
) -> None:
|
||||
connection = mock.Mock(in_atomic_block=True)
|
||||
connections_all.return_value = [connection]
|
||||
|
||||
task_id = send_publish("documents.tasks.train_classifier", (), {})
|
||||
|
||||
connection.close_if_unusable_or_obsolete.assert_not_called()
|
||||
assert PaperlessTask.objects.filter(task_id=task_id).exists()
|
||||
|
||||
def test_creates_task_for_consume_file(
|
||||
self,
|
||||
consume_input_doc,
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
import pytest_mock
|
||||
|
||||
from documents.utils import QuerySetStream
|
||||
|
||||
|
||||
class TestQuerySetStream:
|
||||
def test_len_and_iter_delegate_to_streaming_queryset_methods(
|
||||
self,
|
||||
mocker: pytest_mock.MockerFixture,
|
||||
) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- A mock queryset
|
||||
WHEN:
|
||||
- A QuerySetStream wrapping it is measured and iterated
|
||||
THEN:
|
||||
- len() uses count() (not a materializing len()), and iteration
|
||||
uses .iterator(chunk_size=...) (not plain iteration, which
|
||||
would materialize the whole queryset, plus any prefetch
|
||||
caches, into Django's own result cache at once)
|
||||
"""
|
||||
mock_queryset = mocker.MagicMock()
|
||||
mock_queryset.count.return_value = 42
|
||||
mock_queryset.iterator.return_value = iter(["row-1", "row-2"])
|
||||
streamed = QuerySetStream(mock_queryset, chunk_size=1000)
|
||||
|
||||
assert len(streamed) == 42
|
||||
assert list(streamed) == ["row-1", "row-2"]
|
||||
# count.call_count isn't asserted exactly: list()'s own size-hint
|
||||
# optimization calls len(streamed) again internally, on top of the
|
||||
# explicit len() call above -- both legitimately delegate to
|
||||
# count(), so only the delegation itself (not the call count) is
|
||||
# the thing being verified here.
|
||||
mock_queryset.count.assert_called_with()
|
||||
mock_queryset.iterator.assert_called_once_with(chunk_size=1000)
|
||||
@@ -3,16 +3,24 @@ import logging
|
||||
import shutil
|
||||
from collections.abc import Callable
|
||||
from collections.abc import Iterable
|
||||
from collections.abc import Iterator
|
||||
from os import utime
|
||||
from pathlib import Path
|
||||
from subprocess import CompletedProcess
|
||||
from subprocess import run
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import Generic
|
||||
from typing import TypeVar
|
||||
|
||||
from django.conf import settings
|
||||
from PIL import Image
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from django.db.models import Model
|
||||
from django.db.models import QuerySet
|
||||
|
||||
_T = TypeVar("_T")
|
||||
_M = TypeVar("_M", bound="Model")
|
||||
|
||||
# A function that wraps an iterable — typically used to inject a progress bar.
|
||||
IterWrapper = Callable[[Iterable[_T]], Iterable[_T]]
|
||||
@@ -23,6 +31,40 @@ def identity(iterable: Iterable[_T]) -> Iterable[_T]:
|
||||
return iterable
|
||||
|
||||
|
||||
class QuerySetStream(Generic[_M]):
|
||||
"""Stream a QuerySet via .iterator(chunk_size=...) instead of
|
||||
materializing it (plus any prefetch caches) all at once, while still
|
||||
supporting len() via count() so a progress bar wrapped around this
|
||||
(e.g. via IterWrapper) shows a real total instead of falling back to
|
||||
indeterminate.
|
||||
|
||||
Plain QuerySet iteration (``for row in queryset:``) is not lazy: Django
|
||||
fetches every matching row in one query and caches the fully-hydrated
|
||||
result in the queryset's own ``_result_cache`` before yielding the
|
||||
first item -- wrapping that in a progress bar or any other iterable
|
||||
adapter doesn't change this, since none of them alter how the
|
||||
underlying queryset produces items. ``.iterator(chunk_size=...)`` is
|
||||
the specific Django API that bypasses ``_result_cache`` and streams
|
||||
from a server-side cursor instead, discarding each chunk once consumed
|
||||
(and, since Django 4.1, still honours ``prefetch_related``, running the
|
||||
prefetches one batch at a time rather than for the whole queryset).
|
||||
|
||||
Subclass to layer additional per-batch work on top (see
|
||||
``documents.search._backend._DocumentViewerStream``) by overriding
|
||||
``__iter__`` -- ``__len__`` and the constructor are inherited for free.
|
||||
"""
|
||||
|
||||
def __init__(self, queryset: "QuerySet[_M]", *, chunk_size: int) -> None:
|
||||
self._queryset = queryset
|
||||
self._chunk_size = chunk_size
|
||||
|
||||
def __len__(self) -> int:
|
||||
return self._queryset.count()
|
||||
|
||||
def __iter__(self) -> Iterator[_M]:
|
||||
return iter(self._queryset.iterator(chunk_size=self._chunk_size))
|
||||
|
||||
|
||||
def _coerce_to_path(
|
||||
source: Path | str,
|
||||
dest: Path | str,
|
||||
|
||||
@@ -14,6 +14,7 @@ from filelock import Timeout
|
||||
from documents.models import Document
|
||||
from documents.models import PaperlessTask
|
||||
from documents.utils import IterWrapper
|
||||
from documents.utils import QuerySetStream
|
||||
from documents.utils import identity
|
||||
from paperless.config import AIConfig
|
||||
from paperless_ai.db import db_connection_released
|
||||
@@ -32,6 +33,11 @@ logger = logging.getLogger("paperless_ai.indexing")
|
||||
RAG_NUM_OUTPUT = 512
|
||||
RAG_CHUNK_OVERLAP = 200
|
||||
|
||||
# update_llm_index(): row count per .iterator() batch when streaming
|
||||
# documents for a rebuild/update via QuerySetStream, matching
|
||||
# _DocumentViewerStream's chunk size in documents/search/_backend.py.
|
||||
_INDEX_STREAM_CHUNK_SIZE = 1000
|
||||
|
||||
|
||||
def queue_llm_index_update_if_needed(*, rebuild: bool, reason: str) -> bool:
|
||||
# NOTE: The check-then-enqueue sequence below is non-atomic (TOCTOU): two
|
||||
@@ -385,7 +391,9 @@ def update_llm_index(
|
||||
if rebuild or not store.table_exists():
|
||||
logger.info("Rebuilding LLM index.")
|
||||
store.drop_table()
|
||||
for document in iter_wrapper(documents):
|
||||
for document in iter_wrapper(
|
||||
QuerySetStream(documents, chunk_size=_INDEX_STREAM_CHUNK_SIZE),
|
||||
):
|
||||
nodes = build_document_node(document, chunk_size=chunk_size)
|
||||
_embed_nodes(nodes, embed_model)
|
||||
store.add(nodes)
|
||||
@@ -398,7 +406,9 @@ def update_llm_index(
|
||||
)
|
||||
existing = store.get_modified_times()
|
||||
changed = 0
|
||||
for document in iter_wrapper(scoped_documents):
|
||||
for document in iter_wrapper(
|
||||
QuerySetStream(scoped_documents, chunk_size=_INDEX_STREAM_CHUNK_SIZE),
|
||||
):
|
||||
doc_id = str(document.id)
|
||||
if existing.get(doc_id) == document.modified.isoformat():
|
||||
continue
|
||||
|
||||
@@ -3735,15 +3735,15 @@ crypto = [
|
||||
|
||||
[[package]]
|
||||
name = "pymdown-extensions"
|
||||
version = "10.21.3"
|
||||
version = "11.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "markdown", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||
{ name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/47/67/f1e79672a5f91985577c7984c9709ca110e4fd37fe7fd167b60422e6ccc2/pymdown_extensions-11.0.tar.gz", hash = "sha256:8269cef0247f9e2d0a62fcea10860aba05c1cbab5470fd4b63230b96434dc589", size = 857049, upload-time = "2026-06-23T02:27:45.146Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/af/b6/1ae53367e28b9cffa3be7574e13fbe4589694272fd47710fbdbafd3d63c6/pymdown_extensions-11.0-py3-none-any.whl", hash = "sha256:fbc4acb641814fa9d17521bbd21a5240ef739a662f11c06330c4b78c93e954d6", size = 269415, upload-time = "2026-06-23T02:27:43.826Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user