Compare commits

..
Author SHA1 Message Date
stumpylogandClaude Sonnet 5 7d9ba34582 Fix: address code review feedback on table gateways
- Document sqlite3.Row row_factory precondition in module docstring
- Strengthen test_create_is_idempotent to verify populated table survives
- Collapse three roundtrip tests into one @pytest.mark.parametrize

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-29 13:51:32 -07:00
stumpylogandClaude Sonnet 5 91c3d9caab Feature: add table-gateway module for document_chunks/document_meta/index_meta
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-29 13:42:39 -07:00
shamoonandGitHub 668fa77428 Fix: prevent scale vs page loop in pngx PDF viewer (#13406) 2026-07-29 09:25:37 -07:00
shamoonandGitHub 5bd72014a6 Fix: handle hidden line breaks in email subjects when sending email (#13402) 2026-07-29 14:35:30 +00:00
Trenton HandGitHub bbb9c86ba4 Fix: avoid NotSupportedError from document_importer on MariaDB (#13400) 2026-07-29 07:14:13 -07:00
GitHub Actions 09c9fa03cf Auto translate strings 2026-07-28 18:28:12 +00:00
shamoonandGitHub 04779d72bb Tweak: adjust doc details button toolbar flow (#13382) 2026-07-28 11:26:37 -07:00
Trenton HandGitHub 1b32b9d678 Fix: exclude next-period start from relative date-range filters (#13381)
* Fix: exclude next-period start from relative date-range filters

Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper
bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of
the next period. Use half-open [lo TO hi} for those so e.g. "previous month"
no longer matches the 1st of the current month.

* Adds a regression test down to the second check for the hi range
2026-07-28 16:58:38 +00:00
Trenton HandGitHub 12d318deff Documentation: Add Password Removal workflow action documentation (#13377)
Addresses discussion #13373. Documents that password removal creates a
new document version via re-consumption of the decrypted file rather
than editing in place, and explains why the Consumption Started trigger
produces an initial un-OCR'd version followed by a properly processed
one.
2026-07-28 09:21:14 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9c98f30d3f Chore(deps): Bump pymdown-extensions in the uv group across 1 directory (#13378)
Bumps the uv group with 1 update in the / directory: [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions).


Updates `pymdown-extensions` from 10.21.3 to 11.0
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](https://github.com/facelessuser/pymdown-extensions/compare/10.21.3...11.0)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-version: '11.0'
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 15:29:20 +00:00
shamoonandGitHub 6a1d7b1bca Fix: close non-atomic db connections in before_task_publish (#13366) 2026-07-28 07:21:02 -07:00
shamoonandGitHub de9f520143 Chore/fix: refactor frontend task service (#13365) 2026-07-28 00:34:30 -07:00
shamoonandGitHub a68c487d60 Fix: fix Enter selection in search autocomplete (#13361) 2026-07-27 23:33:13 -07:00
github-actions[bot]andGitHub 217421bceb Documentation: Add v3.0.4 changelog (#13356) 2026-07-27 21:21:10 -07:00
24 changed files with 975 additions and 238 deletions
+46
View File
@@ -1,5 +1,51 @@
# Changelog
## paperless-ngx 3.0.4
### Bug Fixes
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
### Documentation
- Documentation: fix PAPERLESS\_AI\_LLM\_OUTPUT\_LANGUAGE heading level [@SuperSandro2000](https://github.com/SuperSandro2000) ([#13341](https://github.com/paperless-ngx/paperless-ngx/pull/13341))
### Dependencies
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
### All App Changes
<details>
<summary>14 changes</summary>
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
- Performance: prefetch notes and custom fields for LLM index text building [@stumpylog](https://github.com/stumpylog) ([#13350](https://github.com/paperless-ngx/paperless-ngx/pull/13350))
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
- Performance: Scope llm index updates to actually modified documents [@stumpylog](https://github.com/stumpylog) ([#13322](https://github.com/paperless-ngx/paperless-ngx/pull/13322))
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
</details>
## paperless-ngx 3.0.3
### Bug Fixes
+28
View File
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
If a "Move to Trash" action is executed in a consume pipeline, the consumption
will be aborted and the file will be deleted.
##### Password Removal {#workflow-action-password-removal}
"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
- One or more passwords to try, separated by commas or new lines
- Each password is tried in order until one successfully unlocks the document
Password removal never modifies a file in place. Instead, once a working password is found, the
decrypted content is consumed as a new [document version](#document-file-versions), leaving the
original (still encrypted) version in the document's version history.
**Consumption Started**: because this trigger fires before the document exists yet, the password
removal itself is deferred until after the initial consumption of the encrypted file has completed.
OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
extracted text (unless the file already contained extractable text outside of OCR). Immediately
afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
new version of the same document, this time with normal OCR/text extraction applied. In other words,
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
the properly processed version is created.
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
already exists, so password removal happens immediately: the decrypted content is queued for
consumption as a new version right away. Note that if the document's initial consumption also
happened while it was still encrypted, that original version will likewise be missing OCR text.
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
multiple PDF types with different passwords, create separate workflows for each use case.
#### Workflow placeholders
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
+35 -35
View File
@@ -659,7 +659,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">445</context>
<context context-type="linenumber">449</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
@@ -831,7 +831,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">464</context>
<context context-type="linenumber">468</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -1355,7 +1355,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">390</context>
<context context-type="linenumber">394</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1596,7 +1596,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">165</context>
<context context-type="linenumber">169</context>
</context-group>
</trans-unit>
<trans-unit id="2691296884221415710" datatype="html">
@@ -1607,7 +1607,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">170</context>
<context context-type="linenumber">174</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1638,7 +1638,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">174</context>
<context context-type="linenumber">178</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -1669,7 +1669,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">178</context>
<context context-type="linenumber">182</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
@@ -4914,7 +4914,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">356</context>
<context context-type="linenumber">360</context>
</context-group>
</trans-unit>
<trans-unit id="8057014866157903311" datatype="html">
@@ -7760,14 +7760,14 @@
<source>Details</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">160</context>
<context context-type="linenumber">164</context>
</context-group>
</trans-unit>
<trans-unit id="5701618810648052610" datatype="html">
<source>Title</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">163</context>
<context context-type="linenumber">167</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
@@ -7790,14 +7790,14 @@
<source>Date created</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">167</context>
<context context-type="linenumber">171</context>
</context-group>
</trans-unit>
<trans-unit id="5607669932062416162" datatype="html">
<source>Default</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">179</context>
<context context-type="linenumber">183</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
@@ -7808,14 +7808,14 @@
<source>Content</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">286</context>
<context context-type="linenumber">290</context>
</context-group>
</trans-unit>
<trans-unit id="218403386307979629" datatype="html">
<source>Metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">295</context>
<context context-type="linenumber">299</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
@@ -7826,147 +7826,147 @@
<source>Date modified</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">302</context>
<context context-type="linenumber">306</context>
</context-group>
</trans-unit>
<trans-unit id="6392918669949841614" datatype="html">
<source>Date added</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">306</context>
<context context-type="linenumber">310</context>
</context-group>
</trans-unit>
<trans-unit id="146828917013192897" datatype="html">
<source>Media filename</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">310</context>
<context context-type="linenumber">314</context>
</context-group>
</trans-unit>
<trans-unit id="4500855521601039868" datatype="html">
<source>Original filename</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">314</context>
<context context-type="linenumber">318</context>
</context-group>
</trans-unit>
<trans-unit id="2659735245739197634" datatype="html">
<source>Original SHA256 checksum</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">318</context>
<context context-type="linenumber">322</context>
</context-group>
</trans-unit>
<trans-unit id="5888243105821763422" datatype="html">
<source>Original file size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">322</context>
<context context-type="linenumber">326</context>
</context-group>
</trans-unit>
<trans-unit id="2696647325713149563" datatype="html">
<source>Original mime type</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">326</context>
<context context-type="linenumber">330</context>
</context-group>
</trans-unit>
<trans-unit id="6714358112223607756" datatype="html">
<source>Archive SHA256 checksum</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">331</context>
<context context-type="linenumber">335</context>
</context-group>
</trans-unit>
<trans-unit id="6033581412811562084" datatype="html">
<source>Archive file size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">337</context>
<context context-type="linenumber">341</context>
</context-group>
</trans-unit>
<trans-unit id="6992781481378431874" datatype="html">
<source>Original document metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">346</context>
<context context-type="linenumber">350</context>
</context-group>
</trans-unit>
<trans-unit id="2846565152091361585" datatype="html">
<source>Archived document metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">349</context>
<context context-type="linenumber">353</context>
</context-group>
</trans-unit>
<trans-unit id="7206723502037428235" datatype="html">
<source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">368,371</context>
<context context-type="linenumber">372,375</context>
</context-group>
</trans-unit>
<trans-unit id="186236568870281953" datatype="html">
<source>History</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">379</context>
<context context-type="linenumber">383</context>
</context-group>
</trans-unit>
<trans-unit id="8236092845697214347" datatype="html">
<source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="&lt;span class=&quot;badge text-bg-secondary ms-1&quot;&gt;"/><x id="INTERPOLATION" equiv-text="cate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">401,405</context>
<context context-type="linenumber">405,409</context>
</context-group>
</trans-unit>
<trans-unit id="6449374629822973702" datatype="html">
<source>Duplicate documents detected:</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">407</context>
<context context-type="linenumber">411</context>
</context-group>
</trans-unit>
<trans-unit id="14058600336670816" datatype="html">
<source>In trash</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">418</context>
<context context-type="linenumber">422</context>
</context-group>
</trans-unit>
<trans-unit id="5129524307369213584" datatype="html">
<source>Save &amp; next</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">447</context>
<context context-type="linenumber">451</context>
</context-group>
</trans-unit>
<trans-unit id="4910102545766233758" datatype="html">
<source>Save &amp; close</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">449</context>
<context context-type="linenumber">453</context>
</context-group>
</trans-unit>
<trans-unit id="3823219296477075982" datatype="html">
<source>Discard</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">451</context>
<context context-type="linenumber">455</context>
</context-group>
</trans-unit>
<trans-unit id="1309556917227148591" datatype="html">
<source>Document loading...</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">459</context>
<context context-type="linenumber">463</context>
</context-group>
</trans-unit>
<trans-unit id="8191371354890763172" datatype="html">
<source>Enter Password</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">513</context>
<context context-type="linenumber">517</context>
</context-group>
</trans-unit>
<trans-unit id="5758784066858623886" datatype="html">
@@ -21,7 +21,7 @@
</div>
</pngx-page-header>
@if (!tasksService.completedFileTasks && tasksService.loading) {
@if (loading() && pagedTasks().length === 0) {
<div class="spinner-border spinner-border-sm fw-normal ms-2 me-auto" role="status"></div>
<div class="visually-hidden" i18n>Loading...</div>
}
@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
;(component as any).applyScale()
expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
expect(viewer.currentScale).toBe(2)
})
it('does not reapply scale for page-only changes', async () => {
await initComponent()
const pdf = (component as any).pdf as { numPages: number }
pdf.numPages = 3
const viewer = (component as any).pdfViewer as PDFViewer
viewer.setDocument(pdf)
const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
component.page = 2
;(component as any).lastViewerPage = 2
;(component as any).applyViewerState()
component.ngOnChanges({
page: new SimpleChange(1, 2, false),
})
expect(viewer.currentPageNumber).toBe(2)
expect((component as any).lastViewerPage).toBeUndefined()
expect(applyScaleSpy).toHaveBeenCalled()
expect(applyScaleSpy).not.toHaveBeenCalled()
})
it('does not reset the viewer when it is already on the requested page', async () => {
@@ -116,7 +116,10 @@ export class PngxPdfViewerComponent
changes['zoomScale'] ||
changes['rotation']
) {
this.applyViewerState()
// Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
this.applyViewerState(
!!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
)
}
if (changes['searchQuery']) {
@@ -240,7 +243,7 @@ export class PngxPdfViewerComponent
}
}
private applyViewerState(): void {
private applyViewerState(applyScale = true): void {
if (!this.pdfViewer) {
return
}
@@ -264,7 +267,7 @@ export class PngxPdfViewerComponent
if (this.page === this.lastViewerPage) {
this.lastViewerPage = undefined
}
if (hasPages) {
if (hasPages && applyScale) {
this.applyScale()
}
this.dispatchFindIfReady()
@@ -113,8 +113,8 @@
<form [formGroup]='documentForm' (ngSubmit)="save()">
<div class="btn-toolbar mb-1 border-bottom">
<div class="btn-group pb-3">
<div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
<div class="btn-group me-auto">
<button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
<i-bs width="1.2em" height="1.2em" name="x"></i-bs>
</button>
@@ -127,32 +127,36 @@
</div>
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
<div class="btn-group pb-3 ms-auto">
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
[disabled]="!userCanEdit || suggestionsLoading()"
[loading]="suggestionsLoading()"
[suggestions]="suggestions()"
[aiEnabled]="aiEnabled"
(getSuggestions)="getSuggestions()"
(addTag)="createTag($event)"
(addDocumentType)="createDocumentType($event)"
(addCorrespondent)="createCorrespondent($event)">
</pngx-suggestions-dropdown>
<div class="d-flex gap-2">
<div class="btn-group">
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
[disabled]="!userCanEdit || suggestionsLoading()"
[loading]="suggestionsLoading()"
[suggestions]="suggestions()"
[aiEnabled]="aiEnabled"
(getSuggestions)="getSuggestions()"
(addTag)="createTag($event)"
(addDocumentType)="createDocumentType($event)"
(addCorrespondent)="createCorrespondent($event)">
</pngx-suggestions-dropdown>
</div>
<div class="btn-group">
<pngx-custom-fields-dropdown
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
[documentId]="documentId()"
[disabled]="!userCanEdit"
[existingFields]="document()?.custom_fields"
(created)="refreshCustomFields()"
(added)="addField($event)">
</pngx-custom-fields-dropdown>
</div>
</div>
<div class="btn-group pb-3 ms-2">
<pngx-custom-fields-dropdown
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
[documentId]="documentId()"
[disabled]="!userCanEdit"
[existingFields]="document()?.custom_fields"
(created)="refreshCustomFields()"
(added)="addField($event)">
</pngx-custom-fields-dropdown>
<div class="ps-3">
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
</div>
</ng-container>
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
</div>
<ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
@@ -276,7 +280,7 @@
}
</div>
<div class="d-flex border-top pt-3">
<div class="d-flex justify-content-end border-top pt-3">
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
</div>
</ng-template>
@@ -440,7 +444,7 @@
</div>
<ng-template #saveButtons>
<div class="btn-group pb-3 ms-4">
<div class="btn-group">
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
@if (hasNext()) {
@@ -2269,4 +2269,34 @@ describe('FilterEditorComponent', () => {
component.itemSelected({ item: 'world', preventDefault: () => true })
expect(component.textFilter).toEqual('hello world ')
})
it('should choose the active autocomplete item with Enter', () => {
component.textFilterTarget = 'fulltext-query'
jest
.spyOn(searchService, 'autocomplete')
.mockReturnValue(of(['hello', 'help']))
const input = component.textFilterInput.nativeElement as HTMLInputElement
input.value = 'he'
input.dispatchEvent(new Event('input'))
tick(250)
input.dispatchEvent(
new KeyboardEvent('keydown', {
key: 'ArrowDown',
bubbles: true,
cancelable: true,
})
)
input.dispatchEvent(
new KeyboardEvent('keydown', {
key: 'Enter',
bubbles: true,
cancelable: true,
})
)
fixture.detectChanges()
expect(component.textFilter).toEqual('help ')
})
})
@@ -1314,6 +1314,10 @@ export class FilterEditorComponent
textFilterKeydown(event: KeyboardEvent) {
if (event.key == 'Enter') {
if (event.defaultPrevented) {
// NgbTypeahead calls preventDefault, so use that to detect if the Enter key was for the dropdown
return
}
const filterString = (
this.textFilterInput.nativeElement as HTMLInputElement
).value
+56 -76
View File
@@ -50,13 +50,66 @@ describe('TasksService', () => {
req.flush({ count: 0, results: [] })
})
it('does not call tasks api endpoint on reload if already loading', () => {
tasksService.loading = true
it('cancels an in-progress reload when reloading again', () => {
tasksService.reload()
httpTestingController.expectNone(
const staleReload = httpTestingController.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
tasksService.reload()
expect(staleReload.cancelled).toBe(true)
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
.flush({ count: 0, results: [] })
})
it('continues reloading after a reload request fails', () => {
tasksService.reload()
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
.flush('error', { status: 500, statusText: 'error' })
expect(tasksService.loading).toBe(false)
tasksService.reload()
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
.flush({ count: 0, results: [] })
})
it('reloads after dismissing a task while a reload is already in progress', () => {
tasksService.reload()
const staleReload = httpTestingController.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('acknowledged') === 'false'
)
tasksService.dismissTasks(new Set([1])).subscribe()
httpTestingController
.expectOne(`${environment.apiBaseUrl}tasks/acknowledge/`)
.flush([])
expect(staleReload.cancelled).toBe(true)
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('acknowledged') === 'false'
)
.flush({ count: 0, results: [] })
expect(tasksService.needsAttentionTasks).toHaveLength(0)
})
it('calls acknowledge_tasks api endpoint on dismiss and reloads', () => {
@@ -101,79 +154,6 @@ describe('TasksService', () => {
.flush({ count: 0, results: [] })
})
it('groups mixed task types by status when reloading', () => {
expect(tasksService.total).toEqual(0)
const mockTasks = [
{
task_type: PaperlessTaskType.ConsumeFile,
trigger_source: PaperlessTaskTriggerSource.FolderConsume,
status: PaperlessTaskStatus.Success,
acknowledged: false,
task_id: '1234',
input_data: { filename: 'file1.pdf' },
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.SanityCheck,
trigger_source: PaperlessTaskTriggerSource.System,
status: PaperlessTaskStatus.Failure,
acknowledged: false,
task_id: '1235',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.MailFetch,
trigger_source: PaperlessTaskTriggerSource.Scheduled,
status: PaperlessTaskStatus.Pending,
acknowledged: false,
task_id: '1236',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.LlmIndex,
trigger_source: PaperlessTaskTriggerSource.WebUI,
status: PaperlessTaskStatus.Started,
acknowledged: false,
task_id: '1237',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.EmptyTrash,
trigger_source: PaperlessTaskTriggerSource.Manual,
status: PaperlessTaskStatus.Success,
acknowledged: false,
task_id: '1238',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
]
tasksService.reload()
const req = httpTestingController.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('acknowledged') === 'false' &&
req.params.get('page_size') === '1000'
)
req.flush({ count: mockTasks.length, results: mockTasks })
expect(tasksService.allFileTasks).toHaveLength(5)
expect(tasksService.completedFileTasks).toHaveLength(2)
expect(tasksService.failedFileTasks).toHaveLength(1)
expect(tasksService.queuedFileTasks).toHaveLength(1)
expect(tasksService.startedFileTasks).toHaveLength(1)
})
it('includes revoked tasks in needs attention', () => {
const mockTasks = [
{
+39 -50
View File
@@ -1,7 +1,15 @@
import { HttpClient } from '@angular/common/http'
import { Injectable, inject, signal } from '@angular/core'
import { Observable, Subject } from 'rxjs'
import { first, map, takeUntil, tap } from 'rxjs/operators'
import { EMPTY, Observable, Subject } from 'rxjs'
import {
catchError,
finalize,
first,
map,
switchMap,
takeUntil,
tap,
} from 'rxjs/operators'
import {
PaperlessTask,
PaperlessTaskStatus,
@@ -23,44 +31,40 @@ export class TasksService {
public loading: boolean = false
private readonly fileTasks = signal<PaperlessTask[]>([])
private readonly tasks = signal<PaperlessTask[]>([])
private readonly reloadNotifier = new Subject<void>()
private unsubscribeNotifer: Subject<any> = new Subject()
public get total(): number {
return this.fileTasks().length
}
public get allFileTasks(): PaperlessTask[] {
return this.fileTasks().slice(0)
}
public get queuedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Pending
)
}
public get startedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Started
)
}
public get completedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Success
)
}
public get failedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Failure
)
constructor() {
this.reloadNotifier
.pipe(
switchMap(() => {
this.loading = true
return this.http
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
params: {
acknowledged: 'false',
page_size: this.defaultReloadPageSize,
},
})
.pipe(
map((response) => response.results),
takeUntil(this.unsubscribeNotifer),
catchError(() => EMPTY),
finalize(() => {
this.loading = false
})
)
})
)
.subscribe((tasks) => {
this.tasks.set(tasks)
})
}
public get needsAttentionTasks(): PaperlessTask[] {
return this.fileTasks().filter((t) =>
return this.tasks().filter((t) =>
[PaperlessTaskStatus.Failure, PaperlessTaskStatus.Revoked].includes(
t.status
)
@@ -68,22 +72,7 @@ export class TasksService {
}
public reload() {
if (this.loading) return
this.loading = true
this.http
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
params: {
acknowledged: 'false',
page_size: this.defaultReloadPageSize,
},
})
.pipe(map((r) => r.results))
.pipe(takeUntil(this.unsubscribeNotifer), first())
.subscribe((r) => {
this.fileTasks.set(r)
this.loading = false
})
this.reloadNotifier.next()
}
public list(
+3
View File
@@ -36,6 +36,9 @@ def send_email(
TODO: re-evaluate this pending https://code.djangoproject.com/ticket/35581 / https://github.com/django/django/pull/18966
"""
if "\r" in subject or "\n" in subject:
subject = " ".join(line.strip(" \t") for line in subject.splitlines())
email = EmailMessage(
subject=subject,
body=body,
@@ -386,10 +386,19 @@ class Command(CryptMixin, PaperlessCommand):
raise DeserializationError(
f"{model.__name__} has no updatable fields; PK-only models are not supported by the importer",
)
# MySQL/MariaDB support upserts via ON DUPLICATE KEY UPDATE but,
# unlike PostgreSQL/SQLite, cannot target a specific unique field
# for the conflict -- passing unique_fields there raises
# NotSupportedError.
unique_fields = (
[model._meta.pk.attname]
if connection.features.supports_update_conflicts_with_target
else None
)
model.objects.bulk_create( # type: ignore[attr-defined]
instances,
update_conflicts=True,
unique_fields=[model._meta.pk.attname],
unique_fields=unique_fields,
update_fields=update_fields,
)
loaded_models.add(model)
+10 -2
View File
@@ -43,8 +43,16 @@ def _fmt(dt: datetime) -> str:
def _iso_range(lo: datetime, hi: datetime) -> str:
"""Format a [lo TO hi] range string in ISO 8601 for Tantivy query syntax."""
return f"[{_fmt(lo)} TO {_fmt(hi)}]"
"""
Format a half-open ``[lo TO hi)`` range in ISO 8601 for Tantivy query syntax.
``hi`` is always the exclusive ceiling of a computed period (the start of
the *next* day/week/month/quarter/year), so the closing bracket must be
the Tantivy exclusive-range brace ``}`` rather than ``]`` — otherwise the
first instant of the following period (e.g. the 1st of next month) is
incorrectly included in the match.
"""
return f"[{_fmt(lo)} TO {_fmt(hi)}}}"
def _quarter_start(d: date) -> date:
+13 -2
View File
@@ -566,6 +566,17 @@ def translate_range(field: str, lo: str, hi: str, tz: tzinfo) -> str:
lo_pair, hi_pair = hi_pair, lo_pair
lo_iso = _fmt(lo_pair[0]) if lo_pair is not None else OPEN_LO
hi_iso = _fmt(hi_pair[1]) if hi_pair is not None else OPEN_HI
return f"{field}:[{lo_iso} TO {hi_iso}]"
# A bound resolves to (floor, ceil) where floor == ceil for an exact instant
# (a full ISO datetime, "now", or a "+/-N unit" offset) and floor != ceil for
# a coarser period token (year/month/day precision). Only the latter needs a
# half-open close: its ceil is the start of the *next* period and must be
# excluded, or that instant (e.g. the 1st of next month) wrongly matches.
if hi_pair is not None:
hi_iso = _fmt(hi_pair[1])
hi_close = "]" if hi_pair[0] == hi_pair[1] else "}"
else:
hi_iso = OPEN_HI
hi_close = "]"
return f"{field}:[{lo_iso} TO {hi_iso}{hi_close}"
+5
View File
@@ -1134,6 +1134,11 @@ def before_task_publish_handler(
return
try:
# Close stale connections without disrupting a transaction publishing a task
for connection in connections.all(initialized_only=True):
if not connection.in_atomic_block:
connection.close_if_unusable_or_obsolete()
_, task_kwargs, _ = body
task_id = headers["id"]
+3 -1
View File
@@ -32,7 +32,9 @@ AUCKLAND = ZoneInfo("Pacific/Auckland") # UTC+13 in southern-hemisphere summer
def _range(result: str, field: str) -> tuple[str, str]:
m = re.search(rf"{field}:\[(.+?) TO (.+?)\]", result)
# Half-open period ranges close with "}" (exclusive); exact-instant ranges
# (full ISO datetimes, "now", relative offsets) close with "]" (inclusive).
m = re.search(rf"{field}:\[(.+?) TO (.+?)[\]}}]", result)
assert m, f"No range for {field!r} in: {result!r}"
return m.group(1), m.group(2)
+34 -34
View File
@@ -214,27 +214,27 @@ class TestTranslateScalar:
(
"created",
"2020",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
),
(
"created",
"202003",
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
),
(
"created",
"20200115",
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
),
(
"created",
"2020-01-15",
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
),
(
"created",
"2020-03",
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
),
],
)
@@ -248,9 +248,9 @@ class TestTranslateScalar:
assert exc_info.value.value == "202023"
def test_keyword_delegates(self) -> None:
# keyword path produces a range; just assert it is a created range
# keyword path produces a half-open range; just assert it is a created range
out = translate_scalar("created", "today", UTC)
assert out.startswith("created:[") and out.endswith("]")
assert out.startswith("created:[") and out.endswith("}")
def test_14digit_compact_datetime(self) -> None:
out = translate_scalar("created", "20240115120000", UTC)
@@ -279,21 +279,21 @@ class TestTranslateRange:
@pytest.mark.parametrize(
("lo", "hi", "expected"),
[
("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"),
("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"),
(
"202001",
"202006",
"created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z]",
"created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z}",
),
(
"20200101",
"20201231",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
),
(
"2020-01-01",
"2020-12-31",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
),
],
)
@@ -302,7 +302,7 @@ class TestTranslateRange:
def test_reversed_swaps(self):
assert translate_range("created", "2009", "2005", UTC) == (
"created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"
"created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"
)
def test_open_upper(self):
@@ -311,7 +311,7 @@ class TestTranslateRange:
def test_open_lower(self):
out = translate_range("created", "", "2020", UTC)
assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z]"
assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z}}"
def test_invalid_bound_raises(self):
with pytest.raises(InvalidDateQuery) as exc_info:
@@ -334,16 +334,16 @@ class TestTranslateQuery:
[
(
"created:2020",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
),
("tag:foo,bar", "tag:foo AND tag:bar"),
# 'type' is a user-facing alias rewritten to 'document_type' (the real schema field)
("tag:foo,type:bar", "tag:foo AND document_type:bar"),
(
"created:[2020 TO 2021],added:[2022 TO 2023]",
"created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
"created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
" AND "
"added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z]",
"added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z}",
),
# correspondent is not multi-value: comma stays literal inside the value
("correspondent:foo,bar", "correspondent:foo,bar"),
@@ -506,7 +506,7 @@ class TestOperatorNormalization:
def test_date_range_preserved(self) -> None:
out = translate_query("created:[2020 TO 2021]", UTC)
# Must not corrupt the ISO range
assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
def test_date_scalar_with_or(self) -> None:
out = translate_query("created:2020 OR foo", UTC)
@@ -581,42 +581,42 @@ class TestKeywordDateResolution:
[
pytest.param(
"today",
"created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z]",
"created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z}",
id="today",
),
pytest.param(
"yesterday",
"created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z]",
"created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z}",
id="yesterday",
),
pytest.param(
"previous week",
"created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z]",
"created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z}",
id="previous-week",
),
pytest.param(
"this month",
"created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z]",
"created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z}",
id="this-month",
),
pytest.param(
"previous month",
"created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z]",
"created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z}",
id="previous-month",
),
pytest.param(
"this year",
"created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z]",
"created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z}",
id="this-year",
),
pytest.param(
"previous year",
"created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
"created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
id="previous-year",
),
pytest.param(
"previous quarter",
"created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
"created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
id="previous-quarter",
),
],
@@ -637,42 +637,42 @@ class TestKeywordDateResolution:
[
pytest.param(
"today",
"added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z]",
"added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z}",
id="today",
),
pytest.param(
"yesterday",
"added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z]",
"added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z}",
id="yesterday",
),
pytest.param(
"previous week",
"added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z]",
"added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z}",
id="previous-week",
),
pytest.param(
"this month",
"added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z]",
"added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z}",
id="this-month",
),
pytest.param(
"previous month",
"added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z]",
"added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z}",
id="previous-month",
),
pytest.param(
"this year",
"added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z]",
"added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z}",
id="this-year",
),
pytest.param(
"previous year",
"added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z]",
"added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z}",
id="previous-year",
),
pytest.param(
"previous quarter",
"added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z]",
"added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z}",
id="previous-quarter",
),
],
@@ -719,7 +719,7 @@ class TestISODatetimeBounds:
def test_translate_query_text_before_comma_separated_date_clause(self) -> None:
result = translate_query("schäfersee,created:previous year", UTC)
assert result == (
"schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]"
"schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}"
)
def test_invalid_iso_datetime_raises(self) -> None:
+1 -1
View File
@@ -75,7 +75,7 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
{
"documents": [self.doc1.pk, self.doc2.pk],
"addresses": "hello@paperless-ngx.com,test@example.com",
"subject": "Bulk email test",
"subject": "Bulk email\n test",
"message": "Here are your documents",
},
),
+42
View File
@@ -720,6 +720,48 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
self.assertEqual(results[0]["id"], 3)
self.assertEqual(results[0]["title"], "bank statement 3")
def test_search_added_previous_month_excludes_next_period_start(self) -> None:
"""
GIVEN:
- One document added at the last instant of last month
- One document added exactly at the first instant of this month
WHEN:
- Query for documents added in the previous month
THEN:
- Only the document from last month is returned; the document dated
exactly at the start of this month (the exclusive upper bound of
the range) is not
"""
d1 = DocumentFactory.create(
title="end of last month",
content="last instant of last month",
checksum="A",
pk=1,
added=timezone.make_aware(datetime.datetime(2024, 1, 31, 23, 59, 59)),
)
d2 = DocumentFactory.create(
title="start of this month",
content="first instant of this month",
checksum="B",
pk=2,
added=timezone.make_aware(datetime.datetime(2024, 2, 1, 0, 0, 0)),
)
backend = get_backend()
backend.add_or_update(d1)
backend.add_or_update(d2)
with time_machine.travel(
timezone.make_aware(datetime.datetime(2024, 2, 15, 12, 0, 0)),
tick=False,
):
response = self.client.get("/api/documents/?query=added:previous month")
results = response.data["results"]
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], 1)
self.assertEqual(results[0]["title"], "end of last month")
def test_search_added_invalid_date(self) -> None:
"""
GIVEN:
+26
View File
@@ -56,6 +56,32 @@ def send_publish(
@pytest.mark.django_db
class TestBeforeTaskPublishHandler:
@mock.patch("documents.signals.handlers.connections.all")
def test_closes_old_connections_outside_atomic_blocks(
self,
connections_all,
) -> None:
connection = mock.Mock(in_atomic_block=False)
connections_all.return_value = [connection]
task_id = send_publish("documents.tasks.train_classifier", (), {})
connection.close_if_unusable_or_obsolete.assert_called_once_with()
assert PaperlessTask.objects.filter(task_id=task_id).exists()
@mock.patch("documents.signals.handlers.connections.all")
def test_keeps_connections_open_inside_atomic_blocks(
self,
connections_all,
) -> None:
connection = mock.Mock(in_atomic_block=True)
connections_all.return_value = [connection]
task_id = send_publish("documents.tasks.train_classifier", (), {})
connection.close_if_unusable_or_obsolete.assert_not_called()
assert PaperlessTask.objects.filter(task_id=task_id).exists()
def test_creates_task_for_consume_file(
self,
consume_input_doc,
+229
View File
@@ -0,0 +1,229 @@
"""Thin gateways over the plain relational side tables that sit alongside the
vec0 table. Each method takes the sqlite3.Connection to operate on
explicitly, rather than owning one -- the store swaps connections during
compact()/migration, and migrations always work across two connections
(src_conn, dst_conn) at once.
PRECONDITION: Callers must set conn.row_factory = sqlite3.Row before passing a
connection to any of these gateways' read methods. The read methods across all
three classes (DocumentChunksTable.chunk_ids_for_document, IndexMetaTable._get,
DocumentMetaTable.all_modified_times, DocumentMetaTable.copy_all) use
row["column_name"] dictionary-style indexing, which requires sqlite3.Row as the
row factory -- without it, sqlite3.Row is not set, rows are returned as plain
tuples, and tuple indices must be integers, raising TypeError.
"""
import sqlite3
from collections.abc import Iterable
from typing import NamedTuple
class ChunkRow(NamedTuple):
chunk_id: str
document_id: int
class DocumentMetaRow(NamedTuple):
document_id: int
modified: str
class DocumentChunksTable:
"""chunk_id -> document_id, indexed by document_id. Gives O(1)
per-document chunk lookup that vec0's own document_id metadata column
cannot (see PaperlessSqliteVecVectorStore._delete_chunks_by_document_id).
"""
@staticmethod
def create(conn: sqlite3.Connection) -> None:
conn.execute(
"CREATE TABLE IF NOT EXISTS document_chunks "
"(chunk_id TEXT PRIMARY KEY, document_id INTEGER NOT NULL)",
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_document_chunks_document_id "
"ON document_chunks (document_id)",
)
@staticmethod
def insert_many(conn: sqlite3.Connection, rows: Iterable[ChunkRow]) -> None:
"""rows must already be batch-bounded by the caller (e.g. vec0's own
fetchmany() loop) -- this never reads, so it can't itself introduce
an unbounded scan, but a whole-table iterable defeats the point."""
conn.executemany(
"INSERT INTO document_chunks (chunk_id, document_id) VALUES (?, ?)",
rows,
)
@staticmethod
def chunk_ids_for_document(
conn: sqlite3.Connection,
document_id: int,
) -> list[str]:
return [
row["chunk_id"]
for row in conn.execute(
"SELECT chunk_id FROM document_chunks WHERE document_id = ?",
(document_id,),
).fetchall()
]
@staticmethod
def delete_for_document(conn: sqlite3.Connection, document_id: int) -> None:
conn.execute(
"DELETE FROM document_chunks WHERE document_id = ?",
(document_id,),
)
@staticmethod
def delete_all(conn: sqlite3.Connection) -> None:
conn.execute("DELETE FROM document_chunks")
@staticmethod
def count(conn: sqlite3.Connection) -> int:
"""Cheap stand-in for vec0's own row count -- see compact()."""
return conn.execute("SELECT count(*) FROM document_chunks").fetchone()[0]
class DocumentMetaTable:
"""document_id -> modified, one row per document. Lives outside vec0
because vec0 only inlines TEXT metadata up to 12 bytes and `modified`
(an ISO timestamp) is always longer.
"""
@staticmethod
def create(conn: sqlite3.Connection) -> None:
conn.execute(
"CREATE TABLE IF NOT EXISTS document_meta "
"(document_id INTEGER PRIMARY KEY, modified TEXT NOT NULL)",
)
@staticmethod
def upsert_many(
conn: sqlite3.Connection,
rows: Iterable[DocumentMetaRow],
) -> None:
conn.executemany(
"INSERT INTO document_meta (document_id, modified) VALUES (?, ?) "
"ON CONFLICT(document_id) DO UPDATE SET modified = excluded.modified",
rows,
)
@staticmethod
def delete_for_document(conn: sqlite3.Connection, document_id: int) -> None:
conn.execute(
"DELETE FROM document_meta WHERE document_id = ?",
(document_id,),
)
@staticmethod
def delete_all(conn: sqlite3.Connection) -> None:
conn.execute("DELETE FROM document_meta")
@staticmethod
def copy_all(
src_conn: sqlite3.Connection,
dst_conn: sqlite3.Connection,
batch_size: int,
) -> None:
"""Stream document_meta from src_conn into dst_conn in bounded
batches. The *only* sanctioned way to move this table across
connections (compact()/migrations) -- an unbounded fetchall here
would defeat the same OOM-avoidance the vec0 row copy already relies
on. batch_size has no default: forces the call site to think about
it (pass COMPACT_BATCH_SIZE)."""
cursor = src_conn.execute(
"SELECT document_id, modified FROM document_meta",
)
while batch := cursor.fetchmany(batch_size):
DocumentMetaTable.upsert_many(
dst_conn,
(DocumentMetaRow(r["document_id"], r["modified"]) for r in batch),
)
@staticmethod
def all_modified_times(conn: sqlite3.Connection) -> dict[str, str]:
"""Full document_id -> modified map, for get_modified_times()'s
public API only. One unbounded read by design (existing behavior).
Never use this for cross-connection copying; see copy_all()."""
return {
str(row["document_id"]): str(row["modified"] or "")
for row in conn.execute(
"SELECT document_id, modified FROM document_meta",
)
}
class IndexMetaTable:
"""Typed accessors over index_meta's key/value rows -- replaces
PaperlessSqliteVecVectorStore._meta_get_on/_meta_set_on, which returned
untyped str | None regardless of whether the key held an int (dim,
schema_version, total_inserts) or a string (embed_model).
"""
@staticmethod
def create(conn: sqlite3.Connection) -> None:
conn.execute(
"CREATE TABLE IF NOT EXISTS index_meta (key TEXT PRIMARY KEY, value TEXT)",
)
@staticmethod
def _get(conn: sqlite3.Connection, key: str) -> str | None:
row = conn.execute(
"SELECT value FROM index_meta WHERE key = ?",
(key,),
).fetchone()
return row["value"] if row else None
@staticmethod
def _set(conn: sqlite3.Connection, key: str, value: str) -> None:
conn.execute(
"INSERT INTO index_meta (key, value) VALUES (?, ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
(key, value),
)
@staticmethod
def get_dim(conn: sqlite3.Connection) -> int | None:
value = IndexMetaTable._get(conn, "dim")
return int(value) if value is not None else None
@staticmethod
def set_dim(conn: sqlite3.Connection, dim: int) -> None:
IndexMetaTable._set(conn, "dim", str(dim))
@staticmethod
def get_embed_model(conn: sqlite3.Connection) -> str | None:
return IndexMetaTable._get(conn, "embed_model")
@staticmethod
def set_embed_model(conn: sqlite3.Connection, name: str) -> None:
IndexMetaTable._set(conn, "embed_model", name)
@staticmethod
def get_schema_version(conn: sqlite3.Connection) -> int | None:
value = IndexMetaTable._get(conn, "schema_version")
return int(value) if value is not None else None
@staticmethod
def set_schema_version(conn: sqlite3.Connection, version: int) -> None:
IndexMetaTable._set(conn, "schema_version", str(version))
@staticmethod
def get_total_inserts(conn: sqlite3.Connection) -> int:
value = IndexMetaTable._get(conn, "total_inserts")
return int(value) if value is not None else 0
@staticmethod
def increment_total_inserts(conn: sqlite3.Connection, count: int) -> None:
current = IndexMetaTable.get_total_inserts(conn)
IndexMetaTable._set(conn, "total_inserts", str(current + count))
@staticmethod
def reset_total_inserts(conn: sqlite3.Connection, count: int) -> None:
"""Set total_inserts to an absolute value -- distinct from
increment_total_inserts(): used by compact()'s rebuild and by
m0001_v1_to_v2 after copying live rows into a fresh file, where
total_inserts must become exactly the live row count, not add to
whatever the source file's counter held."""
IndexMetaTable._set(conn, "total_inserts", str(count))
+306
View File
@@ -0,0 +1,306 @@
import sqlite3
from collections.abc import Generator
import pytest
from paperless_ai.tables import ChunkRow
from paperless_ai.tables import DocumentChunksTable
from paperless_ai.tables import DocumentMetaRow
from paperless_ai.tables import DocumentMetaTable
from paperless_ai.tables import IndexMetaTable
@pytest.fixture
def conn() -> Generator[sqlite3.Connection, None, None]:
connection = sqlite3.connect(":memory:")
connection.row_factory = sqlite3.Row
try:
yield connection
finally:
connection.close()
class TestDocumentChunksTable:
def test_create_is_idempotent(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- A bare sqlite3 connection
WHEN:
- create() is called, a row is inserted, then create() is called again
THEN:
- No error is raised and the row survives uncorrupted
"""
DocumentChunksTable.create(conn)
DocumentChunksTable.insert_many(conn, [ChunkRow("c1", 1)])
DocumentChunksTable.create(conn)
assert DocumentChunksTable.chunk_ids_for_document(conn, 1) == ["c1"]
def test_insert_many_then_lookup_by_document_id(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- An empty document_chunks table
WHEN:
- Two chunks for document 1 and one for document 2 are inserted
THEN:
- chunk_ids_for_document returns exactly the matching chunk ids
"""
DocumentChunksTable.create(conn)
DocumentChunksTable.insert_many(
conn,
[ChunkRow("c1", 1), ChunkRow("c2", 1), ChunkRow("c3", 2)],
)
assert sorted(DocumentChunksTable.chunk_ids_for_document(conn, 1)) == [
"c1",
"c2",
]
assert DocumentChunksTable.chunk_ids_for_document(conn, 2) == ["c3"]
assert DocumentChunksTable.chunk_ids_for_document(conn, 999) == []
def test_delete_for_document_removes_only_that_document(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- Chunks for two different documents
WHEN:
- delete_for_document() is called for one of them
THEN:
- Only that document's chunks are removed
"""
DocumentChunksTable.create(conn)
DocumentChunksTable.insert_many(
conn,
[ChunkRow("c1", 1), ChunkRow("c2", 2)],
)
DocumentChunksTable.delete_for_document(conn, 1)
assert DocumentChunksTable.chunk_ids_for_document(conn, 1) == []
assert DocumentChunksTable.chunk_ids_for_document(conn, 2) == ["c2"]
def test_delete_all_clears_every_row(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- Chunks for multiple documents
WHEN:
- delete_all() is called
THEN:
- count() returns 0
"""
DocumentChunksTable.create(conn)
DocumentChunksTable.insert_many(
conn,
[ChunkRow("c1", 1), ChunkRow("c2", 2)],
)
DocumentChunksTable.delete_all(conn)
assert DocumentChunksTable.count(conn) == 0
def test_count_reflects_live_rows(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- An empty document_chunks table
WHEN:
- Rows are inserted then one document's rows are deleted
THEN:
- count() reflects the remaining row count
"""
DocumentChunksTable.create(conn)
DocumentChunksTable.insert_many(
conn,
[ChunkRow("c1", 1), ChunkRow("c2", 1), ChunkRow("c3", 2)],
)
assert DocumentChunksTable.count(conn) == 3
DocumentChunksTable.delete_for_document(conn, 1)
assert DocumentChunksTable.count(conn) == 1
class TestDocumentMetaTable:
def test_upsert_many_then_all_modified_times(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- An empty document_meta table
WHEN:
- Two documents' modified timestamps are upserted
THEN:
- all_modified_times() returns both, keyed by str(document_id)
"""
DocumentMetaTable.create(conn)
DocumentMetaTable.upsert_many(
conn,
[
DocumentMetaRow(1, "2026-01-01T00:00:00"),
DocumentMetaRow(2, "2026-02-02T00:00:00"),
],
)
assert DocumentMetaTable.all_modified_times(conn) == {
"1": "2026-01-01T00:00:00",
"2": "2026-02-02T00:00:00",
}
def test_upsert_many_overwrites_existing_value(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- A document_meta row for document 1
WHEN:
- upsert_many() is called again with a new modified value for
the same document_id
THEN:
- The stored value is replaced, not duplicated
"""
DocumentMetaTable.create(conn)
DocumentMetaTable.upsert_many(conn, [DocumentMetaRow(1, "old")])
DocumentMetaTable.upsert_many(conn, [DocumentMetaRow(1, "new")])
assert DocumentMetaTable.all_modified_times(conn) == {"1": "new"}
def test_delete_for_document_removes_only_that_row(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- document_meta rows for two documents
WHEN:
- delete_for_document() is called for one of them
THEN:
- Only that document's row is removed
"""
DocumentMetaTable.create(conn)
DocumentMetaTable.upsert_many(
conn,
[DocumentMetaRow(1, "a"), DocumentMetaRow(2, "b")],
)
DocumentMetaTable.delete_for_document(conn, 1)
assert DocumentMetaTable.all_modified_times(conn) == {"2": "b"}
def test_delete_all_clears_every_row(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- document_meta rows for multiple documents
WHEN:
- delete_all() is called
THEN:
- all_modified_times() returns an empty dict
"""
DocumentMetaTable.create(conn)
DocumentMetaTable.upsert_many(
conn,
[DocumentMetaRow(1, "a"), DocumentMetaRow(2, "b")],
)
DocumentMetaTable.delete_all(conn)
assert DocumentMetaTable.all_modified_times(conn) == {}
def test_copy_all_streams_every_row_to_destination(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- A source connection with document_meta rows for 5 documents
- A separate, empty destination connection
WHEN:
- copy_all() is called with a batch size smaller than the row
count, forcing multiple fetchmany() cycles
THEN:
- Every row is present on the destination connection
"""
DocumentMetaTable.create(conn)
DocumentMetaTable.upsert_many(
conn,
[DocumentMetaRow(i, f"modified-{i}") for i in range(5)],
)
dst_conn = sqlite3.connect(":memory:")
dst_conn.row_factory = sqlite3.Row
try:
DocumentMetaTable.create(dst_conn)
DocumentMetaTable.copy_all(conn, dst_conn, batch_size=2)
assert DocumentMetaTable.all_modified_times(dst_conn) == {
str(i): f"modified-{i}" for i in range(5)
}
finally:
dst_conn.close()
class TestIndexMetaTable:
@pytest.mark.parametrize(
("setter_name", "getter_name", "value"),
[
("set_dim", "get_dim", 384),
("set_embed_model", "get_embed_model", "model-a"),
("set_schema_version", "get_schema_version", 2),
],
)
def test_typed_accessor_roundtrip(
self,
conn: sqlite3.Connection,
setter_name: str,
getter_name: str,
value: int | str,
) -> None:
"""
GIVEN:
- An empty index_meta table
WHEN:
- A typed accessor's setter is called then the getter is read back
THEN:
- The same value is returned, correctly typed (int or str)
"""
IndexMetaTable.create(conn)
getter = getattr(IndexMetaTable, getter_name)
setter = getattr(IndexMetaTable, setter_name)
assert getter(conn) is None
setter(conn, value)
assert getter(conn) == value
def test_total_inserts_starts_at_zero(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- An empty index_meta table
WHEN:
- get_total_inserts() is read before anything is set
THEN:
- 0 is returned
"""
IndexMetaTable.create(conn)
assert IndexMetaTable.get_total_inserts(conn) == 0
def test_increment_total_inserts_accumulates(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- An empty index_meta table
WHEN:
- increment_total_inserts() is called twice
THEN:
- get_total_inserts() returns the running sum
"""
IndexMetaTable.create(conn)
IndexMetaTable.increment_total_inserts(conn, 5)
IndexMetaTable.increment_total_inserts(conn, 3)
assert IndexMetaTable.get_total_inserts(conn) == 8
def test_reset_total_inserts_sets_absolute_value(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- A total_inserts counter already at a high value
WHEN:
- reset_total_inserts() is called with a lower value
THEN:
- get_total_inserts() returns exactly that value, not a sum
"""
IndexMetaTable.create(conn)
IndexMetaTable.increment_total_inserts(conn, 100)
IndexMetaTable.reset_total_inserts(conn, 7)
assert IndexMetaTable.get_total_inserts(conn) == 7
Generated
+3 -3
View File
@@ -3735,15 +3735,15 @@ crypto = [
[[package]]
name = "pymdown-extensions"
version = "10.21.3"
version = "11.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "markdown", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" }
sdist = { url = "https://files.pythonhosted.org/packages/47/67/f1e79672a5f91985577c7984c9709ca110e4fd37fe7fd167b60422e6ccc2/pymdown_extensions-11.0.tar.gz", hash = "sha256:8269cef0247f9e2d0a62fcea10860aba05c1cbab5470fd4b63230b96434dc589", size = 857049, upload-time = "2026-06-23T02:27:45.146Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" },
{ url = "https://files.pythonhosted.org/packages/af/b6/1ae53367e28b9cffa3be7574e13fbe4589694272fd47710fbdbafd3d63c6/pymdown_extensions-11.0-py3-none-any.whl", hash = "sha256:fbc4acb641814fa9d17521bbd21a5240ef739a662f11c06330c4b78c93e954d6", size = 269415, upload-time = "2026-06-23T02:27:43.826Z" },
]
[[package]]