Compare commits

..
Author SHA1 Message Date
stumpylogandClaude Sonnet 5 86846255f4 perf: resolve permitted_document_ids(perm=delete_document, include_deleted=True) once for trash loop
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
2026-07-28 13:11:19 -07:00
stumpylog e4e93f9a9d test: use HTTPStatus enum instead of bare integers in security test assertions 2026-07-28 11:07:55 -07:00
stumpylog 9edcfa8682 perf: resolve permitted_document_ids once for root-document version-listing loop
BulkDownloadView.post() previously called has_perms_owner_aware() per row
inside the loop that resolves each document's root and latest version.
Resolve permitted_document_ids(request.user) once before the loop and check
membership by root_doc.pk instead, consistent with the other consolidated
permission-filtering sites.
2026-07-28 07:55:12 -07:00
stumpylogandClaude Sonnet 5 75f8e9f611 perf: resolve permitted_document_ids(perm=change_document) once before bulk-edit loops
Migrates the bulk document edit permission check in views.py and the
custom-field DOCUMENTLINK validator in serialisers.py off of
has_perms_owner_aware-per-document loops, resolving
permitted_document_ids(user, perm="change_document") once instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
2026-07-28 07:26:42 -07:00
stumpylogandClaude Sonnet 5 6794d6e835 perf: resolve permitted_document_ids once before email/share loops
Replaces per-document has_perms_owner_aware calls in the email-document
action and bulk share-link-bundle creation with a single
permitted_document_ids(request.user) resolution before the loop,
reducing DB round-trips while preserving identical permission
semantics (including the per-document error message on the bundle
endpoint).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
2026-07-27 23:41:22 -07:00
stumpylogandClaude Sonnet 5 09c60ea667 feat: add perm param to permitted_document_ids for change/delete checks
Widens permitted_document_ids(user, *, include_deleted=False) to
permitted_document_ids(user, *, perm="view_document", include_deleted=False)
so Stage 2 callers can check change_document/delete_document permissions
instead of the hardcoded view_document codename. Default is unchanged for
every existing call site.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
2026-07-27 21:36:38 -07:00
stumpylogandClaude Sonnet 5 d4fa852373 perf: migrate 3 serialisers.py Document permission sites to permitted_document_ids
Migrates _get_viewable_duplicates(), PaperlessTaskSerializer.get_duplicate_documents(),
and the ShareLinkBundle document field queryset to use permitted_document_ids()
instead of get_objects_for_user_owner_aware()/get_objects_for_user(), consolidating
onto the shared permission-filtering helper. The is_staff gate in
get_duplicate_documents() is preserved as-is.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
2026-07-27 20:34:31 -07:00
stumpylog b7384c1858 perf: migrate 5 single-call Document permission sites to permitted_document_ids
Swaps get_objects_for_user_owner_aware(user, "view_document", Document) for
Document.objects.filter(id__in=permitted_document_ids(user)) at 5 read-only,
single-call sites: AI chat "ask all documents", bulk-edit
_resolve_document_ids all:true branch, SelectionDataView permission check,
global search docs bucket, and the statistics endpoint's Document branch.

Confirmed all 3 callers of _resolve_document_ids always use the default
"view_document" codename before swapping. Added a regression test pinning
the AI-chat owner/permission boundary through the real API client, and
updated 2 existing mocked tests in test_views.py that asserted on
get_objects_for_user_owner_aware for the chat endpoint.
2026-07-27 20:27:30 -07:00
stumpylogandClaude Haiku 137a86bcb1 refactor: remove redundant deleted_at filter in permitted_document_ids
Document.objects already applies filter(deleted_at__isnull=True) internally
via SoftDeleteManager.get_queryset(), so the conditional filter was redundant.
Simplify to just use manager.all() in both branches — manager selection alone
ensures correct behavior (Document.objects excludes deleted, Document.global_objects
includes all).

Co-Authored-By: Claude Haiku <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session
2026-07-27 20:14:26 -07:00
stumpylog 4352a8be0f feat: add include_deleted param to permitted_document_ids
Widens permitted_document_ids to accept an include_deleted keyword-only
flag (default False, preserving current behavior) so later call sites
that need visibility into soft-deleted documents (e.g. trash restore)
can reuse this permission check instead of duplicating it.
2026-07-27 19:39:20 -07:00
stumpylog 2ece405b6d test: add permission-filtering security regression suite for Document 2026-07-27 15:24:41 -07:00
GitHub Actions 693a111c5a Auto translate strings 2026-07-27 19:35:15 +00:00
Trenton HandGitHub b19edd0b74 Fix: dedupe permission-visible documents when combined with multi-tag ALL filtering (#13331) (#13345)
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
2026-07-27 19:33:26 +00:00
Trenton HandGitHub 0e98a7f1ce Performance: Scope llm index updates to actually modified documents (#13322)
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library

* Perf: select_related/prefetch_related for the scoped LLM index batch

* Perf: select_related/prefetch_related for the full LLM index rebuild path

* Fix: address Copilot review findings on LLM index scoping
2026-07-27 10:23:43 -07:00
GitHub Actions 14517062c4 Auto translate strings 2026-07-27 16:53:33 +00:00
shamoonandGitHub 0c0faf7f80 Fixhancement: pass LLM output language to chat if specified (#13340) 2026-07-27 16:51:31 +00:00
SandroandGitHub eda79603fe Documentation: fix PAPERLESS_AI_LLM_OUTPUT_LANGUAGE heading level (#13341) 2026-07-27 09:37:09 -07:00
shamoonandGitHub 8f017942d9 Chore: resolve npm provenance issues with chokidar and semver (#13323) 2026-07-26 16:25:49 -07:00
GitHub Actions f4e7a2e9e4 Auto translate strings 2026-07-26 22:38:16 +00:00
shamoonandGitHub 317e534aa0 Fix: ensure preview reload on live changes (#13321) 2026-07-26 22:36:03 +00:00
Trenton HandGitHub 3ba6d0325a Fix: clamp out-of-range MailRule.order and ApplicationConfiguration DPI/page fields before smallint migration (#13316) 2026-07-26 22:11:27 +00:00
Trenton HandGitHub 318520a0f1 Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description (#13315) 2026-07-26 22:00:06 +00:00
Trenton HandGitHub ea1f3653a9 Fix: guard build_document_node against stale FK on deleted correspondent/doc type (#13318) 2026-07-26 21:45:59 +00:00
shamoonandGitHub 423d4b9f2d Fix: prevent search filter loss when closing document with Escape key (#13317) 2026-07-26 14:17:26 -07:00
shamoonandGitHub 8bc4e5ee94 Fix: fix frontend permissions display for stats/system perms (#13305) 2026-07-26 02:56:06 -07:00
37 changed files with 997 additions and 190 deletions
+1 -1
View File
@@ -2135,7 +2135,7 @@ used with the OpenAI-compatible backend to target a custom provider or local gat
Defaults to None.
### [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=<str>`](#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE) {#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE}
#### [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=<str>`](#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE) {#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE}
: The language to use for AI suggestions (results may vary by LLM model). If not supplied, defaults to the user's UI language setting or None.
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "paperless-ngx"
version = "3.0.3"
version = "3.0.2"
description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
readme = "README.md"
requires-python = ">=3.11"
@@ -1,4 +1,4 @@
import { expect, test } from '@playwright/test'
import { expect, test, type WebSocketRoute } from '@playwright/test'
import path from 'node:path'
const REQUESTS_HAR = path.join(__dirname, 'requests/api-document-detail.har')
@@ -95,3 +95,60 @@ test('should support quick filters', async ({ page }) => {
.click()
await expect(page).toHaveURL(/tags__id__all=4&sort=created&reverse=1&page=1/)
})
test('should finish reloading the preview after a remote document update', async ({
page,
}) => {
let resolveStatusSocket: (socket: WebSocketRoute) => void
const statusSocketReady = new Promise<WebSocketRoute>((resolve) => {
resolveStatusSocket = resolve
})
await page.routeWebSocket(/\/ws\/status\/$/, (socket) => {
resolveStatusSocket(socket)
})
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
let previewRequestCount = 0
page.on('request', (request) => {
if (request.url().includes('/api/documents/175/preview/')) {
previewRequestCount++
}
})
await page.goto('/documents/175/details')
await page.locator('pngx-document-detail').waitFor()
await expect(page.getByTitle('Storage path', { exact: true })).toHaveText(
/\w+/
)
const previewWasLoaded = await page.evaluate(() => {
const detail = document.querySelector('pngx-document-detail')
const component = (window as any).ng.getComponent(detail)
component.pdfPreviewLoaded({ numPages: 1 })
return component.previewLoaded()
})
expect(previewWasLoaded).toBe(true)
const previewRequestsBeforeReload = previewRequestCount
const statusSocket = await statusSocketReady
const documentReloaded = page.waitForResponse(
(response) =>
response.url().includes('/api/documents/175/?full_perms=true') &&
response.request().method() === 'GET'
)
statusSocket.send(
JSON.stringify({
type: 'document_updated',
data: {
document_id: 175,
modified: '2026-07-26T20:00:00Z',
},
})
)
await documentReloaded
await expect(
page.getByText('Document reloaded with latest changes.').first()
).toBeVisible()
await expect
.poll(() => previewRequestCount)
.toBeGreaterThan(previewRequestsBeforeReload + 1)
})
+49 -49
View File
@@ -1277,7 +1277,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1787</context>
<context context-type="linenumber">1791</context>
</context-group>
</trans-unit>
<trans-unit id="1577733187050997705" datatype="html">
@@ -2184,7 +2184,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">653</context>
<context context-type="linenumber">657</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html</context>
@@ -3087,11 +3087,11 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1405</context>
<context context-type="linenumber">1409</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1788</context>
<context context-type="linenumber">1792</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -3688,7 +3688,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1358</context>
<context context-type="linenumber">1362</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -3800,7 +3800,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1841</context>
<context context-type="linenumber">1845</context>
</context-group>
</trans-unit>
<trans-unit id="6661109599266152398" datatype="html">
@@ -3811,7 +3811,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1842</context>
<context context-type="linenumber">1846</context>
</context-group>
</trans-unit>
<trans-unit id="5162686434580248853" datatype="html">
@@ -3822,7 +3822,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1843</context>
<context context-type="linenumber">1847</context>
</context-group>
</trans-unit>
<trans-unit id="8157388568390631653" datatype="html">
@@ -5724,7 +5724,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1362</context>
<context context-type="linenumber">1366</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -7966,88 +7966,88 @@
<source>Enter Password</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
<context context-type="linenumber">512</context>
<context context-type="linenumber">513</context>
</context-group>
</trans-unit>
<trans-unit id="5758784066858623886" datatype="html">
<source>Error retrieving metadata</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">423</context>
<context context-type="linenumber">424</context>
</context-group>
</trans-unit>
<trans-unit id="2218903673684131427" datatype="html">
<source>An error occurred loading content: <x id="PH" equiv-text="err.message ?? err.toString()"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">525,527</context>
<context context-type="linenumber">526,528</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">982,984</context>
<context context-type="linenumber">986,988</context>
</context-group>
</trans-unit>
<trans-unit id="6357361810318120957" datatype="html">
<source>Document was updated</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">648</context>
<context context-type="linenumber">652</context>
</context-group>
</trans-unit>
<trans-unit id="5154064822428631306" datatype="html">
<source>Document was updated at <x id="PH" equiv-text="formattedModified"/>.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">649</context>
<context context-type="linenumber">653</context>
</context-group>
</trans-unit>
<trans-unit id="8462497568316256794" datatype="html">
<source>Reload to discard your local unsaved edits and load the latest remote version.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">650</context>
<context context-type="linenumber">654</context>
</context-group>
</trans-unit>
<trans-unit id="7967484035994732534" datatype="html">
<source>Reload</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">652</context>
<context context-type="linenumber">656</context>
</context-group>
</trans-unit>
<trans-unit id="2907037627372942104" datatype="html">
<source>Document reloaded with latest changes.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">708</context>
<context context-type="linenumber">712</context>
</context-group>
</trans-unit>
<trans-unit id="6435639868943916539" datatype="html">
<source>Document reloaded.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">719</context>
<context context-type="linenumber">723</context>
</context-group>
</trans-unit>
<trans-unit id="6142395741265832184" datatype="html">
<source>Next document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">821</context>
<context context-type="linenumber">825</context>
</context-group>
</trans-unit>
<trans-unit id="651985345816518480" datatype="html">
<source>Previous document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">831</context>
<context context-type="linenumber">835</context>
</context-group>
</trans-unit>
<trans-unit id="2885986061416655600" datatype="html">
<source>Close document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">839</context>
<context context-type="linenumber">843</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
@@ -8058,67 +8058,67 @@
<source>Save document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">846</context>
<context context-type="linenumber">850</context>
</context-group>
</trans-unit>
<trans-unit id="1784543155727940353" datatype="html">
<source>Save and close / next</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">855</context>
<context context-type="linenumber">859</context>
</context-group>
</trans-unit>
<trans-unit id="7427704425579737895" datatype="html">
<source>Error retrieving version content</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">965</context>
<context context-type="linenumber">969</context>
</context-group>
</trans-unit>
<trans-unit id="3456881259945295697" datatype="html">
<source>Error retrieving suggestions.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1026</context>
<context context-type="linenumber">1030</context>
</context-group>
</trans-unit>
<trans-unit id="2194092841814123758" datatype="html">
<source>Document &quot;<x id="PH" equiv-text="newValues.title"/>&quot; saved successfully.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1238</context>
<context context-type="linenumber">1242</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1265</context>
<context context-type="linenumber">1269</context>
</context-group>
</trans-unit>
<trans-unit id="6626387786259219838" datatype="html">
<source>Error saving document &quot;<x id="PH" equiv-text="this.document().title"/>&quot;</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1271</context>
<context context-type="linenumber">1275</context>
</context-group>
</trans-unit>
<trans-unit id="448882439049417053" datatype="html">
<source>Error saving document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1326</context>
<context context-type="linenumber">1330</context>
</context-group>
</trans-unit>
<trans-unit id="8410796510716511826" datatype="html">
<source>Do you really want to move the document &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash?</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1359</context>
<context context-type="linenumber">1363</context>
</context-group>
</trans-unit>
<trans-unit id="282586936710748252" datatype="html">
<source>Documents can be restored prior to permanent deletion.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1360</context>
<context context-type="linenumber">1364</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8129,14 +8129,14 @@
<source>Error deleting document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1381</context>
<context context-type="linenumber">1385</context>
</context-group>
</trans-unit>
<trans-unit id="619486176823357521" datatype="html">
<source>Reprocess confirm</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1401</context>
<context context-type="linenumber">1405</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8147,102 +8147,102 @@
<source>This operation will permanently recreate the archive file for this document.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1402</context>
<context context-type="linenumber">1406</context>
</context-group>
</trans-unit>
<trans-unit id="302054111564709516" datatype="html">
<source>The archive file will be re-generated with the current settings.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1403</context>
<context context-type="linenumber">1407</context>
</context-group>
</trans-unit>
<trans-unit id="4700389117298802932" datatype="html">
<source>Reprocess operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1413</context>
<context context-type="linenumber">1417</context>
</context-group>
</trans-unit>
<trans-unit id="4409560272830824468" datatype="html">
<source>Error executing operation</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1424</context>
<context context-type="linenumber">1428</context>
</context-group>
</trans-unit>
<trans-unit id="6030453331794586802" datatype="html">
<source>Error downloading document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1487</context>
<context context-type="linenumber">1491</context>
</context-group>
</trans-unit>
<trans-unit id="4458954481601077369" datatype="html">
<source>Page Fit</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1565</context>
<context context-type="linenumber">1569</context>
</context-group>
</trans-unit>
<trans-unit id="4663705961777238777" datatype="html">
<source>PDF edit operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1808</context>
<context context-type="linenumber">1812</context>
</context-group>
</trans-unit>
<trans-unit id="9043972994040261999" datatype="html">
<source>Error executing PDF edit operation</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1820</context>
<context context-type="linenumber">1824</context>
</context-group>
</trans-unit>
<trans-unit id="6172690334763056188" datatype="html">
<source>Please enter the current password before attempting to remove it.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1831</context>
<context context-type="linenumber">1835</context>
</context-group>
</trans-unit>
<trans-unit id="968660764814228922" datatype="html">
<source>Password removal operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1865</context>
<context context-type="linenumber">1869</context>
</context-group>
</trans-unit>
<trans-unit id="2282118435712883014" datatype="html">
<source>Error executing password removal operation</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1879</context>
<context context-type="linenumber">1883</context>
</context-group>
</trans-unit>
<trans-unit id="3740891324955700797" datatype="html">
<source>Print failed.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1925</context>
<context context-type="linenumber">1929</context>
</context-group>
</trans-unit>
<trans-unit id="6457245677384603573" datatype="html">
<source>Error loading document for printing.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">1938</context>
<context context-type="linenumber">1942</context>
</context-group>
</trans-unit>
<trans-unit id="6085793215710522488" datatype="html">
<source>An error occurred loading tiff: <x id="PH" equiv-text="err.toString()"/></source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2008</context>
<context context-type="linenumber">2012</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
<context context-type="linenumber">2014</context>
<context context-type="linenumber">2018</context>
</context-group>
</trans-unit>
<trans-unit id="4958946940233632319" datatype="html">
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "paperless-ngx-ui",
"version": "3.0.3",
"version": "3.0.2",
"scripts": {
"preinstall": "npx only-allow pnpm",
"ng": "ng",
+3
View File
@@ -2,6 +2,9 @@ packages:
- "."
minimumReleaseAge: 10080
trustPolicy: no-downgrade
trustPolicyExclude:
- "chokidar@4.0.3"
- "semver@6.3.1 || 5.7.2"
allowBuilds:
"@parcel/watcher": true
canvas: true
@@ -283,6 +283,22 @@ describe('PngxPdfViewerComponent', () => {
expect(mockViewer.currentPageNumber).toBe(1)
})
it('reloads when the source revision changes', () => {
const resetSpy = jest.spyOn(component as any, 'resetViewerState')
const loadSpy = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
component.src = 'test.pdf'
component.sourceRevision = 1
component.ngOnChanges({
sourceRevision: new SimpleChange(0, 1, false),
})
expect(resetSpy).toHaveBeenCalled()
expect(loadSpy).toHaveBeenCalled()
})
it('applies viewer state after view init when already loaded', () => {
const applySpy = jest.spyOn(component as any, 'applyViewerState')
;(component as any).hasLoaded = true
@@ -43,6 +43,7 @@ export class PngxPdfViewerComponent
private readonly document = inject<Document>(DOCUMENT)
@Input() src!: string
@Input() sourceRevision = 0
@Input() password?: string
@Input() page?: number
@Output() pageChange = new EventEmitter<number>()
@@ -93,7 +94,7 @@ export class PngxPdfViewerComponent
}
ngOnChanges(changes: SimpleChanges): void {
if (changes['src'] || changes['password']) {
if (changes['src'] || changes['sourceRevision'] || changes['password']) {
this.resetViewerState()
if (this.src) {
this.loadDocument()
@@ -472,6 +472,7 @@
<div class="preview-sticky pdf-viewer-container">
<pngx-pdf-viewer
[src]="pdfSource()"
[sourceRevision]="previewRevision()"
[password]="pdfPassword()"
[renderMode]="PdfRenderMode.All"
[page]="previewCurrentPage()" (pageChange)="previewCurrentPage.set($event)"
@@ -1602,6 +1602,7 @@ describe('DocumentDetailComponent', () => {
expect(openDoc.__changedFields).toEqual([])
expect(setDirtySpy).toHaveBeenCalledWith(openDoc, false)
expect(saveSpy).toHaveBeenCalled()
expect(component.previewRevision()).toBe(1)
})
it('should ignore incoming update for a different document id', () => {
@@ -249,6 +249,7 @@ export class DocumentDetailComponent
titleSubject: Subject<string> = new Subject()
readonly previewUrl = signal<string>(undefined)
readonly pdfSource = signal<string>(undefined)
readonly previewRevision = signal(0)
readonly pdfPassword = signal<string>(undefined)
readonly thumbUrl = signal<string>(undefined)
readonly previewText = signal<string>(undefined)
@@ -609,6 +610,9 @@ export class DocumentDetailComponent
.subscribe()
}
this.updateComponent(useDoc)
if (forceRemote) {
this.previewRevision.update((revision) => revision + 1)
}
this.titleSubject
.pipe(
debounceTime(1000),
@@ -25,7 +25,7 @@
<input #textFilterInput class="form-control form-control-sm" type="text"
[disabled]="textFilterModifierIsNull"
[(ngModel)]="textFilter"
(keyup)="textFilterKeyup($event)"
(keydown)="textFilterKeydown($event)"
[ngbTypeahead]="searchAutoComplete"
(selectItem)="itemSelected($event)"
[readonly]="textFilterTarget === 'fulltext-morelike'">
@@ -2180,17 +2180,17 @@ describe('FilterEditorComponent', () => {
it('should support Enter / Esc key on text field', () => {
component.textFilterInput.nativeElement.value = 'foo'
component.textFilterInput.nativeElement.dispatchEvent(
new KeyboardEvent('keyup', { key: 'Enter' })
new KeyboardEvent('keydown', { key: 'Enter' })
)
expect(component.textFilter).toEqual('foo')
component.textFilterInput.nativeElement.value = 'foo bar'
component.textFilterInput.nativeElement.dispatchEvent(
new KeyboardEvent('keyup', { key: 'Escape' })
new KeyboardEvent('keydown', { key: 'Escape' })
)
expect(component.textFilter).toEqual('')
const blurSpy = jest.spyOn(component.textFilterInput.nativeElement, 'blur')
component.textFilterInput.nativeElement.dispatchEvent(
new KeyboardEvent('keyup', { key: 'Escape' })
new KeyboardEvent('keydown', { key: 'Escape' })
)
expect(blurSpy).toHaveBeenCalled()
})
@@ -1312,7 +1312,7 @@ export class FilterEditorComponent
}
}
textFilterKeyup(event: KeyboardEvent) {
textFilterKeydown(event: KeyboardEvent) {
if (event.key == 'Enter') {
const filterString = (
this.textFilterInput.nativeElement as HTMLInputElement
@@ -108,6 +108,18 @@ describe('PermissionsService', () => {
actionKey: 'View', // PermissionAction.View
typeKey: 'Document', // PermissionType.Document
})
expect(
permissionsService.getPermissionKeys('view_global_statistics')
).toEqual({
actionKey: 'View', // PermissionAction.View
typeKey: 'GlobalStatistics', // PermissionType.GlobalStatistics
})
expect(
permissionsService.getPermissionKeys('view_system_monitoring')
).toEqual({
actionKey: 'View', // PermissionAction.View
typeKey: 'SystemMonitoring', // PermissionType.SystemMonitoring
})
})
it('correctly checks explicit global permissions', () => {
+7 -10
View File
@@ -110,19 +110,16 @@ export class PermissionsService {
actionKey: string
typeKey: string
} {
const matches = permissionStr.match(/(.+)_/)
let typeKey
let actionKey
if (matches?.length > 0) {
const action = matches[1]
const actionIndex = Object.values(PermissionAction).indexOf(
action as PermissionAction
)
if (actionIndex > -1) {
actionKey = Object.keys(PermissionAction)[actionIndex]
}
const actionIndex = Object.values(PermissionAction).findIndex((action) =>
permissionStr.startsWith(`${action}_`)
)
if (actionIndex > -1) {
const action = Object.values(PermissionAction)[actionIndex]
actionKey = Object.keys(PermissionAction)[actionIndex]
const typeIndex = Object.values(PermissionType).indexOf(
permissionStr.replace(action, '%s') as PermissionType
permissionStr.replace(`${action}_`, '%s_') as PermissionType
)
if (typeIndex > -1) {
typeKey = Object.keys(PermissionType)[typeIndex]
+1 -1
View File
@@ -8,7 +8,7 @@ export const environment = {
apiVersion: '10', // match src/paperless/settings.py
appTitle: DEFAULT_APP_TITLE,
tag: 'prod',
version: '3.0.3',
version: '3.0.2',
webSocketHost: window.location.host,
webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
webSocketBaseUrl: base_url.pathname + 'ws/',
+27
View File
@@ -37,6 +37,7 @@ from drf_spectacular.utils import extend_schema_field
from guardian.utils import get_group_obj_perms_model
from guardian.utils import get_user_obj_perms_model
from rest_framework import serializers
from rest_framework.filters import BaseFilterBackend
from rest_framework.filters import OrderingFilter
from rest_framework_guardian.filters import ObjectPermissionsFilter
@@ -50,6 +51,7 @@ from documents.models import ShareLink
from documents.models import ShareLinkBundle
from documents.models import StoragePath
from documents.models import Tag
from documents.permissions import permitted_document_ids
if TYPE_CHECKING:
from collections.abc import Callable
@@ -1038,6 +1040,31 @@ class ObjectOwnedOrGrantedPermissionsFilter(ObjectPermissionsFilter):
return objects_with_perms | objects_owned | objects_unowned
class DocumentPermissionsFilter(BaseFilterBackend):
"""
A filter backend limiting Document results to those the requesting user
owns, are unowned, or has explicit (user- or group-level) view
permission on.
Unlike ``ObjectOwnedOrGrantedPermissionsFilter``, this does not build an
``objects_with_perms | objects_owned | objects_unowned`` union of
querysets derived from the same base queryset. When that base queryset
already carries independent joins on a multi-valued relation (e.g. two
separate joins from ``tags__id__all`` filtering on two tags), each
OR-ed branch can end up pairing those joins' aliases differently,
letting more than one row out of the join's cross product satisfy the
combined WHERE -- returning the same document more than once. Filtering
via a single ``id__in`` against ``permitted_document_ids`` (a plain
subquery, not a join) sidesteps that entirely and is also cheaper than
guardian's join-based permission check.
"""
def filter_queryset(self, request, queryset, view):
if request.user.is_superuser:
return queryset
return queryset.filter(id__in=permitted_document_ids(request.user))
class ObjectOwnedPermissionsFilter(ObjectPermissionsFilter):
"""
A filter backend that limits results to those where the requesting user
+17 -8
View File
@@ -163,14 +163,23 @@ def set_permissions_for_object(
)
def _permitted_document_ids(user):
def permitted_document_ids(
user,
*,
perm: str = "view_document",
include_deleted: bool = False,
):
"""
Return a queryset of document IDs the user may view, limited to non-deleted
documents. This intentionally avoids ``get_objects_for_user`` to keep the
subquery small and index-friendly.
Return a queryset of document IDs the user has ``perm`` on (default
``"view_document"``). By default limited to non-deleted documents; pass
``include_deleted=True`` for callers that need to check permission on
soft-deleted documents (e.g. trash restore). This intentionally avoids
``get_objects_for_user`` to keep the subquery small and index-friendly.
"""
base_docs = Document.objects.filter(deleted_at__isnull=True).only("id", "owner")
manager = Document.global_objects if include_deleted else Document.objects
base_docs = manager.all()
base_docs = base_docs.only("id", "owner")
if user is None or not getattr(user, "is_authenticated", False):
# Just Anonymous user e.g. for drf-spectacular
@@ -181,7 +190,7 @@ def _permitted_document_ids(user):
document_ct = ContentType.objects.get_for_model(Document)
perm_filter = {
"permission__codename": "view_document",
"permission__codename": perm,
"permission__content_type": document_ct,
}
@@ -220,7 +229,7 @@ def get_document_count_filter_for_user(user, related_name: str = "documents"):
# Superuser: no permission filtering needed
return Q(**{f"{related_name}__deleted_at__isnull": True})
permitted_ids = _permitted_document_ids(user)
permitted_ids = permitted_document_ids(user)
return Q(**{f"{related_name}__id__in": permitted_ids})
@@ -311,7 +320,7 @@ def annotate_document_count_for_related_queryset(
queryset,
through_model=through_model,
related_object_field=related_object_field,
document_ids=_permitted_document_ids(user),
document_ids=permitted_document_ids(user),
target_field=target_field,
)
+9 -25
View File
@@ -39,7 +39,6 @@ from drf_spectacular.utils import extend_schema_field
from drf_spectacular.utils import extend_schema_serializer
from drf_writable_nested.serializers import NestedUpdateMixin
from guardian.core import ObjectPermissionChecker
from guardian.shortcuts import get_objects_for_user
from guardian.shortcuts import get_users_with_perms
from guardian.utils import get_group_obj_perms_model
from guardian.utils import get_user_obj_perms_model
@@ -80,8 +79,8 @@ from documents.models import WorkflowTrigger
from documents.parsers import is_mime_type_supported
from documents.permissions import get_document_count_filter_for_user
from documents.permissions import get_groups_with_only_permission
from documents.permissions import get_objects_for_user_owner_aware
from documents.permissions import has_perms_owner_aware
from documents.permissions import permitted_document_ids
from documents.permissions import set_permissions_for_object
from documents.regex import validate_regex_pattern
from documents.templating.filepath import validate_filepath_template_and_render
@@ -865,11 +864,8 @@ def validate_documentlink_targets(user, doc_ids):
if user is None:
return
target_documents = Document.objects.filter(id__in=doc_ids).select_related("owner")
if not all(
has_perms_owner_aware(user, "change_document", document)
for document in target_documents
):
permitted_change_ids = set(permitted_document_ids(user, perm="change_document"))
if not set(doc_ids) <= permitted_change_ids:
raise PermissionDenied(
_("Insufficient permissions."),
)
@@ -1011,13 +1007,8 @@ def _get_viewable_duplicates(
).exclude(pk=document.pk)
duplicates = duplicates.filter(root_document__isnull=True)
duplicates = duplicates.order_by("-created")
allowed = get_objects_for_user_owner_aware(
user,
"documents.view_document",
Document,
include_deleted=True,
)
return duplicates.filter(id__in=allowed)
allowed_ids = permitted_document_ids(user, include_deleted=True)
return duplicates.filter(id__in=allowed_ids)
class DuplicateDocumentSummarySerializer(serializers.Serializer[dict[str, Any]]):
@@ -2659,13 +2650,8 @@ class TaskSerializerV9(serializers.ModelSerializer[PaperlessTask]):
user = request.user
qs = Document.global_objects.filter(pk=dup_of)
if not user.is_staff:
with_perms = get_objects_for_user(
user,
"documents.view_document",
qs,
accept_global_perms=False,
)
qs = with_perms | qs.filter(owner=user) | qs.filter(owner__isnull=True)
allowed_ids = permitted_document_ids(user, include_deleted=True)
qs = qs.filter(pk__in=allowed_ids)
return list(qs.values("id", "title", "deleted_at"))
@@ -3515,8 +3501,6 @@ class StoragePathTestSerializer(SerializerWithPerms):
document_field = self.fields.get("document")
if not isinstance(document_field, serializers.PrimaryKeyRelatedField):
return
document_field.queryset = get_objects_for_user_owner_aware(
user,
"documents.view_document",
Document,
document_field.queryset = Document.objects.filter(
id__in=permitted_document_ids(user),
)
+2
View File
@@ -311,6 +311,7 @@ def sanity_check(*, raise_on_error: bool = True) -> str:
def bulk_update_documents(document_ids) -> None:
from documents.search import get_backend
document_ids = list(document_ids)
documents = Document.objects.filter(id__in=document_ids)
for doc in documents:
@@ -331,6 +332,7 @@ def bulk_update_documents(document_ids) -> None:
if ai_config.llm_index_enabled:
update_llm_index(
rebuild=False,
document_ids=document_ids,
)
+88
View File
@@ -14,6 +14,7 @@ from unittest import mock
import celery
from dateutil import parser
from django.conf import settings
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.core import mail
@@ -47,6 +48,8 @@ from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from documents.signals.handlers import run_workflows
from documents.tests.factories import DocumentFactory
from documents.tests.factories import TagFactory
from documents.tests.utils import ConsumeTaskMixin
from documents.tests.utils import DirectoriesMixin
from documents.tests.utils import read_streaming_response
@@ -1212,6 +1215,91 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
[u1_doc1.id],
)
def test_document_owned_and_group_shared_not_duplicated_when_filtering_by_tags(
self,
) -> None:
"""
GIVEN:
- A document owned by a user and also shared with a group the user belongs to
WHEN:
- The user filters documents by more than one tag (tags__id__all)
THEN:
- The document is returned exactly once, not once per permission path
(regression test for https://github.com/paperless-ngx/paperless-ngx/issues/13331)
"""
user = User.objects.create_user("user1")
user.user_permissions.add(*Permission.objects.filter(codename="view_document"))
group = Group.objects.create(name="group1")
user.groups.add(group)
tag1 = TagFactory()
tag2 = TagFactory()
doc = DocumentFactory(title="shared", owner=user)
doc.tags.add(tag1, tag2)
assign_perm("view_document", group, doc)
self.client.force_authenticate(user=user)
response = self.client.get(
f"/api/documents/?tags__id__all={tag1.id},{tag2.id}",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["results"][0]["id"], doc.id)
def test_document_permission_filter_excludes_unrelated_documents(self) -> None:
"""
GIVEN:
- A document owned by one user, with no permission granted to another user
WHEN:
- The unrelated user requests the document list
THEN:
- The document does not appear in their results
"""
owner = User.objects.create_user("owner1")
stranger = User.objects.create_user("stranger1")
stranger.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
DocumentFactory(title="private", owner=owner)
self.client.force_authenticate(user=stranger)
response = self.client.get("/api/documents/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 0)
def test_document_permission_filter_only_visible_to_group_members(self) -> None:
"""
GIVEN:
- A document shared with a group via object permissions
WHEN:
- A group member and a non-member both request the document list
THEN:
- Only the group member sees the document
"""
owner = User.objects.create_user("owner2")
member = User.objects.create_user("member1")
non_member = User.objects.create_user("nonmember1")
for u in (member, non_member):
u.user_permissions.add(*Permission.objects.filter(codename="view_document"))
group = Group.objects.create(name="group2")
member.groups.add(group)
doc = DocumentFactory(title="shared2", owner=owner)
assign_perm("view_document", group, doc)
self.client.force_authenticate(user=member)
response = self.client.get("/api/documents/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["results"][0]["id"], doc.id)
self.client.force_authenticate(user=non_member)
response = self.client.get("/api/documents/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 0)
def test_pagination_results(self) -> None:
"""
GIVEN:
@@ -0,0 +1,392 @@
from __future__ import annotations
from http import HTTPStatus
from unittest.mock import patch
import pytest
from django.contrib.auth.models import AnonymousUser
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.test import override_settings
from guardian.shortcuts import assign_perm
from rest_framework.test import APIClient
from documents.permissions import permitted_document_ids
from documents.serialisers import _get_viewable_duplicates
from documents.tests.factories import DocumentFactory
def assert_visible_document_ids(actual_ids, *, expected_visible, expected_hidden):
actual_ids = set(actual_ids)
for doc_id in expected_visible:
assert doc_id in actual_ids, (
f"document {doc_id} should be visible but was hidden"
)
for doc_id in expected_hidden:
assert doc_id not in actual_ids, (
f"document {doc_id} leaked but should be hidden"
)
@pytest.mark.django_db
class TestPermittedDocumentIdsSecurity:
def test_owner_sees_own_document(self):
user = User.objects.create_user(username="alice")
stranger = User.objects.create_user(username="mallory")
owned = DocumentFactory(owner=user)
strangers_doc = DocumentFactory(owner=stranger)
visible = permitted_document_ids(user)
assert_visible_document_ids(
visible,
expected_visible=[owned.pk],
expected_hidden=[strangers_doc.pk],
)
def test_unowned_document_visible_to_everyone(self):
user = User.objects.create_user(username="alice")
unowned = DocumentFactory(owner=None)
assert_visible_document_ids(
permitted_document_ids(user),
expected_visible=[unowned.pk],
expected_hidden=[],
)
def test_explicit_user_permission_grants_visibility(self):
grantee = User.objects.create_user(username="alice")
stranger = User.objects.create_user(username="mallory")
owner = User.objects.create_user(username="owner")
shared = DocumentFactory(owner=owner)
not_shared = DocumentFactory(owner=owner)
assign_perm("view_document", grantee, shared)
assert_visible_document_ids(
permitted_document_ids(grantee),
expected_visible=[shared.pk],
expected_hidden=[not_shared.pk],
)
assert_visible_document_ids(
permitted_document_ids(stranger),
expected_visible=[],
expected_hidden=[shared.pk, not_shared.pk],
)
def test_explicit_group_permission_grants_visibility_to_members_only(self):
owner = User.objects.create_user(username="owner")
member = User.objects.create_user(username="member")
non_member = User.objects.create_user(username="non_member")
group = Group.objects.create(name="finance")
member.groups.add(group)
shared = DocumentFactory(owner=owner)
assign_perm("view_document", group, shared)
assert_visible_document_ids(
permitted_document_ids(member),
expected_visible=[shared.pk],
expected_hidden=[],
)
assert_visible_document_ids(
permitted_document_ids(non_member),
expected_visible=[],
expected_hidden=[shared.pk],
)
def test_soft_deleted_document_excluded_by_default(self):
owner = User.objects.create_user(username="owner")
doc = DocumentFactory(owner=owner)
doc.delete() # soft delete
assert_visible_document_ids(
permitted_document_ids(owner),
expected_visible=[],
expected_hidden=[doc.pk],
)
def test_superuser_sees_everything_including_no_perm_documents(self):
superuser = User.objects.create_superuser(username="root")
owner = User.objects.create_user(username="owner")
doc = DocumentFactory(owner=owner)
assert_visible_document_ids(
permitted_document_ids(superuser),
expected_visible=[doc.pk],
expected_hidden=[],
)
def test_anonymous_user_sees_only_unowned_documents(self):
owner = User.objects.create_user(username="owner")
owned = DocumentFactory(owner=owner)
unowned = DocumentFactory(owner=None)
assert_visible_document_ids(
permitted_document_ids(AnonymousUser()),
expected_visible=[unowned.pk],
expected_hidden=[owned.pk],
)
@pytest.mark.django_db
class TestPermittedDocumentIdsIncludeDeleted:
def test_include_deleted_true_reveals_soft_deleted_owned_document(self):
owner = User.objects.create_user(username="owner")
doc = DocumentFactory(owner=owner)
doc.delete()
assert_visible_document_ids(
permitted_document_ids(owner, include_deleted=True),
expected_visible=[doc.pk],
expected_hidden=[],
)
def test_include_deleted_true_still_respects_permission_boundary(self):
owner = User.objects.create_user(username="owner")
stranger = User.objects.create_user(username="mallory")
doc = DocumentFactory(owner=owner)
doc.delete()
assert_visible_document_ids(
permitted_document_ids(stranger, include_deleted=True),
expected_visible=[],
expected_hidden=[doc.pk],
)
@pytest.mark.django_db
class TestAiChatAllDocumentsPermissionBoundary:
"""
Regression test pinning the "ask across all documents" AI chat behavior
(ChatStreamingView.post, no document_id) to the same owner/permission
boundary enforced by permitted_document_ids(). This call site was
migrated from get_objects_for_user_owner_aware() to
permitted_document_ids() in Task 5; this test must stay green across
that swap.
"""
ENDPOINT = "/api/documents/chat/"
@override_settings(AI_ENABLED=True)
@patch("documents.views.stream_chat_with_documents")
def test_chat_all_documents_excludes_unshared_document(self, mock_stream_chat):
mock_stream_chat.return_value = iter([b"data"])
owner = User.objects.create_user(username="owner")
asker = User.objects.create_user(username="asker")
asker.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
shared = DocumentFactory(owner=owner)
not_shared = DocumentFactory(owner=owner)
assign_perm("view_document", asker, shared)
client = APIClient()
client.force_authenticate(user=asker)
response = client.post(
self.ENDPOINT,
data={"q": "question"},
format="json",
)
assert response.status_code == HTTPStatus.OK
mock_stream_chat.assert_called_once()
_, kwargs = mock_stream_chat.call_args
visible_ids = {doc.pk for doc in kwargs["documents"]}
assert shared.pk in visible_ids
assert not_shared.pk not in visible_ids
@pytest.mark.django_db
class TestDuplicateDocumentsPermissionBoundary:
def test_get_viewable_duplicates_includes_soft_deleted_but_respects_perms(self):
owner = User.objects.create_user(username="owner")
stranger = User.objects.create_user(username="mallory")
original = DocumentFactory(owner=owner, checksum="dupe-checksum")
dup_visible = DocumentFactory(owner=owner, checksum="dupe-checksum")
dup_hidden = DocumentFactory(owner=owner, checksum="dupe-checksum")
dup_hidden.delete() # soft delete, should still be found (include_deleted=True)
assign_perm("view_document", stranger, dup_visible)
result_owner = _get_viewable_duplicates(original, owner)
assert {d.pk for d in result_owner} == {dup_visible.pk, dup_hidden.pk}
result_stranger = _get_viewable_duplicates(original, stranger)
assert {d.pk for d in result_stranger} == {dup_visible.pk}
@pytest.mark.django_db
class TestPermittedDocumentIdsArbitraryPermission:
def test_change_document_permission_is_distinct_from_view(self):
owner = User.objects.create_user(username="owner")
viewer_only = User.objects.create_user(username="viewer")
editor = User.objects.create_user(username="editor")
doc = DocumentFactory(owner=owner)
assign_perm("view_document", viewer_only, doc)
assign_perm("change_document", editor, doc)
assign_perm("view_document", editor, doc)
assert_visible_document_ids(
permitted_document_ids(editor, perm="change_document"),
expected_visible=[doc.pk],
expected_hidden=[],
)
assert_visible_document_ids(
permitted_document_ids(viewer_only, perm="change_document"),
expected_visible=[],
expected_hidden=[doc.pk],
)
def test_delete_permission_with_include_deleted_for_trash_restore(self):
owner = User.objects.create_user(username="owner")
stranger = User.objects.create_user(username="mallory")
doc = DocumentFactory(owner=owner)
doc.delete()
assert_visible_document_ids(
permitted_document_ids(owner, perm="delete_document", include_deleted=True),
expected_visible=[doc.pk],
expected_hidden=[],
)
assert_visible_document_ids(
permitted_document_ids(
stranger,
perm="delete_document",
include_deleted=True,
),
expected_visible=[],
expected_hidden=[doc.pk],
)
@pytest.mark.django_db
class TestEmailDocumentPermissionBoundary:
def test_email_action_rejects_document_without_view_permission(
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
rest_api_client.force_authenticate(user=requester)
hidden = DocumentFactory(owner=owner)
response = rest_api_client.post(
"/api/documents/email/",
{
"documents": [hidden.pk],
"addresses": "someone@example.com",
"subject": "test",
"message": "test",
},
format="json",
)
assert response.status_code == HTTPStatus.FORBIDDEN
@pytest.mark.django_db
class TestBulkEditChangePermissionBoundary:
def test_bulk_edit_rejects_document_without_change_permission(
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
# grant the global change_document permission so the object-level
# check (not the global has_perm check) is what's under test
requester.user_permissions.add(
Permission.objects.get(codename="change_document"),
)
rest_api_client.force_authenticate(user=requester)
assign_perm(
"view_document",
requester,
DocumentFactory(owner=owner),
) # unrelated grant
target = DocumentFactory(owner=owner)
assign_perm("view_document", requester, target) # view only, NOT change
response = rest_api_client.post(
"/api/documents/bulk_edit/",
{
"documents": [target.pk],
"method": "modify_tags",
"parameters": {"add_tags": [], "remove_tags": []},
},
format="json",
)
assert response.status_code == HTTPStatus.FORBIDDEN
@pytest.mark.django_db
class TestBulkDownloadPermissionChecksRootDocument:
def test_permission_checked_on_root_not_on_version(
self,
rest_api_client,
paperless_dirs,
_media_settings,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
rest_api_client.force_authenticate(user=requester)
root = DocumentFactory(owner=owner)
# a version of root that the requester has NOT been individually granted
version = DocumentFactory(owner=owner, root_document=root, version_index=1)
version.source_path.write_bytes(b"%PDF-1.4 test")
assign_perm("view_document", requester, root) # granted on ROOT only
response = rest_api_client.post(
"/api/documents/bulk_download/",
{"documents": [version.pk]},
format="json",
)
assert (
response.status_code == HTTPStatus.OK
) # visible because root is permitted
stranger = User.objects.create_user(username="mallory")
rest_api_client.force_authenticate(user=stranger)
response = rest_api_client.post(
"/api/documents/bulk_download/",
{"documents": [version.pk]},
format="json",
)
assert response.status_code == HTTPStatus.FORBIDDEN
@pytest.mark.django_db
class TestTrashRestorePermissionBoundary:
def test_restore_rejects_document_without_delete_permission(
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
rest_api_client.force_authenticate(user=requester)
doc = DocumentFactory(owner=owner)
assign_perm("view_document", requester, doc) # view only, NOT delete
doc.delete()
response = rest_api_client.post(
"/api/trash/",
{"documents": [doc.pk], "action": "restore"},
format="json",
)
assert response.status_code == HTTPStatus.FORBIDDEN
def test_restore_allows_document_with_explicit_delete_permission(
self,
rest_api_client,
):
owner = User.objects.create_user(username="owner")
requester = User.objects.create_user(username="requester")
rest_api_client.force_authenticate(user=requester)
doc = DocumentFactory(owner=owner)
assign_perm("delete_document", requester, doc)
doc.delete()
response = rest_api_client.post(
"/api/trash/",
{"documents": [doc.pk], "action": "restore"},
format="json",
)
assert response.status_code == HTTPStatus.OK
@@ -60,7 +60,7 @@ class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("document_ids", response.data)
@mock.patch("documents.views.has_perms_owner_aware", return_value=False)
@mock.patch("documents.views.permitted_document_ids", return_value=set())
def test_create_bundle_rejects_insufficient_permissions(self, perms_mock) -> None:
payload = {
"document_ids": [self.document.pk],
+6 -2
View File
@@ -401,6 +401,10 @@ class TestAIIndex(DirectoriesMixin, TestCase):
"documents.tasks.update_llm_index",
) as update_llm_index,
):
tasks.bulk_update_documents([doc.pk for doc in docs])
doc_ids = [doc.pk for doc in docs]
tasks.bulk_update_documents(doc_ids)
self.assertEqual(update_document_in_llm_index.apply_async.call_count, 0)
update_llm_index.assert_called_once()
update_llm_index.assert_called_once_with(
rebuild=False,
document_ids=doc_ids,
)
+34 -3
View File
@@ -612,11 +612,11 @@ class TestAIChatStreamingView(DirectoriesMixin, TestCase):
self.assertIn(b"AI is required for this feature", response.content)
@patch("documents.views.stream_chat_with_documents")
@patch("documents.views.get_objects_for_user_owner_aware")
@patch("documents.views.permitted_document_ids")
@override_settings(AI_ENABLED=True)
def test_post_no_document_id(self, mock_get_objects, mock_stream_chat) -> None:
def test_post_no_document_id(self, mock_permitted_ids, mock_stream_chat) -> None:
self.grant_view_document_permission()
mock_get_objects.return_value = [self.document]
mock_permitted_ids.return_value = [self.document.pk]
mock_stream_chat.return_value = iter([b"data"])
response = self.client.post(
self.ENDPOINT,
@@ -625,6 +625,37 @@ class TestAIChatStreamingView(DirectoriesMixin, TestCase):
)
self.assertEqual(response.status_code, 200)
self.assertEqual(response["Content-Type"], "text/event-stream")
mock_stream_chat.assert_called_once()
call_kwargs = mock_stream_chat.call_args.kwargs
self.assertEqual(call_kwargs["query_str"], "question")
self.assertEqual(list(call_kwargs["documents"]), [self.document])
self.assertIsNone(call_kwargs["output_language"])
@patch("documents.views.stream_chat_with_documents")
@patch("documents.views.permitted_document_ids")
@override_settings(AI_ENABLED=True)
def test_post_uses_user_display_language(
self,
mock_permitted_ids,
mock_stream_chat,
) -> None:
UiSettings.objects.create(user=self.user, settings={"language": "de-de"})
self.grant_view_document_permission()
mock_permitted_ids.return_value = [self.document.pk]
mock_stream_chat.return_value = iter([b"data"])
response = self.client.post(
self.ENDPOINT,
data='{"q": "question"}',
content_type="application/json",
)
self.assertEqual(response.status_code, 200)
mock_stream_chat.assert_called_once()
call_kwargs = mock_stream_chat.call_args.kwargs
self.assertEqual(call_kwargs["query_str"], "question")
self.assertEqual(list(call_kwargs["documents"]), [self.document])
self.assertEqual(call_kwargs["output_language"], "de-de")
@patch("documents.views.stream_chat_with_documents")
@override_settings(AI_ENABLED=True)
+52 -45
View File
@@ -133,6 +133,7 @@ from documents.file_handling import format_filename
from documents.filters import CorrespondentFilterSet
from documents.filters import CustomFieldFilterSet
from documents.filters import DocumentFilterSet
from documents.filters import DocumentPermissionsFilter
from documents.filters import DocumentsOrderingFilter
from documents.filters import DocumentTypeFilterSet
from documents.filters import ObjectOwnedOrGrantedPermissionsFilter
@@ -176,6 +177,7 @@ from documents.permissions import get_objects_for_user_owner_aware
from documents.permissions import has_global_statistics_permission
from documents.permissions import has_perms_owner_aware
from documents.permissions import has_system_status_permission
from documents.permissions import permitted_document_ids
from documents.permissions import set_permissions_for_object
from documents.plugins.date_parsing import get_date_parser
from documents.schema import generate_object_with_permissions_schema
@@ -653,6 +655,20 @@ class TagViewSet(PermissionsAwareDocumentCountMixin, ModelViewSet[Tag]):
update_document_parent_tags(tag, new_parent)
def _get_llm_output_language(ai_config: AIConfig, request) -> str | None:
output_language = ai_config.llm_output_language
if (
not output_language
and hasattr(request.user, "ui_settings")
and isinstance(
request.user.ui_settings.settings,
dict,
)
):
output_language = request.user.ui_settings.settings.get("language")
return output_language
@extend_schema_view(**generate_object_with_permissions_schema(DocumentTypeSerializer))
class DocumentTypeViewSet(
PermissionsAwareDocumentCountMixin,
@@ -972,7 +988,7 @@ class DocumentViewSet(
DjangoFilterBackend,
SearchFilter,
DocumentsOrderingFilter,
ObjectOwnedOrGrantedPermissionsFilter,
DocumentPermissionsFilter,
)
filterset_class = DocumentFilterSet
search_fields = ("title", "correspondent__name", "effective_content")
@@ -1514,16 +1530,7 @@ class DocumentViewSet(
if not ai_config.ai_enabled:
return HttpResponseBadRequest("AI is required for this feature")
output_language = ai_config.llm_output_language
if (
not output_language
and hasattr(request.user, "ui_settings")
and isinstance(
request.user.ui_settings.settings,
dict,
)
):
output_language = request.user.ui_settings.settings.get("language") or None
output_language = _get_llm_output_language(ai_config=ai_config, request=request)
llm_cache_backend = (
f"{ai_config.llm_backend}:{output_language}"
if output_language
@@ -1918,12 +1925,9 @@ class DocumentViewSet(
use_archive_version = validated_data.get("use_archive_version", True)
documents = Document.objects.select_related("owner").filter(pk__in=document_ids)
for document in documents:
if request.user is not None and not has_perms_owner_aware(
request.user,
"view_document",
document,
):
if request.user is not None:
permitted_ids = set(permitted_document_ids(request.user))
if not all(document.pk in permitted_ids for document in documents):
return HttpResponseForbidden("Insufficient permissions")
try:
@@ -2259,14 +2263,18 @@ class ChatStreamingView(GenericAPIView[Any]):
documents = [document]
else:
documents = get_objects_for_user_owner_aware(
request.user,
"view_document",
Document,
documents = Document.objects.filter(
id__in=permitted_document_ids(request.user),
)
output_language = _get_llm_output_language(ai_config=ai_config, request=request)
response = StreamingHttpResponse(
stream_chat_with_documents(query_str=question, documents=documents),
stream_chat_with_documents(
query_str=question,
documents=documents,
output_language=output_language,
),
content_type="text/event-stream",
)
return response
@@ -2724,10 +2732,8 @@ class DocumentSelectionMixin:
for key, value in filters.items()
if key not in _TANTIVY_SEARCH_PARAM_NAMES
}
permitted_documents = get_objects_for_user_owner_aware(
user,
permission_codename,
Document,
permitted_documents = Document.objects.filter(
id__in=permitted_document_ids(user),
)
# orm-filtered docs
filtered_documents = DocumentFilterSet(
@@ -2776,8 +2782,9 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
)
# check global and object permissions for all documents
permitted_change_ids = set(permitted_document_ids(user, perm="change_document"))
has_perms = user.has_perm("documents.change_document") and all(
has_perms_owner_aware(user, "change_document", doc) for doc in document_objs
doc.pk in permitted_change_ids for doc in document_objs
)
# check ownership for methods that change original document
@@ -3335,10 +3342,8 @@ class SelectionDataView(GenericAPIView[Any]):
serializer.is_valid(raise_exception=True)
ids = serializer.validated_data.get("documents")
permitted_documents = get_objects_for_user_owner_aware(
request.user,
"documents.view_document",
Document,
permitted_documents = Document.objects.filter(
id__in=permitted_document_ids(request.user),
)
if permitted_documents.filter(pk__in=ids).count() != len(ids):
return HttpResponseForbidden("Insufficient permissions")
@@ -3510,10 +3515,8 @@ class GlobalSearchView(PassUserMixin):
OBJECT_LIMIT = 3
docs = []
if request.user.has_perm("documents.view_document"):
all_docs = get_objects_for_user_owner_aware(
request.user,
"view_document",
Document,
all_docs = Document.objects.filter(
id__in=permitted_document_ids(request.user),
)
if db_only:
docs = all_docs.filter(title__icontains=query)[:OBJECT_LIMIT]
@@ -3717,11 +3720,7 @@ class StatisticsView(GenericAPIView[Any]):
documents = (
Document.objects.all()
if can_view_global_stats
else get_objects_for_user_owner_aware(
user,
"documents.view_document",
Document,
)
else Document.objects.filter(id__in=permitted_document_ids(user))
).filter(root_document__isnull=True)
tags = (
Tag.objects.all()
@@ -3838,9 +3837,10 @@ class BulkDownloadView(DocumentSelectionMixin, GenericAPIView[Any]):
content = serializer.validated_data.get("content")
follow_filename_format = serializer.validated_data.get("follow_formatting")
permitted_ids = set(permitted_document_ids(request.user))
for document in documents:
root_doc = get_root_document(document)
if not has_perms_owner_aware(request.user, "view_document", root_doc):
if root_doc.pk not in permitted_ids:
return HttpResponseForbidden("Insufficient permissions")
versioned_documents.append(
get_latest_version_for_root(
@@ -4504,8 +4504,9 @@ class ShareLinkBundleViewSet(PassUserMixin, ModelViewSet[ShareLinkBundle]):
)
documents = list(documents_qs)
permitted_ids = set(permitted_document_ids(request.user))
for document in documents:
if not has_perms_owner_aware(request.user, "view_document", document):
if document.pk not in permitted_ids:
raise ValidationError(
{
"document_ids": _(
@@ -5307,9 +5308,15 @@ class TrashView(ListModelMixin, PassUserMixin):
if doc_ids is not None
else self.filter_queryset(self.get_queryset()).all()
)
for doc in docs:
if not has_perms_owner_aware(request.user, "delete_document", doc):
return HttpResponseForbidden("Insufficient permissions")
permitted_ids = set(
permitted_document_ids(
request.user,
perm="delete_document",
include_deleted=True,
),
)
if not all(doc.pk in permitted_ids for doc in docs):
return HttpResponseForbidden("Insufficient permissions")
action = serializer.validated_data.get("action")
if action == "restore":
for doc in Document.deleted_objects.filter(id__in=doc_ids).all():
+20 -20
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-07-25 07:12+0000\n"
"POT-Creation-Date: 2026-07-27 19:34+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -21,39 +21,39 @@ msgstr ""
msgid "Documents"
msgstr ""
#: documents/filters.py:470
#: documents/filters.py:472
msgid "Value must be valid JSON."
msgstr ""
#: documents/filters.py:489
#: documents/filters.py:491
msgid "Invalid custom field query expression"
msgstr ""
#: documents/filters.py:499
#: documents/filters.py:501
msgid "Invalid expression list. Must be nonempty."
msgstr ""
#: documents/filters.py:520
#: documents/filters.py:522
msgid "Invalid logical operator {op!r}"
msgstr ""
#: documents/filters.py:534
#: documents/filters.py:536
msgid "Maximum number of query conditions exceeded."
msgstr ""
#: documents/filters.py:598
#: documents/filters.py:600
msgid "{name!r} is not a valid custom field."
msgstr ""
#: documents/filters.py:635
#: documents/filters.py:637
msgid "{data_type} does not support query expr {expr!r}."
msgstr ""
#: documents/filters.py:750 documents/models.py:136
#: documents/filters.py:752 documents/models.py:136
msgid "Maximum nesting depth exceeded."
msgstr ""
#: documents/filters.py:1067
#: documents/filters.py:1094
msgid "Custom field not found"
msgstr ""
@@ -1352,7 +1352,7 @@ msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:522 documents/serialisers.py:874
#: documents/serialisers.py:2763 documents/views.py:298 documents/views.py:2541
#: documents/serialisers.py:2763 documents/views.py:299 documents/views.py:2553
#: paperless_mail/serialisers.py:155
msgid "Insufficient permissions."
msgstr ""
@@ -1393,7 +1393,7 @@ msgstr ""
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2849 documents/views.py:4500
#: documents/serialisers.py:2849 documents/views.py:4512
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
@@ -1661,36 +1661,36 @@ msgstr ""
msgid "Unable to parse URI {value}"
msgstr ""
#: documents/views.py:291 documents/views.py:2538
#: documents/views.py:292 documents/views.py:2550
msgid "Invalid more_like_id"
msgstr ""
#: documents/views.py:1556
#: documents/views.py:1562
msgid "Invalid AI configuration."
msgstr ""
#: documents/views.py:1565
#: documents/views.py:1571
msgid "AI backend request timed out."
msgstr ""
#: documents/views.py:2363 documents/views.py:2684
#: documents/views.py:2375 documents/views.py:2696
msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr ""
#: documents/views.py:4512
#: documents/views.py:4524
#, python-format
msgid "Insufficient permissions to share document %(id)s."
msgstr ""
#: documents/views.py:4558
#: documents/views.py:4570
msgid "Bundle is already being processed."
msgstr ""
#: documents/views.py:4619
#: documents/views.py:4631
msgid "The share link bundle is still being prepared. Please try again later."
msgstr ""
#: documents/views.py:4629
#: documents/views.py:4641
msgid "The share link bundle is unavailable."
msgstr ""
@@ -5,12 +5,30 @@ from django.db import migrations
from django.db import models
def clamp_applicationconfiguration_integer_fields(apps, schema_editor):
# Clamp barcode_dpi, barcode_max_pages, image_dpi and pages because of
# PositiveIntegerField --> PositiveSmallIntegerField
ApplicationConfiguration = apps.get_model("paperless", "ApplicationConfiguration")
ApplicationConfiguration.objects.filter(barcode_dpi__gt=32767).update(
barcode_dpi=32767,
)
ApplicationConfiguration.objects.filter(barcode_max_pages__gt=32767).update(
barcode_max_pages=32767,
)
ApplicationConfiguration.objects.filter(image_dpi__gt=32767).update(image_dpi=32767)
ApplicationConfiguration.objects.filter(pages__gt=32767).update(pages=32767)
class Migration(migrations.Migration):
dependencies = [
("paperless", "0006_applicationconfiguration_barcode_tag_split"),
]
operations = [
migrations.RunPython(
clamp_applicationconfiguration_integer_fields,
migrations.RunPython.noop,
),
migrations.AlterField(
model_name="applicationconfiguration",
name="barcode_dpi",
+1 -1
View File
@@ -1,6 +1,6 @@
from typing import Final
__version__: Final[tuple[int, int, int]] = (3, 0, 3)
__version__: Final[tuple[int, int, int]] = (3, 0, 2)
# Version string like X.Y.Z
__full_version_str__: Final[str] = ".".join(map(str, __version__))
# Version string like X.Y
+2
View File
@@ -2,6 +2,8 @@ from pydantic import BaseModel
class DocumentClassifierSchema(BaseModel):
"""Schema for document classification suggestions."""
title: str
tags: list[str]
correspondents: list[str]
+27 -4
View File
@@ -28,11 +28,22 @@ CHAT_PROMPT_TMPL = (
"---------------------\n"
"Using only the context above, answer the query. "
"Do not use prior knowledge.\n"
"{output_language_line}"
"Query: {query_str}\n"
"Answer:"
)
def _build_chat_prompt(output_language: str | None) -> str:
output_language_line = (
f"Respond in {output_language}.\n" if output_language is not None else ""
)
return CHAT_PROMPT_TMPL.replace(
"{output_language_line}",
output_language_line,
)
def _build_document_reference(
document: Document,
title: str | None = None,
@@ -79,15 +90,27 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
)
def stream_chat_with_documents(query_str: str, documents: list[Document]):
def stream_chat_with_documents(
query_str: str,
documents: list[Document],
output_language: str | None = None,
):
try:
yield from _stream_chat_with_documents(query_str, documents)
yield from _stream_chat_with_documents(
query_str,
documents,
output_language=output_language,
)
except Exception as e:
logger.exception("Failed to stream document chat response: %s", e)
yield CHAT_ERROR_MESSAGE
def _stream_chat_with_documents(query_str: str, documents: list[Document]):
def _stream_chat_with_documents(
query_str: str,
documents: list[Document],
output_language: str | None = None,
):
if not documents:
yield CHAT_NO_CONTENT_MESSAGE
return
@@ -125,7 +148,7 @@ def _stream_chat_with_documents(query_str: str, documents: list[Document]):
references = _get_document_references(documents, top_nodes)
prompt_template = PromptTemplate(template=CHAT_PROMPT_TMPL)
prompt_template = PromptTemplate(template=_build_chat_prompt(output_language))
response_synthesizer = get_response_synthesizer(
llm=client.llm,
prompt_helper=get_rag_prompt_helper(
+37 -10
View File
@@ -5,6 +5,7 @@ from datetime import timedelta
from typing import TYPE_CHECKING
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist
from django.utils import timezone
from filelock import FileLock
from filelock import ReadWriteLock
@@ -167,6 +168,19 @@ def write_store(embed_model_name: str | None = None):
yield store
def _safe_related_name(document: Document, field: str) -> str | None:
"""
Returns the ``name`` of a related object (correspondent, document_type,
storage_path), or None if the FK is unset or points at a row that has
since been deleted (e.g. concurrently with this call).
"""
try:
related = getattr(document, field)
except ObjectDoesNotExist:
return None
return related.name if related else None
def build_document_node(
document: Document,
*,
@@ -180,14 +194,10 @@ def build_document_node(
"document_id": str(document.id),
"title": document.title,
"tags": [t.name for t in document.tags.all()],
"correspondent": document.correspondent.name
if document.correspondent
else None,
"document_type": document.document_type.name
if document.document_type
else None,
"correspondent": _safe_related_name(document, "correspondent"),
"document_type": _safe_related_name(document, "document_type"),
"filename": document.filename,
"storage_path": document.storage_path.name if document.storage_path else None,
"storage_path": _safe_related_name(document, "storage_path"),
"archive_serial_number": document.archive_serial_number,
"created": document.created.isoformat() if document.created else None,
"added": document.added.isoformat() if document.added else None,
@@ -318,8 +328,16 @@ def update_llm_index(
*,
iter_wrapper: IterWrapper[Document] = identity,
rebuild=False,
document_ids: Iterable[int] | None = None,
) -> str:
"""Rebuild or incrementally update the LLM index."""
"""Rebuild or incrementally update the LLM index.
``document_ids``, when given, scopes an incremental update to just those
documents instead of scanning the whole library -- callers that already
know which documents changed (e.g. a bulk edit) should pass this to avoid
an O(library size) scan per call. Ignored whenever a rebuild actually
happens, since a rebuild always covers the whole library regardless.
"""
with write_store() as store:
try:
with _exclude_readers():
@@ -335,7 +353,11 @@ def update_llm_index(
"LLM index migration requires re-embedding; forcing rebuild.",
)
rebuild = True
documents = Document.objects.all()
documents = Document.objects.select_related(
"correspondent",
"document_type",
"storage_path",
).prefetch_related("tags")
no_documents = not documents.exists()
# Fast exit before touching config: nothing to index and no existing index.
@@ -369,9 +391,14 @@ def update_llm_index(
store.add(nodes)
msg = "LLM index rebuilt successfully."
else:
scoped_documents = (
documents.filter(id__in=document_ids)
if document_ids is not None
else documents
)
existing = store.get_modified_times()
changed = 0
for document in iter_wrapper(documents):
for document in iter_wrapper(scoped_documents):
doc_id = str(document.id)
if existing.get(doc_id) == document.modified.isoformat():
continue
@@ -8,7 +8,9 @@ from django.test import override_settings
from django.utils import timezone
from llama_index.core.schema import MetadataMode
from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.models import PaperlessTask
from documents.signals import document_consumption_finished
from documents.signals import document_updated
@@ -95,6 +97,38 @@ def test_build_document_node_structured_fields_in_metadata(
assert "modified" in node.metadata
@pytest.mark.django_db
def test_build_document_node_survives_concurrently_deleted_correspondent(
real_document: Document,
) -> None:
"""Regression test for #13314.
If a document's correspondent (or document type) is deleted after the
in-memory Document instance was loaded but before build_document_node
resolves the relation, accessing the FK must not raise -- it should
behave like an unset FK and produce None in the metadata instead of
aborting the whole indexing pass.
"""
correspondent = Correspondent.objects.create(name="Stale Correspondent")
document_type = DocumentType.objects.create(name="Stale Type")
real_document.correspondent = correspondent
real_document.document_type = document_type
real_document.save()
# Re-fetch to get an instance whose correspondent/document_type relations
# are unresolved (not yet cached), mirroring a task that loaded the
# document before the concurrent deletion below.
stale_document = Document.objects.get(pk=real_document.pk)
correspondent.delete()
document_type.delete()
nodes = indexing.build_document_node(stale_document)
assert len(nodes) > 0
assert nodes[0].metadata["correspondent"] is None
assert nodes[0].metadata["document_type"] is None
@pytest.mark.django_db
def test_build_document_node_excludes_document_id_from_llm_context(
real_document: Document,
@@ -163,6 +197,8 @@ def test_update_llm_index(
mock_queryset = MagicMock()
mock_queryset.exists.return_value = True
mock_queryset.__iter__.return_value = iter([real_document])
mock_queryset.select_related.return_value = mock_queryset
mock_queryset.prefetch_related.return_value = mock_queryset
mock_all.return_value = mock_queryset
build_document_node.return_value = []
indexing.update_llm_index(rebuild=True)
@@ -182,6 +218,8 @@ def test_update_llm_index_rebuilds_on_model_name_change(
mock_queryset = MagicMock()
mock_queryset.exists.return_value = True
mock_queryset.__iter__.return_value = iter([real_document])
mock_queryset.select_related.return_value = mock_queryset
mock_queryset.prefetch_related.return_value = mock_queryset
mock_all.return_value = mock_queryset
with patch(
"paperless_ai.indexing.get_configured_model_name",
@@ -194,6 +232,8 @@ def test_update_llm_index_rebuilds_on_model_name_change(
mock_queryset = MagicMock()
mock_queryset.exists.return_value = True
mock_queryset.__iter__.return_value = iter([real_document])
mock_queryset.select_related.return_value = mock_queryset
mock_queryset.prefetch_related.return_value = mock_queryset
mock_all.return_value = mock_queryset
with patch(
"paperless_ai.indexing.get_configured_model_name",
@@ -224,6 +264,8 @@ def test_update_llm_index_partial_update(
mock_queryset = MagicMock()
mock_queryset.exists.return_value = True
mock_queryset.__iter__.return_value = iter([real_document, doc2])
mock_queryset.select_related.return_value = mock_queryset
mock_queryset.prefetch_related.return_value = mock_queryset
mock_all.return_value = mock_queryset
indexing.update_llm_index(rebuild=True)
@@ -252,6 +294,22 @@ def test_update_llm_index_partial_update(
assert store.table_exists(), (
"Expected the vector store table to exist after incremental update"
)
before = store.get_modified_times()
# A further edit, scoped via document_ids to just doc3 -- doc2 must be
# left exactly as it was, proving document_ids restricts the scan
# instead of falling back to the whole library.
doc3.modified = timezone.now()
doc3.save()
result = indexing.update_llm_index(rebuild=False, document_ids=[doc3.pk])
assert result == "LLM index updated successfully."
with indexing.get_vector_store() as store:
after = store.get_modified_times()
assert after[str(doc3.pk)] == doc3.modified.isoformat()
assert after[str(doc2.pk)] == before[str(doc2.pk)]
@pytest.mark.django_db
+21
View File
@@ -12,6 +12,7 @@ from paperless_ai import chat
from paperless_ai import indexing
from paperless_ai.chat import CHAT_ERROR_MESSAGE
from paperless_ai.chat import CHAT_METADATA_DELIMITER
from paperless_ai.chat import _build_chat_prompt
from paperless_ai.chat import stream_chat_with_documents
@@ -59,6 +60,26 @@ def assert_chat_output(
}
@pytest.mark.parametrize(
("output_language", "expected_language_line"),
[
(None, ""),
("de-de", "Respond in de-de.\n"),
],
)
def test_build_chat_prompt(
output_language,
expected_language_line,
) -> None:
prompt = _build_chat_prompt(output_language)
assert "{output_language_line}" not in prompt
assert (
prompt.split("Do not use prior knowledge.\n", maxsplit=1)[1]
== f"{expected_language_line}Query: {{query_str}}\nAnswer:"
)
@pytest.mark.django_db
def test_stream_chat_with_one_document_retrieval(
mock_document,
@@ -10,6 +10,24 @@ def clamp_mailrule_maximum_age(apps, schema_editor):
MailRule.objects.filter(maximum_age__gt=32767).update(maximum_age=32767)
def clamp_mailrule_order(apps, schema_editor):
# The order field of MailRule is only used for relative sorting
# (account.rules.order_by("order")), so out-of-range values must be
# renumbered by rank rather than clamped to a single value, which
# would collapse distinct rules to the same order.
MailRule = apps.get_model("paperless_mail", "MailRule")
if (
MailRule.objects.filter(order__gt=32767).exists()
or MailRule.objects.filter(
order__lt=-32768,
).exists()
): # pragma: no cover
for index, rule_id in enumerate(
MailRule.objects.order_by("order", "pk").values_list("pk", flat=True),
):
MailRule.objects.filter(pk=rule_id).update(order=index)
class Migration(migrations.Migration):
# The data update must commit before PostgreSQL rewrites the table, otherwise
# pending foreign-key trigger events can block the subsequent ALTER TABLE.
@@ -135,6 +153,10 @@ class Migration(migrations.Migration):
verbose_name="maximum age",
),
),
migrations.RunPython(
clamp_mailrule_order,
migrations.RunPython.noop,
),
migrations.AlterField(
model_name="mailrule",
name="order",
Generated
+1 -1
View File
@@ -2880,7 +2880,7 @@ wheels = [
[[package]]
name = "paperless-ngx"
version = "3.0.3"
version = "3.0.2"
source = { virtual = "." }
dependencies = [
{ name = "azure-ai-documentintelligence", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },