Compare commits

..
Author SHA1 Message Date
Trenton Holmes 8308ab0552 Fix: pass document chat queries as a QuerySet instead of a materialized list
In tracemalloc based profiling, not materializing the whole Document list
reduced memory to approximately 20% of the baseline, with a peak memory
that scaled with the library size.  Now, the lazt queryset is used and only
the needed pk value is actually contributing to memory
2026-08-10 19:39:57 -07:00
31 changed files with 204 additions and 577 deletions
+4 -5
View File
@@ -948,11 +948,10 @@ for display in the web interface.
!!! note !!! note
The **remote OCR parser** (Azure AI) also honors this setting: when The **remote OCR parser** (Azure AI) always produces a searchable
no archive is requested (`never`, or `auto` with a born-digital PDF), PDF and stores it as the archive copy, regardless of this setting.
the remote engine is skipped entirely and locally-extracted text is `ARCHIVE_FILE_GENERATION=never` has no effect when the remote
used instead, avoiding an unnecessary API call and a duplicate text parser handles a document.
layer.
#### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN} #### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
+4 -5
View File
@@ -187,11 +187,10 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
### Remote OCR parser ### Remote OCR parser
If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is If you use the **remote OCR parser** (Azure AI), note that it always produces a
honored the same way as for the local engine: when no archive is requested searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
(`never`, or `auto` with a born-digital PDF), the remote engine is skipped has no effect for documents handled by the remote parser - the archive is produced
entirely and locally-extracted text is used instead, avoiding an unnecessary unconditionally by the remote engine.
API call and a duplicate text layer.
## Search Index (Whoosh -> Tantivy) ## Search Index (Whoosh -> Tantivy)
+1 -3
View File
@@ -576,9 +576,7 @@ The following workflow action types are available:
- Tags, correspondent, document type and storage path - Tags, correspondent, document type and storage path
- Document owner - Document owner
- View and / or edit permissions to users or groups - View and / or edit permissions to users or groups
- Custom fields, optionally with a value. If no value is set, the field is only added to the - Custom fields. Note that no value for the field will be set
document and any value it may already have is left untouched. If a value is set, it will
overwrite an existing value of that field on the document.
##### Removal {#workflow-action-removal} ##### Removal {#workflow-action-removal}
+6 -13
View File
@@ -1703,7 +1703,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context> <context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">34</context> <context context-type="linenumber">28</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context> <context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
@@ -3279,7 +3279,7 @@
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context> <context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">46</context> <context context-type="linenumber">40</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context> <context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
@@ -7070,39 +7070,32 @@
<context context-type="linenumber">143</context> <context context-type="linenumber">143</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="8336346011691074629" datatype="html">
<source>No suggestions</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">11,12</context>
</context-group>
</trans-unit>
<trans-unit id="5320136382998259826" datatype="html"> <trans-unit id="5320136382998259826" datatype="html">
<source>Suggest</source> <source>Suggest</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context> <context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">13,14</context> <context context-type="linenumber">8,9</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="6934085657687954669" datatype="html"> <trans-unit id="6934085657687954669" datatype="html">
<source>Show suggestions</source> <source>Show suggestions</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context> <context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">23,24</context> <context context-type="linenumber">17,18</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="3834115140127576673" datatype="html"> <trans-unit id="3834115140127576673" datatype="html">
<source>No novel suggestions</source> <source>No novel suggestions</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context> <context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">30,31</context> <context context-type="linenumber">24,25</context>
</context-group> </context-group>
</trans-unit> </trans-unit>
<trans-unit id="4369111787961525769" datatype="html"> <trans-unit id="4369111787961525769" datatype="html">
<source>Document Types</source> <source>Document Types</source>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context> <context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
<context context-type="linenumber">40</context> <context context-type="linenumber">34</context>
</context-group> </context-group>
<context-group purpose="location"> <context-group purpose="location">
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context> <context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
+1 -1
View File
@@ -66,5 +66,5 @@
"ts-node": "~10.9.1", "ts-node": "~10.9.1",
"typescript": "^6.0.3" "typescript": "^6.0.3"
}, },
"packageManager": "pnpm@11.15.1" "packageManager": "pnpm@10.26.0"
} }
-1
View File
@@ -5,7 +5,6 @@ trustPolicy: no-downgrade
trustPolicyExclude: trustPolicyExclude:
- "chokidar@4.0.3" - "chokidar@4.0.3"
- "semver@6.3.1 || 5.7.2" - "semver@6.3.1 || 5.7.2"
blockExoticSubdeps: true
allowBuilds: allowBuilds:
"@parcel/watcher": true "@parcel/watcher": true
canvas: true canvas: true
@@ -52,10 +52,10 @@ describe('CustomFieldsValuesComponent', () => {
}) })
it('should set selectedFields and map values correctly', () => { it('should set selectedFields and map values correctly', () => {
component.value = { 1: 'value1', 3: 0, 4: false } component.value = { 1: 'value1' }
component.selectedFields = [1, 2, 3, 4] component.selectedFields = [1, 2]
expect(component.selectedFields).toEqual([1, 2, 3, 4]) expect(component.selectedFields).toEqual([1, 2])
expect(component.value).toEqual({ 1: 'value1', 2: null, 3: 0, 4: false }) expect(component.value).toEqual({ 1: 'value1', 2: null })
}) })
it('should return the correct custom field by id', () => { it('should return the correct custom field by id', () => {
@@ -77,7 +77,7 @@ export class CustomFieldsValuesComponent extends AbstractInputComponent<Object>
this._selectedFields = newFields this._selectedFields = newFields
// map the selected fields to an object with field_id as key and value as value // map the selected fields to an object with field_id as key and value as value
this.value = newFields.reduce((acc, fieldId) => { this.value = newFields.reduce((acc, fieldId) => {
acc[fieldId] = this.value?.[fieldId] ?? null acc[fieldId] = this.value?.[fieldId] || null
return acc return acc
}, {}) }, {})
this.onChange(this.value) this.onChange(this.value)
@@ -2,16 +2,10 @@
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())"> <button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())">
@if (loading()) { @if (loading()) {
<div class="spinner-border spinner-border-sm" role="status"></div> <div class="spinner-border spinner-border-sm" role="status"></div>
} @else if (noSuggestions) {
<i-bs width="1.2em" height="1.2em" name="check-circle"></i-bs>
} @else { } @else {
<i-bs width="1.2em" height="1.2em" name="stars"></i-bs> <i-bs width="1.2em" height="1.2em" name="stars"></i-bs>
} }
@if (noSuggestions) { <span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
<span class="d-none d-lg-inline ps-1" i18n>No suggestions</span>
} @else {
<span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
}
@if (totalSuggestions > 0) { @if (totalSuggestions > 0) {
<span class="badge bg-primary ms-2">{{ totalSuggestions }}</span> <span class="badge bg-primary ms-2">{{ totalSuggestions }}</span>
} }
@@ -25,7 +19,7 @@
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown"> <div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
<div class="list-group list-group-flush small pb-0"> <div class="list-group list-group-flush small pb-0">
@if (totalSuggestions === 0) { @if (!suggestions()?.suggested_tags && !suggestions()?.suggested_document_types && !suggestions()?.suggested_correspondents) {
<div class="list-group-item text-muted fst-italic"> <div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>No novel suggestions</small> <small class="text-muted small fst-italic" i18n>No novel suggestions</small>
</div> </div>
@@ -30,34 +30,6 @@ describe('SuggestionsDropdownComponent', () => {
expect(component.totalSuggestions).toBe(4) expect(component.totalSuggestions).toBe(4)
}) })
it('should show when a completed request returned no suggestions', () => {
fixture.componentRef.setInput('suggestions', {
correspondents: [],
tags: [],
document_types: [],
storage_paths: [],
dates: [],
})
fixture.detectChanges()
expect(component.noSuggestions).toBeTruthy()
expect(fixture.nativeElement.textContent).toContain('No suggestions')
})
it('should not show the empty state before a request or with suggestions', () => {
expect(component.noSuggestions).toBeFalsy()
fixture.componentRef.setInput('suggestions', {
correspondents: [],
tags: [42],
document_types: [],
storage_paths: [],
dates: [],
})
expect(component.noSuggestions).toBeFalsy()
})
it('should emit getSuggestions when clickSuggest is called and suggestions are null', () => { it('should emit getSuggestions when clickSuggest is called and suggestions are null', () => {
jest.spyOn(component.getSuggestions, 'emit') jest.spyOn(component.getSuggestions, 'emit')
fixture.componentRef.setInput('suggestions', null) fixture.componentRef.setInput('suggestions', null)
@@ -87,6 +59,5 @@ describe('SuggestionsDropdownComponent', () => {
}) })
component.clickSuggest() component.clickSuggest()
expect(component.dropdown.open).toBeTruthy() expect(component.dropdown.open).toBeTruthy()
expect(fixture.nativeElement.textContent).toContain('No novel suggestions')
}) })
}) })
@@ -61,21 +61,4 @@ export class SuggestionsDropdownComponent {
this.suggestions()?.suggested_document_types?.length || 0 this.suggestions()?.suggested_document_types?.length || 0
) )
} }
get noSuggestions(): boolean {
const suggestions = this.suggestions()
return (
suggestions != null &&
!suggestions.title &&
!suggestions.tags?.length &&
!suggestions.suggested_tags?.length &&
!suggestions.correspondents?.length &&
!suggestions.suggested_correspondents?.length &&
!suggestions.document_types?.length &&
!suggestions.suggested_document_types?.length &&
!suggestions.storage_paths?.length &&
!suggestions.suggested_storage_paths?.length &&
!suggestions.dates?.length
)
}
} }
-6
View File
@@ -1047,12 +1047,6 @@ class PermittedObjectsFilter(BaseFilterBackend):
perm_codename: str | None = None perm_codename: str | None = None
def filter_queryset(self, request, queryset, view): def filter_queryset(self, request, queryset, view):
# Before the superuser and owner-only paths, neither of which consults
# permitted_object_ids. Scoped to authenticated users so anonymous
# access (AnonymousUser.is_active is False) keeps its existing
# unowned-only behaviour.
if request.user.is_authenticated and not request.user.is_active:
return queryset.none()
if request.user.is_superuser: if request.user.is_superuser:
return queryset return queryset
if not self.include_granted: if not self.include_granted:
+3 -18
View File
@@ -54,15 +54,11 @@ class PaperlessObjectPermissions(DjangoObjectPermissions):
class PaperlessAdminPermissions(BasePermission): class PaperlessAdminPermissions(BasePermission):
def has_permission(self, request, view): def has_permission(self, request, view):
return request.user.is_active and request.user.is_staff return request.user.is_staff
def has_global_statistics_permission(user: User | None) -> bool: def has_global_statistics_permission(user: User | None) -> bool:
if ( if user is None or not getattr(user, "is_authenticated", False):
user is None
or not getattr(user, "is_active", False)
or not getattr(user, "is_authenticated", False)
):
return False return False
return getattr(user, "is_superuser", False) or user.has_perm( return getattr(user, "is_superuser", False) or user.has_perm(
@@ -71,11 +67,7 @@ def has_global_statistics_permission(user: User | None) -> bool:
def has_system_status_permission(user: User | None) -> bool: def has_system_status_permission(user: User | None) -> bool:
if ( if user is None or not getattr(user, "is_authenticated", False):
user is None
or not getattr(user, "is_active", False)
or not getattr(user, "is_authenticated", False)
):
return False return False
return ( return (
@@ -196,13 +188,6 @@ def permitted_object_ids(
if user is None or not getattr(user, "is_authenticated", False): if user is None or not getattr(user, "is_authenticated", False):
return base_qs.filter(owner__isnull=True).values_list("id", flat=True) return base_qs.filter(owner__isnull=True).values_list("id", flat=True)
# Deactivated users get nothing, deactivated superusers included, so this
# has to come before the superuser shortcut. guardian's
# ObjectPermissionChecker denies inactive users, but get_objects_for_user
# (the pattern this replaces) does not, so it would not be inherited.
if not getattr(user, "is_active", False):
return base_qs.none().values_list("id", flat=True)
if getattr(user, "is_superuser", False): if getattr(user, "is_superuser", False):
return base_qs.values_list("id", flat=True) return base_qs.values_list("id", flat=True)
-7
View File
@@ -3213,13 +3213,6 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
{"assign_title": f'Invalid f-string detected: "{e.args[0]}"'}, {"assign_title": f'Invalid f-string detected: "{e.args[0]}"'},
) )
if attrs.get("assign_custom_fields_values"):
# Empty strings treated as None to avoid unexpected behavior
attrs["assign_custom_fields_values"] = {
field_id: (None if value == "" else value)
for field_id, value in attrs["assign_custom_fields_values"].items()
}
if ( if (
"type" in attrs "type" in attrs
and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL
@@ -422,11 +422,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
json.dumps( json.dumps(
{ {
"assign_title": "", "assign_title": "",
"assign_custom_fields": [self.cf1.id, self.cf2.id],
"assign_custom_fields_values": {
str(self.cf1.id): "",
str(self.cf2.id): 0,
},
}, },
), ),
content_type="application/json", content_type="application/json",
@@ -434,10 +429,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_201_CREATED) self.assertEqual(response.status_code, status.HTTP_201_CREATED)
action = WorkflowAction.objects.get(id=response.data["id"]) action = WorkflowAction.objects.get(id=response.data["id"])
self.assertIsNone(action.assign_title) self.assertIsNone(action.assign_title)
self.assertEqual(
action.assign_custom_fields_values,
{str(self.cf1.id): None, str(self.cf2.id): 0},
)
response = self.client.post( response = self.client.post(
self.ENDPOINT_TRIGGERS, self.ENDPOINT_TRIGGERS,
@@ -496,28 +496,6 @@ class TestPermittedObjectIdsGenericModels:
expected_hidden=[strangers.pk], expected_hidden=[strangers.pk],
) )
@pytest.mark.parametrize("is_superuser", [False, True])
def test_inactive_user_sees_nothing(self, model, factory, perm, is_superuser):
suffix = f"{model.__name__}_{is_superuser}"
user = User.objects.create_user(
username=f"inactive_{suffix}",
is_active=False,
is_superuser=is_superuser,
)
other = User.objects.create_user(username=f"other_{suffix}")
granted = factory(owner=other)
assign_perm(perm, user, granted)
assert_visible_document_ids(
permitted_object_ids(user, model, perm),
expected_visible=[],
expected_hidden=[
factory(owner=None).pk,
factory(owner=user).pk,
granted.pk,
],
)
def test_unowned_object_visible_to_everyone(self, model, factory, perm): def test_unowned_object_visible_to_everyone(self, model, factory, perm):
user = User.objects.create_user(username=f"user_{model.__name__}") user = User.objects.create_user(username=f"user_{model.__name__}")
unowned = factory(owner=None) unowned = factory(owner=None)
@@ -68,44 +68,3 @@ class TestPermittedObjectsFilter:
visible_ids = set(result.values_list("id", flat=True)) visible_ids = set(result.values_list("id", flat=True))
assert visible_ids == {owned.pk} assert visible_ids == {owned.pk}
assert granted.pk not in visible_ids assert granted.pk not in visible_ids
@pytest.mark.parametrize(
("username", "is_superuser"),
[("inactive", False), ("inactive_super", True)],
)
def test_inactive_user_sees_nothing(self, username: str, *, is_superuser: bool):
user = User.objects.create_user(
username=username,
is_active=False,
is_superuser=is_superuser,
)
TagFactory(owner=None)
TagFactory(owner=user)
granted = TagFactory(owner=User.objects.create_user(username=f"o_{username}"))
assign_perm("view_tag", user, granted)
request = APIRequestFactory().get("/")
request.user = user
result = PermittedObjectsFilter().filter_queryset(
request,
Tag.objects.all(),
_DummyView(),
)
assert result.count() == 0
def test_inactive_user_sees_nothing_with_include_granted_false(self):
user = User.objects.create_user(username="inactive_owner", is_active=False)
TagFactory(owner=user)
TagFactory(owner=None)
request = APIRequestFactory().get("/")
request.user = user
class _OwnerOnlyFilter(PermittedObjectsFilter):
include_granted = False
result = _OwnerOnlyFilter().filter_queryset(
request,
Tag.objects.all(),
_DummyView(),
)
assert result.count() == 0
-49
View File
@@ -2000,55 +2000,6 @@ class TestWorkflows(
r"Doc added in \w{3,}", r"Doc added in \w{3,}",
) # Match any 3-letter month name ) # Match any 3-letter month name
def test_document_updated_workflow_existing_custom_field_empty_value(self) -> None:
"""
GIVEN:
- Existing workflow with UPDATED trigger and action that assigns a custom field
with an empty value
WHEN:
- Document is updated that already contains the field with a value
THEN:
- The existing value is left untouched, see GH #13627
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
filter_has_document_type=self.dt,
)
action = WorkflowAction.objects.create()
action.assign_custom_fields.add(self.cf1)
action.assign_custom_fields_values = {self.cf1.pk: ""}
action.save()
w = Workflow.objects.create(
name="Workflow 1",
order=0,
)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
correspondent=self.c,
original_filename="sample.pdf",
)
CustomFieldInstance.objects.create(
document=doc,
field=self.cf1,
value_text="existing value",
)
superuser = User.objects.create_superuser("superuser")
self.client.force_authenticate(user=superuser)
self.client.patch(
f"/api/documents/{doc.id}/",
{"document_type": self.dt.id},
format="json",
)
doc.refresh_from_db()
self.assertEqual(doc.custom_fields.get(field=self.cf1).value, "existing value")
def test_document_updated_workflow_existing_custom_field(self) -> None: def test_document_updated_workflow_existing_custom_field(self) -> None:
""" """
GIVEN: GIVEN:
+1 -1
View File
@@ -2267,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
if not has_perms_owner_aware(request.user, "view_document", document): if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions") return HttpResponseForbidden("Insufficient permissions")
documents = [document] documents = Document.objects.filter(pk=document.pk)
else: else:
documents = Document.objects.filter( documents = Document.objects.filter(
id__in=permitted_document_ids(request.user), id__in=permitted_document_ids(request.user),
+1 -2
View File
@@ -105,8 +105,7 @@ def apply_assignment_to_document(
field=field, field=field,
document=document, document=document,
).first() ).first()
# empty string is indistinguishable from no value in the UI if instance and args[value_field_name] is not None:
if instance and args[value_field_name] not in (None, ""):
setattr(instance, value_field_name, args[value_field_name]) setattr(instance, value_field_name, args[value_field_name])
instance.save() instance.save()
elif not instance: elif not instance:
+20 -20
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr "" msgstr ""
"Project-Id-Version: paperless-ngx\n" "Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n" "Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-08-10 02:25+0000\n" "POT-Creation-Date: 2026-08-07 20:00+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n" "PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n" "Last-Translator: \n"
"Language-Team: English\n" "Language-Team: English\n"
@@ -21,39 +21,39 @@ msgstr ""
msgid "Documents" msgid "Documents"
msgstr "" msgstr ""
#: documents/filters.py:471 #: documents/filters.py:472
msgid "Value must be valid JSON." msgid "Value must be valid JSON."
msgstr "" msgstr ""
#: documents/filters.py:490 #: documents/filters.py:491
msgid "Invalid custom field query expression" msgid "Invalid custom field query expression"
msgstr "" msgstr ""
#: documents/filters.py:500 #: documents/filters.py:501
msgid "Invalid expression list. Must be nonempty." msgid "Invalid expression list. Must be nonempty."
msgstr "" msgstr ""
#: documents/filters.py:521 #: documents/filters.py:522
msgid "Invalid logical operator {op!r}" msgid "Invalid logical operator {op!r}"
msgstr "" msgstr ""
#: documents/filters.py:535 #: documents/filters.py:536
msgid "Maximum number of query conditions exceeded." msgid "Maximum number of query conditions exceeded."
msgstr "" msgstr ""
#: documents/filters.py:599 #: documents/filters.py:600
msgid "{name!r} is not a valid custom field." msgid "{name!r} is not a valid custom field."
msgstr "" msgstr ""
#: documents/filters.py:636 #: documents/filters.py:637
msgid "{data_type} does not support query expr {expr!r}." msgid "{data_type} does not support query expr {expr!r}."
msgstr "" msgstr ""
#: documents/filters.py:755 documents/models.py:136 #: documents/filters.py:756 documents/models.py:136
msgid "Maximum nesting depth exceeded." msgid "Maximum nesting depth exceeded."
msgstr "" msgstr ""
#: documents/filters.py:1079 #: documents/filters.py:1098
msgid "Custom field not found" msgid "Custom field not found"
msgstr "" msgstr ""
@@ -1352,7 +1352,7 @@ msgid "workflow runs"
msgstr "" msgstr ""
#: documents/serialisers.py:521 documents/serialisers.py:873 #: documents/serialisers.py:521 documents/serialisers.py:873
#: documents/serialisers.py:2767 documents/views.py:299 documents/views.py:2555 #: documents/serialisers.py:2767 documents/views.py:300 documents/views.py:2556
#: paperless_mail/serialisers.py:155 #: paperless_mail/serialisers.py:155
msgid "Insufficient permissions." msgid "Insufficient permissions."
msgstr "" msgstr ""
@@ -1393,7 +1393,7 @@ msgstr ""
msgid "Duplicate document identifiers are not allowed." msgid "Duplicate document identifiers are not allowed."
msgstr "" msgstr ""
#: documents/serialisers.py:2853 documents/views.py:4509 #: documents/serialisers.py:2853 documents/views.py:4510
#, python-format #, python-format
msgid "Documents not found: %(ids)s" msgid "Documents not found: %(ids)s"
msgstr "" msgstr ""
@@ -1661,36 +1661,36 @@ msgstr ""
msgid "Unable to parse URI {value}" msgid "Unable to parse URI {value}"
msgstr "" msgstr ""
#: documents/views.py:292 documents/views.py:2552 #: documents/views.py:293 documents/views.py:2553
msgid "Invalid more_like_id" msgid "Invalid more_like_id"
msgstr "" msgstr ""
#: documents/views.py:1566 #: documents/views.py:1567
msgid "Invalid AI configuration." msgid "Invalid AI configuration."
msgstr "" msgstr ""
#: documents/views.py:1575 #: documents/views.py:1576
msgid "AI backend request timed out." msgid "AI backend request timed out."
msgstr "" msgstr ""
#: documents/views.py:2377 documents/views.py:2698 #: documents/views.py:2378 documents/views.py:2699
msgid "Specify only one of text, title_search, query, or more_like_id." msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr "" msgstr ""
#: documents/views.py:4522 #: documents/views.py:4523
#, python-format #, python-format
msgid "Insufficient permissions to share document %(id)s." msgid "Insufficient permissions to share document %(id)s."
msgstr "" msgstr ""
#: documents/views.py:4568 #: documents/views.py:4569
msgid "Bundle is already being processed." msgid "Bundle is already being processed."
msgstr "" msgstr ""
#: documents/views.py:4629 #: documents/views.py:4630
msgid "The share link bundle is still being prepared. Please try again later." msgid "The share link bundle is still being prepared. Please try again later."
msgstr "" msgstr ""
#: documents/views.py:4639 #: documents/views.py:4640
msgid "The share link bundle is unavailable." msgid "The share link bundle is unavailable."
msgstr "" msgstr ""
+1 -4
View File
@@ -19,10 +19,7 @@ class AutoLoginMiddleware(MiddlewareMixin):
if request.path.startswith("/api/token/") and request.method == "POST": if request.path.startswith("/api/token/") and request.method == "POST":
return None return None
try: try:
request.user = User.objects.get( request.user = User.objects.get(username=settings.AUTO_LOGIN_USERNAME)
username=settings.AUTO_LOGIN_USERNAME,
is_active=True,
)
auth.login( auth.login(
request=request, request=request,
user=request.user, user=request.user,
+7 -33
View File
@@ -3,9 +3,7 @@ Built-in remote-OCR document parser.
Handles documents by sending them to a configured remote OCR engine Handles documents by sending them to a configured remote OCR engine
(currently Azure AI Vision / Document Intelligence) and retrieving both (currently Azure AI Vision / Document Intelligence) and retrieving both
the extracted text and a searchable PDF with an embedded text layer. For the extracted text and a searchable PDF with an embedded text layer.
born-digital PDFs that need no archive copy, the remote call is skipped
entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
When no engine is configured, ``score()`` returns ``None`` so the parser When no engine is configured, ``score()`` returns ``None`` so the parser
is effectively invisible to the registry the tesseract parser handles is effectively invisible to the registry the tesseract parser handles
@@ -24,8 +22,6 @@ from typing import Self
from django.conf import settings from django.conf import settings
from documents.parsers import ParseError from documents.parsers import ParseError
from paperless.parsers.utils import extract_pdf_text
from paperless.parsers.utils import post_process_text
from paperless.version import __full_version_str__ from paperless.version import __full_version_str__
if TYPE_CHECKING: if TYPE_CHECKING:
@@ -74,11 +70,8 @@ class RemoteDocumentParser:
"""Parse documents via a remote OCR API (currently Azure AI Vision). """Parse documents via a remote OCR API (currently Azure AI Vision).
This parser sends documents to a remote engine that returns both This parser sends documents to a remote engine that returns both
extracted text and a searchable PDF with an embedded text layer, extracted text and a searchable PDF with an embedded text layer.
except when ``parse()`` is called with ``produce_archive=False`` for It does not depend on Tesseract or ocrmypdf.
a PDF, in which case the remote call is skipped and only locally
extracted text is returned (no archive). It does not depend on
Tesseract or ocrmypdf.
Class attributes Class attributes
---------------- ----------------
@@ -167,11 +160,8 @@ class RemoteDocumentParser:
Returns Returns
------- -------
bool bool
Always True the remote engine is capable of returning a PDF Always True the remote engine always returns a PDF with an
with an embedded text layer to serve as the archive copy. embedded text layer that serves as the archive copy.
Whether it actually does so for a given document depends on
``produce_archive`` passed to :meth:`parse` (see there for when
the remote engine call, and thus archive generation, is skipped).
""" """
return True return True
@@ -228,12 +218,6 @@ class RemoteDocumentParser:
) -> None: ) -> None:
"""Send the document to the remote engine and store results. """Send the document to the remote engine and store results.
When *produce_archive* is False for a PDF, the caller (via
``documents.consumer.should_produce_archive``) has already determined
that the document is born-digital and needs no archive skip the
remote engine entirely rather than re-OCRing it and creating a
duplicate text layer.
Parameters Parameters
---------- ----------
document_path: document_path:
@@ -241,8 +225,8 @@ class RemoteDocumentParser:
mime_type: mime_type:
Detected MIME type of the document. Detected MIME type of the document.
produce_archive: produce_archive:
Whether an archive copy is wanted. For PDFs, False skips the Ignored the remote engine always returns a searchable PDF,
remote engine and uses locally-extracted text instead. which is stored as the archive copy regardless of this flag.
""" """
config = RemoteEngineConfig( config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE, engine=settings.REMOTE_OCR_ENGINE,
@@ -257,16 +241,6 @@ class RemoteDocumentParser:
self._text = "" self._text = ""
return return
if not produce_archive and mime_type == "application/pdf":
logger.debug(
"Remote OCR: skipped — no archive requested, "
"using locally-extracted text",
)
self._text = (
post_process_text(extract_pdf_text(document_path, log=logger)) or ""
)
return
if config.engine == "azureai": if config.engine == "azureai":
self._text = self._azure_ai_vision_parse(document_path, config) self._text = self._azure_ai_vision_parse(document_path, config)
-1
View File
@@ -217,7 +217,6 @@ class ApplicationConfigurationSerializer(
llm_api_key = ObfuscatedPasswordField( llm_api_key = ObfuscatedPasswordField(
required=False, required=False,
allow_null=True, allow_null=True,
max_length=1024,
) )
def run_validation(self, data): def run_validation(self, data):
@@ -337,117 +337,6 @@ class TestRemoteParserParse:
assert remote_parser.get_date() is None assert remote_parser.get_date() is None
# ---------------------------------------------------------------------------
# parse() — produce_archive=False skips the remote engine (PDFs only)
# ---------------------------------------------------------------------------
class TestRemoteParserSkipsWhenNoArchiveWanted:
"""When the caller has already decided no archive is needed for a PDF
(documents.consumer.should_produce_archive), the remote engine call is
skipped entirely in favor of locally-extracted text.
"""
def test_pdf_skips_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: Azure is never invoked, no archive is produced, and text
comes from local pdftotext extraction
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_not_called()
assert remote_parser.get_archive_path() is None
assert remote_parser.get_text() != ""
def test_pdf_no_archive_requested_text_matches_local_extraction(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
mocker: MockerFixture,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: the returned text is exactly the locally-extracted text,
not anything from the (unused) Azure mock
"""
mocker.patch(
"paperless.parsers.remote.extract_pdf_text",
return_value="Local digital text.",
)
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == "Local digital text."
def test_pdf_no_archive_requested_closes_no_client(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.close.assert_not_called()
def test_non_pdf_still_calls_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
Images have no local-text fallback, so produce_archive=False does
not skip the remote engine for non-PDF MIME types.
"""
remote_parser.parse(
simple_digital_pdf_file,
"image/png",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_called_once()
assert remote_parser.get_text() == _DEFAULT_TEXT
@pytest.mark.usefixtures("no_engine_settings")
def test_unconfigured_engine_takes_precedence_over_skip(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
) -> None:
"""An unconfigured engine still short-circuits before the
produce_archive check, returning empty text as before.
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == ""
assert remote_parser.get_archive_path() is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# parse() — Azure failure path # parse() — Azure failure path
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -1,53 +0,0 @@
from django.contrib.auth.models import AnonymousUser
from django.contrib.auth.models import User
from django.test import RequestFactory
from django.test import TestCase
from django.test import override_settings
from paperless.auth import AutoLoginMiddleware
@override_settings(AUTO_LOGIN_USERNAME="autologin")
class TestAutoLoginMiddleware(TestCase):
def setUp(self) -> None:
super().setUp()
self.factory = RequestFactory()
self.middleware = AutoLoginMiddleware(lambda request: None)
def _process(self, request):
# login() needs a session to write to
request.session = self.client.session
self.middleware.process_request(request)
return request
def test_active_user_is_logged_in(self) -> None:
"""
GIVEN:
- AUTO_LOGIN_USERNAME names an active user
WHEN:
- A request is processed by the middleware
THEN:
- That user is attached to the request
"""
user = User.objects.create_user(username="autologin")
request = self._process(self.factory.get("/"))
self.assertEqual(request.user, user)
def test_deactivated_user_is_not_logged_in(self) -> None:
"""
GIVEN:
- AUTO_LOGIN_USERNAME names a user who has been deactivated
WHEN:
- A request is processed by the middleware
THEN:
- The request is left anonymous rather than authenticated as them
"""
User.objects.create_user(username="autologin", is_active=False)
request = self.factory.get("/")
request.user = AnonymousUser()
self._process(request)
self.assertFalse(request.user.is_authenticated)
+21 -6
View File
@@ -2,6 +2,8 @@ import json
import logging import logging
import sys import sys
from django.db.models import QuerySet
from documents.models import Document from documents.models import Document
from paperless.config import AIConfig from paperless.config import AIConfig
from paperless_ai.client import AIClient from paperless_ai.client import AIClient
@@ -82,10 +84,21 @@ def _build_document_reference(
def _get_document_references( def _get_document_references(
documents: list[Document], documents: QuerySet[Document],
top_nodes: list, top_nodes: list,
) -> list[dict[str, int | str]]: ) -> list[dict[str, int | str]]:
allowed_documents = {doc.pk: doc for doc in documents} candidate_ids: set[int] = set()
for node in top_nodes:
try:
candidate_ids.add(int(node.metadata["document_id"]))
except (KeyError, TypeError, ValueError): # pragma: no cover
continue
if not candidate_ids:
return []
allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
references: list[dict[str, int | str]] = [] references: list[dict[str, int | str]] = []
seen_document_ids: set[int] = set() seen_document_ids: set[int] = set()
@@ -119,7 +132,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
def stream_chat_with_documents( def stream_chat_with_documents(
query_str: str, query_str: str,
documents: list[Document], documents: QuerySet[Document],
output_language: str | None = None, output_language: str | None = None,
): ):
try: try:
@@ -135,10 +148,10 @@ def stream_chat_with_documents(
def _stream_chat_with_documents( def _stream_chat_with_documents(
query_str: str, query_str: str,
documents: list[Document], documents: QuerySet[Document],
output_language: str | None = None, output_language: str | None = None,
): ):
if not documents: if not documents.exists():
yield CHAT_NO_CONTENT_MESSAGE yield CHAT_NO_CONTENT_MESSAGE
return return
@@ -148,7 +161,9 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig() config = AIConfig()
filters = _document_id_filters(str(doc.pk) for doc in documents) filters = _document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
# Hold the shared read lock for the whole operation: the query engine # Hold the shared read lock for the whole operation: the query engine
# retrieves from the vector store again during synthesis, so the connection # retrieves from the vector store again during synthesis, so the connection
+101 -46
View File
@@ -3,10 +3,12 @@ from unittest.mock import MagicMock
from unittest.mock import patch from unittest.mock import patch
import pytest import pytest
from django.db.models.signals import post_init
from llama_index.core import settings as llama_settings from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode from llama_index.core.schema import TextNode
from documents.models import Document
from documents.tests.factories import DocumentFactory from documents.tests.factories import DocumentFactory
from paperless_ai import chat from paperless_ai import chat
from paperless_ai import indexing from paperless_ai import indexing
@@ -36,16 +38,6 @@ def patch_embed_nodes():
yield mock_embed_nodes yield mock_embed_nodes
@pytest.fixture
def mock_document():
doc = MagicMock()
doc.pk = 1
doc.title = "Test Document"
doc.filename = "test_file.pdf"
doc.content = "This is the document content."
return doc
def assert_chat_output( def assert_chat_output(
output: list[str], output: list[str],
*, *,
@@ -61,6 +53,13 @@ def assert_chat_output(
} }
def _fake_documents_queryset(pks: list[int]) -> MagicMock:
qs = MagicMock()
qs.exists.return_value = bool(pks)
qs.values_list.return_value = pks
return qs
@pytest.mark.parametrize( @pytest.mark.parametrize(
("output_language", "expected_language_line"), ("output_language", "expected_language_line"),
[ [
@@ -107,9 +106,10 @@ def test_build_refine_prompt(
@pytest.mark.django_db @pytest.mark.django_db
def test_stream_chat_with_one_document_retrieval( def test_stream_chat_with_one_document_retrieval(
mock_document,
patch_embed_nodes, patch_embed_nodes,
) -> None: ) -> None:
document = DocumentFactory.create(title="Test Document", content="ignored")
documents = Document.objects.filter(pk=document.pk)
with ( with (
patch("paperless_ai.chat.AIClient") as mock_client_cls, patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index, patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -124,22 +124,19 @@ def test_stream_chat_with_one_document_retrieval(
mock_client_cls.return_value = mock_client mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock() mock_client.llm = MagicMock()
mock_node = TextNode(
text="This is node content.",
metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
)
mock_index = MagicMock() mock_index = MagicMock()
# Simulate get_nodes returning nodes (content exists) mock_index.vector_store.get_nodes.return_value = [
mock_index.vector_store.get_nodes.return_value = [mock_node] TextNode(
text="This is node content.",
metadata={"document_id": str(document.pk), "title": "Test Document"},
),
]
mock_load_index.return_value = mock_index mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock() mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [ mock_retriever_instance.retrieve.return_value = [
MagicMock( MagicMock(
metadata={ metadata={"document_id": str(document.pk), "title": "Test Document"},
"document_id": str(mock_document.pk),
"title": "Test Document",
},
), ),
] ]
@@ -153,7 +150,7 @@ def test_stream_chat_with_one_document_retrieval(
"llama_index.core.retrievers.VectorIndexRetriever", "llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance, return_value=mock_retriever_instance,
): ):
output = list(stream_chat_with_documents("What is this?", [mock_document])) output = list(stream_chat_with_documents("What is this?", documents))
mock_query_engine.query.assert_called_once_with("What is this?") mock_query_engine.query.assert_called_once_with("What is this?")
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
@@ -166,13 +163,16 @@ def test_stream_chat_with_one_document_retrieval(
output, output,
expected_chunks=["chunk1", "chunk2"], expected_chunks=["chunk1", "chunk2"],
expected_references=[ expected_references=[
{"id": mock_document.pk, "title": "Test Document"}, {"id": document.pk, "title": "Test Document"},
], ],
) )
@pytest.mark.django_db @pytest.mark.django_db
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None: def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
doc1 = DocumentFactory.create(title="Document 1", content="ignored")
doc2 = DocumentFactory.create(title="Document 2", content="ignored")
documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
with ( with (
patch("paperless_ai.chat.AIClient") as mock_client_cls, patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index, patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_client_cls.return_value = mock_client mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock() mock_client.llm = MagicMock()
mock_node1 = TextNode(
text="Content for doc 1.",
metadata={"document_id": "1", "title": "Document 1"},
)
mock_node2 = TextNode(
text="Content for doc 2.",
metadata={"document_id": "2", "title": "Document 2"},
)
mock_index = MagicMock() mock_index = MagicMock()
# Simulate get_nodes returning nodes (content exists) mock_index.vector_store.get_nodes.return_value = [
mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2] TextNode(
text="Content for doc 1.",
metadata={"document_id": str(doc1.pk), "title": "Document 1"},
),
TextNode(
text="Content for doc 2.",
metadata={"document_id": str(doc2.pk), "title": "Document 2"},
),
]
mock_load_index.return_value = mock_index mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock() mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [ mock_retriever_instance.retrieve.return_value = [
MagicMock(metadata={"document_id": "1", "title": "Document 1"}), MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
MagicMock(metadata={"document_id": "2", "title": "Document 2"}), MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
] ]
mock_response_stream = MagicMock() mock_response_stream = MagicMock()
@@ -210,14 +210,11 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_query_engine_cls.return_value = mock_query_engine mock_query_engine_cls.return_value = mock_query_engine
mock_query_engine.query.return_value = mock_response_stream mock_query_engine.query.return_value = mock_response_stream
doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
with patch( with patch(
"llama_index.core.retrievers.VectorIndexRetriever", "llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance, return_value=mock_retriever_instance,
): ):
output = list(stream_chat_with_documents("What's up?", [doc1, doc2])) output = list(stream_chat_with_documents("What's up?", documents))
mock_query_engine.query.assert_called_once_with("What's up?") mock_query_engine.query.assert_called_once_with("What's up?")
patch_embed_nodes.assert_not_called() patch_embed_nodes.assert_not_called()
@@ -225,15 +222,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
output, output,
expected_chunks=["chunk1", "chunk2"], expected_chunks=["chunk1", "chunk2"],
expected_references=[ expected_references=[
{"id": 1, "title": "Document 1"}, {"id": doc1.pk, "title": "Document 1"},
{"id": 2, "title": "Document 2"}, {"id": doc2.pk, "title": "Document 2"},
], ],
) )
def test_stream_chat_empty_document_list() -> None: def test_stream_chat_empty_document_list() -> None:
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index: with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
output = list(stream_chat_with_documents("Any info?", [])) output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
mock_load_index.assert_not_called() mock_load_index.assert_not_called()
assert output == ["Sorry, I couldn't find any content to answer your question."] assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -253,7 +250,9 @@ def test_stream_chat_no_matching_nodes() -> None:
mock_index.vector_store.get_nodes.return_value = [] mock_index.vector_store.get_nodes.return_value = []
mock_load_index.return_value = mock_index mock_load_index.return_value = mock_index
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)])) output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
assert output == ["Sorry, I couldn't find any content to answer your question."] assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -282,7 +281,9 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
) )
mock_retriever_cls.return_value = mock_retriever mock_retriever_cls.return_value = mock_retriever
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)])) output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
assert output == [CHAT_ERROR_MESSAGE] assert output == [CHAT_ERROR_MESSAGE]
assert "Failed to stream document chat response" in caplog.text assert "Failed to stream document chat response" in caplog.text
@@ -298,7 +299,12 @@ class TestStreamChatRetrieval:
) -> None: ) -> None:
doc = DocumentFactory.create(content="hello world") doc = DocumentFactory.create(content="hello world")
# Nothing indexed for this document yet. # Nothing indexed for this document yet.
out = list(chat.stream_chat_with_documents("question?", [doc])) out = list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=doc.pk),
),
)
assert chat.CHAT_NO_CONTENT_MESSAGE in out assert chat.CHAT_NO_CONTENT_MESSAGE in out
def test_chat_filter_contains_only_requested_document_ids( def test_chat_filter_contains_only_requested_document_ids(
@@ -332,7 +338,12 @@ class TestStreamChatRetrieval:
side_effect=capture_retriever, side_effect=capture_retriever,
) )
list(chat.stream_chat_with_documents("question?", [included])) list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=included.pk),
),
)
assert captured_filters, "VectorIndexRetriever was never constructed" assert captured_filters, "VectorIndexRetriever was never constructed"
filt = captured_filters[0] filt = captured_filters[0]
@@ -340,3 +351,47 @@ class TestStreamChatRetrieval:
filter_values = filt.filters[0].value filter_values = filt.filters[0].value
assert str(included.pk) in filter_values assert str(included.pk) in filter_values
assert str(excluded.pk) not in filter_values assert str(excluded.pk) not in filter_values
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
django_assert_num_queries,
) -> None:
"""Building references must not hydrate every document the caller is
permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
the retriever actually returned nodes for.
"""
referenced = DocumentFactory.create(title="Referenced Document")
# Many more documents are "accessible" but never referenced by a node.
DocumentFactory.create_batch(200)
documents = Document.objects.all()
top_nodes = [
MagicMock(
metadata={
"document_id": str(referenced.pk),
"title": "Referenced Document",
},
),
]
hydrated_count = 0
def _count_hydration(sender, instance, **kwargs):
nonlocal hydrated_count
hydrated_count += 1
post_init.connect(_count_hydration, sender=Document)
try:
# One query: `documents.filter(pk__in=candidate_ids)` for the single
# referenced id. No query should scale with the 200 unreferenced documents.
with django_assert_num_queries(1):
references = chat._get_document_references(documents, top_nodes)
finally:
post_init.disconnect(_count_hydration, sender=Document)
# The bug this guards against: the old code hydrated all 201 accessible
# documents via `{doc.pk: doc for doc in documents}` before filtering by
# top_nodes. Only the referenced document should ever be constructed.
assert hydrated_count == 1
assert references == [{"id": referenced.pk, "title": "Referenced Document"}]
-27
View File
@@ -757,30 +757,3 @@ class TestAPIProcessedMails(DirectoriesMixin, APITestCase):
format="json", format="json",
) )
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
def test_bulk_delete_processed_mails_rejects_mixed_batch_atomically(self) -> None:
"""
GIVEN:
- A permitted processed mail and one the user may not delete
WHEN:
- API call bulk deletes both in a single request
THEN:
- The request is rejected and neither mail is deleted
"""
user2 = User.objects.create_user(username="temp_admin2")
rule = MailRuleFactory()
# Created first so it sorts ahead of the forbidden mail, i.e. the
# permission check has to cover the whole batch before deleting rather
# than rejecting only once it reaches the forbidden one.
pm_owned = ProcessedMailFactory(rule=rule, owner=self.user)
pm_forbidden = ProcessedMailFactory(rule=rule, owner=user2)
response = self.client.post(
f"{self.ENDPOINT}bulk_delete/",
data={"mail_ids": [pm_owned.id, pm_forbidden.id]},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
self.assertTrue(ProcessedMail.objects.filter(id=pm_owned.id).exists())
self.assertTrue(ProcessedMail.objects.filter(id=pm_forbidden.id).exists())
+4 -12
View File
@@ -27,7 +27,6 @@ from documents.filters import PermittedObjectsFilter
from documents.models import PaperlessTask from documents.models import PaperlessTask
from documents.permissions import PaperlessObjectPermissions from documents.permissions import PaperlessObjectPermissions
from documents.permissions import has_perms_owner_aware from documents.permissions import has_perms_owner_aware
from documents.permissions import permitted_object_ids
from documents.views import PassUserMixin from documents.views import PassUserMixin
from paperless.views import StandardPagination from paperless.views import StandardPagination
from paperless_mail.filters import ProcessedMailFilterSet from paperless_mail.filters import ProcessedMailFilterSet
@@ -212,17 +211,10 @@ class ProcessedMailViewSet(PassUserMixin, ReadOnlyModelViewSet[ProcessedMail]):
): ):
return HttpResponseBadRequest("mail_ids must be a list of integers") return HttpResponseBadRequest("mail_ids must be a list of integers")
mails = ProcessedMail.objects.filter(id__in=mail_ids) mails = ProcessedMail.objects.filter(id__in=mail_ids)
# Check every id up front so an unpermitted one rejects the whole for mail in mails:
# request rather than deleting the mails ahead of it first. if not has_perms_owner_aware(request.user, "delete_processedmail", mail):
if mails.exclude( return HttpResponseForbidden("Insufficient permissions")
pk__in=permitted_object_ids( mail.delete()
request.user,
ProcessedMail,
"delete_processedmail",
),
).exists():
return HttpResponseForbidden("Insufficient permissions")
mails.delete()
return Response({"result": "OK", "deleted_mail_ids": mail_ids}) return Response({"result": "OK", "deleted_mail_ids": mail_ids})
Generated
+22 -22
View File
@@ -1298,16 +1298,16 @@ wheels = [
[[package]] [[package]]
name = "fpdf2" name = "fpdf2"
version = "2.8.8" version = "2.8.7"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/1e/bc/8fd4321aed40cadadddc8f311c65b6082346b252bca048f7b476d8f35d72/fpdf2-2.8.8.tar.gz", hash = "sha256:9e94e155e85e8053329a9a1fce8b566fd7a7c5bb79e98a1a3952d379b947c5b9", size = 374689, upload-time = "2026-08-09T23:32:45.334Z" } sdist = { url = "https://files.pythonhosted.org/packages/27/f2/72feae0b2827ed38013e4307b14f95bf0b3d124adfef4d38a7d57533f7be/fpdf2-2.8.7.tar.gz", hash = "sha256:7060ccee5a9c7ab0a271fb765a36a23639f83ef8996c34e3d46af0a17ede57f9", size = 362351, upload-time = "2026-02-28T05:39:16.456Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/f5/be/af012eda9507494f28b99b077423806c43a11573eb6225dd46f19ae2d263/fpdf2-2.8.8-py3-none-any.whl", hash = "sha256:3557a478fc577a929c94aace9666aed4dcc432b5ab6764232e6a59f1ccd75f17", size = 337000, upload-time = "2026-08-09T23:32:43.728Z" }, { url = "https://files.pythonhosted.org/packages/66/0a/cf50ecffa1e3747ed9380a3adfc829259f1f86b3fdbd9e505af789003141/fpdf2-2.8.7-py3-none-any.whl", hash = "sha256:d391fc508a3ce02fc43a577c830cda4fe6f37646f2d143d489839940932fbc19", size = 327056, upload-time = "2026-02-28T05:39:14.619Z" },
] ]
[[package]] [[package]]
@@ -2927,8 +2927,8 @@ dependencies = [
{ name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" },
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
{ name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -4511,8 +4511,8 @@ dependencies = [
{ name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" },
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
{ name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -4957,17 +4957,18 @@ name = "torch"
version = "2.13.0" version = "2.13.0"
source = { registry = "https://download.pytorch.org/whl/cpu" } source = { registry = "https://download.pytorch.org/whl/cpu" }
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and sys_platform == 'darwin'",
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'", "python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
"python_full_version < '3.12' and sys_platform == 'darwin'", "python_full_version < '3.12' and sys_platform == 'darwin'",
] ]
dependencies = [ dependencies = [
{ name = "filelock", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "filelock", marker = "sys_platform == 'darwin'" },
{ name = "fsspec", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "fsspec", marker = "sys_platform == 'darwin'" },
{ name = "jinja2", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "jinja2", marker = "sys_platform == 'darwin'" },
{ name = "networkx", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "networkx", marker = "sys_platform == 'darwin'" },
{ name = "setuptools", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "setuptools", marker = "sys_platform == 'darwin'" },
{ name = "sympy", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "sympy", marker = "sys_platform == 'darwin'" },
{ name = "typing-extensions", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" }, { name = "typing-extensions", marker = "sys_platform == 'darwin'" },
] ]
wheels = [ wheels = [
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" }, { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" },
@@ -4982,7 +4983,6 @@ name = "torch"
version = "2.13.0+cpu" version = "2.13.0+cpu"
source = { registry = "https://download.pytorch.org/whl/cpu" } source = { registry = "https://download.pytorch.org/whl/cpu" }
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.15' and sys_platform == 'darwin'",
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'", "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'", "python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
"python_full_version >= '3.15' and sys_platform == 'linux'", "python_full_version >= '3.15' and sys_platform == 'linux'",
@@ -4990,13 +4990,13 @@ resolution-markers = [
"python_full_version < '3.12' and sys_platform == 'linux'", "python_full_version < '3.12' and sys_platform == 'linux'",
] ]
dependencies = [ dependencies = [
{ name = "filelock", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "filelock", marker = "sys_platform == 'linux'" },
{ name = "fsspec", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "fsspec", marker = "sys_platform == 'linux'" },
{ name = "jinja2", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "jinja2", marker = "sys_platform == 'linux'" },
{ name = "networkx", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "networkx", marker = "sys_platform == 'linux'" },
{ name = "setuptools", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "setuptools", marker = "sys_platform == 'linux'" },
{ name = "sympy", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "sympy", marker = "sys_platform == 'linux'" },
{ name = "typing-extensions", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" }, { name = "typing-extensions", marker = "sys_platform == 'linux'" },
] ]
wheels = [ wheels = [
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" }, { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" },