Compare commits

...
Author SHA1 Message Date
dependabot[bot] 47288d4559 Chore(deps): Bump the pre-commit-dependencies group with 3 updates
Bumps the pre-commit-dependencies group with 3 updates: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit), [https://github.com/tox-dev/pyproject-fmt](https://github.com/tox-dev/pyproject-fmt) and [https://github.com/AleksaC/hadolint-py](https://github.com/AleksaC/hadolint-py).


Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.16.1 to 0.16.4
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](https://github.com/astral-sh/ruff-pre-commit/compare/v0.16.1...v0.16.4)

Updates `https://github.com/tox-dev/pyproject-fmt` from v2.26.0 to 2.28.1
- [Release notes](https://github.com/tox-dev/pyproject-fmt/releases)
- [Commits](https://github.com/tox-dev/pyproject-fmt/compare/v2.26.0...v2.28.1)

Updates `https://github.com/AleksaC/hadolint-py` from v2.14.0 to 2.15.1
- [Commits](https://github.com/AleksaC/hadolint-py/compare/v2.14.0...v2.15.1)

---
updated-dependencies:
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.16.4
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/tox-dev/pyproject-fmt
  dependency-version: 2.28.1
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/AleksaC/hadolint-py
  dependency-version: 2.15.1
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-01 20:23:31 +00:00
Trenton H d78754bff1 Security: validate remote OCR endpoint against internal SSRF (#13897)
* Security: validate remote OCR endpoint against internal SSRF

Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.

* Validates te outbound url again right before use

* cover empty-value branch of validate_remote_ocr_endpoint because coverage

* re-validate remote OCR endpoint on every outbound request
2026-09-01 20:22:10 +00:00
shamoon 5c5b1ee6b5 Fix: fix slim sidebar saved view dragging appearance (#13906) 2026-09-01 13:02:57 -07:00
GitHub Actions 08f2f4bfe2 Auto translate strings 2026-09-01 19:54:47 +00:00
Trenton H f993462973 Security: Minor additional hardening (#13898)
* Security: bump jinja2 floor to 3.1.6 (CVE-2025-27516)

* Security: anchor the /share/ URL pattern

* Security: handle missing file on public share view without 500

* Security: scope correspondent last_correspondence to permitted documents

* Security: disable PUT/PATCH on share link bundles
2026-09-01 19:53:28 +00:00
shamoon ae70b8d60f Chore: consolidate pickle hmac signing (#13899) 2026-09-01 12:41:45 -07:00
shamoon 38db6b51db Fix: use signal-backed queries input in CF dropdown to reflect changes immediately under zoneless (#13901) 2026-09-01 11:52:53 -07:00
GitHub Actions 31e9f4272c Auto translate strings 2026-09-01 16:56:33 +00:00
shamoon b8659c1af3 Fix: use root doc metadata for filename generation (#13893) 2026-09-01 09:55:04 -07:00
shamoon 741115b36b Fix: some css cleanup (#13891) 2026-09-01 09:17:27 -07:00
github-actions[bot] 1211db5cbb Documentation: Add v3.1.2 changelog (#13890) 2026-09-01 08:33:32 -07:00
30 changed files with 829 additions and 444 deletions
+3 -3
View File
@@ -50,18 +50,18 @@ repos:
- 'prettier-plugin-organize-imports@4.3.0'
# Python hooks
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.1
rev: v0.16.4
hooks:
- id: ruff-check
- id: ruff-format
- repo: https://github.com/tox-dev/pyproject-fmt
rev: "v2.26.0"
rev: "v2.28.1"
hooks:
- id: pyproject-fmt
additional_dependencies: [tomli]
# Dockerfile hooks
- repo: https://github.com/AleksaC/hadolint-py
rev: v2.14.0
rev: v2.15.1
hooks:
- id: hadolint
# Shell script hooks
+21
View File
@@ -1,5 +1,26 @@
# Changelog
## paperless-ngx 3.1.2
### Bug Fixes
- Fix: fix dark mode select disabled color, ensure disabled cursor on display mode dropdown [@shamoon](https://github.com/shamoon) ([#13881](https://github.com/paperless-ngx/paperless-ngx/pull/13881))
- Fix: add disable to the drag-drop list component [@shamoon](https://github.com/shamoon) ([#13880](https://github.com/paperless-ngx/paperless-ngx/pull/13880))
### Documentation
- Chore: update screenshots for v3+ [@shamoon](https://github.com/shamoon) ([#13883](https://github.com/paperless-ngx/paperless-ngx/pull/13883))
### All App Changes
<details>
<summary>2 changes</summary>
- Fix: fix dark mode select disabled color, ensure disabled cursor on display mode dropdown [@shamoon](https://github.com/shamoon) ([#13881](https://github.com/paperless-ngx/paperless-ngx/pull/13881))
- Fix: add disable to the drag-drop list component [@shamoon](https://github.com/shamoon) ([#13880](https://github.com/paperless-ngx/paperless-ngx/pull/13880))
</details>
## paperless-ngx 3.1.1
### Bug Fixes
+6
View File
@@ -2088,6 +2088,12 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to "always".
#### [`PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=<bool>`](#PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS) {#PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS}
: If set to false, Paperless blocks remote OCR endpoint URLs that resolve to non-public addresses (e.g., localhost, etc).
Defaults to True.
## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
+1 -1
View File
@@ -47,7 +47,7 @@ dependencies = [
"httpx-oauth~=0.17",
"ijson>=3.5.1",
"imap-tools~=1.14.0",
"jinja2~=3.1.5",
"jinja2~=3.1.6",
"langdetect~=1.0.9",
"llama-index-core>=0.14.23",
"llama-index-embeddings-huggingface>=0.6.1",
@@ -111,7 +111,7 @@
</h6>
<ul class="nav flex-column mb-2" cdkDropList (cdkDropListDropped)="onDrop($event)">
@for (view of savedViewService.sidebarViews; track view.id) {
<li class="nav-item w-100 app-link" cdkDrag [cdkDragDisabled]="!settingsService.organizingSidebarSavedViews() || !canSaveSettings"
<li class="nav-item app-link" cdkDrag [cdkDragDisabled]="!settingsService.organizingSidebarSavedViews() || !canSaveSettings"
cdkDragPreviewContainer="parent" cdkDragPreviewClass="navItemDrag" (cdkDragStarted)="onDragStart($event)"
(cdkDragEnded)="onDragEnd($event)">
<a class="nav-link" routerLink="view/{{view.id}}"
@@ -128,7 +128,7 @@
}
</a>
@if (settingsService.organizingSidebarSavedViews() && canSaveSettings) {
<div class="position-absolute end-0 top-0 px-3 py-2" [class.me-n3]="slimSidebarEnabled" cdkDragHandle>
<div class="position-absolute end-0 top-0 px-1 py-2" [class.me-n2]="slimSidebarEnabled" cdkDragHandle>
<i-bs name="grip-vertical"></i-bs>
</div>
}
@@ -332,7 +332,7 @@
</li>
<li class="nav-item" [class.visually-hidden]="slimSidebarEnabled">
<div class="text-muted small d-flex align-items-center flex-wrap nav-label">
<div class="me-3">
<div class="me-2">
<a class="text-muted text-decoration-none" target="_blank" rel="noopener noreferrer"
href="https://github.com/paperless-ngx/paperless-ngx" ngbPopover="GitHub" i18n-ngbPopover
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
@@ -341,7 +341,7 @@
</a>
</div>
@if (!settingsService.updateCheckingIsSet || appRemoteVersion()) {
<div class="version-check">
<div class="version-check d-flex align-items-center">
<ng-template #updateAvailablePopContent>
<span class="small">Paperless-ngx {{ appRemoteVersion().version }} <ng-container i18n>is
available.</ng-container><br /><ng-container i18n>Click to view.</ng-container></span>
@@ -1,6 +1,6 @@
@if (useDropdown) {
<div class="btn-group w-100" role="group" ngbDropdown #dropdown="ngbDropdown" (openChange)="onOpenChange($event)" [popperOptions]="popperOptions">
<button class="btn btn-sm btn-outline-primary" id="dropdown_toggle" ngbDropdownToggle [disabled]="disabled" [aria-label]="title">
<button class="btn btn-sm" [ngClass]="!editing && isActive ? 'btn-primary' : 'btn-outline-primary'" id="dropdown_toggle" ngbDropdownToggle [disabled]="disabled" [aria-label]="title">
<i-bs name="{{icon}}"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div>
@if (isActive) {
<pngx-clearable-badge [selected]="isActive" (cleared)="reset()"></pngx-clearable-badge>
@@ -1,5 +1,6 @@
import {
getLocaleNumberSymbol,
NgClass,
NgTemplateOutlet,
NumberSymbol,
} from '@angular/common'
@@ -48,25 +49,26 @@ import { ClearableBadgeComponent } from '../clearable-badge/clearable-badge.comp
import { DocumentLinkComponent } from '../input/document-link/document-link.component'
export class CustomFieldQueriesModel {
private _queries: CustomFieldQueryElement[] = []
private readonly _queries = signal<CustomFieldQueryElement[]>([])
private rootSubscriptions: Subscription[] = []
public readonly changed = new Subject<CustomFieldQueriesModel>()
public get queries(): CustomFieldQueryElement[] {
return this._queries
return this._queries()
}
public set queries(value: CustomFieldQueryElement[]) {
this.teardownRootSubscriptions()
this._queries = value ?? []
for (const element of this._queries) {
const queries = value ?? []
for (const element of queries) {
this.rootSubscriptions.push(
element.changed.subscribe(() => {
this.changed.next(this)
})
)
}
this._queries.set(queries)
}
public clear(fireEvent = true) {
@@ -209,6 +211,7 @@ export class CustomFieldQueriesModel {
DocumentLinkComponent,
ReactiveFormsModule,
NgbDatepickerModule,
NgClass,
NgTemplateOutlet,
NgSelectModule,
NgxBootstrapIconsModule,
@@ -116,7 +116,7 @@
</pngx-page-header>
<div class="row sticky-top py-3 mt-n2 mt-md-n3 bg-body">
<div class="row sticky-top py-3 mt-n2 mt-md-n3 bg-body rounded shadow-sm">
<pngx-filter-editor [hidden]="isBulkEditing" [disabled]="isBulkEditing" [filterRules]="list.filterRules" (filterRulesChange)="onFilterRulesChange($event)" (resetFilterRules)="onFilterRulesReset($event)" [unmodifiedFilterRules]="unmodifiedFilterRules()" [selectionData]="list.selectionData" #filterEditor></pngx-filter-editor>
<pngx-bulk-editor [hidden]="!isBulkEditing" [disabled]="!isBulkEditing"></pngx-bulk-editor>
</div>
@@ -1034,6 +1034,49 @@ describe('FilterEditorComponent', () => {
).toEqual([42, CustomFieldQueryOperator.Exists, 'true'])
})
it('should reflect ingested custom field query rules in the dropdown toggle', () => {
const dropdown = fixture.debugElement.query(
By.css('pngx-custom-fields-query-dropdown')
)
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).toBeNull()
// switching to a view with a custom field query
component.filterRules = [
{
rule_type: FILTER_CUSTOM_FIELDS_QUERY,
value: '["OR",[[42,"exists","true"]]]',
},
]
fixture.detectChanges()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).not.toBeNull()
expect(
dropdown.nativeElement
.querySelector('#dropdown_toggle')
.classList.contains('btn-primary')
).toBeTruthy()
// and back to a view without one
component.filterRules = [
{
rule_type: FILTER_HAS_TAGS_ALL,
value: '19',
},
]
fixture.detectChanges()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).toBeNull()
expect(
dropdown.nativeElement
.querySelector('#dropdown_toggle')
.classList.contains('btn-primary')
).toBeFalsy()
})
it('should ingest filter rules for owner', () => {
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NONE
+5 -16
View File
@@ -47,6 +47,8 @@ $grid-breakpoints: (
);
:root {
--bs-border-radius: #{$border-radius};
@each $name, $value in $grid-breakpoints {
--bs-breakpoint-#{$name}: #{$value};
}
@@ -78,19 +80,12 @@ body {
}
.btn {
--bs-btn-border-radius: .425rem;
--bs-border-radius-sm: .425rem;
--bs-border-radius-sm: #{$border-radius};
font-weight: 500;
}
.form-control,
.form-select,
.input-group-text {
border-radius: .425rem;
}
.pagination, .input-group {
--bs-border-radius-sm: .425rem;
--bs-border-radius-sm: #{$border-radius};
}
@media(min-width: 768px) {
@@ -689,10 +684,6 @@ table.table {
--bs-toast-max-width: var(--pngx-toast-max-width);
}
.alert {
--bs-border-radius: .425rem;
}
.alert-primary {
--bs-alert-color: var(--bs-primary);
--bs-alert-bg: var(--pngx-primary-faded);
@@ -824,8 +815,6 @@ code {
--bs-accordion-bg: var(--bs-light);
--bs-accordion-active-color: var(--bs-primary);
--bs-accordion-active-bg: var(--pngx-bg-alt);
--bs-border-radius: .425rem;
--bs-accordion-inner-border-radius: calc(.425rem - 1px);
}
.accordion-button::after {
@@ -849,7 +838,7 @@ code {
}
/* Animate items as they're being sorted. */
.cdk-drop-list-dragging .cdk-drag {
.cdk-drop-list-dragging .cdk-drag:not(.cdk-drag-preview) {
transition: transform 250ms cubic-bezier(0, 0, 0.2, 1);
}
+1
View File
@@ -113,6 +113,7 @@ $form-check-radio-checked-bg-image-dark: url("data:image/svg+xml,%3csvg xmlns='h
--bs-tertiary-bg: var(--pngx-bg-darker);
--bs-dark-border-subtle: var(--pngx-bg-darker);
--bs-border-color-translucent: rgba(0, 0, 0, .175); // override bs
--bs-box-shadow-sm: 0 0.125rem 0.25rem rgba(0, 0, 0, 0.15); // slightly darker than bs default
.text-dark, .text-light {
color: var(--bs-body-color) !important;
+8 -3
View File
@@ -16,6 +16,9 @@ from django.core.cache import cache
from django.core.cache import caches
from documents.models import Document
from paperless.signed_pickle import SignedPickleError
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
if TYPE_CHECKING:
from django.core.cache.backends.base import BaseCache
@@ -118,9 +121,11 @@ class StoredLRUCache(LRUCache):
serialized_data = self._backend.get(self._backend_key)
try:
self._data = (
pickle.loads(serialized_data) if serialized_data else OrderedDict()
signed_pickle_loads(serialized_data)
if serialized_data
else OrderedDict()
)
except pickle.PickleError:
except (SignedPickleError, pickle.PickleError):
logger.warning(
"Cache exists in backend but could not be read (possibly invalid format)",
)
@@ -132,7 +137,7 @@ class StoredLRUCache(LRUCache):
"""
self._backend.set(
self._backend_key,
pickle.dumps(self._data),
signed_pickle_dumps(self._data),
self.backend_ttl,
)
+13 -3
View File
@@ -28,6 +28,9 @@ from documents.caching import CLASSIFIER_VERSION_KEY
from documents.caching import StoredLRUCache
from documents.models import Document
from documents.models import MatchingModel
from paperless.signed_pickle import SignedPickleError
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
logger = logging.getLogger("paperless.classifier")
@@ -527,10 +530,17 @@ class DocumentClassifier:
serialized_result = read_cache.get(key)
if serialized_result is None:
result = self.data_vectorizer.transform([self.preprocess_content(content)])
read_cache.set(key, pickle.dumps(result), CACHE_5_MINUTES)
read_cache.set(key, signed_pickle_dumps(result), CACHE_5_MINUTES)
else:
read_cache.touch(key, CACHE_5_MINUTES)
result = pickle.loads(serialized_result)
try:
result = signed_pickle_loads(serialized_result)
except SignedPickleError:
result = self.data_vectorizer.transform(
[self.preprocess_content(content)],
)
read_cache.set(key, signed_pickle_dumps(result), CACHE_5_MINUTES)
else:
read_cache.touch(key, CACHE_5_MINUTES)
return result
def predict_correspondent(self, content: str) -> int | None:
+5 -1
View File
@@ -462,7 +462,11 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
"""
Returns a sanitized filename for the document, not including any paths.
"""
result = str(self)
# Root owns metadata for all versions
context_document = (
self.root_document if self.root_document_id is not None else self
)
result = str(context_document)
if counter:
result += f"_{counter:02}"
@@ -1063,3 +1063,79 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("non-public address", str(response.data).lower())
@override_settings(REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=False)
def test_update_remote_ocr_endpoint_blocks_internal_endpoint_when_disallowed(
self,
) -> None:
"""
GIVEN:
- Internal remote OCR endpoints are disallowed
WHEN:
- The config is updated with a remote OCR endpoint resolving internally
THEN:
- The request is rejected
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_endpoint": "http://127.0.0.1:5000",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("non-public address", str(response.data).lower())
@override_settings(REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=True)
def test_update_remote_ocr_endpoint_allows_internal_endpoint_by_default(
self,
) -> None:
"""
GIVEN:
- Internal remote OCR endpoints are allowed (the default)
WHEN:
- The config is updated with a remote OCR endpoint resolving internally
THEN:
- The request is accepted, preserving existing self-hosted deployments
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_endpoint": "http://127.0.0.1:5000",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
response.data["remote_ocr_endpoint"],
"http://127.0.0.1:5000",
)
@override_settings(REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=False)
def test_update_remote_ocr_endpoint_empty_value_skips_validation(
self,
) -> None:
"""
GIVEN:
- Internal remote OCR endpoints are disallowed
WHEN:
- The config is updated with an empty remote OCR endpoint
THEN:
- The request is accepted; clearing the field never needs
outbound URL validation
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_endpoint": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["remote_ocr_endpoint"], "")
+27
View File
@@ -102,6 +102,7 @@ class TestApiObjects(DirectoriesMixin, APITestCase):
- API is called
THEN:
- Last correspondence date is returned only if requested for list, and for detail
- The date is scoped to documents the requesting user may view
"""
Document.objects.create(
@@ -145,6 +146,32 @@ class TestApiObjects(DirectoriesMixin, APITestCase):
response.data["last_correspondence"],
)
# A newer document owned by another user must not leak through the
# aggregate for a non-superuser who cannot view it
other = User.objects.create_user(username="other")
Document.objects.create(
mime_type="application/pdf",
correspondent=self.c1,
created=datetime.date(2023, 6, 1),
checksum="hidden",
owner=other,
)
user = User.objects.create_user(username="regular")
user.user_permissions.add(
Permission.objects.get(codename="view_correspondent"),
)
self.client.force_authenticate(user=user)
response = self.client.get("/api/correspondents/?last_correspondence=true")
self.assertEqual(response.status_code, status.HTTP_200_OK)
result = next(r for r in response.data["results"] if r["id"] == self.c1.id)
self.assertIn("2022-01-02", result["last_correspondence"])
response = self.client.get(f"/api/correspondents/{self.c1.id}/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertIn("2022-01-02", response.data["last_correspondence"])
def test_paginated_objects_include_all_only_for_legacy_version(self) -> None:
response_v10 = self.client.get("/api/correspondents/")
self.assertEqual(response_v10.status_code, status.HTTP_200_OK)
+16 -3
View File
@@ -1,6 +1,7 @@
import pickle
from documents.caching import StoredLRUCache
from paperless.signed_pickle import HMAC_SIZE
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
def test_lru_cache_entries() -> None:
@@ -42,4 +43,16 @@ def test_stored_lru_cache_key_ttl(mocker) -> None:
key, data, timeout = mock_backend.set.call_args[0]
assert key == "test_key"
assert timeout == 321
assert pickle.loads(data) == {"x": "X", "y": "Y"}
assert signed_pickle_loads(data) == {"x": "X", "y": "Y"}
def test_stored_lru_cache_rejects_tampered_data(mocker) -> None:
serialized_data = bytearray(signed_pickle_dumps({"x": "X"}))
serialized_data[HMAC_SIZE] ^= 0xFF
mock_backend = mocker.Mock()
mock_backend.get.return_value = bytes(serialized_data)
cache = StoredLRUCache("test_key", backend=mock_backend)
cache.load()
assert cache.get("x") is None
+23
View File
@@ -19,6 +19,8 @@ from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.utils import DirectoriesMixin
from paperless.signed_pickle import HMAC_SIZE
from paperless.signed_pickle import signed_pickle_dumps
def dummy_preprocess(content: str, **kwargs):
@@ -265,6 +267,27 @@ class TestClassifier(DirectoriesMixin, TestCase):
self.assertEqual(mock_preprocess_content.call_count, 2)
self.assertEqual(mock_transform.call_count, 2)
def test_vectorize_recomputes_tampered_cache_entry(self) -> None:
cached = bytearray(signed_pickle_dumps(["cached vector"]))
cached[HMAC_SIZE] ^= 0xFF
self.classifier.data_vectorizer = mock.Mock()
self.classifier.data_vectorizer.transform.return_value = ["fresh vector"]
with (
mock.patch(
"documents.classifier.read_cache.get",
return_value=bytes(cached),
),
mock.patch("documents.classifier.read_cache.set") as cache_set,
mock.patch("documents.classifier.read_cache.touch") as cache_touch,
):
result = self.classifier._vectorize("content")
self.assertEqual(result, ["fresh vector"])
self.classifier.data_vectorizer.transform.assert_called_once()
cache_set.assert_called_once()
cache_touch.assert_not_called()
def test_no_retrain_if_no_change(self) -> None:
"""
GIVEN:
@@ -156,6 +156,40 @@ class TestDocument(TestCase):
)
self.assertEqual(doc.get_public_filename(), "2020-12-25 test")
def test_version_file_name_uses_root_document_metadata(self) -> None:
root_correspondent = Correspondent.objects.create(name="Root correspondent")
version_correspondent = Correspondent.objects.create(
name="Version correspondent",
)
root = Document.objects.create(
mime_type="application/pdf",
title="Root title",
created=date(2020, 12, 25),
correspondent=root_correspondent,
)
version = Document.objects.create(
mime_type="application/pdf",
title="Version title",
created=date(1990, 1, 1),
correspondent=version_correspondent,
root_document=root,
version_index=1,
)
self.assertEqual(
version.get_public_filename(),
"2020-12-25 Root correspondent Root title.pdf",
)
root.title = "Updated root title"
root.save(update_fields=("title",))
version.refresh_from_db()
self.assertEqual(
version.get_public_filename(),
"2020-12-25 Root correspondent Updated root title.pdf",
)
def test_suggestion_content_uses_latest_version_content_for_root_documents(
self,
) -> None:
@@ -192,6 +192,50 @@ class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_302_FOUND)
self.assertIn("sharelink_notfound=1", response["Location"])
def test_share_link_missing_file_redirects(self) -> None:
"""
GIVEN:
- A share link whose document file is missing from disk
WHEN:
- The public share link is requested anonymously
THEN:
- The user is redirected to login instead of a 500 error
"""
doc = DocumentFactory.create(filename="missing-original.pdf")
share_link = ShareLink.objects.create(
slug="missingfilelink",
document=doc,
file_version=ShareLink.FileVersion.ORIGINAL,
)
self.client.logout()
response = self.client.get(f"/share/{share_link.slug}/")
self.assertEqual(response.status_code, status.HTTP_302_FOUND)
self.assertIn("sharelink_notfound=1", response["Location"])
def test_download_ready_bundle_missing_file_returns_503(self) -> None:
"""
GIVEN:
- A READY bundle whose zip file is missing from disk
WHEN:
- The public share link is requested anonymously
THEN:
- A 503 is returned instead of a 500 error
"""
bundle = ShareLinkBundle.objects.create(
slug="missingbundlefile",
file_version=ShareLink.FileVersion.ARCHIVE,
status=ShareLinkBundle.Status.READY,
file_path="bundles/gone.zip",
)
bundle.documents.set([self.document])
self.client.logout()
response = self.client.get(f"/share/{bundle.slug}/")
self.assertEqual(response.status_code, status.HTTP_503_SERVICE_UNAVAILABLE)
class ShareLinkBundleTaskTests(DirectoriesMixin, APITestCase):
def setUp(self) -> None:
+26 -9
View File
@@ -577,13 +577,19 @@ class CorrespondentViewSet(
def list(self, request, *args, **kwargs):
if request.query_params.get("last_correspondence", None):
self.queryset = self.queryset.annotate(
last_correspondence=Max("documents__created"),
last_correspondence=Max(
"documents__created",
filter=self.get_document_count_filter(),
),
)
return super().list(request, *args, **kwargs)
def retrieve(self, request, *args, **kwargs):
self.queryset = self.queryset.annotate(
last_correspondence=Max("documents__created"),
last_correspondence=Max(
"documents__created",
filter=self.get_document_count_filter(),
),
)
return super().retrieve(request, *args, **kwargs)
@@ -4573,6 +4579,10 @@ class ShareLinkViewSet(
class ShareLinkBundleViewSet(PassUserMixin, ModelViewSet[ShareLinkBundle]):
model = ShareLinkBundle
# Bundles are immutable once created; rebuild via the dedicated action
# rather than PUT/PATCH.
http_method_names = ["get", "post", "delete", "head", "options"]
queryset = ShareLinkBundle.objects.all()
serializer_class = ShareLinkBundleSerializer
@@ -4707,12 +4717,15 @@ class SharedLinkView(View):
and share_link.expiration < timezone.now()
):
return HttpResponseRedirect("/accounts/login/?sharelink_expired=1")
return serve_file(
doc=share_link.document,
use_archive=share_link.file_version == ShareLink.FileVersion.ARCHIVE
and share_link.document.has_archive_version,
disposition="inline",
)
try:
return serve_file(
doc=share_link.document,
use_archive=share_link.file_version == ShareLink.FileVersion.ARCHIVE
and share_link.document.has_archive_version,
disposition="inline",
)
except FileNotFoundError:
return HttpResponseRedirect("/accounts/login/?sharelink_notfound=1")
bundle = ShareLinkBundle.objects.filter(slug=slug).first()
if bundle is None:
@@ -4734,7 +4747,11 @@ class SharedLinkView(View):
file_path = bundle.absolute_file_path
if bundle.status == ShareLinkBundle.Status.FAILED or file_path is None:
if (
bundle.status == ShareLinkBundle.Status.FAILED
or file_path is None
or not file_path.exists()
):
return HttpResponse(
_(
"The share link bundle is unavailable.",
File diff suppressed because it is too large Load Diff
+3 -31
View File
@@ -1,12 +1,12 @@
import hmac
import os
import pickle
from hashlib import sha256
from celery import Celery
from celery.signals import worker_process_init
from kombu.serialization import register
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "paperless.settings")
@@ -18,34 +18,6 @@ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "paperless.settings")
# on the worker side using Django's SECRET_KEY.
# ---------------------------------------------------------------------------
HMAC_SIZE = 32 # SHA-256 digest length
def _get_signing_key() -> bytes:
from django.conf import settings
return settings.SECRET_KEY.encode()
def signed_pickle_dumps(obj: object) -> bytes:
data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
signature = hmac.new(_get_signing_key(), data, sha256).digest()
return signature + data
def signed_pickle_loads(payload: bytes) -> object:
if len(payload) < HMAC_SIZE:
msg = "Signed-pickle payload too short"
raise ValueError(msg)
signature = payload[:HMAC_SIZE]
data = payload[HMAC_SIZE:]
expected = hmac.new(_get_signing_key(), data, sha256).digest()
if not hmac.compare_digest(signature, expected):
msg = "Signed-pickle HMAC verification failed — message may have been tampered with"
raise ValueError(msg)
return pickle.loads(data)
register(
"signed-pickle",
signed_pickle_dumps,
+38
View File
@@ -32,6 +32,8 @@ if TYPE_CHECKING:
import datetime
from types import TracebackType
from azure.core.pipeline import PipelineRequest
from paperless.parsers import MetadataEntry
from paperless.parsers import ParserContext
@@ -436,9 +438,45 @@ class RemoteDocumentParser:
from azure.ai.documentintelligence.models import DocumentContentFormat
from azure.core.credentials import AzureKeyCredential
from paperless.network import validate_outbound_http_url
allow_internal = settings.REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS
try:
validate_outbound_http_url(config.endpoint, allow_internal=allow_internal)
except ValueError as e:
raise ParseError(f"Invalid remote OCR endpoint: {e}") from e
def _revalidate_request_host(request: PipelineRequest) -> None:
"""Re-validates the destination host of every request sent.
The check above only covers the moment the client is built. A
single analysis involves several requests spread over the
polling loop below, and any one of them can be redirected.
Wiring this through ``raw_request_hook`` (Azure's built-in
CustomHookPolicy) rather than a custom policy means it runs
*after* RedirectPolicy in the pipeline, so it sees - and
re-checks - every actual outbound URL, including redirect
targets, not just the original request.
"""
validate_outbound_http_url(
request.http_request.url,
allow_internal=allow_internal,
)
client = DocumentIntelligenceClient(
endpoint=config.endpoint,
credential=AzureKeyCredential(config.api_key),
raw_request_hook=_revalidate_request_host,
# AzureKeyCredential is sent as Ocp-Apim-Subscription-Key, which
# Azure's default SensitiveHeaderCleanupPolicy does not strip on
# a cross-domain redirect (only Authorization and
# x-ms-authorization-auxiliary are, by default).
blocked_redirect_headers=[
"Authorization",
"x-ms-authorization-auxiliary",
"Ocp-Apim-Subscription-Key",
],
)
try:
+16
View File
@@ -305,6 +305,22 @@ class ApplicationConfigurationSerializer(
validate_llm_embedding_endpoint = validate_llm_endpoint
def validate_remote_ocr_endpoint(self, value: str | None) -> str | None:
if not value:
return value
try:
validate_outbound_http_url(
value,
allow_internal=settings.REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS,
)
except ValueError as e:
raise serializers.ValidationError(
f"Invalid remote OCR endpoint: {e.args[0]}, see logs for details",
) from e
return value
class Meta:
model = ApplicationConfiguration
fields = "__all__"
+4
View File
@@ -1208,6 +1208,10 @@ REMOTE_OCR_MODE = get_choice_from_env(
{"always", "workflow_only"},
default="always",
)
REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS = get_bool_from_env(
"PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS",
"true",
)
################################################################################
# AI Settings #
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
import hmac
import pickle
from hashlib import sha256
from typing import Any
from django.conf import settings
HMAC_SIZE = sha256().digest_size
class SignedPickleError(ValueError):
"""Raised when a signed pickle payload cannot be authenticated."""
def _get_signing_key() -> bytes:
return settings.SECRET_KEY.encode()
def signed_pickle_dumps(obj: object) -> bytes:
data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
signature = hmac.new(_get_signing_key(), data, sha256).digest()
return signature + data
def signed_pickle_loads(payload: bytes) -> Any:
if len(payload) <= HMAC_SIZE:
msg = "Signed-pickle payload too short"
raise SignedPickleError(msg)
signature = payload[:HMAC_SIZE]
data = payload[HMAC_SIZE:]
expected = hmac.new(_get_signing_key(), data, sha256).digest()
if not hmac.compare_digest(signature, expected):
msg = "Signed-pickle HMAC verification failed; payload may have been tampered with"
raise SignedPickleError(msg)
return pickle.loads(data)
+1 -1
View File
@@ -6,9 +6,9 @@ from pathlib import Path
import pytest
from django.test import override_settings
from paperless.celery import HMAC_SIZE
from paperless.celery import signed_pickle_dumps
from paperless.celery import signed_pickle_loads
from paperless.signed_pickle import HMAC_SIZE
class TestSignedPickleSerializer:
+1 -1
View File
@@ -295,7 +295,7 @@ urlpatterns = [
],
),
),
re_path(r"share/(?P<slug>\w+)/?$", SharedLinkView.as_view()),
re_path(r"^share/(?P<slug>\w+)/?$", SharedLinkView.as_view()),
re_path(r"^favicon.ico$", FaviconView.as_view(), name="favicon"),
re_path(r"admin/", admin.site.urls),
re_path(
Generated
+1 -1
View File
@@ -3053,7 +3053,7 @@ requires-dist = [
{ name = "httpx-oauth", specifier = "~=0.17" },
{ name = "ijson", specifier = ">=3.5.1" },
{ name = "imap-tools", specifier = "~=1.14.0" },
{ name = "jinja2", specifier = "~=3.1.5" },
{ name = "jinja2", specifier = "~=3.1.6" },
{ name = "langdetect", specifier = "~=1.0.9" },
{ name = "llama-index-core", specifier = ">=0.14.23" },
{ name = "llama-index-embeddings-huggingface", specifier = ">=0.6.1" },