Compare commits

...
Author SHA1 Message Date
dependabot[bot] 47288d4559 Chore(deps): Bump the pre-commit-dependencies group with 3 updates
Bumps the pre-commit-dependencies group with 3 updates: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit), [https://github.com/tox-dev/pyproject-fmt](https://github.com/tox-dev/pyproject-fmt) and [https://github.com/AleksaC/hadolint-py](https://github.com/AleksaC/hadolint-py).


Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.16.1 to 0.16.4
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](https://github.com/astral-sh/ruff-pre-commit/compare/v0.16.1...v0.16.4)

Updates `https://github.com/tox-dev/pyproject-fmt` from v2.26.0 to 2.28.1
- [Release notes](https://github.com/tox-dev/pyproject-fmt/releases)
- [Commits](https://github.com/tox-dev/pyproject-fmt/compare/v2.26.0...v2.28.1)

Updates `https://github.com/AleksaC/hadolint-py` from v2.14.0 to 2.15.1
- [Commits](https://github.com/AleksaC/hadolint-py/compare/v2.14.0...v2.15.1)

---
updated-dependencies:
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.16.4
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/tox-dev/pyproject-fmt
  dependency-version: 2.28.1
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/AleksaC/hadolint-py
  dependency-version: 2.15.1
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-01 20:23:31 +00:00
Trenton H d78754bff1 Security: validate remote OCR endpoint against internal SSRF (#13897)
* Security: validate remote OCR endpoint against internal SSRF

Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.

* Validates te outbound url again right before use

* cover empty-value branch of validate_remote_ocr_endpoint because coverage

* re-validate remote OCR endpoint on every outbound request
2026-09-01 20:22:10 +00:00
shamoon 5c5b1ee6b5 Fix: fix slim sidebar saved view dragging appearance (#13906) 2026-09-01 13:02:57 -07:00
GitHub Actions 08f2f4bfe2 Auto translate strings 2026-09-01 19:54:47 +00:00
Trenton H f993462973 Security: Minor additional hardening (#13898)
* Security: bump jinja2 floor to 3.1.6 (CVE-2025-27516)

* Security: anchor the /share/ URL pattern

* Security: handle missing file on public share view without 500

* Security: scope correspondent last_correspondence to permitted documents

* Security: disable PUT/PATCH on share link bundles
2026-09-01 19:53:28 +00:00
shamoon ae70b8d60f Chore: consolidate pickle hmac signing (#13899) 2026-09-01 12:41:45 -07:00
shamoon 38db6b51db Fix: use signal-backed queries input in CF dropdown to reflect changes immediately under zoneless (#13901) 2026-09-01 11:52:53 -07:00
GitHub Actions 31e9f4272c Auto translate strings 2026-09-01 16:56:33 +00:00
shamoon b8659c1af3 Fix: use root doc metadata for filename generation (#13893) 2026-09-01 09:55:04 -07:00
shamoon 741115b36b Fix: some css cleanup (#13891) 2026-09-01 09:17:27 -07:00
github-actions[bot] 1211db5cbb Documentation: Add v3.1.2 changelog (#13890) 2026-09-01 08:33:32 -07:00
30 changed files with 829 additions and 444 deletions
+3 -3
View File
@@ -50,18 +50,18 @@ repos:
- 'prettier-plugin-organize-imports@4.3.0' - 'prettier-plugin-organize-imports@4.3.0'
# Python hooks # Python hooks
- repo: https://github.com/astral-sh/ruff-pre-commit - repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.1 rev: v0.16.4
hooks: hooks:
- id: ruff-check - id: ruff-check
- id: ruff-format - id: ruff-format
- repo: https://github.com/tox-dev/pyproject-fmt - repo: https://github.com/tox-dev/pyproject-fmt
rev: "v2.26.0" rev: "v2.28.1"
hooks: hooks:
- id: pyproject-fmt - id: pyproject-fmt
additional_dependencies: [tomli] additional_dependencies: [tomli]
# Dockerfile hooks # Dockerfile hooks
- repo: https://github.com/AleksaC/hadolint-py - repo: https://github.com/AleksaC/hadolint-py
rev: v2.14.0 rev: v2.15.1
hooks: hooks:
- id: hadolint - id: hadolint
# Shell script hooks # Shell script hooks
+21
View File
@@ -1,5 +1,26 @@
# Changelog # Changelog
## paperless-ngx 3.1.2
### Bug Fixes
- Fix: fix dark mode select disabled color, ensure disabled cursor on display mode dropdown [@shamoon](https://github.com/shamoon) ([#13881](https://github.com/paperless-ngx/paperless-ngx/pull/13881))
- Fix: add disable to the drag-drop list component [@shamoon](https://github.com/shamoon) ([#13880](https://github.com/paperless-ngx/paperless-ngx/pull/13880))
### Documentation
- Chore: update screenshots for v3+ [@shamoon](https://github.com/shamoon) ([#13883](https://github.com/paperless-ngx/paperless-ngx/pull/13883))
### All App Changes
<details>
<summary>2 changes</summary>
- Fix: fix dark mode select disabled color, ensure disabled cursor on display mode dropdown [@shamoon](https://github.com/shamoon) ([#13881](https://github.com/paperless-ngx/paperless-ngx/pull/13881))
- Fix: add disable to the drag-drop list component [@shamoon](https://github.com/shamoon) ([#13880](https://github.com/paperless-ngx/paperless-ngx/pull/13880))
</details>
## paperless-ngx 3.1.1 ## paperless-ngx 3.1.1
### Bug Fixes ### Bug Fixes
+6
View File
@@ -2088,6 +2088,12 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to "always". Defaults to "always".
#### [`PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=<bool>`](#PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS) {#PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS}
: If set to false, Paperless blocks remote OCR endpoint URLs that resolve to non-public addresses (e.g., localhost, etc).
Defaults to True.
## AI {#ai} ## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED} #### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
+1 -1
View File
@@ -47,7 +47,7 @@ dependencies = [
"httpx-oauth~=0.17", "httpx-oauth~=0.17",
"ijson>=3.5.1", "ijson>=3.5.1",
"imap-tools~=1.14.0", "imap-tools~=1.14.0",
"jinja2~=3.1.5", "jinja2~=3.1.6",
"langdetect~=1.0.9", "langdetect~=1.0.9",
"llama-index-core>=0.14.23", "llama-index-core>=0.14.23",
"llama-index-embeddings-huggingface>=0.6.1", "llama-index-embeddings-huggingface>=0.6.1",
@@ -111,7 +111,7 @@
</h6> </h6>
<ul class="nav flex-column mb-2" cdkDropList (cdkDropListDropped)="onDrop($event)"> <ul class="nav flex-column mb-2" cdkDropList (cdkDropListDropped)="onDrop($event)">
@for (view of savedViewService.sidebarViews; track view.id) { @for (view of savedViewService.sidebarViews; track view.id) {
<li class="nav-item w-100 app-link" cdkDrag [cdkDragDisabled]="!settingsService.organizingSidebarSavedViews() || !canSaveSettings" <li class="nav-item app-link" cdkDrag [cdkDragDisabled]="!settingsService.organizingSidebarSavedViews() || !canSaveSettings"
cdkDragPreviewContainer="parent" cdkDragPreviewClass="navItemDrag" (cdkDragStarted)="onDragStart($event)" cdkDragPreviewContainer="parent" cdkDragPreviewClass="navItemDrag" (cdkDragStarted)="onDragStart($event)"
(cdkDragEnded)="onDragEnd($event)"> (cdkDragEnded)="onDragEnd($event)">
<a class="nav-link" routerLink="view/{{view.id}}" <a class="nav-link" routerLink="view/{{view.id}}"
@@ -128,7 +128,7 @@
} }
</a> </a>
@if (settingsService.organizingSidebarSavedViews() && canSaveSettings) { @if (settingsService.organizingSidebarSavedViews() && canSaveSettings) {
<div class="position-absolute end-0 top-0 px-3 py-2" [class.me-n3]="slimSidebarEnabled" cdkDragHandle> <div class="position-absolute end-0 top-0 px-1 py-2" [class.me-n2]="slimSidebarEnabled" cdkDragHandle>
<i-bs name="grip-vertical"></i-bs> <i-bs name="grip-vertical"></i-bs>
</div> </div>
} }
@@ -332,7 +332,7 @@
</li> </li>
<li class="nav-item" [class.visually-hidden]="slimSidebarEnabled"> <li class="nav-item" [class.visually-hidden]="slimSidebarEnabled">
<div class="text-muted small d-flex align-items-center flex-wrap nav-label"> <div class="text-muted small d-flex align-items-center flex-wrap nav-label">
<div class="me-3"> <div class="me-2">
<a class="text-muted text-decoration-none" target="_blank" rel="noopener noreferrer" <a class="text-muted text-decoration-none" target="_blank" rel="noopener noreferrer"
href="https://github.com/paperless-ngx/paperless-ngx" ngbPopover="GitHub" i18n-ngbPopover href="https://github.com/paperless-ngx/paperless-ngx" ngbPopover="GitHub" i18n-ngbPopover
[disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body" [disablePopover]="!slimSidebarPopoversEnabled" placement="end" container="body"
@@ -341,7 +341,7 @@
</a> </a>
</div> </div>
@if (!settingsService.updateCheckingIsSet || appRemoteVersion()) { @if (!settingsService.updateCheckingIsSet || appRemoteVersion()) {
<div class="version-check"> <div class="version-check d-flex align-items-center">
<ng-template #updateAvailablePopContent> <ng-template #updateAvailablePopContent>
<span class="small">Paperless-ngx {{ appRemoteVersion().version }} <ng-container i18n>is <span class="small">Paperless-ngx {{ appRemoteVersion().version }} <ng-container i18n>is
available.</ng-container><br /><ng-container i18n>Click to view.</ng-container></span> available.</ng-container><br /><ng-container i18n>Click to view.</ng-container></span>
@@ -1,6 +1,6 @@
@if (useDropdown) { @if (useDropdown) {
<div class="btn-group w-100" role="group" ngbDropdown #dropdown="ngbDropdown" (openChange)="onOpenChange($event)" [popperOptions]="popperOptions"> <div class="btn-group w-100" role="group" ngbDropdown #dropdown="ngbDropdown" (openChange)="onOpenChange($event)" [popperOptions]="popperOptions">
<button class="btn btn-sm btn-outline-primary" id="dropdown_toggle" ngbDropdownToggle [disabled]="disabled" [aria-label]="title"> <button class="btn btn-sm" [ngClass]="!editing && isActive ? 'btn-primary' : 'btn-outline-primary'" id="dropdown_toggle" ngbDropdownToggle [disabled]="disabled" [aria-label]="title">
<i-bs name="{{icon}}"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div> <i-bs name="{{icon}}"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div>
@if (isActive) { @if (isActive) {
<pngx-clearable-badge [selected]="isActive" (cleared)="reset()"></pngx-clearable-badge> <pngx-clearable-badge [selected]="isActive" (cleared)="reset()"></pngx-clearable-badge>
@@ -1,5 +1,6 @@
import { import {
getLocaleNumberSymbol, getLocaleNumberSymbol,
NgClass,
NgTemplateOutlet, NgTemplateOutlet,
NumberSymbol, NumberSymbol,
} from '@angular/common' } from '@angular/common'
@@ -48,25 +49,26 @@ import { ClearableBadgeComponent } from '../clearable-badge/clearable-badge.comp
import { DocumentLinkComponent } from '../input/document-link/document-link.component' import { DocumentLinkComponent } from '../input/document-link/document-link.component'
export class CustomFieldQueriesModel { export class CustomFieldQueriesModel {
private _queries: CustomFieldQueryElement[] = [] private readonly _queries = signal<CustomFieldQueryElement[]>([])
private rootSubscriptions: Subscription[] = [] private rootSubscriptions: Subscription[] = []
public readonly changed = new Subject<CustomFieldQueriesModel>() public readonly changed = new Subject<CustomFieldQueriesModel>()
public get queries(): CustomFieldQueryElement[] { public get queries(): CustomFieldQueryElement[] {
return this._queries return this._queries()
} }
public set queries(value: CustomFieldQueryElement[]) { public set queries(value: CustomFieldQueryElement[]) {
this.teardownRootSubscriptions() this.teardownRootSubscriptions()
this._queries = value ?? [] const queries = value ?? []
for (const element of this._queries) { for (const element of queries) {
this.rootSubscriptions.push( this.rootSubscriptions.push(
element.changed.subscribe(() => { element.changed.subscribe(() => {
this.changed.next(this) this.changed.next(this)
}) })
) )
} }
this._queries.set(queries)
} }
public clear(fireEvent = true) { public clear(fireEvent = true) {
@@ -209,6 +211,7 @@ export class CustomFieldQueriesModel {
DocumentLinkComponent, DocumentLinkComponent,
ReactiveFormsModule, ReactiveFormsModule,
NgbDatepickerModule, NgbDatepickerModule,
NgClass,
NgTemplateOutlet, NgTemplateOutlet,
NgSelectModule, NgSelectModule,
NgxBootstrapIconsModule, NgxBootstrapIconsModule,
@@ -116,7 +116,7 @@
</pngx-page-header> </pngx-page-header>
<div class="row sticky-top py-3 mt-n2 mt-md-n3 bg-body"> <div class="row sticky-top py-3 mt-n2 mt-md-n3 bg-body rounded shadow-sm">
<pngx-filter-editor [hidden]="isBulkEditing" [disabled]="isBulkEditing" [filterRules]="list.filterRules" (filterRulesChange)="onFilterRulesChange($event)" (resetFilterRules)="onFilterRulesReset($event)" [unmodifiedFilterRules]="unmodifiedFilterRules()" [selectionData]="list.selectionData" #filterEditor></pngx-filter-editor> <pngx-filter-editor [hidden]="isBulkEditing" [disabled]="isBulkEditing" [filterRules]="list.filterRules" (filterRulesChange)="onFilterRulesChange($event)" (resetFilterRules)="onFilterRulesReset($event)" [unmodifiedFilterRules]="unmodifiedFilterRules()" [selectionData]="list.selectionData" #filterEditor></pngx-filter-editor>
<pngx-bulk-editor [hidden]="!isBulkEditing" [disabled]="!isBulkEditing"></pngx-bulk-editor> <pngx-bulk-editor [hidden]="!isBulkEditing" [disabled]="!isBulkEditing"></pngx-bulk-editor>
</div> </div>
@@ -1034,6 +1034,49 @@ describe('FilterEditorComponent', () => {
).toEqual([42, CustomFieldQueryOperator.Exists, 'true']) ).toEqual([42, CustomFieldQueryOperator.Exists, 'true'])
}) })
it('should reflect ingested custom field query rules in the dropdown toggle', () => {
const dropdown = fixture.debugElement.query(
By.css('pngx-custom-fields-query-dropdown')
)
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).toBeNull()
// switching to a view with a custom field query
component.filterRules = [
{
rule_type: FILTER_CUSTOM_FIELDS_QUERY,
value: '["OR",[[42,"exists","true"]]]',
},
]
fixture.detectChanges()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).not.toBeNull()
expect(
dropdown.nativeElement
.querySelector('#dropdown_toggle')
.classList.contains('btn-primary')
).toBeTruthy()
// and back to a view without one
component.filterRules = [
{
rule_type: FILTER_HAS_TAGS_ALL,
value: '19',
},
]
fixture.detectChanges()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).toBeNull()
expect(
dropdown.nativeElement
.querySelector('#dropdown_toggle')
.classList.contains('btn-primary')
).toBeFalsy()
})
it('should ingest filter rules for owner', () => { it('should ingest filter rules for owner', () => {
expect(component.permissionsSelectionModel.ownerFilter).toEqual( expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NONE OwnerFilterType.NONE
+5 -16
View File
@@ -47,6 +47,8 @@ $grid-breakpoints: (
); );
:root { :root {
--bs-border-radius: #{$border-radius};
@each $name, $value in $grid-breakpoints { @each $name, $value in $grid-breakpoints {
--bs-breakpoint-#{$name}: #{$value}; --bs-breakpoint-#{$name}: #{$value};
} }
@@ -78,19 +80,12 @@ body {
} }
.btn { .btn {
--bs-btn-border-radius: .425rem; --bs-border-radius-sm: #{$border-radius};
--bs-border-radius-sm: .425rem;
font-weight: 500; font-weight: 500;
} }
.form-control,
.form-select,
.input-group-text {
border-radius: .425rem;
}
.pagination, .input-group { .pagination, .input-group {
--bs-border-radius-sm: .425rem; --bs-border-radius-sm: #{$border-radius};
} }
@media(min-width: 768px) { @media(min-width: 768px) {
@@ -689,10 +684,6 @@ table.table {
--bs-toast-max-width: var(--pngx-toast-max-width); --bs-toast-max-width: var(--pngx-toast-max-width);
} }
.alert {
--bs-border-radius: .425rem;
}
.alert-primary { .alert-primary {
--bs-alert-color: var(--bs-primary); --bs-alert-color: var(--bs-primary);
--bs-alert-bg: var(--pngx-primary-faded); --bs-alert-bg: var(--pngx-primary-faded);
@@ -824,8 +815,6 @@ code {
--bs-accordion-bg: var(--bs-light); --bs-accordion-bg: var(--bs-light);
--bs-accordion-active-color: var(--bs-primary); --bs-accordion-active-color: var(--bs-primary);
--bs-accordion-active-bg: var(--pngx-bg-alt); --bs-accordion-active-bg: var(--pngx-bg-alt);
--bs-border-radius: .425rem;
--bs-accordion-inner-border-radius: calc(.425rem - 1px);
} }
.accordion-button::after { .accordion-button::after {
@@ -849,7 +838,7 @@ code {
} }
/* Animate items as they're being sorted. */ /* Animate items as they're being sorted. */
.cdk-drop-list-dragging .cdk-drag { .cdk-drop-list-dragging .cdk-drag:not(.cdk-drag-preview) {
transition: transform 250ms cubic-bezier(0, 0, 0.2, 1); transition: transform 250ms cubic-bezier(0, 0, 0.2, 1);
} }
+1
View File
@@ -113,6 +113,7 @@ $form-check-radio-checked-bg-image-dark: url("data:image/svg+xml,%3csvg xmlns='h
--bs-tertiary-bg: var(--pngx-bg-darker); --bs-tertiary-bg: var(--pngx-bg-darker);
--bs-dark-border-subtle: var(--pngx-bg-darker); --bs-dark-border-subtle: var(--pngx-bg-darker);
--bs-border-color-translucent: rgba(0, 0, 0, .175); // override bs --bs-border-color-translucent: rgba(0, 0, 0, .175); // override bs
--bs-box-shadow-sm: 0 0.125rem 0.25rem rgba(0, 0, 0, 0.15); // slightly darker than bs default
.text-dark, .text-light { .text-dark, .text-light {
color: var(--bs-body-color) !important; color: var(--bs-body-color) !important;
+8 -3
View File
@@ -16,6 +16,9 @@ from django.core.cache import cache
from django.core.cache import caches from django.core.cache import caches
from documents.models import Document from documents.models import Document
from paperless.signed_pickle import SignedPickleError
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
if TYPE_CHECKING: if TYPE_CHECKING:
from django.core.cache.backends.base import BaseCache from django.core.cache.backends.base import BaseCache
@@ -118,9 +121,11 @@ class StoredLRUCache(LRUCache):
serialized_data = self._backend.get(self._backend_key) serialized_data = self._backend.get(self._backend_key)
try: try:
self._data = ( self._data = (
pickle.loads(serialized_data) if serialized_data else OrderedDict() signed_pickle_loads(serialized_data)
if serialized_data
else OrderedDict()
) )
except pickle.PickleError: except (SignedPickleError, pickle.PickleError):
logger.warning( logger.warning(
"Cache exists in backend but could not be read (possibly invalid format)", "Cache exists in backend but could not be read (possibly invalid format)",
) )
@@ -132,7 +137,7 @@ class StoredLRUCache(LRUCache):
""" """
self._backend.set( self._backend.set(
self._backend_key, self._backend_key,
pickle.dumps(self._data), signed_pickle_dumps(self._data),
self.backend_ttl, self.backend_ttl,
) )
+13 -3
View File
@@ -28,6 +28,9 @@ from documents.caching import CLASSIFIER_VERSION_KEY
from documents.caching import StoredLRUCache from documents.caching import StoredLRUCache
from documents.models import Document from documents.models import Document
from documents.models import MatchingModel from documents.models import MatchingModel
from paperless.signed_pickle import SignedPickleError
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
logger = logging.getLogger("paperless.classifier") logger = logging.getLogger("paperless.classifier")
@@ -527,10 +530,17 @@ class DocumentClassifier:
serialized_result = read_cache.get(key) serialized_result = read_cache.get(key)
if serialized_result is None: if serialized_result is None:
result = self.data_vectorizer.transform([self.preprocess_content(content)]) result = self.data_vectorizer.transform([self.preprocess_content(content)])
read_cache.set(key, pickle.dumps(result), CACHE_5_MINUTES) read_cache.set(key, signed_pickle_dumps(result), CACHE_5_MINUTES)
else: else:
read_cache.touch(key, CACHE_5_MINUTES) try:
result = pickle.loads(serialized_result) result = signed_pickle_loads(serialized_result)
except SignedPickleError:
result = self.data_vectorizer.transform(
[self.preprocess_content(content)],
)
read_cache.set(key, signed_pickle_dumps(result), CACHE_5_MINUTES)
else:
read_cache.touch(key, CACHE_5_MINUTES)
return result return result
def predict_correspondent(self, content: str) -> int | None: def predict_correspondent(self, content: str) -> int | None:
+5 -1
View File
@@ -462,7 +462,11 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
""" """
Returns a sanitized filename for the document, not including any paths. Returns a sanitized filename for the document, not including any paths.
""" """
result = str(self) # Root owns metadata for all versions
context_document = (
self.root_document if self.root_document_id is not None else self
)
result = str(context_document)
if counter: if counter:
result += f"_{counter:02}" result += f"_{counter:02}"
@@ -1063,3 +1063,79 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
) )
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("non-public address", str(response.data).lower()) self.assertIn("non-public address", str(response.data).lower())
@override_settings(REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=False)
def test_update_remote_ocr_endpoint_blocks_internal_endpoint_when_disallowed(
self,
) -> None:
"""
GIVEN:
- Internal remote OCR endpoints are disallowed
WHEN:
- The config is updated with a remote OCR endpoint resolving internally
THEN:
- The request is rejected
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_endpoint": "http://127.0.0.1:5000",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("non-public address", str(response.data).lower())
@override_settings(REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=True)
def test_update_remote_ocr_endpoint_allows_internal_endpoint_by_default(
self,
) -> None:
"""
GIVEN:
- Internal remote OCR endpoints are allowed (the default)
WHEN:
- The config is updated with a remote OCR endpoint resolving internally
THEN:
- The request is accepted, preserving existing self-hosted deployments
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_endpoint": "http://127.0.0.1:5000",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(
response.data["remote_ocr_endpoint"],
"http://127.0.0.1:5000",
)
@override_settings(REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS=False)
def test_update_remote_ocr_endpoint_empty_value_skips_validation(
self,
) -> None:
"""
GIVEN:
- Internal remote OCR endpoints are disallowed
WHEN:
- The config is updated with an empty remote OCR endpoint
THEN:
- The request is accepted; clearing the field never needs
outbound URL validation
"""
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_endpoint": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["remote_ocr_endpoint"], "")
+27
View File
@@ -102,6 +102,7 @@ class TestApiObjects(DirectoriesMixin, APITestCase):
- API is called - API is called
THEN: THEN:
- Last correspondence date is returned only if requested for list, and for detail - Last correspondence date is returned only if requested for list, and for detail
- The date is scoped to documents the requesting user may view
""" """
Document.objects.create( Document.objects.create(
@@ -145,6 +146,32 @@ class TestApiObjects(DirectoriesMixin, APITestCase):
response.data["last_correspondence"], response.data["last_correspondence"],
) )
# A newer document owned by another user must not leak through the
# aggregate for a non-superuser who cannot view it
other = User.objects.create_user(username="other")
Document.objects.create(
mime_type="application/pdf",
correspondent=self.c1,
created=datetime.date(2023, 6, 1),
checksum="hidden",
owner=other,
)
user = User.objects.create_user(username="regular")
user.user_permissions.add(
Permission.objects.get(codename="view_correspondent"),
)
self.client.force_authenticate(user=user)
response = self.client.get("/api/correspondents/?last_correspondence=true")
self.assertEqual(response.status_code, status.HTTP_200_OK)
result = next(r for r in response.data["results"] if r["id"] == self.c1.id)
self.assertIn("2022-01-02", result["last_correspondence"])
response = self.client.get(f"/api/correspondents/{self.c1.id}/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertIn("2022-01-02", response.data["last_correspondence"])
def test_paginated_objects_include_all_only_for_legacy_version(self) -> None: def test_paginated_objects_include_all_only_for_legacy_version(self) -> None:
response_v10 = self.client.get("/api/correspondents/") response_v10 = self.client.get("/api/correspondents/")
self.assertEqual(response_v10.status_code, status.HTTP_200_OK) self.assertEqual(response_v10.status_code, status.HTTP_200_OK)
+16 -3
View File
@@ -1,6 +1,7 @@
import pickle
from documents.caching import StoredLRUCache from documents.caching import StoredLRUCache
from paperless.signed_pickle import HMAC_SIZE
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
def test_lru_cache_entries() -> None: def test_lru_cache_entries() -> None:
@@ -42,4 +43,16 @@ def test_stored_lru_cache_key_ttl(mocker) -> None:
key, data, timeout = mock_backend.set.call_args[0] key, data, timeout = mock_backend.set.call_args[0]
assert key == "test_key" assert key == "test_key"
assert timeout == 321 assert timeout == 321
assert pickle.loads(data) == {"x": "X", "y": "Y"} assert signed_pickle_loads(data) == {"x": "X", "y": "Y"}
def test_stored_lru_cache_rejects_tampered_data(mocker) -> None:
serialized_data = bytearray(signed_pickle_dumps({"x": "X"}))
serialized_data[HMAC_SIZE] ^= 0xFF
mock_backend = mocker.Mock()
mock_backend.get.return_value = bytes(serialized_data)
cache = StoredLRUCache("test_key", backend=mock_backend)
cache.load()
assert cache.get("x") is None
+23
View File
@@ -19,6 +19,8 @@ from documents.models import MatchingModel
from documents.models import StoragePath from documents.models import StoragePath
from documents.models import Tag from documents.models import Tag
from documents.tests.utils import DirectoriesMixin from documents.tests.utils import DirectoriesMixin
from paperless.signed_pickle import HMAC_SIZE
from paperless.signed_pickle import signed_pickle_dumps
def dummy_preprocess(content: str, **kwargs): def dummy_preprocess(content: str, **kwargs):
@@ -265,6 +267,27 @@ class TestClassifier(DirectoriesMixin, TestCase):
self.assertEqual(mock_preprocess_content.call_count, 2) self.assertEqual(mock_preprocess_content.call_count, 2)
self.assertEqual(mock_transform.call_count, 2) self.assertEqual(mock_transform.call_count, 2)
def test_vectorize_recomputes_tampered_cache_entry(self) -> None:
cached = bytearray(signed_pickle_dumps(["cached vector"]))
cached[HMAC_SIZE] ^= 0xFF
self.classifier.data_vectorizer = mock.Mock()
self.classifier.data_vectorizer.transform.return_value = ["fresh vector"]
with (
mock.patch(
"documents.classifier.read_cache.get",
return_value=bytes(cached),
),
mock.patch("documents.classifier.read_cache.set") as cache_set,
mock.patch("documents.classifier.read_cache.touch") as cache_touch,
):
result = self.classifier._vectorize("content")
self.assertEqual(result, ["fresh vector"])
self.classifier.data_vectorizer.transform.assert_called_once()
cache_set.assert_called_once()
cache_touch.assert_not_called()
def test_no_retrain_if_no_change(self) -> None: def test_no_retrain_if_no_change(self) -> None:
""" """
GIVEN: GIVEN:
@@ -156,6 +156,40 @@ class TestDocument(TestCase):
) )
self.assertEqual(doc.get_public_filename(), "2020-12-25 test") self.assertEqual(doc.get_public_filename(), "2020-12-25 test")
def test_version_file_name_uses_root_document_metadata(self) -> None:
root_correspondent = Correspondent.objects.create(name="Root correspondent")
version_correspondent = Correspondent.objects.create(
name="Version correspondent",
)
root = Document.objects.create(
mime_type="application/pdf",
title="Root title",
created=date(2020, 12, 25),
correspondent=root_correspondent,
)
version = Document.objects.create(
mime_type="application/pdf",
title="Version title",
created=date(1990, 1, 1),
correspondent=version_correspondent,
root_document=root,
version_index=1,
)
self.assertEqual(
version.get_public_filename(),
"2020-12-25 Root correspondent Root title.pdf",
)
root.title = "Updated root title"
root.save(update_fields=("title",))
version.refresh_from_db()
self.assertEqual(
version.get_public_filename(),
"2020-12-25 Root correspondent Updated root title.pdf",
)
def test_suggestion_content_uses_latest_version_content_for_root_documents( def test_suggestion_content_uses_latest_version_content_for_root_documents(
self, self,
) -> None: ) -> None:
@@ -192,6 +192,50 @@ class ShareLinkBundleAPITests(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_302_FOUND) self.assertEqual(response.status_code, status.HTTP_302_FOUND)
self.assertIn("sharelink_notfound=1", response["Location"]) self.assertIn("sharelink_notfound=1", response["Location"])
def test_share_link_missing_file_redirects(self) -> None:
"""
GIVEN:
- A share link whose document file is missing from disk
WHEN:
- The public share link is requested anonymously
THEN:
- The user is redirected to login instead of a 500 error
"""
doc = DocumentFactory.create(filename="missing-original.pdf")
share_link = ShareLink.objects.create(
slug="missingfilelink",
document=doc,
file_version=ShareLink.FileVersion.ORIGINAL,
)
self.client.logout()
response = self.client.get(f"/share/{share_link.slug}/")
self.assertEqual(response.status_code, status.HTTP_302_FOUND)
self.assertIn("sharelink_notfound=1", response["Location"])
def test_download_ready_bundle_missing_file_returns_503(self) -> None:
"""
GIVEN:
- A READY bundle whose zip file is missing from disk
WHEN:
- The public share link is requested anonymously
THEN:
- A 503 is returned instead of a 500 error
"""
bundle = ShareLinkBundle.objects.create(
slug="missingbundlefile",
file_version=ShareLink.FileVersion.ARCHIVE,
status=ShareLinkBundle.Status.READY,
file_path="bundles/gone.zip",
)
bundle.documents.set([self.document])
self.client.logout()
response = self.client.get(f"/share/{bundle.slug}/")
self.assertEqual(response.status_code, status.HTTP_503_SERVICE_UNAVAILABLE)
class ShareLinkBundleTaskTests(DirectoriesMixin, APITestCase): class ShareLinkBundleTaskTests(DirectoriesMixin, APITestCase):
def setUp(self) -> None: def setUp(self) -> None:
+26 -9
View File
@@ -577,13 +577,19 @@ class CorrespondentViewSet(
def list(self, request, *args, **kwargs): def list(self, request, *args, **kwargs):
if request.query_params.get("last_correspondence", None): if request.query_params.get("last_correspondence", None):
self.queryset = self.queryset.annotate( self.queryset = self.queryset.annotate(
last_correspondence=Max("documents__created"), last_correspondence=Max(
"documents__created",
filter=self.get_document_count_filter(),
),
) )
return super().list(request, *args, **kwargs) return super().list(request, *args, **kwargs)
def retrieve(self, request, *args, **kwargs): def retrieve(self, request, *args, **kwargs):
self.queryset = self.queryset.annotate( self.queryset = self.queryset.annotate(
last_correspondence=Max("documents__created"), last_correspondence=Max(
"documents__created",
filter=self.get_document_count_filter(),
),
) )
return super().retrieve(request, *args, **kwargs) return super().retrieve(request, *args, **kwargs)
@@ -4573,6 +4579,10 @@ class ShareLinkViewSet(
class ShareLinkBundleViewSet(PassUserMixin, ModelViewSet[ShareLinkBundle]): class ShareLinkBundleViewSet(PassUserMixin, ModelViewSet[ShareLinkBundle]):
model = ShareLinkBundle model = ShareLinkBundle
# Bundles are immutable once created; rebuild via the dedicated action
# rather than PUT/PATCH.
http_method_names = ["get", "post", "delete", "head", "options"]
queryset = ShareLinkBundle.objects.all() queryset = ShareLinkBundle.objects.all()
serializer_class = ShareLinkBundleSerializer serializer_class = ShareLinkBundleSerializer
@@ -4707,12 +4717,15 @@ class SharedLinkView(View):
and share_link.expiration < timezone.now() and share_link.expiration < timezone.now()
): ):
return HttpResponseRedirect("/accounts/login/?sharelink_expired=1") return HttpResponseRedirect("/accounts/login/?sharelink_expired=1")
return serve_file( try:
doc=share_link.document, return serve_file(
use_archive=share_link.file_version == ShareLink.FileVersion.ARCHIVE doc=share_link.document,
and share_link.document.has_archive_version, use_archive=share_link.file_version == ShareLink.FileVersion.ARCHIVE
disposition="inline", and share_link.document.has_archive_version,
) disposition="inline",
)
except FileNotFoundError:
return HttpResponseRedirect("/accounts/login/?sharelink_notfound=1")
bundle = ShareLinkBundle.objects.filter(slug=slug).first() bundle = ShareLinkBundle.objects.filter(slug=slug).first()
if bundle is None: if bundle is None:
@@ -4734,7 +4747,11 @@ class SharedLinkView(View):
file_path = bundle.absolute_file_path file_path = bundle.absolute_file_path
if bundle.status == ShareLinkBundle.Status.FAILED or file_path is None: if (
bundle.status == ShareLinkBundle.Status.FAILED
or file_path is None
or not file_path.exists()
):
return HttpResponse( return HttpResponse(
_( _(
"The share link bundle is unavailable.", "The share link bundle is unavailable.",
File diff suppressed because it is too large Load Diff
+3 -31
View File
@@ -1,12 +1,12 @@
import hmac
import os import os
import pickle
from hashlib import sha256
from celery import Celery from celery import Celery
from celery.signals import worker_process_init from celery.signals import worker_process_init
from kombu.serialization import register from kombu.serialization import register
from paperless.signed_pickle import signed_pickle_dumps
from paperless.signed_pickle import signed_pickle_loads
# Set the default Django settings module for the 'celery' program. # Set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "paperless.settings") os.environ.setdefault("DJANGO_SETTINGS_MODULE", "paperless.settings")
@@ -18,34 +18,6 @@ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "paperless.settings")
# on the worker side using Django's SECRET_KEY. # on the worker side using Django's SECRET_KEY.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
HMAC_SIZE = 32 # SHA-256 digest length
def _get_signing_key() -> bytes:
from django.conf import settings
return settings.SECRET_KEY.encode()
def signed_pickle_dumps(obj: object) -> bytes:
data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
signature = hmac.new(_get_signing_key(), data, sha256).digest()
return signature + data
def signed_pickle_loads(payload: bytes) -> object:
if len(payload) < HMAC_SIZE:
msg = "Signed-pickle payload too short"
raise ValueError(msg)
signature = payload[:HMAC_SIZE]
data = payload[HMAC_SIZE:]
expected = hmac.new(_get_signing_key(), data, sha256).digest()
if not hmac.compare_digest(signature, expected):
msg = "Signed-pickle HMAC verification failed — message may have been tampered with"
raise ValueError(msg)
return pickle.loads(data)
register( register(
"signed-pickle", "signed-pickle",
signed_pickle_dumps, signed_pickle_dumps,
+38
View File
@@ -32,6 +32,8 @@ if TYPE_CHECKING:
import datetime import datetime
from types import TracebackType from types import TracebackType
from azure.core.pipeline import PipelineRequest
from paperless.parsers import MetadataEntry from paperless.parsers import MetadataEntry
from paperless.parsers import ParserContext from paperless.parsers import ParserContext
@@ -436,9 +438,45 @@ class RemoteDocumentParser:
from azure.ai.documentintelligence.models import DocumentContentFormat from azure.ai.documentintelligence.models import DocumentContentFormat
from azure.core.credentials import AzureKeyCredential from azure.core.credentials import AzureKeyCredential
from paperless.network import validate_outbound_http_url
allow_internal = settings.REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS
try:
validate_outbound_http_url(config.endpoint, allow_internal=allow_internal)
except ValueError as e:
raise ParseError(f"Invalid remote OCR endpoint: {e}") from e
def _revalidate_request_host(request: PipelineRequest) -> None:
"""Re-validates the destination host of every request sent.
The check above only covers the moment the client is built. A
single analysis involves several requests spread over the
polling loop below, and any one of them can be redirected.
Wiring this through ``raw_request_hook`` (Azure's built-in
CustomHookPolicy) rather than a custom policy means it runs
*after* RedirectPolicy in the pipeline, so it sees - and
re-checks - every actual outbound URL, including redirect
targets, not just the original request.
"""
validate_outbound_http_url(
request.http_request.url,
allow_internal=allow_internal,
)
client = DocumentIntelligenceClient( client = DocumentIntelligenceClient(
endpoint=config.endpoint, endpoint=config.endpoint,
credential=AzureKeyCredential(config.api_key), credential=AzureKeyCredential(config.api_key),
raw_request_hook=_revalidate_request_host,
# AzureKeyCredential is sent as Ocp-Apim-Subscription-Key, which
# Azure's default SensitiveHeaderCleanupPolicy does not strip on
# a cross-domain redirect (only Authorization and
# x-ms-authorization-auxiliary are, by default).
blocked_redirect_headers=[
"Authorization",
"x-ms-authorization-auxiliary",
"Ocp-Apim-Subscription-Key",
],
) )
try: try:
+16
View File
@@ -305,6 +305,22 @@ class ApplicationConfigurationSerializer(
validate_llm_embedding_endpoint = validate_llm_endpoint validate_llm_embedding_endpoint = validate_llm_endpoint
def validate_remote_ocr_endpoint(self, value: str | None) -> str | None:
if not value:
return value
try:
validate_outbound_http_url(
value,
allow_internal=settings.REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS,
)
except ValueError as e:
raise serializers.ValidationError(
f"Invalid remote OCR endpoint: {e.args[0]}, see logs for details",
) from e
return value
class Meta: class Meta:
model = ApplicationConfiguration model = ApplicationConfiguration
fields = "__all__" fields = "__all__"
+4
View File
@@ -1208,6 +1208,10 @@ REMOTE_OCR_MODE = get_choice_from_env(
{"always", "workflow_only"}, {"always", "workflow_only"},
default="always", default="always",
) )
REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS = get_bool_from_env(
"PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS",
"true",
)
################################################################################ ################################################################################
# AI Settings # # AI Settings #
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
import hmac
import pickle
from hashlib import sha256
from typing import Any
from django.conf import settings
HMAC_SIZE = sha256().digest_size
class SignedPickleError(ValueError):
"""Raised when a signed pickle payload cannot be authenticated."""
def _get_signing_key() -> bytes:
return settings.SECRET_KEY.encode()
def signed_pickle_dumps(obj: object) -> bytes:
data = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
signature = hmac.new(_get_signing_key(), data, sha256).digest()
return signature + data
def signed_pickle_loads(payload: bytes) -> Any:
if len(payload) <= HMAC_SIZE:
msg = "Signed-pickle payload too short"
raise SignedPickleError(msg)
signature = payload[:HMAC_SIZE]
data = payload[HMAC_SIZE:]
expected = hmac.new(_get_signing_key(), data, sha256).digest()
if not hmac.compare_digest(signature, expected):
msg = "Signed-pickle HMAC verification failed; payload may have been tampered with"
raise SignedPickleError(msg)
return pickle.loads(data)
+1 -1
View File
@@ -6,9 +6,9 @@ from pathlib import Path
import pytest import pytest
from django.test import override_settings from django.test import override_settings
from paperless.celery import HMAC_SIZE
from paperless.celery import signed_pickle_dumps from paperless.celery import signed_pickle_dumps
from paperless.celery import signed_pickle_loads from paperless.celery import signed_pickle_loads
from paperless.signed_pickle import HMAC_SIZE
class TestSignedPickleSerializer: class TestSignedPickleSerializer:
+1 -1
View File
@@ -295,7 +295,7 @@ urlpatterns = [
], ],
), ),
), ),
re_path(r"share/(?P<slug>\w+)/?$", SharedLinkView.as_view()), re_path(r"^share/(?P<slug>\w+)/?$", SharedLinkView.as_view()),
re_path(r"^favicon.ico$", FaviconView.as_view(), name="favicon"), re_path(r"^favicon.ico$", FaviconView.as_view(), name="favicon"),
re_path(r"admin/", admin.site.urls), re_path(r"admin/", admin.site.urls),
re_path( re_path(
Generated
+1 -1
View File
@@ -3053,7 +3053,7 @@ requires-dist = [
{ name = "httpx-oauth", specifier = "~=0.17" }, { name = "httpx-oauth", specifier = "~=0.17" },
{ name = "ijson", specifier = ">=3.5.1" }, { name = "ijson", specifier = ">=3.5.1" },
{ name = "imap-tools", specifier = "~=1.14.0" }, { name = "imap-tools", specifier = "~=1.14.0" },
{ name = "jinja2", specifier = "~=3.1.5" }, { name = "jinja2", specifier = "~=3.1.6" },
{ name = "langdetect", specifier = "~=1.0.9" }, { name = "langdetect", specifier = "~=1.0.9" },
{ name = "llama-index-core", specifier = ">=0.14.23" }, { name = "llama-index-core", specifier = ">=0.14.23" },
{ name = "llama-index-embeddings-huggingface", specifier = ">=0.6.1" }, { name = "llama-index-embeddings-huggingface", specifier = ">=0.6.1" },