Make testing easier

Fix: fix allowed_hosts default
Auto translate strings
2026-11-11 00:41:02 +00:00 · 2026-11-10 15:21:06 -08:00 · 2026-11-10 07:53:53 -08:00 · 2026-11-09 06:53:56 +00:00 · 2026-11-08 22:52:34 -08:00 · 2026-11-07 12:24:29 -08:00
20 changed files with 533 additions and 162 deletions
@@ -56,6 +56,9 @@ We may close reports that are:

 The following are not generally considered vulnerabilities unless accompanied by a concrete, reproducible impact in Paperless-ngx:

+- password length inferred from masked mail-account passwords by users who already have permission to view the account, without disclosure of the password or token itself
+- disclosure limited to document IDs, existence, page counts, checksums, or modification/deletion timestamps, without access to document contents or an unintended privileged action
+- authenticated access to non-secret application settings, such as service endpoint URLs, model names, or configured environment variable names, without disclosure of credentials or secret values
 - large uploads or resource usage that do not bypass documented limits or privileges
 - IDOR / access control claims regarding the ability to attach an un-viewable object to a document. This is expected behavior.
 - claims based solely on the presence of a library, framework feature or code pattern without a working exploit
@@ -501,20 +501,20 @@ See the [Django project documentation on the settings](https://docs.djangoprojec

 #### [`PAPERLESS_ALLOWED_HOSTS=<comma-separated-list>`](#PAPERLESS_ALLOWED_HOSTS) {#PAPERLESS_ALLOWED_HOSTS}

-: If you're planning on putting Paperless on the open internet, then
-you really should set this value to the domain name you're using.
-Failing to do so leaves you open to HTTP host header attacks.
+: If you expose Paperless to the internet, set `PAPERLESS_URL` to its public URL
+or list its public hostname in `PAPERLESS_ALLOWED_HOSTS` to help protect against
+HTTP Host header attacks.
 You can read more about this in [the Django project's documentation](https://docs.djangoproject.com/en/4.1/topics/security/#host-header-validation)

-    Just remember that this is a comma-separated list, so
-    "example.com" is fine, as is "example.com,www.example.com", but
-    NOT " example.com" or "example.com,"
+    Separate multiple hostnames with commas, as in `example.com,www.example.com`.

-    Can also be set using PAPERLESS_URL (see above).
-
-    "localhost" is always allowed for docker healthcheck
-
-    Defaults to "\*", which is all hosts.
+    - With neither setting configured, Paperless accepts any host.
+    - With only `PAPERLESS_URL` set, Paperless accepts the hostname from that URL
+      and `localhost` (used by the Docker health check).
+    - `PAPERLESS_ALLOWED_HOSTS` adds hosts to `localhost` and the hostname from
+      `PAPERLESS_URL`, if set. A blank list adds no hosts.
+    - With `PAPERLESS_ALLOWED_HOSTS=*`, Paperless accepts any host, even when
+      `PAPERLESS_URL` is set.

 #### [`PAPERLESS_CORS_ALLOWED_HOSTS=<comma-separated-list>`](#PAPERLESS_CORS_ALLOWED_HOSTS) {#PAPERLESS_CORS_ALLOWED_HOSTS}

@@ -4030,14 +4030,21 @@
        <source>Sources</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
-          <context context-type="linenumber">46,47</context>
+          <context context-type="linenumber">47,48</context>
+        </context-group>
+      </trans-unit>
+      <trans-unit id="6025275430560167531" datatype="html">
+        <source>+<x id="INTERPOLATION" equiv-text="{{ hiddenReferenceCount(message) }}"/> more</source>
+        <context-group purpose="location">
+          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
+          <context context-type="linenumber">49,50</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8066608938393600549" datatype="html">
        <source>Message</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
-          <context context-type="linenumber">61,62</context>
+          <context context-type="linenumber">66,67</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/common/email-document-dialog/email-document-dialog.component.html</context>
@@ -4048,22 +4055,22 @@
        <source>Stop</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
-          <context context-type="linenumber">74</context>
+          <context context-type="linenumber">79</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
-          <context context-type="linenumber">74</context>
+          <context context-type="linenumber">79</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6490688569532630280" datatype="html">
        <source>Send</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
-          <context context-type="linenumber">78</context>
+          <context context-type="linenumber">83</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
-          <context context-type="linenumber">78</context>
+          <context context-type="linenumber">83</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
@@ -4090,14 +4097,14 @@
        <source>Ask a question about this document...</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.ts</context>
-          <context context-type="linenumber">61</context>
+          <context context-type="linenumber">83</context>
        </context-group>
      </trans-unit>
      <trans-unit id="224049052163028286" datatype="html">
        <source>Ask a question about a document...</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/chat/chat/chat.component.ts</context>
-          <context context-type="linenumber">62</context>
+          <context context-type="linenumber">84</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8700121026680200191" datatype="html">
@@ -6984,7 +6991,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/common/input/select/select.component.ts</context>
-          <context context-type="linenumber">179</context>
+          <context context-type="linenumber">185</context>
        </context-group>
      </trans-unit>
      <trans-unit id="1880237574877817137" datatype="html">
@@ -7091,7 +7098,7 @@
        <source>Private</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/common/input/select/select.component.ts</context>
-          <context context-type="linenumber">72</context>
+          <context context-type="linenumber">73</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/common/tag/tag.component.html</context>
@@ -7114,7 +7121,7 @@
        <source>No items found</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/common/input/select/select.component.ts</context>
-          <context context-type="linenumber">113</context>
+          <context context-type="linenumber">114</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6541407358060244620" datatype="html">
@@ -42,11 +42,16 @@
              <span class="text-break">{{ message.content }}</span>
            }
            @if (message.role === 'assistant' && !message.isStreaming && message.references?.length) {
-              <div class="chat-references list-group mt-3">
-                <div class="list-group-item text-muted small py-1" i18n>Sources</div>
-                @for (reference of message.references; track reference.id) {
-                  <a class="list-group-item list-group-item-action d-flex align-items-center text-primary" [routerLink]="['/documents', reference.id]" [title]="reference.title">
-                    <pngx-bs-icon width="0.9em" height="0.9em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
+              <div class="chat-references mt-2 pt-2 border-top">
+                <div class="d-flex align-items-center text-muted mb-1">
+                  <span i18n>Sources</span>
+                  @if (hiddenReferenceCount(message) > 0 && !expandedReferences().has(message)) {
+                    <button type="button" class="badge rounded-pill bg-body-secondary text-body-secondary fw-normal border-0 ms-2" (click)="expandReferences(message)" i18n>+{{ hiddenReferenceCount(message) }} more</button>
+                  }
+                </div>
+                @for (reference of visibleReferences(message); track reference.id) {
+                  <a class="reference-link d-flex align-items-center lh-sm text-primary text-decoration-none py-1" [routerLink]="['/documents', reference.id]" [title]="reference.title">
+                    <pngx-bs-icon width="0.8em" height="0.8em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
                  </a>
                }
              </div>
@@ -12,6 +12,10 @@
  display: none;
 }

+.message .flex-grow-1 {
+  min-width: 0;
+}
+
 .dropdown-item {
  white-space: initial;
 }
@@ -14,6 +14,7 @@ import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon
 import { MarkdownPipe } from 'src/app/pipes/markdown.pipe'
 import {
  ChatMessage,
+  ChatReference,
  ChatService,
  parseChatResponse,
 } from 'src/app/services/chat.service'
@@ -40,8 +41,29 @@ export class ChatComponent implements OnInit {
  readonly input = signal('')
  readonly documentId = signal<number>(undefined)
  readonly slowResponse = signal(false)
+  readonly expandedReferences = signal<Set<ChatMessage>>(new Set())
  public popperOptions = pngxPopperOptions

+  static readonly COLLAPSED_REFERENCE_COUNT = 2
+
+  public visibleReferences(message: ChatMessage): ChatReference[] {
+    if (this.expandedReferences().has(message)) {
+      return message.references
+    }
+    return message.references.slice(0, ChatComponent.COLLAPSED_REFERENCE_COUNT)
+  }
+
+  public hiddenReferenceCount(message: ChatMessage): number {
+    return Math.max(
+      0,
+      message.references.length - ChatComponent.COLLAPSED_REFERENCE_COUNT
+    )
+  }
+
+  public expandReferences(message: ChatMessage): void {
+    this.expandedReferences.update((expanded) => new Set(expanded).add(message))
+  }
+
  private chatService: ChatService = inject(ChatService)
  private router: Router = inject(Router)

@@ -36,7 +36,7 @@
    [appendTo]="appendTo"
    [loading]="loading()"
    [typeahead]="documentsInput$"
-    (mousedown)="$event.stopImmediatePropagation()"
+    (mousedown)="appendTo && $event.stopImmediatePropagation()"
    (change)="onChange(selectedDocumentIDs)">
    <ng-template ng-label-tmp let-document="item">
      <div class="d-flex align-items-center">
@@ -97,6 +97,7 @@ describe('SelectComponent', () => {
    let createNewVal
    component.createNew.subscribe((v) => (createNewVal = v))
    expect(component.allowCreateNew).toBeTruthy()
+    const closeSpy = jest.spyOn(component.ngSelectComponent, 'close')
    component.onSearch({ term: 'foo' })
    component.addItem(undefined)
    expect(createNewVal).toEqual('foo')
@@ -105,6 +106,8 @@ describe('SelectComponent', () => {
    component.onSearch({ term: 'baz' })
    component.clickNew()
    expect(createNewVal).toEqual('baz')
+    // dropdown should close when creating a new item so it isn't left open behind the modal
+    expect(closeSpy).toHaveBeenCalled()
  })

  it('should search items by independent normalized terms', () => {
@@ -4,6 +4,7 @@ import {
  forwardRef,
  Input,
  Output,
+  ViewChild,
 } from '@angular/core'
 import {
  FormsModule,
@@ -11,7 +12,7 @@ import {
  ReactiveFormsModule,
 } from '@angular/forms'
 import { RouterModule } from '@angular/router'
-import { NgSelectModule } from '@ng-select/ng-select'
+import { NgSelectComponent, NgSelectModule } from '@ng-select/ng-select'
 import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon/bootstrap-icon.component'
 import { matchesSearchText } from 'src/app/utils/text-search'
 import { AbstractInputComponent } from '../abstract-input'
@@ -124,6 +125,9 @@ export class SelectComponent extends AbstractInputComponent<number | string> {
  @Output()
  filterDocuments = new EventEmitter<any[]>()

+  @ViewChild(NgSelectComponent)
+  ngSelectComponent: NgSelectComponent
+
  public addItemRef: (name) => void

  private _lastSearchTerm: string
@@ -150,11 +154,13 @@ export class SelectComponent extends AbstractInputComponent<number | string> {
    if (name) this.createNew.next(name)
    else this.createNew.next(this._lastSearchTerm)
    this.clearLastSearchTerm()
+    this.ngSelectComponent?.close()
  }

  clickNew() {
    this.createNew.next(this._lastSearchTerm)
    this.clearLastSearchTerm()
+    this.ngSelectComponent?.close()
  }

  clearLastSearchTerm() {
@@ -31,6 +31,7 @@ from guardian.core import ObjectPermissionChecker
 from guardian.models import GroupObjectPermission
 from guardian.models import UserObjectPermission
 from rest_framework.permissions import BasePermission
+from rest_framework.permissions import DjangoModelPermissions
 from rest_framework.permissions import DjangoObjectPermissions

 from documents.models import Document
@@ -63,6 +64,17 @@ class PermissionGranteesUpdate(TypedDict, total=False):
 PermissionsSpec: TypeAlias = Mapping[PermissionAction, PermissionGranteesUpdate]


+class PaperlessModelPermissions(DjangoModelPermissions):
+    """Require basic global model permissions."""
+
+    perms_map = {
+        **DjangoModelPermissions.perms_map,
+        "GET": ["%(app_label)s.view_%(model_name)s"],
+        "OPTIONS": ["%(app_label)s.view_%(model_name)s"],
+        "HEAD": ["%(app_label)s.view_%(model_name)s"],
+    }
+
+
 class PaperlessObjectPermissions(DjangoObjectPermissions):
    """
    A permissions backend that checks for object-level permissions
@@ -576,12 +588,9 @@ def annotate_document_count_by_ids(
                       or CustomFieldInstance)
        related_object_field: field on the relation pointing back to queryset pk
        document_ids: the document ids to count against -- a concrete list/set,
-                       or a simple (already resolved) queryset of ids. Callers
-                       that need this filtered by a complex condition (e.g. a
-                       permission check) should resolve it to a concrete list
-                       first if the same ids will be reused across multiple
-                       calls, rather than passing the complex queryset itself
-                       into each -- see ``_get_selection_data_for_queryset``.
+                       or a simple (already resolved) queryset of ids. Keep
+                       the list small: it is embedded in the statement as
+                       literal parameters.
        target_field: field on the relation pointing to Document id
    """

@@ -15,6 +15,7 @@ from paperless.models import ColorConvertChoices
 from paperless_testing.dirs import DirectoriesMixin
 from paperless_testing.factories import UserFactory
 from paperless_testing.http import read_streaming_response
+from paperless_testing.permissions import grant_global


 class TestApiAppConfig(DirectoriesMixin, APITestCase):
@@ -26,6 +27,67 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
        user = UserFactory(username="temp_admin", superuser=True)
        self.client.force_authenticate(user=user)

+    def test_api_read_config_requires_view_permission(self) -> None:
+        for permissions in (
+            (),
+            ("change_applicationconfiguration",),
+            ("view_document",),
+        ):
+            user = UserFactory()
+            grant_global(user, *permissions)
+            self.client.force_authenticate(user=user)
+
+            for endpoint in (self.ENDPOINT, f"{self.ENDPOINT}1/"):
+                for method in ("get", "head", "options"):
+                    with self.subTest(
+                        permissions=permissions,
+                        endpoint=endpoint,
+                        method=method,
+                    ):
+                        response = getattr(self.client, method)(endpoint)
+                        self.assertEqual(
+                            response.status_code,
+                            status.HTTP_403_FORBIDDEN,
+                        )
+
+    def test_api_read_config_with_view_permission(self) -> None:
+        user = UserFactory()
+        grant_global(user, "view_applicationconfiguration")
+        self.client.force_authenticate(user=user)
+
+        for endpoint in (self.ENDPOINT, f"{self.ENDPOINT}1/"):
+            for method in ("get", "head", "options"):
+                with self.subTest(endpoint=endpoint, method=method):
+                    response = getattr(self.client, method)(endpoint)
+                    self.assertEqual(response.status_code, status.HTTP_200_OK)
+
+    def test_api_view_config_permission_does_not_allow_writes(self) -> None:
+        user = UserFactory()
+        grant_global(user, "view_applicationconfiguration")
+        self.client.force_authenticate(user=user)
+
+        for method in ("put", "patch", "delete"):
+            with self.subTest(method=method):
+                response = getattr(self.client, method)(f"{self.ENDPOINT}1/")
+                self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
+
+    def test_api_change_config_permission_allows_updates(self) -> None:
+        user = UserFactory()
+        grant_global(user, "change_applicationconfiguration")
+        self.client.force_authenticate(user=user)
+
+        response = self.client.patch(
+            f"{self.ENDPOINT}1/",
+            {"app_title": "Updated title"},
+            format="json",
+        )
+
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(
+            ApplicationConfiguration.objects.get().app_title,
+            "Updated title",
+        )
+
    def test_api_get_config(self) -> None:
        """
        GIVEN:
@@ -1141,8 +1141,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
        self.assertCountEqual(
            response.data["selected_document_types"],
            [
-                {"id": self.c1.id, "document_count": 1},
-                {"id": self.c2.id, "document_count": 0},
+                {"id": self.dt1.id, "document_count": 1},
+                {"id": self.dt2.id, "document_count": 0},
            ],
        )

@@ -49,6 +49,7 @@ from documents.models import WorkflowTrigger
 from documents.search import get_backend
 from documents.signals.handlers import run_workflows
 from documents.tests.utils import ConsumeTaskMixin
+from documents.views import DocumentViewSet
 from paperless_testing.dirs import DirectoriesMixin
 from paperless_testing.factories import DocumentFactory
 from paperless_testing.factories import TagFactory
@@ -1755,12 +1756,113 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        }
        self.assertEqual(counts_by_field[custom_field.id], 1)

+    def test_selection_data_counts_with_shared_by_filter(self) -> None:
+        """
+        GIVEN:
+            - Several documents owned by one user with the same correspondent
+              and document type, only one of which is shared
+        WHEN:
+            - Requesting the document list filtered by shared_by__id with
+              include_selection_data=true
+        THEN:
+            - Counts reflect only the shared document, not every document the
+              owner has with that correspondent or document type, regardless of
+              the batch size
+        """
+        owner = User.objects.create_user(username="owner")
+        other = User.objects.create_user(username="other")
+        correspondent = Correspondent.objects.create(name="c1")
+        doc_type = DocumentType.objects.create(name="dt1")
+        tag = Tag.objects.create(name="tag")
+        docs = [
+            Document.objects.create(
+                checksum=str(i),
+                owner=owner,
+                correspondent=correspondent,
+                document_type=doc_type,
+            )
+            for i in range(3)
+        ]
+        for doc in docs:
+            doc.tags.add(tag)
+        grant_object(other, docs[0], "view_document")
+
+        for batch_size in (10_000, 1):
+            with (
+                self.subTest(batch_size=batch_size),
+                mock.patch.object(
+                    DocumentViewSet,
+                    "selection_data_batch_size",
+                    batch_size,
+                ),
+            ):
+                response = self.client.get(
+                    f"/api/documents/?shared_by__id={owner.id}"
+                    "&include_selection_data=true",
+                )
+
+                self.assertEqual(response.status_code, status.HTTP_200_OK)
+                self.assertEqual(response.data["count"], 1)
+                selection_data = response.data["selection_data"]
+                for key, obj in (
+                    ("selected_correspondents", correspondent),
+                    ("selected_document_types", doc_type),
+                    ("selected_tags", tag),
+                ):
+                    counts = {
+                        item["id"]: item["document_count"]
+                        for item in selection_data[key]
+                    }
+                    self.assertEqual(counts[obj.id], 1, key)
+
+    def test_selection_data_counts_documents_once_when_filter_joins(self) -> None:
+        """
+        GIVEN:
+            - A document matching a multi-valued filter through two tags
+        WHEN:
+            - Requesting the document list with include_selection_data=true
+        THEN:
+            - The document is counted once per correspondent, not once per
+              matching tag, regardless of the batch size
+        """
+        correspondent = Correspondent.objects.create(name="c1")
+        tag_a = Tag.objects.create(name="a")
+        tag_b = Tag.objects.create(name="b")
+        both = Document.objects.create(checksum="1", correspondent=correspondent)
+        both.tags.add(tag_a, tag_b)
+        only_a = Document.objects.create(checksum="2", correspondent=correspondent)
+        only_a.tags.add(tag_a)
+
+        for batch_size in (10_000, 1):
+            with (
+                self.subTest(batch_size=batch_size),
+                mock.patch.object(
+                    DocumentViewSet,
+                    "selection_data_batch_size",
+                    batch_size,
+                ),
+            ):
+                response = self.client.get(
+                    f"/api/documents/?tags__id__in={tag_a.id},{tag_b.id}"
+                    "&include_selection_data=true",
+                )
+
+                self.assertEqual(response.status_code, status.HTTP_200_OK)
+                counts = {
+                    item["id"]: item["document_count"]
+                    for item in response.data["selection_data"][
+                        "selected_correspondents"
+                    ]
+                }
+                self.assertEqual(counts[correspondent.id], 2)
+
    def test_statistics(self) -> None:
        doc1 = Document.objects.create(
            title="none1",
            checksum="A",
            mime_type="application/pdf",
            content="abc",
+            archive_serial_number=42,
        )
        Document.objects.create(
            title="none2",
@@ -1804,6 +1906,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        self.assertEqual(response.data["correspondent_count"], 2)
        self.assertEqual(response.data["document_type_count"], 1)
        self.assertEqual(response.data["storage_path_count"], 2)
+        self.assertEqual(response.data["current_asn"], 42)

    def test_statistics_excludes_document_versions(self) -> None:
        root = Document.objects.create(
@@ -1811,6 +1914,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
            checksum="A",
            mime_type="application/pdf",
            content="root",
+            archive_serial_number=42,
        )
        version = Document.objects.create(
            title="version",
@@ -1819,6 +1923,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
            content="version",
            root_document=root,
            version_index=1,
+            archive_serial_number=99,
        )
        tag_inbox = Tag.objects.create(name="t1", is_inbox_tag=True)
        version.tags.add(tag_inbox)
@@ -1829,6 +1934,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        self.assertEqual(response.data["documents_total"], 1)
        self.assertEqual(response.data["documents_inbox"], 0)
        self.assertEqual(response.data["character_count"], 4)
+        self.assertEqual(response.data["current_asn"], 42)
        self.assertEqual(
            response.data["document_file_type_counts"][0]["mime_type_count"],
            1,
@@ -1842,6 +1948,45 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        self.assertEqual(response.data["documents_inbox"], None)
        self.assertEqual(response.data["inbox_tags"], None)

+    def test_statistics_current_asn_respects_document_permissions(self) -> None:
+        viewer = UserFactory()
+        grant_global(viewer, "view_document")
+        DocumentFactory(owner=viewer, archive_serial_number=12)
+        shared = DocumentFactory(owner=self.user, archive_serial_number=23)
+        grant_object(viewer, shared, "view_document")
+        DocumentFactory(owner=self.user, archive_serial_number=99)
+
+        self.client.force_authenticate(user=viewer)
+        response = self.client.get("/api/statistics/")
+
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data["current_asn"], 23)
+
+    def test_statistics_current_asn_without_visible_documents(self) -> None:
+        DocumentFactory(owner=self.user, archive_serial_number=99)
+
+        for permissions in ((), ("view_document",)):
+            viewer = UserFactory()
+            grant_global(viewer, *permissions)
+            self.client.force_authenticate(user=viewer)
+
+            with self.subTest(permissions=permissions):
+                response = self.client.get("/api/statistics/")
+
+                self.assertEqual(response.status_code, status.HTTP_200_OK)
+                self.assertEqual(response.data["current_asn"], 0)
+
+    def test_statistics_current_asn_with_global_statistics_permission(self) -> None:
+        viewer = UserFactory()
+        grant_global(viewer, "view_global_statistics")
+        DocumentFactory(owner=self.user, archive_serial_number=99)
+
+        self.client.force_authenticate(user=viewer)
+        response = self.client.get("/api/statistics/")
+
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        self.assertEqual(response.data["current_asn"], 99)
+
    def test_statistics_multiple_users(self) -> None:
        """
        GIVEN:
@@ -25,6 +25,7 @@ from documents.models import Tag
 from documents.models import Workflow
 from documents.search import get_backend
 from documents.search import reset_backend
+from documents.views import DocumentViewSet
 from paperless_mail.models import MailAccount
 from paperless_mail.models import MailRule
 from paperless_testing.dirs import DirectoriesMixin
@@ -345,38 +346,39 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
        storage_path = StoragePath.objects.create(name="sp1")
        tag = Tag.objects.create(name="tag")

-        matching_doc = Document.objects.create(
-            title="bank statement",
-            content="bank content",
-            checksum="A",
-            correspondent=correspondent,
-            document_type=doc_type,
-            storage_path=storage_path,
-        )
-        matching_doc.tags.add(tag)
+        for checksum, title in (
+            ("A", "bank statement"),
+            ("B", "bank account"),
+            ("C", "credit card"),
+        ):
+            doc = Document.objects.create(
+                title=title,
+                checksum=checksum,
+                correspondent=correspondent,
+                document_type=doc_type,
+                storage_path=storage_path,
+            )
+            doc.tags.add(tag)
+            get_backend().add_or_update(doc)

-        get_backend().add_or_update(matching_doc)
-
-        response = self.client.get(
-            "/api/documents/?query=bank&include_selection_data=true",
-        )
+        with mock.patch.object(DocumentViewSet, "selection_data_batch_size", 1):
+            response = self.client.get(
+                "/api/documents/?query=bank&page_size=1&include_selection_data=true",
+            )

        self.assertEqual(response.status_code, status.HTTP_200_OK)
-        self.assertIn("selection_data", response.data)
-
-        selected_correspondent = next(
-            item
-            for item in response.data["selection_data"]["selected_correspondents"]
-            if item["id"] == correspondent.id
-        )
-        selected_tag = next(
-            item
-            for item in response.data["selection_data"]["selected_tags"]
-            if item["id"] == tag.id
-        )
-
-        self.assertEqual(selected_correspondent["document_count"], 1)
-        self.assertEqual(selected_tag["document_count"], 1)
+        self.assertEqual(response.data["count"], 2)
+        self.assertEqual(len(response.data["results"]), 1)
+        for key, pk in (
+            ("selected_correspondents", correspondent.pk),
+            ("selected_document_types", doc_type.pk),
+            ("selected_storage_paths", storage_path.pk),
+            ("selected_tags", tag.pk),
+        ):
+            self.assertEqual(
+                response.data["selection_data"][key],
+                [{"id": pk, "document_count": 2}],
+            )

    def test_search_custom_field_ordering(self) -> None:
        custom_field = CustomField.objects.create(
@@ -5,11 +5,13 @@ import platform
 import re
 import tempfile
 import zipfile
+from collections import Counter
 from collections import deque
 from collections.abc import Callable
 from datetime import datetime
 from datetime import timedelta
 from http import HTTPStatus
+from itertools import islice
 from pathlib import Path
 from time import mktime
 from time import sleep
@@ -168,7 +170,6 @@ from documents.permissions import PaperlessNotePermissions
 from documents.permissions import PaperlessObjectPermissions
 from documents.permissions import TrashPermissions
 from documents.permissions import ViewDocumentsPermissions
-from documents.permissions import annotate_document_count_by_ids
 from documents.permissions import annotate_document_count_for_related_queryset
 from documents.permissions import get_document_count_filter_for_user
 from documents.permissions import get_permission_grantees
@@ -1014,71 +1015,80 @@ class DocumentViewSet(
        "custom_field_",
    )

-    def _get_selection_data_for_queryset(self, queryset):
-        # Resolve once instead of once per model below. `queryset` can carry an
-        # arbitrarily expensive WHERE clause (user filters plus the permission
-        # filter); re-embedding it as a subquery inside 5 separate Count(...)
-        # calls forces the database to re-evaluate that whole thing 5 times, and
-        # -- for FK relations especially -- can defeat semi-join planning
-        # entirely at scale. A concrete id list is cheap to reuse.
-        # order_by() drops the default/user ordering -- irrelevant for a plain
-        # id list, but left in place it forces a sort over the full filtered
-        # set before the ids can even be collected.
-        document_ids = list(queryset.order_by().values_list("pk", flat=True))
+    # Bound the IN clauses used for search results and through-table lookups.
+    selection_data_batch_size = 10_000

-        correspondents = Correspondent.objects.annotate(
-            document_count=Count(
-                "documents",
-                filter=Q(documents__id__in=document_ids),
-                distinct=True,
-            ),
-        )
-        document_types = DocumentType.objects.annotate(
-            document_count=Count(
-                "documents",
-                filter=Q(documents__id__in=document_ids),
-                distinct=True,
-            ),
-        )
-        storage_paths = StoragePath.objects.annotate(
-            document_count=Count(
-                "documents",
-                filter=Q(documents__id__in=document_ids),
-                distinct=True,
-            ),
-        )
-        # Tag and CustomField reach Document through an M2M/through-model table;
-        # a plain Count(filter=...) there is a much more expensive plan than the
-        # FK relations above once the bridge table is large -- see
-        # annotate_document_count_by_ids() for why.
-        tags = annotate_document_count_by_ids(
-            Tag.objects.all(),
-            through_model=Document.tags.through,
-            related_object_field="tag_id",
-            document_ids=document_ids,
-        )
-        custom_fields = annotate_document_count_by_ids(
-            CustomField.objects.all(),
-            through_model=CustomFieldInstance,
-            related_object_field="field_id",
-            document_ids=document_ids,
+    def _get_selection_data_for_queryset(
+        self,
+        queryset: QuerySet[Document],
+        *,
+        search_ids: list[int] | None = None,
+    ) -> dict[str, list[dict[str, int]]]:
+        # Keep permission and filter predicates at document granularity.
+        # Reusing the filtered queryset as a subquery for each count is slow.
+        rows = (
+            queryset.order_by()
+            .prefetch_related(None)
+            .values_list(
+                "pk",
+                "correspondent_id",
+                "document_type_id",
+                "storage_path_id",
+            )
        )
+        batch_size = self.selection_data_batch_size
+
+        def document_rows():
+            if search_ids is None:
+                yield from rows.iterator()
+            else:
+                for start in range(0, len(search_ids), batch_size):
+                    yield from rows.filter(
+                        pk__in=search_ids[start : start + batch_size],
+                    ).iterator()
+
+        correspondents: Counter[int | None] = Counter()
+        document_types: Counter[int | None] = Counter()
+        storage_paths: Counter[int | None] = Counter()
+        document_ids: set[int] = set()
+        for pk, correspondent, document_type, storage_path in document_rows():
+            # Filters that join multi-valued relations can repeat a document.
+            if pk in document_ids:
+                continue
+            document_ids.add(pk)
+            correspondents[correspondent] += 1
+            document_types[document_type] += 1
+            storage_paths[storage_path] += 1
+
+        tags: Counter[int | None] = Counter()
+        custom_fields: Counter[int | None] = Counter()
+        ids = iter(document_ids)
+        while batch := list(islice(ids, batch_size)):
+            for model, field, counts in (
+                (Document.tags.through, "tag_id", tags),
+                (CustomFieldInstance, "field_id", custom_fields),
+            ):
+                counts.update(
+                    dict(
+                        model.objects.filter(document_id__in=batch)
+                        .order_by()
+                        .values_list(field)
+                        .annotate(document_count=Count("document_id")),
+                    ),
+                )
+
+        def selection(model: type[Model], counts: Counter[int | None]):
+            return [
+                {"id": pk, "document_count": counts[pk]}
+                for pk in model._default_manager.values_list("pk", flat=True)
+            ]
+
        return {
-            "selected_correspondents": [
-                {"id": t.id, "document_count": t.document_count} for t in correspondents
-            ],
-            "selected_tags": [
-                {"id": t.id, "document_count": t.document_count} for t in tags
-            ],
-            "selected_document_types": [
-                {"id": t.id, "document_count": t.document_count} for t in document_types
-            ],
-            "selected_storage_paths": [
-                {"id": t.id, "document_count": t.document_count} for t in storage_paths
-            ],
-            "selected_custom_fields": [
-                {"id": t.id, "document_count": t.document_count} for t in custom_fields
-            ],
+            "selected_correspondents": selection(Correspondent, correspondents),
+            "selected_tags": selection(Tag, tags),
+            "selected_document_types": selection(DocumentType, document_types),
+            "selected_storage_paths": selection(StoragePath, storage_paths),
+            "selected_custom_fields": selection(CustomField, custom_fields),
        }

    @classmethod
@@ -2725,13 +2735,10 @@ class UnifiedSearchViewSet(DocumentViewSet):
                if get_boolean(
                    str(request.query_params.get("include_selection_data", "false")),
                ):
-                    # NOTE: pk__in=ordered_ids generates a large SQL IN clause
-                    # for big result sets.  Acceptable today but may need a temp
-                    # table or chunked approach if selection_data becomes slow
-                    # at scale (tens of thousands of matching documents).
                    response.data["selection_data"] = (
                        self._get_selection_data_for_queryset(
-                            filtered_qs.filter(pk__in=result.ordered_ids),
+                            filtered_qs,
+                            search_ids=result.ordered_ids,
                        )
                    )
                return response
@@ -4071,7 +4078,7 @@ class StatisticsView(GenericAPIView[Any]):
            for row in mime_type_stats
        ]

-        current_asn = Document.objects.aggregate(
+        current_asn = documents.aggregate(
            Max("archive_serial_number", default=0),
        ).get(
            "archive_serial_number__max",
@@ -2,7 +2,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: paperless-ngx\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2026-11-06 20:19+0000\n"
+"POT-Creation-Date: 2026-11-07 02:45+0000\n"
 "PO-Revision-Date: 2022-02-17 04:17\n"
 "Last-Translator: \n"
 "Language-Team: English\n"
@@ -1661,8 +1661,8 @@ msgid "workflow runs"
 msgstr ""

 #: documents/serialisers.py:483 documents/serialisers.py:844
-#: documents/serialisers.py:2850 documents/views.py:348 documents/views.py:614
-#: documents/views.py:2748 paperless_mail/serialisers.py:156
+#: documents/serialisers.py:2850 documents/views.py:349 documents/views.py:615
+#: documents/views.py:2755 paperless_mail/serialisers.py:156
 msgid "Insufficient permissions."
 msgstr ""

@@ -1702,7 +1702,7 @@ msgstr ""
 msgid "Duplicate document identifiers are not allowed."
 msgstr ""

-#: documents/serialisers.py:2936 documents/views.py:4798
+#: documents/serialisers.py:2936 documents/views.py:4805
 #, python-format
 msgid "Documents not found: %(ids)s"
 msgstr ""
@@ -1953,40 +1953,40 @@ msgstr ""
 msgid ", "
 msgstr ""

-#: documents/views.py:303
+#: documents/views.py:304
 msgid "Specify only one of text, title_search, query, or more_like_id."
 msgstr ""

-#: documents/views.py:341 documents/views.py:2745
+#: documents/views.py:342 documents/views.py:2752
 msgid "Invalid more_like_id"
 msgstr ""

-#: documents/views.py:1669
+#: documents/views.py:1679
 msgid "Invalid AI configuration."
 msgstr ""

-#: documents/views.py:1680
+#: documents/views.py:1690
 msgid "AI backend request timed out."
 msgstr ""

-#: documents/views.py:1692
+#: documents/views.py:1702
 msgid "AI backend rejected the request. Check logs for details."
 msgstr ""

-#: documents/views.py:4814
+#: documents/views.py:4821
 #, python-format
 msgid "Insufficient permissions to share document %(id)s."
 msgstr ""

-#: documents/views.py:4861
+#: documents/views.py:4868
 msgid "Bundle is already being processed."
 msgstr ""

-#: documents/views.py:4927
+#: documents/views.py:4934
 msgid "The share link bundle is still being prepared. Please try again later."
 msgstr ""

-#: documents/views.py:4941
+#: documents/views.py:4948
 msgid "The share link bundle is unavailable."
 msgstr ""

@@ -450,23 +450,34 @@ CORS_EXPOSE_HEADERS = [
    "Content-Disposition",
 ]

-ALLOWED_HOSTS = get_list_from_env("PAPERLESS_ALLOWED_HOSTS", default=["*"])
-if ALLOWED_HOSTS != ["*"]:
-    # always allow localhost. Necessary e.g. for healthcheck in docker.
-    ALLOWED_HOSTS.append("localhost")
+
+def _get_allowed_hosts() -> list[str]:
+    if "PAPERLESS_ALLOWED_HOSTS" in os.environ or os.getenv("PAPERLESS_URL"):
+        allowed_hosts = get_list_from_env("PAPERLESS_ALLOWED_HOSTS")
+    else:
+        allowed_hosts = ["*"]
+    if allowed_hosts != ["*"]:
+        # Always allow localhost for the Docker health check.
+        allowed_hosts.append("localhost")
+    return allowed_hosts
+
+
+ALLOWED_HOSTS = _get_allowed_hosts()


 def _parse_paperless_url():
    global CSRF_TRUSTED_ORIGINS, CORS_ALLOWED_ORIGINS, ALLOWED_HOSTS
    url = os.getenv("PAPERLESS_URL")
    if url:
-        hostname = urlparse(url).hostname
-        if not hostname:
+        hostname = (urlparse(url).hostname or "").removesuffix(".")
+        if not hostname or hostname == "*" or hostname.startswith("."):
            raise ImproperlyConfigured(
-                f"PAPERLESS_URL {url!r} has no hostname, expected a URL like https://example.com",
+                f"PAPERLESS_URL {url!r} has no valid hostname, expected a URL like https://example.com",
            )
        CSRF_TRUSTED_ORIGINS.append(url)
        CORS_ALLOWED_ORIGINS.append(url)
+        if ":" in hostname and not hostname.startswith("["):
+            hostname = f"[{hostname}]"  # Add brackets around IPv6 addresses
        ALLOWED_HOSTS.append(hostname)

    return url
@@ -3,9 +3,14 @@ from unittest import TestCase
 from unittest import mock

 import pytest
+from django.core.exceptions import DisallowedHost
 from django.core.exceptions import ImproperlyConfigured
+from django.http import HttpRequest
+from django.test import override_settings

+import paperless.settings as paperless_settings
 from paperless.settings import _get_allauth_trusted_proxy_count
+from paperless.settings import _get_allowed_hosts
 from paperless.settings import _get_classifier_language_setting
 from paperless.settings import _get_search_language_setting
 from paperless.settings import _parse_paperless_url
@@ -168,8 +173,88 @@ class TestPaperlessURLSettings(TestCase):
            self.assertIn(url, settings.CORS_ALLOWED_ORIGINS)


-@pytest.mark.parametrize("url", ["example.com", "https://", "/just/a/path"])
-def test_paperless_url_without_hostname_is_rejected(
+@pytest.mark.parametrize(
+    ("url", "allowed_hosts", "expected", "accepted_host", "rejected_host"),
+    [
+        (None, None, ["*"], "other.example.com", None),
+        (None, "", ["localhost"], "localhost:8000", "other.example.com"),
+        (
+            None,
+            "lan.example.com",
+            ["lan.example.com", "localhost"],
+            "lan.example.com",
+            "other.example.com",
+        ),
+        (None, "*", ["*"], "other.example.com", None),
+        (
+            "https://example.com:8443",
+            None,
+            ["localhost", "example.com"],
+            "example.com:8443",
+            "other.example.com",
+        ),
+        (
+            "https://example.com.",
+            "",
+            ["localhost", "example.com"],
+            "example.com.",
+            "other.example.com",
+        ),
+        (
+            "http://[::1]:8000",
+            None,
+            ["localhost", "[::1]"],
+            "[::1]:8000",
+            "other.example.com",
+        ),
+        ("https://example.com", "*", ["*", "example.com"], "other.example.com", None),
+        (
+            "https://example.com",
+            "lan.example.com",
+            ["lan.example.com", "localhost", "example.com"],
+            "lan.example.com",
+            "other.example.com",
+        ),
+    ],
+)
+def test_allowed_hosts_at_startup(
+    monkeypatch: pytest.MonkeyPatch,
+    url: str | None,
+    allowed_hosts: str | None,
+    expected: list[str],
+    accepted_host: str,
+    rejected_host: str | None,
+) -> None:
+    for name, value in (
+        ("PAPERLESS_URL", url),
+        ("PAPERLESS_ALLOWED_HOSTS", allowed_hosts),
+    ):
+        if value is None:
+            monkeypatch.delenv(name, raising=False)
+        else:
+            monkeypatch.setenv(name, value)
+
+    monkeypatch.setattr(paperless_settings, "ALLOWED_HOSTS", _get_allowed_hosts())
+    monkeypatch.setattr(paperless_settings, "CSRF_TRUSTED_ORIGINS", [])
+    monkeypatch.setattr(paperless_settings, "CORS_ALLOWED_ORIGINS", [])
+    assert _parse_paperless_url() == url
+    assert expected == paperless_settings.ALLOWED_HOSTS
+
+    request = HttpRequest()
+    with override_settings(ALLOWED_HOSTS=expected):
+        request.META["HTTP_HOST"] = accepted_host
+        assert request.get_host() == accepted_host
+        if rejected_host:
+            request.META["HTTP_HOST"] = rejected_host
+            with pytest.raises(DisallowedHost):
+                request.get_host()
+
+
+@pytest.mark.parametrize(
+    "url",
+    ["example.com", "https://", "/just/a/path", "https://*", "https://.example.com"],
+)
+def test_paperless_url_without_valid_hostname_is_rejected(
    monkeypatch: pytest.MonkeyPatch,
    url: str,
 ) -> None:
@@ -179,9 +264,9 @@ def test_paperless_url_without_hostname_is_rejected(
    WHEN:
        - The URL is parsed
    THEN:
-        - A configuration error is raised instead of adding None to ALLOWED_HOSTS
+        - A configuration error is raised for missing or wildcard hostnames
    """
    monkeypatch.setenv("PAPERLESS_URL", url)

-    with pytest.raises(ImproperlyConfigured, match="no hostname"):
+    with pytest.raises(ImproperlyConfigured, match="no valid hostname"):
        _parse_paperless_url()
@@ -32,13 +32,13 @@ from rest_framework.fields import BooleanField
 from rest_framework.filters import OrderingFilter
 from rest_framework.generics import GenericAPIView
 from rest_framework.pagination import PageNumberPagination
-from rest_framework.permissions import DjangoModelPermissions
 from rest_framework.permissions import IsAuthenticated
 from rest_framework.response import Response
 from rest_framework.throttling import ScopedRateThrottle
 from rest_framework.viewsets import ModelViewSet

 from documents.models import PaperlessTask
+from documents.permissions import PaperlessModelPermissions
 from documents.permissions import PaperlessObjectPermissions
 from documents.tasks import llmindex_index
 from paperless.filters import GroupFilterSet
@@ -414,7 +414,7 @@ class ApplicationConfigurationViewSet(ModelViewSet[ApplicationConfiguration]):
    queryset = ApplicationConfiguration.objects

    serializer_class = ApplicationConfigurationSerializer
-    permission_classes = (IsAuthenticated, DjangoModelPermissions)
+    permission_classes = (IsAuthenticated, PaperlessModelPermissions)

    @extend_schema(exclude=True)
    def create(self, request, *args, **kwargs):
@@ -22,8 +22,8 @@ logger = logging.getLogger("paperless_ai.chat")
 CHAT_METADATA_DELIMITER = "\n\n__PAPERLESS_CHAT_METADATA__"
 CHAT_ERROR_MESSAGE = "Sorry, something went wrong while generating a response."
 CHAT_NO_CONTENT_MESSAGE = "Sorry, I couldn't find any content to answer your question."
-MAX_CHAT_REFERENCES = 3
 CHAT_RETRIEVER_TOP_K = 5
+MAX_CHAT_REFERENCES = CHAT_RETRIEVER_TOP_K


 def _build_chat_prompt(output_language: str | None) -> str:
Author	SHA1	Message	Date
shamoon	d618a61788	Make testing easier	2026-11-10 15:21:06 -08:00
shamoon	7b3daa313a	Fix: fix allowed_hosts default	2026-11-10 07:53:53 -08:00
GitHub Actions	936b1b3bfa	Auto translate strings	2026-11-09 06:53:56 +00:00
shamoon	e0c33b3d39	Tweak: chat reference tweaks (#14723 )	2026-11-08 22:52:34 -08:00
shamoon	32bc5522e9	Update SECURITY.md	2026-11-07 12:24:29 -08:00
GitHub Actions	d5698329f5	Auto translate strings	2026-11-07 20:22:08 +00:00
shamoon	2e0d92d8f5	Fix: two dropdown-related UI fixes (#14717 )	2026-11-07 12:20:31 -08:00
shamoon	0736c07d8e	Fix: use filtered documents for statistics ASN max (#14710 )	2026-11-07 07:52:23 -08:00
shamoon	53d9aa34a1	FIx: make config view permissions consistent (#14709 )	2026-11-07 07:52:12 -08:00
GitHub Actions	19fdc3b815	Auto translate strings	2026-11-07 02:45:43 +00:00
Trenton H Claude Sonnet 5.5 Codex shamoon	5357be2d43	Performance: stop embedding every document ID in selection_data queries (#14597 ) * Fix: count selection_data from one pass over the filtered documents Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Chore: use document type ids in the bulk edit selection_data test Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Batch search selection data queries Co-Authored-By: Codex <267193182+codex@users.noreply.github.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com> Co-authored-by: Codex <267193182+codex@users.noreply.github.com>	2026-11-06 18:44:18 -08:00