New Crowdin translations by GitHub Action

* Perf: batch the repeated lookups in modify_custom_fields

`modify_custom_fields()` re-resolved the same objects inside its
per-document loop: `custom_fields.get(id=field_id)` re-ran a CustomField
query for every document, and doc link fields called
`Document.objects.get(id=doc_id)` a second time for a document that was
already known.

Resolve both up front with `in_bulk()` and hand the resolved objects to
`update_or_create()` rather than bare ids. Passing the objects also
populates the FK cache on the newly created instance, so auditlog's
post_save receiver touching `.document`/`.field` no longer costs a reload
per row. The document map defers `content`, the one field here that is
both large and unused. The symmetrical-link removal pass and
`remove_doclink()` get `select_related()` for the same auditlog reason.

Measured over 50 documents, sqlite, audit log enabled:

                        before   after
  add 3 string fields    1502    1054
  update 1 string field   451     403
  add doc link            851     653
  remove doc link         604     354

`update_or_create()` is kept as-is. Dropping it for a hand-rolled
get-or-construct loop removes a further ~4 statements per row, but those
are the SAVEPOINT/RELEASE pairs of its `transaction.atomic()`, and the
`select_for_update()` and IntegrityError fallback that go with them. The
(document, field) unique constraint depends on that when two bulk edits
overlap, and the wall clock did not move to pay for it (331 ms vs 310 ms
for the string case above).

Also normalises the field ids to int once at the top so the old dict API,
whose keys may arrive as strings, indexes the resolved map correctly.

The `if custom_field:` branch it replaces was dead: `.get()` raises
DoesNotExist, it never returns None.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fixes the comment

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

docs/configuration.md

@@ -1200,6 +1200,15 @@ still perform some basic text pre-processing before matching.
 
     Defaults to true, enabling the feature.
 
+#### [`PAPERLESS_CLASSIFIER_MATCH_THRESHOLD=<float>`](#PAPERLESS_CLASSIFIER_MATCH_THRESHOLD) {#PAPERLESS_CLASSIFIER_MATCH_THRESHOLD}
+
+: Sets the minimum confidence score (0.0-1.0) required for the automatic
+classifier to assign a correspondent, document type, or storage path to a
+document. Predictions below this threshold are discarded and the field is
+left unassigned, preventing low-confidence guesses from being applied.
+
+    Defaults to 0.6.
+
 #### [`PAPERLESS_DATE_PARSER_LANGUAGES=<lang>`](#PAPERLESS_DATE_PARSER_LANGUAGES) {#PAPERLESS_DATE_PARSER_LANGUAGES}
 
 : Specifies which language Paperless should use when parsing dates from documents.

src/documents/bulk_edit.py

@@ -2,6 +2,7 @@ from __future__ import annotations
 
 import logging
 import tempfile
+import uuid
 from pathlib import Path
 from typing import TYPE_CHECKING
 from typing import Literal
@@ -298,53 +299,55 @@ def modify_custom_fields(
 ) -> Literal["OK"]:
     qs = Document.objects.filter(id__in=doc_ids).only("pk")
     affected_docs = list(qs.values_list("pk", flat=True))
-    # Ensure add_custom_fields is a list of tuples, supports old API
+    # Ensure add_custom_fields is a list of (int, value) tuples, supports old API
     add_custom_fields = (
-        add_custom_fields.items()
+        [(int(field), value) for field, value in add_custom_fields.items()]
         if isinstance(add_custom_fields, dict)
-        else [(field, None) for field in add_custom_fields]
+        else [(int(field), None) for field in add_custom_fields]
     )
 
-    custom_fields = CustomField.objects.filter(
+    # Resolved once, instead of re-querying the same field for every document
-        id__in=[int(field) for field, _ in add_custom_fields],
+    custom_fields_by_id: dict[int, CustomField] = CustomField.objects.in_bulk(
-    ).distinct()
+        [field_id for field_id, _ in add_custom_fields],
+    )
+    # Passed to update_or_create() below rather than a bare id, so the FK is
+    # cached on the created instance and auditlog's post_save receiver does
+    # not reload it per row. Only needed for additions. content is deferred:
+    # the one field here that is both large and unused.
+    docs_by_id: dict[int, Document] = (
+        Document.objects.defer("content").in_bulk(affected_docs)
+        if add_custom_fields
+        else {}
+    )
     for field_id, value in add_custom_fields:
-        for doc_id in affected_docs:
+        custom_field = custom_fields_by_id[field_id]
-            defaults = {}
-            custom_field = custom_fields.get(id=field_id)
-            if custom_field:
         value_field = CustomFieldInstance.TYPE_TO_DATA_STORE_NAME_MAP[
             custom_field.data_type
         ]
-                defaults[value_field] = value
+        is_doclink = custom_field.data_type == CustomField.FieldDataType.DOCUMENTLINK
-                if (
+        for doc_id in affected_docs:
-                    custom_field.data_type == CustomField.FieldDataType.DOCUMENTLINK
+            if is_doclink and value and doc_id in value:
-                    and value
-                    and doc_id in value
-                ):
                 # Prevent self-linking
                 continue
             CustomFieldInstance.objects.update_or_create(
-                document_id=doc_id,
+                document=docs_by_id[doc_id],
-                field_id=field_id,
+                field=custom_field,
-                defaults=defaults,
+                defaults={value_field: value},
             )
-            if custom_field.data_type == CustomField.FieldDataType.DOCUMENTLINK:
+            if is_doclink:
-                doc = Document.objects.get(id=doc_id)
+                reflect_doclinks(docs_by_id[doc_id], custom_field, value)
-                reflect_doclinks(doc, custom_field, value)
 
-    # For doc link fields that are being removed, remove symmetrical links
+    # For doc link fields that are being removed, remove symmetrical links.
+    # select_related avoids a per-instance reload of the document and field.
     for doclink_being_removed_instance in CustomFieldInstance.objects.filter(
         document_id__in=affected_docs,
         field__id__in=remove_custom_fields,
         field__data_type=CustomField.FieldDataType.DOCUMENTLINK,
         value_document_ids__isnull=False,
-    ):
+    ).select_related("field", "document"):
         for target_doc_id in doclink_being_removed_instance.value:
             remove_doclink(
-                document=Document.objects.get(
+                document=doclink_being_removed_instance.document,
-                    id=doclink_being_removed_instance.document.id,
-                ),
                 field=doclink_being_removed_instance.field,
                 target_doc_id=target_doc_id,
             )
@@ -379,7 +382,7 @@ def delete(doc_ids: list[int]) -> Literal["OK"]:
         )
         delete_ids = list({*doc_ids, *version_ids})
 
-        Document.objects.filter(id__in=delete_ids).delete()
+        Document.objects.filter(id__in=delete_ids).delete(transaction_id=uuid.uuid4())
 
         from documents.search import get_backend
 
@@ -1177,10 +1180,13 @@ def remove_doclink(
     """
     Removes a 'symmetrical' link to `document` from the target document's existing custom field instance
     """
-    target_doc_field_instance = CustomFieldInstance.objects.filter(
+    # select_related: a signal receiver (auditlog) touches .document/.field on
-        document_id=target_doc_id,
+    # the save() below, without this that is a per-call reload query
-        field=field,
+    target_doc_field_instance = (
-    ).first()
+        CustomFieldInstance.objects.filter(document_id=target_doc_id, field=field)
+        .select_related("document", "field")
+        .first()
+    )
     if (
         target_doc_field_instance is not None
         and document.id in target_doc_field_instance.value

src/documents/classifier.py

@@ -34,6 +34,27 @@ from paperless.signed_pickle import signed_pickle_loads
 
 logger = logging.getLogger("paperless.classifier")
 
+
+def _predict_with_threshold(classifier, X, threshold: float) -> int | None:
+    """
+    Return the predicted class id, or None if:
+    - the prediction is -1 (no match), or
+    - the winning class probability is below the configured threshold.
+
+    Using predict_proba() instead of predict() lets us apply a minimum-confidence
+    cutoff so that uncertain predictions are discarded rather than assigned.
+    """
+    probas = classifier.predict_proba(X)[0]
+    best_idx = int(probas.argmax())
+    best_class = int(classifier.classes_[best_idx])
+
+    if best_class == -1:
+        return None
+    if threshold > 0.0 and probas[best_idx] < threshold:
+        return None
+    return best_class
+
+
 ADVANCED_TEXT_PROCESSING_ENABLED = (
     settings.NLTK_LANGUAGE is not None and settings.NLTK_ENABLED
 )
@@ -102,7 +123,8 @@ class DocumentClassifier:
     # v8 - Added storage path classifier
     # v9 - Changed from hashing to time/ids for re-train check
     # v10 - HMAC-signed model file
-    FORMAT_VERSION = 10
+    # v11 - Use sample_weight for balanced training; predict_proba with threshold
+    FORMAT_VERSION = 11
 
     HMAC_SIZE = 32  # SHA-256 digest length
 
@@ -324,6 +346,13 @@ class DocumentClassifier:
         from sklearn.preprocessing import LabelBinarizer
         from sklearn.preprocessing import MultiLabelBinarizer
 
+        # MLPClassifier does not support class_weight directly
+        # (https://github.com/scikit-learn/scikit-learn/issues/9113), so we use
+        # compute_sample_weight to balance classes during training and prevent
+        # over-represented correspondents from dominating predictions.
+        # https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_sample_weight.html
+        from sklearn.utils.class_weight import compute_sample_weight
+
         # Step 2: vectorize data
         logger.debug("Vectorizing data...")
         notify("Vectorizing document content...")
@@ -369,7 +398,7 @@ class DocumentClassifier:
                 self.tags_binarizer = MultiLabelBinarizer()
                 labels_tags_vectorized = self.tags_binarizer.fit_transform(labels_tags)
 
-            self.tags_classifier = MLPClassifier(tol=0.01)
+            self.tags_classifier = MLPClassifier(tol=0.01, random_state=0)
             self.tags_classifier.fit(data_vectorized, labels_tags_vectorized)
         else:
             self.tags_classifier = None
@@ -380,8 +409,12 @@ class DocumentClassifier:
             notify(
                 f"Training correspondent classifier ({num_correspondents} correspondent(s))...",
             )
-            self.correspondent_classifier = MLPClassifier(tol=0.01)
+            self.correspondent_classifier = MLPClassifier(tol=0.01, random_state=0)
-            self.correspondent_classifier.fit(data_vectorized, labels_correspondent)
+            self.correspondent_classifier.fit(
+                data_vectorized,
+                labels_correspondent,
+                sample_weight=compute_sample_weight("balanced", labels_correspondent),
+            )
         else:
             self.correspondent_classifier = None
             logger.debug(
@@ -393,8 +426,12 @@ class DocumentClassifier:
             notify(
                 f"Training document type classifier ({num_document_types} type(s))...",
             )
-            self.document_type_classifier = MLPClassifier(tol=0.01)
+            self.document_type_classifier = MLPClassifier(tol=0.01, random_state=0)
-            self.document_type_classifier.fit(data_vectorized, labels_document_type)
+            self.document_type_classifier.fit(
+                data_vectorized,
+                labels_document_type,
+                sample_weight=compute_sample_weight("balanced", labels_document_type),
+            )
         else:
             self.document_type_classifier = None
             logger.debug(
@@ -406,10 +443,11 @@ class DocumentClassifier:
                 "Training storage paths classifier...",
             )
             notify(f"Training storage path classifier ({num_storage_paths} path(s))...")
-            self.storage_path_classifier = MLPClassifier(tol=0.01)
+            self.storage_path_classifier = MLPClassifier(tol=0.01, random_state=0)
             self.storage_path_classifier.fit(
                 data_vectorized,
                 labels_storage_path,
+                sample_weight=compute_sample_weight("balanced", labels_storage_path),
             )
         else:
             self.storage_path_classifier = None
@@ -546,23 +584,23 @@ class DocumentClassifier:
     def predict_correspondent(self, content: str) -> int | None:
         if self.correspondent_classifier:
             X = self._vectorize(content)
-            correspondent_id = self.correspondent_classifier.predict(X)
+            predicted_id = _predict_with_threshold(
-            if correspondent_id != -1:
+                self.correspondent_classifier,
-                return correspondent_id
+                X,
-            else:
+                settings.CLASSIFIER_MATCH_THRESHOLD,
-                return None
+            )
-        else:
+            return predicted_id
         return None
 
     def predict_document_type(self, content: str) -> int | None:
         if self.document_type_classifier:
             X = self._vectorize(content)
-            document_type_id = self.document_type_classifier.predict(X)
+            predicted_id = _predict_with_threshold(
-            if document_type_id != -1:
+                self.document_type_classifier,
-                return document_type_id
+                X,
-            else:
+                settings.CLASSIFIER_MATCH_THRESHOLD,
-                return None
+            )
-        else:
+            return predicted_id
         return None
 
     def predict_tags(self, content: str) -> list[int]:
@@ -589,10 +627,10 @@ class DocumentClassifier:
     def predict_storage_path(self, content: str) -> int | None:
         if self.storage_path_classifier:
             X = self._vectorize(content)
-            storage_path_id = self.storage_path_classifier.predict(X)
+            predicted_id = _predict_with_threshold(
-            if storage_path_id != -1:
+                self.storage_path_classifier,
-                return storage_path_id
+                X,
-            else:
+                settings.CLASSIFIER_MATCH_THRESHOLD,
-                return None
+            )
-        else:
+            return predicted_id
         return None

src/documents/management/commands/document_consumer.py

@@ -156,6 +156,15 @@ class FileStabilityTracker:
                     logger.debug(f"File disappeared during stability check: {path}")
                 continue
 
+            # Stable, but empty: some scanners create a zero byte placeholder
+            # and only write the page some time later. Consuming it now can
+            # only fail  so drop it and let the writer's next event
+            # (or the periodic rescan) bring it back once it has content
+            if not tracked.last_size:
+                to_remove.append(path)
+                logger.debug("Ignoring stable but empty file: %s", path)
+                continue
+
             # File is stable, we can return it
             to_yield.append(path)
             logger.info(f"File is stable: {path}")

src/documents/models.py

@@ -1,4 +1,5 @@
 import datetime
+import uuid
 from pathlib import Path
 from typing import Final
 
@@ -514,13 +515,20 @@ class Document(SoftDeleteModel, ModelWithOwner):  # type: ignore[django-manager-
     def delete(
         self,
         *args,
+        transaction_id=None,
         **kwargs,
     ):
-        # If deleting a root document, move all its versions to trash as well.
+        # Versions must share the root's transaction ID so they are restored
+        # together by django-softdelete.
+        if transaction_id is None:
+            transaction_id = uuid.uuid4()
         if self.root_document_id is None:
-            Document.objects.filter(root_document=self).delete()
+            Document.objects.filter(root_document=self).delete(
+                transaction_id=transaction_id,
+            )
         return super().delete(
             *args,
+            transaction_id=transaction_id,
             **kwargs,
         )
 

src/documents/tests/test_api_trash.py

@@ -207,3 +207,65 @@ class TestTrashAPI(DirectoriesMixin, APITestCase):
         )
         self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
         self.assertIn("have not yet been deleted", resp.data["documents"][0])
+
+    def _make_versioned_document(self) -> tuple[Document, list[Document]]:
+        root = Document.objects.create(
+            title="root",
+            content="root-content",
+            checksum="root",
+            mime_type="application/pdf",
+        )
+        versions = [
+            Document.objects.create(
+                title=f"v{index}",
+                content=f"v{index}-content",
+                checksum=f"v{index}",
+                mime_type="application/pdf",
+                root_document=root,
+                version_index=index,
+            )
+            for index in range(1, 3)
+        ]
+        return root, versions
+
+    def test_api_trash_restore_document_restores_its_versions(self) -> None:
+        """
+        GIVEN:
+            - Existing document with two versions
+        WHEN:
+            - API request to delete the document
+            - API request to restore it from the trash
+        THEN:
+            - Only the document itself is listed in the trash
+            - A version cannot be restored without its root
+            - The document is restored together with all of its versions
+        """
+        root, versions = self._make_versioned_document()
+
+        self.client.force_login(user=self.user)
+        self.client.delete(f"/api/documents/{root.pk}/")
+        self.assertEqual(Document.deleted_objects.count(), 3)
+
+        resp = self.client.get("/api/trash/")
+        self.assertEqual(resp.status_code, status.HTTP_200_OK)
+        self.assertEqual(resp.data["count"], 1)
+        self.assertEqual(resp.data["results"][0]["id"], root.pk)
+
+        # A version cannot be restored while its root remains in the trash.
+        resp = self.client.post(
+            "/api/trash/",
+            {"action": "restore", "documents": [versions[0].pk]},
+        )
+        self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
+        self.assertIn("Restore the root document", resp.data["documents"][0])
+
+        resp = self.client.post(
+            "/api/trash/",
+            {"action": "restore", "documents": [root.pk]},
+        )
+        self.assertEqual(resp.status_code, status.HTTP_200_OK)
+        self.assertEqual(Document.deleted_objects.count(), 0)
+        self.assertCountEqual(
+            Document.objects.filter(root_document=root).values_list("id", flat=True),
+            [version.pk for version in versions],
+        )

src/documents/tests/test_bulk_edit.py

@@ -392,6 +392,11 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
         self.assertFalse(Document.objects.filter(id=self.doc1.id).exists())
         self.assertFalse(Document.objects.filter(id=version.id).exists())
 
+        Document.deleted_objects.get(id=self.doc1.id).restore(strict=False)
+
+        self.assertTrue(Document.objects.filter(id=self.doc1.id).exists())
+        self.assertTrue(Document.objects.filter(id=version.id).exists())
+
     def test_delete_version_document_keeps_root(self) -> None:
         version = Document.objects.create(
             checksum="A-v1",

src/documents/tests/test_classifier.py

@@ -3,6 +3,7 @@ import warnings
 from pathlib import Path
 from unittest import mock
 
+import numpy as np
 import pytest
 from django.conf import settings
 from django.test import TestCase
@@ -11,6 +12,7 @@ from django.test import override_settings
 from documents.classifier import ClassifierModelCorruptError
 from documents.classifier import DocumentClassifier
 from documents.classifier import IncompatibleClassifierVersionError
+from documents.classifier import _predict_with_threshold
 from documents.classifier import load_classifier
 from documents.models import Correspondent
 from documents.models import Document
@@ -625,6 +627,103 @@ class TestClassifier(DirectoriesMixin, TestCase):
         self.assertEqual(self.classifier.predict_storage_path(doc1.content), sp.pk)
         self.assertIsNone(self.classifier.predict_storage_path(doc2.content))
 
+    def test_predict_rejects_prediction_below_match_threshold(self) -> None:
+        """
+        GIVEN:
+            - Classifiers trained against test data with confident predictions
+        WHEN:
+            - CLASSIFIER_MATCH_THRESHOLD exceeds the model's confidence
+        THEN:
+            - Every predict_* method discards the match in favor of no match
+        """
+        c1 = Correspondent.objects.create(
+            name="c1",
+            matching_algorithm=Correspondent.MATCH_AUTO,
+        )
+        dt1 = DocumentType.objects.create(
+            name="dt1",
+            matching_algorithm=DocumentType.MATCH_AUTO,
+        )
+        sp1 = StoragePath.objects.create(
+            name="sp1",
+            matching_algorithm=StoragePath.MATCH_AUTO,
+        )
+
+        doc1 = Document.objects.create(
+            title="doc1",
+            content="this is a document from c1",
+            correspondent=c1,
+            document_type=dt1,
+            storage_path=sp1,
+            checksum="A",
+        )
+        Document.objects.create(
+            title="doc2",
+            content="this is a document from no one",
+            checksum="B",
+        )
+
+        self.classifier.train()
+
+        predictors = {
+            "correspondent": self.classifier.predict_correspondent,
+            "document_type": self.classifier.predict_document_type,
+            "storage_path": self.classifier.predict_storage_path,
+        }
+        # No real prediction can reach a confidence this high, so this
+        # isolates the threshold check from the model's actual output.
+        with override_settings(CLASSIFIER_MATCH_THRESHOLD=0.999999):
+            for name, predict in predictors.items():
+                with self.subTest(field=name):
+                    self.assertIsNone(predict(doc1.content))
+
+    def test_train_uses_balanced_sample_weight(self) -> None:
+        """
+        GIVEN:
+            - A training set with correspondents, document types and storage paths
+        WHEN:
+            - The classifier is trained
+        THEN:
+            - Each MLP classifier is fit with balanced sample weights, so that
+              over-represented classes don't dominate predictions
+        """
+        c1 = Correspondent.objects.create(
+            name="c1",
+            matching_algorithm=Correspondent.MATCH_AUTO,
+        )
+        dt1 = DocumentType.objects.create(
+            name="dt1",
+            matching_algorithm=DocumentType.MATCH_AUTO,
+        )
+        sp1 = StoragePath.objects.create(
+            name="sp1",
+            matching_algorithm=StoragePath.MATCH_AUTO,
+        )
+
+        Document.objects.create(
+            title="doc1",
+            content="this is a document from c1",
+            correspondent=c1,
+            document_type=dt1,
+            storage_path=sp1,
+            checksum="A",
+        )
+        Document.objects.create(
+            title="doc2",
+            content="this is a document from no one",
+            checksum="B",
+        )
+
+        with mock.patch(
+            "sklearn.utils.class_weight.compute_sample_weight",
+            return_value=None,
+        ) as mocked_compute_sample_weight:
+            self.classifier.train()
+
+        self.assertEqual(mocked_compute_sample_weight.call_count, 3)
+        for call in mocked_compute_sample_weight.call_args_list:
+            self.assertEqual(call.args[0], "balanced")
+
     def test_one_tag_predict(self) -> None:
         t1 = Tag.objects.create(name="t1", matching_algorithm=Tag.MATCH_AUTO, pk=12)
 
@@ -810,6 +909,52 @@ class TestClassifier(DirectoriesMixin, TestCase):
             load_classifier(raise_exception=True)
 
 
+class _StubProbaClassifier:
+    """
+    A fake scikit-learn classifier exposing just enough of the API for
+    `_predict_with_threshold`: `classes_` and `predict_proba`.
+    """
+
+    def __init__(self, classes: list[int], probabilities: list[float]) -> None:
+        self.classes_ = np.array(classes)
+        self._probabilities = np.array([probabilities])
+
+    def predict_proba(self, X) -> np.ndarray:
+        return self._probabilities
+
+
+@pytest.mark.parametrize(
+    ("classes", "probabilities", "threshold", "expected"),
+    [
+        # confident prediction above the threshold is returned
+        ([-1, 3], [0.1, 0.9], 0.6, 3),
+        # prediction below the threshold is discarded
+        ([-1, 3], [0.45, 0.55], 0.6, None),
+        # boundary: exactly at the threshold is accepted, not discarded
+        ([-1, 3], [0.4, 0.6], 0.6, 3),
+        # the winning class is the "no match" pseudo-class, regardless of its
+        # own confidence
+        ([-1, 3], [0.99, 0.01], 0.0, None),
+        # threshold of 0.0 disables the confidence check entirely
+        ([-1, 3], [0.45, 0.55], 0.0, 3),
+    ],
+)
+def test_predict_with_threshold(classes, probabilities, threshold, expected) -> None:
+    classifier = _StubProbaClassifier(classes, probabilities)
+    result = _predict_with_threshold(classifier, X=None, threshold=threshold)
+    assert result == expected
+
+
+def test_classifier_match_threshold_default() -> None:
+    """
+    GIVEN:
+        - No PAPERLESS_CLASSIFIER_MATCH_THRESHOLD environment variable is set
+    THEN:
+        - The classifier match threshold defaults to 0.6
+    """
+    assert settings.CLASSIFIER_MATCH_THRESHOLD == 0.6
+
+
 def test_preprocess_content() -> None:
     """
     GIVEN:

src/documents/tests/test_document_model.py

@@ -110,7 +110,7 @@ class TestDocument(TestCase):
             checksum="checksum",
             mime_type="application/pdf",
         )
-        Document.objects.create(
+        version = Document.objects.create(
             root_document=root,
             correspondent=root.correspondent,
             title="Version",
@@ -124,6 +124,10 @@ class TestDocument(TestCase):
         self.assertEqual(Document.objects.count(), 0)
         self.assertEqual(Document.deleted_objects.count(), 2)
 
+        root.restore(strict=False)
+
+        self.assertTrue(Document.objects.filter(pk=version.pk).exists())
+
     def test_file_name(self) -> None:
         doc = Document(
             mime_type="application/pdf",

src/documents/tests/test_management_consumer.py

@@ -136,6 +136,23 @@ def wait_for_mock_call(
     return False
 
 
+def sleep_past_stability(
+    owner: FileStabilityTracker | ConsumerThread,
+    *,
+    windows: float = 1.5,
+) -> None:
+    """
+    Block until a tracked file's stability window has certainly elapsed.
+
+    Args:
+        owner: The tracker, or the consumer thread running one, whose
+               configured stability delay sets the wait.
+        windows: How many stability windows to wait, giving slop for a slow
+                 or loaded test runner.
+    """
+    sleep(owner.stability_delay * windows)
+
+
 class TestTrackedFile:
     """Tests for the TrackedFile dataclass."""
 
@@ -261,6 +278,56 @@ class TestFileStabilityTracker:
         assert len(stable) == 0
         assert stability_tracker.pending_count == 1
 
+    def test_get_stable_files_skips_empty_file(
+        self,
+        stability_tracker: FileStabilityTracker,
+        tmp_path: Path,
+    ) -> None:
+        """
+        GIVEN:
+            - A zero byte file, tracked and past its stability delay
+        WHEN:
+            - Stable files are collected
+        THEN:
+            - The file is not yielded for consumption
+            - The file is dropped from tracking rather than held, so an
+              abandoned placeholder does not keep the watch loop awake
+        """
+        empty = tmp_path / "scan.pdf"
+        empty.write_bytes(b"")
+        stability_tracker.track(empty, Change.added)
+        sleep_past_stability(stability_tracker)
+
+        stable = list(stability_tracker.get_stable_files())
+
+        assert stable == []
+        assert stability_tracker.pending_count == 0
+
+    def test_empty_file_is_yielded_once_content_arrives(
+        self,
+        stability_tracker: FileStabilityTracker,
+        tmp_path: Path,
+    ) -> None:
+        """
+        GIVEN:
+            - A zero byte file which was dropped from tracking while empty
+        WHEN:
+            - The writer fills the file and a new event re-tracks it
+        THEN:
+            - The file is yielded for consumption once it is stable
+        """
+        target = tmp_path / "scan.pdf"
+        target.write_bytes(b"")
+        stability_tracker.track(target, Change.added)
+        sleep_past_stability(stability_tracker)
+        assert list(stability_tracker.get_stable_files()) == []
+
+        target.write_bytes(b"%PDF-1.4 content")
+        stability_tracker.track(target, Change.modified)
+        sleep_past_stability(stability_tracker)
+
+        assert list(stability_tracker.get_stable_files()) == [target]
+
     def test_get_stable_files_deleted_during_check(self, temp_file: Path) -> None:
         """Test deleted file is not returned during stability check."""
         tracker = FileStabilityTracker(stability_delay=0.1)
@@ -879,6 +946,51 @@ class TestCommandWatch:
 
         mock_consume_file_delay.apply_async.assert_called()
 
+    def test_scanner_placeholder_is_not_consumed_while_empty(
+        self,
+        consumption_dir: Path,
+        sample_pdf: Path,
+        mock_consume_file_delay: MagicMock,
+        start_consumer: Callable[..., ConsumerThread],
+    ) -> None:
+        """
+        GIVEN:
+            - A scanner which creates a zero byte placeholder and only writes
+              the page some time later (GH discussion #13969)
+        WHEN:
+            - The placeholder sits untouched well past the stability delay
+            - The scanner then writes the real content
+        THEN:
+            - The empty placeholder is never queued, as it could only fail
+              with "Unsupported mime type inode/x-empty"
+            - The file is queued exactly once, when the content lands
+        """
+        thread = start_consumer(stability_delay=0.2)
+
+        target = consumption_dir / "scan.pdf"
+        target.write_bytes(b"")  # the scanner's placeholder
+
+        # Well past the stability delay: the old behaviour queued it here.
+        sleep_past_stability(thread, windows=5)
+        if thread.exception:
+            raise thread.exception
+        assert mock_consume_file_delay.apply_async.call_count == 0
+
+        shutil.copy(sample_pdf, target)  # the scanner finishes the page
+
+        assert wait_for_mock_call(
+            mock_consume_file_delay.apply_async,
+            timeout_s=5.0,
+        )
+        if thread.exception:
+            raise thread.exception
+
+        assert mock_consume_file_delay.apply_async.call_count == 1
+        queued_doc = mock_consume_file_delay.apply_async.call_args.kwargs["kwargs"][
+            "input_doc"
+        ]
+        assert queued_doc.original_file.name == "scan.pdf"
+
     def test_ignores_macos_files(
         self,
         consumption_dir: Path,

src/documents/tests/test_views.py

@@ -32,6 +32,7 @@ from documents.signals.handlers import update_llm_suggestions_cache
 from documents.tests.utils import DirectoriesMixin
 from documents.tests.utils import read_streaming_response
 from paperless.models import ApplicationConfiguration
+from paperless_ai.exceptions import LLMProviderError
 from paperless_ai.exceptions import LLMTimeoutError
 
 
@@ -737,6 +738,38 @@ class TestAISuggestions(DirectoriesMixin, TestCase):
             get_llm_suggestion_cache(self.document.pk, backend="openai-like"),
         )
 
+    @patch("documents.views.get_ai_document_classification")
+    @override_settings(
+        AI_ENABLED=True,
+        LLM_BACKEND="openai-like",
+    )
+    def test_ai_suggestions_with_llm_provider_error(
+        self,
+        mock_get_ai_classification,
+    ) -> None:
+        mock_get_ai_classification.side_effect = LLMProviderError(
+            "confidential provider response",
+        )
+
+        self.client.force_login(user=self.user)
+        response = self.client.get(
+            f"/api/documents/{self.document.pk}/ai_suggestions/",
+        )
+
+        self.assertEqual(response.status_code, status.HTTP_502_BAD_GATEWAY)
+        self.assertEqual(
+            response.json(),
+            {
+                "ai": [
+                    "AI backend rejected the request. Check logs for details.",
+                ],
+            },
+        )
+        self.assertNotIn("confidential provider response", response.content.decode())
+        self.assertIsNone(
+            get_llm_suggestion_cache(self.document.pk, backend="openai-like"),
+        )
+
     @patch("documents.views.get_ai_document_classification")
     @override_settings(
         AI_ENABLED=True,

src/documents/views.py

@@ -252,6 +252,7 @@ from paperless.views import StandardPagination
 from paperless_ai.ai_classifier import get_ai_document_classification
 from paperless_ai.ai_classifier import get_llm_output_language
 from paperless_ai.chat import stream_chat_with_documents
+from paperless_ai.exceptions import LLMProviderError
 from paperless_ai.exceptions import LLMTimeoutError
 from paperless_ai.matching import extract_unmatched_names
 from paperless_ai.matching import match_correspondents_by_name
@@ -1603,6 +1604,22 @@ class DocumentViewSet(
                     {"ai": [_("AI backend request timed out.")]},
                     status=status.HTTP_503_SERVICE_UNAVAILABLE,
                 )
+            except LLMProviderError:
+                logger.exception(
+                    "AI backend rejected the request for document %s",
+                    doc.pk,
+                )
+                return Response(
+                    {
+                        "ai": [
+                            _(
+                                "AI backend rejected the request. "
+                                "Check logs for details.",
+                            ),
+                        ],
+                    },
+                    status=status.HTTP_502_BAD_GATEWAY,
+                )
             set_llm_suggestions_cache(
                 doc.pk,
                 llm_suggestions,
@@ -5437,7 +5454,10 @@ class TrashView(ListModelMixin, PassUserMixin):
 
     model = Document
 
-    queryset = Document.deleted_objects.all()
+    # A version is listed separately only when its root is not in the trash.
+    queryset = Document.deleted_objects.exclude(
+        root_document_id__in=Document.deleted_objects.values("id"),
+    )
 
     def get(self, request: Request, format: str | None = None) -> Response:
         self.serializer_class = DocumentSerializer
@@ -5468,7 +5488,15 @@ class TrashView(ListModelMixin, PassUserMixin):
             return HttpResponseForbidden("Insufficient permissions")
         action = serializer.validated_data.get("action")
         if action == "restore":
-            restored = list(Document.deleted_objects.filter(id__in=doc_ids))
+            restored = list(self.get_queryset().filter(id__in=doc_ids))
+            if len(restored) != len(doc_ids):
+                raise ValidationError(
+                    {
+                        "documents": [
+                            "Restore the root document instead of one of its versions.",
+                        ],
+                    },
+                )
             for doc in restored:
                 doc.restore(strict=False)
             if restored: