Changelog v3.0.2 - GHA (#13244)

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

docs/changelog.md

@@ -1,5 +1,64 @@
 # Changelog
 
+## paperless-ngx 3.0.2
+
+### Bug Fixes
+
+- Fix: fix broken migration in 3.0.1 [@shamoon](https://github.com/shamoon) ([#13242](https://github.com/paperless-ngx/paperless-ngx/pull/13242))
+
+### All App Changes
+
+- Fix: fix broken migration in 3.0.1 [@shamoon](https://github.com/shamoon) ([#13242](https://github.com/paperless-ngx/paperless-ngx/pull/13242))
+
+## paperless-ngx 3.0.1
+
+### Bug Fixes
+
+- Fix: selection\_data re-derives the filtered document set 5 times over [@stumpylog](https://github.com/stumpylog) ([#13229](https://github.com/paperless-ngx/paperless-ngx/pull/13229))
+- Fix: exclude source document from its own RAG similarity results [@stumpylog](https://github.com/stumpylog) ([#13233](https://github.com/paperless-ngx/paperless-ngx/pull/13233))
+- Fix: split mailrule maximum\_age clamp into its own migration [@stumpylog](https://github.com/stumpylog) ([#13231](https://github.com/paperless-ngx/paperless-ngx/pull/13231))
+- Chore: mark yes in confirm button for translation [@shamoon](https://github.com/shamoon) ([#13225](https://github.com/paperless-ngx/paperless-ngx/pull/13225))
+- Fix: correct URL for W001 check [@shamoon](https://github.com/shamoon) ([#13220](https://github.com/paperless-ngx/paperless-ngx/pull/13220))
+- Fix: ensure create dialog uses correct attribute type [@shamoon](https://github.com/shamoon) ([#13221](https://github.com/paperless-ngx/paperless-ngx/pull/13221))
+- Fix: correct database engine from postgres to postgresql [@OberstVonGatow](https://github.com/OberstVonGatow) ([#13213](https://github.com/paperless-ngx/paperless-ngx/pull/13213))
+- Fix: remove unnecessary .distinct() dominating document list queries [@stumpylog](https://github.com/stumpylog) ([#13205](https://github.com/paperless-ngx/paperless-ngx/pull/13205))
+- Fix: also wrap non-breaking words in chat messages, handle whitespace [@shamoon](https://github.com/shamoon) ([#13211](https://github.com/paperless-ngx/paperless-ngx/pull/13211))
+- Fix: tag/custom-field document\_count scales badly with tag count [@stumpylog](https://github.com/stumpylog) ([#13203](https://github.com/paperless-ngx/paperless-ngx/pull/13203))
+- Fix: batch document user\_can\_change checks to avoid per-row N+1 [@stumpylog](https://github.com/stumpylog) ([#13204](https://github.com/paperless-ngx/paperless-ngx/pull/13204))
+- Fix: fix app title restoration [@shamoon](https://github.com/shamoon) ([#13208](https://github.com/paperless-ngx/paperless-ngx/pull/13208))
+- Fix: handle long wrapping titles in AI chat document list [@shamoon](https://github.com/shamoon) ([#13206](https://github.com/paperless-ngx/paperless-ngx/pull/13206))
+
+### Dependencies
+
+- Chore(deps-dev): Bump postcss from 8.5.6 to 8.5.22 in /src/paperless\_mail/templates in the npm\_and\_yarn group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13236](https://github.com/paperless-ngx/paperless-ngx/pull/13236))
+- Chore(deps): Bump pyasn1 from 0.6.3 to 0.6.4 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13237](https://github.com/paperless-ngx/paperless-ngx/pull/13237))
+- Chore(deps): Bump the uv group across 1 directory with 8 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#13189](https://github.com/paperless-ngx/paperless-ngx/pull/13189))
+
+### All App Changes
+
+<details>
+<summary>17 changes</summary>
+
+- Chore(deps-dev): Bump postcss from 8.5.6 to 8.5.22 in /src/paperless\_mail/templates in the npm\_and\_yarn group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13236](https://github.com/paperless-ngx/paperless-ngx/pull/13236))
+- Fix: selection\_data re-derives the filtered document set 5 times over [@stumpylog](https://github.com/stumpylog) ([#13229](https://github.com/paperless-ngx/paperless-ngx/pull/13229))
+- Chore: reword "Settings > Default permissions" description for clarity [@NotaInutilis](https://github.com/NotaInutilis) ([#13232](https://github.com/paperless-ngx/paperless-ngx/pull/13232))
+- Chore(deps): Bump pyasn1 from 0.6.3 to 0.6.4 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13237](https://github.com/paperless-ngx/paperless-ngx/pull/13237))
+- Chore(deps): Bump the uv group across 1 directory with 8 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#13189](https://github.com/paperless-ngx/paperless-ngx/pull/13189))
+- Fix: exclude source document from its own RAG similarity results [@stumpylog](https://github.com/stumpylog) ([#13233](https://github.com/paperless-ngx/paperless-ngx/pull/13233))
+- Fix: split mailrule maximum\_age clamp into its own migration [@stumpylog](https://github.com/stumpylog) ([#13231](https://github.com/paperless-ngx/paperless-ngx/pull/13231))
+- Chore: mark yes in confirm button for translation [@shamoon](https://github.com/shamoon) ([#13225](https://github.com/paperless-ngx/paperless-ngx/pull/13225))
+- Fix: correct URL for W001 check [@shamoon](https://github.com/shamoon) ([#13220](https://github.com/paperless-ngx/paperless-ngx/pull/13220))
+- Fix: ensure create dialog uses correct attribute type [@shamoon](https://github.com/shamoon) ([#13221](https://github.com/paperless-ngx/paperless-ngx/pull/13221))
+- Fix: correct database engine from postgres to postgresql [@OberstVonGatow](https://github.com/OberstVonGatow) ([#13213](https://github.com/paperless-ngx/paperless-ngx/pull/13213))
+- Fix: remove unnecessary .distinct() dominating document list queries [@stumpylog](https://github.com/stumpylog) ([#13205](https://github.com/paperless-ngx/paperless-ngx/pull/13205))
+- Fix: also wrap non-breaking words in chat messages, handle whitespace [@shamoon](https://github.com/shamoon) ([#13211](https://github.com/paperless-ngx/paperless-ngx/pull/13211))
+- Fix: tag/custom-field document\_count scales badly with tag count [@stumpylog](https://github.com/stumpylog) ([#13203](https://github.com/paperless-ngx/paperless-ngx/pull/13203))
+- Fix: batch document user\_can\_change checks to avoid per-row N+1 [@stumpylog](https://github.com/stumpylog) ([#13204](https://github.com/paperless-ngx/paperless-ngx/pull/13204))
+- Fix: fix app title restoration [@shamoon](https://github.com/shamoon) ([#13208](https://github.com/paperless-ngx/paperless-ngx/pull/13208))
+- Fix: handle long wrapping titles in AI chat document list [@shamoon](https://github.com/shamoon) ([#13206](https://github.com/paperless-ngx/paperless-ngx/pull/13206))
+
+</details>
+
 ## paperless-ngx 3.0.0
 
 ### Breaking Changes

docs/configuration.md

@@ -934,11 +934,10 @@ for display in the web interface.
 
     !!! note
 
-        The **remote OCR parser** (Azure AI) also honors this setting: when
-        no archive is requested (`never`, or `auto` with a born-digital PDF),
-        the remote engine is skipped entirely and locally-extracted text is
-        used instead, avoiding an unnecessary API call and a duplicate text
-        layer.
+        The **remote OCR parser** (Azure AI) always produces a searchable
+        PDF and stores it as the archive copy, regardless of this setting.
+        `ARCHIVE_FILE_GENERATION=never` has no effect when the remote
+        parser handles a document.
 
 #### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
 

docs/migration-v3.md

@@ -187,11 +187,10 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
 
 ### Remote OCR parser
 
-If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
-honored the same way as for the local engine: when no archive is requested
-(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
-entirely and locally-extracted text is used instead, avoiding an unnecessary
-API call and a duplicate text layer.
+If you use the **remote OCR parser** (Azure AI), note that it always produces a
+searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
+has no effect for documents handled by the remote parser — the archive is produced
+unconditionally by the remote engine.
 
 ## Search Index (Whoosh -> Tantivy)
 

src/documents/search/_backend.py

@@ -475,13 +475,7 @@ class TantivyBackend:
         note_texts: list[str] = []
         for note in document.notes.all():
             num_notes += 1
-            doc.add_json(
-                "notes",
-                {
-                    "note": note.note,
-                    "user": note.user.username if note.user else None,
-                },
-            )
+            doc.add_json("notes", {"note": note.note, "user": note.user.username})
             note_texts.append(note.note)
         if note_texts:
             doc.add_text("notes_text", " ".join(note_texts))

src/documents/tests/search/test_backend.py

@@ -844,23 +844,6 @@ class TestFieldHandling:
             f"Expected 1, got {len(ids)}. Note content should be searchable via notes.note: prefix."
         )
 
-    def test_notes_without_user_are_indexed(self, backend: TantivyBackend) -> None:
-        """Notes whose user was deleted (SET_NULL) must not break indexing."""
-        doc = Document.objects.create(
-            title="Doc with orphaned note",
-            content="test",
-            checksum="NT2",
-            pk=81,
-        )
-        Note.objects.create(document=doc, note="Orphaned note", user=None)
-
-        backend.add_or_update(doc)
-
-        ids = backend.search_ids("notes.note:orphaned", user=None)
-        assert len(ids) == 1, (
-            f"Expected 1, got {len(ids)}. Notes without a user should still be indexed."
-        )
-
 
 class TestHighlightHits:
     """Test highlight_hits returns proper HTML strings, not raw Snippet objects."""

src/paperless/parsers/remote.py

@@ -3,9 +3,7 @@ Built-in remote-OCR document parser.
 
 Handles documents by sending them to a configured remote OCR engine
 (currently Azure AI Vision / Document Intelligence) and retrieving both
-the extracted text and a searchable PDF with an embedded text layer. For
-born-digital PDFs that need no archive copy, the remote call is skipped
-entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
+the extracted text and a searchable PDF with an embedded text layer.
 
 When no engine is configured, ``score()`` returns ``None`` so the parser
 is effectively invisible to the registry — the tesseract parser handles
@@ -23,8 +21,6 @@ from typing import Self
 
 from django.conf import settings
 
-from paperless.parsers.utils import extract_pdf_text
-from paperless.parsers.utils import post_process_text
 from paperless.version import __full_version_str__
 
 if TYPE_CHECKING:
@@ -73,11 +69,8 @@ class RemoteDocumentParser:
     """Parse documents via a remote OCR API (currently Azure AI Vision).
 
     This parser sends documents to a remote engine that returns both
-    extracted text and a searchable PDF with an embedded text layer,
-    except when ``parse()`` is called with ``produce_archive=False`` for
-    a PDF, in which case the remote call is skipped and only locally
-    extracted text is returned (no archive). It does not depend on
-    Tesseract or ocrmypdf.
+    extracted text and a searchable PDF with an embedded text layer.
+    It does not depend on Tesseract or ocrmypdf.
 
     Class attributes
     ----------------
@@ -166,11 +159,8 @@ class RemoteDocumentParser:
         Returns
         -------
         bool
-            Always True — the remote engine is capable of returning a PDF
-            with an embedded text layer to serve as the archive copy.
-            Whether it actually does so for a given document depends on
-            ``produce_archive`` passed to :meth:`parse` (see there for when
-            the remote engine call, and thus archive generation, is skipped).
+            Always True — the remote engine always returns a PDF with an
+            embedded text layer that serves as the archive copy.
         """
         return True
 
@@ -227,12 +217,6 @@ class RemoteDocumentParser:
     ) -> None:
         """Send the document to the remote engine and store results.
 
-        When *produce_archive* is False for a PDF, the caller (via
-        ``documents.consumer.should_produce_archive``) has already determined
-        that the document is born-digital and needs no archive — skip the
-        remote engine entirely rather than re-OCRing it and creating a
-        duplicate text layer.
-
         Parameters
         ----------
         document_path:
@@ -240,8 +224,8 @@ class RemoteDocumentParser:
         mime_type:
             Detected MIME type of the document.
         produce_archive:
-            Whether an archive copy is wanted. For PDFs, False skips the
-            remote engine and uses locally-extracted text instead.
+            Ignored — the remote engine always returns a searchable PDF,
+            which is stored as the archive copy regardless of this flag.
         """
         config = RemoteEngineConfig(
             engine=settings.REMOTE_OCR_ENGINE,
@@ -256,16 +240,6 @@ class RemoteDocumentParser:
             self._text = ""
             return
 
-        if not produce_archive and mime_type == "application/pdf":
-            logger.debug(
-                "Remote OCR: skipped — no archive requested, "
-                "using locally-extracted text",
-            )
-            self._text = (
-                post_process_text(extract_pdf_text(document_path, log=logger)) or ""
-            )
-            return
-
         if config.engine == "azureai":
             self._text = self._azure_ai_vision_parse(document_path, config)
 

src/paperless/parsers/tesseract.py

@@ -3,6 +3,7 @@ from __future__ import annotations
 import importlib.resources
 import logging
 import os
+import re
 import shutil
 import tempfile
 from pathlib import Path
@@ -24,9 +25,9 @@ from paperless.config import OcrConfig
 from paperless.models import CleanChoices
 from paperless.models import ModeChoices
 from paperless.models import OutputTypeChoices
+from paperless.parsers.utils import PDF_TEXT_MIN_LENGTH
 from paperless.parsers.utils import extract_pdf_text
-from paperless.parsers.utils import pdf_has_digital_text
-from paperless.parsers.utils import post_process_text
+from paperless.parsers.utils import is_tagged_pdf
 from paperless.parsers.utils import read_file_handle_unicode_errors
 from paperless.version import __full_version_str__
 
@@ -509,10 +510,8 @@ class RasterisedDocumentParser:
 
         if mime_type == "application/pdf":
             text_original = self.extract_text(None, document_path)
-            original_has_text = pdf_has_digital_text(
-                document_path,
-                text_original,
-                log=self.log,
+            original_has_text = is_tagged_pdf(document_path, log=self.log) or (
+                text_original is not None and len(text_original) > PDF_TEXT_MIN_LENGTH
             )
         else:
             text_original = None
@@ -657,3 +656,17 @@ class RasterisedDocumentParser:
                     f"No text was found in {document_path}, the content will be empty.",
                 )
                 self.text = ""
+
+
+def post_process_text(text: str | None) -> str | None:
+    if not text:
+        return None
+
+    collapsed_spaces = re.sub(r"([^\S\r\n]+)", " ", text)
+    no_leading_whitespace = re.sub(r"([\n\r]+)([^\S\n\r]+)", "\\1", collapsed_spaces)
+    no_trailing_whitespace = re.sub(r"([^\S\n\r]+)$", "", no_leading_whitespace)
+
+    # TODO: this needs a rework
+    # replace \0 prevents issues with saving to postgres.
+    # text may contain \0 when this character is present in PDF files.
+    return no_trailing_whitespace.strip().replace("\0", " ")

src/paperless/parsers/utils.py

@@ -65,63 +65,6 @@ def is_tagged_pdf(
         return False
 
 
-def pdf_has_digital_text(
-    path: Path,
-    text: str | None,
-    log: logging.Logger | None = None,
-) -> bool:
-    """Return True if a PDF already has a usable, born-digital text layer.
-
-    Combines the tagged-PDF check with an extracted-text length check.
-    Shared by the tesseract and remote OCR parsers to decide whether
-    OCR_MODE=auto/off should skip (re-)OCRing a document.
-
-    Parameters
-    ----------
-    path:
-        Absolute path to the PDF file.
-    text:
-        Text already extracted from the PDF (e.g. via ``extract_pdf_text``),
-        or ``None``.
-    log:
-        Logger for warnings.  Falls back to the module-level logger when omitted.
-
-    Returns
-    -------
-    bool
-        ``True`` when the document already contains a text layer.
-    """
-    return is_tagged_pdf(path, log=log) or (
-        text is not None and len(text) > PDF_TEXT_MIN_LENGTH
-    )
-
-
-def post_process_text(text: str | None) -> str | None:
-    """Normalise whitespace in extracted OCR/PDF text and strip NUL bytes.
-
-    Parameters
-    ----------
-    text:
-        Raw extracted text, or ``None``.
-
-    Returns
-    -------
-    str | None
-        Cleaned text, or ``None`` when *text* is falsy.
-    """
-    if not text:
-        return None
-
-    collapsed_spaces = re.sub(r"([^\S\r\n]+)", " ", text)
-    no_leading_whitespace = re.sub(r"([\n\r]+)([^\S\n\r]+)", "\\1", collapsed_spaces)
-    no_trailing_whitespace = re.sub(r"([^\S\n\r]+)$", "", no_leading_whitespace)
-
-    # TODO: this needs a rework
-    # replace \0 prevents issues with saving to postgres.
-    # text may contain \0 when this character is present in PDF files.
-    return no_trailing_whitespace.strip().replace("\0", " ")
-
-
 def extract_pdf_text(
     path: Path,
     log: logging.Logger | None = None,

src/paperless/tests/parsers/test_remote_parser.py

@@ -336,117 +336,6 @@ class TestRemoteParserParse:
         assert remote_parser.get_date() is None
 
 
-# ---------------------------------------------------------------------------
-# parse() — produce_archive=False skips the remote engine (PDFs only)
-# ---------------------------------------------------------------------------
-
-
-class TestRemoteParserSkipsWhenNoArchiveWanted:
-    """When the caller has already decided no archive is needed for a PDF
-    (documents.consumer.should_produce_archive), the remote engine call is
-    skipped entirely in favor of locally-extracted text.
-    """
-
-    def test_pdf_skips_azure_when_no_archive_requested(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-    ) -> None:
-        """
-        GIVEN: produce_archive=False for a PDF
-        WHEN:  parse() is called
-        THEN:  Azure is never invoked, no archive is produced, and text
-               comes from local pdftotext extraction
-        """
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        azure_client.begin_analyze_document.assert_not_called()
-        assert remote_parser.get_archive_path() is None
-        assert remote_parser.get_text() != ""
-
-    def test_pdf_no_archive_requested_text_matches_local_extraction(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-        mocker: MockerFixture,
-    ) -> None:
-        """
-        GIVEN: produce_archive=False for a PDF
-        WHEN:  parse() is called
-        THEN:  the returned text is exactly the locally-extracted text,
-               not anything from the (unused) Azure mock
-        """
-        mocker.patch(
-            "paperless.parsers.remote.extract_pdf_text",
-            return_value="Local digital text.",
-        )
-
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        assert remote_parser.get_text() == "Local digital text."
-
-    def test_pdf_no_archive_requested_closes_no_client(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-    ) -> None:
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        azure_client.close.assert_not_called()
-
-    def test_non_pdf_still_calls_azure_when_no_archive_requested(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-        azure_client: Mock,
-    ) -> None:
-        """
-        Images have no local-text fallback, so produce_archive=False does
-        not skip the remote engine for non-PDF MIME types.
-        """
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "image/png",
-            produce_archive=False,
-        )
-
-        azure_client.begin_analyze_document.assert_called_once()
-        assert remote_parser.get_text() == _DEFAULT_TEXT
-
-    @pytest.mark.usefixtures("no_engine_settings")
-    def test_unconfigured_engine_takes_precedence_over_skip(
-        self,
-        remote_parser: RemoteDocumentParser,
-        simple_digital_pdf_file: Path,
-    ) -> None:
-        """An unconfigured engine still short-circuits before the
-        produce_archive check, returning empty text as before.
-        """
-        remote_parser.parse(
-            simple_digital_pdf_file,
-            "application/pdf",
-            produce_archive=False,
-        )
-
-        assert remote_parser.get_text() == ""
-        assert remote_parser.get_archive_path() is None
-
-
 # ---------------------------------------------------------------------------
 # parse() — Azure failure path
 # ---------------------------------------------------------------------------

src/paperless/tests/parsers/test_tesseract_parser.py

@@ -21,7 +21,7 @@ from documents.parsers import run_convert
 from paperless.models import ModeChoices
 from paperless.parsers import ParserProtocol
 from paperless.parsers.tesseract import RasterisedDocumentParser
-from paperless.parsers.utils import post_process_text
+from paperless.parsers.tesseract import post_process_text
 
 if TYPE_CHECKING:
     from pathlib import Path