Fix: consolidate and extend unicode NFC normalization for filenames and matching

Consolidates the scattered unicodedata.normalize("NFC", ...) calls introduced
by my earlier filename/path normalization work into a single
documents.utils.normalize_unicode() helper, and closes several gaps where
NFD-normalized filenames could still slip through unnormalized:

- Document.get_public_filename() now normalizes, fixing exported filenames
  built from an NFD title/correspondent name (the default, non-format export
  path was not covered by the earlier fix).
- DocumentViewSet.update_version() now normalizes the uploaded filename,
  matching PostDocumentView's existing behavior.
- ConsumerPlugin normalizes self.filename once at consumption time, covering
  the title fallback and Document.original_filename for every document
  source (consume folder, mail, API, barcode splits).
- Workflow trigger and mail rule filename/path matching (documents/matching.py,
  paperless_mail/mail.py) now normalize the document/attachment side before
  comparing, so an NFD filename matches an NFC-typed filter pattern instead of
  silently failing to match.
- WorkflowTriggerSerializer and MailRuleSerializer normalize filter_filename/
  filter_path and the attachment include/exclude patterns once at write time,
  so the read side isn't re-normalizing an already-canonical value on every
  match.

.claude/skills/paperless-benchmarking/SKILL.md

@@ -1,187 +0,0 @@
----
-name: paperless-benchmarking
-description: Use when profiling paperless-ngx performance, running `manage.py benchmark`, investigating a slow query or endpoint, or deciding whether a profiling finding should become a permanent registered scenario. Covers command reference, the fork/merge-back branch workflow for perf investigations, and how to read query-plan output.
----
-
-# Paperless-ngx Benchmarking
-
-This repo has a built-in benchmarking/profiling tool: `manage.py benchmark`, in
-the `paperless_benchmark` Django app. It replaces ad hoc standalone scripts —
-use it instead of writing new one-off seed/timing scripts.
-
-## Command reference
-
-```
-manage.py benchmark seed --tier {home,medium,large} [--reset --yes-i-know-this-wipes-the-database] [--seed N]
-manage.py benchmark run --repeat 5 [--label baseline]
-manage.py benchmark profile <scenario_name> [--repeat 5] [--explain]
-manage.py benchmark list-scenarios
-```
-
-- **`seed`** builds a realistic dataset at one of three scales: `home` (500
-  documents — fast, use this for iteration), `medium` (20,000 — the default
-  when `--tier` is omitted; a multi-minute seed), `large` (360,000 — matches
-  the scale reported in real large-install bug reports; slow, only use it
-  when a finding needs confirming at real scale). `--reset` wipes any
-  previously-seeded benchmark data first — but it is destructive and
-  irreversible: it deletes **all** users, **all** groups, and **all**
-  documents/tags/correspondents/document types/storage paths in the target
-  database, not just benchmark-created rows. Only run it against a disposable
-  benchmark database, never a real install. Because of that, `--reset` also
-  requires passing `--yes-i-know-this-wipes-the-database` in the same
-  invocation, or the command raises an error and does nothing. Omit both
-  flags if you want to layer more data onto an existing seed instead. `seed`
-  creates two named users, `perf_target` (mixed owned/shared documents,
-  realistic guardian permission grants) and `perf_admin` (superuser), plus a
-  general user/group pool with realistic permission-row ratios.
-- **`run`** times the 3 built-in API endpoint benchmarks
-  (`/api/documents/`, `/api/documents/?page_size=50`, `/api/tags/?page_size=100000`) for both
-  `perf_target` and `perf_admin`, reporting min/median/max wall-clock and SQL
-  query count. Requires `seed` to have already run — it reuses that data, it
-  does not seed its own.
-- **`profile`** times one named scenario from the registry (see
-  `list-scenarios`) via best-of-N repeat timing and SQL query count, against
-  `perf_target`. `--explain` additionally captures and prints the query plan:
-  real `EXPLAIN ANALYZE` execution stats on PostgreSQL/MariaDB, or
-  `EXPLAIN QUERY PLAN` (plan only, no real timing/row counts — clearly labeled
-  as such) on SQLite. Like `run`, `profile` requires `seed` to have already
-  run — it does not seed its own data either.
-- Every `run`/`profile` invocation appends a JSON line to
-  `benchmark_results/history.jsonl` at the repo root (local-only, gitignored —
-  never commit this file). Use it to compare a `before`/`after` pair across
-  two invocations without hand-copying numbers.
-- Full chain example: `seed --reset` once, then `run` and `profile` as many
-  times as you want against that same seeded data — no need to reseed between
-  them.
-
-## Adding a new scenario
-
-A "scenario" is a named, registered query/operation that `profile` can time
-and explain. To add one, edit `src/paperless_benchmark/scenarios.py`: write a
-`_<name>_run(user)` function (returns whatever `run_profile` should time) and
-optionally a `_<name>_queryset(user)` function (returns the `QuerySet` for
-`--explain` to analyze), then `register(Scenario(name=..., describe=...,
-run=..., queryset_for_explain=...))` at module level. Both functions receive
-the already-seeded `perf_target` user — they should not seed their own data.
-
-## Branch workflow
-
-`tools/benchmark-management-commands` is a **long-lived tooling branch**, not
-a feature branch that gets merged and closed:
-
-1. It is periodically brought up to date with `dev` (merge `dev` into it) so
-   the tooling doesn't drift from the schema/codebase it profiles. Do this
-   before starting a new investigation if it's been a while since the last
-   sync.
-2. **Every performance investigation forks its own branch from
-   `tools/benchmark-management-commands`** (not from `dev`). Do the
-   investigation there: write throwaway profiling code, try fixes, capture
-   before/after numbers.
-3. **That investigation branch never merges into `dev` or production.** Its
-   only job is to produce evidence and, optionally, a reusable scenario.
-4. If the investigation turns up a scenario worth keeping permanently (see
-   "When to graduate a scenario" below), open a PR that adds **just that
-   scenario** back into `tools/benchmark-management-commands` — not the rest
-   of the investigation branch's throwaway code.
-5. Any actual production fix the investigation motivates (e.g. an ORM query
-   change) goes into its own normal feature branch off `dev`, following the
-   project's regular contribution process — profiling evidence informs that
-   PR's description, but the profiling code itself does not travel with it.
-
-## Reading query-plan output
-
-- **PostgreSQL** `EXPLAIN ANALYZE`: look for `Seq Scan` on a large table
-  (missing index), a large gap between `rows=N` (planner's estimate) and the
-  actual row count in parentheses (stale statistics or a bad cardinality
-  estimate), and nested-loop joins driven by an outer relation with many
-  rows (usually the N+1 pattern this tool exists to catch).
-- **MariaDB**: verified against a real MariaDB 12.3 container that MariaDB
-  does NOT accept MySQL 8.0.18+'s `EXPLAIN ANALYZE` syntax (it's a 1064
-  syntax error) -- `capture_explain()` instead runs MariaDB's own
-  `ANALYZE <statement>` form (no `EXPLAIN` keyword), which returns a
-  tabular plan with real per-row execution columns: `rows` (estimate) vs.
-  `r_rows` (actual), and `filtered` vs. `r_filtered`. A large gap between
-  `rows` and `r_rows`, or `type: ALL` (full table scan) on a large table,
-  are the signals to look for -- the same underlying concerns as Postgres's
-  `Seq Scan`/estimate-vs-actual gap, just in MariaDB's column-based output
-  instead of Postgres's nested-tree text format.
-- **SQLite** `EXPLAIN QUERY PLAN`: no real timing/row-count data, only the
-  chosen access path (`SCAN` vs `SEARCH`, which index if any). Useful for
-  confirming an index is even being considered, not for judging real-world
-  cost — corroborate any SQLite finding against Postgres/MariaDB before
-  trusting it, since planner behavior differs meaningfully between them.
-- Compare query **count**, not just timing, between before/after: a fix that
-  keeps the same wall-clock time but drops query count from O(n) to O(1) is
-  still a real, durable improvement — timing alone is noisy and
-  environment-dependent, query count is not.
-
-## Cleaning up after an interrupted run
-
-If a `seed`/`run`/`profile` invocation gets killed mid-run (Ctrl-C, `kill -9`,
-a timed-out SSH session, etc.), check whether it left anything behind before
-trusting the next benchmark's numbers. This was verified for real: a
-`benchmark seed --tier large --reset ...` was started against both a fresh
-PostgreSQL 18 container and a fresh MariaDB 12.3 container and `kill -9`'d a
-few seconds into document seeding. In both cases, the database-side
-connection disappeared immediately -- no stuck backend, no lingering query,
-no held lock was observed in either backend once the killed process's PID
-was confirmed gone. That said, this was one interruption point (mid
-bulk-seed, between chunks); a run killed mid-query, or a driver/network
-hiccup that doesn't cleanly close the socket, could behave differently, so
-still check before trusting a number if any run in the session was
-interrupted:
-
-- **PostgreSQL**: look for leftover connections against the benchmark
-  database:
-
-  ```sql
-  SELECT pid, state, query, query_start
-  FROM pg_stat_activity
-  WHERE datname = current_database() AND pid <> pg_backend_pid();
-  ```
-
-  If a stuck backend shows up, clear it with:
-
-  ```sql
-  SELECT pg_terminate_backend(pid)
-  FROM pg_stat_activity
-  WHERE datname = current_database() AND pid <> pg_backend_pid();
-  ```
-
-- **MariaDB**: look for leftover connections/queries:
-
-  ```sql
-  SHOW FULL PROCESSLIST;
-  ```
-
-  If a stuck connection shows up (anything other than your current admin
-  session), clear it with:
-
-  ```sql
-  KILL <id>;
-  ```
-
-A stray connection left running concurrently with a subsequent benchmark run
-would add real, contaminating load (extra queries competing for the same
-rows, possibly held locks slowing the next run's timings) -- cheap to rule
-out, expensive to silently trust a number that was actually measured
-alongside a zombie connection.
-
-## When to graduate a one-off finding into a permanent scenario
-
-Register a scenario (rather than leaving it as throwaway code on the
-investigation branch) when **both** are true:
-
-- The query pattern is one this codebase is likely to regress on again (e.g.
-  it involves a permission-check join, a bulk operation, or anything else
-  with an easy-to-reintroduce N+1) — not a one-time fluke specific to this
-  investigation.
-- Re-running it later, against a fresh seed, would still produce a
-  meaningful signal (it doesn't depend on investigation-specific throwaway
-  data or a fix that's already permanently landed and can't regress the same
-  way).
-
-If a finding doesn't meet both bars, keep it as disposable code on the
-investigation branch and let the branch's evidence (captured in the PR
-description of whatever production fix it motivates) be the permanent
-record instead.

.gitignore

@@ -115,6 +115,3 @@ celerybeat-schedule*
 
 # Git worktree local folder
 .worktrees
-
-# Benchmark tooling output (local only, never committed)
-/benchmark_results/

src/documents/consumer.py

@@ -52,6 +52,7 @@ from documents.templating.workflows import parse_w_workflow_placeholders
 from documents.utils import compute_checksum
 from documents.utils import copy_basic_file_stats
 from documents.utils import copy_file_with_basic_stats
+from documents.utils import normalize_unicode
 from documents.utils import run_subprocess
 from paperless.config import OcrConfig
 from paperless.config import RemoteOCRConfig
@@ -201,7 +202,9 @@ class ConsumerPluginMixin:
 
         self.renew_logging_group()
 
-        self.filename = self.metadata.filename or self.input_doc.original_file.name
+        self.filename = normalize_unicode(
+            self.metadata.filename or self.input_doc.original_file.name,
+        )
 
     def _send_progress(
         self,

src/documents/matching.py

@@ -21,6 +21,7 @@ from documents.models import Workflow
 from documents.models import WorkflowTrigger
 from documents.permissions import permitted_object_ids
 from documents.regex import safe_regex_search
+from documents.utils import normalize_unicode
 
 if TYPE_CHECKING:
     from django.db.models import QuerySet
@@ -311,11 +312,12 @@ def consumable_document_matches_workflow(
         trigger_matched = False
 
     # Document filename vs trigger filename
+    document_filename = normalize_unicode(document.original_file.name)
     if (
         trigger.filter_filename is not None
         and len(trigger.filter_filename) > 0
         and not fnmatch(
-            document.original_file.name.lower(),
+            document_filename.lower(),
             trigger.filter_filename.lower(),
         )
     ):
@@ -328,10 +330,12 @@ def consumable_document_matches_workflow(
     # Document path vs trigger path
 
     # Use the original_path if set, else us the original_file
-    match_against = (
-        document.original_path
-        if document.original_path is not None
-        else document.original_file
+    match_against = normalize_unicode(
+        str(
+            document.original_path
+            if document.original_path is not None
+            else document.original_file,
+        ),
     )
 
     if (
@@ -536,7 +540,7 @@ def existing_document_matches_workflow(
         and len(trigger.filter_filename) > 0
         and document.original_filename is not None
         and not fnmatch(
-            document.original_filename.lower(),
+            normalize_unicode(document.original_filename).lower(),
             trigger.filter_filename.lower(),
         )
     ):

src/documents/models.py

@@ -27,6 +27,7 @@ from django_softdelete.models import SoftDeleteModel
 
 from documents.data_models import DocumentSource
 from documents.parsers import get_default_file_extension
+from documents.utils import normalize_unicode
 
 
 class ModelWithOwner(models.Model):
@@ -467,7 +468,7 @@ class Document(SoftDeleteModel, ModelWithOwner):  # type: ignore[django-manager-
         context_document = (
             self.root_document if self.root_document_id is not None else self
         )
-        result = str(context_document)
+        result = normalize_unicode(str(context_document))
 
         if counter:
             result += f"_{counter:02}"

src/documents/serialisers.py

@@ -87,6 +87,7 @@ from documents.regex import validate_regex_pattern
 from documents.templating.filepath import validate_filepath_template_and_render
 from documents.templating.utils import convert_format_str_to_template_format
 from documents.templating.workflows import validate_workflow_template
+from documents.utils import normalize_unicode
 from documents.validators import uri_validator
 from documents.validators import url_validator
 from documents.versioning import sort_versions_newest_first
@@ -3120,6 +3121,13 @@ class WorkflowTriggerSerializer(serializers.ModelSerializer[WorkflowTrigger]):
         ):
             attrs["filter_path"] = None
 
+        # Normalize once at write time, since these are matched against many
+        # documents but edited rarely
+        if attrs.get("filter_filename") is not None:
+            attrs["filter_filename"] = normalize_unicode(attrs["filter_filename"])
+        if attrs.get("filter_path") is not None:
+            attrs["filter_path"] = normalize_unicode(attrs["filter_path"])
+
         if (
             "filter_custom_field_query" in attrs
             and attrs["filter_custom_field_query"] is not None

src/documents/templating/filepath.py

@@ -1,7 +1,6 @@
 import logging
 import os
 import re
-import unicodedata
 from collections.abc import Iterable
 from pathlib import PurePath
 
@@ -26,6 +25,7 @@ from documents.templating.environment import _template_environment
 from documents.templating.filters import format_datetime
 from documents.templating.filters import get_cf_value
 from documents.templating.filters import localize_date
+from documents.utils import normalize_unicode
 
 logger = logging.getLogger("paperless.templating")
 
@@ -42,7 +42,7 @@ class FilePathTemplate(Template):
             3. Removing extra spaces before and after forward slashes
             4. Preserving spaces in other parts of the path
             """
-            value = unicodedata.normalize("NFC", value)
+            value = normalize_unicode(value)
             value = value.replace("\n", "").replace("\r", "")
             value = re.sub(r"\s*/\s*", "/", value)
 
@@ -184,17 +184,17 @@ def get_basic_metadata_context(
     """
     return {
         "title": pathvalidate.sanitize_filename(
-            unicodedata.normalize("NFC", document.title),
+            normalize_unicode(document.title),
             replacement_text="-",
         ),
         "correspondent": pathvalidate.sanitize_filename(
-            unicodedata.normalize("NFC", document.correspondent.name),
+            normalize_unicode(document.correspondent.name),
             replacement_text="-",
         )
         if document.correspondent
         else no_value_default,
         "document_type": pathvalidate.sanitize_filename(
-            unicodedata.normalize("NFC", document.document_type.name),
+            normalize_unicode(document.document_type.name),
             replacement_text="-",
         )
         if document.document_type
@@ -205,8 +205,7 @@ def get_basic_metadata_context(
         "owner_username": document.owner.username
         if document.owner
         else no_value_default,
-        "original_name": unicodedata.normalize(
-            "NFC",
+        "original_name": normalize_unicode(
             PurePath(document.original_filename).with_suffix("").name,
         )
         if document.original_filename
@@ -275,12 +274,12 @@ def get_tags_context(tags: Iterable[Tag]) -> dict[str, str | list[str]]:
     return {
         "tag_list": pathvalidate.sanitize_filename(
             ",".join(
-                sorted(unicodedata.normalize("NFC", tag.name) for tag in tags),
+                sorted(normalize_unicode(tag.name) for tag in tags),
             ),
             replacement_text="-",
         ),
         # Assumed to be ordered, but a template could loop through to find what they want
-        "tag_name_list": [unicodedata.normalize("NFC", x.name) for x in tags],
+        "tag_name_list": [normalize_unicode(x.name) for x in tags],
     }
 
 
@@ -307,7 +306,7 @@ def get_custom_fields_context(
             CustomField.FieldDataType.LONG_TEXT,
         }:
             value = pathvalidate.sanitize_filename(
-                unicodedata.normalize("NFC", field_instance.value),
+                normalize_unicode(field_instance.value),
                 replacement_text="-",
             )
         elif (
@@ -316,8 +315,7 @@ def get_custom_fields_context(
         ):
             options = field_instance.field.extra_data["select_options"]
             value = pathvalidate.sanitize_filename(
-                unicodedata.normalize(
-                    "NFC",
+                normalize_unicode(
                     next(
                         option["label"]
                         for option in options
@@ -330,7 +328,7 @@ def get_custom_fields_context(
             value = field_instance.value
         field_data["custom_fields"][
             pathvalidate.sanitize_filename(
-                unicodedata.normalize("NFC", field_instance.field.name),
+                normalize_unicode(field_instance.field.name),
                 replacement_text="-",
             )
         ] = {

src/documents/tests/test_api_update_version_nfc.py

@@ -0,0 +1,69 @@
+import unicodedata
+from typing import TYPE_CHECKING
+from unittest import mock
+
+import celery.result
+import pytest
+from django.core.files.uploadedfile import SimpleUploadedFile
+
+from documents.models import Document
+
+if TYPE_CHECKING:
+    from documents.data_models import ConsumableDocument
+
+
+@pytest.fixture()
+def consume_file_mock():
+    with mock.patch("documents.tasks.consume_file.apply_async") as m:
+        m.return_value = celery.result.AsyncResult(id="test-task-id")
+        yield m
+
+
+@pytest.fixture()
+def directories(tmp_path, settings, _media_settings):
+    scratch = tmp_path / "scratch"
+    scratch.mkdir()
+    settings.SCRATCH_DIR = scratch
+    return scratch
+
+
+@pytest.mark.django_db
+class TestUpdateVersionNFCNormalization:
+    def test_nfd_filename_normalized_to_nfc(
+        self,
+        admin_client,
+        consume_file_mock: mock.MagicMock,
+        directories,
+    ):
+        """Uploaded new-version file with NFD filename must have its temp name stored as NFC."""
+        document = Document.objects.create(
+            title="Test",
+            content="content",
+            checksum="checksum",
+            mime_type="application/pdf",
+        )
+
+        nfd = unicodedata.normalize("NFD", "Rechnung März.pdf")
+        nfc = unicodedata.normalize("NFC", "Rechnung März.pdf")
+
+        assert nfd != nfc
+
+        uploaded = SimpleUploadedFile(
+            nfd,
+            b"%PDF-1.4 test",
+            content_type="application/pdf",
+        )
+        response = admin_client.post(
+            f"/api/documents/{document.pk}/update_version/",
+            {"document": uploaded},
+        )
+
+        assert response.status_code == 200
+
+        task_kwargs = consume_file_mock.call_args.kwargs["kwargs"]
+        input_doc: ConsumableDocument = task_kwargs["input_doc"]
+
+        assert input_doc.original_file.name == nfc, (
+            f"Expected NFC filename {nfc!r}, got {input_doc.original_file.name!r}"
+        )
+        assert unicodedata.is_normalized("NFC", input_doc.original_file.name)

src/documents/tests/test_get_public_filename_nfc.py

@@ -0,0 +1,48 @@
+import unicodedata
+from datetime import date
+
+import pytest
+
+from documents.models import Correspondent
+from documents.models import Document
+
+
+@pytest.mark.django_db
+class TestGetPublicFilenameNfc:
+    def test_normalizes_nfd_title_to_nfc(self) -> None:
+        nfd_title = unicodedata.normalize("NFD", "Gehaltserhöhung")
+        assert not unicodedata.is_normalized("NFC", nfd_title)
+
+        doc = Document(
+            mime_type="application/pdf",
+            title=nfd_title,
+            created=date(2025, 10, 17),
+        )
+
+        result = doc.get_public_filename()
+
+        assert unicodedata.is_normalized("NFC", result)
+        assert (
+            result
+            == "2025-10-17 "
+            + unicodedata.normalize(
+                "NFC",
+                nfd_title,
+            )
+            + ".pdf"
+        )
+
+    def test_normalizes_nfd_correspondent_name_to_nfc(self) -> None:
+        nfd_name = unicodedata.normalize("NFD", "Müller GmbH")
+        correspondent = Correspondent.objects.create(name=nfd_name)
+
+        doc = Document.objects.create(
+            mime_type="application/pdf",
+            title="Rechnung",
+            created=date(2025, 10, 17),
+            correspondent=correspondent,
+        )
+
+        result = doc.get_public_filename()
+
+        assert unicodedata.is_normalized("NFC", result)

src/documents/tests/test_matching_nfc.py

@@ -0,0 +1,80 @@
+import unicodedata
+
+import pytest
+
+from documents.data_models import ConsumableDocument
+from documents.data_models import DocumentSource
+from documents.matching import consumable_document_matches_workflow
+from documents.matching import existing_document_matches_workflow
+from documents.models import Document
+from documents.models import Workflow
+from documents.models import WorkflowTrigger
+
+
+@pytest.mark.django_db
+class TestMatchingNfcNormalization:
+    def test_consumable_document_filename_nfd_matches_nfc_pattern(
+        self,
+        tmp_path,
+    ) -> None:
+        """
+        GIVEN:
+            - A file on disk whose name is NFD-normalized
+            - A workflow trigger filename filter typed as NFC
+        WHEN:
+            - The consumable document is checked against the trigger
+        THEN:
+            - It matches, because both sides are normalized before comparing
+        """
+        nfd_name = unicodedata.normalize("NFD", "Gehaltserhöhung.pdf")
+        nfc_pattern = unicodedata.normalize("NFC", "*Gehaltserhöhung*")
+        assert nfd_name != unicodedata.normalize("NFC", nfd_name)
+
+        file_path = tmp_path / nfd_name
+        file_path.write_bytes(b"%PDF-1.4 test")
+
+        document = ConsumableDocument(
+            source=DocumentSource.ConsumeFolder,
+            original_file=file_path,
+        )
+        trigger = WorkflowTrigger(
+            type=WorkflowTrigger.WorkflowTriggerType.CONSUMPTION,
+            filter_filename=nfc_pattern,
+            sources=[],
+        )
+
+        matched, reason = consumable_document_matches_workflow(document, trigger)
+
+        assert matched, reason
+
+    def test_existing_document_filename_nfd_matches_nfc_pattern(self) -> None:
+        """
+        GIVEN:
+            - A Document whose original_filename is NFD-normalized (e.g. from
+              before normalization was applied at consumption time)
+            - A workflow trigger filename filter typed as NFC
+        WHEN:
+            - The document is checked against the trigger
+        THEN:
+            - It matches, because both sides are normalized before comparing
+        """
+        nfd_name = unicodedata.normalize("NFD", "Gehaltserhöhung.pdf")
+        nfc_pattern = unicodedata.normalize("NFC", "*Gehaltserhöhung*")
+
+        document = Document.objects.create(
+            title="Test",
+            content="content",
+            checksum="checksum",
+            mime_type="application/pdf",
+            original_filename=nfd_name,
+        )
+        workflow = Workflow.objects.create(name="Test workflow", order=0)
+        trigger = WorkflowTrigger.objects.create(
+            type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
+            filter_filename=nfc_pattern,
+        )
+        workflow.triggers.add(trigger)
+
+        matched, reason = existing_document_matches_workflow(document, trigger)
+
+        assert matched, reason

src/documents/tests/test_normalize_unicode.py

@@ -0,0 +1,6 @@
+from documents.utils import normalize_unicode
+
+
+class TestNormalizeUnicode:
+    def test_none_passes_through(self) -> None:
+        assert normalize_unicode(None) is None

src/documents/utils.py

@@ -1,6 +1,7 @@
 import hashlib
 import logging
 import shutil
+import unicodedata
 from collections.abc import Callable
 from collections.abc import Iterable
 from collections.abc import Iterator
@@ -31,6 +32,25 @@ def identity(iterable: Iterable[_T]) -> Iterable[_T]:
     return iterable
 
 
+def normalize_unicode(value: str | None) -> str | None:
+    """
+    Normalize a string to Unicode NFC form, or return None unchanged.
+
+    This is the single normalization pass for any user- or filesystem-supplied
+    text that ends up in a filename, path, or is compared/matched against one
+    (titles, correspondent/tag/type names, uploaded filenames, workflow and
+    mail rule filename/path filters). Composed (NFC) and decomposed (NFD)
+    forms of the same visible text are different byte sequences, which breaks
+    exact comparisons and filesystem lookups even though the text looks
+    identical. Always normalize through this function rather than calling
+    unicodedata.normalize() directly, so every call site agrees on the same
+    form.
+    """
+    if value is None:
+        return None
+    return unicodedata.normalize("NFC", value)
+
+
 class QuerySetStream(Generic[_M]):
     """Stream a QuerySet via .iterator(chunk_size=...) instead of
     materializing it (plus any prefetch caches) all at once, while still

src/documents/views.py

@@ -231,6 +231,7 @@ from documents.tasks import sanity_check
 from documents.tasks import train_classifier
 from documents.tasks import update_document_parent_tags
 from documents.utils import get_boolean
+from documents.utils import normalize_unicode
 from documents.versioning import VersionResolutionError
 from documents.versioning import annotate_effective_content
 from documents.versioning import get_latest_version_for_root
@@ -2068,6 +2069,7 @@ class DocumentViewSet(
 
         try:
             doc_name, doc_data = serializer.validated_data.get("document")
+            doc_name = normalize_unicode(doc_name)
             version_label = serializer.validated_data.get("version_label")
 
             t = int(mktime(datetime.now().timetuple()))
@@ -3334,7 +3336,7 @@ class PostDocumentView(GenericAPIView[Any]):
         serializer.is_valid(raise_exception=True)
 
         doc_name, doc_data = serializer.validated_data.get("document")
-        doc_name = normalize("NFC", doc_name)
+        doc_name = normalize_unicode(doc_name)
         correspondent_id = serializer.validated_data.get("correspondent")
         document_type_id = serializer.validated_data.get("document_type")
         storage_path_id = serializer.validated_data.get("storage_path")

src/paperless/settings/__init__.py

@@ -150,7 +150,6 @@ INSTALLED_APPS = [
     "drf_spectacular",
     "drf_spectacular_sidecar",
     "treenode",
-    "paperless_benchmark.apps.PaperlessBenchmarkConfig",
     *env_apps,
 ]
 

src/paperless_benchmark/apps.py

@@ -1,8 +0,0 @@
-from django.apps import AppConfig
-from django.utils.translation import gettext_lazy as _
-
-
-class PaperlessBenchmarkConfig(AppConfig):
-    name = "paperless_benchmark"
-
-    verbose_name = _("Paperless benchmark")

src/paperless_benchmark/db.py

@@ -1,136 +0,0 @@
-from __future__ import annotations
-
-from typing import TYPE_CHECKING
-
-from django.db import connection
-
-if TYPE_CHECKING:
-    from django.db.models import QuerySet
-
-
-def _reset_table_names() -> list[str]:
-    from guardian.models import GroupObjectPermission
-    from guardian.models import UserObjectPermission
-
-    from documents.models import Correspondent
-    from documents.models import Document
-    from documents.models import DocumentType
-    from documents.models import StoragePath
-    from documents.models import Tag
-
-    return [
-        Document.tags.through._meta.db_table,
-        Document._meta.db_table,
-        Tag._meta.db_table,
-        Correspondent._meta.db_table,
-        DocumentType._meta.db_table,
-        StoragePath._meta.db_table,
-        UserObjectPermission._meta.db_table,
-        GroupObjectPermission._meta.db_table,
-    ]
-
-
-def _delete_all_users_and_groups() -> None:
-    # ASSUMPTION: this tool assumes a disposable benchmark database, never
-    # point it at a real install. This deletes EVERY user and group in the
-    # database (not just benchmark-created ones) -- there is no way to
-    # distinguish "real" users from seeded ones, so this is only safe against
-    # a database that exists solely to run this benchmarking tool. The
-    # `benchmark seed --reset` CLI path requires an explicit
-    # `--yes-i-know-this-wipes-the-database` flag before reaching here; do
-    # not remove that guard.
-    from django.contrib.auth import get_user_model
-    from django.contrib.auth.models import Group
-
-    get_user_model().objects.all().delete()
-    Group.objects.all().delete()
-
-
-def _reset_postgresql() -> None:
-    tables = _reset_table_names()
-    with connection.cursor() as cursor:
-        cursor.execute(f"TRUNCATE TABLE {', '.join(tables)} RESTART IDENTITY CASCADE;")
-    _delete_all_users_and_groups()
-
-
-def _reset_mariadb() -> None:
-    # MariaDB's TRUNCATE has no CASCADE clause and refuses to truncate a
-    # table referenced by a foreign key while checks are enabled, so
-    # checks are disabled for the duration of the reset.
-    tables = _reset_table_names()
-    with connection.cursor() as cursor:
-        cursor.execute("SET FOREIGN_KEY_CHECKS = 0;")
-        try:
-            for table in tables:
-                cursor.execute(f"TRUNCATE TABLE {table};")
-        finally:
-            cursor.execute("SET FOREIGN_KEY_CHECKS = 1;")
-    _delete_all_users_and_groups()
-
-
-def _reset_sqlite() -> None:
-    from documents.models import Correspondent
-    from documents.models import Document
-    from documents.models import DocumentType
-    from documents.models import StoragePath
-    from documents.models import Tag
-
-    Document.global_objects.all().delete()
-    Tag.objects.all().delete()
-    Correspondent.objects.all().delete()
-    DocumentType.objects.all().delete()
-    StoragePath.objects.all().delete()
-    _delete_all_users_and_groups()
-
-
-def reset_benchmark_data() -> None:
-    """
-    Remove all previously-seeded benchmark data (documents, tags,
-    correspondents, document types, storage paths, guardian permission
-    rows, users, and groups) so a fresh `benchmark seed` run starts
-    from an empty slate. Dispatches per-backend because TRUNCATE syntax
-    and cascade behavior differ across the 3 supported databases.
-    """
-    if connection.vendor == "postgresql":
-        _reset_postgresql()
-    elif connection.vendor == "mysql":
-        # MariaDB also reports vendor == "mysql" under Django's mysql backend.
-        _reset_mariadb()
-    else:
-        _reset_sqlite()
-
-
-def capture_explain(queryset: QuerySet) -> str:
-    """
-    Return the query plan for `queryset` using the current backend's
-    explain facility. PostgreSQL supports `EXPLAIN ANALYZE {sql}` (real
-    execution stats). MariaDB does NOT accept that syntax -- verified
-    against a real MariaDB 12.3 container: `EXPLAIN ANALYZE {sql}` raises a
-    1064 syntax error, while MariaDB's own `ANALYZE {sql}` form (no
-    `EXPLAIN` keyword) works and returns real per-row execution stats
-    (`r_rows`, `r_filtered`, etc. columns) -- this is MariaDB's
-    EXPLAIN-ANALYZE-equivalent, distinct from MySQL 8.0.18+'s
-    `EXPLAIN ANALYZE` syntax, which MariaDB does not implement. SQLite only
-    supports EXPLAIN QUERY PLAN (the chosen plan, not real timing/row
-    counts) -- that output is clearly labeled rather than silently looking
-    equivalent to the other two backends' output.
-    """
-    sql, params = queryset.query.sql_with_params()
-    with connection.cursor() as cursor:
-        if connection.vendor == "postgresql":
-            cursor.execute(f"EXPLAIN ANALYZE {sql}", params)
-            return "\n".join(str(row[0]) for row in cursor.fetchall())
-        if connection.vendor == "mysql":
-            # MariaDB also reports vendor == "mysql" under Django's mysql
-            # backend. Unlike MySQL 8.0.18+, MariaDB has no `EXPLAIN
-            # ANALYZE` syntax -- its equivalent is `ANALYZE <statement>`.
-            cursor.execute(f"ANALYZE {sql}", params)
-            columns = [c[0] for c in cursor.description]
-            header = " | ".join(columns)
-            rows = "\n".join(
-                " | ".join(str(c) for c in row) for row in cursor.fetchall()
-            )
-            return f"{header}\n{rows}"
-        cursor.execute(f"EXPLAIN QUERY PLAN {sql}", params)
-        rows = "\n".join(" | ".join(str(c) for c in row) for row in cursor.fetchall())
-        return f"(plan only -- no execution stats on SQLite)\n{rows}"

src/paperless_benchmark/endpoints.py

@@ -1,81 +0,0 @@
-# src/paperless_benchmark/endpoints.py
-from __future__ import annotations
-
-import statistics
-import time
-from dataclasses import dataclass
-from typing import TYPE_CHECKING
-
-if TYPE_CHECKING:
-    from django.contrib.auth.models import User
-    from rest_framework.test import APIClient
-
-ENDPOINTS: tuple[tuple[str, str], ...] = (
-    ("documents_default", "/api/documents/"),
-    ("documents_page50", "/api/documents/?page_size=50"),
-    ("tags_all", "/api/tags/?page_size=100000"),
-)
-
-
-@dataclass(frozen=True, slots=True)
-class EndpointTiming:
-    user_label: str
-    endpoint_name: str
-    query_count: int
-    min_ms: float
-    median_ms: float
-    max_ms: float
-
-
-def _timed_requests(client: APIClient, url: str, n: int) -> list[float]:
-    times = []
-    for _ in range(n):
-        t0 = time.perf_counter()
-        resp = client.get(url)
-        t1 = time.perf_counter()
-        if resp.status_code != 200:
-            raise RuntimeError(
-                f"GET {url} -> {resp.status_code}: {resp.content[:300]!r}",
-            )
-        times.append(t1 - t0)
-    return times
-
-
-def _query_count(client: APIClient, url: str) -> int:
-    from django.db import connection
-    from django.test.utils import CaptureQueriesContext
-
-    with CaptureQueriesContext(connection) as ctx:
-        resp = client.get(url)
-    if resp.status_code != 200:
-        raise RuntimeError(f"GET {url} -> {resp.status_code}: {resp.content[:300]!r}")
-    return len(ctx.captured_queries)
-
-
-def run_endpoint_benchmarks(
-    *,
-    perf_target: User,
-    perf_admin: User,
-    repeat: int,
-) -> list[EndpointTiming]:
-    from rest_framework.test import APIClient
-
-    results: list[EndpointTiming] = []
-    for user_label, user in (("target", perf_target), ("admin", perf_admin)):
-        client = APIClient()
-        client.force_authenticate(user=user)
-        for name, url in ENDPOINTS:
-            client.get(url)  # warm-up request, not counted
-            qcount = _query_count(client, url)
-            times_ms = [t * 1000 for t in _timed_requests(client, url, repeat)]
-            results.append(
-                EndpointTiming(
-                    user_label=user_label,
-                    endpoint_name=name,
-                    query_count=qcount,
-                    min_ms=min(times_ms),
-                    median_ms=statistics.median(times_ms),
-                    max_ms=max(times_ms),
-                ),
-            )
-    return results

src/paperless_benchmark/harness.py

@@ -1,56 +0,0 @@
-# src/paperless_benchmark/harness.py
-from __future__ import annotations
-
-import time
-from dataclasses import dataclass
-from typing import TYPE_CHECKING
-from typing import Generic
-from typing import TypeVar
-
-from django.db import connection
-from django.test.utils import CaptureQueriesContext
-
-if TYPE_CHECKING:
-    from collections.abc import Callable
-
-T = TypeVar("T")
-
-
-@dataclass(frozen=True, slots=True)
-class ProfileResult(Generic[T]):
-    best_seconds: float
-    all_seconds: tuple[float, ...]
-    query_count: int
-    result: T
-
-
-def run_profile(fn: Callable[[], T], *, repeat: int = 5) -> ProfileResult[T]:
-    """
-    Call `fn` `repeat` times, capturing wall-clock time for every call and
-    the SQL query count for the final call. Returns the best (minimum)
-    time across all repeats, since the first call(s) can be skewed by
-    connection warm-up or cold caches.
-    """
-    if repeat < 1:
-        raise ValueError("repeat must be >= 1")
-
-    all_seconds: list[float] = []
-    result: T | None = None
-    query_count = 0
-    for i in range(repeat):
-        with CaptureQueriesContext(connection) as ctx:
-            start = time.perf_counter()
-            result = fn()
-            all_seconds.append(time.perf_counter() - start)
-        if i == repeat - 1:
-            query_count = len(ctx.captured_queries)
-    # Purely a type-narrowing aid for the type checker: the `repeat < 1`
-    # guard above already turns the one case that could leave `result`
-    # unset into a clear ValueError, so this is unreachable in practice.
-    assert result is not None
-    return ProfileResult(
-        best_seconds=min(all_seconds),
-        all_seconds=tuple(all_seconds),
-        query_count=query_count,
-        result=result,
-    )

src/paperless_benchmark/management/commands/benchmark.py

@@ -1,222 +0,0 @@
-# src/paperless_benchmark/management/commands/benchmark.py
-from __future__ import annotations
-
-from typing import Any
-
-from django.core.management.base import BaseCommand
-from django.core.management.base import CommandError
-from django.core.management.base import CommandParser
-
-
-class Command(BaseCommand):
-    help = "Seed, run, and profile paperless-ngx performance benchmarks."
-
-    def add_arguments(self, parser: CommandParser) -> None:
-        parser.add_argument(
-            "action",
-            choices=["seed", "run", "profile", "list-scenarios"],
-            help="Which benchmark action to perform.",
-        )
-        parser.add_argument(
-            "scenario",
-            nargs="?",
-            default=None,
-            help="Scenario name (required for `profile`; see `list-scenarios`).",
-        )
-        parser.add_argument(
-            "--tier",
-            choices=["home", "medium", "large"],
-            default="medium",
-            help="Dataset scale tier for `seed` (default: medium).",
-        )
-        parser.add_argument(
-            "--reset",
-            action="store_true",
-            default=False,
-            help="For `seed`: truncate existing benchmark data first.",
-        )
-        parser.add_argument(
-            "--yes-i-know-this-wipes-the-database",
-            action="store_true",
-            default=False,
-            help=(
-                "Required alongside --reset: confirms you understand `seed "
-                "--reset` deletes ALL users, ALL groups, and ALL documents/"
-                "tags/correspondents/document types/storage paths in this "
-                "database, not just benchmark-created ones."
-            ),
-        )
-        parser.add_argument(
-            "--seed",
-            type=int,
-            default=42,
-            help="RNG seed for reproducible datasets (default: 42).",
-        )
-        parser.add_argument(
-            "--repeat",
-            type=int,
-            default=5,
-            help="Number of timed repetitions for `run`/`profile` (default: 5).",
-        )
-        parser.add_argument(
-            "--label",
-            default="baseline",
-            help="Free-text tag for a `run`, printed and recorded in history only.",
-        )
-        parser.add_argument(
-            "--explain",
-            action="store_true",
-            default=False,
-            help="For `profile`: also capture and print the query plan.",
-        )
-
-    def handle(self, *args: Any, **options: Any) -> None:
-        action = options["action"]
-        if action == "seed":
-            self._handle_seed(options)
-        elif action == "run":
-            self._handle_run(options)
-        elif action == "profile":
-            self._handle_profile(options)
-        else:
-            self._handle_list_scenarios()
-
-    def _handle_seed(self, options: dict[str, Any]) -> None:
-        from paperless_benchmark.db import reset_benchmark_data
-        from paperless_benchmark.seeding import seed_benchmark_dataset
-
-        if options["reset"]:
-            if not options["yes_i_know_this_wipes_the_database"]:
-                raise CommandError(
-                    "--reset requires --yes-i-know-this-wipes-the-database. "
-                    "This deletes ALL users, ALL groups, and ALL documents, "
-                    "tags, correspondents, document types, and storage paths "
-                    "in this database -- not just benchmark-created ones. "
-                    "Only run this against a disposable benchmark database, "
-                    "never a real install. Re-run with "
-                    "--reset --yes-i-know-this-wipes-the-database to proceed.",
-                )
-            self.stdout.write("Resetting existing benchmark data...")
-            reset_benchmark_data()
-
-        data = seed_benchmark_dataset(options["tier"], seed=options["seed"])
-        self.stdout.write(
-            self.style.SUCCESS(
-                f"Seeded tier={options['tier']!r}: {data.documents} documents, "
-                f"{len(data.users)} users, {len(data.groups)} groups.",
-            ),
-        )
-
-    def _handle_run(self, options: dict[str, Any]) -> None:
-        from django.contrib.auth import get_user_model
-        from django.db import connection
-
-        from documents.models import Document
-        from paperless_benchmark.endpoints import run_endpoint_benchmarks
-        from paperless_benchmark.results import append_history
-
-        user_model = get_user_model()
-        try:
-            perf_target = user_model.objects.get(username="perf_target")
-            perf_admin = user_model.objects.get(username="perf_admin")
-        except user_model.DoesNotExist as e:
-            raise CommandError(
-                "No benchmark dataset found. Run `manage.py benchmark seed` first.",
-            ) from e
-
-        db_vendor = connection.vendor
-        document_count = Document.objects.count()
-
-        results = run_endpoint_benchmarks(
-            perf_target=perf_target,
-            perf_admin=perf_admin,
-            repeat=options["repeat"],
-        )
-
-        self.stdout.write(f"# label={options['label']} repeat={options['repeat']}")
-        self.stdout.write(
-            f"{'user':7s} {'endpoint':20s} {'queries':>8s} "
-            f"{'min_ms':>9s} {'median_ms':>10s} {'max_ms':>9s}",
-        )
-        for r in results:
-            self.stdout.write(
-                f"{r.user_label:7s} {r.endpoint_name:20s} {r.query_count:8d} "
-                f"{r.min_ms:9.1f} {r.median_ms:10.1f} {r.max_ms:9.1f}",
-            )
-            append_history(
-                {
-                    "mode": "run",
-                    "label": options["label"],
-                    "user": r.user_label,
-                    "endpoint": r.endpoint_name,
-                    "query_count": r.query_count,
-                    "min_ms": r.min_ms,
-                    "median_ms": r.median_ms,
-                    "max_ms": r.max_ms,
-                    "db_vendor": db_vendor,
-                    "document_count": document_count,
-                },
-            )
-
-    def _handle_profile(self, options: dict[str, Any]) -> None:
-        from django.contrib.auth import get_user_model
-        from django.db import connection
-
-        from documents.models import Document
-        from paperless_benchmark.db import capture_explain
-        from paperless_benchmark.harness import run_profile
-        from paperless_benchmark.results import append_history
-        from paperless_benchmark.scenarios import get as get_scenario
-
-        if not options["scenario"]:
-            raise CommandError(
-                "`profile` requires a scenario name; see `list-scenarios`.",
-            )
-
-        scenario = get_scenario(options["scenario"])
-
-        user_model = get_user_model()
-        try:
-            perf_target = user_model.objects.get(username="perf_target")
-        except user_model.DoesNotExist as e:
-            raise CommandError(
-                "No benchmark dataset found. Run `manage.py benchmark seed` first.",
-            ) from e
-
-        profile = run_profile(
-            lambda: scenario.run(perf_target),
-            repeat=options["repeat"],
-        )
-        self.stdout.write(
-            f"{scenario.name}: best={profile.best_seconds:.4f}s "
-            f"queries={profile.query_count}",
-        )
-
-        if options["explain"]:
-            if scenario.queryset_for_explain is not None:
-                plan = capture_explain(scenario.queryset_for_explain(perf_target))
-                self.stdout.write(plan)
-            else:
-                self.stdout.write(
-                    self.style.WARNING(
-                        f"--explain was requested but scenario {scenario.name!r} "
-                        "does not support it (no queryset_for_explain); skipping.",
-                    ),
-                )
-
-        append_history(
-            {
-                "mode": "profile",
-                "scenario": scenario.name,
-                "best_seconds": profile.best_seconds,
-                "query_count": profile.query_count,
-                "db_vendor": connection.vendor,
-                "document_count": Document.objects.count(),
-            },
-        )
-
-    def _handle_list_scenarios(self) -> None:
-        from paperless_benchmark.scenarios import all_scenarios
-
-        for scenario in all_scenarios():
-            self.stdout.write(f"{scenario.name}: {scenario.describe}")

src/paperless_benchmark/results.py

@@ -1,39 +0,0 @@
-# src/paperless_benchmark/results.py
-from __future__ import annotations
-
-import json
-import subprocess
-from datetime import UTC
-from datetime import datetime
-from pathlib import Path
-from typing import Any
-
-RESULTS_DIR = Path(__file__).resolve().parent.parent.parent / "benchmark_results"
-
-
-def _current_git_ref() -> str:
-    result = subprocess.run(
-        ["git", "rev-parse", "--short", "HEAD"],
-        capture_output=True,
-        text=True,
-        check=False,
-    )
-    return result.stdout.strip() or "unknown"
-
-
-def append_history(entry: dict[str, Any], *, code_ref: str | None = None) -> None:
-    """
-    Append one line to benchmark_results/history.jsonl -- a local-only,
-    append-only, cross-session record of every `benchmark run`/`profile`
-    invocation. Unlike a single overwritten snapshot file, this survives
-    across sessions so a benchmarking effort picked back up days later has
-    a full timeline instead of only the most recent result.
-    """
-    RESULTS_DIR.mkdir(exist_ok=True)
-    record = {
-        "timestamp": datetime.now(UTC).isoformat(),
-        "code_ref": code_ref or _current_git_ref(),
-        **entry,
-    }
-    with (RESULTS_DIR / "history.jsonl").open("a") as f:
-        f.write(json.dumps(record) + "\n")

src/paperless_benchmark/scenarios.py

@@ -1,110 +0,0 @@
-# src/paperless_benchmark/scenarios.py
-from __future__ import annotations
-
-from dataclasses import dataclass
-from typing import TYPE_CHECKING
-from typing import Any
-
-if TYPE_CHECKING:
-    from collections.abc import Callable
-
-    from django.contrib.auth.models import User
-    from django.db.models import QuerySet
-
-
-@dataclass(frozen=True, slots=True)
-class Scenario:
-    name: str
-    describe: str
-    run: Callable[[User], Any]
-    queryset_for_explain: Callable[[User], QuerySet] | None = None
-
-
-_SCENARIOS: dict[str, Scenario] = {}
-
-
-def register(scenario: Scenario) -> None:
-    _SCENARIOS[scenario.name] = scenario
-
-
-def get(name: str) -> Scenario:
-    from django.core.management.base import CommandError
-
-    try:
-        return _SCENARIOS[name]
-    except KeyError:
-        available = ", ".join(sorted(_SCENARIOS)) or "(none registered)"
-        raise CommandError(
-            f"Unknown benchmark scenario {name!r}. Available: {available}",
-        ) from None
-
-
-def all_scenarios() -> tuple[Scenario, ...]:
-    return tuple(_SCENARIOS.values())
-
-
-def _guardian_visibility_query_run(user: User) -> list[int]:
-    from documents.models import Document
-    from documents.permissions import get_objects_for_user_owner_aware
-
-    return list(
-        get_objects_for_user_owner_aware(
-            user,
-            "documents.view_document",
-            Document,
-        ).values_list("id", flat=True),
-    )
-
-
-def _guardian_visibility_query_queryset(user: User) -> QuerySet:
-    from documents.models import Document
-    from documents.permissions import get_objects_for_user_owner_aware
-
-    return get_objects_for_user_owner_aware(user, "documents.view_document", Document)
-
-
-register(
-    Scenario(
-        name="guardian_visibility_query",
-        describe=(
-            "Document-visibility queryset for a user with mixed owned/shared "
-            "documents -- exercises documents.permissions."
-            "get_objects_for_user_owner_aware's guardian permission join."
-        ),
-        run=_guardian_visibility_query_run,
-        queryset_for_explain=_guardian_visibility_query_queryset,
-    ),
-)
-
-
-def _permitted_document_ids_run(user: User) -> list[int]:
-    from documents.models import Document
-    from documents.permissions import permitted_document_ids
-
-    return list(
-        Document.objects.filter(id__in=permitted_document_ids(user)).values_list(
-            "id",
-            flat=True,
-        ),
-    )
-
-
-def _permitted_document_ids_queryset(user: User) -> QuerySet:
-    from documents.models import Document
-    from documents.permissions import permitted_document_ids
-
-    return Document.objects.filter(id__in=permitted_document_ids(user))
-
-
-register(
-    Scenario(
-        name="permitted_document_ids",
-        describe=(
-            "Document-visibility query built from documents.permissions."
-            "permitted_document_ids -- the resolved-ID-set alternative to "
-            "guardian_visibility_query, for side-by-side comparison."
-        ),
-        run=_permitted_document_ids_run,
-        queryset_for_explain=_permitted_document_ids_queryset,
-    ),
-)

src/paperless_benchmark/seeding.py

@@ -1,445 +0,0 @@
-# src/paperless_benchmark/seeding.py
-from __future__ import annotations
-
-import datetime
-import random
-import time
-from dataclasses import dataclass
-from typing import TYPE_CHECKING
-from typing import Literal
-
-if TYPE_CHECKING:
-    from django.contrib.auth.models import Group
-    from django.contrib.auth.models import User
-
-Tier = Literal["home", "medium", "large"]
-
-CHUNK_SIZE = 5_000
-
-MIME_TYPES = (
-    "application/pdf",
-    "image/png",
-    "image/jpeg",
-    "text/plain",
-)
-
-# Ownership split for documents, mirroring the shape used in the #11950
-# perf-benchmark dataset (owned-by-target / owned-by-other / unowned).
-OWNED_BY_TARGET_FRACTION = 0.60
-OWNED_BY_OTHER_FRACTION = 0.30
-# remainder (0.10) is unowned
-
-# Of documents owned by "other" users, the fraction explicitly shared
-# (view, or view+change) with perf_target via guardian permissions --
-# this is what exercises the get_user_can_change() per-row N+1 that the
-# `run` endpoint benchmarks measure.
-SHARED_WITH_TARGET_FRACTION = 0.5
-SHARED_WITH_CHANGE_FRACTION = 0.5
-
-# Guardian permission-row ratios measured from a real install (discussion
-# #13276): 1,414 user-perm rows / 27,232 group-perm rows over 12,000
-# documents. Layered across ALL owned documents for the general user/group
-# pool (not just perf_target's shares), so `profile` scenarios exercise a
-# realistic permission-join shape for arbitrary users, not only perf_target.
-USER_PERM_ROWS_PER_DOC = 1_414 / 12_000
-GROUP_PERM_ROWS_PER_DOC = 27_232 / 12_000
-
-
-@dataclass(frozen=True, slots=True)
-class _TierCounts:
-    documents: int
-    tags: int
-    correspondents: int
-    document_types: int
-    storage_paths: int
-    other_users: int
-    groups: int
-    tags_per_doc: tuple[int, int]
-
-
-TIERS: dict[Tier, _TierCounts] = {
-    "home": _TierCounts(
-        documents=500,
-        tags=20,
-        correspondents=10,
-        document_types=8,
-        storage_paths=5,
-        other_users=3,
-        groups=2,
-        tags_per_doc=(1, 3),
-    ),
-    "medium": _TierCounts(
-        documents=20_000,
-        tags=100,
-        correspondents=300,
-        document_types=50,
-        storage_paths=20,
-        other_users=10,
-        groups=5,
-        tags_per_doc=(2, 6),
-    ),
-    "large": _TierCounts(
-        documents=360_000,
-        tags=1_000,
-        correspondents=5_000,
-        document_types=300,
-        storage_paths=50,
-        other_users=25,
-        groups=10,
-        tags_per_doc=(3, 7),
-    ),
-}
-
-
-def log(msg: str) -> None:
-    print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True)  # noqa: T201
-
-
-@dataclass(frozen=True, slots=True)
-class SeededData:
-    """
-    Summary of a completed seed run. `documents`/`tags`/`correspondents`/
-    `document_types`/`storage_paths` are counts, not the seeded ORM
-    instances: at the `large` tier (360,000 documents) holding every
-    instance in memory simultaneously is a real risk for zero benefit, since
-    no caller consumes anything but the counts. `perf_target`/`perf_admin`/
-    `users`/`groups` stay as real objects -- at most ~26 users/12 groups even
-    at `large` tier, and small enough to be useful to a future caller.
-    """
-
-    perf_target: User
-    perf_admin: User
-    users: tuple[User, ...]
-    groups: tuple[Group, ...]
-    documents: int
-    tags: int
-    correspondents: int
-    document_types: int
-    storage_paths: int
-
-
-def _grant_model_level_permissions(user: User) -> None:
-    """
-    Grant perf_target Django model-level view/add/change permissions on
-    Document and Tag, on top of the per-object guardian grants seeding
-    creates elsewhere. DRF's PaperlessObjectPermissions checks model-level
-    permissions before guardian's object-level ones are ever consulted, so
-    without this perf_target gets a blanket 403 on /api/documents/ and
-    /api/tags/ regardless of which documents guardian says it can see.
-    """
-    from django.contrib.auth.models import Permission
-    from django.contrib.contenttypes.models import ContentType
-
-    from documents.models import Document
-    from documents.models import Tag
-
-    for model in (Document, Tag):
-        content_type = ContentType.objects.get_for_model(model)
-        codenames = [
-            f"{action}_{model._meta.model_name}" for action in ("view", "add", "change")
-        ]
-        perms = Permission.objects.filter(
-            content_type=content_type,
-            codename__in=codenames,
-        )
-        user.user_permissions.add(*perms)
-
-
-def _create_users_and_groups(
-    counts: _TierCounts,
-) -> tuple[User, User, tuple[User, ...], tuple[Group, ...]]:
-    from django.contrib.auth.models import Group
-
-    from documents.tests.factories import UserFactory
-
-    perf_target = UserFactory.create(username="perf_target")
-    _grant_model_level_permissions(perf_target)
-    perf_admin = UserFactory.create(username="perf_admin", superuser=True)
-    other_users = tuple(UserFactory.create_batch(counts.other_users))
-    groups = tuple(
-        Group.objects.create(name=f"benchmark_group_{i}") for i in range(counts.groups)
-    )
-    log(
-        f"Created users: 1 target, 1 superuser, {len(other_users)} other, "
-        f"{len(groups)} groups.",
-    )
-    return perf_target, perf_admin, other_users, groups
-
-
-def _create_lookup_tables(counts: _TierCounts):
-    from documents.models import Correspondent
-    from documents.models import DocumentType
-    from documents.models import StoragePath
-    from documents.models import Tag
-    from documents.tests.factories import CorrespondentFactory
-    from documents.tests.factories import DocumentTypeFactory
-    from documents.tests.factories import StoragePathFactory
-    from documents.tests.factories import TagFactory
-
-    tags = tuple(Tag.objects.bulk_create(TagFactory.build_batch(counts.tags)))
-    correspondents = tuple(
-        Correspondent.objects.bulk_create(
-            CorrespondentFactory.build_batch(counts.correspondents),
-        ),
-    )
-    document_types = tuple(
-        DocumentType.objects.bulk_create(
-            DocumentTypeFactory.build_batch(counts.document_types),
-        ),
-    )
-    storage_paths = tuple(
-        StoragePath.objects.bulk_create(
-            StoragePathFactory.build_batch(counts.storage_paths),
-        ),
-    )
-    log(
-        f"Created {len(tags)} tags, {len(correspondents)} correspondents, "
-        f"{len(document_types)} document types, {len(storage_paths)} storage paths.",
-    )
-    return tags, correspondents, document_types, storage_paths
-
-
-def _assign_owner(rng: random.Random, perf_target: User, other_users: tuple[User, ...]):
-    roll = rng.random()
-    if roll < OWNED_BY_TARGET_FRACTION:
-        return perf_target, "target"
-    if roll < OWNED_BY_TARGET_FRACTION + OWNED_BY_OTHER_FRACTION:
-        return rng.choice(other_users), "other"
-    return None, "unowned"
-
-
-def _seed_documents(
-    rng: random.Random,
-    counts: _TierCounts,
-    tags,
-    correspondents,
-    document_types,
-    storage_paths,
-    perf_target,
-    other_users,
-) -> tuple[int, list[int]]:
-    """
-    Bulk-create `counts.documents` documents in chunks. Returns the total
-    document count plus a lightweight list of pks for documents that ended
-    up with an owner (target or other) -- that's all
-    `_grant_general_permissions` needs to sample from, so full `Document`
-    instances aren't accumulated across chunks (a real memory concern at the
-    `large` tier's 360,000 documents).
-    """
-    from django.contrib.auth.models import Permission
-    from django.contrib.contenttypes.models import ContentType
-    from guardian.models import UserObjectPermission
-
-    from documents.models import Document
-    from documents.tests.factories import DocumentFactory
-
-    tag_ids = [t.pk for t in tags]
-    correspondent_ids = [c.pk for c in correspondents]
-    document_type_ids = [d.pk for d in document_types]
-    storage_path_ids = [s.pk for s in storage_paths]
-
-    doc_content_type = ContentType.objects.get_for_model(Document)
-    view_perm = Permission.objects.get(
-        codename="view_document",
-        content_type=doc_content_type,
-    )
-    change_perm = Permission.objects.get(
-        codename="change_document",
-        content_type=doc_content_type,
-    )
-    through_model = Document.tags.through
-
-    document_count = 0
-    owned_document_pks: list[int] = []
-    remaining = counts.documents
-    while remaining > 0:
-        chunk_n = min(CHUNK_SIZE, remaining)
-        remaining -= chunk_n
-
-        batch = []
-        owner_buckets = []
-        for _ in range(chunk_n):
-            doc = DocumentFactory.build(
-                mime_type=rng.choice(MIME_TYPES),
-                page_count=rng.randint(1, 30),
-                correspondent_id=(
-                    rng.choice(correspondent_ids)
-                    if correspondent_ids and rng.random() < 0.8
-                    else None
-                ),
-                document_type_id=(
-                    rng.choice(document_type_ids)
-                    if document_type_ids and rng.random() < 0.6
-                    else None
-                ),
-                storage_path_id=(
-                    rng.choice(storage_path_ids)
-                    if storage_path_ids and rng.random() < 0.8
-                    else None
-                ),
-                # Document.created is a plain DateField (default: today).
-                # Leaving it unset would give every seeded document the same
-                # date, collapsing Document's ("-created",) ordering index
-                # into a single-valued sort key -- spread it over a
-                # realistic multi-year window instead.
-                created=datetime.date.today()
-                - datetime.timedelta(days=rng.randint(0, 365 * 3)),
-            )
-            owner, bucket = _assign_owner(rng, perf_target, other_users)
-            doc.owner_id = owner.pk if owner else None
-            batch.append(doc)
-            owner_buckets.append(bucket)
-
-        created = Document.objects.bulk_create(batch, batch_size=CHUNK_SIZE)
-
-        through_rows = []
-        for doc in created:
-            k = rng.randint(*counts.tags_per_doc)
-            for tag_id in rng.sample(tag_ids, min(k, len(tag_ids))):
-                through_rows.append(through_model(document_id=doc.pk, tag_id=tag_id))
-        if through_rows:
-            through_model.objects.bulk_create(through_rows, batch_size=CHUNK_SIZE)
-
-        perm_rows = []
-        for doc, bucket in zip(created, owner_buckets, strict=True):
-            if bucket == "unowned":
-                continue
-            owned_document_pks.append(doc.pk)
-
-            if bucket != "other":
-                continue
-            if rng.random() >= SHARED_WITH_TARGET_FRACTION:
-                continue
-            perm_rows.append(
-                UserObjectPermission(
-                    permission=view_perm,
-                    content_type=doc_content_type,
-                    object_pk=str(doc.pk),
-                    user=perf_target,
-                ),
-            )
-            if rng.random() < SHARED_WITH_CHANGE_FRACTION:
-                perm_rows.append(
-                    UserObjectPermission(
-                        permission=change_perm,
-                        content_type=doc_content_type,
-                        object_pk=str(doc.pk),
-                        user=perf_target,
-                    ),
-                )
-        if perm_rows:
-            UserObjectPermission.objects.bulk_create(perm_rows, batch_size=CHUNK_SIZE)
-
-        document_count += len(created)
-        log(f"  {document_count}/{counts.documents} documents seeded")
-
-    return document_count, owned_document_pks
-
-
-def _grant_general_permissions(
-    rng: random.Random,
-    owned_document_pks: list[int],
-    users,
-    groups,
-) -> None:
-    """
-    Layer realistic (issue #13276-derived) guardian permission-row ratios
-    across owned documents for the general user/group pool, so `profile`
-    scenarios exercise the same permission-join shape regardless of which
-    user they check visibility for (not just perf_target).
-    """
-    from django.contrib.auth.models import Permission
-    from django.contrib.contenttypes.models import ContentType
-    from guardian.models import GroupObjectPermission
-    from guardian.models import UserObjectPermission
-
-    from documents.models import Document
-
-    if not owned_document_pks or not users:
-        return
-
-    doc_content_type = ContentType.objects.get_for_model(Document)
-    view_perm = Permission.objects.get(
-        codename="view_document",
-        content_type=doc_content_type,
-    )
-
-    n_user_perms = round(len(owned_document_pks) * USER_PERM_ROWS_PER_DOC)
-    n_group_perms = (
-        round(len(owned_document_pks) * GROUP_PERM_ROWS_PER_DOC) if groups else 0
-    )
-
-    user_rows = [
-        UserObjectPermission(
-            permission=view_perm,
-            content_type=doc_content_type,
-            object_pk=str(rng.choice(owned_document_pks)),
-            user=rng.choice(users),
-        )
-        for _ in range(n_user_perms)
-    ]
-    if user_rows:
-        UserObjectPermission.objects.bulk_create(
-            user_rows,
-            batch_size=CHUNK_SIZE,
-            ignore_conflicts=True,
-        )
-
-    group_rows = [
-        GroupObjectPermission(
-            permission=view_perm,
-            content_type=doc_content_type,
-            object_pk=str(rng.choice(owned_document_pks)),
-            group=rng.choice(groups),
-        )
-        for _ in range(n_group_perms)
-    ]
-    if group_rows:
-        GroupObjectPermission.objects.bulk_create(
-            group_rows,
-            batch_size=CHUNK_SIZE,
-            ignore_conflicts=True,
-        )
-
-    log(f"  Granted {len(user_rows)} user perms, {len(group_rows)} group perms.")
-
-
-def seed_benchmark_dataset(tier: Tier, *, seed: int = 42) -> SeededData:
-    """
-    Build a benchmark dataset at the given scale tier: a named perf_target
-    (mixed owned/shared documents) and perf_admin (superuser) for endpoint
-    benchmarking, plus a general user/group pool with realistic guardian
-    permission-row ratios for profile scenarios.
-    """
-    counts = TIERS[tier]
-    rng = random.Random(seed)
-
-    log(f"Seeding tier={tier!r}")
-    perf_target, perf_admin, other_users, groups = _create_users_and_groups(counts)
-    tags, correspondents, document_types, storage_paths = _create_lookup_tables(counts)
-    document_count, owned_document_pks = _seed_documents(
-        rng,
-        counts,
-        tags,
-        correspondents,
-        document_types,
-        storage_paths,
-        perf_target,
-        other_users,
-    )
-    all_users = (perf_target, *other_users)
-    _grant_general_permissions(rng, owned_document_pks, all_users, groups)
-
-    log(f"Done. {document_count} documents seeded for tier={tier!r}.")
-
-    return SeededData(
-        perf_target=perf_target,
-        perf_admin=perf_admin,
-        users=all_users,
-        groups=groups,
-        documents=document_count,
-        tags=len(tags),
-        correspondents=len(correspondents),
-        document_types=len(document_types),
-        storage_paths=len(storage_paths),
-    )

src/paperless_mail/mail.py

@@ -6,7 +6,6 @@ import socket
 import ssl
 import tempfile
 import traceback
-import unicodedata
 from datetime import date
 from datetime import timedelta
 from fnmatch import fnmatch
@@ -45,6 +44,7 @@ from documents.models import Correspondent
 from documents.models import PaperlessTask
 from documents.parsers import is_mime_type_supported
 from documents.tasks import consume_file
+from documents.utils import normalize_unicode
 from paperless.network import is_public_ip
 from paperless.network import resolve_hostname_ips
 from paperless_mail.models import MailAccount
@@ -617,10 +617,10 @@ class MailAccountHandler(LoggingMixin):
         rule: MailRule,
     ) -> str | None:
         if rule.assign_title_from == MailRule.TitleSource.FROM_SUBJECT:
-            return unicodedata.normalize("NFC", message.subject)
+            return normalize_unicode(message.subject)
 
         elif rule.assign_title_from == MailRule.TitleSource.FROM_FILENAME:
-            return unicodedata.normalize("NFC", Path(att.filename).stem)
+            return normalize_unicode(Path(att.filename).stem)
 
         elif rule.assign_title_from == MailRule.TitleSource.NONE:
             return None
@@ -1004,6 +1004,8 @@ class MailAccountHandler(LoggingMixin):
         consume_tasks = []
 
         for att in message.attachments:
+            attachment_filename = normalize_unicode(att.filename)
+
             if (
                 att.content_disposition != "attachment"
                 and rule.attachment_type
@@ -1018,7 +1020,7 @@ class MailAccountHandler(LoggingMixin):
 
             if not self.filename_inclusion_matches(
                 rule.filter_attachment_filename_include,
-                att.filename,
+                attachment_filename,
             ):
                 # Force the filename and pattern to the lowercase
                 # as this is system dependent otherwise
@@ -1030,7 +1032,7 @@ class MailAccountHandler(LoggingMixin):
                 continue
             elif self.filename_exclusion_matches(
                 rule.filter_attachment_filename_exclude,
-                att.filename,
+                attachment_filename,
             ):
                 self.log.debug(
                     f"Rule {rule}: "
@@ -1064,7 +1066,7 @@ class MailAccountHandler(LoggingMixin):
                 )
 
                 attachment_name = pathvalidate.sanitize_filename(
-                    unicodedata.normalize("NFC", att.filename),
+                    attachment_filename,
                 )
                 if attachment_name:
                     temp_filename = temp_dir / attachment_name
@@ -1175,7 +1177,7 @@ class MailAccountHandler(LoggingMixin):
         doc_overrides = DocumentMetadataOverrides(
             title=message.subject,
             filename=pathvalidate.sanitize_filename(
-                unicodedata.normalize("NFC", f"{message.subject}.eml"),
+                normalize_unicode(f"{message.subject}.eml"),
             ),
             correspondent_id=correspondent.id if correspondent else None,
             document_type_id=doc_type.id if doc_type else None,

src/paperless_mail/serialisers.py

@@ -8,6 +8,7 @@ from documents.serialisers import CorrespondentField
 from documents.serialisers import DocumentTypeField
 from documents.serialisers import OwnedObjectSerializer
 from documents.serialisers import TagsField
+from documents.utils import normalize_unicode
 from paperless_mail.models import MailAccount
 from paperless_mail.models import MailRule
 from paperless_mail.models import ProcessedMail
@@ -161,6 +162,12 @@ class MailRuleSerializer(OwnedObjectSerializer):
             raise serializers.ValidationError("Maximum mail age is unreasonably large.")
         return value
 
+    def validate_filter_attachment_filename_include(self, value):
+        return normalize_unicode(value)
+
+    def validate_filter_attachment_filename_exclude(self, value):
+        return normalize_unicode(value)
+
 
 class ProcessedMailSerializer(OwnedObjectSerializer):
     class Meta: