Fix: preserve document fields during Gotenberg conversion (#13271)

Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.

.github/workflows/ci-release.yml

@@ -86,7 +86,7 @@ jobs:
       # ---- Prepare Release ----
       - name: Generate requirements file
         run: |
-          uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt
+          uv export --quiet --no-dev --all-extras --emit-index-url --format requirements-txt --output-file requirements.txt
       - name: Compile messages
         env:
           PAPERLESS_SECRET_KEY: "ci-release-not-a-real-secret"

docs/migration-v3.md

@@ -133,7 +133,7 @@ PAPERLESS_DB_OPTIONS="sslmode=require,sslrootcert=/certs/ca.pem,pool.max_size=10
 
 ## OCR and Archive File Generation Settings
 
-The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** — old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
+The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** - old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
 
 ### Removed settings
 
@@ -166,7 +166,7 @@ Remove any `PAPERLESS_OCR_SKIP_ARCHIVE_FILE` variable from your environment. If
 # v2: skip OCR when text present, always archive
 PAPERLESS_OCR_MODE=skip
 # v3: equivalent (auto is the new default)
-# No change needed — auto is the default
+# No change needed - auto is the default
 
 # v2: skip OCR when text present, skip archive too
 PAPERLESS_OCR_MODE=skip_noarchive
@@ -189,7 +189,7 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
 
 If you use the **remote OCR parser** (Azure AI), note that it always produces a
 searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
-has no effect for documents handled by the remote parser — the archive is produced
+has no effect for documents handled by the remote parser - the archive is produced
 unconditionally by the remote engine.
 
 ## Search Index (Whoosh -> Tantivy)
@@ -327,6 +327,64 @@ behind a reverse proxy may need to set
 [`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER),
 or both, to avoid `403 Forbidden` errors on login.
 
+## Minimum CPU Requirements (NumPy Baseline)
+
+Starting with NumPy 2.4.0, official `manylinux` x86_64 wheels are compiled with a minimum
+CPU baseline of `x86-64-v2`, which requires SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, and
+CMPXCHG16B. This is a [deliberate upstream change](https://github.com/numpy/numpy/issues/27851)
+citing that these instructions have been present in over 99.7% of CPUs since 2008
+([Intel](<https://en.wikipedia.org/wiki/Nehalem_(microarchitecture)>)) or 2011
+([AMD](<https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)>)).
+
+NumPy is a dependency of the document classifier (via scikit-learn), so any CPU that
+predates SSE4.2 support will crash with `SIGILL` (illegal instruction) when the classifier
+is loaded or trained, regardless of whether AI features are enabled.
+
+This differs from NumPy's optional SIMD dispatch (e.g. AVX2, AVX512), which is detected and
+selected safely at runtime - the `x86-64-v2` requirement above is a hard floor baked into the
+wheel, with no runtime fallback.
+
+### Affected hardware
+
+CPUs older than roughly 2008 (Intel) or 2011 (AMD) that lack SSE4.2 support. This is more
+likely to affect low-power or embedded hardware - e.g. early Atom, Celeron, or pre-Bulldozer
+AMD chips - than typical desktop or server hardware from the last decade.
+
+Check for SSE4.2 support with:
+
+```bash
+grep -o -m1 sse4_2 /proc/cpuinfo
+```
+
+If this prints nothing, your CPU is affected.
+
+### Symptoms
+
+The Celery worker (and potentially the web server) repeatedly crashes and restarts with a
+`SIGILL` error, typically visible in `dmesg`/`journalctl` as a `trap invalid opcode` inside
+`_multiarray_umath...so`. Because the classifier is trained on a periodic schedule
+(hourly, by default), affected instances see intermittent, hard-to-reproduce document
+consumption failures whenever that scheduled task runs and takes down the worker process
+mid-task.
+
+### Action Required (for affected hardware only)
+
+There is no way to make the classifier itself work on such CPUs - it requires an unofficial
+NumPy build with `cpu-baseline=none`, which is not something we can ship. The practical
+path forward is to stop the classifier from ever loading or training, which avoids
+importing NumPy at all:
+
+```bash
+PAPERLESS_TRAIN_TASK_CRON=disable
+```
+
+This disables the periodic classifier training task (see
+[`PAPERLESS_TRAIN_TASK_CRON`](configuration.md#PAPERLESS_TRAIN_TASK_CRON)). Automatic
+matching based on the classifier (suggested correspondents, document types, tags, and
+storage paths from trained rules) will no longer be available, but rule-based matching is
+unaffected, and document consumption itself will no longer be at risk of crashing the
+worker.
+
 ## Database Migrations
 
 Some integer fields have been changed to smaller types to reduce database size. If you have any `MailRule` records with a `maximum_age` greater than 32767, they will be clamped to 32767 during the migration to avoid errors during migration.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "paperless-ngx"
-version = "3.0.1"
+version = "3.0.2"
 description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
 readme = "README.md"
 requires-python = ">=3.11"

src-ui/package.json

@@ -1,6 +1,6 @@
 {
   "name": "paperless-ngx-ui",
-  "version": "3.0.1",
+  "version": "3.0.2",
   "scripts": {
     "preinstall": "npx only-allow pnpm",
     "ng": "ng",

src-ui/src/environments/environment.prod.ts

@@ -8,7 +8,7 @@ export const environment = {
   apiVersion: '10', // match src/paperless/settings.py
   appTitle: DEFAULT_APP_TITLE,
   tag: 'prod',
-  version: '3.0.1',
+  version: '3.0.2',
   webSocketHost: window.location.host,
   webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
   webSocketBaseUrl: base_url.pathname + 'ws/',

src/documents/search/_backend.py

@@ -475,7 +475,13 @@ class TantivyBackend:
         note_texts: list[str] = []
         for note in document.notes.all():
             num_notes += 1
-            doc.add_json("notes", {"note": note.note, "user": note.user.username})
+            doc.add_json(
+                "notes",
+                {
+                    "note": note.note,
+                    "user": note.user.username if note.user else None,
+                },
+            )
             note_texts.append(note.note)
         if note_texts:
             doc.add_text("notes_text", " ".join(note_texts))

src/documents/signals/handlers.py

@@ -1264,7 +1264,17 @@ def task_failure_handler(
             "error_message": str(exception) if exception else "Unknown error",
         }
         if traceback:
-            tb_str = "".join(_tb.format_tb(traceback))
+            # billiard/celery pass a pre-formatted string instead of a real
+            # traceback object when the worker process itself died (e.g.
+            # WorkerLostError from a SIGILL) since there's no live traceback
+            # to walk in that case.
+            tb_str = (
+                traceback
+                if isinstance(traceback, str)
+                else "".join(
+                    _tb.format_tb(traceback),
+                )
+            )
             result_data["traceback"] = tb_str[:5000]
 
         now = timezone.now()

src/documents/tests/search/test_backend.py

@@ -844,6 +844,23 @@ class TestFieldHandling:
             f"Expected 1, got {len(ids)}. Note content should be searchable via notes.note: prefix."
         )
 
+    def test_notes_without_user_are_indexed(self, backend: TantivyBackend) -> None:
+        """Notes whose user was deleted (SET_NULL) must not break indexing."""
+        doc = Document.objects.create(
+            title="Doc with orphaned note",
+            content="test",
+            checksum="NT2",
+            pk=81,
+        )
+        Note.objects.create(document=doc, note="Orphaned note", user=None)
+
+        backend.add_or_update(doc)
+
+        ids = backend.search_ids("notes.note:orphaned", user=None)
+        assert len(ids) == 1, (
+            f"Expected 1, got {len(ids)}. Notes without a user should still be indexed."
+        )
+
 
 class TestHighlightHits:
     """Test highlight_hits returns proper HTML strings, not raw Snippet objects."""

src/paperless/parsers/mail.py

@@ -272,10 +272,7 @@ class MailDocumentParser:
         logger.debug("Building formatted text from email")
         self._text = build_formatted_text(mail)
 
-        if is_naive(mail.date):
-            self._date = make_aware(mail.date)
-        else:
-            self._date = mail.date
+        self._date = mail.date
 
         logger.debug("Creating a PDF from the email")
         if self._mailrule_id:
@@ -502,6 +499,9 @@ class MailDocumentParser:
                 f"Could not parse {filepath}: {err}",
             ) from err
 
+        if is_naive(parsed.date):
+            parsed.date = make_aware(parsed.date)
+
         return parsed
 
     def tika_parse(self, html: str) -> str:

src/paperless/parsers/tika.py

@@ -421,6 +421,11 @@ class TikaDocumentParser:
         logger.info("Converting %s to PDF as %s", document_path, pdf_path)
 
         with self._gotenberg_client.libre_office.to_pdf() as route:
+            # Preserve document fields as authored. updateIndexes (Gotenberg's
+            # default) triggers a refresh() that rewrites dynamic fields like
+            # auto-dates to the current date.
+            route.update_indexes(update_indexes=False)
+
             # Set the output format of the resulting PDF.
             # OutputTypeConfig reads the database-stored ApplicationConfiguration
             # first, then falls back to the PAPERLESS_OCR_OUTPUT_TYPE env var.

src/paperless/tests/parsers/test_mail_parser.py

@@ -145,6 +145,41 @@ class TestEmailFileParsing:
         assert parsed_msg.text == "This is just a simple Text Mail.\n"
         assert parsed_msg.to == ("some@one.de",)
 
+    def test_parse_file_to_message_makes_naive_date_aware(
+        self,
+        mocker: MockerFixture,
+        mail_parser: MailDocumentParser,
+        simple_txt_email_file: Path,
+    ) -> None:
+        """
+        GIVEN:
+            - An email whose parsed date is naive (no tzinfo)
+        WHEN:
+            - The .eml file is parsed into a MailMessage
+        THEN:
+            - The resulting message date is made timezone-aware
+        """
+        mock_message = mocker.Mock(
+            from_values="mail@someserver.de",
+            date=datetime.datetime(2022, 10, 12, 21, 40, 43),
+        )
+        mocker.patch(
+            "paperless.parsers.mail.MailMessage.from_bytes",
+            return_value=mock_message,
+        )
+
+        parsed_msg = mail_parser.parse_file_to_message(simple_txt_email_file)
+
+        assert timezone.is_aware(parsed_msg.date)
+        assert parsed_msg.date.replace(tzinfo=None) == datetime.datetime(
+            2022,
+            10,
+            12,
+            21,
+            40,
+            43,
+        )
+
 
 class TestEmailMetadataExtraction:
     """

src/paperless/tests/parsers/test_tika_parser.py

@@ -223,4 +223,11 @@ class TestTikaParser:
 
         assert form_field_found
 
+        # Field updates must be disabled so LibreOffice preserves document
+        # fields (e.g. auto-date fields) as authored rather than refreshing
+        # them to the current date.
+        assert any(
+            b'name="updateIndexes"' in part and b"false" in part for part in parts
+        )
+
         httpx_mock.reset()

src/paperless/version.py

@@ -1,6 +1,6 @@
 from typing import Final
 
-__version__: Final[tuple[int, int, int]] = (3, 0, 1)
+__version__: Final[tuple[int, int, int]] = (3, 0, 2)
 # Version string like X.Y.Z
 __full_version_str__: Final[str] = ".".join(map(str, __version__))
 # Version string like X.Y

src/paperless_mail/migrations/0001_1_clamp_mailrule_maximum_age.py

@@ -1,27 +0,0 @@
-# Split out of 0002_optimize_integer_field_sizes.py into its own migration
-# (own transaction). Running this UPDATE in the same transaction as the
-# later ALTER TABLE on paperless_mail_mailrule can fail on PostgreSQL with
-# "cannot ALTER TABLE because it has pending trigger events", since the
-# update queues FK trigger events against that table that are still pending
-# when the subsequent AlterField tries to rewrite it. Committing the clamp
-# here first avoids that.
-from django.db import migrations
-
-
-def clamp_mailrule_maximum_age(apps, schema_editor):
-    # Clamp the maximum_age field of MailRule because of PositiveIntegerField --> PositiveSmallIntegerField
-    MailRule = apps.get_model("paperless_mail", "MailRule")
-    MailRule.objects.filter(maximum_age__gt=32767).update(maximum_age=32767)
-
-
-class Migration(migrations.Migration):
-    dependencies = [
-        ("paperless_mail", "0001_squashed"),
-    ]
-
-    operations = [
-        migrations.RunPython(
-            clamp_mailrule_maximum_age,
-            migrations.RunPython.noop,
-        ),
-    ]

src/paperless_mail/migrations/0002_optimize_integer_field_sizes.py

@@ -4,9 +4,19 @@ from django.db import migrations
 from django.db import models
 
 
+def clamp_mailrule_maximum_age(apps, schema_editor):
+    # Clamp the maximum_age field of MailRule because of PositiveIntegerField --> PositiveSmallIntegerField
+    MailRule = apps.get_model("paperless_mail", "MailRule")
+    MailRule.objects.filter(maximum_age__gt=32767).update(maximum_age=32767)
+
+
 class Migration(migrations.Migration):
+    # The data update must commit before PostgreSQL rewrites the table, otherwise
+    # pending foreign-key trigger events can block the subsequent ALTER TABLE.
+    atomic = False
+
     dependencies = [
-        ("paperless_mail", "0001_1_clamp_mailrule_maximum_age"),
+        ("paperless_mail", "0001_squashed"),
     ]
 
     operations = [
@@ -112,6 +122,10 @@ class Migration(migrations.Migration):
                 verbose_name="consumption scope",
             ),
         ),
+        migrations.RunPython(
+            clamp_mailrule_maximum_age,
+            migrations.RunPython.noop,
+        ),
         migrations.AlterField(
             model_name="mailrule",
             name="maximum_age",

src/paperless_mail/tests/test_migrations.py

@@ -0,0 +1,37 @@
+import importlib
+
+from django.db import migrations
+
+
+def test_optimize_integer_fields_preserves_released_migration_history():
+    migration_module = importlib.import_module(
+        "paperless_mail.migrations.0002_optimize_integer_field_sizes",
+    )
+
+    assert migration_module.Migration.dependencies == [
+        ("paperless_mail", "0001_squashed"),
+    ]
+
+
+def test_mailrule_maximum_age_is_clamped_before_non_atomic_alter():
+    migration_module = importlib.import_module(
+        "paperless_mail.migrations.0002_optimize_integer_field_sizes",
+    )
+    migration = migration_module.Migration
+
+    clamp_index = next(
+        index
+        for index, operation in enumerate(migration.operations)
+        if isinstance(operation, migrations.RunPython)
+        and operation.code is migration_module.clamp_mailrule_maximum_age
+    )
+    alter_index = next(
+        index
+        for index, operation in enumerate(migration.operations)
+        if isinstance(operation, migrations.AlterField)
+        and operation.model_name == "mailrule"
+        and operation.name == "maximum_age"
+    )
+
+    assert migration.atomic is False
+    assert clamp_index < alter_index

uv.lock

@@ -2880,7 +2880,7 @@ wheels = [
 
 [[package]]
 name = "paperless-ngx"
-version = "3.0.1"
+version = "3.0.2"
 source = { virtual = "." }
 dependencies = [
     { name = "azure-ai-documentintelligence", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },