Fix: preserve document fields during Gotenberg conversion (#13271)

Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.

.github/workflows/ci-release.yml

@@ -86,7 +86,7 @@ jobs:
       # ---- Prepare Release ----
       - name: Generate requirements file
         run: |
-          uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt
+          uv export --quiet --no-dev --all-extras --emit-index-url --format requirements-txt --output-file requirements.txt
       - name: Compile messages
         env:
           PAPERLESS_SECRET_KEY: "ci-release-not-a-real-secret"

docs/migration-v3.md

@@ -133,7 +133,7 @@ PAPERLESS_DB_OPTIONS="sslmode=require,sslrootcert=/certs/ca.pem,pool.max_size=10
 
 ## OCR and Archive File Generation Settings
 
-The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** — old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
+The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** - old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
 
 ### Removed settings
 
@@ -166,7 +166,7 @@ Remove any `PAPERLESS_OCR_SKIP_ARCHIVE_FILE` variable from your environment. If
 # v2: skip OCR when text present, always archive
 PAPERLESS_OCR_MODE=skip
 # v3: equivalent (auto is the new default)
-# No change needed — auto is the default
+# No change needed - auto is the default
 
 # v2: skip OCR when text present, skip archive too
 PAPERLESS_OCR_MODE=skip_noarchive
@@ -189,7 +189,7 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
 
 If you use the **remote OCR parser** (Azure AI), note that it always produces a
 searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
-has no effect for documents handled by the remote parser — the archive is produced
+has no effect for documents handled by the remote parser - the archive is produced
 unconditionally by the remote engine.
 
 ## Search Index (Whoosh -> Tantivy)
@@ -327,6 +327,64 @@ behind a reverse proxy may need to set
 [`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER),
 or both, to avoid `403 Forbidden` errors on login.
 
+## Minimum CPU Requirements (NumPy Baseline)
+
+Starting with NumPy 2.4.0, official `manylinux` x86_64 wheels are compiled with a minimum
+CPU baseline of `x86-64-v2`, which requires SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, and
+CMPXCHG16B. This is a [deliberate upstream change](https://github.com/numpy/numpy/issues/27851)
+citing that these instructions have been present in over 99.7% of CPUs since 2008
+([Intel](<https://en.wikipedia.org/wiki/Nehalem_(microarchitecture)>)) or 2011
+([AMD](<https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)>)).
+
+NumPy is a dependency of the document classifier (via scikit-learn), so any CPU that
+predates SSE4.2 support will crash with `SIGILL` (illegal instruction) when the classifier
+is loaded or trained, regardless of whether AI features are enabled.
+
+This differs from NumPy's optional SIMD dispatch (e.g. AVX2, AVX512), which is detected and
+selected safely at runtime - the `x86-64-v2` requirement above is a hard floor baked into the
+wheel, with no runtime fallback.
+
+### Affected hardware
+
+CPUs older than roughly 2008 (Intel) or 2011 (AMD) that lack SSE4.2 support. This is more
+likely to affect low-power or embedded hardware - e.g. early Atom, Celeron, or pre-Bulldozer
+AMD chips - than typical desktop or server hardware from the last decade.
+
+Check for SSE4.2 support with:
+
+```bash
+grep -o -m1 sse4_2 /proc/cpuinfo
+```
+
+If this prints nothing, your CPU is affected.
+
+### Symptoms
+
+The Celery worker (and potentially the web server) repeatedly crashes and restarts with a
+`SIGILL` error, typically visible in `dmesg`/`journalctl` as a `trap invalid opcode` inside
+`_multiarray_umath...so`. Because the classifier is trained on a periodic schedule
+(hourly, by default), affected instances see intermittent, hard-to-reproduce document
+consumption failures whenever that scheduled task runs and takes down the worker process
+mid-task.
+
+### Action Required (for affected hardware only)
+
+There is no way to make the classifier itself work on such CPUs - it requires an unofficial
+NumPy build with `cpu-baseline=none`, which is not something we can ship. The practical
+path forward is to stop the classifier from ever loading or training, which avoids
+importing NumPy at all:
+
+```bash
+PAPERLESS_TRAIN_TASK_CRON=disable
+```
+
+This disables the periodic classifier training task (see
+[`PAPERLESS_TRAIN_TASK_CRON`](configuration.md#PAPERLESS_TRAIN_TASK_CRON)). Automatic
+matching based on the classifier (suggested correspondents, document types, tags, and
+storage paths from trained rules) will no longer be available, but rule-based matching is
+unaffected, and document consumption itself will no longer be at risk of crashing the
+worker.
+
 ## Database Migrations
 
 Some integer fields have been changed to smaller types to reduce database size. If you have any `MailRule` records with a `maximum_age` greater than 32767, they will be clamped to 32767 during the migration to avoid errors during migration.

src/documents/search/_backend.py

@@ -475,7 +475,13 @@ class TantivyBackend:
         note_texts: list[str] = []
         for note in document.notes.all():
             num_notes += 1
-            doc.add_json("notes", {"note": note.note, "user": note.user.username})
+            doc.add_json(
+                "notes",
+                {
+                    "note": note.note,
+                    "user": note.user.username if note.user else None,
+                },
+            )
             note_texts.append(note.note)
         if note_texts:
             doc.add_text("notes_text", " ".join(note_texts))

src/documents/signals/handlers.py

@@ -1264,7 +1264,17 @@ def task_failure_handler(
             "error_message": str(exception) if exception else "Unknown error",
         }
         if traceback:
-            tb_str = "".join(_tb.format_tb(traceback))
+            # billiard/celery pass a pre-formatted string instead of a real
+            # traceback object when the worker process itself died (e.g.
+            # WorkerLostError from a SIGILL) since there's no live traceback
+            # to walk in that case.
+            tb_str = (
+                traceback
+                if isinstance(traceback, str)
+                else "".join(
+                    _tb.format_tb(traceback),
+                )
+            )
             result_data["traceback"] = tb_str[:5000]
 
         now = timezone.now()

src/documents/tests/search/test_backend.py

@@ -844,6 +844,23 @@ class TestFieldHandling:
             f"Expected 1, got {len(ids)}. Note content should be searchable via notes.note: prefix."
         )
 
+    def test_notes_without_user_are_indexed(self, backend: TantivyBackend) -> None:
+        """Notes whose user was deleted (SET_NULL) must not break indexing."""
+        doc = Document.objects.create(
+            title="Doc with orphaned note",
+            content="test",
+            checksum="NT2",
+            pk=81,
+        )
+        Note.objects.create(document=doc, note="Orphaned note", user=None)
+
+        backend.add_or_update(doc)
+
+        ids = backend.search_ids("notes.note:orphaned", user=None)
+        assert len(ids) == 1, (
+            f"Expected 1, got {len(ids)}. Notes without a user should still be indexed."
+        )
+
 
 class TestHighlightHits:
     """Test highlight_hits returns proper HTML strings, not raw Snippet objects."""

src/paperless/parsers/mail.py

@@ -272,10 +272,7 @@ class MailDocumentParser:
         logger.debug("Building formatted text from email")
         self._text = build_formatted_text(mail)
 
-        if is_naive(mail.date):
-            self._date = make_aware(mail.date)
-        else:
-            self._date = mail.date
+        self._date = mail.date
 
         logger.debug("Creating a PDF from the email")
         if self._mailrule_id:
@@ -502,6 +499,9 @@ class MailDocumentParser:
                 f"Could not parse {filepath}: {err}",
             ) from err
 
+        if is_naive(parsed.date):
+            parsed.date = make_aware(parsed.date)
+
         return parsed
 
     def tika_parse(self, html: str) -> str:

src/paperless/parsers/tika.py

@@ -421,6 +421,11 @@ class TikaDocumentParser:
         logger.info("Converting %s to PDF as %s", document_path, pdf_path)
 
         with self._gotenberg_client.libre_office.to_pdf() as route:
+            # Preserve document fields as authored. updateIndexes (Gotenberg's
+            # default) triggers a refresh() that rewrites dynamic fields like
+            # auto-dates to the current date.
+            route.update_indexes(update_indexes=False)
+
             # Set the output format of the resulting PDF.
             # OutputTypeConfig reads the database-stored ApplicationConfiguration
             # first, then falls back to the PAPERLESS_OCR_OUTPUT_TYPE env var.

src/paperless/tests/parsers/test_mail_parser.py

@@ -145,6 +145,41 @@ class TestEmailFileParsing:
         assert parsed_msg.text == "This is just a simple Text Mail.\n"
         assert parsed_msg.to == ("some@one.de",)
 
+    def test_parse_file_to_message_makes_naive_date_aware(
+        self,
+        mocker: MockerFixture,
+        mail_parser: MailDocumentParser,
+        simple_txt_email_file: Path,
+    ) -> None:
+        """
+        GIVEN:
+            - An email whose parsed date is naive (no tzinfo)
+        WHEN:
+            - The .eml file is parsed into a MailMessage
+        THEN:
+            - The resulting message date is made timezone-aware
+        """
+        mock_message = mocker.Mock(
+            from_values="mail@someserver.de",
+            date=datetime.datetime(2022, 10, 12, 21, 40, 43),
+        )
+        mocker.patch(
+            "paperless.parsers.mail.MailMessage.from_bytes",
+            return_value=mock_message,
+        )
+
+        parsed_msg = mail_parser.parse_file_to_message(simple_txt_email_file)
+
+        assert timezone.is_aware(parsed_msg.date)
+        assert parsed_msg.date.replace(tzinfo=None) == datetime.datetime(
+            2022,
+            10,
+            12,
+            21,
+            40,
+            43,
+        )
+
 
 class TestEmailMetadataExtraction:
     """

src/paperless/tests/parsers/test_tika_parser.py

@@ -223,4 +223,11 @@ class TestTikaParser:
 
         assert form_field_found
 
+        # Field updates must be disabled so LibreOffice preserves document
+        # fields (e.g. auto-date fields) as authored rather than refreshing
+        # them to the current date.
+        assert any(
+            b'name="updateIndexes"' in part and b"false" in part for part in parts
+        )
+
         httpx_mock.reset()