From d76dbf5366763e57724d923501fbba964b13879f Mon Sep 17 00:00:00 2001 From: Trenton H <797416+stumpylog@users.noreply.github.com> Date: Fri, 24 Jul 2026 09:18:07 -0700 Subject: [PATCH] Documentation: Add the NumPy CPU baseline increase to the migration guide (#13269) --- docs/migration-v3.md | 64 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 61 insertions(+), 3 deletions(-) diff --git a/docs/migration-v3.md b/docs/migration-v3.md index 6b8dec659..a8b75480b 100644 --- a/docs/migration-v3.md +++ b/docs/migration-v3.md @@ -133,7 +133,7 @@ PAPERLESS_DB_OPTIONS="sslmode=require,sslrootcert=/certs/ca.pem,pool.max_size=10 ## OCR and Archive File Generation Settings -The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** — old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment. +The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** - old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment. ### Removed settings @@ -166,7 +166,7 @@ Remove any `PAPERLESS_OCR_SKIP_ARCHIVE_FILE` variable from your environment. If # v2: skip OCR when text present, always archive PAPERLESS_OCR_MODE=skip # v3: equivalent (auto is the new default) -# No change needed — auto is the default +# No change needed - auto is the default # v2: skip OCR when text present, skip archive too PAPERLESS_OCR_MODE=skip_noarchive @@ -189,7 +189,7 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto If you use the **remote OCR parser** (Azure AI), note that it always produces a searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never` -has no effect for documents handled by the remote parser — the archive is produced +has no effect for documents handled by the remote parser - the archive is produced unconditionally by the remote engine. ## Search Index (Whoosh -> Tantivy) @@ -327,6 +327,64 @@ behind a reverse proxy may need to set [`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER), or both, to avoid `403 Forbidden` errors on login. +## Minimum CPU Requirements (NumPy Baseline) + +Starting with NumPy 2.4.0, official `manylinux` x86_64 wheels are compiled with a minimum +CPU baseline of `x86-64-v2`, which requires SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, and +CMPXCHG16B. This is a [deliberate upstream change](https://github.com/numpy/numpy/issues/27851) +citing that these instructions have been present in over 99.7% of CPUs since 2008 +([Intel]()) or 2011 +([AMD]()). + +NumPy is a dependency of the document classifier (via scikit-learn), so any CPU that +predates SSE4.2 support will crash with `SIGILL` (illegal instruction) when the classifier +is loaded or trained, regardless of whether AI features are enabled. + +This differs from NumPy's optional SIMD dispatch (e.g. AVX2, AVX512), which is detected and +selected safely at runtime - the `x86-64-v2` requirement above is a hard floor baked into the +wheel, with no runtime fallback. + +### Affected hardware + +CPUs older than roughly 2008 (Intel) or 2011 (AMD) that lack SSE4.2 support. This is more +likely to affect low-power or embedded hardware - e.g. early Atom, Celeron, or pre-Bulldozer +AMD chips - than typical desktop or server hardware from the last decade. + +Check for SSE4.2 support with: + +```bash +grep -o -m1 sse4_2 /proc/cpuinfo +``` + +If this prints nothing, your CPU is affected. + +### Symptoms + +The Celery worker (and potentially the web server) repeatedly crashes and restarts with a +`SIGILL` error, typically visible in `dmesg`/`journalctl` as a `trap invalid opcode` inside +`_multiarray_umath...so`. Because the classifier is trained on a periodic schedule +(hourly, by default), affected instances see intermittent, hard-to-reproduce document +consumption failures whenever that scheduled task runs and takes down the worker process +mid-task. + +### Action Required (for affected hardware only) + +There is no way to make the classifier itself work on such CPUs - it requires an unofficial +NumPy build with `cpu-baseline=none`, which is not something we can ship. The practical +path forward is to stop the classifier from ever loading or training, which avoids +importing NumPy at all: + +```bash +PAPERLESS_TRAIN_TASK_CRON=disable +``` + +This disables the periodic classifier training task (see +[`PAPERLESS_TRAIN_TASK_CRON`](configuration.md#PAPERLESS_TRAIN_TASK_CRON)). Automatic +matching based on the classifier (suggested correspondents, document types, tags, and +storage paths from trained rules) will no longer be available, but rule-based matching is +unaffected, and document consumption itself will no longer be at risk of crashing the +worker. + ## Database Migrations Some integer fields have been changed to smaller types to reduce database size. If you have any `MailRule` records with a `maximum_age` greater than 32767, they will be clamped to 32767 during the migration to avoid errors during migration.