Performance: Preprocess classifier text with Tantivy instead of NLTK (#14127)

* Preprocesses classifier content with Tantivy instead of NLTK

Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.

Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.

* Copies packages instead of hardlinking them in backend CI, some NLTK thing

* Adds a normalization to NFC to better fit what Tantivy expects

docs/configuration.md

@@ -413,18 +413,12 @@ details.
 
     Defaults to `PAPERLESS_DATA_DIR/log/`.
 
-#### [`PAPERLESS_NLTK_DIR=<path>`](#PAPERLESS_NLTK_DIR) {#PAPERLESS_NLTK_DIR}
+#### ~~[`PAPERLESS_NLTK_DIR`](#PAPERLESS_NLTK_DIR)~~ {#PAPERLESS_NLTK_DIR}
 
-: This is where paperless will search for the data required for NLTK
-processing, if you are using it. If you are using the Docker image,
-this should not be changed, as the data is included in the image
-already.
+!!! failure "Removed in v3.2"
 
-Previously, the location defaulted to `PAPERLESS_DATA_DIR/nltk`.
-Unless you are using this in a bare metal install or other setup,
-this folder is no longer needed and can be removed manually.
-
-Defaults to `/usr/share/nltk_data`
+    Removed and ignored. Any previously downloaded NLTK data folder can be
+    deleted.
 
 #### [`PAPERLESS_MODEL_FILE=<path>`](#PAPERLESS_MODEL_FILE) {#PAPERLESS_MODEL_FILE}
 
@@ -1190,15 +1184,14 @@ for details on how to set it.
 
     Defaults to UTC.
 
-#### [`PAPERLESS_ENABLE_NLTK=<bool>`](#PAPERLESS_ENABLE_NLTK) {#PAPERLESS_ENABLE_NLTK}
+#### ~~[`PAPERLESS_ENABLE_NLTK`](#PAPERLESS_ENABLE_NLTK)~~ {#PAPERLESS_ENABLE_NLTK}
 
-: Enables or disables the advanced natural language processing
-used during automatic classification. If disabled, paperless will
-still perform some basic text pre-processing before matching.
+!!! failure "Removed in v3.2"
 
-: See also `PAPERLESS_NLTK_DIR`.
-
-    Defaults to true, enabling the feature.
+    Removed and ignored. Automatic classification always removes stop words
+    and stems words when the primary OCR language is Danish, Dutch, English,
+    Finnish, French, German, Italian, Norwegian, Portuguese, Russian, Spanish
+    or Swedish. Other languages are only lowercased and split into words.
 
 #### [`PAPERLESS_CLASSIFIER_MATCH_THRESHOLD=<float>`](#PAPERLESS_CLASSIFIER_MATCH_THRESHOLD) {#PAPERLESS_CLASSIFIER_MATCH_THRESHOLD}
 

docs/setup.md

@@ -430,11 +430,6 @@ to a positive number to enable polling and disable native filesystem notificatio
 This will reduce the size of generated PDF documents. You'll most likely need to compile this yourself, because this
 software has been patented until around 2017 and binary packages are not available for most distributions.
 
-**Optional: download the NLTK data**
-If using the NLTK machine-learning processing (see [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) for details),
-download the NLTK data for the Snowball Stemmer, Stopwords and Punkt tokenizer to `/usr/share/nltk_data`. Refer to the [NLTK
-instructions](https://www.nltk.org/data.html) for details on how to download the data.
-
 #### After installation
 
 Your Paperless-ngx instance should now be accessible at `http://localhost:8000` (or similar, depending on your configuration).
@@ -650,9 +645,6 @@ hardware, but a few settings can improve performance:
   `PAPERLESS_OCR_CLEAN=none`. This will speed up OCR times and use
   less memory at the expense of slightly worse OCR results.
 - If using Docker, consider setting [`PAPERLESS_WEBSERVER_WORKERS`](configuration.md#PAPERLESS_WEBSERVER_WORKERS) to 1. This will save some memory.
-- Consider setting [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) to false, to disable the
-  more advanced language processing, which can take more memory and
-  processing time.
 
 For details, refer to [configuration](configuration.md).