amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-09-21 17:08:33 +00:00
Code Issues Projects Releases Wiki Activity

Performance: Preprocess classifier text with Tantivy instead of NLTK (#14127)

Browse Source 
* Preprocesses classifier content with Tantivy instead of NLTK

Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.

Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.

* Copies packages instead of hardlinking them in backend CI, some NLTK thing

* Adds a normalization to NFC to better fit what Tantivy expects
This commit is contained in:
Trenton H
2026-09-16 07:35:26 -07:00
committed by GitHub
parent 989f138556
commit 530059c5c0
15 changed files with 441 additions and 378 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/setup.md
-8
View File
@@ -430,11 +430,6 @@ to a positive number to enable polling and disable native filesystem notificatio
This will reduce the size of generated PDF documents. You'll most likely need to compile this yourself, because this
software has been patented until around 2017 and binary packages are not available for most distributions.
**Optional: download the NLTK data**
If using the NLTK machine-learning processing (see [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) for details),
download the NLTK data for the Snowball Stemmer, Stopwords and Punkt tokenizer to `/usr/share/nltk_data`. Refer to the [NLTK
instructions](https://www.nltk.org/data.html) for details on how to download the data.
#### After installation
Your Paperless-ngx instance should now be accessible at `http://localhost:8000` (or similar, depending on your configuration).
@@ -650,9 +645,6 @@ hardware, but a few settings can improve performance:
`PAPERLESS_OCR_CLEAN=none`. This will speed up OCR times and use
less memory at the expense of slightly worse OCR results.
- If using Docker, consider setting [`PAPERLESS_WEBSERVER_WORKERS`](configuration.md#PAPERLESS_WEBSERVER_WORKERS) to 1. This will save some memory.
- Consider setting [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) to false, to disable the
more advanced language processing, which can take more memory and
processing time.
For details, refer to [configuration](configuration.md).