stumpylog
e9b56abfe9
Chore: Move unittest directory setup onto the shared fixture
...
The unittest side of the suite built its temp directory tree with
tempfile.mkdtemp and a manually enabled override_settings, cleaned up only if
tearDown ran. That is now gone. DirectoriesMixin lives alongside the layout it
bridges and does nothing but hand the paperless_dirs fixture to TestCase
subclasses as self.dirs, so both halves of the suite get the same twelve
settings, the same directory shapes and cleanup owned by tmp_path.
The mixin moves to paperless_testing.dirs rather than staying in the documents
test utilities, because modules in paperless and paperless_mail import it
across the app boundary. The thirty-eight consuming modules change only their
import line; self.dirs.scratch_dir and its siblings keep working.
2026-09-21 08:08:49 -07:00
stumpylog
e4bbe2123d
Chore: Move model factories to the shared test layer
...
The model factories lived in the documents test package, but three other
apps needed them. The AI, mail and testing suites all reached across an app
boundary to import from documents.tests.factories, which made a private test
package into a shared dependency.
The factories now live in the shared testing package, where cross-app use is
the intended use.
2026-09-21 08:08:49 -07:00
shamoon
405ab39b40
Chore: drop classifier match default threshold to 0.3
2026-09-18 19:10:13 -07:00
Trenton H
530059c5c0
Performance: Preprocess classifier text with Tantivy instead of NLTK ( #14127 )
...
* Preprocesses classifier content with Tantivy instead of NLTK
Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.
Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.
* Copies packages instead of hardlinking them in backend CI, some NLTK thing
* Adds a normalization to NFC to better fit what Tantivy expects
2026-09-16 07:35:26 -07:00
Trenton H
f919c981e1
Peformance: Improves the memory efficency of classifier training ( #14124 )
2026-09-15 13:35:44 -07:00
Trenton H
ec4e45ecf4
Performance: Streams the classifier pickle file during save to file ( #14121 )
2026-09-15 19:35:18 +00:00
Trenton H
ff1e61b162
Fix: Use prefetching to reduce query counts during classifier training ( #14122 )
2026-09-15 11:12:08 -07:00
Philipp Defner
3e56dace73
Enhancement: Improve matching for correspondents, storage path and labels by removing bias + adding minimum match threshold ( #12164 )
2026-09-09 22:40:24 +00:00
shamoon
ae70b8d60f
Chore: consolidate pickle hmac signing ( #13899 )
2026-09-01 12:41:45 -07:00
Trenton H
dda05a7c00
Security: Improve overall security in a few ways ( #12501 )
...
- Make sure we're always using regex with timeouts for user controlled data
- Adds rate limiting to the token endpoint (configurable)
- Signs the classifier pickle file with the SECRET_KEY and refuse to load one which doesn't verify.
- Require the user to set a secret key, instead of falling back to our old hard coded one
2026-04-02 15:30:26 -07:00
Sebastian Steinbeißer
3b5ffbf9fa
Chore(mypy): Annotate None returns for typing improvements ( #11213 )
2026-02-02 08:44:12 -08:00
Sebastian Steinbeißer
d2064a2535
Chore: switch from os.path to pathlib.Path ( #10539 )
2025-09-03 08:12:41 -07:00
Antoine Mérino
1bee1495cf
Performance: Classifier performance optimizations ( #10363 )
2025-08-06 16:00:11 -04:00
Sebastian Steinbeißer
76d363f22d
Chore: switch from os.path to pathlib.Path ( #9060 )
2025-03-05 21:06:01 +00:00
Sebastian Steinbeißer and shamoon
935d077836
Chore: Switch from os.path to pathlib.Path ( #8325 )
...
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com >
2025-01-06 12:12:27 -08:00
Trenton H
41a3c7c89b
Fix: Catch new warning when loading the classifier ( #5395 )
2024-01-14 13:21:17 -08:00
shamoon
f525ac0af6
Chore: add pre-commit hook for codespell ( #5324 )
2024-01-08 13:03:05 -08:00
Trenton H
6f163111ce
Upgrades black to v23, upgrades ruff
2023-04-26 09:35:27 -07:00
Trenton H
3bcbd05252
Fixes ruff not running isort against the codebase
2023-04-26 09:35:27 -07:00
Trenton H
ce41ac9158
Configures ruff as the one stop linter and resolves warnings it raised
2023-04-01 17:03:52 -07:00
Trenton Holmes
6b939f7567
Returns to using hashing against primary keys, at least for fields. Improves testing coverage
2023-02-28 08:13:10 -08:00
Trenton Holmes
c958a7c593
Changes from a hash based system to a time based system to prevent extra retrains
2023-02-28 08:13:10 -08:00
Trenton H
d7939ca958
Fixes some sample test files showing as modified after running tests
2023-01-05 08:39:48 -08:00
Trenton Holmes
f7cd6974c5
Mock out the nltk portions so the data doesn't need to be downloaded
2022-10-10 08:58:23 -07:00
Trenton Holmes
77fbbe95ff
Updates the classifier to catch warnings from scikit-learn and rebuild the model file when this happens
2022-07-05 08:20:35 +02:00
Trenton Holmes
1aeb95396b
Updates scikit-learn and fixes the warning this normally generates in the unit tests
2022-07-05 08:20:35 +02:00
69ef26dab0
Feature: Dynamic document storage pathes ( #916 )
...
* Added devcontainer
* Add feature storage pathes
* Exclude tests and add versioning
* Check escaping
* Check escaping
* Check quoting
* Echo
* Escape
* Escape :
* Double escape \
* Escaping
* Remove if
* Escape colon
* Missing \
* Esacpe :
* Escape all
* test
* Remove sed
* Fix exclude
* Remove SED command
* Add LD_LIBRARY_PATH
* Adjusted to v1.7
* Updated test-cases
* Remove devcontainer
* Removed internal build-file
* Run pre-commit
* Corrected flak8 error
* Adjusted to v1.7
* Updated test-cases
* Corrected flak8 error
* Adjusted to new plural translations
* Small adjustments due to code-review backend
* Adjusted line-break
* Removed PAPERLESS prefix from settings variables
* Corrected style change due to search+replace
* First documentation draft
* Revert changes to Pipfile
* Add sphinx-autobuild with keep-outdated
* Revert merge error that results in wrong storage path is evaluated
* Adjust styles of generated files ...
* Adds additional testing to cover dynamic storage path functionality
* Remove unnecessary condition
* Add hint to edit storage path dialog
* Correct spelling of pathes to paths
* Minor documentation tweaks
* Minor typo
* improving wrapping of filter editor buttons with new storage path button
* Update .gitignore
* Fix select border radius in non input-groups
* Better storage path edit hint
* Add note to edit storage path dialog re document_renamer
* Add note to bulk edit storage path re document_renamer
* Rename FILTER_STORAGE_DIRECTORY to PATH
* Fix broken filter rule parsing
* Show default storage if unspecified
* Remove note re storage path on bulk edit
* Add basic validation of filename variables
Co-authored-by: Markus Kling <markus@markus-kling.net >
Co-authored-by: Trenton Holmes <holmes.trenton@gmail.com >
Co-authored-by: Michael Shamoon <4887959+shamoon@users.noreply.github.com >
Co-authored-by: Quinn Casey <quinn@quinncasey.com >
2022-05-19 14:42:25 -07:00
Trenton Holmes
3003bdd507
Runs pyupgrade to Python 3.8+ and adds a hook for it
2022-05-06 09:04:08 -07:00
Trenton Holmes
1771d18a21
Runs the pre-commit hooks over all the Python files
2022-03-11 11:34:28 -08:00
kpj
fc695896dd
Format Python code with black
2022-02-27 15:26:41 +01:00
jonaswinkler
b48e67d714
revert a faulty change that caused memory usage to explode #537
2021-02-13 19:51:04 +01:00
jonaswinkler
ffe96c8fff
classifier caching
2021-02-06 20:54:58 +01:00
jonaswinkler
d8e0ef257e
don't load sklearn libraries unless needed
2021-02-04 15:15:11 +01:00
jonaswinkler
87a18eae2d
centralized classifier loading, better error handling, no error messages when auto matching is not used
2021-01-30 14:22:23 +01:00
jonaswinkler
bc4192e7d1
more tests and bugfixes.
2020-11-27 15:36:32 +01:00
jonaswinkler
4bf0d834a0
improved test cases. Python 3.6 compatibility.
2020-11-26 22:17:14 +01:00
Jonas Winkler
30acfdd3f1
tests for the classifier and fixes for edge cases with minimal data.
2020-11-26 14:18:34 +01:00
Jonas Winkler
727f86c369
codestyle
2020-11-18 22:41:14 +01:00
Jonas Winkler
43b5d61483
codestyle changes
2020-11-17 17:13:08 +01:00
Jonas Winkler
7164f21fbe
a new test for the document classifier
2020-11-17 02:50:17 +01:00
stumpylog
shamoon
Philipp Defner
Sebastian Steinbeißer
Antoine Mérino
Trenton Holmes
Trenton Holmes
kpj
jonaswinkler