amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-09-22 17:38:32 +00:00
Code Issues Projects Releases Wiki Activity
12,136 Commits 27 Branches 233 Tags
530059c5c0be4e6ac2c5022ea00871b46988a5d5
Commit Graph
196 Commits
Author SHA1 Message Date
Trenton H 530059c5c0 Performance: Preprocess classifier text with Tantivy instead of NLTK (#14127) 
* Preprocesses classifier content with Tantivy instead of NLTK

Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.

Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.

* Copies packages instead of hardlinking them in backend CI, some NLTK thing

* Adds a normalization to NFC to better fit what Tantivy expects
 2026-09-16 07:35:26 -07:00
shamoon c9a5607902 Enhancement: allow regex timeout configuration (#14085) 2026-09-13 21:57:24 +00:00
Philipp Defner 3e56dace73 Enhancement: Improve matching for correspondents, storage path and labels by removing bias + adding minimum match threshold (#12164) 2026-09-09 22:40:24 +00:00
Trenton H d78754bff1 Security: validate remote OCR endpoint against internal SSRF (#13897) 
* Security: validate remote OCR endpoint against internal SSRF

Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.

* Validates te outbound url again right before use

* cover empty-value branch of validate_remote_ocr_endpoint because coverage

* re-validate remote OCR endpoint on every outbound request
 2026-09-01 20:22:10 +00:00
shamoon 94651d817d Merge branch 'dev' 2026-08-26 18:42:21 -07:00
shamoon aad75b4981 Docs: clarify app config trust 2026-08-26 01:39:50 -07:00
shamoon 7f1609332a Allow parsers to declare uses remote, and remote ocr_mode (#13634) 
* uses_remote_service + allow_remote to allow opt-in / out of remote OCR

* Add to parser dev docs

* remote_ocr_mode config setting

* Checks for remote_ocr_mode and fix import

* Update config.component.spec.ts

* More tests for remote_ocr_mode

* Docs for remote_ocr_mode

* Ok, wire up the remote_ocr_mode with allow_remote for consumer

* Update consumer.py

* Format remote OCR mode check tests

* Use get_choice_from_env
 2026-08-24 14:43:04 -07:00
shamoon a510d03c77 Merge branch 'main' into dev 2026-08-22 05:27:10 -07:00
shamoon bed95ea301 Tweakhancement: add jitter to IMAP polling schedule (#13734) 2026-08-20 17:19:47 +00:00
BeSovereignshamoonSoleroTGstumpylog
 		f1c8a72f26 Enhancement: sync OIDC groups to superuser and staff roles (#13060) 
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Co-authored-by: SoleroTG <github-29h@solero.quietmail.eu>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
 2026-08-19 15:27:56 +00:00
Trenton H 6a02b87dde Feature: Updates remote OCR parser to respect the OCR mode setting (#13408) 
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks

* Updates the documentation to be correct about the respecting now

* merge conflict fixing
 2026-08-11 19:23:46 +00:00
shamoon e2c284f64e Documentation: add wiki links for AI stuff and parser plugins (#13626) 2026-08-09 19:05:32 -07:00
shamoon 882dfa78d2 Documentation: PAPERLESS_CONSUMER_IGNORE_PATTERNS clarifications (#13489) 2026-08-01 14:20:31 -07:00
Trenton H dcff067dc1 Fix: don't skip OCR/archive for tagged PDFs with no actual text (#13351) 2026-07-27 16:52:29 -07:00
Sandro eda79603fe Documentation: fix PAPERLESS_AI_LLM_OUTPUT_LANGUAGE heading level (#13341) 2026-07-27 09:37:09 -07:00
shamoon 4f0845b094 Fixhancement: PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT (#13281) 2026-07-25 00:12:12 -07:00
shamoon c2d9c52cde Merge branch 'main' into beta 2026-07-22 10:30:57 -07:00
Trenton Handshamoon bb5d7438b1 Documentation (beta): Updates documentation for new v3 features (#13033) 
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-06-18 16:20:31 -07:00
Trenton Handupmcplanetracker a009ea1f04 Chore(beta): Update suggested broker to Valkey + Redis agnostic documentation (#13032) 
Co-authored-by: upmcplanetracker <219436948+upmcplanetracker@users.noreply.github.com>
 2026-06-18 19:54:15 +00:00
shamoon 262183e848 Enhancement (beta): support LLM timeout config (#13002) 2026-06-18 08:35:11 -07:00
shamoon fd0168ac6f Documentation: fix PAPERLESS_DATE_PARSER_LANGUAGES formatting (#12997) 2026-06-12 23:26:45 -07:00
shamoon 289d797837 Merge branch 'dev' into beta 2026-06-03 15:12:44 -07:00
shamoon 1663ed170c Enhancement (beta): add direct LLM language setting (#12906) 2026-06-03 15:53:22 +00:00
shamoon ab8fe0521b Merge branch 'beta' into dev 2026-06-02 08:32:54 -07:00
shamoon 2638554969 Merge branch 'main' into dev 2026-06-02 08:32:43 -07:00
shamoon f6c865bf47 Enhancement: AI LLM chunk size and context window config (#12891) 2026-06-01 17:56:21 +00:00
Matt Van Horn 45ba35dd3a docs: remove duplicate words in three files (#12852) 2026-05-26 06:40:30 -07:00
Trenton H 9a68dcdddf Fix: Allow setting allauth rate limit configuration settings (#12798) 2026-05-14 07:29:49 -07:00
shamoon 79d0a04df0 Enhancement: support ollama embeddings (#12753) 2026-05-09 00:06:14 +00:00
shamoon 69cb4d06c6 Enhancement (dev): Use OpenAI-like backend (#12668) 2026-04-28 10:06:59 -07:00
Trenton H 21db608d57 Feature: Tune SQLite default settings for increased speed (#12580) 2026-04-15 15:26:00 -07:00
Trenton HClaude Sonnet 4.6shamoon
 		c232d443fa Breaking: Decouple OCR control from archive file control (#12448) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-04-06 15:50:21 -07:00
shamoon e7c7978d67 Enhancement: allow opt-in blocking internal mail hosts (#12502) 2026-04-03 03:24:28 +00:00
Trenton H dda05a7c00 Security: Improve overall security in a few ways (#12501) 
- Make sure we're always using regex with timeouts for user controlled data
- Adds rate limiting to the token endpoint (configurable)
- Signs the classifier pickle file with the SECRET_KEY and refuse to load one which doesn't verify.
- Require the user to set a secret key, instead of falling back to our old hard coded one
 2026-04-02 15:30:26 -07:00
Trenton HClaude Sonnet 4.6Antoine Mérino
 		aed9abe48c Feature: Replace Whoosh with tantivy search backend (#12471) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Antoine Mérino <3023499+Merinorus@users.noreply.github.com>
 2026-04-02 12:38:22 -07:00
shamoon d671e34559 Documentation: OIDC token_auth_method setting for v3 (#12398) 2026-03-19 22:03:00 -07:00
shamoon 4d4f30b5f8 Security: validate outbound llm URLs and block internal endpoints 2026-03-16 22:58:16 -07:00
dependabot[bot]andshamoon e2947ccff2 Chore(deps): Bump the pre-commit-dependencies group with 4 updates (#12323) 
* Chore(deps): Bump the pre-commit-dependencies group with 4 updates

---
updated-dependencies:
- dependency-name: https://github.com/codespell-project/codespell
  dependency-version: 2.4.2
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: prettier
  dependency-version: 3.8.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pre-commit-dependencies
- dependency-name: prettier-plugin-organize-imports
  dependency-version: 4.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/lovesegfault/beautysh
  dependency-version: 6.4.3
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Drop this, it seems more trouble than its worth

* Re-run prek with new prettier

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-03-12 16:29:57 +00:00
Trenton H 5988d5896b Breaking: Refactor advanced database settings to allow more user configuration (#12165) 2026-02-27 14:37:26 -08:00
Trenton H 8db1c4e08b Breaking: Remove pybzar as a barcode reader (#12065) 2026-02-13 08:14:00 -08:00
Christoph Schober d16d3fb618 Feature: support split documents based on tag barcodes (#11645) 2026-01-29 08:05:33 -08:00
shamoon 1f074390e4 Feature: sharelink bundles (#11682) 2026-01-27 18:54:51 +00:00
shamoon 4428354150 Feature: allow duplicates with warnings, UI for discovery (#11815) 2026-01-26 18:55:08 +00:00
Gabgobieandshamoon e9e138e62c Enhancement: configurable SSO groups claim (#11841) 
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-01-26 18:31:01 +00:00
Trenton H 51b466a86b Feature: Simplify and improve the consumer (#11753) 2026-01-21 14:37:48 -08:00
shamoon 94a5af66eb Fix default llama3.1 2026-01-14 15:36:01 -08:00
shamoon 9c0de249a6 Merge branch 'main' into dev 2026-01-13 11:53:40 -08:00
shamoon e940764fe0 Feature: Paperless AI (#10319) 2026-01-13 16:24:42 +00:00
shamoon 58d88440f1 Feature: Remote OCR (Azure AI) (#10320) 2026-01-08 21:49:17 +00:00
Antoine Mérino 65aed2405c Documentation: update notes for DB pool size (#11600) 2025-12-30 13:06:21 -08:00