Commit Graph
105 Commits
Author SHA1 Message Date
701735f6e5 Chore: Drop old signal and unneeded apps, transition to parser registry instead (#12405)
* refactor: switch consumer and callers to ParserRegistry (Phase 4)

Replace all Django signal-based parser discovery with direct registry
calls. Removes `_parser_cleanup`, `parser_is_new_style` shims, and all
old-style isinstance checks. All parser instantiation now uses the
`with parser_class() as parser:` context manager pattern.

- documents/parsers.py: delegate to get_parser_registry(); drop lru_cache
- documents/consumer.py: use registry + context manager; remove shims
- documents/tasks.py: same pattern
- documents/management/commands/document_thumbnails.py: same pattern
- documents/views.py: get_metadata uses context manager
- documents/checks.py: use get_parser_registry().all_parsers()
- paperless/parsers/registry.py: add all_parsers() public method
- tests: update mocks to target documents.consumer.get_parser_class_for_mime_type

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: drop get_parser_class_for_mime_type; callers use registry directly

All callers now call get_parser_registry().get_parser_for_file() with
the actual filename and path, enabling score() to use file extension
hints. The MIME-only helper is removed.

- consumer.py: passes self.filename + self.working_copy
- tasks.py: passes document.original_filename + document.source_path
- document_thumbnails.py: same pattern
- views.py: passes Path(file).name + Path(file)
- parsers.py: internal helpers inline the registry call with filename=""
- test_parsers.py: drop TestParserDiscovery (was testing mock behavior);
  TestParserAvailability uses registry directly
- test_consumer.py: mocks switch to documents.consumer.get_parser_registry

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: remove document_consumer_declaration signal infrastructure

Remove the document_consumer_declaration signal that was previously used
for parser registration. Each parser app no longer connects to this signal,
and the signal declaration itself has been removed from documents/signals.

Changes:
- Remove document_consumer_declaration from documents/signals/__init__.py
- Remove ready() methods and signal imports from all parser app configs
- Delete signal shim files (signals.py) from all parser apps:
  - paperless_tesseract/signals.py
  - paperless_text/signals.py
  - paperless_tika/signals.py
  - paperless_mail/signals.py
  - paperless_remote/signals.py

Parser discovery now happens exclusively through the ParserRegistry
system introduced in the previous refactor phases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: remove empty paperless_text and paperless_tika Django apps

After parser classes were moved to paperless/parsers/ in the plugin
refactor, these Django apps contained only empty AppConfig classes
with no models, views, tasks, migrations, or other functionality.

- Remove paperless_text and paperless_tika from INSTALLED_APPS
- Delete empty app directories entirely
- Update pyproject.toml test exclusions
- Clean stale mypy baseline entries for moved parser files

paperless_remote app is retained as it contains meaningful system
checks for Azure AI configuration.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Moves the checks and tests to the main application and removes the old applications

* Adds a comment to satisy Sonar

* refactor: remove automatic log_summary() call from get_parser_registry()

The summary was logged once per process, causing it to appear repeatedly
during Docker startup (management commands, web server, each Celery
worker subprocess). External parsers are already announced individually
at INFO when discovered; the full summary is redundant noise.
log_summary() is retained on ParserRegistry for manual/debug use.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Cleans up the duplicate test file/fixture

* Fixes a race condition where webserver threads could race to populate the registry

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-22 06:53:32 -07:00
Trenton HandGitHub 6a87c3f4dd Fixes handling the case where there is no status reported from celery (due to external termination of the worker) (#12040) 2026-02-08 17:26:35 -08:00
Trenton HandGitHub 2ec8ec96c8 Feature: Enable users to customize date parsing via plugins (#11931) 2026-02-03 20:09:13 +00:00
Sebastian SteinbeißerandGitHub 3b5ffbf9fa Chore(mypy): Annotate None returns for typing improvements (#11213) 2026-02-02 08:44:12 -08:00
Sebastian SteinbeißerandGitHub d2064a2535 Chore: switch from os.path to pathlib.Path (#10539) 2025-09-03 08:12:41 -07:00
shamoonandGitHub cb927c5b22 Fix: include application config language settings for dateparser auto-detection (#10722) 2025-08-31 15:22:39 -07:00
6591d5da63 Performance: Add support for configuring date parser languages (#10181)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2025-07-01 05:57:38 +00:00
Trenton HandGitHub 827fcba277 Chore: Reduce imports for a slight memory improvement (#9217) 2025-02-24 15:06:14 -08:00
Sebastian SteinbeißerandGitHub e560fa3be0 Chore: Enable ruff FBT (#8645) 2025-02-07 09:12:03 -08:00
XstreamGitandGitHub 046d8456e2 Tweak: improve date matching regex for dates after numbers (#8964) 2025-02-06 19:09:35 +00:00
Trenton HandGitHub e6f59472e4 Chore: Drop Python 3.9 support (#7774) 2024-09-26 12:22:24 -07:00
c92c3e224a Feature: page count (#7750)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-09-25 08:22:12 -07:00
Trenton HandGitHub 3cf73a77ac Chore: Initial conversion to pytest fixtures (#7110) 2024-07-08 07:46:20 -07:00
Fabien DubuyandGitHub 4f1185c65d Enhancement: improve date parsing with accented characters (#7100) 2024-06-26 20:47:37 -07:00
Trenton HandGitHub b720aa3cd1 Chore: Convert the consumer to a plugin (#6361) 2024-04-18 02:59:14 +00:00
Trenton HandGitHub 2c43b06910 Chore: Standardize subprocess running and logging (#6275) 2024-04-04 13:11:43 -07:00
Trenton HandGitHub 13201dbfff Ensure all creations of directories create the parents too (#5711) 2024-02-10 11:02:40 -08:00
Henning BandGitHub 38a817e887 Enhancement: Respect PDF cropbox for thumbnail generation (#5531)
* Use the convert argument '-define "pdf:use-cropbox=true"' when creating thumbnails. That way cropboxes, if present, gets respected for thumbnail generation
2024-02-01 11:25:57 -08:00
shamoonandGitHub ba0f4718e5 Fix: Modify one of date regexes (#5540) 2024-01-25 21:29:22 +00:00
shamoonandGitHub f525ac0af6 Chore: add pre-commit hook for codespell (#5324) 2024-01-08 13:03:05 -08:00
Trenton HandGitHub a82e3771ae Fix: Allows pre-consume scripts to modify the working path again (#5260)
* Allows pre-consume scripts to modify the working path again and generally cleans up some confusion about working copy vs original
2024-01-05 21:01:57 -08:00
061f33fb05 Feature: Allow setting backend configuration settings via the UI (#5126)
* Saving some start on this

* At least partially working for the tesseract parser

* Problems with migration testing need to figure out

* Work around that error

* Fixes max m_pixels

* Moving the settings to main paperless application

* Starting some consumer options

* More fixes and work

* Fixes these last tests

* Fix max_length on OcrSettings.mode field

* Fix all fields on Common & Ocr settings serializers

* Umbrellla config view

* Revert "Umbrellla config view"

This reverts commit fbaf9f4be30f89afeb509099180158a3406416a5.

* Updates to use a single configuration object for all settings

* Squashed commit of the following:

commit 8a0a49dd57
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 23:02:47 2023 -0800

    Fix formatting

commit 66b2d90c50
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 22:36:35 2023 -0800

    Refactor frontend data models

commit 5723bd8dd8
Author: Adam Bogdał <adam@bogdal.pl>
Date:   Wed Dec 20 01:17:43 2023 +0100

    Fix: speed up admin panel for installs with a large number of documents (#5052)

commit 9b08ce1761
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 15:18:51 2023 -0800

    Update PULL_REQUEST_TEMPLATE.md

commit a6248bec2d
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 15:02:05 2023 -0800

    Chore: Update Angular to v17 (#4980)

commit b1f6f52486
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 13:53:56 2023 -0800

    Fix: Dont allow null custom_fields property via API (#5063)

commit 638d9970fd
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 13:43:50 2023 -0800

    Enhancement: symmetric document links (#4907)

commit 5e8de4c1da
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 12:45:04 2023 -0800

    Enhancement: shared icon & shared by me filter (#4859)

commit 088bad9030
Author: Trenton H <797416+stumpylog@users.noreply.github.com>
Date:   Tue Dec 19 12:04:03 2023 -0800

    Bulk updates all the backend libraries (#5061)

* Saving some work on frontend config

* Very basic but dynamically-generated config form

* Saving work on slightly less ugly frontend config

* JSON validation for user_args field

* Fully dynamic config form

* Adds in some additional validators for a nicer error message

* Cleaning up the testing and coverage more

* Reverts unintentional change

* Adds documentation about the settings and the precedence

* Couple more commenting and style fixes

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2023-12-29 15:42:56 -08:00
Trenton HandGitHub 5b502b1e1a Use the original image file for the checksum, not the maybe alpha removed version (#4781) 2023-12-02 16:18:06 -08:00
Trenton HandGitHub facb7226fe Chore: Backend bulk updates (#4509) 2023-11-13 17:09:56 +00:00
Trenton H 1a87c730bc Logs the errors during thumbnail generation, updates default to be WebP 2023-09-12 12:16:22 -07:00
Trenton Holmes 650c816a7b Removes support for Python 3.8 and lower from the code base 2023-09-10 11:42:59 -07:00
brainrecursionandTrenton H 0918eab004 Remove debug 2023-07-30 08:34:33 -07:00
brainrecursionandTrenton H 157240351f Add support for more date formats 2023-07-30 08:34:33 -07:00
Trenton H 9f5d47c320 Fixes issues with copy2 or copystat and SELinux see #3665 2023-07-22 06:27:49 -07:00
Johannes PlunienandTrenton H 83a760644d Copy default thumbnail if thumbnail generation fails
Fix #3631
2023-06-20 11:28:46 -07:00
Trenton H 452c79f9a1 Improves the logging mixin and allows it to be typed better 2023-05-23 17:16:39 -07:00
Trenton Holmes 3205d52331 Changes the error mode to replace instead of ignore, to better highlight where a problem happened 2023-05-13 09:29:18 -07:00
Trenton H 111960c530 Adds better handling for files with invalid utf8 content 2023-05-13 09:29:18 -07:00
Trenton H 6f163111ce Upgrades black to v23, upgrades ruff 2023-04-26 09:35:27 -07:00
Trenton H 3bcbd05252 Fixes ruff not running isort against the codebase 2023-04-26 09:35:27 -07:00
Trenton H ce41ac9158 Configures ruff as the one stop linter and resolves warnings it raised 2023-04-01 17:03:52 -07:00
Trenton H c21775980f relock with Python 3.8.15 2023-01-06 17:59:39 -08:00
Trenton H d19bf59f47 Cleans up and improves parser discovery testing, simplifies the determination of supported or not supported extensions and mime types 2023-01-05 08:39:48 -08:00
Trenton H 914661fdbb Don't allow an exception when trying to parse a date cause complete failure 2022-11-17 13:37:37 -08:00
Matthias Eck 3d0a26fdb1 fix(parsers|test_api): fix failed tests 2022-08-06 19:19:10 +02:00
Matthias Eck a5d2ae2588 feat(parsers): add generator for date parsing 2022-08-06 13:03:20 +02:00
Trenton Holmes e8868d7ebf Entirely removes the optipng, updates ghostscript fall back to also use WebP. Updates the conversion to use a multiprocessing pool 2022-06-11 08:38:49 -07:00
Michael Shamoon 58f2c6a5fc webp thumbnail support with png fallback 2022-06-10 02:28:13 -07:00
shamoonandGitHub 536576518e Merge pull request #721 from paperless-ngx/bug-fix-date-ignore
Fix Ignore Date Parsing
2022-05-10 16:45:58 -07:00
Trenton HolmesandTrenton Holmes 5b96944940 Updates the ignore date parsing to utilize the settings defined date order, instead of guessing a bit 2022-05-08 16:57:35 -07:00
Trenton Holmes 8a6aaf4e2d Adds additional testing for both date parsing and consumed document created date 2022-05-08 16:57:35 -07:00
Trenton Holmes 3003bdd507 Runs pyupgrade to Python 3.8+ and adds a hook for it 2022-05-06 09:04:08 -07:00
Fantasticle 0baacbef98 update new regex pattern for second boundary 2022-03-31 09:37:15 +02:00
fantasticleandGitHub 1ecb26a3fb Update regex date match patterns 2022-03-30 12:19:30 +02:00
Simon SiebertandJohann Bauer 54cbacf4f4 Update parsers.py and test_consumer.py 2022-03-14 19:03:09 +01:00