mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-09-22 09:28:32 +00:00
0c8ca5684dae91f8a603dfd04f18539e27ffc8ce
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 stumpylog
|
0c8ca5684d |
Chore: Move model factories to the shared test layer
The model factories lived in the documents test package, but three other apps needed them. The AI, mail and testing suites all reached across an app boundary to import from documents.tests.factories, which made a private test package into a shared dependency. The factories now live in the shared testing package, where cross-app use is the intended use. |
||
|
stumpylog
|
4abdd6d8b6 |
Chore: Add a shared test support layer
All four Django apps have test code in common, but the only place to put it was the documents app's own tests package, so paperless, paperless_ai and paperless_mail each reached across an app boundary to import helpers and relied on fixtures that were only defined for the documents test path. This adds a root src/conftest.py holding the fixtures every app needs and an new src/paperless_testing package for shared helpers a test names |
||
|
stumpylog
|
46db7217ee | Chore: Remove unused test helpers | ||
|
Trenton H
|
fdd32b73bb | Chore: Move the search tests onto the shared indexing fixtures (#14159) | ||
 dependabot[bot]andTrenton Holmes
|
f287a4cb8c |
Chore(deps): Bump the utilities-minor group across 1 directory with 11 updates (#13988)
* Chore(deps): Bump the utilities-minor group across 1 directory with 11 updates Bumps the utilities-minor group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [django-guardian](https://github.com/django-guardian/django-guardian) | `3.3.3` | `3.4.0` | | [django-treenode](https://github.com/fabiocaccamo/django-treenode) | `0.24.0` | `0.25.0` | | [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) | `2026.7.1` | `2026.8.1` | | [imap-tools](https://github.com/ikvk/imap_tools) | `1.14.0` | `1.15.0` | | [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF) | `17.10.0` | `17.11.0` | | [prek](https://github.com/j178/prek) | `0.4.11` | `0.5.0` | | [faker](https://github.com/joke2k/faker) | `40.36.0` | `40.37.0` | | [pytest-django](https://github.com/pytest-dev/pytest-django) | `4.12.0` | `4.14.0` | | [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) | `16.4` | `16.6` | | [time-machine](https://github.com/adamchainz/time-machine) | `3.2.0` | `3.5.0` | | [types-pygments](https://github.com/python/typeshed) | `2.20.0.20260408` | `2.21.0.20260819` | Updates `django-guardian` from 3.3.3 to 3.4.0 - [Release notes](https://github.com/django-guardian/django-guardian/releases) - [Commits](https://github.com/django-guardian/django-guardian/compare/3.3.3...3.4.0) Updates `django-treenode` from 0.24.0 to 0.25.0 - [Release notes](https://github.com/fabiocaccamo/django-treenode/releases) - [Changelog](https://github.com/fabiocaccamo/django-treenode/blob/main/CHANGELOG.md) - [Commits](https://github.com/fabiocaccamo/django-treenode/compare/0.24.0...0.25.0) Updates `drf-spectacular-sidecar` from 2026.7.1 to 2026.8.1 - [Commits](https://github.com/tfranzel/drf-spectacular-sidecar/compare/2026.7.1...2026.8.1) Updates `imap-tools` from 1.14.0 to 1.15.0 - [Release notes](https://github.com/ikvk/imap_tools/releases) - [Changelog](https://github.com/ikvk/imap_tools/blob/master/docs/release_notes.rst) - [Commits](https://github.com/ikvk/imap_tools/compare/v1.14.0...v1.15.0) Updates `ocrmypdf` from 17.10.0 to 17.11.0 - [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases) - [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v17.10.0...v17.11.0) Updates `prek` from 0.4.11 to 0.5.0 - [Release notes](https://github.com/j178/prek/releases) - [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md) - [Commits](https://github.com/j178/prek/compare/v0.4.11...v0.5.0) Updates `faker` from 40.36.0 to 40.37.0 - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](https://github.com/joke2k/faker/compare/v40.36.0...v40.37.0) Updates `pytest-django` from 4.12.0 to 4.14.0 - [Release notes](https://github.com/pytest-dev/pytest-django/releases) - [Changelog](https://github.com/pytest-dev/pytest-django/blob/main/docs/changelog.rst) - [Commits](https://github.com/pytest-dev/pytest-django/compare/v4.12.0...v4.14.0) Updates `pytest-rerunfailures` from 16.4 to 16.6 - [Changelog](https://github.com/pytest-dev/pytest-rerunfailures/blob/master/CHANGES.rst) - [Commits](https://github.com/pytest-dev/pytest-rerunfailures/compare/16.4...16.6) Updates `time-machine` from 3.2.0 to 3.5.0 - [Changelog](https://github.com/adamchainz/time-machine/blob/main/docs/changelog.rst) - [Commits](https://github.com/adamchainz/time-machine/compare/3.2.0...3.5.0) Updates `types-pygments` from 2.20.0.20260408 to 2.21.0.20260819 - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: django-guardian dependency-version: 3.4.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: django-treenode dependency-version: 0.25.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: drf-spectacular-sidecar dependency-version: 2026.8.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: faker dependency-version: 40.37.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: imap-tools dependency-version: 1.15.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: ocrmypdf dependency-version: 17.11.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: prek dependency-version: 0.5.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: pytest-django dependency-version: 4.14.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: pytest-rerunfailures dependency-version: '16.6' dependency-type: direct:development update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: time-machine dependency-version: 3.5.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: utilities-minor - dependency-name: types-pygments dependency-version: 2.21.0.20260819 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: utilities-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Handles pytest-django * Formatting --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Trenton Holmes <797416+stumpylog@users.noreply.github.com> |
||
|
Trenton H
|
5202dc0748 | Fix: Clear ContentType/guardian caches at import and test cases (#12758) | ||
 Trenton HandClaude Sonnet 4.6
|
8e67828bd7 |
Feature: Redesign the task system (#12584)
* feat(tasks): replace PaperlessTask model with structured redesign Drop the old string-based PaperlessTask table and recreate it with Status/TaskType/TriggerSource enums, JSONField result storage, and duration tracking fields. Update all call sites to use the new API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): rewrite signal handlers to track all task types Replace the old consume_file-only handler with a full rewrite that tracks 6 task types (consume_file, train_classifier, sanity_check, index_optimize, llm_index, mail_fetch) with proper trigger source detection, input data extraction, legacy result string parsing, duration/wait time recording, and structured error capture on failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): add traceback and revoked state coverage to signal tests * refactor(tasks): remove manual PaperlessTask creation and scheduled/auto params All task records are now created exclusively via Celery signals (Task 2). Removed PaperlessTask creation/update from train_classifier, sanity_check, llmindex_index, and check_sanity. Removed scheduled= and auto= parameters from all 7 call sites. Updated apply_async callers to use trigger_source headers instead. Exceptions now propagate naturally from task functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): auto-inject trigger_source=scheduled header for all beat tasks Inject `headers: {"trigger_source": "scheduled"}` into every Celery beat schedule entry so signal handlers can identify scheduler-originated tasks without per-task instrumentation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): update serializer, filter, and viewset with v9 backwards compat - Replace TasksViewSerializer/RunTaskViewSerializer with TaskSerializerV10 (new field names), TaskSerializerV9 (v9 compat), TaskSummarySerializer, and RunTaskSerializer - Add AcknowledgeTasksViewSerializer unchanged (kept existing validation) - Expand PaperlessTaskFilterSet with MultipleChoiceFilter for task_type, trigger_source, status; add is_complete, date_created_after/before filters - Replace TasksViewSet.get_serializer_class() to branch on request.version - Add get_queryset() v9 compat for task_name/type query params - Add acknowledge_all, summary, active actions to TasksViewSet - Rewrite run action to use apply_async with trigger_source header - Add timedelta import to views.py; add MultipleChoiceFilter/DateTimeFilter to filters.py imports Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): add read_only_fields to TaskSerializerV9, enforce admin via permission_classes on run action * test(tasks): rewrite API task tests for redesigned model and v9 compat Replaces the old Django TestCase-based tests with pytest-style classes using PaperlessTaskFactory. Covers v10 field names, v9 backwards-compat field mapping, filtering, ordering, acknowledge, acknowledge_all, summary, active, and run endpoints. Also adds PaperlessTaskFactory to factories.py and fixes a redundant source= kwarg in TaskSerializerV10.related_document_ids. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): fix two spec gaps in task API test suite Move test_list_is_owner_aware to TestGetTasksV10 (it tests GET /api/tasks/, not acknowledge). Add test_related_document_ids_includes_duplicate_of to cover the duplicate_of path in the related_document_ids property. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): address code quality review findings Remove trivial field-existence tests per project conventions. Fix potentially flaky ordering test to use explicit date_created values. Add is_complete=false filter test, v9 type filter input direction test, and tighten TestActive second test to target REVOKED specifically. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): update TaskAdmin for redesigned model Add date_created, duration_seconds to list_display; add trigger_source to list_filter; add input_data, duration_seconds, wait_time_seconds to readonly_fields. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): update Angular types and service for task redesign Replace PaperlessTaskName/PaperlessTaskType/PaperlessTaskStatus enums with new PaperlessTaskType, PaperlessTaskTriggerSource, PaperlessTaskStatus enums. Update PaperlessTask interface to new field names (task_type, trigger_source, input_data, result_message, related_document_ids). Update TasksService to filter by task_type instead of task_name. Update tasks component and system-status-dialog to use new field names. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(tasks): remove django-celery-results PaperlessTask now tracks all task results via Celery signals. The django-celery-results DB backend was write-only -- nothing reads from it. Drop the package and add a migration to clean up the orphaned tables. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: fix remaining tests broken by task system redesign Update all tests that created PaperlessTask objects with old field names to use PaperlessTaskFactory and new field names (task_type, trigger_source, status, result_message). Use apply_async instead of delay where mocked. Drop TestCheckSanityTaskRecording — tests PaperlessTask creation that was intentionally removed from check_sanity(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): improve test_api_tasks.py structure and add api marker - Move admin_client, v9_client, user_client fixtures to conftest.py so they can be reused by other API tests; all three now build on the rest_api_client fixture instead of creating APIClient() directly - Move regular_user fixture to conftest.py (was already done, now also used by the new client fixtures) - Add docstrings to every test method describing the behaviour under test - Move timedelta/timezone imports to module level - Register 'api' pytest marker in pyproject.toml and apply pytestmark to the entire file so all 40 tests are selectable via -m api Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(tasks): simplify task tracking code after redesign - Extract COMPLETE_STATUSES as a class constant on PaperlessTask, eliminating the repeated status tuple across models.py, views.py (3×), and filters.py - Extract _CELERY_STATE_TO_STATUS as a module-level constant instead of rebuilding the dict on every task_postrun - Extract _V9_TYPE_TO_TRIGGER_SOURCE and _RUNNABLE_TASKS as class constants on TasksViewSet instead of rebuilding on every request - Extract _TRIGGER_SOURCE_TO_V9_TYPE as a class constant on TaskSerializerV9 instead of rebuilding per serialized object - Extract _get_consume_args helper to deduplicate identical arg extraction logic in _extract_input_data, _determine_trigger_source, and _extract_owner_id - Move inline imports (re, traceback) and Avg to module level - Fix _DOCUMENT_SOURCE_TO_TRIGGER type annotation key type to DocumentSource instead of Any - Remove redundant truthiness checks in SystemStatusView branches already guarded by an is-None check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(tasks): add docstrings and rename _parse_legacy_result - Add docstrings to _extract_input_data, _determine_trigger_source, _extract_owner_id explaining what each helper does and why - Rename _parse_legacy_result -> _parse_consume_result: the function parses current consume_file string outputs (consumer.py returns "New document id N created" and "It is a duplicate of X (#N)"), not legacy data; the old name was misleading Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): extend and harden the task system redesign - TaskType: add EMPTY_TRASH, CHECK_WORKFLOWS, CLEANUP_SHARE_LINKS; remove INDEX_REBUILD (no backing task — beat schedule uses index_optimize) - TRACKED_TASKS: wire up all nine task types including the three new ones and llmindex_index / process_mail_accounts - Add task_revoked_handler so cancelled/expired tasks are marked REVOKED - Fix double-write: task_postrun_handler no longer overwrites result_data when status is already FAILURE (task_failure_handler owns that write) - v9 serialiser: map EMAIL_CONSUME and FOLDER_CONSUME to AUTO_TASK - views: scope task list to owner for regular users, admins see all; validate ?days= query param and return 400 on bad input - tests: add test_list_admin_sees_all_tasks; rename/fix test_parses_duplicate_string (duplicates produce SUCCESS, not FAILURE); use PaperlessTaskFactory in modified tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): fix MAIL_FETCH null input_data and postrun double-query - _extract_input_data: return {} instead of {"account_ids": None} when process_mail_accounts is called without an explicit account list (the normal beat-scheduled path); add test to cover this path - task_postrun_handler: replace filter().first() + filter().update() with get() + save(update_fields=[...]) — single fetch, single write, consistent with task_prerun_handler Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): add queryset stub to satisfy drf-spectacular schema generation TasksViewSet.get_queryset() accesses request.user, which drf-spectacular cannot provide during static schema generation. Adding a class-level queryset = PaperlessTask.objects.none() gives spectacular a model to introspect without invoking get_queryset(), eliminating both warnings and the test_valid_schema failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): fill coverage gaps in task system - test_task_signals: add TestTaskRevokedHandler (marks REVOKED, ignores None request, ignores unknown id); switch existing direct PaperlessTask.objects.create calls to PaperlessTaskFactory; import pytest_mock and use MockerFixture typing on mocker params - test_api_tasks: add test_rejects_invalid_days_param to TestSummary - tasks.service.spec: add dismissAllTasks test (POST acknowledge_all + reload) - models: add pragma: no cover to __str__, is_complete, and related_document_ids (trivial delegates, covered indirectly) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Well, that was a bad push. * Fixes v9 API compatability with testing coverage * fix(tasks): restore INDEX_OPTIMIZE enum and remove no-op run button INDEX_OPTIMIZE was dropped from the TaskType enum but still referenced in _RUNNABLE_TASKS (views.py) and the frontend system-status-dialog, causing an AttributeError at import time. Restore the enum value in the model and migration so the serializer accepts it, but remove it from _RUNNABLE_TASKS since index_optimize is a Tantivy no-op. Remove the frontend "Run Task" button for index optimization accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tasks): v9 type filter now matches all equivalent trigger sources The v9 ?type= query param mapped each value to a single TriggerSource, but the serializer maps multiple sources to the same v9 type value. A task serialized as "auto_task" would not appear when filtering by ?type=auto_task if its trigger_source was email_consume or folder_consume. Same issue for "manual_task" missing web_ui and api_upload sources. Changed to trigger_source__in with the full set of sources for each v9 type value. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tasks): give task_failure_handler full ownership of FAILURE path task_postrun_handler now early-returns for FAILURE states instead of redundantly writing status and date_done. task_failure_handler now computes duration_seconds and wait_time_seconds so failed tasks get complete timing data. This eliminates a wasted .get() + .save() round trip on every failed task and gives each handler a clean, non-overlapping responsibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tasks): resolve trigger_source header via TriggerSource enum lookup Replace two hardcoded string comparisons ("scheduled", "system") with a single TriggerSource(header_source) lookup so the enum values are the single source of truth. Any valid TriggerSource DB value passed in the header is accepted; invalid values fall through to the document-source / MANUAL logic. Update tests to pass enum values in headers rather than raw strings, and add a test for the invalid-header fallback path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): use TriggerSource enum values at all apply_async call sites Replace raw strings ("system", "manual") with PaperlessTask.TriggerSource enum values in the three callers that can import models. The settings file remains a raw string (models cannot be imported at settings load time) with a comment pointing to the enum value it must match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): parametrize repetitive test cases in task test files test_api_tasks.py: - Collapse six trigger_source->v9-type tests into one parametrized test, adding the previously untested API_UPLOAD case - Collapse three task_name mapping tests (two remaps + pass-through) into one parametrized test - Collapse two acknowledge_all status tests into one parametrized test - Collapse two run-endpoint 400 tests into one parametrized test - Update run/ assertions to use TriggerSource enum values test_task_signals.py: - Collapse three trigger_source header tests into one parametrized test - Collapse two DocumentSource->TriggerSource mapping tests into one parametrized test - Collapse two prerun ignore-invalid-id tests into one parametrized test All parametrize cases use pytest.param with descriptive ids. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Handle JSON serialization for datetime and Path. Further restrist the v9 permissions as Copilot suggests * That should fix the generated schema/browser * Use XSerializer for the schema * A few more basic cases I see no value in covering * Drops the migration related stuff too. Just in case we want it again or it confuses people * fix: annotate tasks_summary_retrieve as array of TaskSummarySerializer Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: annotate tasks_active_retrieve as array of TaskSerializerV10 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Restore task running to superuser only * Removes the acknowledge/dismiss all stuff * Aligns v10 and v9 task permissions with each other * Short blurb just to warn users about the tasks being cleared --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
aed9abe48c |
Feature: Replace Whoosh with tantivy search backend (#12471)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Antoine Mérino <3023499+Merinorus@users.noreply.github.com> |
||
|
Trenton H
|
9383471fa0 | Feature: Transition all checksums to use SHA256 (#12432) | ||
|
Trenton H
|
43406f44f2 | Feature: Improve the retagger output using rich (#12194) | ||
|
Trenton H
|
e58a35d40c | Feature: Transition sanity check to rich and improve output (#12182) | ||
|
Trenton H
|
ec34197b59 | Chore: Switch remote version check to HTTPx (#9232) | ||
|
Trenton H
|
3cf73a77ac | Chore: Initial conversion to pytest fixtures (#7110) |