Commit Graph
2268 Commits
Author SHA1 Message Date
stumpylogandClaude Opus 4.6 9521e89680 fix(tasks): restore INDEX_OPTIMIZE enum and remove no-op run button
INDEX_OPTIMIZE was dropped from the TaskType enum but still referenced
in _RUNNABLE_TASKS (views.py) and the frontend system-status-dialog,
causing an AttributeError at import time. Restore the enum value in the
model and migration so the serializer accepts it, but remove it from
_RUNNABLE_TASKS since index_optimize is a Tantivy no-op. Remove the
frontend "Run Task" button for index optimization accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-16 09:53:32 -07:00
stumpylog 14931c02e6 Fixes v9 API compatability with testing coverage 2026-04-16 09:14:00 -07:00
stumpylog c213e854dc Well, that was a bad push. 2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 1012dc887b test(tasks): fill coverage gaps in task system
- test_task_signals: add TestTaskRevokedHandler (marks REVOKED, ignores
  None request, ignores unknown id); switch existing direct
  PaperlessTask.objects.create calls to PaperlessTaskFactory; import
  pytest_mock and use MockerFixture typing on mocker params
- test_api_tasks: add test_rejects_invalid_days_param to TestSummary
- tasks.service.spec: add dismissAllTasks test (POST acknowledge_all +
  reload)
- models: add pragma: no cover to __str__, is_complete, and
  related_document_ids (trivial delegates, covered indirectly)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 cf341c19fe fix(tasks): add queryset stub to satisfy drf-spectacular schema generation
TasksViewSet.get_queryset() accesses request.user, which drf-spectacular
cannot provide during static schema generation.  Adding a class-level
queryset = PaperlessTask.objects.none() gives spectacular a model to
introspect without invoking get_queryset(), eliminating both warnings
and the test_valid_schema failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 8bba782ad9 fix(tasks): fix MAIL_FETCH null input_data and postrun double-query
- _extract_input_data: return {} instead of {"account_ids": None} when
  process_mail_accounts is called without an explicit account list (the
  normal beat-scheduled path); add test to cover this path
- task_postrun_handler: replace filter().first() + filter().update() with
  get() + save(update_fields=[...]) — single fetch, single write,
  consistent with task_prerun_handler

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 e86c37f717 feat(tasks): extend and harden the task system redesign
- TaskType: add EMPTY_TRASH, CHECK_WORKFLOWS, CLEANUP_SHARE_LINKS;
  remove INDEX_REBUILD (no backing task — beat schedule uses index_optimize)
- TRACKED_TASKS: wire up all nine task types including the three new ones
  and llmindex_index / process_mail_accounts
- Add task_revoked_handler so cancelled/expired tasks are marked REVOKED
- Fix double-write: task_postrun_handler no longer overwrites result_data
  when status is already FAILURE (task_failure_handler owns that write)
- v9 serialiser: map EMAIL_CONSUME and FOLDER_CONSUME to AUTO_TASK
- views: scope task list to owner for regular users, admins see all;
  validate ?days= query param and return 400 on bad input
- tests: add test_list_admin_sees_all_tasks; rename/fix
  test_parses_duplicate_string (duplicates produce SUCCESS, not FAILURE);
  use PaperlessTaskFactory in modified tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 979d8a67f0 refactor(tasks): add docstrings and rename _parse_legacy_result
- Add docstrings to _extract_input_data, _determine_trigger_source,
  _extract_owner_id explaining what each helper does and why
- Rename _parse_legacy_result -> _parse_consume_result: the function
  parses current consume_file string outputs (consumer.py returns
  "New document id N created" and "It is a duplicate of X (#N)"),
  not legacy data; the old name was misleading

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 0bc9227d6f refactor(tasks): simplify task tracking code after redesign
- Extract COMPLETE_STATUSES as a class constant on PaperlessTask,
  eliminating the repeated status tuple across models.py, views.py (3×),
  and filters.py
- Extract _CELERY_STATE_TO_STATUS as a module-level constant instead of
  rebuilding the dict on every task_postrun
- Extract _V9_TYPE_TO_TRIGGER_SOURCE and _RUNNABLE_TASKS as class
  constants on TasksViewSet instead of rebuilding on every request
- Extract _TRIGGER_SOURCE_TO_V9_TYPE as a class constant on
  TaskSerializerV9 instead of rebuilding per serialized object
- Extract _get_consume_args helper to deduplicate identical arg
  extraction logic in _extract_input_data, _determine_trigger_source,
  and _extract_owner_id
- Move inline imports (re, traceback) and Avg to module level
- Fix _DOCUMENT_SOURCE_TO_TRIGGER type annotation key type to
  DocumentSource instead of Any
- Remove redundant truthiness checks in SystemStatusView branches
  already guarded by an is-None check

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 e37ecc2b43 test(tasks): improve test_api_tasks.py structure and add api marker
- Move admin_client, v9_client, user_client fixtures to conftest.py so
  they can be reused by other API tests; all three now build on the
  rest_api_client fixture instead of creating APIClient() directly
- Move regular_user fixture to conftest.py (was already done, now also
  used by the new client fixtures)
- Add docstrings to every test method describing the behaviour under test
- Move timedelta/timezone imports to module level
- Register 'api' pytest marker in pyproject.toml and apply pytestmark to
  the entire file so all 40 tests are selectable via -m api

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 a6c93b7703 test: fix remaining tests broken by task system redesign
Update all tests that created PaperlessTask objects with old field names
to use PaperlessTaskFactory and new field names (task_type, trigger_source,
status, result_message). Use apply_async instead of delay where mocked.
Drop TestCheckSanityTaskRecording — tests PaperlessTask creation that was
intentionally removed from check_sanity().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 61bfe25e17 chore(tasks): remove django-celery-results
PaperlessTask now tracks all task results via Celery signals. The
django-celery-results DB backend was write-only -- nothing reads
from it. Drop the package and add a migration to clean up the
orphaned tables.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 587c2fdfdf feat(tasks): update TaskAdmin for redesigned model
Add date_created, duration_seconds to list_display; add trigger_source
to list_filter; add input_data, duration_seconds, wait_time_seconds to
readonly_fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 e82fed504d test(tasks): address code quality review findings
Remove trivial field-existence tests per project conventions. Fix
potentially flaky ordering test to use explicit date_created values.
Add is_complete=false filter test, v9 type filter input direction test,
and tighten TestActive second test to target REVOKED specifically.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 39534d2227 test(tasks): fix two spec gaps in task API test suite
Move test_list_is_owner_aware to TestGetTasksV10 (it tests GET /api/tasks/,
not acknowledge). Add test_related_document_ids_includes_duplicate_of to
cover the duplicate_of path in the related_document_ids property.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 bc63ca5806 test(tasks): rewrite API task tests for redesigned model and v9 compat
Replaces the old Django TestCase-based tests with pytest-style classes using
PaperlessTaskFactory. Covers v10 field names, v9 backwards-compat field
mapping, filtering, ordering, acknowledge, acknowledge_all, summary, active,
and run endpoints. Also adds PaperlessTaskFactory to factories.py and fixes
a redundant source= kwarg in TaskSerializerV10.related_document_ids.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylog 1ae146a38b fix(tasks): add read_only_fields to TaskSerializerV9, enforce admin via permission_classes on run action 2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 39c7a04dd2 feat(tasks): update serializer, filter, and viewset with v9 backwards compat
- Replace TasksViewSerializer/RunTaskViewSerializer with TaskSerializerV10
  (new field names), TaskSerializerV9 (v9 compat), TaskSummarySerializer,
  and RunTaskSerializer
- Add AcknowledgeTasksViewSerializer unchanged (kept existing validation)
- Expand PaperlessTaskFilterSet with MultipleChoiceFilter for task_type,
  trigger_source, status; add is_complete, date_created_after/before filters
- Replace TasksViewSet.get_serializer_class() to branch on request.version
- Add get_queryset() v9 compat for task_name/type query params
- Add acknowledge_all, summary, active actions to TasksViewSet
- Rewrite run action to use apply_async with trigger_source header
- Add timedelta import to views.py; add MultipleChoiceFilter/DateTimeFilter
  to filters.py imports

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 b241250741 refactor(tasks): remove manual PaperlessTask creation and scheduled/auto params
All task records are now created exclusively via Celery signals (Task 2).
Removed PaperlessTask creation/update from train_classifier, sanity_check,
llmindex_index, and check_sanity. Removed scheduled= and auto= parameters
from all 7 call sites. Updated apply_async callers to use trigger_source
headers instead. Exceptions now propagate naturally from task functions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylog efeb5b447c test(tasks): add traceback and revoked state coverage to signal tests 2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 ad47e96df1 feat(tasks): rewrite signal handlers to track all task types
Replace the old consume_file-only handler with a full rewrite that tracks
6 task types (consume_file, train_classifier, sanity_check, index_optimize,
llm_index, mail_fetch) with proper trigger source detection, input data
extraction, legacy result string parsing, duration/wait time recording,
and structured error capture on failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
stumpylogandClaude Sonnet 4.6 ef4e3d31ef feat(tasks): replace PaperlessTask model with structured redesign
Drop the old string-based PaperlessTask table and recreate it with
Status/TaskType/TriggerSource enums, JSONField result storage, and
duration tracking fields. Update all call sites to use the new API.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 18:34:34 -07:00
3ffbb8862c Feature: paginate search highlights and remove 10k document search limit (#12518)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-04-15 23:20:31 +00:00
2fd1a1cf3a Feature: Document fuzzy match improvements (#12579)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 20:59:43 +00:00
Gaëtan GOUZIandGitHub 02008e40d8 fix: Return HTTP 400 instead of HTTP 500 on DELETE /api/documents/{id}/notes/ with missing or invalid note id (#12582) 2026-04-15 11:56:15 -07:00
shamoon 8f036c9521 Merge branch 'main' into dev
# Conflicts:
#	docs/usage.md
#	src/documents/signals/handlers.py
#	src/documents/tests/test_api_documents.py
#	src/documents/views.py
2026-04-14 15:11:23 -07:00
Trenton HandGitHub 17c13c1a03 Breaking: Remove the positional arguments from the pre/post consume scripts. Environment has been preferred for quite some time now (#12573) 2026-04-14 21:18:20 +00:00
3b6edcdd8e Chore: Add generic type params and update our baselines (#12566)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 14:12:59 -07:00
Trenton HandGitHub 8c1225e120 Fixes an N+1 query in matching with the version content fetching by prefetching versions (#12562) 2026-04-13 13:10:28 -07:00
Trenton HandGitHub 54d5269145 Fix: Use an iterator in the sanity checking (#12563) 2026-04-13 12:32:22 -07:00
shamoon df3b656352 Add tests 2026-04-10 12:06:28 -07:00
shamoonandGitHub 51e721733f Enhancement: validate and sanitize uploaded logos (#12551) 2026-04-10 11:50:58 -07:00
shamoonandGitHub 4629bbf83e Enhancement: add view_global_statistics and view_system_status permissions (#12530) 2026-04-08 15:39:47 +00:00
c232d443fa Breaking: Decouple OCR control from archive file control (#12448)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-04-06 15:50:21 -07:00
Trenton HandGitHub 5f5fb263c9 Fix: Don't create a new note highlight generator per note in the loop (#12512) 2026-04-03 17:34:15 -07:00
shamoonandGitHub b807b107ad Enhancement: include sharelinks + bundles in export/import (#12479) 2026-04-03 21:51:57 +00:00
Trenton HandGitHub c2f02851da Chore: Better typed status manager messages (#12509) 2026-04-03 21:18:01 +00:00
shamoonandGitHub 566afdffca Enhancement: unify text search to use tantivy (#12485) 2026-04-03 13:53:45 -07:00
Trenton HandGitHub f32ad98d8e Feature: Update consumer logging to include task ID for log correlation (#12510) 2026-04-03 13:31:40 -07:00
Trenton HandGitHub dda05a7c00 Security: Improve overall security in a few ways (#12501)
- Make sure we're always using regex with timeouts for user controlled data
- Adds rate limiting to the token endpoint (configurable)
- Signs the classifier pickle file with the SECRET_KEY and refuse to load one which doesn't verify.
- Require the user to set a secret key, instead of falling back to our old hard coded one
2026-04-02 15:30:26 -07:00
Trenton HandGitHub 376af81b9c Fix: Resolve another TC assuming an object has been created somewhere (#12503) 2026-04-02 14:58:28 -07:00
aed9abe48c Feature: Replace Whoosh with tantivy search backend (#12471)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Antoine Mérino <3023499+Merinorus@users.noreply.github.com>
2026-04-02 12:38:22 -07:00
shamoonandGitHub 2f5bcdf66e Fix: dont defer tag change application in workflows (#12478) 2026-04-02 11:54:37 -07:00
shamoonandGitHub d2328b776a Performance: support bulk edit without id lists (#12355) 2026-03-31 18:23:28 +00:00
shamoonandGitHub 245514ad10 Performance: deprecate and remove usage of all in API results (#12309) 2026-03-31 07:55:59 -07:00
shamoonandGitHub f715533770 Performance: support passing selection data with filtered document requests (#12300) 2026-03-30 16:38:52 +00:00
shamoonandGitHub 501cdd92d2 Fix: limit share link viewset actions (#12461) 2026-03-30 09:34:13 -07:00
Jan KleineandGitHub 0292edbee7 Fixhancement: include trashed documents in document exporter/importer (#12425) 2026-03-30 16:30:22 +00:00
shamoonandGitHub 129da3ade7 Tweakhancement: show file extension in StoragePath test (#12452) 2026-03-28 13:58:33 -07:00
Trenton HandGitHub 9383471fa0 Feature: Transition all checksums to use SHA256 (#12432) 2026-03-26 11:28:02 -07:00