Feature: Redesign the task system (#12584)

* feat(tasks): replace PaperlessTask model with structured redesign

Drop the old string-based PaperlessTask table and recreate it with
Status/TaskType/TriggerSource enums, JSONField result storage, and
duration tracking fields. Update all call sites to use the new API.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tasks): rewrite signal handlers to track all task types

Replace the old consume_file-only handler with a full rewrite that tracks
6 task types (consume_file, train_classifier, sanity_check, index_optimize,
llm_index, mail_fetch) with proper trigger source detection, input data
extraction, legacy result string parsing, duration/wait time recording,
and structured error capture on failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tasks): add traceback and revoked state coverage to signal tests

* refactor(tasks): remove manual PaperlessTask creation and scheduled/auto params

All task records are now created exclusively via Celery signals (Task 2).
Removed PaperlessTask creation/update from train_classifier, sanity_check,
llmindex_index, and check_sanity. Removed scheduled= and auto= parameters
from all 7 call sites. Updated apply_async callers to use trigger_source
headers instead. Exceptions now propagate naturally from task functions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tasks): auto-inject trigger_source=scheduled header for all beat tasks

Inject `headers: {"trigger_source": "scheduled"}` into every Celery beat
schedule entry so signal handlers can identify scheduler-originated tasks
without per-task instrumentation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tasks): update serializer, filter, and viewset with v9 backwards compat

- Replace TasksViewSerializer/RunTaskViewSerializer with TaskSerializerV10
  (new field names), TaskSerializerV9 (v9 compat), TaskSummarySerializer,
  and RunTaskSerializer
- Add AcknowledgeTasksViewSerializer unchanged (kept existing validation)
- Expand PaperlessTaskFilterSet with MultipleChoiceFilter for task_type,
  trigger_source, status; add is_complete, date_created_after/before filters
- Replace TasksViewSet.get_serializer_class() to branch on request.version
- Add get_queryset() v9 compat for task_name/type query params
- Add acknowledge_all, summary, active actions to TasksViewSet
- Rewrite run action to use apply_async with trigger_source header
- Add timedelta import to views.py; add MultipleChoiceFilter/DateTimeFilter
  to filters.py imports

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tasks): add read_only_fields to TaskSerializerV9, enforce admin via permission_classes on run action

* test(tasks): rewrite API task tests for redesigned model and v9 compat

Replaces the old Django TestCase-based tests with pytest-style classes using
PaperlessTaskFactory. Covers v10 field names, v9 backwards-compat field
mapping, filtering, ordering, acknowledge, acknowledge_all, summary, active,
and run endpoints. Also adds PaperlessTaskFactory to factories.py and fixes
a redundant source= kwarg in TaskSerializerV10.related_document_ids.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tasks): fix two spec gaps in task API test suite

Move test_list_is_owner_aware to TestGetTasksV10 (it tests GET /api/tasks/,
not acknowledge). Add test_related_document_ids_includes_duplicate_of to
cover the duplicate_of path in the related_document_ids property.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tasks): address code quality review findings

Remove trivial field-existence tests per project conventions. Fix
potentially flaky ordering test to use explicit date_created values.
Add is_complete=false filter test, v9 type filter input direction test,
and tighten TestActive second test to target REVOKED specifically.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tasks): update TaskAdmin for redesigned model

Add date_created, duration_seconds to list_display; add trigger_source
to list_filter; add input_data, duration_seconds, wait_time_seconds to
readonly_fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tasks): update Angular types and service for task redesign

Replace PaperlessTaskName/PaperlessTaskType/PaperlessTaskStatus enums
with new PaperlessTaskType, PaperlessTaskTriggerSource, PaperlessTaskStatus
enums. Update PaperlessTask interface to new field names (task_type,
trigger_source, input_data, result_message, related_document_ids).
Update TasksService to filter by task_type instead of task_name.
Update tasks component and system-status-dialog to use new field names.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(tasks): remove django-celery-results

PaperlessTask now tracks all task results via Celery signals. The
django-celery-results DB backend was write-only -- nothing reads
from it. Drop the package and add a migration to clean up the
orphaned tables.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: fix remaining tests broken by task system redesign

Update all tests that created PaperlessTask objects with old field names
to use PaperlessTaskFactory and new field names (task_type, trigger_source,
status, result_message). Use apply_async instead of delay where mocked.
Drop TestCheckSanityTaskRecording — tests PaperlessTask creation that was
intentionally removed from check_sanity().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tasks): improve test_api_tasks.py structure and add api marker

- Move admin_client, v9_client, user_client fixtures to conftest.py so
  they can be reused by other API tests; all three now build on the
  rest_api_client fixture instead of creating APIClient() directly
- Move regular_user fixture to conftest.py (was already done, now also
  used by the new client fixtures)
- Add docstrings to every test method describing the behaviour under test
- Move timedelta/timezone imports to module level
- Register 'api' pytest marker in pyproject.toml and apply pytestmark to
  the entire file so all 40 tests are selectable via -m api

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(tasks): simplify task tracking code after redesign

- Extract COMPLETE_STATUSES as a class constant on PaperlessTask,
  eliminating the repeated status tuple across models.py, views.py (3×),
  and filters.py
- Extract _CELERY_STATE_TO_STATUS as a module-level constant instead of
  rebuilding the dict on every task_postrun
- Extract _V9_TYPE_TO_TRIGGER_SOURCE and _RUNNABLE_TASKS as class
  constants on TasksViewSet instead of rebuilding on every request
- Extract _TRIGGER_SOURCE_TO_V9_TYPE as a class constant on
  TaskSerializerV9 instead of rebuilding per serialized object
- Extract _get_consume_args helper to deduplicate identical arg
  extraction logic in _extract_input_data, _determine_trigger_source,
  and _extract_owner_id
- Move inline imports (re, traceback) and Avg to module level
- Fix _DOCUMENT_SOURCE_TO_TRIGGER type annotation key type to
  DocumentSource instead of Any
- Remove redundant truthiness checks in SystemStatusView branches
  already guarded by an is-None check

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(tasks): add docstrings and rename _parse_legacy_result

- Add docstrings to _extract_input_data, _determine_trigger_source,
  _extract_owner_id explaining what each helper does and why
- Rename _parse_legacy_result -> _parse_consume_result: the function
  parses current consume_file string outputs (consumer.py returns
  "New document id N created" and "It is a duplicate of X (#N)"),
  not legacy data; the old name was misleading

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tasks): extend and harden the task system redesign

- TaskType: add EMPTY_TRASH, CHECK_WORKFLOWS, CLEANUP_SHARE_LINKS;
  remove INDEX_REBUILD (no backing task — beat schedule uses index_optimize)
- TRACKED_TASKS: wire up all nine task types including the three new ones
  and llmindex_index / process_mail_accounts
- Add task_revoked_handler so cancelled/expired tasks are marked REVOKED
- Fix double-write: task_postrun_handler no longer overwrites result_data
  when status is already FAILURE (task_failure_handler owns that write)
- v9 serialiser: map EMAIL_CONSUME and FOLDER_CONSUME to AUTO_TASK
- views: scope task list to owner for regular users, admins see all;
  validate ?days= query param and return 400 on bad input
- tests: add test_list_admin_sees_all_tasks; rename/fix
  test_parses_duplicate_string (duplicates produce SUCCESS, not FAILURE);
  use PaperlessTaskFactory in modified tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tasks): fix MAIL_FETCH null input_data and postrun double-query

- _extract_input_data: return {} instead of {"account_ids": None} when
  process_mail_accounts is called without an explicit account list (the
  normal beat-scheduled path); add test to cover this path
- task_postrun_handler: replace filter().first() + filter().update() with
  get() + save(update_fields=[...]) — single fetch, single write,
  consistent with task_prerun_handler

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tasks): add queryset stub to satisfy drf-spectacular schema generation

TasksViewSet.get_queryset() accesses request.user, which drf-spectacular
cannot provide during static schema generation.  Adding a class-level
queryset = PaperlessTask.objects.none() gives spectacular a model to
introspect without invoking get_queryset(), eliminating both warnings
and the test_valid_schema failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tasks): fill coverage gaps in task system

- test_task_signals: add TestTaskRevokedHandler (marks REVOKED, ignores
  None request, ignores unknown id); switch existing direct
  PaperlessTask.objects.create calls to PaperlessTaskFactory; import
  pytest_mock and use MockerFixture typing on mocker params
- test_api_tasks: add test_rejects_invalid_days_param to TestSummary
- tasks.service.spec: add dismissAllTasks test (POST acknowledge_all +
  reload)
- models: add pragma: no cover to __str__, is_complete, and
  related_document_ids (trivial delegates, covered indirectly)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Well, that was a bad push.

* Fixes v9 API compatability with testing coverage

* fix(tasks): restore INDEX_OPTIMIZE enum and remove no-op run button

INDEX_OPTIMIZE was dropped from the TaskType enum but still referenced
in _RUNNABLE_TASKS (views.py) and the frontend system-status-dialog,
causing an AttributeError at import time. Restore the enum value in the
model and migration so the serializer accepts it, but remove it from
_RUNNABLE_TASKS since index_optimize is a Tantivy no-op. Remove the
frontend "Run Task" button for index optimization accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(tasks): v9 type filter now matches all equivalent trigger sources

The v9 ?type= query param mapped each value to a single TriggerSource,
but the serializer maps multiple sources to the same v9 type value.
A task serialized as "auto_task" would not appear when filtering by
?type=auto_task if its trigger_source was email_consume or
folder_consume. Same issue for "manual_task" missing web_ui and
api_upload sources. Changed to trigger_source__in with the full set
of sources for each v9 type value.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(tasks): give task_failure_handler full ownership of FAILURE path

task_postrun_handler now early-returns for FAILURE states instead of
redundantly writing status and date_done. task_failure_handler now
computes duration_seconds and wait_time_seconds so failed tasks get
complete timing data. This eliminates a wasted .get() + .save() round
trip on every failed task and gives each handler a clean, non-overlapping
responsibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(tasks): resolve trigger_source header via TriggerSource enum lookup

Replace two hardcoded string comparisons ("scheduled", "system") with a
single TriggerSource(header_source) lookup so the enum values are the
single source of truth. Any valid TriggerSource DB value passed in the
header is accepted; invalid values fall through to the document-source /
MANUAL logic. Update tests to pass enum values in headers rather than raw
strings, and add a test for the invalid-header fallback path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tasks): use TriggerSource enum values at all apply_async call sites

Replace raw strings ("system", "manual") with PaperlessTask.TriggerSource
enum values in the three callers that can import models. The settings
file remains a raw string (models cannot be imported at settings load
time) with a comment pointing to the enum value it must match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tasks): parametrize repetitive test cases in task test files

test_api_tasks.py:
- Collapse six trigger_source->v9-type tests into one parametrized test,
  adding the previously untested API_UPLOAD case
- Collapse three task_name mapping tests (two remaps + pass-through)
  into one parametrized test
- Collapse two acknowledge_all status tests into one parametrized test
- Collapse two run-endpoint 400 tests into one parametrized test
- Update run/ assertions to use TriggerSource enum values

test_task_signals.py:
- Collapse three trigger_source header tests into one parametrized test
- Collapse two DocumentSource->TriggerSource mapping tests into one
  parametrized test
- Collapse two prerun ignore-invalid-id tests into one parametrized test

All parametrize cases use pytest.param with descriptive ids.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Handle JSON serialization for datetime and Path.  Further restrist the v9 permissions as Copilot suggests

* That should fix the generated schema/browser

* Use XSerializer for the schema

* A few more basic cases I see no value in covering

* Drops the migration related stuff too.  Just in case we want it again or it confuses people

* fix: annotate tasks_summary_retrieve as array of TaskSummarySerializer

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: annotate tasks_active_retrieve as array of TaskSerializerV10

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Restore task running to superuser only

* Removes the acknowledge/dismiss all stuff

* Aligns v10 and v9 task permissions with each other

* Short blurb just to warn users about the tasks being cleared

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

src/documents/models.py

@@ -3,7 +3,6 @@ from pathlib import Path
 from typing import Final
 
 import pathvalidate
-from celery import states
 from django.conf import settings
 from django.contrib.auth.models import Group
 from django.contrib.auth.models import User
@@ -663,97 +662,174 @@ class UiSettings(models.Model):
 
 
 class PaperlessTask(ModelWithOwner):
-    ALL_STATES = sorted(states.ALL_STATES)
-    TASK_STATE_CHOICES = sorted(zip(ALL_STATES, ALL_STATES))
+    """
+    Tracks background task execution for user visibility and debugging.
+
+    State transitions:
+        PENDING -> STARTED -> SUCCESS
+        PENDING -> STARTED -> FAILURE
+        PENDING -> REVOKED (if cancelled before starting)
+    """
+
+    class Status(models.TextChoices):
+        PENDING = "pending", _("Pending")
+        STARTED = "started", _("Started")
+        SUCCESS = "success", _("Success")
+        FAILURE = "failure", _("Failure")
+        REVOKED = "revoked", _("Revoked")
 
     class TaskType(models.TextChoices):
-        AUTO = ("auto_task", _("Auto Task"))
-        SCHEDULED_TASK = ("scheduled_task", _("Scheduled Task"))
-        MANUAL_TASK = ("manual_task", _("Manual Task"))
+        CONSUME_FILE = "consume_file", _("Consume File")
+        TRAIN_CLASSIFIER = "train_classifier", _("Train Classifier")
+        SANITY_CHECK = "sanity_check", _("Sanity Check")
+        INDEX_OPTIMIZE = "index_optimize", _("Index Optimize")
+        MAIL_FETCH = "mail_fetch", _("Mail Fetch")
+        LLM_INDEX = "llm_index", _("LLM Index")
+        EMPTY_TRASH = "empty_trash", _("Empty Trash")
+        CHECK_WORKFLOWS = "check_workflows", _("Check Workflows")
+        BULK_UPDATE = "bulk_update", _("Bulk Update")
+        REPROCESS_DOCUMENT = "reprocess_document", _("Reprocess Document")
+        BUILD_SHARE_LINK = "build_share_link", _("Build Share Link")
+        BULK_DELETE = "bulk_delete", _("Bulk Delete")
 
-    class TaskName(models.TextChoices):
-        CONSUME_FILE = ("consume_file", _("Consume File"))
-        TRAIN_CLASSIFIER = ("train_classifier", _("Train Classifier"))
-        CHECK_SANITY = ("check_sanity", _("Check Sanity"))
-        INDEX_OPTIMIZE = ("index_optimize", _("Index Optimize"))
-        LLMINDEX_UPDATE = ("llmindex_update", _("LLM Index Update"))
+    COMPLETE_STATUSES = (
+        Status.SUCCESS,
+        Status.FAILURE,
+        Status.REVOKED,
+    )
 
+    class TriggerSource(models.TextChoices):
+        SCHEDULED = "scheduled", _("Scheduled")  # Celery beat
+        WEB_UI = "web_ui", _("Web UI")  # Document uploaded via web
+        API_UPLOAD = "api_upload", _("API Upload")  # Document uploaded via API
+        FOLDER_CONSUME = "folder_consume", _("Folder Consume")  # Consume folder
+        EMAIL_CONSUME = "email_consume", _("Email Consume")  # Email attachment
+        SYSTEM = "system", _("System")  # Auto-triggered (self-heal, config side-effect)
+        MANUAL = "manual", _("Manual")  # User explicitly ran via /api/tasks/run/
+
+    # Identification
     task_id = models.CharField(
-        max_length=255,
+        max_length=72,
         unique=True,
         verbose_name=_("Task ID"),
-        help_text=_("Celery ID for the Task that was run"),
+        help_text=_("Celery task ID"),
     )
 
-    acknowledged = models.BooleanField(
-        default=False,
-        verbose_name=_("Acknowledged"),
-        help_text=_("If the task is acknowledged via the frontend or API"),
+    task_type = models.CharField(
+        max_length=50,
+        choices=TaskType.choices,
+        verbose_name=_("Task Type"),
+        help_text=_("The kind of work being performed"),
+        db_index=True,
     )
 
-    task_file_name = models.CharField(
-        null=True,
-        max_length=255,
-        verbose_name=_("Task Filename"),
-        help_text=_("Name of the file which the Task was run for"),
-    )
-
-    task_name = models.CharField(
-        null=True,
-        max_length=255,
-        choices=TaskName.choices,
-        verbose_name=_("Task Name"),
-        help_text=_("Name of the task that was run"),
+    trigger_source = models.CharField(
+        max_length=50,
+        choices=TriggerSource.choices,
+        verbose_name=_("Trigger Source"),
+        help_text=_("What initiated this task"),
+        db_index=True,
     )
 
+    # State tracking
     status = models.CharField(
         max_length=30,
-        default=states.PENDING,
-        choices=TASK_STATE_CHOICES,
-        verbose_name=_("Task State"),
-        help_text=_("Current state of the task being run"),
+        choices=Status.choices,
+        default=Status.PENDING,
+        verbose_name=_("Status"),
+        db_index=True,
     )
 
+    # Timestamps
     date_created = models.DateTimeField(
-        null=True,
         default=timezone.now,
-        verbose_name=_("Created DateTime"),
-        help_text=_("Datetime field when the task result was created in UTC"),
+        verbose_name=_("Created"),
+        db_index=True,
     )
 
     date_started = models.DateTimeField(
         null=True,
-        default=None,
-        verbose_name=_("Started DateTime"),
-        help_text=_("Datetime field when the task was started in UTC"),
+        blank=True,
+        verbose_name=_("Started"),
     )
 
     date_done = models.DateTimeField(
         null=True,
-        default=None,
-        verbose_name=_("Completed DateTime"),
-        help_text=_("Datetime field when the task was completed in UTC"),
+        blank=True,
+        verbose_name=_("Completed"),
+        db_index=True,
     )
 
-    result = models.TextField(
+    # Duration fields -- populated by task_postrun signal handler
+    duration_seconds = models.FloatField(
         null=True,
-        default=None,
+        blank=True,
+        verbose_name=_("Duration (seconds)"),
+        help_text=_("Elapsed time from start to completion"),
+    )
+
+    wait_time_seconds = models.FloatField(
+        null=True,
+        blank=True,
+        verbose_name=_("Wait Time (seconds)"),
+        help_text=_("Time from task creation to worker pickup"),
+    )
+
+    # Input/Output data
+    input_data = models.JSONField(
+        default=dict,
+        blank=True,
+        verbose_name=_("Input Data"),
+        help_text=_("Structured input parameters for the task"),
+    )
+
+    result_data = models.JSONField(
+        null=True,
+        blank=True,
         verbose_name=_("Result Data"),
-        help_text=_(
-            "The data returned by the task",
-        ),
+        help_text=_("Structured result data from task execution"),
     )
 
-    type = models.CharField(
-        max_length=30,
-        choices=TaskType.choices,
-        default=TaskType.AUTO,
-        verbose_name=_("Task Type"),
-        help_text=_("The type of task that was run"),
+    result_message = models.TextField(
+        null=True,
+        blank=True,
+        verbose_name=_("Result Message"),
+        help_text=_("Human-readable result message"),
     )
 
-    def __str__(self) -> str:
-        return f"Task {self.task_id}"
+    # Acknowledgment
+    acknowledged = models.BooleanField(
+        default=False,
+        verbose_name=_("Acknowledged"),
+        db_index=True,
+    )
+
+    class Meta:
+        verbose_name = _("Task")
+        verbose_name_plural = _("Tasks")
+        ordering = ["-date_created"]
+        indexes = [
+            models.Index(fields=["status", "date_created"]),
+            models.Index(fields=["task_type", "status"]),
+            models.Index(fields=["owner", "acknowledged", "date_created"]),
+        ]
+
+    def __str__(self) -> str:  # pragma: no cover
+        return f"{self.get_task_type_display()} [{self.task_id[:8]}]"
+
+    @property
+    def is_complete(self) -> bool:  # pragma: no cover
+        return self.status in self.COMPLETE_STATUSES
+
+    @property
+    def related_document_ids(self) -> list[int]:  # pragma: no cover
+        if not self.result_data:
+            return []
+        if doc_id := self.result_data.get("document_id"):
+            return [doc_id]
+        if dup_id := self.result_data.get("duplicate_of"):
+            return [dup_id]
+        return []
 
 
 class Note(SoftDeleteModel):