mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-04-21 23:39:28 +00:00
* feat(tasks): replace PaperlessTask model with structured redesign Drop the old string-based PaperlessTask table and recreate it with Status/TaskType/TriggerSource enums, JSONField result storage, and duration tracking fields. Update all call sites to use the new API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): rewrite signal handlers to track all task types Replace the old consume_file-only handler with a full rewrite that tracks 6 task types (consume_file, train_classifier, sanity_check, index_optimize, llm_index, mail_fetch) with proper trigger source detection, input data extraction, legacy result string parsing, duration/wait time recording, and structured error capture on failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): add traceback and revoked state coverage to signal tests * refactor(tasks): remove manual PaperlessTask creation and scheduled/auto params All task records are now created exclusively via Celery signals (Task 2). Removed PaperlessTask creation/update from train_classifier, sanity_check, llmindex_index, and check_sanity. Removed scheduled= and auto= parameters from all 7 call sites. Updated apply_async callers to use trigger_source headers instead. Exceptions now propagate naturally from task functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): auto-inject trigger_source=scheduled header for all beat tasks Inject `headers: {"trigger_source": "scheduled"}` into every Celery beat schedule entry so signal handlers can identify scheduler-originated tasks without per-task instrumentation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): update serializer, filter, and viewset with v9 backwards compat - Replace TasksViewSerializer/RunTaskViewSerializer with TaskSerializerV10 (new field names), TaskSerializerV9 (v9 compat), TaskSummarySerializer, and RunTaskSerializer - Add AcknowledgeTasksViewSerializer unchanged (kept existing validation) - Expand PaperlessTaskFilterSet with MultipleChoiceFilter for task_type, trigger_source, status; add is_complete, date_created_after/before filters - Replace TasksViewSet.get_serializer_class() to branch on request.version - Add get_queryset() v9 compat for task_name/type query params - Add acknowledge_all, summary, active actions to TasksViewSet - Rewrite run action to use apply_async with trigger_source header - Add timedelta import to views.py; add MultipleChoiceFilter/DateTimeFilter to filters.py imports Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): add read_only_fields to TaskSerializerV9, enforce admin via permission_classes on run action * test(tasks): rewrite API task tests for redesigned model and v9 compat Replaces the old Django TestCase-based tests with pytest-style classes using PaperlessTaskFactory. Covers v10 field names, v9 backwards-compat field mapping, filtering, ordering, acknowledge, acknowledge_all, summary, active, and run endpoints. Also adds PaperlessTaskFactory to factories.py and fixes a redundant source= kwarg in TaskSerializerV10.related_document_ids. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): fix two spec gaps in task API test suite Move test_list_is_owner_aware to TestGetTasksV10 (it tests GET /api/tasks/, not acknowledge). Add test_related_document_ids_includes_duplicate_of to cover the duplicate_of path in the related_document_ids property. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): address code quality review findings Remove trivial field-existence tests per project conventions. Fix potentially flaky ordering test to use explicit date_created values. Add is_complete=false filter test, v9 type filter input direction test, and tighten TestActive second test to target REVOKED specifically. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): update TaskAdmin for redesigned model Add date_created, duration_seconds to list_display; add trigger_source to list_filter; add input_data, duration_seconds, wait_time_seconds to readonly_fields. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): update Angular types and service for task redesign Replace PaperlessTaskName/PaperlessTaskType/PaperlessTaskStatus enums with new PaperlessTaskType, PaperlessTaskTriggerSource, PaperlessTaskStatus enums. Update PaperlessTask interface to new field names (task_type, trigger_source, input_data, result_message, related_document_ids). Update TasksService to filter by task_type instead of task_name. Update tasks component and system-status-dialog to use new field names. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(tasks): remove django-celery-results PaperlessTask now tracks all task results via Celery signals. The django-celery-results DB backend was write-only -- nothing reads from it. Drop the package and add a migration to clean up the orphaned tables. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: fix remaining tests broken by task system redesign Update all tests that created PaperlessTask objects with old field names to use PaperlessTaskFactory and new field names (task_type, trigger_source, status, result_message). Use apply_async instead of delay where mocked. Drop TestCheckSanityTaskRecording — tests PaperlessTask creation that was intentionally removed from check_sanity(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): improve test_api_tasks.py structure and add api marker - Move admin_client, v9_client, user_client fixtures to conftest.py so they can be reused by other API tests; all three now build on the rest_api_client fixture instead of creating APIClient() directly - Move regular_user fixture to conftest.py (was already done, now also used by the new client fixtures) - Add docstrings to every test method describing the behaviour under test - Move timedelta/timezone imports to module level - Register 'api' pytest marker in pyproject.toml and apply pytestmark to the entire file so all 40 tests are selectable via -m api Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(tasks): simplify task tracking code after redesign - Extract COMPLETE_STATUSES as a class constant on PaperlessTask, eliminating the repeated status tuple across models.py, views.py (3×), and filters.py - Extract _CELERY_STATE_TO_STATUS as a module-level constant instead of rebuilding the dict on every task_postrun - Extract _V9_TYPE_TO_TRIGGER_SOURCE and _RUNNABLE_TASKS as class constants on TasksViewSet instead of rebuilding on every request - Extract _TRIGGER_SOURCE_TO_V9_TYPE as a class constant on TaskSerializerV9 instead of rebuilding per serialized object - Extract _get_consume_args helper to deduplicate identical arg extraction logic in _extract_input_data, _determine_trigger_source, and _extract_owner_id - Move inline imports (re, traceback) and Avg to module level - Fix _DOCUMENT_SOURCE_TO_TRIGGER type annotation key type to DocumentSource instead of Any - Remove redundant truthiness checks in SystemStatusView branches already guarded by an is-None check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(tasks): add docstrings and rename _parse_legacy_result - Add docstrings to _extract_input_data, _determine_trigger_source, _extract_owner_id explaining what each helper does and why - Rename _parse_legacy_result -> _parse_consume_result: the function parses current consume_file string outputs (consumer.py returns "New document id N created" and "It is a duplicate of X (#N)"), not legacy data; the old name was misleading Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(tasks): extend and harden the task system redesign - TaskType: add EMPTY_TRASH, CHECK_WORKFLOWS, CLEANUP_SHARE_LINKS; remove INDEX_REBUILD (no backing task — beat schedule uses index_optimize) - TRACKED_TASKS: wire up all nine task types including the three new ones and llmindex_index / process_mail_accounts - Add task_revoked_handler so cancelled/expired tasks are marked REVOKED - Fix double-write: task_postrun_handler no longer overwrites result_data when status is already FAILURE (task_failure_handler owns that write) - v9 serialiser: map EMAIL_CONSUME and FOLDER_CONSUME to AUTO_TASK - views: scope task list to owner for regular users, admins see all; validate ?days= query param and return 400 on bad input - tests: add test_list_admin_sees_all_tasks; rename/fix test_parses_duplicate_string (duplicates produce SUCCESS, not FAILURE); use PaperlessTaskFactory in modified tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): fix MAIL_FETCH null input_data and postrun double-query - _extract_input_data: return {} instead of {"account_ids": None} when process_mail_accounts is called without an explicit account list (the normal beat-scheduled path); add test to cover this path - task_postrun_handler: replace filter().first() + filter().update() with get() + save(update_fields=[...]) — single fetch, single write, consistent with task_prerun_handler Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): add queryset stub to satisfy drf-spectacular schema generation TasksViewSet.get_queryset() accesses request.user, which drf-spectacular cannot provide during static schema generation. Adding a class-level queryset = PaperlessTask.objects.none() gives spectacular a model to introspect without invoking get_queryset(), eliminating both warnings and the test_valid_schema failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): fill coverage gaps in task system - test_task_signals: add TestTaskRevokedHandler (marks REVOKED, ignores None request, ignores unknown id); switch existing direct PaperlessTask.objects.create calls to PaperlessTaskFactory; import pytest_mock and use MockerFixture typing on mocker params - test_api_tasks: add test_rejects_invalid_days_param to TestSummary - tasks.service.spec: add dismissAllTasks test (POST acknowledge_all + reload) - models: add pragma: no cover to __str__, is_complete, and related_document_ids (trivial delegates, covered indirectly) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Well, that was a bad push. * Fixes v9 API compatability with testing coverage * fix(tasks): restore INDEX_OPTIMIZE enum and remove no-op run button INDEX_OPTIMIZE was dropped from the TaskType enum but still referenced in _RUNNABLE_TASKS (views.py) and the frontend system-status-dialog, causing an AttributeError at import time. Restore the enum value in the model and migration so the serializer accepts it, but remove it from _RUNNABLE_TASKS since index_optimize is a Tantivy no-op. Remove the frontend "Run Task" button for index optimization accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tasks): v9 type filter now matches all equivalent trigger sources The v9 ?type= query param mapped each value to a single TriggerSource, but the serializer maps multiple sources to the same v9 type value. A task serialized as "auto_task" would not appear when filtering by ?type=auto_task if its trigger_source was email_consume or folder_consume. Same issue for "manual_task" missing web_ui and api_upload sources. Changed to trigger_source__in with the full set of sources for each v9 type value. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tasks): give task_failure_handler full ownership of FAILURE path task_postrun_handler now early-returns for FAILURE states instead of redundantly writing status and date_done. task_failure_handler now computes duration_seconds and wait_time_seconds so failed tasks get complete timing data. This eliminates a wasted .get() + .save() round trip on every failed task and gives each handler a clean, non-overlapping responsibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tasks): resolve trigger_source header via TriggerSource enum lookup Replace two hardcoded string comparisons ("scheduled", "system") with a single TriggerSource(header_source) lookup so the enum values are the single source of truth. Any valid TriggerSource DB value passed in the header is accepted; invalid values fall through to the document-source / MANUAL logic. Update tests to pass enum values in headers rather than raw strings, and add a test for the invalid-header fallback path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tasks): use TriggerSource enum values at all apply_async call sites Replace raw strings ("system", "manual") with PaperlessTask.TriggerSource enum values in the three callers that can import models. The settings file remains a raw string (models cannot be imported at settings load time) with a comment pointing to the enum value it must match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(tasks): parametrize repetitive test cases in task test files test_api_tasks.py: - Collapse six trigger_source->v9-type tests into one parametrized test, adding the previously untested API_UPLOAD case - Collapse three task_name mapping tests (two remaps + pass-through) into one parametrized test - Collapse two acknowledge_all status tests into one parametrized test - Collapse two run-endpoint 400 tests into one parametrized test - Update run/ assertions to use TriggerSource enum values test_task_signals.py: - Collapse three trigger_source header tests into one parametrized test - Collapse two DocumentSource->TriggerSource mapping tests into one parametrized test - Collapse two prerun ignore-invalid-id tests into one parametrized test All parametrize cases use pytest.param with descriptive ids. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Handle JSON serialization for datetime and Path. Further restrist the v9 permissions as Copilot suggests * That should fix the generated schema/browser * Use XSerializer for the schema * A few more basic cases I see no value in covering * Drops the migration related stuff too. Just in case we want it again or it confuses people * fix: annotate tasks_summary_retrieve as array of TaskSummarySerializer Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: annotate tasks_active_retrieve as array of TaskSerializerV10 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Restore task running to superuser only * Removes the acknowledge/dismiss all stuff * Aligns v10 and v9 task permissions with each other * Short blurb just to warn users about the tasks being cleared --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
270 lines
7.5 KiB
TypeScript
270 lines
7.5 KiB
TypeScript
import { NgTemplateOutlet, SlicePipe } from '@angular/common'
|
|
import { Component, inject, OnDestroy, OnInit } from '@angular/core'
|
|
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
|
|
import { Router } from '@angular/router'
|
|
import {
|
|
NgbCollapseModule,
|
|
NgbDropdownModule,
|
|
NgbModal,
|
|
NgbNavModule,
|
|
NgbPaginationModule,
|
|
NgbPopoverModule,
|
|
} from '@ng-bootstrap/ng-bootstrap'
|
|
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
|
import {
|
|
debounceTime,
|
|
distinctUntilChanged,
|
|
filter,
|
|
first,
|
|
Subject,
|
|
takeUntil,
|
|
timer,
|
|
} from 'rxjs'
|
|
import { PaperlessTask } from 'src/app/data/paperless-task'
|
|
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
|
|
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
|
|
import { TasksService } from 'src/app/services/tasks.service'
|
|
import { ToastService } from 'src/app/services/toast.service'
|
|
import { ConfirmDialogComponent } from '../../common/confirm-dialog/confirm-dialog.component'
|
|
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
|
|
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
|
|
|
|
export enum TaskTab {
|
|
Queued = 'queued',
|
|
Started = 'started',
|
|
Completed = 'completed',
|
|
Failed = 'failed',
|
|
}
|
|
|
|
enum TaskFilterTargetID {
|
|
Name,
|
|
Result,
|
|
}
|
|
|
|
const FILTER_TARGETS = [
|
|
{ id: TaskFilterTargetID.Name, name: $localize`Name` },
|
|
{ id: TaskFilterTargetID.Result, name: $localize`Result` },
|
|
]
|
|
|
|
@Component({
|
|
selector: 'pngx-tasks',
|
|
templateUrl: './tasks.component.html',
|
|
styleUrls: ['./tasks.component.scss'],
|
|
imports: [
|
|
PageHeaderComponent,
|
|
IfPermissionsDirective,
|
|
CustomDatePipe,
|
|
SlicePipe,
|
|
FormsModule,
|
|
ReactiveFormsModule,
|
|
NgTemplateOutlet,
|
|
NgbCollapseModule,
|
|
NgbDropdownModule,
|
|
NgbNavModule,
|
|
NgbPaginationModule,
|
|
NgbPopoverModule,
|
|
NgxBootstrapIconsModule,
|
|
],
|
|
})
|
|
export class TasksComponent
|
|
extends LoadingComponentWithPermissions
|
|
implements OnInit, OnDestroy
|
|
{
|
|
tasksService = inject(TasksService)
|
|
private modalService = inject(NgbModal)
|
|
private readonly router = inject(Router)
|
|
private readonly toastService = inject(ToastService)
|
|
|
|
public activeTab: TaskTab
|
|
public selectedTasks: Set<number> = new Set()
|
|
public togggleAll: boolean = false
|
|
public expandedTask: number
|
|
|
|
public pageSize: number = 25
|
|
public page: number = 1
|
|
|
|
public autoRefreshEnabled: boolean = true
|
|
|
|
private _filterText: string = ''
|
|
get filterText() {
|
|
return this._filterText
|
|
}
|
|
set filterText(value: string) {
|
|
this.filterDebounce.next(value)
|
|
}
|
|
|
|
public filterTargetID: TaskFilterTargetID = TaskFilterTargetID.Name
|
|
public get filterTargetName(): string {
|
|
return this.filterTargets.find((t) => t.id == this.filterTargetID).name
|
|
}
|
|
private filterDebounce: Subject<string> = new Subject<string>()
|
|
|
|
public get filterTargets(): Array<{ id: number; name: string }> {
|
|
return [TaskTab.Failed, TaskTab.Completed].includes(this.activeTab)
|
|
? FILTER_TARGETS
|
|
: FILTER_TARGETS.slice(0, 1)
|
|
}
|
|
|
|
get dismissButtonText(): string {
|
|
return this.selectedTasks.size > 0
|
|
? $localize`Dismiss selected`
|
|
: $localize`Dismiss all`
|
|
}
|
|
|
|
ngOnInit() {
|
|
this.tasksService.reload()
|
|
timer(5000, 5000)
|
|
.pipe(
|
|
filter(() => this.autoRefreshEnabled),
|
|
takeUntil(this.unsubscribeNotifier)
|
|
)
|
|
.subscribe(() => {
|
|
this.tasksService.reload()
|
|
})
|
|
|
|
this.filterDebounce
|
|
.pipe(
|
|
takeUntil(this.unsubscribeNotifier),
|
|
debounceTime(100),
|
|
distinctUntilChanged(),
|
|
filter((query) => !query.length || query.length > 2)
|
|
)
|
|
.subscribe((query) => (this._filterText = query))
|
|
}
|
|
|
|
ngOnDestroy() {
|
|
super.ngOnDestroy()
|
|
this.tasksService.cancelPending()
|
|
}
|
|
|
|
dismissTask(task: PaperlessTask) {
|
|
this.dismissTasks(task)
|
|
}
|
|
|
|
dismissTasks(task: PaperlessTask = undefined) {
|
|
let tasks = task ? new Set([task.id]) : new Set(this.selectedTasks.values())
|
|
if (!task && tasks.size == 0)
|
|
tasks = new Set(this.tasksService.allFileTasks.map((t) => t.id))
|
|
if (tasks.size > 1) {
|
|
let modal = this.modalService.open(ConfirmDialogComponent, {
|
|
backdrop: 'static',
|
|
})
|
|
modal.componentInstance.title = $localize`Confirm Dismiss All`
|
|
modal.componentInstance.messageBold = $localize`Dismiss all ${tasks.size} tasks?`
|
|
modal.componentInstance.btnClass = 'btn-warning'
|
|
modal.componentInstance.btnCaption = $localize`Dismiss`
|
|
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
|
|
modal.componentInstance.buttonsEnabled = false
|
|
modal.close()
|
|
this.tasksService.dismissTasks(tasks).subscribe({
|
|
error: (e) => {
|
|
this.toastService.showError($localize`Error dismissing tasks`, e)
|
|
modal.componentInstance.buttonsEnabled = true
|
|
},
|
|
})
|
|
this.clearSelection()
|
|
})
|
|
} else {
|
|
this.tasksService.dismissTasks(tasks).subscribe({
|
|
error: (e) =>
|
|
this.toastService.showError($localize`Error dismissing task`, e),
|
|
})
|
|
this.clearSelection()
|
|
}
|
|
}
|
|
|
|
dismissAndGo(task: PaperlessTask) {
|
|
this.dismissTask(task)
|
|
this.router.navigate(['documents', task.related_document_ids?.[0]])
|
|
}
|
|
|
|
expandTask(task: PaperlessTask) {
|
|
this.expandedTask = this.expandedTask == task.id ? undefined : task.id
|
|
}
|
|
|
|
toggleSelected(task: PaperlessTask) {
|
|
this.selectedTasks.has(task.id)
|
|
? this.selectedTasks.delete(task.id)
|
|
: this.selectedTasks.add(task.id)
|
|
}
|
|
|
|
get currentTasks(): PaperlessTask[] {
|
|
let tasks: PaperlessTask[] = []
|
|
switch (this.activeTab) {
|
|
case TaskTab.Queued:
|
|
tasks = this.tasksService.queuedFileTasks
|
|
break
|
|
case TaskTab.Started:
|
|
tasks = this.tasksService.startedFileTasks
|
|
break
|
|
case TaskTab.Completed:
|
|
tasks = this.tasksService.completedFileTasks
|
|
break
|
|
case TaskTab.Failed:
|
|
tasks = this.tasksService.failedFileTasks
|
|
break
|
|
}
|
|
if (this._filterText.length) {
|
|
tasks = tasks.filter((t) => {
|
|
if (this.filterTargetID == TaskFilterTargetID.Name) {
|
|
return (t.input_data?.filename as string)
|
|
?.toLowerCase()
|
|
.includes(this._filterText.toLowerCase())
|
|
} else if (this.filterTargetID == TaskFilterTargetID.Result) {
|
|
return t.result_message
|
|
?.toLowerCase()
|
|
.includes(this._filterText.toLowerCase())
|
|
}
|
|
})
|
|
}
|
|
return tasks
|
|
}
|
|
|
|
toggleAll(event: PointerEvent) {
|
|
if ((event.target as HTMLInputElement).checked) {
|
|
this.selectedTasks = new Set(this.currentTasks.map((t) => t.id))
|
|
} else {
|
|
this.clearSelection()
|
|
}
|
|
}
|
|
|
|
clearSelection() {
|
|
this.togggleAll = false
|
|
this.selectedTasks.clear()
|
|
}
|
|
|
|
duringTabChange() {
|
|
this.page = 1
|
|
}
|
|
|
|
beforeTabChange() {
|
|
this.resetFilter()
|
|
this.filterTargetID = TaskFilterTargetID.Name
|
|
}
|
|
|
|
get activeTabLocalized(): string {
|
|
switch (this.activeTab) {
|
|
case TaskTab.Queued:
|
|
return $localize`queued`
|
|
case TaskTab.Started:
|
|
return $localize`started`
|
|
case TaskTab.Completed:
|
|
return $localize`completed`
|
|
case TaskTab.Failed:
|
|
return $localize`failed`
|
|
}
|
|
}
|
|
|
|
public resetFilter() {
|
|
this._filterText = ''
|
|
}
|
|
|
|
filterInputKeyup(event: KeyboardEvent) {
|
|
if (event.key == 'Enter') {
|
|
this._filterText = (event.target as HTMLInputElement).value
|
|
} else if (event.key === 'Escape') {
|
|
this.resetFilter()
|
|
}
|
|
}
|
|
}
|