amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-08-18 16:53:21 +00:00
Code Issues Projects Releases Wiki Activity

Idea: redesign scheduled-backup export module against real export/ package

Browse Source 
Section 2 assumed a bare documents/export.py module, but PRs #13490/#13661
already refactored the exporter into a documents/export/ package with
ExportSink abstractions and zip-compression options. Reworks the export
extraction (ExportOptions/ExportError/ExportRunner/run_export) to fit the
actual current architecture, including the progress-bar and CryptMixin
coupling the original spec didn't account for.
This commit is contained in:
stumpylog
2026-08-18 09:27:55 -07:00
parent bcb4d1265c
commit 152d291e7d
1 changed files with 87 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-05-15-scheduled-backup-design.md
+87 -23
View File
@@ -49,7 +49,7 @@ class BackupConfiguration(AbstractSingletonModel):
```
- `output_dir` blank/empty means backup is disabled (the task treats it as a no-op).
- `output_dir` must be an absolute path. The serializer validates this via a custom validator; `run_export` also calls `.resolve()` on the path unconditionally.
- `output_dir` must be an absolute path. The serializer validates this via a custom validator; `ExportRunner.run()` also calls `.resolve()` on the path unconditionally.
- `keep_count = 0` means keep all backups; no rotation is performed.
### Migration
@@ -66,7 +66,9 @@ The migration is created in `src/paperless/migrations/` (not `src/documents/migr
## Section 2: Export Module
New module `src/documents/export.py` contains the export contract and core logic, extracted from `document_exporter`'s `handle()` method.
**Revised 2026-08-18** against the export refactor that has landed since this spec was written (PRs #13490 and #13661): `document_exporter`'s logic is no longer a flat `Command.handle()` body. There is now a `src/documents/export/` **package** (`compression.py`: `COMPRESSION_CHOICES`/`COMPRESSION_METHODS`/`compression_available`/`level_error`; `sinks.py`: `ExportSink`/`DirectoryExportSink`/`ZipExportSink`/`StreamingManifestWriter`), and `document_exporter.Command` (`CryptMixin`, `PaperlessCommand`) builds a `sink: ExportSink` in `handle()` and calls `self.dump(sink)`, followed by 7 more helper methods (`generate_base_name`, `generate_document_targets`, `copy_document_files`, `generate_share_link_bundle_target`, `copy_share_link_bundle_file`, `_encrypt_record_inline`, `_write_split_manifest`) that `dump()` calls into — all still `self.`-based methods on the `Command` class, not free functions. This is a materially bigger extraction than "move `handle()`'s body," because `dump()` and its helpers also depend on three CLI-specific things: `self.track(...)` (a `PaperlessCommand`/rich-progress-bar wrapper, `document_exporter.py:442`), one `self.stdout.write(self.style.NOTICE(...))` call (`document_exporter.py:380-384`, the no-passphrase warning), and mixing in `CryptMixin` for `setup_crypto`/`_encrypt_record_inline`. A scheduled Celery task has none of `self.track`/`self.stdout`/`self.style` available, so the extraction has to account for that rather than being a verbatim cut-paste.
New module `src/documents/export/service.py` (inside the existing package — **not** a bare `src/documents/export.py`, since that name is already taken by the package).
### `ExportOptions` dataclass
@@ -84,24 +86,83 @@ class ExportOptions:
split_manifest: bool = False
zip_export: bool = False
zip_name: str | None = None # None -> default date-based name
zip_compression: str | None = None # None -> "deflated" default
zip_compression_level: int | None = None
data_only: bool = False
passphrase: str | None = None
batch_size: int = 500
```
`zip_name = None` means the caller wants the default date-based name. `run_export` resolves `None` internally to `f"export-{timezone.localdate().isoformat()}"` before use — callers never need to supply a default. The scheduled task always passes an explicit timestamped name.
Same shape as originally designed, plus `zip_compression`/`zip_compression_level` — both exist on the current CLI (`document_exporter.py:200-220`, added by #13661) and were missing from the original dataclass sketch. `zip_name = None` means the caller wants the default date-based name; resolved internally to `f"export-{timezone.localdate().isoformat()}"` before use. The scheduled task always passes an explicit timestamped name.
### `run_export(options: ExportOptions) -> None`
### `ExportError` exception
The body of the current `Command.handle()` in `document_exporter` moves here, reading from `ExportOptions` instead of parsed CLI options. No behaviour changes.
```python
class ExportError(Exception):
"""Raised for export-time validation failures (bad target path, unavailable
compression method/level, incompatible option combinations)."""
```
Replaces the current `CommandError` raises inside `handle()`/`dump()` (`document_exporter.py:263-300`) for the logic that moves into the service layer — `CommandError` is Django-management-command-specific and isn't appropriate to raise from a Celery task. `document_exporter.Command` catches `ExportError` at its call site and re-raises as `CommandError` for CLI-appropriate display; `scheduled_backup` lets it propagate (Celery/the signal handlers mark the task failed).
### `ExportRunner` class
Owns the state `dump()` and its 7 helper methods currently keep on `self` (`exported_files`, and the option fields), decoupled from `BaseCommand`:
```python
class ExportRunner(CryptMixin):
def __init__(
self,
options: ExportOptions,
*,
progress: Callable[[Iterable, ...], Iterable] | None = None,
) -> None:
self.options = options
self._track = progress or (lambda iterable, **_: iterable) # no-op when not run interactively
self.exported_files: set[str] = set()
def run(self) -> None:
# 1. Validate target (exists/is_dir/writable) -> ExportError, not CommandError
# 2. Validate zip compression method/level -> ExportError
# 3. Build sink (ZipExportSink / DirectoryExportSink), same logic as
# today's handle() lines 302-317
# 4. with FileLock(settings.MEDIA_LOCK), sink:
# self.dump(sink)
...
def dump(self, sink: ExportSink) -> None:
# Same body as today's Command.dump() (document_exporter.py:323-499),
# with self.X -> self.options.X for option fields, self.track(...) ->
# self._track(...), and the self.stdout.write(self.style.NOTICE(...))
# no-passphrase warning -> logger.warning(...) (module logger
# "documents.export").
...
# generate_base_name / generate_document_targets / copy_document_files /
# generate_share_link_bundle_target / copy_share_link_bundle_file /
# _encrypt_record_inline / _write_split_manifest move here unchanged
# (document_exporter.py:500-657) - no self.X reference in them needs
# anything Command-specific beyond what's already covered above.
```
`CryptMixin` (`documents/management/commands/mixins.py`) has no `BaseCommand`/`self.stdout` coupling of its own — it's a plain mixin (`setup_crypto`/`encrypt_string`/`decrypt_string`), safe to use outside a management command.
### `run_export()` convenience function
```python
def run_export(options: ExportOptions, *, progress: Callable[..., Iterable] | None = None) -> None:
ExportRunner(options, progress=progress).run()
```
The single public entry point both callers use — `scheduled_backup` calls it with no `progress`; `document_exporter.Command` passes `progress=self.track` so the CLI keeps its rich progress bar.
### Refactored `document_exporter` management command
Becomes a thin CLI adapter:
`add_arguments` is unchanged. `handle()` becomes a thin adapter:
1. Parse arguments (unchanged)
2. Construct `ExportOptions` from parsed args
3. Call `run_export(options)`
2. Construct `ExportOptions` from parsed args (including the now-present `zip_compression`/`zip_compression_level`)
3. Call `run_export(options, progress=self.track)`, catching `ExportError` and re-raising as `CommandError(str(e))`
---
@@ -109,6 +170,8 @@ Becomes a thin CLI adapter:
### `scheduled_backup` task in `src/documents/tasks.py`
Imports `ExportOptions`, `ExportError`, `run_export` from `documents.export.service`.
```
1. Load BackupConfiguration (singleton)
2. If output_dir is blank, log a debug message and return (no-op, no PaperlessTask created)
@@ -133,6 +196,7 @@ Key design notes:
- Rotation uses `export-*.zip` glob, not `*.zip`, to avoid matching zip files in the directory that paperless did not create.
- Rotation occurs only after a successful export, so a failed run does not consume a rotation slot.
- The timestamp format `YYYY-MM-DD-HHMMSS` in local time ensures multiple runs per day produce distinct filenames without collision.
- `run_export()` raising `ExportError` (bad `output_dir`, unavailable compression) is not caught in the task — it propagates, Celery marks the task failed, and `task_failure_handler` (Section 3, PaperlessTask integration below) updates the `PaperlessTask` record accordingly. No separate error handling needed here.
### PaperlessTask integration
@@ -199,27 +263,27 @@ The component performs a GET to `/api/backup_config/` on load and a PATCH on sav
## File Change Summary
| File | Change |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `src/paperless/models.py` | Add `BackupConfiguration` model |
| `src/paperless/serialisers.py` | Add `BackupConfigurationSerializer` |
| `src/paperless/views.py` | Add `BackupConfigurationViewSet` |
| `src/paperless/urls.py` | Register `/api/backup_config/` route |
| `src/paperless/settings/custom.py` | Add `PAPERLESS_EXPORT_TASK_CRON` beat entry |
| `src/documents/export.py` | New module: `ExportOptions`, `run_export()` |
| `src/documents/management/commands/document_exporter.py` | Thin wrapper around `run_export()` |
| `src/documents/models.py` | Add `PaperlessTask.TaskType.BACKUP` |
| `src/documents/signals/handlers.py` | Add `BACKUP` to `TRACKED_TASKS`; add `_backup_task_is_tracked()`; update all 5 signal handlers |
| `src/documents/tasks.py` | Add `scheduled_backup` task |
| `src-ui/` | New or extended settings component (location TBD) |
| `src/paperless/migrations/` | New migration for `BackupConfiguration` |
| File | Change |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `src/paperless/models.py` | Add `BackupConfiguration` model |
| `src/paperless/serialisers.py` | Add `BackupConfigurationSerializer` |
| `src/paperless/views.py` | Add `BackupConfigurationViewSet` |
| `src/paperless/urls.py` | Register `/api/backup_config/` route |
| `src/paperless/settings/custom.py` | Add `PAPERLESS_EXPORT_TASK_CRON` beat entry |
| `src/documents/export/service.py` | New module: `ExportOptions`, `ExportError`, `ExportRunner`, `run_export()` |
| `src/documents/management/commands/document_exporter.py` | `Command.handle()` becomes a thin adapter; `dump()` and its 7 helper methods move to `ExportRunner` |
| `src/documents/models.py` | Add `PaperlessTask.TaskType.BACKUP` |
| `src/documents/signals/handlers.py` | Add `BACKUP` to `TRACKED_TASKS`; add `_backup_task_is_tracked()`; update all 5 signal handlers |
| `src/documents/tasks.py` | Add `scheduled_backup` task |
| `src-ui/` | New or extended settings component (location TBD) |
| `src/paperless/migrations/` | New migration for `BackupConfiguration` |
---
## Testing
- **`src/paperless/tests/test_backup_config.py`** — model, serializer, API (GET/PATCH)
- **`src/documents/tests/test_export.py`** — new unit tests for `run_export()` directly; `test_management_exporter.py` retains its existing CLI wiring tests and gains tests for the thin-wrapper behaviour
- **`src/documents/tests/export/test_service.py`** — new unit tests for `ExportRunner`/`run_export()` directly, alongside the existing `src/documents/tests/export/test_compression.py` and `test_sinks.py` (the `export/` test package already exists and mirrors the `documents/export/` package structure — follow that convention rather than a flat `test_export.py`). `test_management_exporter.py` retains its existing CLI wiring tests and gains tests for the thin-adapter behaviour (arg parsing → `ExportOptions` → `run_export`, `ExportError` → `CommandError` translation)
- **`src/documents/tests/test_tasks_backup.py`** — `scheduled_backup` task: no-op when `output_dir` blank, export called with correct options, rotation deletes correct files, rotation skipped when `keep_count=0`
- **`src/documents/tests/test_task_signals.py`** — signal handler behaviour for `BACKUP` task type: no record created when `output_dir` blank, all downstream handlers skip cleanly when no record exists, normal lifecycle when `output_dir` is set
- Frontend unit tests for the settings component