mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-19 01:03:18 +00:00
Saves some other ideas and moves a few to done
This commit is contained in:
stumpylog
@@ -1,337 +0,0 @@
|
||||
# Export Sink Architecture — Design
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Branch base:** `dev`
|
||||
**Status:** Approved design, pending implementation plan
|
||||
|
||||
## Problem
|
||||
|
||||
The `document_exporter` management command can export to a folder or to a zip
|
||||
file, but the zip support is bolted on rather than designed in:
|
||||
|
||||
- **Zip mode is a temp-dir detour.** `handle()` redirects `self.target` to a
|
||||
`tempfile.TemporaryDirectory` in `SCRATCH_DIR`, runs the entire export against
|
||||
that directory, then calls `shutil.make_archive` to zip the whole tree and
|
||||
cleans the temp dir up (`document_exporter.py:322-358`). The export is written
|
||||
to disk twice (loose files, then the zip).
|
||||
|
||||
- **An attempted "direct to zip" refactor leaks the destination everywhere.**
|
||||
The prior work on `feature-direct-zip-export` threads `if self.zip_export:`
|
||||
branches through `check_and_copy`, `check_and_write_json`,
|
||||
`_write_split_manifest`, `dump`, `handle`, and `StreamingManifestWriter`. Each
|
||||
write site grew a second code path plus a `.resolve().relative_to(self.target)`
|
||||
arcname dance. The destination became a cross-cutting concern smeared across
|
||||
the command.
|
||||
|
||||
- **The command owns logic that isn't about the export contents.** Incremental
|
||||
sync — the `files_in_export_dir` snapshot, the `--compare-checksums` /
|
||||
`--compare-json` skip-if-unchanged checks, and the `--delete` stale-file prune —
|
||||
is interleaved with the logic that decides _what_ to export. These behaviors
|
||||
only make sense for a folder destination, yet they live in the command body.
|
||||
|
||||
- **Atomicity is informal.** A backup must never look complete when it isn't.
|
||||
The temp-dir approach happens to be safe (the zip is built last), but there is
|
||||
no explicit "produce the archive only if the whole run succeeded" contract, and
|
||||
the direct-to-zip branch had to hand-manage a `.tmp` file inline.
|
||||
|
||||
## Goal
|
||||
|
||||
Separate **what** is exported (the command's job) from **where/how** it lands
|
||||
(the destination's job), behind a small `ExportSink` abstraction. The command
|
||||
declares files, JSON blobs, and a streamed manifest; the sink decides whether and
|
||||
how to persist each one. Folder and zip become two interchangeable sinks, and a
|
||||
future `S3ExportSink` is a third implementation rather than a fourth set of
|
||||
branches. The zip is produced **only** if the entire export succeeds.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- New `documents/export/` package with the `ExportSink` ABC and two concrete
|
||||
sinks (`DirectoryExportSink`, `ZipExportSink`).
|
||||
- Move all incremental-sync machinery (snapshot, compare, prune) out of the
|
||||
command and into `DirectoryExportSink`.
|
||||
- Rewrite `document_exporter.handle()` / `dump()` to be destination-agnostic.
|
||||
- Simplify `StreamingManifestWriter` to write to a sink-provided handle.
|
||||
- Unit tests for each sink; keep existing command-level tests green.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- `bulk_download.py` / `BulkArchiveStrategy` and share-link bundle zipping. Those
|
||||
select _which document files_ go in and stream to an HTTP response with no
|
||||
atomic-finalize requirement — a different axis from the backup sink. Untouched.
|
||||
- Actually implementing an S3 (or any cloud) sink. The interface is designed to
|
||||
_allow_ one; we do not build one (YAGNI).
|
||||
- Changing the export's on-disk/in-zip layout, manifest schema, crypto, or any
|
||||
CLI flag's meaning. Behavior is preserved; only the destination plumbing moves.
|
||||
- Zip compression control (method / level). The `ZipExportSink` keeps today's
|
||||
fixed `ZIP_DEFLATED` here; making compression configurable is a follow-up —
|
||||
see `2026-06-16-export-zip-compression-design.md`, which depends on this
|
||||
refactor landing first. The sink is the single seam that makes it a small,
|
||||
isolated change.
|
||||
|
||||
## Decisions
|
||||
|
||||
These were settled during brainstorming:
|
||||
|
||||
1. **Scope is the `document_exporter` command only.** Design the interface so an
|
||||
S3 sink could be added later; do not refactor `bulk_download` or share bundles.
|
||||
2. **`--compare-*` are folder-only (hard error with `--zip`); `--delete` is kept
|
||||
for both.** `--compare-checksums` / `--compare-json` are genuine no-ops in zip
|
||||
mode today (the temp dir is always empty, so the compare always copies), so
|
||||
combining either with `--zip` raises a `CommandError` up front. **`--delete`,
|
||||
however, is an existing tested feature in zip mode** — it wipes the destination
|
||||
directory of pre-existing files/dirs before the archive lands
|
||||
(`test_export_zipped_with_delete`). Its meaning differs by destination: folder
|
||||
`--delete` prunes stale exported files; zip `--delete` clears the target dir.
|
||||
Both are preserved — `--delete` is a parameter of _both_ sinks, not an error.
|
||||
3. **The zip manifest spools to a temp file, not memory.** The sink exposes a
|
||||
streaming-write handle. The zip sink streams the manifest to a single temp
|
||||
file in `SCRATCH_DIR` and adds it as the manifest entry at finalize, keeping
|
||||
peak memory flat regardless of library size. The only "temp" artifact is one
|
||||
manifest file, not a whole export tree.
|
||||
|
||||
## Architecture
|
||||
|
||||
### The `ExportSink` interface
|
||||
|
||||
New module `documents/export/sinks.py`:
|
||||
|
||||
```python
|
||||
class ExportSink(AbstractContextManager):
|
||||
"""Destination for a document export.
|
||||
|
||||
The command declares export contents via the three verbs below; the sink
|
||||
decides whether and how to persist each item. arcname is always a relative
|
||||
POSIX path (e.g. "manifest.json", "originals/foo.pdf").
|
||||
"""
|
||||
|
||||
def add_file(
|
||||
self,
|
||||
source: Path,
|
||||
arcname: str,
|
||||
*,
|
||||
checksum: str | None = None,
|
||||
) -> None:
|
||||
"""Persist an existing file at the relative arcname."""
|
||||
|
||||
def add_json(self, content: list | dict, arcname: str) -> None:
|
||||
"""Persist JSON-serializable content at the relative arcname."""
|
||||
|
||||
def stream(self, arcname: str) -> ContextManager[TextIO]:
|
||||
"""Yield a writable text handle for incrementally produced content.
|
||||
|
||||
Reserved for the bulk manifest. At most one stream may be open at a
|
||||
time; add_file/add_json may be called freely while it is open.
|
||||
"""
|
||||
|
||||
# __enter__ opens the sink and returns self.
|
||||
# __exit__ calls finalize() on success, abort() on exception.
|
||||
```
|
||||
|
||||
**Contract / invariants** (the checklist a future sink author honors):
|
||||
|
||||
- `arcname` is relative and **POSIX-style (forward slashes)**; the sink maps it to
|
||||
its own namespace (folder: joined under the target; zip: the entry name). The
|
||||
command must build arcnames with `Path(...).as_posix()` — `str(Path(...))`
|
||||
yields backslashes on Windows, which corrupts zip entry names and makes the
|
||||
manifest's stored paths non-portable. The same string is used both as the sink
|
||||
key and as the value stored in the manifest (`EXPORTER_FILE_NAME` etc.), so it
|
||||
must be POSIX at the point of construction. (The share-link bundle path already
|
||||
uses `.as_posix()`; the document targets currently do not and must be fixed.)
|
||||
- At most one `stream()` is open at a time. It is the manifest. `add_file` /
|
||||
`add_json` may be interleaved with an open stream — implementations that can't
|
||||
interleave a real stream (zip, S3) must spool the stream to a side buffer and
|
||||
emit it at `finalize()`.
|
||||
- The sink is a context manager. Normal exit finalizes; an exception aborts.
|
||||
**No partial or failed run may leave a "complete-looking" artifact.**
|
||||
|
||||
### `DirectoryExportSink(target, *, compare_checksums, compare_json, delete)`
|
||||
|
||||
Owns everything the command currently does for folder mode:
|
||||
|
||||
- On open: snapshot existing files under `target` (today's `files_in_export_dir`).
|
||||
- `add_file`: the `check_and_copy` skip logic (mtime/size, or checksum when
|
||||
`compare_checksums`), then copy with stat preservation. Records the arcname as
|
||||
"seen this run".
|
||||
- `add_json`: the `check_and_write_json` blake2b compare-or-write (honoring
|
||||
`compare_json`). Records the arcname as seen.
|
||||
- `stream`: yields a handle writing to `<arcname>.tmp`; on context close, applies
|
||||
the `compare_json` blake2b compare and renames-or-discards (today's
|
||||
`StreamingManifestWriter` finalize). Records the arcname as seen.
|
||||
- `finalize()` (success only): if `delete`, prune every snapshot file not seen
|
||||
this run and clean up emptied directories (today's stale-delete pass).
|
||||
- `abort()` (on exception): discard any in-flight `.tmp`; leave existing files
|
||||
intact; do **not** run the prune.
|
||||
|
||||
The folder sink is inherently in-place/incremental, not atomic — that is its
|
||||
nature and is unchanged. Its safety is the per-file `.tmp`+rename it already does.
|
||||
|
||||
### `ZipExportSink(target, zip_name, *, delete)`
|
||||
|
||||
- On open: ensure `SCRATCH_DIR` exists (`mkdir(parents=True, exist_ok=True)` —
|
||||
today's `handle()` does this before using it; the sink must do it now), then
|
||||
open a `zipfile.ZipFile` at `<target>/<zip_name>.zip.tmp` (`ZIP_DEFLATED`,
|
||||
`allowZip64=True`). The `.zip.tmp` lives in the same directory as the final
|
||||
`.zip` so the finalize rename is atomic (same filesystem).
|
||||
- `add_file` / `add_json`: write the entry directly, first emitting directory
|
||||
marker entries for parent paths so every zip viewer shows the folder structure
|
||||
(today's `_ensure_zip_dirs`). A _flat_ export (no `--use-folder-prefix`, no
|
||||
nested arcnames) has no parent dirs, so it emits **zero** markers — matching
|
||||
today's `make_archive` output for flat trees (keeps the `namelist()` count
|
||||
assertions in `test_export_zipped` valid). Nested/prefixed exports gain marker
|
||||
entries; any count assertion on those must be audited.
|
||||
- `stream`: yields a handle writing to a single temp file in `SCRATCH_DIR`.
|
||||
- `finalize()` (success only): add the spooled manifest temp file as its entry,
|
||||
close the zip, then **if `delete`, wipe the destination directory** of every
|
||||
pre-existing file/dir except the in-progress `.zip.tmp` and any prior `.zip`
|
||||
(today's zip `--delete` behavior), then atomically rename `.zip.tmp` → `.zip`.
|
||||
- `abort()` (on exception): close the zip, unlink the `.zip.tmp`, delete the
|
||||
manifest temp file. **A `.zip` therefore exists only after a fully successful
|
||||
run**, and on abort the destination is never wiped.
|
||||
- Rejects `compare_*` (the command guards this before constructing the sink). It
|
||||
does **not** reject `delete` — that is a supported zip behavior (see above).
|
||||
|
||||
### Command changes (`document_exporter.py`)
|
||||
|
||||
- **`handle()`**: validate the target, then _up front_ raise `CommandError` if
|
||||
`--compare-checksums` or `--compare-json` is combined with `--zip` (those are
|
||||
no-ops in zip mode). `--delete` is **not** rejected — it is passed to whichever
|
||||
sink is built. Construct the appropriate sink (`delete=` passed to both). Run
|
||||
the export as `with sink: self.dump(sink)`. Delete the temp-dir /
|
||||
`shutil.make_archive` block entirely.
|
||||
- **`--data-only`**: unchanged in meaning — it simply skips every `sink.add_file`
|
||||
call (no document/thumbnail/archive/bundle files) while the manifest stream and
|
||||
`metadata.json` are still written. Works identically for both sinks; no sink
|
||||
code is data-only-aware. (`test_export_data_only` and its zip equivalent stay
|
||||
green.)
|
||||
- **`dump(sink)`**: destination-agnostic. Builds relative arcnames and calls
|
||||
`sink.add_file(...)`, `sink.add_json(...)`, and `sink.stream("manifest.json")`.
|
||||
`self.files_in_export_dir`, `check_and_copy`, `check_and_write_json`, and the
|
||||
stale-delete pass are removed (their logic now lives in the folder sink).
|
||||
- **`generate_document_targets`**: returns relative arcnames
|
||||
(`originals/<name>`, `<name>-thumbnail.webp`, `archive/<name>-archive.pdf`)
|
||||
instead of absolute `self.target / ...` paths. It already writes the relative
|
||||
name into `document_dict[EXPORTER_FILE_NAME]` etc.; we just drop the absolute
|
||||
half.
|
||||
- **`StreamingManifestWriter`**: simplified to write JSON-array records to the
|
||||
text handle returned by `sink.stream("manifest.json")`. It no longer knows
|
||||
folder vs zip, owns no `.tmp` logic, and has no compare/zip parameters — that
|
||||
behavior moved into each sink's `stream()`.
|
||||
- **Crypto / passphrase** handling stays in the command: it transforms record
|
||||
_contents_ before they reach the sink, which is independent of destination.
|
||||
- **Progress tracking stays in the command — the sinks know nothing about it.**
|
||||
`PaperlessCommand.track()` wraps the _document iterable_ in `dump()` and ticks
|
||||
the Rich bar once per document. That loop stays in the command; each iteration
|
||||
calls `sink.add_file(...)`, so the per-document progress is preserved
|
||||
unchanged. The sinks deliberately do **not** depend on `PaperlessCommand`,
|
||||
`track()`, or Rich — coupling the destination abstraction to the command
|
||||
framework would defeat the isolation goal and make the sinks impossible to unit
|
||||
-test without a full command. (A sink is a plain context-managed I/O object; it
|
||||
is constructed by `handle()` and exercised directly in `test_sinks.py`.) If
|
||||
finer-grained progress is ever wanted for a single very large file, that is a
|
||||
future enhancement layered via an optional callback — not a `PaperlessCommand`
|
||||
dependency, and out of scope here.
|
||||
|
||||
### How `--split-manifest` fits (no sink special-casing)
|
||||
|
||||
`--split-manifest` is purely a command-level choice and touches no sink code:
|
||||
|
||||
- The single bulk `manifest.json` is always the one and only `sink.stream(...)`
|
||||
handle. In split mode it simply carries fewer record types (document records,
|
||||
notes, and custom-field-instances are redirected out).
|
||||
- Per-document `<base>-manifest.json` files are small _complete_ JSON blobs — they
|
||||
were never streamed. `_write_split_manifest` collapses to building the content
|
||||
list and one `sink.add_json(content, "<base>-manifest.json")` call, exactly
|
||||
like `metadata.json`.
|
||||
|
||||
Because the manifest stream is backed by its own handle (a `.tmp` file in the
|
||||
folder sink, a `SCRATCH_DIR` temp file in the zip sink) and never an open zip
|
||||
entry, the per-document `add_json` / `add_file` calls made _while the bulk
|
||||
manifest stream is open_ never collide with it.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
handle(options)
|
||||
├─ validate target; reject --compare-* + --zip → CommandError (--delete allowed)
|
||||
├─ sink = DirectoryExportSink(..., delete=…) | ZipExportSink(..., delete=…)
|
||||
└─ with FileLock(MEDIA_LOCK), sink:
|
||||
dump(sink)
|
||||
├─ with sink.stream("manifest.json") as mh:
|
||||
│ writer = StreamingManifestWriter(mh)
|
||||
│ ├─ global querysets → writer.write_batch(...) (encrypted inline)
|
||||
│ ├─ per document:
|
||||
│ │ ├─ sink.add_file(source, "originals/…", checksum=…)
|
||||
│ │ ├─ sink.add_file(thumb, "…-thumbnail.webp")
|
||||
│ │ ├─ sink.add_file(archive,"archive/…-archive.pdf", checksum=…)
|
||||
│ │ └─ split? sink.add_json(doc_bundle, "…-manifest.json")
|
||||
│ │ : writer.write_record(doc_record)
|
||||
│ └─ per share-link bundle: sink.add_file(...) + writer.write_record(...)
|
||||
└─ sink.add_json(metadata, "metadata.json")
|
||||
(success → sink.finalize(); exception → sink.abort())
|
||||
```
|
||||
|
||||
## Error handling & atomicity
|
||||
|
||||
- Any exception in `dump()` propagates through `with sink:` → `__exit__` →
|
||||
`abort()`. Zip: the `.zip.tmp` and the manifest temp file are deleted, and the
|
||||
destination is **not** wiped; **no `.zip` is produced.** Folder: in-flight
|
||||
`.tmp` files are discarded, existing files are left intact, and the stale-prune
|
||||
does not run.
|
||||
- `finalize()` runs only on clean exit, after all contents are written. For the
|
||||
zip: optionally wipe the destination (`--delete`), then the single `.zip.tmp` →
|
||||
`.zip` rename (atomic on the same filesystem). For the folder: the optional
|
||||
stale-delete prune.
|
||||
- **Honest limits of the atomicity guarantee.** The guarantee is "no
|
||||
_complete-looking_ `.zip` after a failed run," not "no leftovers." If the
|
||||
process is `SIGKILL`ed or the rename itself fails _after_ the zip is closed, a
|
||||
`.zip.tmp` may be orphaned — that is the safe direction (no false-complete
|
||||
`.zip`), but stale `.zip.tmp` files are **not** auto-cleaned on a later run
|
||||
(matching the prior branch). `KeyboardInterrupt` is a `BaseException` but
|
||||
`__exit__` still runs, so `abort()` fires normally. The rename being atomic and
|
||||
these runs not racing each other both rely on `FileLock(settings.MEDIA_LOCK)`,
|
||||
which serializes exports; concurrent same-`--zip-name` runs are out of scope.
|
||||
- The `FileLock(settings.MEDIA_LOCK)` wrapping is unchanged.
|
||||
|
||||
## Testing
|
||||
|
||||
New `documents/export/tests/test_sinks.py`, unit-testing each sink in isolation
|
||||
(pytest classes, factory-boy factories, the `mocker` fixture, `parametrize`, full
|
||||
type annotations; run on the Linux VM):
|
||||
|
||||
- **Round-trip** (both sinks, parametrized): `add_file` + `add_json` + a streamed
|
||||
manifest produce the expected files/entries with correct relative arcnames.
|
||||
- **Folder incremental**: unchanged file is skipped under `compare_checksums` and
|
||||
under `compare_json`; `delete` prunes a snapshot file not written this run and
|
||||
removes emptied directories; without `delete`, stale files remain.
|
||||
- **Zip atomicity**: injecting an exception mid-export (via `mocker`) leaves no
|
||||
`.zip` and no leftover `.zip.tmp`, and does not wipe the destination even with
|
||||
`--delete`; a clean run yields exactly the `.zip`. A nested/prefixed export has
|
||||
directory marker entries; a flat export has none.
|
||||
- **Zip `--delete`**: a clean `--zip --delete` run wipes pre-existing
|
||||
files/dirs in the destination and produces the `.zip` (preserves
|
||||
`test_export_zipped_with_delete`).
|
||||
- **POSIX arcnames**: nested arcnames are stored with forward slashes in both the
|
||||
zip entry names and the manifest values, regardless of host OS (guards the
|
||||
Windows backslash bug).
|
||||
- **`--data-only`**: both sinks produce only `manifest.json` + `metadata.json`,
|
||||
no document files.
|
||||
- **Stream contract**: opening a second concurrent `stream()` is rejected;
|
||||
`add_file`/`add_json` while a stream is open succeed.
|
||||
- **Command guard**: `--zip` with `--compare-checksums` or `--compare-json`
|
||||
raises `CommandError`; `--zip --delete` does **not** error.
|
||||
|
||||
Existing `test_management_exporter.py` and `test_management_importer.py` stay
|
||||
green unchanged — the export's external behavior (layout, manifest, round-trip
|
||||
import, `--zip --delete`, `--data-only`) is preserved.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Behavior drift in the folder path.** The incremental logic is subtle
|
||||
(mtime/size vs checksum, blake2b json compare, empty-dir cleanup). Mitigation:
|
||||
move it verbatim into the sink and lean on the unchanged command-level tests
|
||||
plus new focused sink tests.
|
||||
- **Manifest interleaving in zip mode.** Relies on the spool-to-temp-file
|
||||
decision; the stream contract makes this explicit and the stream-contract test
|
||||
guards it.
|
||||
@@ -1,236 +0,0 @@
|
||||
# Export Zip Compression Control — Design
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Branch base:** `dev`
|
||||
**Status:** Design complete (zstd facts verified on CPython 3.14.3) — **depends on**
|
||||
`2026-06-16-export-sink-architecture-design.md` being implemented first.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
This builds directly on the export sink refactor. It assumes `ZipExportSink`
|
||||
already exists and is the single place that owns `zipfile.ZipFile` creation and
|
||||
entry writes. Do not start this until that refactor has landed; without it, the
|
||||
change would have to touch the command's zip branches again.
|
||||
|
||||
## Problem
|
||||
|
||||
Zip export is hardwired to `ZIP_DEFLATED` at the library default level. Users
|
||||
have no way to trade speed against archive size — a fast `ZIP_STORED` pass for a
|
||||
quick local copy, or a maximal `ZIP_LZMA` pass for the smallest off-site backup.
|
||||
The sink refactor turns "which compression" into a single constructor argument,
|
||||
so exposing it is now a small, isolated change.
|
||||
|
||||
## Goal
|
||||
|
||||
Let the operator choose the zip compression method and level from the CLI, with
|
||||
behavior identical to today when the flags are omitted. All knowledge of
|
||||
compression stays inside `ZipExportSink`; the command only parses flags and maps
|
||||
them to sink arguments.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- `ZipExportSink` gains `compression: int` and `compresslevel: int | None`
|
||||
constructor parameters (default `ZIP_DEFLATED`, `None` → library default),
|
||||
passed straight to `zipfile.ZipFile(...)`.
|
||||
- New `document_exporter` flags: `--zip-compression` and
|
||||
`--zip-compression-level`, valid only with `--zip`.
|
||||
- Validation: method availability, level range per method, and the
|
||||
requires-`--zip` guard.
|
||||
- Import-side: a pre-extract support check in `document_importer` that turns an
|
||||
unsupported codec into a clear `CommandError` (the importer otherwise decompresses
|
||||
transparently via `ZipFile.extractall`).
|
||||
- Docs: add both flags and the zstd-portability caveat to `docs/administration.md`
|
||||
(the `document_exporter` option list, lines ~257-270 and the `-z`/`-zn` section,
|
||||
lines ~328-330). New flags are long-form only (`--zip-compression`,
|
||||
`--zip-compression-level`) — no short aliases, to avoid `-zc`/`-zl` collisions
|
||||
with the existing `-z`/`-zn`.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Compression for any non-zip sink (folder has none; a future S3 sink would
|
||||
handle its own object storage compression separately).
|
||||
- Changing the default. Omitting the flags must produce a byte-compatible-method
|
||||
archive to today's (`ZIP_DEFLATED`, default level).
|
||||
|
||||
## Design
|
||||
|
||||
### `ZipExportSink` changes
|
||||
|
||||
The base sink's signature is `ZipExportSink(target, zip_name, *, delete)`; this
|
||||
adds two keyword-only params after `delete`:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
target: Path,
|
||||
zip_name: str,
|
||||
*,
|
||||
delete: bool = False,
|
||||
compression: int = zipfile.ZIP_DEFLATED,
|
||||
compresslevel: int | None = None,
|
||||
) -> None:
|
||||
...
|
||||
# opened in __enter__:
|
||||
self._zip = zipfile.ZipFile(
|
||||
self._tmp_path,
|
||||
"w",
|
||||
compression=compression,
|
||||
compresslevel=compresslevel,
|
||||
allowZip64=True,
|
||||
)
|
||||
```
|
||||
|
||||
`ZipFile` applies `compression`/`compresslevel` as the default for every
|
||||
`write`/`writestr` (verified: a `ZipFile(..., compression=ZIP_BZIP2)` yields
|
||||
entries with `compress_type == ZIP_BZIP2` without per-call args), so `add_file` /
|
||||
`add_json` / the manifest entry need no changes. Directory marker entries are
|
||||
empty so their compressed payload is zero, but they are still _tagged_ with the
|
||||
chosen `compress_type` — harmless, but tests that read `infolist()` should filter
|
||||
or account for marker entries (see Testing).
|
||||
|
||||
### CLI flags (`document_exporter`)
|
||||
|
||||
- `--zip-compression {stored,deflated,bzip2,lzma}` — and `zstd` **when the
|
||||
runtime supports it** (see below). Maps to the matching `zipfile.ZIP_*`
|
||||
constant. Default `deflated`.
|
||||
- `--zip-compression-level N` — integer. Per-method accepted ranges (verified
|
||||
against the [3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html#zipfile.ZipFile)):
|
||||
- `deflated`: **0–9** (`zlib` also accepts `-1` = "default", identical to
|
||||
omitting the flag / `compresslevel=None`).
|
||||
- `bzip2`: **1–9** (`0` is invalid for bzip2).
|
||||
- `lzma`, `stored`: level has **no effect** — passing `--zip-compression-level`
|
||||
with either is a `CommandError`, not a silent accept (consistent with the
|
||||
base refactor's fail-fast posture).
|
||||
- `zstd`: **-131072 … 22** (the documented commonly-accepted range; the
|
||||
authoritative bounds are
|
||||
`compression.zstd.CompressionParameter.compression_level.bounds()`).
|
||||
|
||||
Default: unset → library default (`compresslevel=None`).
|
||||
|
||||
Both flags require `--zip`; passing either without `--zip` raises a
|
||||
`CommandError`, matching the incremental-flag rule from the base refactor.
|
||||
|
||||
**Why validate up front (not let `zipfile` raise) — verified on 3.14.3:** an
|
||||
invalid level does _not_ fail at `ZipFile(...)` construction — it fails at the
|
||||
**first `write`/`writestr` call**, with an opaque message
|
||||
(`ValueError: Invalid initialization option` for deflated > 9, or
|
||||
`ValueError: compresslevel must be between 1 and 9` for bzip2). Worse, on context
|
||||
exit the half-initialized write handle emits a secondary
|
||||
`AttributeError: '_ZipWriteFile' object has no attribute '_compressor'` during GC
|
||||
finalization, so the user sees stack-trace noise unrelated to the real cause.
|
||||
Up-front validation turns all of that into a single clean `CommandError`.
|
||||
|
||||
### Validation (in `handle()`, before constructing the sink)
|
||||
|
||||
1. **Requires `--zip`.** Either flag without `--zip` → `CommandError`.
|
||||
2. **Method availability — via a named, patchable seam.** Expose a module-level
|
||||
helper `compression_available(method: str) -> bool` that does
|
||||
`try: import bz2 / import lzma / from compression import zstd except ImportError:
|
||||
return False` — **not** `importlib.util.find_spec`, which can report a stdlib
|
||||
C-extension as present when importing it actually fails. `stored`/`deflated`
|
||||
are always available (`zlib` is a hard CPython dependency). For `zstd` the probe
|
||||
must import `compression.zstd` (3.14+), not merely check that
|
||||
`zipfile.ZIP_ZSTANDARD` exists. Making this a named function is also what lets
|
||||
the test patch "method unavailable" with `mocker`. If the chosen method is
|
||||
unavailable, raise a `CommandError` naming the missing capability — `zipfile`
|
||||
itself would otherwise raise a bare `RuntimeError`
|
||||
("Compression requires the (missing) … module").
|
||||
3. **Level range.** Reject an out-of-range `--zip-compression-level` for the
|
||||
chosen method with a clear `CommandError`; reject the flag entirely for
|
||||
`stored`/`lzma` (see above).
|
||||
|
||||
### zstd (Python 3.14+)
|
||||
|
||||
**Verified empirically on CPython 3.14.3** (via `uv run --python 3.14 --no-project`)
|
||||
and against [PEP 784](https://peps.python.org/pep-0784/) +
|
||||
[the 3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html):
|
||||
|
||||
- The compression-method constant is **`zipfile.ZIP_ZSTANDARD`** (added 3.14; its
|
||||
numeric value is `93`). It does **not** exist on < 3.14.
|
||||
- It is backed by the new **`compression.zstd`** stdlib module (PEP 784 added a
|
||||
`compression` namespace package; legacy `bz2`/`lzma`/`zlib` imports are
|
||||
unchanged). `zipfile` raises `RuntimeError` if `compression.zstd` is
|
||||
unavailable when zstd is requested.
|
||||
- Accepted `compresslevel` is **`-131072 … 22`**, confirmed at runtime via
|
||||
`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`.
|
||||
|
||||
Gate everything zstd-related at runtime so nothing is imported or referenced on
|
||||
< 3.14 (the project targets Python ≥ 3.11):
|
||||
|
||||
```python
|
||||
_ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None) # None before 3.14
|
||||
```
|
||||
|
||||
Presence of the _constant_ does not guarantee the _codec_ is usable, so the
|
||||
availability probe (validation step 2) imports `compression.zstd`, not merely
|
||||
checks the constant.
|
||||
|
||||
Keep `zstd` in the `--zip-compression` `choices` **always** (even on < 3.14), and
|
||||
reject it in validation with a friendly "zstd requires Python 3.14+" message. If
|
||||
it were dropped from `choices` on older runtimes, argparse would emit a generic
|
||||
"invalid choice" that reads as though the option never existed — worse UX.
|
||||
|
||||
### Import-side compatibility
|
||||
|
||||
`document_importer` reads zips with `ZipFile(self.source).extractall(...)`
|
||||
(`document_importer.py:453`), which decompresses each entry transparently using
|
||||
whatever method it was stored with — **provided the matching module exists on the
|
||||
importing machine.**
|
||||
|
||||
The failure mode when it doesn't is unfriendly and must be handled: a zstd (or
|
||||
otherwise unsupported) entry raises a bare `NotImplementedError` **per-entry,
|
||||
during `extractall`** — _not_ at `ZipFile(self.source)` open, and `is_zipfile()`
|
||||
still returns true (a zstd archive is a valid zip container). So the importer
|
||||
enters the zip branch, creates its temp dir, may partially extract other entries,
|
||||
then blows up mid-extract with no context. **Mitigation (in scope here):** before
|
||||
extracting, inspect `ZipFile(self.source).infolist()` compress types and, if any
|
||||
is unsupported on this runtime, raise a `CommandError` naming the method and the
|
||||
requirement (e.g. "this archive uses zstd, which needs Python 3.14+") instead of
|
||||
letting `NotImplementedError` escape.
|
||||
|
||||
Per-method summary (document in help text + `administration.md`):
|
||||
|
||||
- `deflated`/`stored`: universally importable.
|
||||
- `bzip2`/`lzma`: importable wherever the `bz2`/`lzma` modules are present
|
||||
(essentially always).
|
||||
- `zstd`: importable only on Python 3.14+. An archive compressed with `zstd` is
|
||||
**not** importable on older runtimes.
|
||||
|
||||
## Testing
|
||||
|
||||
New cases in the sink tests and an export→import round-trip
|
||||
(pytest classes, factory-boy, `mocker`, `parametrize`, typed; run on the VM):
|
||||
|
||||
- **Round-trip per method.** Parametrize over the available methods (skip `zstd`
|
||||
below 3.14, skip `bzip2`/`lzma` if the module is somehow absent): export a
|
||||
small library, import it back, assert documents/manifest match.
|
||||
- **Method is applied.** Assert each written _file_ entry's `compress_type`
|
||||
equals the requested method (read back via `ZipFile.infolist()`), filtering out
|
||||
directory marker entries (which are tagged but empty).
|
||||
- **Level affects size — robustly.** Do **not** compare deflate level 9 vs 1
|
||||
(on small or incompressible fixtures level 9 can equal or slightly exceed level
|
||||
1, causing flaky CI). Instead assert that a compressing method on a
|
||||
moderately-compressible fixture yields a total smaller than `stored`
|
||||
(`ZIP_STORED`), which is a stable invariant.
|
||||
- **Validation.** Each flag without `--zip` → `CommandError`; out-of-range level
|
||||
(`--zip-compression-level 99`) → a clean `CommandError` from validation
|
||||
(asserting we never reach the `writestr` that would raise the masked
|
||||
`ValueError`); `--zip-compression-level` with `stored`/`lzma` → `CommandError`;
|
||||
unavailable method (patch the named availability seam with `mocker`) →
|
||||
`CommandError`; on < 3.14, `--zip-compression zstd` → the friendly
|
||||
"requires 3.14+" `CommandError`.
|
||||
- **Import pre-check.** An archive containing an unsupported compress type
|
||||
produces a `CommandError` from the importer naming the method, not a raw
|
||||
`NotImplementedError` (simulate by patching the importer's support probe).
|
||||
- **Default unchanged.** Omitting both flags yields file entries with
|
||||
`compress_type == ZIP_DEFLATED`, identical to pre-feature behavior.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Foot-gun archives.** A user could produce a `zstd`/`lzma` archive their
|
||||
import target can't read. Mitigation: explicit help text and the import-side
|
||||
notes above; the default stays the universally-readable `deflated`.
|
||||
- **Optional-module assumptions.** Don't assume `bz2`/`lzma` are always compiled
|
||||
in; probe and error clearly. Mitigation: the availability validation step.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Split `documents/views.py` and `documents/serialisers.py` into modules
|
||||
|
||||
## Problem
|
||||
|
||||
`src/documents/views.py` (5,395 lines) and `src/documents/serialisers.py`
|
||||
(3,532 lines) have grown into monolithic files covering every REST resource
|
||||
in the `documents` app: correspondents, tags, document types, storage paths,
|
||||
custom fields, the core document viewset and search, chat, bulk-edit
|
||||
operations, sharing, saved views, tasks, workflows, and system/UI settings.
|
||||
Their size makes them hard to navigate, hard to review incrementally, and
|
||||
increases the chance of unrelated changes colliding in the same file.
|
||||
|
||||
This document specifies splitting both files into packages, one module per
|
||||
domain area, with no behavior change.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No behavior change. Class names, method bodies, and public API responses
|
||||
are unchanged — this is a pure move/reorganize.
|
||||
- No change to `test_views.py` or `test_api_documents.py`. They exercise the
|
||||
moved classes via imports or via the live API; class names and behavior
|
||||
don't change, so they need no edits. Splitting those test files is a
|
||||
separate, later task if desired.
|
||||
- No change to the frontend, migrations, or any other app beyond the three
|
||||
files that import from `documents.views` / `documents.serialisers`
|
||||
(`paperless/urls.py`, `paperless_mail/views.py`,
|
||||
`paperless_mail/serialisers.py`).
|
||||
- This work happens as its own branch/PR against `dev`, after the in-flight
|
||||
`feature-ai-taxonomy-hints-v2` work merges — not layered on top of it.
|
||||
|
||||
## Architecture
|
||||
|
||||
`documents/views.py` becomes the package `documents/views/`, and
|
||||
`documents/serialisers.py` becomes `documents/serialisers/`. Each gets one
|
||||
module per domain area (table below). Neither package's `__init__.py`
|
||||
re-exports its submodules' contents — it stays empty (or a short docstring
|
||||
only). The three external call sites that currently do
|
||||
`from documents.views import X` / `from documents.serialisers import X` are
|
||||
updated to import from the specific submodule instead
|
||||
(`from documents.views.workflows import WorkflowViewSet`, etc.). This avoids
|
||||
adding an indirection layer that could quietly regrow into a second dumping
|
||||
ground, at the cost of touching those three files.
|
||||
|
||||
### Import direction
|
||||
|
||||
`views/*` modules may import from `serialisers/*` modules; `serialisers/*`
|
||||
modules never import from `views/*`. This keeps the dependency graph acyclic
|
||||
by construction — there is no case in the current code where a serializer
|
||||
needs a view.
|
||||
|
||||
Domain module names are the same across both packages (e.g. `bulk_edit.py`
|
||||
exists in both), which makes the natural import `from documents.serialisers.bulk_edit import BulkEditSerializer`
|
||||
inside `documents/views/bulk_edit.py` easy to find, but a view is free to
|
||||
import a serializer from a different domain module when needed (e.g. a
|
||||
`documents.py` view using a `metadata.py` field serializer) — that's a plain
|
||||
cross-module import, not a cycle risk, since the reverse direction never
|
||||
happens.
|
||||
|
||||
## Module breakdown — `documents/views/`
|
||||
|
||||
| Module | Contents |
|
||||
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `base.py` | Shared mixins/helpers: `PassUserMixin`, `BulkPermissionMixin`, `PermissionsAwareDocumentCountMixin`, `DocumentSelectionMixin`, `DocumentOperationPermissionMixin`, `SearchParams`/`SearchResultPage`/`ResolvedRequestDocs`, `_get_tantivy_query_and_mode`, `_get_more_like_id`, `serve_file` |
|
||||
| `index.py` | `IndexView`, `serve_logo` |
|
||||
| `metadata.py` | `CorrespondentViewSet`, `TagViewSet`, `DocumentTypeViewSet`, `StoragePathViewSet`, `CustomFieldViewSet`, `_get_llm_output_language` |
|
||||
| `documents.py` | `EmailDocumentDetailSchema`, `DocumentViewSet`, `UnifiedSearchViewSet` |
|
||||
| `upload.py` | `PostDocumentView` |
|
||||
| `chat.py` | `ChatStreamingSerializer`, `ChatStreamingView` |
|
||||
| `search.py` | `SearchAutoCompleteView`, `GlobalSearchView`, `SelectionDataView`, `StatisticsView` |
|
||||
| `bulk_edit.py` | `BulkEditView`, `RotateDocumentsView`, `MergeDocumentsView`, `DeleteDocumentsView`, `ReprocessDocumentsView`, `EditPdfDocumentsView`, `RemovePasswordDocumentsView`, `BulkEditObjectsView`, `BulkDownloadView` |
|
||||
| `sharing.py` | `ShareLinkViewSet`, `ShareLinkBundleViewSet`, `SharedLinkView` |
|
||||
| `saved_views.py` | `SavedViewViewSet` |
|
||||
| `tasks.py` | `_TasksViewSetSchema`, `TasksViewSet` |
|
||||
| `workflows.py` | `WorkflowTriggerViewSet`, `WorkflowActionViewSet`, `WorkflowViewSet` |
|
||||
| `system.py` | `UiSettingsView`, `RemoteVersionView`, `SystemStatusView`, `TrashView` |
|
||||
| `logs.py` | `LogViewSet` |
|
||||
|
||||
`documents.py` remains the largest module at roughly 1,600 lines
|
||||
(`DocumentViewSet` alone is ~1,300 lines in the current file); every other
|
||||
module is well under 500 lines.
|
||||
|
||||
## Module breakdown — `documents/serialisers/`
|
||||
|
||||
| Module | Contents |
|
||||
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `base.py` | `DynamicFieldsModelSerializer`, `DocumentUpdateFieldsModelSerializer`, `MatchingModelSerializer`, `SetPermissionsMixin`, `SerializerWithPerms`, `SetPermissionsSerializer`, `OwnedObjectSerializer`, `OwnedObjectListSerializer`, `ReadWriteSerializerMethodField`, `DocumentListSerializer`, `DocumentSelectionSerializer`, `SourceModeValidationMixin`, `BasicUserSerializer`, `NotesSerializer` |
|
||||
| `metadata.py` | `CorrespondentSerializer`, `DocumentTypeSerializer`, `DeprecatedColors`, `ColorField`, `TagSerializer`, `CorrespondentField`, `TagsField`, `DocumentTypeField`, `StoragePathField`, `StoragePathSerializer`, `StoragePathTestSerializer`, `CustomFieldSerializer`, `CustomFieldInstanceSerializer`, `validate_documentlink_targets` |
|
||||
| `documents.py` | `DocumentSerializer`, `SearchResultListSerializer`, `SearchResultSerializer`, `DuplicateDocumentSummarySerializer`, `_DocumentVersionInfo`, `DocumentVersionInfoSerializer`, `DocumentVersionSerializer`, `DocumentVersionLabelSerializer`, `_get_viewable_duplicates` |
|
||||
| `upload.py` | `PostDocumentSerializer` |
|
||||
| `saved_views.py` | `SavedViewFilterRuleSerializer`, `SavedViewSerializer` |
|
||||
| `bulk_edit.py` | `RotateDocumentsSerializer`, `MergeDocumentsSerializer`, `EditPdfDocumentsSerializer`, `RemovePasswordDocumentsSerializer`, `DeleteDocumentsSerializer`, `ReprocessDocumentsSerializer`, `BulkEditSerializer`, `BulkDownloadSerializer`, `BulkEditObjectsSerializer` |
|
||||
| `sharing.py` | `EmailSerializer`, `ShareLinkSerializer`, `ShareLinkBundleSerializer` |
|
||||
| `tasks.py` | `TaskSerializerV10`, `TaskSerializerV9`, `TaskSummarySerializer`, `RunTaskSerializer`, `AcknowledgeTasksViewSerializer` |
|
||||
| `workflows.py` | `WorkflowTriggerSerializer`, `WorkflowActionEmailSerializer`, `WorkflowActionWebhookSerializer`, `WorkflowActionSerializer`, `WorkflowSerializer` |
|
||||
| `system.py` | `UiSettingsViewSerializer`, `TrashSerializer` |
|
||||
|
||||
Note: `ChatStreamingSerializer` is defined in `views.py` today (not
|
||||
`serialisers.py`), directly above `ChatStreamingView`. It moves with
|
||||
`ChatStreamingView` into `documents/views/chat.py` rather than into the
|
||||
serialisers package, preserving its current co-location.
|
||||
|
||||
## External call sites to update
|
||||
|
||||
Only three files import from these two modules today, and all move to
|
||||
importing from the specific new submodule:
|
||||
|
||||
- `src/paperless/urls.py` — ~34 `from documents.views import X` lines, one
|
||||
per viewset/view used in URL routing. Each becomes
|
||||
`from documents.views.<domain> import X`.
|
||||
- `src/paperless_mail/views.py` — `from documents.views import PassUserMixin`
|
||||
becomes `from documents.views.base import PassUserMixin`.
|
||||
- `src/paperless_mail/serialisers.py` — `CorrespondentField`,
|
||||
`DocumentTypeField`, `OwnedObjectSerializer`, `TagsField` move to
|
||||
`from documents.serialisers.metadata import CorrespondentField, DocumentTypeField, TagsField`
|
||||
and `from documents.serialisers.base import OwnedObjectSerializer`.
|
||||
|
||||
## Migration order
|
||||
|
||||
1. Split `serialisers.py` into `documents/serialisers/` first — serializers
|
||||
have no dependency on views, so this half can be verified in isolation.
|
||||
Run the full backend test suite after this step.
|
||||
2. Split `views.py` into `documents/views/`, importing from the new
|
||||
`documents/serialisers/*` modules per the table above. Run the full
|
||||
backend test suite.
|
||||
3. Update the three external call sites (`paperless/urls.py`,
|
||||
`paperless_mail/views.py`, `paperless_mail/serialisers.py`).
|
||||
4. Run `ruff check` / `ruff format` and the full backend test suite once
|
||||
more end to end.
|
||||
|
||||
Splitting serialisers before views (rather than in parallel) means step 2
|
||||
can immediately import finished, correctly-located serializer modules
|
||||
instead of guessing at not-yet-final paths.
|
||||
|
||||
## Risks / error handling
|
||||
|
||||
- **Circular imports**: prevented by construction (serialisers never import
|
||||
from views — see Import direction above). If a genuine cross-domain need
|
||||
is discovered during implementation that seems to require a
|
||||
views→views import cycle (e.g. `UnifiedSearchViewSet` extending
|
||||
`DocumentViewSet` from a different module — both already live in
|
||||
`documents.py` so this doesn't arise), resolve it by moving the shared
|
||||
piece to `base.py` rather than introducing a cycle.
|
||||
- **Missed re-export consumers**: verified via a full-repo grep for
|
||||
`from documents.views import` / `from documents.serialisers import` /
|
||||
`documents.views.` / `documents.serialisers.` before considering the split
|
||||
complete, in case something beyond the three known call sites appears
|
||||
(e.g. in a management command or a rarely-run script).
|
||||
- **Silent behavior drift during move**: since this is a pure reorganization,
|
||||
the full test suite passing after each step (rather than only at the end)
|
||||
is the primary safety net; no new tests are required for this refactor
|
||||
itself.
|
||||
|
||||
## Testing
|
||||
|
||||
No new tests. Existing coverage (`test_views.py`, `test_api_documents.py`,
|
||||
and the rest of the `documents` test suite) is run after each migration step
|
||||
per the ordering above, and must pass unchanged — a failure indicates the
|
||||
move altered behavior, not that new coverage is needed.
|
||||
Reference in New Issue
Block a user