paperless-ngx/docs/superpowers/specs/2026-06-16-export-zip-compression-design.md

Export Zip Compression Control — Design

Date: 2026-06-16 Branch base: dev Status: Design complete (zstd facts verified on CPython 3.14.3) — depends on 2026-06-16-export-sink-architecture-design.md being implemented first.

Prerequisite

This builds directly on the export sink refactor. It assumes ZipExportSink already exists and is the single place that owns zipfile.ZipFile creation and entry writes. Do not start this until that refactor has landed; without it, the change would have to touch the command's zip branches again.

Problem

Zip export is hardwired to ZIP_DEFLATED at the library default level. Users have no way to trade speed against archive size — a fast ZIP_STORED pass for a quick local copy, or a maximal ZIP_LZMA pass for the smallest off-site backup. The sink refactor turns "which compression" into a single constructor argument, so exposing it is now a small, isolated change.

Goal

Let the operator choose the zip compression method and level from the CLI, with behavior identical to today when the flags are omitted. All knowledge of compression stays inside ZipExportSink; the command only parses flags and maps them to sink arguments.

Scope

In scope:

• ZipExportSink gains compression: int and compresslevel: int | None constructor parameters (default ZIP_DEFLATED, None → library default), passed straight to zipfile.ZipFile(...).
• New document_exporter flags: --zip-compression and --zip-compression-level, valid only with --zip.
• Validation: method availability, level range per method, and the requires---zip guard.
• Import-side: a pre-extract support check in document_importer that turns an unsupported codec into a clear CommandError (the importer otherwise decompresses transparently via ZipFile.extractall).
• Docs: add both flags and the zstd-portability caveat to docs/administration.md (the document_exporter option list, lines ~257-270 and the -z/-zn section, lines ~328-330). New flags are long-form only (--zip-compression, --zip-compression-level) — no short aliases, to avoid -zc/-zl collisions with the existing -z/-zn.

Out of scope:

• Compression for any non-zip sink (folder has none; a future S3 sink would handle its own object storage compression separately).
• Changing the default. Omitting the flags must produce a byte-compatible-method archive to today's (ZIP_DEFLATED, default level).

Design

ZipExportSink changes

The base sink's signature is ZipExportSink(target, zip_name, *, delete); this adds two keyword-only params after delete:

def __init__(
    self,
    target: Path,
    zip_name: str,
    *,
    delete: bool = False,
    compression: int = zipfile.ZIP_DEFLATED,
    compresslevel: int | None = None,
) -> None:
    ...
    # opened in __enter__:
    self._zip = zipfile.ZipFile(
        self._tmp_path,
        "w",
        compression=compression,
        compresslevel=compresslevel,
        allowZip64=True,
    )

ZipFile applies compression/compresslevel as the default for every write/writestr (verified: a ZipFile(..., compression=ZIP_BZIP2) yields entries with compress_type == ZIP_BZIP2 without per-call args), so add_file / add_json / the manifest entry need no changes. Directory marker entries are empty so their compressed payload is zero, but they are still tagged with the chosen compress_type — harmless, but tests that read infolist() should filter or account for marker entries (see Testing).

CLI flags (document_exporter)

• --zip-compression {stored,deflated,bzip2,lzma} — and zstd when the runtime supports it (see below). Maps to the matching zipfile.ZIP_* constant. Default deflated.

• --zip-compression-level N — integer. Per-method accepted ranges (verified against the 3.14 zipfile docs):

  • deflated: 0–9 (zlib also accepts -1 = "default", identical to omitting the flag / compresslevel=None).
  • bzip2: 1–9 (0 is invalid for bzip2).
  • lzma, stored: level has no effect — passing --zip-compression-level with either is a CommandError, not a silent accept (consistent with the base refactor's fail-fast posture).
  • zstd: -131072 … 22 (the documented commonly-accepted range; the authoritative bounds are compression.zstd.CompressionParameter.compression_level.bounds()).

  Default: unset → library default (compresslevel=None).

Both flags require --zip; passing either without --zip raises a CommandError, matching the incremental-flag rule from the base refactor.

Why validate up front (not let zipfile raise) — verified on 3.14.3: an invalid level does not fail at ZipFile(...) construction — it fails at the first write/writestr call, with an opaque message (ValueError: Invalid initialization option for deflated > 9, or ValueError: compresslevel must be between 1 and 9 for bzip2). Worse, on context exit the half-initialized write handle emits a secondary AttributeError: '_ZipWriteFile' object has no attribute '_compressor' during GC finalization, so the user sees stack-trace noise unrelated to the real cause. Up-front validation turns all of that into a single clean CommandError.

Validation (in handle(), before constructing the sink)

1. Requires --zip. Either flag without --zip → CommandError.
2. Method availability — via a named, patchable seam. Expose a module-level helper compression_available(method: str) -> bool that does try: import bz2 / import lzma / from compression import zstd except ImportError: return False — not importlib.util.find_spec, which can report a stdlib C-extension as present when importing it actually fails. stored/deflated are always available (zlib is a hard CPython dependency). For zstd the probe must import compression.zstd (3.14+), not merely check that zipfile.ZIP_ZSTANDARD exists. Making this a named function is also what lets the test patch "method unavailable" with mocker. If the chosen method is unavailable, raise a CommandError naming the missing capability — zipfile itself would otherwise raise a bare RuntimeError ("Compression requires the (missing) … module").
3. Level range. Reject an out-of-range --zip-compression-level for the chosen method with a clear CommandError; reject the flag entirely for stored/lzma (see above).

zstd (Python 3.14+)

Verified empirically on CPython 3.14.3 (via uv run --python 3.14 --no-project) and against PEP 784 + the 3.14 zipfile docs:

• The compression-method constant is zipfile.ZIP_ZSTANDARD (added 3.14; its numeric value is 93). It does not exist on < 3.14.
• It is backed by the new compression.zstd stdlib module (PEP 784 added a compression namespace package; legacy bz2/lzma/zlib imports are unchanged). zipfile raises RuntimeError if compression.zstd is unavailable when zstd is requested.
• Accepted compresslevel is -131072 … 22, confirmed at runtime via compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22).

Gate everything zstd-related at runtime so nothing is imported or referenced on < 3.14 (the project targets Python ≥ 3.11):

_ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)  # None before 3.14

Presence of the constant does not guarantee the codec is usable, so the availability probe (validation step 2) imports compression.zstd, not merely checks the constant.

Keep zstd in the --zip-compression choices always (even on < 3.14), and reject it in validation with a friendly "zstd requires Python 3.14+" message. If it were dropped from choices on older runtimes, argparse would emit a generic "invalid choice" that reads as though the option never existed — worse UX.

Import-side compatibility

document_importer reads zips with ZipFile(self.source).extractall(...) (document_importer.py:453), which decompresses each entry transparently using whatever method it was stored with — provided the matching module exists on the importing machine.

The failure mode when it doesn't is unfriendly and must be handled: a zstd (or otherwise unsupported) entry raises a bare NotImplementedError per-entry, during extractall — not at ZipFile(self.source) open, and is_zipfile() still returns true (a zstd archive is a valid zip container). So the importer enters the zip branch, creates its temp dir, may partially extract other entries, then blows up mid-extract with no context. Mitigation (in scope here): before extracting, inspect ZipFile(self.source).infolist() compress types and, if any is unsupported on this runtime, raise a CommandError naming the method and the requirement (e.g. "this archive uses zstd, which needs Python 3.14+") instead of letting NotImplementedError escape.

Per-method summary (document in help text + administration.md):

• deflated/stored: universally importable.
• bzip2/lzma: importable wherever the bz2/lzma modules are present (essentially always).
• zstd: importable only on Python 3.14+. An archive compressed with zstd is not importable on older runtimes.

Testing

New cases in the sink tests and an export→import round-trip (pytest classes, factory-boy, mocker, parametrize, typed; run on the VM):

• Round-trip per method. Parametrize over the available methods (skip zstd below 3.14, skip bzip2/lzma if the module is somehow absent): export a small library, import it back, assert documents/manifest match.
• Method is applied. Assert each written file entry's compress_type equals the requested method (read back via ZipFile.infolist()), filtering out directory marker entries (which are tagged but empty).
• Level affects size — robustly. Do not compare deflate level 9 vs 1 (on small or incompressible fixtures level 9 can equal or slightly exceed level 1, causing flaky CI). Instead assert that a compressing method on a moderately-compressible fixture yields a total smaller than stored (ZIP_STORED), which is a stable invariant.
• Validation. Each flag without --zip → CommandError; out-of-range level (--zip-compression-level 99) → a clean CommandError from validation (asserting we never reach the writestr that would raise the masked ValueError); --zip-compression-level with stored/lzma → CommandError; unavailable method (patch the named availability seam with mocker) → CommandError; on < 3.14, --zip-compression zstd → the friendly "requires 3.14+" CommandError.
• Import pre-check. An archive containing an unsupported compress type produces a CommandError from the importer naming the method, not a raw NotImplementedError (simulate by patching the importer's support probe).
• Default unchanged. Omitting both flags yields file entries with compress_type == ZIP_DEFLATED, identical to pre-feature behavior.

Risks

• Foot-gun archives. A user could produce a zstd/lzma archive their import target can't read. Mitigation: explicit help text and the import-side notes above; the default stays the universally-readable deflated.
• Optional-module assumptions. Don't assume bz2/lzma are always compiled in; probe and error clearly. Mitigation: the availability validation step.